Yann Collet
463a0fe38b
simplified optimal parser
...
removed "cached" structure.
prices are now saved in the optimal table.
Primarily done for simplification.
Might improve speed by a little.
But actually, and surprisingly, also improves ratio in some circumstances.
2018-05-29 14:07:25 -07:00
Yann Collet
bb6eaf6495
Merge pull request #1153 from facebook/dynThreshold
...
changed dynamic fse threshold for offset
2018-05-26 08:43:45 -07:00
Yann Collet
e916c365a1
fixed minor visual warning
2018-05-25 20:43:09 -07:00
Yann Collet
c6ccf435ae
Merge pull request #1152 from facebook/lowCompression
...
btultra accepts blocks with poorer compression ratio
2018-05-25 17:46:11 -07:00
Yann Collet
a7fdceeccd
changed dynamic fse threshold for offset
...
recent experienced showed that
default distribution table for offset
can get it wrong pretty quickly with the nb of symbols,
while it remains a reasonable choice much longer for lengths symbols.
Changed the formula,
so that dynamic threshold is now 32 symbols for offsets.
It remains at 64 symbols for lengths.
Detection based on defaultNormLog
2018-05-25 17:41:16 -07:00
Yann Collet
4b3a36d5d8
Merge branch 'dev' into lowCompression
2018-05-25 15:45:03 -07:00
Yann Collet
744f36aac4
Merge pull request #1141 from facebook/staticDictCost
...
Random stuff on High Compression mode
2018-05-25 15:43:32 -07:00
Yann Collet
5f177f1c53
btultra accepts blocks with poorer compression ratio
...
zstd rejects blocks which do not compress by at least a certain amount.
In which case, such block is simply emitted uncompressed (even if a little bit of compression could be achieved).
This is better for decompression speed, hence for energy.
The logic is controlled by ZSTD_minGain().
The rule is applied uniformly, at all compression levels.
This change makes btultra accepts blocks with poor compression ratios.
We presume that users of btultra mode prefers compression ratio over some decompress speed gains.
The threshold for minimum gain is lowered for btultra
from s>>6 (~1.5% minimum gain)
to s>>7 (~0.8% minimum gain).
This is a prudent change.
Not sure if it's large enough.
2018-05-25 15:19:52 -07:00
Yann Collet
e2c0e3d437
slightly nudge choices towards less sequences
...
also slightly improve some strange detrimental corner cases.
2018-05-25 14:52:21 -07:00
W. Felix Handte
5b292b5685
Check Long + 1 Matches in Both Prefix and Dict in Bothe Short Match Paths
2018-05-25 13:13:57 -04:00
W. Felix Handte
88b733b380
Interleave Prefix and Dict Searches
2018-05-25 13:13:57 -04:00
W. Felix Handte
1850025156
Refactor ZSTD_dfast to Use gotos
2018-05-25 13:13:57 -04:00
W. Felix Handte
43606f9c83
... When I Said "HashTable", I Meant "ChainTable"
2018-05-25 13:13:28 -04:00
W. Felix Handte
ec7efe88f5
Fix Off-By-One Error
2018-05-25 13:13:28 -04:00
W. Felix Handte
2bfe43267e
Disallow Too-Long Repcodes When Using an Attached Dict
2018-05-25 13:13:28 -04:00
W. Felix Handte
b97ad3f457
Port Changes Made to ZSTD_fast to ZSTD_dfast
2018-05-25 13:13:28 -04:00
W. Felix Handte
2313cca1b7
Implement Second Repcode Check
2018-05-25 13:13:28 -04:00
W. Felix Handte
0998f10813
Implement First Repcode Check
2018-05-25 13:13:28 -04:00
W. Felix Handte
50c5b2bb90
Find Dict Hash Table Matches
2018-05-25 13:13:28 -04:00
W. Felix Handte
7a25f7ef5b
Existing Repcode Check Only Applies to noDict Case
2018-05-25 13:13:28 -04:00
W. Felix Handte
8b241da4df
Properly Initialize Repcode Values
2018-05-25 13:13:28 -04:00
W. Felix Handte
7097a03749
Add Necessary Dict Variables
2018-05-25 13:13:28 -04:00
W. Felix Handte
aacbbf4f9a
Rename 'lowest' to 'localLowest' to Prepare to Introduce Dict Indices
2018-05-25 13:13:28 -04:00
W. Felix Handte
c10d1b4011
Skeleton for In-Place Impl for ZSTD_dfast
2018-05-25 13:13:28 -04:00
Yann Collet
f6ad59ab5c
Merge branch 'dev' into staticDictCost
2018-05-24 16:21:02 -07:00
Yann Collet
a7c75740b3
Merge pull request #1149 from terrelln/fuzz-py
...
Small fixes to fuzz.py
2018-05-24 15:56:09 -07:00
Nick Terrell
fdd4d8510f
Improve compiler detection to work on Mac
2018-05-24 14:21:12 -07:00
Nick Terrell
ac852abb8b
Define BIT_DEBUG for --debug
2018-05-24 14:21:12 -07:00
Nick Terrell
2a9975f77b
Increase the maximum file size
2018-05-24 14:21:12 -07:00
Nick Terrell
e712a3a0a3
Small fixes to fuzz.py
2018-05-24 14:21:12 -07:00
Yann Collet
47d2331ed8
Merge pull request #1150 from facebook/fracFse
...
fix corner case when requiring cost of an FSE symbol
2018-05-24 14:19:30 -07:00
Yann Collet
b5ef32fea7
Merge branch 'dev' into fracFse
2018-05-24 14:09:49 -07:00
Yann Collet
776128d16f
fix corner case when requiring cost of an FSE symbol
...
ensure that, when frequency[symbol]==0,
result is (tableLog + 1) bits
with both upper-bit and fractional-bit estimates.
Also : enable BIT_DEBUG in /tests
2018-05-24 13:59:11 -07:00
Yann Collet
08c5be5db3
Merge pull request #1117 from felixhandte/zstd-fast-in-place-dict
...
ZSTD_fast: Support Searching the Dictionary Context In-Place
2018-05-23 19:32:25 -07:00
Nick Terrell
06b70179da
Work around bug in zstd decoder ( #1147 )
...
Work around bug in zstd decoder
Pull request #1144 exercised a new path in the zstd decoder that proved to
be buggy. Avoid the extremely rare bug by emitting an uncompressed block.
2018-05-23 18:02:30 -07:00
Yann Collet
712318a244
Merge pull request #1146 from terrelln/fse-fix
...
[zstd] Fix decompression edge case
2018-05-23 16:41:42 -07:00
Nick Terrell
f2d0924b87
Variable declarations
2018-05-23 14:58:58 -07:00
W. Felix Handte
d9c7e67125
Assert that Dict and Current Window are Adjacent in Index Space
2018-05-23 17:53:03 -04:00
W. Felix Handte
298d24fa57
Make loadedDictEnd an Index, not the Dict Len
2018-05-23 17:53:03 -04:00
W. Felix Handte
7ef85e0618
Fixes in re Comments
2018-05-23 17:53:03 -04:00
W. Felix Handte
582b7f85ed
Don't Attach Empty Dict Contents
...
In weird corner cases, they produce unexpected results...
2018-05-23 17:53:03 -04:00
W. Felix Handte
9c92223468
Avoid Undefined Behavior in Match Ptr Calculation
2018-05-23 17:53:03 -04:00
W. Felix Handte
a44ab3b475
Remove Out-of-Date Comment
2018-05-23 17:53:03 -04:00
W. Felix Handte
95bdf20a87
Moar Renames
2018-05-23 17:53:03 -04:00
W. Felix Handte
7e0402e738
Also Attach Dict When Source Size is Unknown
2018-05-23 17:53:03 -04:00
W. Felix Handte
3ba70cc759
Clear the Dictionary When Sliding the Window
2018-05-23 17:53:03 -04:00
W. Felix Handte
b05ae9b608
Refine ip Initialization to Avoid ARM Weirdness
2018-05-23 17:53:03 -04:00
W. Felix Handte
1a7b34ef28
Use New Index Invariant to Simplify Conditionals
2018-05-23 17:53:03 -04:00
W. Felix Handte
2d598e6fed
Force Working Context Indices Greater than Dict Indices
2018-05-23 17:53:03 -04:00
W. Felix Handte
d005e5daf4
Whitespace Fix
2018-05-23 17:53:03 -04:00
