AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,017 Commits 1 Branch 0 Tags 27 MiB
432186cbea
Commit Graph

8017 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
senhuang42
432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
senhuang42
0e8ac6b995 Add fCtx to FIO_openDstFile() 2020-09-24 15:49:30 -04:00
senhuang42
21cd640b93 Add unit tests to guard against bad stdin 2020-09-22 14:55:41 -04:00
senhuang42
7aa3da1cd7 Use IS_CONSOLE macro to detect that we're indeed using a console 2020-09-22 14:15:52 -04:00
Yann Collet
05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor 
Make multifile (de)compression print out a summary
 2020-09-14 14:06:26 -07:00
Yann Collet
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads 
Allow environment variable to specify number of threads for compression
 2020-09-14 14:04:19 -07:00
Yann Collet
517956e67d
Merge pull request #2306 from facebook/check_endDirective 
check endDirective in ZSTD_compressStream2()
 2020-09-14 13:57:55 -07:00
Yann Collet
dec1a78d3e minor fix casting for Visual 2020-09-14 11:46:23 -07:00
Yann Collet
c91a0855f8 check endDirective in ZSTD_compressStream2() 
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
 2020-09-14 10:56:08 -07:00
Felix Handte
d903b552c8
Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain 
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
 2020-09-11 01:39:23 -04:00
W. Felix Handte
d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte
6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte
b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte
c5fab8848a Document searchFuncs Table 2020-09-10 22:10:02 -04:00
W. Felix Handte
85a95840e4 Further Consolidate Dict Mode Checks 2020-09-10 22:10:02 -04:00
W. Felix Handte
032010fcc1 Improve Documentation Slightly 2020-09-10 22:10:02 -04:00
W. Felix Handte
0faefbf1b3 Make DDSS Selection Override ForceCopy Directive 2020-09-10 22:10:02 -04:00
W. Felix Handte
2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte
efa33861f2 Attempt to Fix MSVC Warnings 2020-09-10 22:10:02 -04:00
W. Felix Handte
ed43832770 Simplify Match Limit Checks 
Seems like a ~1.25% speedup.
 2020-09-10 22:10:02 -04:00
W. Felix Handte
06d240b8a7 Use All Available Space in the Hash Table to Extent Chain Table Reach 
Rather than restrict our temp chain table to 2 ** chainLog entries, this
commit uses all available space to reach further back to gather longer
chains to pack into the DDSS chain table.
 2020-09-10 22:10:02 -04:00
W. Felix Handte
b2b0641ea0 Rewrite Table Fill to Retain Cache Entries Beyond Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte
916238d9dc Avoid Malloc in Table Fill; Pack Tmp Structure into Hash Table 2020-09-10 22:10:02 -04:00
W. Felix Handte
f42c5bddd9 Truncate Chain at Last Possible Attempt 
Make the chain table denser?
 2020-09-10 22:10:02 -04:00
W. Felix Handte
20a020edbc Prefetch Chain Table Matches 2020-09-10 22:10:02 -04:00
W. Felix Handte
9b9feb84f2 Lay Out Chain Table Chains Contiguously 
Rather than interleave all of the chain table entries, tying each entry's
position to the corresponding position in the input, this commit changes the
layout so that all the entries in a single chain are laid out next to each
other. The last entry in the hash table's bucket for this hash is now a packed
pointer of position + length of this chain.

This cannot be merged as written, since it allocates temporary memory inside
ZSTD_dedicatedDictSearch_lazy_loadDictionary().
 2020-09-10 22:10:02 -04:00
W. Felix Handte
66509c7bf4 Only Insert Positions Inside the Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte
13c5ec3e41 Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk 
The load algorithm requires we do it all in one go.
 2020-09-10 22:10:02 -04:00
Felix Handte
005ceaa052
Merge pull request #2276 from felixhandte/dedicated-dict-search-structure 
DDSS For Lazy: Implement a Dedicated Dictionary Hash Table
 2020-09-10 22:09:25 -04:00
W. Felix Handte
07793547e6 Fix Bug: Only Use DDSS Insertion on CDict MatchStates 
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
d214d8c859 Shorten Dict Mode Conditionals in Order to Improve Readability 2020-09-10 18:51:52 -04:00
W. Felix Handte
f49c1563ff Force-Inline ZSTD_insertAndFindFirstIndex_internal() 
Without this, gcc was declining to inline the function in `ZSTD_noDict` mode,
resulting in a ~10% slowdown.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
cab86b074f Clean Up Search Function Selection 2020-09-10 18:51:52 -04:00
W. Felix Handte
2ffbde0d95 Fix -Wshorten-64-to-32 Error 2020-09-10 18:51:52 -04:00
W. Felix Handte
7b5d2f72ea Adjust Working Context Table Sizes Back Down 2020-09-10 18:51:52 -04:00
W. Felix Handte
c09454e28f Add Warning Comment to ZSTD_createCDict_advanced2() Declaration 2020-09-10 18:51:52 -04:00
W. Felix Handte
d332f57897 Permit Matching Against Lowest Valid Position 
This comparison was previously faulty: the lowest valid position is itself
valid, and we should therefore be allowed to match against it.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
a3659fe1ef Make ZSTD_dedicatedDictSearch_getCParams Wrap ZSTD_getCParams 
Fixes up bounds-checking, and lets us clean up what is at the moment an
unnecessary duplication of the default cparams tables.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
b81f3a37f9 Easy: Fix Test 2020-09-10 18:51:52 -04:00
W. Felix Handte
7b9a755ac9 Remove Chain Limit on Hash Cache Entries; Slightly Improve Compression 
Entries in the hashTable chain cache aren't subject to the same aliasing that
the circular chain table is subject to. As such, we don't need to stop when we
cross the chain limit. We can delve deeper. :)
 2020-09-10 18:51:52 -04:00
W. Felix Handte
e8b4011b52 Split Lookups in Hash Cache and Chain Table into Two Loops 
Sliiiight speedup.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
9e83c782f8 Simplify DDS Hash Table Construction 
No need to walk the chainTable; we can just keep shifting the entries in the
hashTable.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
ad9f98ac3f Document the ZSTD_c_enableDedicatedDictSearch Parameter 2020-09-10 18:51:52 -04:00
W. Felix Handte
5390fee4f7 Rename and Move DD_BLOG Constant to ZSTD_LAZY_DDSS_BUCKET_LOG 2020-09-10 18:51:52 -04:00
W. Felix Handte
5e91ae27eb Prefetch First Batch of Match Positions; +11% Speed in Level 5 w/ 1 Dict 2020-09-10 18:51:52 -04:00
W. Felix Handte
df386b3d8d Fix Off-By-One Error in Counting DDS Search Attempts 
This caused us to double-search the first position and fail to search the
last position in the chain, slowing down search and making it less effective.
 2020-09-10 18:51:52 -04:00
W. Felix Handte
d46306087b Enable Dedicated Dict Search in the CLI 2020-09-10 18:51:52 -04:00
W. Felix Handte
914bfe7ee4 Init CCtx's Local Dict with CCtxParams 2020-09-10 18:51:52 -04:00
W. Felix Handte
db2aa25252 Decision for Whether to Attach Should be Based on CDict Config, not CCtx 2020-09-10 18:51:52 -04:00
W. Felix Handte
a494111385 Move Prefetch Before Insertion; Speed Up ~6% 2020-09-10 18:51:52 -04:00