AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,151 Commits 1 Branch 0 Tags 27 MiB
590f7f55f0
Commit Graph

1432 Commits

Author SHA1 Message Date
Yann Collet
f5d5cd3b40
Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser 
Integrate long distance matches into optimal parser
 2020-10-13 13:09:07 -07:00
Nick Terrell
d5c688e8ae Fix ZSTD_adjustCParams_internal() to handle dictionary logic 
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.

Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.

Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.

Adds a simple test to ensure that we aren't downsizing too far.
 2020-10-12 12:50:04 -07:00
Nick Terrell
7083f79008 [bug] Fix dictContentType when reprocessing cdict 
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
 2020-10-12 12:46:10 -07:00
Yann Collet
b951ad20a2
Merge pull request #2329 from senhuang42/prevent_summary_updates_when_using_stdout 
Prevent summary updates when using stdout
 2020-10-09 01:01:36 -07:00
Yann Collet
c3ee284ca2
Merge pull request #2319 from facebook/fullbench_stream2 
update fullbench for compressStream2()
 2020-10-09 00:40:59 -07:00
senhuang42
e96ea5d147 Fix static analyze fuzzer.c error 2020-10-07 13:56:25 -04:00
senhuang42
b8bfc4e63d Add cSize regression test to fuzzer.c 2020-10-07 13:56:25 -04:00
senhuang42
429dec4f42 Add DEBUGLOG() calls in ldm helpers 2020-10-07 13:56:25 -04:00
senhuang42
cfd2aec1b7 Add unit tests into playTests.sh 2020-10-07 13:56:25 -04:00
senhuang42
7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
Nick Terrell
0057c4acf7
Merge pull request #2333 from terrelln/stable-dst 
Reset all decompression parameters in ZSTD_DCtx_reset()
 2020-10-01 18:56:11 -07:00
Nick Terrell
2e7d174130 Reset all decompression parameters in ZSTD_DCtx_reset() 
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
  resetting parameters.
* Add a test case.
 2020-10-01 14:19:21 -07:00
Yann Collet
83461ce963
Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts 
Don't let warning messages consume input from stdin
 2020-09-30 08:26:50 -07:00
senhuang42
9f7212a48b Update unit tests 2020-09-24 16:44:33 -04:00
Yann Collet
c6c0a57c53
Merge pull request #2315 from senhuang42/allow_zstd_suffix 
Support .zstd suffix only for decompression
 2020-09-24 09:44:48 -07:00
senhuang42
21cd640b93 Add unit tests to guard against bad stdin 2020-09-22 14:55:41 -04:00
senhuang42
7aa3da1cd7 Use IS_CONSOLE macro to detect that we're indeed using a console 2020-09-22 14:15:52 -04:00
Nick Terrell
973f2adeec [tests] Don't write to stdout 2020-09-22 00:40:27 -07:00
Yann Collet
5618e000bd update fullbench for compressStream2() 
makes it possible to measure scenarios such as #2314
 2020-09-21 07:19:20 -07:00
Felix Handte
200c960f1d
Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation 
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
 2020-09-18 14:02:14 -04:00
senhuang42
07034952df Add -f to .zstd decompression CLI test 2020-09-18 13:01:45 -04:00
senhuang42
6b6cc80196 Support .zstd suffix only for decompression 2020-09-18 12:49:51 -04:00
W. Felix Handte
9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
W. Felix Handte
f23a321781 Update Regression Test Results 2020-09-17 12:23:05 -04:00
Yann Collet
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads 
Allow environment variable to specify number of threads for compression
 2020-09-14 14:04:19 -07:00
Yann Collet
dec1a78d3e minor fix casting for Visual 2020-09-14 11:46:23 -07:00
Yann Collet
c91a0855f8 check endDirective in ZSTD_compressStream2() 
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
 2020-09-14 10:56:08 -07:00
W. Felix Handte
d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte
6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte
b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte
2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte
b81f3a37f9 Easy: Fix Test 2020-09-10 18:51:52 -04:00
W. Felix Handte
2cf6cfc55f Add Fuzzer Test for the Various Dict Attachment Strategies 2020-09-10 18:51:52 -04:00
Nick Terrell
a90779397a [lib] Reduce zstd stack usage by 1KB 2020-09-09 14:35:39 -07:00
senhuang42
a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42
0a170b20a8 Add ZSTD_NUMTHREADS tests to playTests.sh 2020-09-08 10:34:50 -04:00
senhuang42
3aec385a10 Fix merge conflicts 2020-08-26 15:43:38 -04:00
Yann Collet
a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files 
Refreshing progress bar for processing multiple files
 2020-08-26 11:54:50 -07:00
Nick Terrell
cf83aceaf3
Merge pull request #2282 from terrelln/ncount-fix 
[bug] Fix FSE_readNCount()
 2020-08-26 10:31:07 -07:00
senhuang42
a73e131f10 Adjust playTests.sh refuse overwrite test to include -q 2020-08-26 11:40:05 -04:00
Nick Terrell
ae163015b1 [fuzz] Fix stream_decompress timeouts 2020-08-25 17:13:09 -07:00
Nick Terrell
49eeb2d1fc [fuzz] Disable superblock expansion test 2020-08-25 17:13:06 -07:00
Nick Terrell
4193638996 [bug] Fix FSE_readNCount() 
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
  correctly
 2020-08-25 15:42:41 -07:00
Yann Collet
f82d9865b9
Merge pull request #2278 from senhuang42/ignore_checksum_advanced_param 
New advanced decompression param to ignore checksums
 2020-08-25 12:08:53 -07:00
Nick Terrell
614e446000
Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
senhuang42
dde97de6c4 Only ask to proceed if using --rm, otherwise just display warning. -f bypasses it all. More robust tests 2020-08-24 20:20:39 -04:00
senhuang42
1acf243540 Add a warning whenever (de)compressing multiple files into one source, or into stdout 2020-08-24 19:10:03 -04:00
Nick Terrell
52f33a1da5 Fix compiler warnings 2020-08-24 16:09:45 -07:00
senhuang42
a030560d62 Add new DCtx param: validateChecksum and update unit tests 2020-08-24 17:28:00 -04:00
Nick Terrell
1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00