AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,368 Commits 1 Branch 0 Tags 27 MiB
d533af14e5
Commit Graph

8368 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
senhuang42
a36fdada57 Add algorithm to remove all delimiters 2020-11-02 10:46:52 -05:00
senhuang42
435a3a0428 Update seqCollector definition 2020-11-02 10:19:26 -05:00
senhuang42
3327932609 Update ZSTD_getSequences function signature 2020-11-02 10:17:59 -05:00
senhuang42
7397d0102f Add new enum for different sequence formats for ingestion/extraction 2020-11-02 10:15:53 -05:00
sen
f54109c589
Merge pull request #2378 from senhuang42/free_cress_ptr 
[minor] Pass cRess_t by const ptr instead of by value
 2020-11-01 20:33:25 -05:00
Nick Terrell
37d546c445
Merge pull request #2379 from terrelln/regression-test 
[regression] Updates results.csv & add README
 2020-10-30 15:09:38 -07:00
Nick Terrell
7205e609a9
Merge pull request #2354 from terrelln/stable-buffer 
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
 2020-10-30 15:06:56 -07:00
Nick Terrell
a446fa33dc [regression] Add README explaining the test 2020-10-30 13:55:52 -07:00
Nick Terrell
222916a5d3 [regression] Update results.csv 
https://github.com/facebook/zstd/pull/2339 removes the single-pass zstdmt API.
This changes the compressed size, because we no longer take the # of threads into
account when deciding the job size.
 2020-10-30 13:54:30 -07:00
sen
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api 
Refine external ZSTD_Sequence API
 2020-10-30 15:47:25 -04:00
Nick Terrell
2ebf6d5588 [test] Add unit tests for ZSTD_c_stable{In,Out}Buffer 2020-10-30 10:55:34 -07:00
Nick Terrell
d4e021fe35 [lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set 
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
 2020-10-30 10:55:34 -07:00
Nick Terrell
24f72789e2 [lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set 
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6bd6b6f7d3 [cwksp] Return NULL when 0 bytes are requested 
This ensures that the buffer is never used.
 2020-10-30 10:55:34 -07:00
Nick Terrell
fcf81cee5e [lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set 
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6d5dc93d4e [lib] Compress directly into output when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
 2020-10-30 10:55:34 -07:00
Nick Terrell
987cb4ca6a [lib] Take the shortcut when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
 2020-10-30 10:55:34 -07:00
Nick Terrell
809b2f2071 [lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2() 
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.

An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
 2020-10-30 10:55:34 -07:00
Nick Terrell
c74be3f6de [lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set 
Adds the validation of the input/output buffers only. They are still
unused.
 2020-10-30 10:55:34 -07:00
Nick Terrell
e3e0775cc8 [API] Add ZSTD_c_stable{In,Out}Buffer parameters 
This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.
 2020-10-30 10:54:39 -07:00
Nick Terrell
e2581d9572 [lib] Set appliedParams in zstdmt mode 
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
 2020-10-30 10:54:38 -07:00
senhuang42
f0da97642a Specify that getSequences() will always emit block boundary sequences 2020-10-30 12:31:17 -04:00
senhuang42
536e89c723 Sequence extractor should update CBlockState 2020-10-30 12:13:19 -04:00
senhuang42
32cac2627a Emit last literals of 0 size as well, to indicate block boundary 2020-10-29 16:41:17 -04:00
senhuang42
69bd5f0654 Correct literalsRead calculation to include longLength 2020-10-29 14:49:37 -04:00
sen
ff93440fc6
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase 
Add a test case for LDM + opt parser with small uncompressible block
 2020-10-29 09:32:05 -04:00
senhuang42
7198ebb213 Un-mix declarations and code 2020-10-28 18:51:03 -04:00
senhuang42
59624f3163 Remove implicit typecast to appease appVeyor windows build 2020-10-28 16:25:09 -04:00
senhuang42
60a52c29e6 Add check for allocation 2020-10-28 16:22:22 -04:00
Yann Collet
bb4ea40e4d
Merge pull request #2373 from facebook/libzstd_autoconf_full 
Automatic detection of configuration changes
 2020-10-28 11:57:08 -07:00
Yann Collet
09e3bb95d2 Merge branch 'dev' into libzstd_autoconf_full 2020-10-28 10:53:08 -07:00
Yann Collet
0adce4631d Merge branch 'libzstd_autoconf_full' of github.com:facebook/zstd into libzstd_autoconf_full 2020-10-28 10:25:55 -07:00
Yann Collet
f6ecf1568f minor Makefile refactor 
hopefully improving readability
 2020-10-28 09:39:15 -07:00
senhuang42
3ed5d053d8 Clarify comments in zstd.h some more 2020-10-28 09:53:09 -04:00
Nick Terrell
599ff58e08
Merge pull request #2339 from terrelln/zstdmt-stability 
Fix zstdmt stability issues and clean up the zstdmt code
 2020-10-27 19:43:13 -07:00
Yann Collet
ceccd7ae2d Merge branch 'dev' into libzstd_autoconf_full 2020-10-27 15:45:30 -07:00
Yann Collet
2d2507b9db
Merge pull request #2374 from bket/portability 
'head -c BYTES' is non-portable
 2020-10-27 14:15:35 -07:00
senhuang42
169fc07aa1 Move test to appropriate location 2020-10-27 16:59:43 -04:00
senhuang42
db0b5d7d1e Add test to fuzzer.c 2020-10-27 16:57:24 -04:00
sen
17b700d78a
Merge pull request #2366 from senhuang42/enable_ldm_by_default 
Enable LDM by default if window size >= 128MB and strategy uses opt parser
 2020-10-27 14:59:28 -04:00
Nick Terrell
0953645837
Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue 
Fix long distance matcher OSS-fuzz issue
 2020-10-27 11:13:03 -07:00
senhuang42
3163909d14 Remove unused variable position 2020-10-27 12:58:12 -04:00
senhuang42
dc448563e9 Add test compatibility with last literals in sequences 2020-10-27 12:35:28 -04:00
Björn Ketelaars
1f661b5f6b 'head -c BYTES' is non-portable 2020-10-27 16:55:23 +01:00
senhuang42
1d221ecc03 Add support for representing last literals in the extracted seqs 2020-10-27 11:19:48 -04:00
senhuang42
9171f920cd Improve documentation of seqStore_t 2020-10-27 10:50:22 -04:00
senhuang42
96b0ff7886 Improve documentation regarding various operations in copyBlockSequences 2020-10-27 10:36:06 -04:00
senhuang42
3a11c7eb03 Modify ZSTD_copyBlockSequences to agree with new API 2020-10-27 10:31:40 -04:00
senhuang42
761f40d1c6 Clarify and modify ZSTD_Sequence definition 2020-10-27 09:41:32 -04:00
Yann Collet
59ab4dcd32 Merge branch 'dev' into libzstd_autoconf_full 2020-10-26 15:17:39 -07:00