AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,367 Commits 1 Branch 0 Tags 27 MiB
4c5f337248
Commit Graph

3654 Commits

Author SHA1 Message Date
senhuang42
74e95c05cc Add ZSTD_SequenceRange to count ranges in array of ZSTD_Sequence 2020-11-16 10:49:16 -05:00
senhuang42
89f3848310 Add support for repcodes 2020-11-16 10:49:16 -05:00
senhuang42
3e930fd044 Code cleanup, add debuglog statments 2020-11-16 10:49:16 -05:00
senhuang42
086513b5b9 Implement first pass at compressSequences() 2020-11-16 10:49:16 -05:00
senhuang42
a9327b1e9b Add initial function prototype for ZSTD_compressSequences_ext (to be renamed later) 2020-11-16 10:33:35 -05:00
senhuang42
9d936d61d2 Reduce number of memcpy() calls 2020-11-13 19:43:30 -05:00
senhuang42
be4ac6c5bc Use existing repcode update function to implement updates 2020-11-12 16:51:12 -05:00
senhuang42
674c9b9235 Add in proper block repcode histories 2020-11-12 15:34:37 -05:00
senhuang42
06c7f14066 Let block reps persist 2020-11-12 12:24:44 -05:00
senhuang42
396275068c Fix incorrect repcode setting 2020-11-12 11:57:01 -05:00
senhuang42
1a8af0de73 Improve unit test 2020-11-12 11:09:09 -05:00
senhuang42
4d4fd2c55f Overhaul repcode handling logic 2020-11-12 10:59:35 -05:00
Yann Collet
69b8361b0c
Merge pull request #2388 from facebook/fix2386 
fix incorrect assert
 2020-11-06 11:38:08 -08:00
sen
f62edf0fe9
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api 
Add enum to define ZSTD_Sequence type and update sequence extraction API
 2020-11-06 13:00:31 -05:00
Yann Collet
95e74616d5 fix multiple minor conversion warnings 
unrelated to #2386, just cleaning up while I'm updating this file ...
 2020-11-06 09:57:05 -08:00
Yann Collet
2769e4d459 fix incorrect assert 
fix #2386, reported by @Neumann-A
 2020-11-06 09:44:04 -08:00
senhuang42
7d1dea070c Update unit tests 2020-11-06 11:10:37 -05:00
senhuang42
779df995c6 Implement mergeGeneratedSequences() 2020-11-06 10:55:46 -05:00
senhuang42
51abd58208 Rename getSequences() to generateSequences() 2020-11-06 10:53:22 -05:00
senhuang42
261ea69661 Add new mergeGeneratedSequences() function 2020-11-06 10:52:34 -05:00
Luke Pitt
eac309c71b Add ZSTD_getDictID_fromCDict function to experimental section 2020-11-04 11:37:37 +00:00
senhuang42
f782cac3d4 Change block delimiter removing to linear time approach 2020-11-02 17:06:20 -05:00
senhuang42
3c9b43da1d Remove trailing comma 2020-11-02 11:53:58 -05:00
senhuang42
3434049c1f Use ZSTD_memmove() instead of memmove() 2020-11-02 11:43:19 -05:00
senhuang42
d4d0346b40 Update name of enum, clarify documentation 2020-11-02 11:38:17 -05:00
senhuang42
e6178f837f Revert unnecessary seqCollector adjustment 2020-11-02 10:59:20 -05:00
senhuang42
e8501e00b8 Fix incorrect index increment in merge algorithm 2020-11-02 10:58:41 -05:00
senhuang42
a36fdada57 Add algorithm to remove all delimiters 2020-11-02 10:46:52 -05:00
senhuang42
435a3a0428 Update seqCollector definition 2020-11-02 10:19:26 -05:00
senhuang42
3327932609 Update ZSTD_getSequences function signature 2020-11-02 10:17:59 -05:00
senhuang42
7397d0102f Add new enum for different sequence formats for ingestion/extraction 2020-11-02 10:15:53 -05:00
Nick Terrell
7205e609a9
Merge pull request #2354 from terrelln/stable-buffer 
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
 2020-10-30 15:06:56 -07:00
sen
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api 
Refine external ZSTD_Sequence API
 2020-10-30 15:47:25 -04:00
Nick Terrell
d4e021fe35 [lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set 
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
 2020-10-30 10:55:34 -07:00
Nick Terrell
24f72789e2 [lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set 
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6bd6b6f7d3 [cwksp] Return NULL when 0 bytes are requested 
This ensures that the buffer is never used.
 2020-10-30 10:55:34 -07:00
Nick Terrell
fcf81cee5e [lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set 
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6d5dc93d4e [lib] Compress directly into output when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
 2020-10-30 10:55:34 -07:00
Nick Terrell
987cb4ca6a [lib] Take the shortcut when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
 2020-10-30 10:55:34 -07:00
Nick Terrell
809b2f2071 [lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2() 
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.

An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
 2020-10-30 10:55:34 -07:00
Nick Terrell
c74be3f6de [lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set 
Adds the validation of the input/output buffers only. They are still
unused.
 2020-10-30 10:55:34 -07:00
Nick Terrell
e3e0775cc8 [API] Add ZSTD_c_stable{In,Out}Buffer parameters 
This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.
 2020-10-30 10:54:39 -07:00
Nick Terrell
e2581d9572 [lib] Set appliedParams in zstdmt mode 
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
 2020-10-30 10:54:38 -07:00
senhuang42
f0da97642a Specify that getSequences() will always emit block boundary sequences 2020-10-30 12:31:17 -04:00
senhuang42
536e89c723 Sequence extractor should update CBlockState 2020-10-30 12:13:19 -04:00
senhuang42
32cac2627a Emit last literals of 0 size as well, to indicate block boundary 2020-10-29 16:41:17 -04:00
senhuang42
69bd5f0654 Correct literalsRead calculation to include longLength 2020-10-29 14:49:37 -04:00
senhuang42
59624f3163 Remove implicit typecast to appease appVeyor windows build 2020-10-28 16:25:09 -04:00
Yann Collet
09e3bb95d2 Merge branch 'dev' into libzstd_autoconf_full 2020-10-28 10:53:08 -07:00
Yann Collet
0adce4631d Merge branch 'libzstd_autoconf_full' of github.com:facebook/zstd into libzstd_autoconf_full 2020-10-28 10:25:55 -07:00
Yann Collet
f6ecf1568f minor Makefile refactor 
hopefully improving readability
 2020-10-28 09:39:15 -07:00
senhuang42
3ed5d053d8 Clarify comments in zstd.h some more 2020-10-28 09:53:09 -04:00
Nick Terrell
599ff58e08
Merge pull request #2339 from terrelln/zstdmt-stability 
Fix zstdmt stability issues and clean up the zstdmt code
 2020-10-27 19:43:13 -07:00
Yann Collet
ceccd7ae2d Merge branch 'dev' into libzstd_autoconf_full 2020-10-27 15:45:30 -07:00
Yann Collet
2d2507b9db
Merge pull request #2374 from bket/portability 
'head -c BYTES' is non-portable
 2020-10-27 14:15:35 -07:00
sen
17b700d78a
Merge pull request #2366 from senhuang42/enable_ldm_by_default 
Enable LDM by default if window size >= 128MB and strategy uses opt parser
 2020-10-27 14:59:28 -04:00
Nick Terrell
0953645837
Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue 
Fix long distance matcher OSS-fuzz issue
 2020-10-27 11:13:03 -07:00
senhuang42
3163909d14 Remove unused variable position 2020-10-27 12:58:12 -04:00
senhuang42
dc448563e9 Add test compatibility with last literals in sequences 2020-10-27 12:35:28 -04:00
Björn Ketelaars
1f661b5f6b 'head -c BYTES' is non-portable 2020-10-27 16:55:23 +01:00
senhuang42
1d221ecc03 Add support for representing last literals in the extracted seqs 2020-10-27 11:19:48 -04:00
senhuang42
9171f920cd Improve documentation of seqStore_t 2020-10-27 10:50:22 -04:00
senhuang42
96b0ff7886 Improve documentation regarding various operations in copyBlockSequences 2020-10-27 10:36:06 -04:00
senhuang42
3a11c7eb03 Modify ZSTD_copyBlockSequences to agree with new API 2020-10-27 10:31:40 -04:00
senhuang42
761f40d1c6 Clarify and modify ZSTD_Sequence definition 2020-10-27 09:41:32 -04:00
Yann Collet
456db0c377 make install only rebuild binaries if they don't exist 
Now `make` followed by `make install` doesn't rebuild binaries

also : only generated target directories if they don't already exist
 2020-10-23 16:46:49 -07:00
Yann Collet
a6ee614a44 make zstd is now differentiated from zstd-nomt 
avoids mixing object files using different flags
 2020-10-23 16:08:21 -07:00
Yann Collet
89b961ea46 simplified silent mode maintenance 2020-10-23 10:41:17 -07:00
Yann Collet
ffe8d9e428 fix partial lib test 2020-10-23 10:27:12 -07:00
Yann Collet
b5d4728713 simplified silent mode 2020-10-23 10:22:52 -07:00
Yann Collet
a7ad05bf57 fixed building libzstd with manual BUILD_DIR 
and when HASH is not found
 2020-10-23 10:14:04 -07:00
Yann Collet
d3f1a9b5bd fix partial-build test 
sometimes, the scope difference is solely determined by the list of source files,
not by the flags.
 2020-10-22 21:36:09 -07:00
Yann Collet
a912ef0952 can integrate later dynamic flags changes 
for example `libzstd-mt` is `differentiated from `libzstd`
 2020-10-22 18:48:06 -07:00
Yann Collet
f90424da2d
Merge pull request #2368 from facebook/progressive_libzstd 
faster rebuild of libzstd
 2020-10-22 17:36:56 -07:00
Yann Collet
ce6cd07c33 updated build documentation 2020-10-22 12:31:23 -07:00
Yann Collet
e3867fb735 fixed libzstd.dll compilation on mingw 
and zstd linking
 2020-10-22 11:52:19 -07:00
Yann Collet
494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet
dd24496951 programs/zstd also automatically generate object dir per conf 
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
 2020-10-21 23:38:33 -07:00
Yann Collet
0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00
Yann Collet
d0436b2a45 automatically detect configuration changes 
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
 2020-10-21 19:22:45 -07:00
Yann Collet
8a453a34c5 automatic %.h header dependency tracking 
also : BUILD_DIR can be manually specified
 2020-10-21 17:25:07 -07:00
Yann Collet
2224ec33ed
Merge pull request #2367 from facebook/progressive_build 
faster rebuild of zstd
 2020-10-21 15:43:14 -07:00
Yann Collet
2b99bc29bf consolidated vpath 2020-10-21 04:01:01 -07:00
Yann Collet
e8eb2939fe store %.o object files into obj/ 
both static and dynamic libraries have their own object directory
 2020-10-21 03:44:38 -07:00
Yann Collet
3e519be965 minor cleaning 2020-10-21 03:22:27 -07:00
Yann Collet
911dbdbb4b build libzstd.so from object files 
%.o object files generated for dynamic library
must be different from those generated for static library.

Due to this difference, %.o were so far only generated for the static library.
The dynamic library was rebuilt from %.c source.

This meant that, for every minor change, the entire dynamic library had to be rebuilt.

This is fixed in this PR :
only the modified %.c source get rebuilt.
 2020-10-20 22:19:57 -07:00
senhuang42
8bdb32aebe Add a function for LDM enable check 2020-10-20 13:46:02 -04:00
senhuang42
578e889ec1 Move ldm enable to compressStream2() 2020-10-20 13:04:45 -04:00
senhuang42
d28d8a1d72 Include LDM tables size for CCtx size estimation where relevant 2020-10-20 09:21:30 -04:00
senhuang42
b1c7fc5768 Add compatibility for multithreading 2020-10-19 12:07:06 -04:00
senhuang42
aad436da37 Document ldm enabled by default in zstd.h 2020-10-19 11:02:29 -04:00
senhuang42
590f7f55f0 Add ldm enable condition in ZSTD_resetCCtx_internal 2020-10-19 10:26:17 -04:00
senhuang42
4d01979b62 Expose and call ZSTD_ldm_skipRawSeqStoreBytes() 2020-10-16 20:30:00 -04:00
Yann Collet
a0ec50c2dc
Merge pull request #2355 from senhuang42/change_ldm_mt_config 
Reduce --long mode MT jobsize at higher levels
 2020-10-16 13:35:50 -07:00
Yann Collet
314c7df170 minor : change test order 
to reduce a warning with `clang` on Windows
 2020-10-16 13:26:47 -07:00
senhuang42
f49926edf4 Change cycleLog adjustment to +3 from +4 2020-10-15 09:56:05 -04:00
senhuang42
ee84817fe7 Reset posInSequence when using ZSTD_referenceExternalSequences() 2020-10-14 22:06:08 -04:00
senhuang42
d0550bb18f Clarify argument names, fix DEBUGLOG() statements 2020-10-14 15:45:43 -04:00
senhuang42
3f99c9b38d Adjust match backwards count args 2020-10-14 15:23:03 -04:00
senhuang42
bf0d559449 Introduce, implement, and call ZSTD_ldm_countBackwardsMatch_2segments() 2020-10-14 12:58:06 -04:00