AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,784 Commits 1 Branch 0 Tags 27 MiB
0783492178
Commit Graph

5784 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
W. Felix Handte
0783492178 Bump Split Log Support to ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
e4ac4a0f16 Support Split Logs in ZSTD_greedy..ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
e710dc3369 Bump Split Log Support to ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
22fcb8d4c7 Support Split Logs in ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
a232b3bb7c Bump Split Log Support to ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
fe96e98f81 Support a Separate Hash Log in ZSTD_fast 2018-09-28 17:12:54 -07:00
W. Felix Handte
bc880ebe8f Stop Passing in hashLog and stepSize to ZSTD_compressBlock_fast_generic 2018-09-28 17:12:54 -07:00
W. Felix Handte
b3107c7799 Temporary Commit to Retain Requested Hash and Chain Logs During Dict Attach 2018-09-28 17:12:54 -07:00
W. Felix Handte
34e0193129 Allow Setting Hash and Chain Logs on Contexts with Attached CDict 2018-09-28 17:12:54 -07:00
W. Felix Handte
eae8232f50 For Supported Strategies, Attach Dict Even When Params Don't Match 2018-09-28 17:12:54 -07:00
W. Felix Handte
01ff945eae Split Attach and Copy Reset Strategies into Separate Implementation Functions 2018-09-28 17:12:54 -07:00
W. Felix Handte
a6d6bbeae1 Pull Attachment Decision into Separate Function 2018-09-28 17:12:54 -07:00
W. Felix Handte
b7fba599ae And Then Avoid the Unused Parameter Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
1f188ae655 Move Asserts into Function to Avoid Unused Function Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
7212b5e5c2 Move Match State CParams Setting into resetCCtx and continueCCtx 2018-09-28 17:12:54 -07:00
W. Felix Handte
01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
W. Felix Handte
50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
W. Felix Handte
14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
W. Felix Handte
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
W. Felix Handte
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
W. Felix Handte
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
Nick Terrell
e06f91a169
Merge pull request #1343 from terrelln/faster-ci 
[zstreamtest] Reduce memory of newapi tests
 2018-09-27 19:29:02 -07:00
Yann Collet
8b86004cff
Merge pull request #1345 from facebook/fixLegacy 
fixed legacy compilation tests
 2018-09-27 19:23:27 -07:00
Yann Collet
ff36513556 fixed longmatch test too 2018-09-27 18:24:41 -07:00
Yann Collet
73773c6b6a fixed legacy compilation tests 
for some reason, these tests started failing recently on CircleCI
 2018-09-27 18:15:14 -07:00
Yann Collet
7318753d47
Merge pull request #1344 from terrelln/grub-fixes 
Fixes for GRUB
 2018-09-27 17:36:54 -07:00
Nick Terrell
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
Nick Terrell
d8c73cd607 Reset number of threads less often 2018-09-27 15:49:31 -07:00
Nick Terrell
aec1a3ec58 Change byte to value to avoid a GRUB typedef 2018-09-27 15:24:48 -07:00
Nick Terrell
109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Nick Terrell
7ee910e86b More aggressive limitations 2018-09-27 13:55:24 -07:00
Nick Terrell
ca0cfa3dbd [zstreamtest] Reduce memory of newapi tests 
We could allocate up to 2^28 bytes of memory when using 2 threads with
window log = 24. Now, we limit it to 2^26 bytes of memory when not running
big tests.

I chose max window log = 22 since that is the maximum source size when
big tests are disabled. Hopefully this will be enough to reduce or
eliminate the test failures.
 2018-09-27 12:48:29 -07:00
Yann Collet
2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd 
fix : huge (>4GB) chain of blocks
 2018-09-27 10:20:14 -07:00
Yann Collet
404a7bfed0 moved again overflow correction 
cannot work from within ZSTD_compressBlock()
 2018-09-26 18:06:53 -07:00
Yann Collet
9bce916732
Merge pull request #1340 from terrelln/generate-artifacts 
Publish artifacts with CircleCI
 2018-09-26 15:52:23 -07:00
Yann Collet
0e2dbac18a changed overflow correction place 
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
 2018-09-26 15:35:38 -07:00
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors 
fixed usage of grep in Makefile
 2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt 
Adaptive compression
 2018-09-26 14:39:08 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks 
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
 2018-09-26 14:24:28 -07:00
Yann Collet
3dae90c4e0
Merge pull request #1338 from facebook/rfc 
updated format documentation
 2018-09-26 13:40:35 -07:00
Yann Collet
65ed6eeefb
Merge pull request #1337 from facebook/test_failure 
fixed `!` tests
 2018-09-26 13:40:20 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout 
Changed default legacy support to v0.5+
 2018-09-26 13:39:05 -07:00
Nick Terrell
3ff6040848 Publish artifacts with CircleCI 
* Updates CircleCI to use workflows.
  We can now specify any number of test jobs to run in parallel.
* Switch the image to `buildpack-deps:trusty` which is only 500 MB
  instead of 7 GB, so that saves 7 minutes to download it if it isn't
  already cached on the host.
* Publish the source tarball and sha256sum as artifacts.
* If the `GITHUB_TOKEN` environment variable is set, we will also
  add the tarball + sha256sum to the tagged release, after manual
  approval.
 2018-09-26 13:23:28 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile 
when terminal uses colors
as suggested by @danielshir (#1294)
 2018-09-25 16:56:53 -07:00
Yann Collet
72a3adf826 updated format documentation 
to match last edits of RFC8478.
 2018-09-25 16:34:26 -07:00