AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,605 Commits 1 Branch 0 Tags 27 MiB
e8a7f5d3ce
Commit Graph

6605 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
e8a7f5d3ce double-fast: changed the trade-off for a smaller positive change 
same number of complementary insertions, just organized differently
(long at `ip-2`, short at `ip-1`).
 2019-07-12 11:34:53 -07:00
Yann Collet
d1327738c2 updated double_fast complementary insertion 
in a way which is more favorable to compression ratio,
though very slightly slower (~-1%).

More details in the PR.
 2019-07-11 15:25:22 -07:00
Yann Collet
b01c1c679f
Merge pull request #1675 from ephiepark/dev 
Factor out the logic to build sequences
 2019-07-10 13:32:31 -07:00
Yann Collet
2387d574cb updated .gitignore 2019-07-09 14:54:48 -07:00
Yann Collet
34a1a37246 updated .gitignore rule 2019-07-09 14:44:22 -07:00
Yann Collet
8eda16c9f6
Merge pull request #1677 from LeeYoung624/gitignore_fix 
fix gitignore errors
 2019-07-09 14:36:38 -07:00
Yann Collet
b8ec4b0fd6 updated version number (to v1.4.1) 
also : added doc on context re-use, as suggested by @scherepanov at #1676
 2019-07-09 11:43:59 -07:00
LeeYoung624
654cb9d439 fix gitignore errors 2019-07-09 21:08:13 +08:00
Yann Collet
096714d1b8
Merge pull request #1671 from ephiepark/dev 
Adding targetCBlockSize param
 2019-07-03 17:47:44 -07:00
Ephraim Park
f57ac7b09e Factor out the logic to build sequences 2019-07-03 15:42:38 -07:00
Ephraim Park
9007701670 Adding targetCBlockSize param 2019-07-03 15:41:52 -07:00
Nick Terrell
e962f07d19
[fuzz] Add a compression fuzzer with randomly sized output buffer (#1670) 2019-07-02 22:05:07 -07:00
Nick Terrell
6c92ba774e
ZSTD_compressSequences_internal assert op <= oend (#1667) 
When we wrote one byte beyond the end of the buffer for RLE
blocks back in 1.3.7, we would then have `op > oend`. That is
a problem when we use `oend - op` for the size of the destination
buffer, and allows further writes beyond the end of the buffer for
the rest of the function. Lets assert that it doesn't happen.
 2019-07-02 15:45:47 -07:00
Yann Collet
857e608b51
Merge pull request #1658 from facebook/memset 
memset() rather than reduceIndex()
 2019-07-01 15:01:43 -07:00
Yann Collet
4d611ca405
Merge pull request #1664 from ephiepark/dev 
decodecorpus
 2019-07-01 14:13:49 -07:00
Ephraim Park
01f5b5d918
Merge pull request #7 from ephiepark/decodecorpus 
reflect code review comments
 2019-07-01 10:18:59 -07:00
Ephraim Park
28309520c0 reflect code review comments 2019-07-01 10:17:30 -07:00
Yann Collet
068cc18927
Merge pull request #1665 from lzutao/meson-deprecated-warnings 
meson: Fix deprecated build warnings on build options
 2019-06-30 11:36:35 -07:00
Lzu Tao
132a1ad291 meson: Always build gen_html on build machine 
Because we use gen_html as a generator instead of a binary to
run on host machine.
 2019-06-29 02:22:39 +07:00
Lzu Tao
8e590a1af3 meson: Fix deprecated build warnings on build options 
Meson now reserves the `build_` prefix for options
 2019-06-29 02:22:39 +07:00
Lzu Tao
4d156854fe meson: Beautify travis config 2019-06-29 01:42:17 +07:00
Ephraim Park
9f4d71f48d
Merge pull request #6 from ephiepark/decodecorpus 
Add test case for short bistream
 2019-06-27 17:38:50 -07:00
Ephraim Park
01e8384000 Add test case for short bistream 2019-06-27 17:37:39 -07:00
Tyler-Tran
c55d2e7ba3 Adding shrinking flag for cover and fastcover (#1656) 
* Changed ERROR(GENERIC) excluding inits

* editing git ignore

* Edited init functions to size_t returns

* moved declarations earlier

* resolved issues with changes to init functions

* fixed style and an error check

* attempting to add tests that might trigger changes

* added && die to cases expecting to fail

* resolved no die on expected failed command

* fixed accel to be incorrect value

* Adding an automated shrinking option

* Fixing build

* finalizing fixes

* fix?

* Removing added comment in cover.h

* Styling fixes

* Merging with fb dev

* removing megic number for default regression

* Requested revisions

* fixing support for fast cover

* fixing casting errors

* parenthesis fix

* fixing some build nits

* resolving travis ci syntax

* might resolve all compilation issues

* removed unused variable

* remodeling the selectDict function

* fixing bad memory access

* fixing error checks

* fixed erroring check in selectDict

* fixing mixed declarations

* modify mixed declaration

* fixing nits and adding test cases

* Adding requested changes + fixed bug for error checking

* switched double comparison from != to <

* fixed declaration typing

* refactoring COVER_best_finish() and changing shrinkDict

* removing the const's

* modifying ZDICT_optimizeTrainFromBuffer_cover functions

* fixing potential bad memcpy

* fixing the error function for dict size
 2019-06-27 16:26:57 -07:00
Ephraim Park
36d0bc2617
Merge pull request #5 from ephiepark/decodecorpus 
Decodecorpus
 2019-06-27 15:02:48 -07:00
Ephraim Park
c7c1ba3a19 Fix a constraint stricter than the spec 2019-06-26 16:43:37 -07:00
Ephraim Park
734eff70b8 enable repeat mode on rle 2019-06-26 16:39:00 -07:00
Yann Collet
621adde3b2 changed naming to ZSTD_indexTooCloseToMax() 
Also : minor speed optimization :
shortcut to ZSTD_reset_matchState() rather than the full reset process.
It still needs to be completed with ZSTD_continueCCtx() for proper initialization.

Also : changed position of LDM hash tables in the context,
so that the "regular" hash tables can be at a predictable position,
hence allowing the shortcut to ZSTD_reset_matchState() without complex conditions.
 2019-06-24 14:39:29 -07:00
Nick Terrell
9038579ab2
Merge pull request #1659 from terrelln/big-dict 
Fix data corruption in niche use case
 2019-06-24 12:40:58 -07:00
Nick Terrell
8e16b872e5 [tests] Add tests for big dictionaries 2019-06-21 17:58:24 -07:00
Yann Collet
45c9fbd6d9 prefer memset() rather than reduceIndex() when close to index range limit 
by disabling continue mode when index is close to limit.
 2019-06-21 16:19:21 -07:00
Yann Collet
944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Nick Terrell
674534a700 [zstd] Fix data corruption in niche use case 
* Extract the overflow correction into a helper function.
* Load the dictionary `ZSTD_CHUNKSIZE_MAX = 512 MB` bytes at a time
  and overflow correct between each chunk.

Data corruption could happen when all these conditions are true:

* You are using multithreading mode
* Your overlap size is >= 512 MB (implies window size >= 512 MB)
* You are using a strategy >= ZSTD_btlazy
* You are compressing more than 4 GB

The problem is that when loading a large dictionary we don't do
overflow correction. We can only load 512 MB at a time, and may
need to do overflow correction before each chunk.
 2019-06-21 15:47:31 -07:00
Nick Terrell
4156060ca4 [zstdmt] Update assert to use ZSTD_WINDOWLOG_MAX 2019-06-21 15:39:33 -07:00
Nick Terrell
95e2b430ea [opt] Add asserts for corruption in ZSTD_updateTree() 2019-06-21 15:22:29 -07:00
Yann Collet
b26818fe73
Merge pull request #1626 from LeeYoung624/dev 
add cmake lz4 support
 2019-06-19 13:35:33 -07:00
Nick Terrell
aeb86b109a
Merge pull request #1655 from terrelln/regression-test 
[regression] Update results for small wlog patch PR#1624
 2019-06-17 11:57:41 -07:00
Nick Terrell
06d5e5ff5a [regression] Update results for small wlog patch PR#1624 2019-06-17 10:40:33 -07:00
Yann Collet
837ff9d206
Merge pull request #1647 from LeeYoung624/cmake_bug_fix 
CMake bug fix: didn't install zstdless and zstdgrep.
 2019-06-14 17:32:05 -07:00
Yann Collet
1166782eb7
Merge pull request #1640 from felixhandte/lstat-macro-guard 
Protect lstat() With Better Macro Guard
 2019-06-14 17:30:03 -07:00
Yann Collet
9af909bf35
Merge pull request #1624 from facebook/smallwlog 
Improves compression ratio for small windowLog
 2019-06-14 17:28:21 -07:00
Yann Collet
018df9c9b1
Merge pull request #1650 from scharan/RemoveExportsForStaticLibrary 
Remove ZSTD_DLL_EXPORT=1 for static lib
 2019-06-14 17:09:00 -07:00
Yann Collet
b0592024e2
Merge pull request #1644 from chungy/chmod_600 
[programs] set chmod 600 after opening destination file
 2019-06-13 17:25:31 -07:00
Sai Charan Koduru
f547dca9e3 Remove ZSTD_DLL_EXPORT=1 for static lib 
As a principle, static libs should not dllexport methods, that should only be used when building DLLs.

Case in point: when static libs with dllexport directives are linked into DLLs created with a .def file, the VC++ compiler exports the dllexported methods into the DLL, in addition to the exports listed in the .def file. This will result in undesired link dependencies and is not the correct thing to do.
 2019-06-12 12:26:36 -07:00
LeeYoung624
a4c0c27410 Install zstdless & zstdgrep as 'PROGRAMS' in CMake 2019-06-12 09:57:28 +08:00
Felix Handte
344f942573
Merge pull request #1620 from michaelforney/test-no-threads 
Skip --adapt and --rsyncable tests when built without thread support
 2019-06-11 15:05:25 -04:00
Nick Terrell
81276c4ee4
Merge pull request #1646 from terrelln/oss-fuzz 
[fuzz] Remove max_len from the options
 2019-06-10 22:13:16 -07:00
Lee
bc400ed13b
Merge pull request #4 from facebook/dev 
merge
 2019-06-11 11:25:26 +08:00
Yann Collet
c601622aa7
Merge pull request #1619 from j301scott/dev 
CMake: Check for existing custom target 'uninstall'
 2019-06-10 11:56:49 -07:00
Nick Terrell
6810dd6191 [fuzz] Remove max_len from the options 2019-06-10 11:05:45 -07:00