AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,661 Commits 1 Branch 0 Tags 27 MiB
f8a05932b9
Commit Graph

4661 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
f8a05932b9 added make list capability to tests/Makefile 2018-01-13 21:54:21 -08:00
Yann Collet
9477f6529d
Merge pull request #984 from terrelln/dict-load 
Load more dictionary positions into table if empty
 2018-01-13 13:20:42 -08:00
Nick Terrell
9a211d1f05 Load more dictionary positions into table if empty 
If the hash table is empty load positions into the hash table
that we would otherwise skip.

| Level | Data Set     | Improvement |
|-------|--------------|-------------|
| 1     | github       | 0.44%       |
| 1     | hg-changelog | 0.13%       |
| 1     | hg-commands  | 1.28%       |
| 1     | hg-manifest  | 0.70%       |
| 3     | github       | 0.74%       |
| 3     | hg-changelog | 0.87%       |
| 3     | hg-commands  | 1.74%       |
| 3     | hg-manifest  | 0.23%       |
 2018-01-12 16:17:22 -08:00
Yann Collet
863b2f8db4
Merge pull request #983 from terrelln/dict-wlog 
Increase windowLog from CDict based on the srcSize when known
 2018-01-12 07:47:43 -08:00
Yann Collet
9d65a5c316
Merge pull request #970 from facebook/dubtlazy 
Delayed update Binary Tree for Lazy strategy
 2018-01-12 07:28:59 -08:00
Nick Terrell
4b7c4e5f41 Add test for cdict window log adjustment 2018-01-11 16:45:16 -08:00
Nick Terrell
b610b777d3 Increase windowLog from CDict based on the srcSize when known 2018-01-11 16:23:21 -08:00
Yann Collet
cacf47cbee Merge branch 'dev' into dubtlazy 
and fixed conflicts
 2018-01-11 13:25:08 -08:00
Yann Collet
04c00f9388
Merge pull request #982 from facebook/fix304 
Fix for #304 and #977 : error during dictionary creation
 2018-01-11 13:20:59 -08:00
Yann Collet
b9a14900ff changed function name to ZSTD_DUBT_findBestMatch() 2018-01-11 12:38:31 -08:00
Yann Collet
1d623e60a1
Merge pull request #981 from facebook/fix976 
fixed bug #976, reported by @indygreg
 2018-01-11 11:40:07 -08:00
Yann Collet
752bae4a48 added warning message 
when pathological dataset is detected
(note : cover_optimize needs -v to display the warning)
 2018-01-11 11:29:28 -08:00
Yann Collet
e8093dde09 fixed #304 
Pathological samples may result in literal section being incompressible.
This case is now detected,
and literal distribution is replaced by one that can be written into the dictionary.
 2018-01-11 11:16:32 -08:00
Yann Collet
218e9fe0fc added a test case for dictBuilder failure 
cyclic data set makes the entropy stage fails
now, onto a fix for #304 ...
 2018-01-11 09:42:38 -08:00
Yann Collet
55cdbdb465
Merge pull request #980 from conor42/dev 
Fix break condition in decompression noise test
 2018-01-11 08:33:35 -08:00
Yann Collet
2103a62b3d fixed minor warning on prototype definition 2018-01-11 04:49:19 -08:00
Yann Collet
ff795580f2 fixed bug #976, reported by @indygreg 
constants in zstd.h should not depend on MIN() macro which existence is not guaranteed.

Added a test to check the specific constants.
The test is a bit too specific.
But I have found no way to control a more generic "are all macro already defined" condition,
especially as this is a valid construction (the missing macro might be defined later, intentionnally).
 2018-01-10 20:33:45 -08:00
conor42
0e88f6e97b
Fix break condition in decompression noise test 
The bug prevents noise being added
 2018-01-11 11:42:58 +10:00
Yann Collet
06995775b0
Merge pull request #979 from facebook/memdoc 
Update API documentation related to memory management capabilities
 2018-01-10 17:39:37 -08:00
Yann Collet
1edf33764e
Merge pull request #974 from terrelln/dstfile 
[fileio] Improve safety of output file modifications
 2018-01-10 19:02:48 +01:00
Yann Collet
292eeb672f api doc : grouped all ZSTD_create*_advanced() functions together 
in a new "custom memory allocator" paragraph
which is itself part of "memory management" category.

This makes it simpler to see the relation between the type and its usages.
 2018-01-10 09:07:47 -08:00
Yann Collet
3ea156368c API doc : grouped ZSTD_initStatic*() together 
within "memory management" category.
 2018-01-10 08:49:50 -08:00
Yann Collet
b17fb488b0 fixed msan test 
a pointer calculation was wrong in a corner case
 2018-01-06 20:50:36 +01:00
Yann Collet
76321d6bbc
Merge pull request #965 from facebook/tests_obj 
%.o objects files in /tests
 2018-01-06 13:15:53 +01:00
Yann Collet
658d6b8588 Merge branch 'dev' into dubtlazy 2018-01-06 12:40:58 +01:00
Yann Collet
a927fae2a1 fixed ZSTD_reduceIndex() 
following suggestions from @terrelln.
Also added some comments to present logic behind ZSTD_preserveUnsortedMark().
 2018-01-06 12:31:26 +01:00
Yann Collet
3d578eef0c minor tests/Makefile change, as suggested by @terrelln 
https://github.com/facebook/zstd/pull/965#discussion_r160008006
 2018-01-06 07:13:09 +01:00
Yann Collet
752880ffed
Merge pull request #963 from facebook/benchfix 
fix: bench can accept hlog custom parameter
 2018-01-06 06:57:02 +01:00
Nick Terrell
ed9611dc62 [fileio] Don't call FIO_remove() on stdout or /dev/null 2018-01-05 11:50:24 -08:00
Nick Terrell
282ad05e0a [fileio] Use FIO_remove() everywhere for safety 2018-01-05 11:44:45 -08:00
Nick Terrell
fd63140e1c [util] Refuse to set file stat on non-regular file 2018-01-05 11:44:25 -08:00
Yann Collet
dce386f658
Merge pull request #972 from pixelb/bz1530049 
zstd: fix crash when not overwriting existing files
 2018-01-04 12:14:50 +01:00
Yann Collet
f51e861665
Merge pull request #973 from terrelln/test-case 
Add test case for PR #972
 2018-01-04 12:14:36 +01:00
Nick Terrell
8adebbd0f8 Add test case for PR #972 2018-01-03 15:52:18 -08:00
Pádraig Brady
e0596715dc zstd: fix crash when not overwriting existing files 
This fixes the following crash:
  $ touch exists
  $ programs/zstd -r examples/ -o exists
  zstd: exists already exists; not overwritten
  Segmentation fault (core dumped)

* programs/fileio.c (FIO_compressMultipleFilenames):
Handle the case where we're not overwriting the destination.

Reported at https://bugzilla.redhat.com/1530049
 2018-01-02 15:24:09 +00:00
Yann Collet
2eff217136 updated /lib documentation 2017-12-31 15:50:00 +01:00
Yann Collet
b6887b6d43
Merge pull request #968 from shawnl/dev 
meson: get soversion right
 2017-12-30 16:16:37 +01:00
Yann Collet
00db4dbbb3 fixed minor argument property for Visual 2017-12-30 15:42:28 +01:00
Yann Collet
76e802cde6 Merge branch 'dubtlazy' of github.com:facebook/zstd into dubtlazy 2017-12-30 15:21:18 +01:00
Yann Collet
f597f55675 improved btlazy2 : list of unsorted candidates can reach extDict 
It used to stop on reaching extDict, for simplification.
As a consequence, there was a small loss of performance each time the round buffer would restart from beginning.
It's not a large difference though, just several hundreds of bytes on silesia.
This patch fixes it.
 2017-12-30 15:12:59 +01:00
Yann Collet
a68b76afef updated compression level table for btlazy2 
now selected for levels 13, 14 and 15.

Also : dropped the requirement for monotonic memory budget increase of compression levels,,
which was required for ZSTD_estimateCCtxSize()
in order to ensure that a memory budget for level L is large enough for any level <= L.
This condition is now ensured at run time inside ZSTD_estimateCCtxSize().
 2017-12-30 11:40:35 +01:00
Yann Collet
706b5430bd updated compression level table for btlazy2 
now selected for levels 13, 14 and 15.

Also : dropped the requirement for monotonic memory budget increase of compression levels,,
which was required for ZSTD_estimateCCtxSize()
in order to ensure that a memory budget for level L is large enough for any level <= L.
This condition is now ensured at run time inside ZSTD_estimateCCtxSize().
 2017-12-30 11:37:18 +01:00
Yann Collet
eb52e2f45e simplify ZSTD_preserveUnsortedMark() implementation 
since no compiler attempts to auto-vectorize it.
 2017-12-30 11:13:52 +01:00
Shawn Landden
ea41b580eb meson: allow -Dlegacy_support=true, fix -Dlegacy_support=0 2017-12-29 10:18:33 -08:00
Yann Collet
77fc611abe Merge branch 'benchfix' into dubtlazy 2017-12-29 19:16:58 +01:00
Yann Collet
d228b6b0d0 btlazy2 : optimization for dictionary compression 
we want the dictionary table to be fully sorted,
not just lazily filled.
Dictionary loading is a bit more intensive,
but it saves cpu cycles for match search during compression.
 2017-12-29 19:14:18 +01:00
Yann Collet
02f64ef955 btlazy2: fixed interaction between unsortedMark and reduceTable 2017-12-29 19:08:51 +01:00
Yann Collet
4c7f137bd2 add test case which reliably produces btlazy2 rescale overflow bug 
The unsorted_mark is handled like any index,
which fails after a rescale.
 2017-12-29 17:40:36 +01:00
Yann Collet
64482c2c97 fixed bug in dubt 
the chain of unsorted candidates could grow beyond lowLimit.
 2017-12-29 17:04:37 +01:00
Yann Collet
f36da5b4d9 minor speed optimization : index overflow prevention 
new code supposed to be easier to auto-vectorize
 2017-12-29 14:40:33 +01:00