AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,884 Commits 1 Branch 0 Tags 27 MiB
e068bd01df
Commit Graph

6884 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Tyler-Tran
c55d2e7ba3 Adding shrinking flag for cover and fastcover (#1656) 
* Changed ERROR(GENERIC) excluding inits

* editing git ignore

* Edited init functions to size_t returns

* moved declarations earlier

* resolved issues with changes to init functions

* fixed style and an error check

* attempting to add tests that might trigger changes

* added && die to cases expecting to fail

* resolved no die on expected failed command

* fixed accel to be incorrect value

* Adding an automated shrinking option

* Fixing build

* finalizing fixes

* fix?

* Removing added comment in cover.h

* Styling fixes

* Merging with fb dev

* removing megic number for default regression

* Requested revisions

* fixing support for fast cover

* fixing casting errors

* parenthesis fix

* fixing some build nits

* resolving travis ci syntax

* might resolve all compilation issues

* removed unused variable

* remodeling the selectDict function

* fixing bad memory access

* fixing error checks

* fixed erroring check in selectDict

* fixing mixed declarations

* modify mixed declaration

* fixing nits and adding test cases

* Adding requested changes + fixed bug for error checking

* switched double comparison from != to <

* fixed declaration typing

* refactoring COVER_best_finish() and changing shrinkDict

* removing the const's

* modifying ZDICT_optimizeTrainFromBuffer_cover functions

* fixing potential bad memcpy

* fixing the error function for dict size
 2019-06-27 16:26:57 -07:00
Ephraim Park
36d0bc2617
Merge pull request #5 from ephiepark/decodecorpus 
Decodecorpus
 2019-06-27 15:02:48 -07:00
Ephraim Park
c7c1ba3a19 Fix a constraint stricter than the spec 2019-06-26 16:43:37 -07:00
Ephraim Park
734eff70b8 enable repeat mode on rle 2019-06-26 16:39:00 -07:00
Yann Collet
621adde3b2 changed naming to ZSTD_indexTooCloseToMax() 
Also : minor speed optimization :
shortcut to ZSTD_reset_matchState() rather than the full reset process.
It still needs to be completed with ZSTD_continueCCtx() for proper initialization.

Also : changed position of LDM hash tables in the context,
so that the "regular" hash tables can be at a predictable position,
hence allowing the shortcut to ZSTD_reset_matchState() without complex conditions.
 2019-06-24 14:39:29 -07:00
Nick Terrell
9038579ab2
Merge pull request #1659 from terrelln/big-dict 
Fix data corruption in niche use case
 2019-06-24 12:40:58 -07:00
Nick Terrell
8e16b872e5 [tests] Add tests for big dictionaries 2019-06-21 17:58:24 -07:00
Yann Collet
45c9fbd6d9 prefer memset() rather than reduceIndex() when close to index range limit 
by disabling continue mode when index is close to limit.
 2019-06-21 16:19:21 -07:00
Yann Collet
944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Nick Terrell
674534a700 [zstd] Fix data corruption in niche use case 
* Extract the overflow correction into a helper function.
* Load the dictionary `ZSTD_CHUNKSIZE_MAX = 512 MB` bytes at a time
  and overflow correct between each chunk.

Data corruption could happen when all these conditions are true:

* You are using multithreading mode
* Your overlap size is >= 512 MB (implies window size >= 512 MB)
* You are using a strategy >= ZSTD_btlazy
* You are compressing more than 4 GB

The problem is that when loading a large dictionary we don't do
overflow correction. We can only load 512 MB at a time, and may
need to do overflow correction before each chunk.
 2019-06-21 15:47:31 -07:00
Nick Terrell
4156060ca4 [zstdmt] Update assert to use ZSTD_WINDOWLOG_MAX 2019-06-21 15:39:33 -07:00
Nick Terrell
95e2b430ea [opt] Add asserts for corruption in ZSTD_updateTree() 2019-06-21 15:22:29 -07:00
Yann Collet
b26818fe73
Merge pull request #1626 from LeeYoung624/dev 
add cmake lz4 support
 2019-06-19 13:35:33 -07:00
Nick Terrell
aeb86b109a
Merge pull request #1655 from terrelln/regression-test 
[regression] Update results for small wlog patch PR#1624
 2019-06-17 11:57:41 -07:00
Nick Terrell
06d5e5ff5a [regression] Update results for small wlog patch PR#1624 2019-06-17 10:40:33 -07:00
Yann Collet
837ff9d206
Merge pull request #1647 from LeeYoung624/cmake_bug_fix 
CMake bug fix: didn't install zstdless and zstdgrep.
 2019-06-14 17:32:05 -07:00
Yann Collet
1166782eb7
Merge pull request #1640 from felixhandte/lstat-macro-guard 
Protect lstat() With Better Macro Guard
 2019-06-14 17:30:03 -07:00
Yann Collet
9af909bf35
Merge pull request #1624 from facebook/smallwlog 
Improves compression ratio for small windowLog
 2019-06-14 17:28:21 -07:00
Yann Collet
018df9c9b1
Merge pull request #1650 from scharan/RemoveExportsForStaticLibrary 
Remove ZSTD_DLL_EXPORT=1 for static lib
 2019-06-14 17:09:00 -07:00
Yann Collet
b0592024e2
Merge pull request #1644 from chungy/chmod_600 
[programs] set chmod 600 after opening destination file
 2019-06-13 17:25:31 -07:00
Sai Charan Koduru
f547dca9e3 Remove ZSTD_DLL_EXPORT=1 for static lib 
As a principle, static libs should not dllexport methods, that should only be used when building DLLs.

Case in point: when static libs with dllexport directives are linked into DLLs created with a .def file, the VC++ compiler exports the dllexported methods into the DLL, in addition to the exports listed in the .def file. This will result in undesired link dependencies and is not the correct thing to do.
 2019-06-12 12:26:36 -07:00
LeeYoung624
a4c0c27410 Install zstdless & zstdgrep as 'PROGRAMS' in CMake 2019-06-12 09:57:28 +08:00
Felix Handte
344f942573
Merge pull request #1620 from michaelforney/test-no-threads 
Skip --adapt and --rsyncable tests when built without thread support
 2019-06-11 15:05:25 -04:00
Nick Terrell
81276c4ee4
Merge pull request #1646 from terrelln/oss-fuzz 
[fuzz] Remove max_len from the options
 2019-06-10 22:13:16 -07:00
Lee
bc400ed13b
Merge pull request #4 from facebook/dev 
merge
 2019-06-11 11:25:26 +08:00
Yann Collet
c601622aa7
Merge pull request #1619 from j301scott/dev 
CMake: Check for existing custom target 'uninstall'
 2019-06-10 11:56:49 -07:00
Nick Terrell
6810dd6191 [fuzz] Remove max_len from the options 2019-06-10 11:05:45 -07:00
W. Felix Handte
d8970199f8 Add Contbuild Test for C99 Build 2019-06-10 11:22:45 -04:00
Felix Handte
e79dbb8074
Merge pull request #1642 from Absotively/dev 
VS2010 project settings improvements
 2019-06-10 10:39:25 -04:00
Mike Swanson
af80f6dfac [programs] Don’t try to chmod a dst file if it can’t be opened 
Repairs an oversight in my last commit, thanks @Cyan4973
 2019-06-09 01:52:45 -07:00
Mike Swanson
3968160a91 [programs] set chmod 600 after opening destination file 
This resolves a race condition where zstd or unzstd may expose read
permissions beyond the original file allowed.  Mode 600 is used
temporarily during the compression and decompression write stage
and the new file inherits the original file’s mode at the end.

Fixes #1630
 2019-06-08 21:54:02 -07:00
Jen Pollock
176a543dde In VS2010+, turn off assembler output for libzstd & libzstd-dll, and don't export functions from libzstd 2019-06-07 17:21:51 -06:00
W. Felix Handte
d2c48042e6 Switch Macro Guarding lstat() 2019-06-07 15:32:28 -04:00
W. Felix Handte
b878bd08dd Clean Up Temp Files Produced By playTests.sh 2019-06-07 15:32:28 -04:00
Nick Terrell
d06c15cc89
Merge pull request #1635 from terrelln/opt-opt 
[libzstd] Optimize ZSTD_insertBt1() for repetitive data
 2019-06-06 13:15:23 -07:00
Nick Terrell
f3800ba073 [regression] Update results.csv 2019-06-05 20:40:52 -07:00
Nick Terrell
cdb9481e38 [libzstd] Optimize ZSTD_insertBt1() for repetitive data 
We would only skip at most 192 bytes at a time before this diff.
This was added to optimize long matches and skip the middle of the
match. However, it doesn't handle the case of repetitive data.

This patch keeps the optimization, but also handles repetitive data
by taking the max of the two return values.

```
> for n in $(seq 9); do echo strategy=$n; dd status=none if=/dev/zero bs=1024k count=1000 | command time -f %U ./zstd --zstd=strategy=$n >/dev/null; done
strategy=1
0.27
strategy=2
0.23
strategy=3
0.27
strategy=4
0.43
strategy=5
0.56
strategy=6
0.43
strategy=7
0.34
strategy=8
0.34
strategy=9
0.35
```

At level 19 with multithreading the compressed size of `silesia.tar` regresses 300 bytes, and `enwik8` regresses 100 bytes.
In single threaded mode `enwik8` is also within 100 bytes, and I didn't test `silesia.tar`.

Fixes Issue #1634.
 2019-06-05 20:34:00 -07:00
Yann Collet
c0eb6c9c81
Merge pull request #1632 from ephiepark/dev 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-05 12:30:21 -07:00
Felix Handte
015985c1f9
Merge pull request #1633 from yk-tanigawa/patch-1 
Update README.md
 2019-06-05 14:55:19 -04:00
Yosuke Tanigawa
f46a3603c9
Update README.md 
Fix minor typo in the documentation.
 2019-06-05 09:11:21 -07:00
Ephraim Park
286394aa9b Merge branch 'dev' of https://github.com/ephiepark/zstd into dev 2019-06-04 21:14:20 -07:00
Ephraim Park
c96535f67e circleci : remove install dependencies command from config.yml 2019-06-04 21:13:40 -07:00
Ephraim Park
3e9572bfa9 circleci : use custom docker image with installations 2019-06-04 21:13:40 -07:00
Ephraim Park
f1bf281715
Merge pull request #2 from ephiepark/circleci 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-04 20:53:29 -07:00
Yann Collet
c6ca65ab78
Merge pull request #1631 from ephiepark/dev 
zstdcli : expose cpu load indicator for each file on -vv mode
 2019-06-04 15:14:35 -07:00
Yann Collet
61d912f3ff
Merge pull request #1629 from facebook/streamSize 
Added comments on I/O buffer sizes for streaming
 2019-06-04 12:38:43 -07:00
Ephraim Park
a6a7f08eaf circleci : remove install dependencies command from config.yml 2019-06-04 12:18:33 -07:00
Ephraim Park
a371df534d circleci : use custom docker image with installations 2019-06-04 11:05:42 -07:00
Yann Collet
b3af1873a0 better title formatting for html documentation 
must pay attention to /** and /*! patterns.
 2019-06-04 10:35:40 -07:00
Yann Collet
b5c98fbfd0 Added comments on I/O buffer sizes for streaming 
It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
 2019-06-04 10:26:16 -07:00