AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,148 Commits 1 Branch 0 Tags 27 MiB
4b141b63e0
Commit Graph

1252 Commits

Author SHA1 Message Date
Sen Huang
97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang
e1edc554a3 Added 2 unit tests: one for sanity, one for correctnesson fixed dict 2019-11-08 13:57:25 -05:00
Felix Handte
32f989c39a
Merge pull request #1869 from bket/openbsd 
'head -c BYTES' is non-portable.
 2019-11-06 16:54:03 -05:00
Björn Ketelaars
e488cabf25 'date -r FILE' is non-portable. 
date(1) is used to display the last modification time of a file, which
is not supported on OpenBSD, FreeBSD and Darwin. Instead use stat(1).

Tested on OpenBSD.
 2019-11-06 13:55:59 +01:00
Björn Ketelaars
66b6688274 'head -c BYTES' is non-portable. 
Similar to #1321: The '-c' option for 'head' is non-portable (not in
POSIX).
 2019-11-06 11:39:59 +01:00
Nick Terrell
60205fec02 Fix 2 bugs in dictionary loading 
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
 2019-11-01 16:52:07 -07:00
Nick Terrell
75e7c0d107 [fuzz] Add dictionary_loader fuzzer 
* Adds the fuzzer
* Adds an additional `InputType` for the fuzzer

I ran the fuzzer for about 10 minutes and it found 2 bugs:

* Catches the original bug without any help
* Catches an additional bug with 8-byte dictionaries
 2019-11-01 15:54:24 -07:00
Yann Collet
47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes 
Gating named pipe support on windows
 2019-10-30 12:18:58 -07:00
Shashank Tavildar
6d0b7bd6ce Changed extension comparision logic, added new test cases 2019-10-29 13:36:09 -07:00
Shashank Tavildar
9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Shashank Tavildar
5e6dbad6c1 Deleting test directory before creation to pass travis-ci test 2019-10-29 09:54:54 -07:00
Shashank Tavildar
850ba66139 Minor fixes in test 2019-10-28 22:24:01 -07:00
Bimba Shrestha
da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar
0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00
Shashank Tavildar
0e9a37daeb Fixing tests and moving flag to advanced usage 2019-10-28 15:22:26 -07:00
Shashank Tavildar
f422914619 Resolving local merge conflicts 2019-10-28 15:15:13 -07:00
Shashank Tavildar
02433e0b15 Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test 2019-10-28 14:54:54 -07:00
Shashank Tavildar
48f856640e Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-28 13:57:40 -07:00
Yann Collet
b4037b18ef
Merge pull request #1848 from felixhandte/tzst-ext-tmp 
Support Decompressing Short Tar Extensions
 2019-10-28 10:08:20 -07:00
Shashank Tavildar
55ee7d56e4 Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-25 15:49:11 -07:00
Bimba Shrestha
5e59d51991 Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes 2019-10-25 15:43:41 -07:00
Bimba Shrestha
5b45ff4f1c Gating named file support on windows 2019-10-25 11:32:38 -07:00
W. Felix Handte
4eccc82875 Minor Fixes 2019-10-25 13:59:19 -04:00
Yann Collet
c2140e9db0
Merge pull request #1845 from facebook/zbuff 
improve deprecation warning macro
 2019-10-25 09:59:00 -07:00
W. Felix Handte
24499036ba Add Tests 2019-10-24 20:19:10 -04:00
Yann Collet
37f90930da
Merge pull request #1839 from bimbashrestha/named_pipes 
Allowing named pipes to go through in zstdcli
 2019-10-24 10:53:12 -07:00
Yann Collet
63e435dda1 improve deprecation warning macro 
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
 2019-10-23 11:59:32 -07:00
Nick Terrell
3c8ff35013
Merge pull request #1840 from terrelln/regression-fix 
[regression] Fix the old streaming regression test
 2019-10-22 17:58:31 -07:00
Bimba Shrestha
10f7147013 Adding shell test and removing bash test file 2019-10-22 17:09:44 -07:00
Yann Collet
09d0b52998
Merge pull request #1838 from facebook/fixAdvanced 
fix initCStream_advanced() for fast strategies
 2019-10-22 16:22:55 -07:00
Bimba Shrestha
edae3c8f3c Adding bash test for named pipes 2019-10-22 16:15:59 -07:00
Nick Terrell
cf21003995 [regression] Fix the old streaming regression test 
* A copy-paste error made it so we weren't running the advanced/cdict
  streaming tests with the old API.
* Clean up the old streaming tests to skip incompatible configs.
* Update `results.csv`.

The tests now catch the bug in #1787.
 2019-10-22 15:43:17 -07:00
Yann Collet
514f941a70
Merge pull request #1836 from facebook/dirFlat2 
Added --output-dir-flat=
 2019-10-22 15:19:04 -07:00
Yann Collet
5d5c895b18 fix initCStream_advanced() for fast strategies 
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
 2019-10-22 15:01:38 -07:00
Nick Terrell
b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data 
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
 2019-10-21 21:16:17 -07:00
Nick Terrell
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset 
[zstdmt] Don't memset the jobDescription
 2019-10-21 15:53:57 -07:00
Yann Collet
989e0f0cee Added --output-dir-flat= 
which is a preferred construction when applicable.

Also, added --output-dir-flat into man page.
 2019-10-21 14:39:18 -07:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix 
Windows fixes
 2019-10-21 13:45:17 -07:00
Yann Collet
2ad75bb574 validated 'make test' for mingw environment 2019-10-18 15:37:07 -07:00
Yann Collet
85a016ed92 made make check faster 
for shorter feedback loop
 2019-10-18 15:21:50 -07:00
Nick Terrell
243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Yann Collet
29e46ed0bd fix test on windows 
isDirectory() doesn't work on Windows
if directory name is followed by '/'
 2019-10-18 14:28:34 -07:00
Nick Terrell
8c11f089a1 [fuzz] Increase output buffer size of stream_round_trip 
Fixes OSS-Fuzz crash.
Credit to OSS-Fuzz
 2019-10-18 13:39:08 -07:00
Yann Collet
ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Yann Collet
ba7e2b6da7 tests: can override isTerminal with environment variable 2019-10-17 15:07:47 -07:00
Yann Collet
6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet
b3e92ad346
Merge pull request #1812 from senhuang42/output-dir-flat 
Support for --output-dir-flat: output into a directory
 2019-10-14 09:25:05 -07:00
Yann Collet
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api 
Adding api for extracting sequences from seqstore
 2019-10-10 13:11:18 -07:00
Bimba Shrestha
36528b96c4 Manually moving instead of memcpy on decoder and using genBuffer() 2019-10-03 09:26:51 -07:00
Sen Huang
f80437c586 Add support for --output-dir-flat 
New flag to specify output directory destination for multiple files.
 2019-10-02 11:08:20 -04:00