AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,740 Commits 1 Branch 0 Tags 27 MiB
727bb7f090
Commit Graph

681 Commits

Author SHA1 Message Date
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
d6394cc4c3 fixed test-zstd-nolegacy 2017-08-20 10:15:44 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
8049556928 Merge pull request #778 from terrelln/bad-huff 
[libzstd] Fix bug in Huffman decompresser
 2017-08-07 14:05:58 -07:00
Nick Terrell
abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Stella Lau
e1abc2a367 Switch the sleep function to UTIL_sleepMilli 2017-08-07 11:49:13 -07:00
Stella Lau
1e366f9dea Add test for deadlock 2017-08-02 11:27:50 -07:00
Stella Lau
5adceeed01 Allow queueSize=0 in pool.c and update poolTests 2017-07-31 10:10:16 -07:00
Yann Collet
38ba7002f2 fixed minor warning on unused variable in shell function 2017-07-20 18:39:04 -07:00
Yann Collet
5e6c5203f3 fixed fuzzer test for non OS-X platforms 2017-07-20 15:11:56 -07:00
Yann Collet
1ca1288689 added --memtest=# command to fuzzer 
to jump directly to relevant test section
 2017-07-19 16:01:16 -07:00
Yann Collet
44b0838253 Merge pull request #770 from terrelln/test-mode 
[zstdcli] Fix -t in streaming mode
 2017-07-18 15:40:59 -07:00
Nick Terrell
d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Nick Terrell
cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Yann Collet
fa3aa04ccd Merge pull request #761 from paulcruz74/file-rename 
renamed pool.c to poolTests.c
 2017-07-14 09:09:45 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet
ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Yann Collet
4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547 optimized memory usage for ZSTDMT_compress() 
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.

In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).

To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
 2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d fix : custom allocator correctly propagated to child contexts 2017-07-10 14:21:40 -07:00
Yann Collet
ee3423d709 extended fuzzer MT memory tests 2017-07-10 14:09:16 -07:00
Yann Collet
88da8f1816 fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() 
also : compile fuzzer with MT enabled
 2017-07-10 14:02:33 -07:00
Yann Collet
f9524cf366 added --memtest to fuzzer 2017-07-10 13:48:41 -07:00
Yann Collet
e32fb0c1fe added ZSTD_sizeof_CCtx() test 2017-07-10 12:29:57 -07:00
Paul Cruz
89190ef07d renamed pool.c to poolTests.c 2017-07-10 11:32:30 -07:00
Yann Collet
ed0243a63c removed zbufftest from list of all tests 2017-07-07 16:16:14 -07:00
Yann Collet
990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet
7758ed8458 fixed fullbench, part 2 2017-07-06 02:48:00 -07:00
Yann Collet
9b2c1acfc0 fixed fullbench 2017-07-06 02:22:57 -07:00
Yann Collet
27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
Yann Collet
5051dd39ca Merge pull request #743 from facebook/fullbench 
compress_generic() automatic optimization opportunities
 2017-07-03 21:26:38 -07:00
Yann Collet
95c4a6e2c8 Merge pull request #745 from terrelln/libfuzzer 
[fuzz] Add libFuzzer targets
 2017-07-03 15:15:20 -07:00
Nick Terrell
bea0f0cfa0 [fuzz] Move from fuzz/ to tests/fuzz/ 2017-07-03 12:40:12 -07:00
cyan4973
4b26306cb8 blindfix : fullbench's one-time leak, detected by valgrind 2017-07-01 08:03:59 -07:00
cyan4973
c07e43c2b5 added --show-leak-kind=all to valgrind tests 2017-07-01 07:05:11 -07:00
cyan4973
b5bb7c6d95 fixed Visual compilation of fullbench-dll 2017-06-29 19:59:37 -07:00
Yann Collet
e7e5a8cef7 made fullbench compatible with multi-threading 
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
 2017-06-29 18:56:24 -07:00
Yann Collet
afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode 
to avoid requesting too much memory in 32-bits mode during MT tests
 2017-06-29 18:19:09 -07:00
Yann Collet
2e84bec9ac updated fullbench to also measure ZSTD_compress_generic() 
will make it possible to visualize
optimization opportunity for ZSTD_e_end
 2017-06-29 13:03:10 -07:00
Yann Collet
7303ed5d74 minor : specify compression level in tags 2017-06-29 11:30:31 -07:00
Yann Collet
590937df20 Merge pull request #739 from facebook/refPrefix 
ZSTD_refPrefix
 2017-06-29 04:36:03 -07:00
Yann Collet
811deaea6f Merge pull request #736 from terrelln/cover-default-api 
[zdict] Make COVER the default algorithm
 2017-06-28 20:25:36 -07:00
Yann Collet
ad5f29918e Merge pull request #737 from facebook/advancedAPI2 
simplified API for *initStatic*()
 2017-06-28 20:25:22 -07:00
Yann Collet
62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Paul Cruz
7ac4724bd2 removed fnum from DISPLAY statements 2017-06-28 13:00:49 -07:00
Paul Cruz
e667d33b0b fixed generation of buggy test, corrected DISPLAY statements for errors 2017-06-28 12:19:37 -07:00
Yann Collet
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet
1fd2df9c2c fixed zstreamtest 
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
 2017-06-28 09:57:59 -07:00
Yann Collet
2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet
b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet
fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Yann Collet
09a5bbe22e Merge pull request #735 from terrelln/xz-default 
[zstdcli] Support xz by default when liblzma is available
 2017-06-27 09:42:02 -07:00
Nick Terrell
5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Yann Collet
20eeb243d1 Merge pull request #729 from paulcruz74/corpus 
Corpus
 2017-06-26 17:47:28 -07:00
Yann Collet
dde10b23fe refactored ZSTD_estimateDStreamSize() 
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
 2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570 ZSTD_estimateCDictSize_advanced() 
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
 2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
Yann Collet
31af8290d1 ZSTD_estimateCCtx_advanced() 
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.

ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
 2017-06-26 15:52:39 -07:00
Nick Terrell
6aeb50ea0d [zstdcli] Add unlzma and unxz symlinks 2017-06-26 11:24:36 -07:00
Paul Cruz
298fe9f23b added cli tests to test-all 2017-06-23 17:10:41 -07:00
Paul Cruz
3a295a91f8 added additional condition so large offsets into the dictionary are not generated past windowSize 2017-06-23 15:54:51 -07:00
Paul Cruz
2085375816 fixed bug detected by the API test 2017-06-23 13:44:24 -07:00
Paul Cruz
8cd134559d type warnings 2017-06-23 12:00:48 -07:00
Paul Cruz
4219acc60a fixed bus error bug 2017-06-23 11:22:29 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
Yann Collet
32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests 
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
 2017-06-23 00:09:02 -07:00
Paul Cruz
2e8cc6f12a added sizeof for clarity 2017-06-22 15:52:33 -07:00
Paul Cruz
6d7f442120 changed makefile test 2017-06-22 15:44:01 -07:00
Paul Cruz
b325a2e4db changed assignment 2017-06-22 15:36:28 -07:00
Paul Cruz
2935239544 added description for dictionary cli tests 2017-06-22 15:23:59 -07:00
Paul Cruz
97a8f89bec made changes to cli test so it runs properly 2017-06-22 15:13:41 -07:00
Paul Cruz
829eb29033 added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset 2017-06-22 14:43:44 -07:00
Paul Cruz
028fb9d01b added cli test case in Makefile 2017-06-22 13:53:39 -07:00
Paul Cruz
98751f69e7 should be updating seed whenever multiple files are generated 2017-06-22 10:23:36 -07:00
Paul Cruz
84cfa07d2d changed format of command to --use-dict=# 2017-06-22 10:04:14 -07:00
Yann Collet
f99c2c1a22 reduced --no-big-tests even more 
to pass tests on qemu-aarch64
 2017-06-21 23:35:58 -07:00
Paul Cruz
04094f37e9 fixed offset in this case os that it always goes past src start 2017-06-21 18:47:40 -07:00
Yann Collet
49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00
Paul Cruz
0950b3159a more meaningful names for count variables 2017-06-21 18:30:27 -07:00
Paul Cruz
0b6eedeace malloc samples instead of static allocation 2017-06-21 18:24:19 -07:00
Yann Collet
4d3bdcf130 reduced CLevelMax for --no-big-tests 2017-06-21 18:20:02 -07:00
Paul Cruz
52a13abf3f removed comment regarding windowSize requirement 2017-06-21 18:13:52 -07:00
Paul Cruz
32e36c2d9a removed requirement that offset be less than windowSize 2017-06-21 18:00:47 -07:00
Yann Collet
bfc2f00080 --no-big-tests for zstreamtest 
Apply --no-big-tests for tsan tests
 2017-06-21 17:57:14 -07:00
Paul Cruz
eeff2d0e3c added const x2 2017-06-21 17:48:37 -07:00
Paul Cruz
04253e21bb added const 2017-06-21 17:48:01 -07:00
Paul Cruz
2ceaec1aad got rid of unnecessary casting to void* 2017-06-21 17:46:37 -07:00
Paul Cruz
ccae9ec807 added comment addressing use of bits in frame header descriptor 2017-06-21 17:43:21 -07:00
Yann Collet
dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet
c16748b137 added test cases for new command --list 2017-06-21 12:09:53 -07:00
Yann Collet
7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Paul Cruz
db3606e249 added test for files with no frame content size 2017-06-20 17:43:36 -07:00
Paul Cruz
acf0df678c fixed strange spacing 2017-06-20 15:41:43 -07:00
Paul Cruz
69bc4fab25 free the context object once done with decompression 2017-06-20 15:37:14 -07:00
Paul Cruz
ceb4f65620 allocate statically so we don't have to deal with freeing 2017-06-20 15:31:54 -07:00
Paul Cruz
a73c2a444a added tests with null files, changed condition to check that the file is more than 0 bytes 2017-06-20 14:33:08 -07:00
Paul Cruz
58c19b4202 spacing matters for test scripts 2017-06-20 14:14:53 -07:00
Yann Collet
ff8f83bd47 fixed fuzzer test 2017-06-20 12:17:32 -07:00
Paul Cruz
b07d0af2ba added additional tests for error detection 2017-06-20 11:54:44 -07:00
Paul Cruz
43ffcdde40 const dictOffset 2017-06-20 11:27:39 -07:00
Paul Cruz
1e0d125e76 const value 2017-06-20 11:15:26 -07:00
Paul Cruz
0034129270 made changes to accommodate single file use case 2017-06-20 11:07:38 -07:00
Paul Cruz
6a99de2dd8 change test to list info on .zst files 2017-06-20 10:45:06 -07:00
Paul Cruz
36e14b1963 added description to advanced usage, changed to use-dict 2017-06-20 10:34:34 -07:00
Paul Cruz
15102069a1 removed testing code from generation code 2017-06-20 10:25:16 -07:00
Paul Cruz
ff5589c091 added comments for clarity 2017-06-20 10:17:38 -07:00
Paul Cruz
551fb7ca05 changed error condition 2017-06-20 10:14:03 -07:00
Paul Cruz
b8237c6c66 refactoring for readability 2017-06-20 10:12:13 -07:00
Paul Cruz
6f17de2132 changed computation for readability 2017-06-20 09:50:28 -07:00
Paul Cruz
4a4f9ba262 const value that doesn't change 2017-06-19 17:46:21 -07:00
Paul Cruz
3a6856edca changed scope to avoid errors 2017-06-19 17:43:16 -07:00
Paul Cruz
7d2b5613b4 adhere to min dict size 2017-06-19 17:37:14 -07:00
Paul Cruz
ea2af1ffc4 changed dictionary to stay reasonable size during test 2017-06-19 17:34:50 -07:00
Paul Cruz
fdbb07283a added test to runTestMode 2017-06-19 17:29:15 -07:00
Paul Cruz
bdec7b8ce7 fixed for loop declaration, fixed test case 2017-06-19 17:18:43 -07:00
Paul Cruz
23098b53e8 wrote a test for dictionary corpus 2017-06-19 17:13:00 -07:00
Paul Cruz
7a26390989 changing for loop declaration 2017-06-19 16:39:57 -07:00
Paul Cruz
147ef05f12 for loop declaration not allowed 2017-06-19 16:38:51 -07:00
Paul Cruz
791352ad0e scope change 2017-06-19 16:32:38 -07:00
Paul Cruz
365e265a8e changed scope so that warning disappears 2017-06-19 16:28:40 -07:00
Paul Cruz
f4f928595b changed scope to get rid of mixed code/variables warning 2017-06-19 16:27:05 -07:00
Paul Cruz
7ea025cc7d added in struct to keep functions more orderly 2017-06-19 16:23:00 -07:00
Paul Cruz
b91b810c64 made some small changes to dictInfo struct and initDictInfo 2017-06-19 16:05:05 -07:00
Paul Cruz
1ee4f6b36c cleaning up code 2017-06-19 15:55:01 -07:00
Paul Cruz
2a52d72712 added comment to anotate decompressed error code 2017-06-19 14:21:35 -07:00
Paul Cruz
73ebb646d8 small stylistic changes 2017-06-19 14:20:35 -07:00
Paul Cruz
34f4e5cab2 changed if statement styling 2017-06-19 14:07:56 -07:00
Paul Cruz
1fc1a35dc4 added calls to free to match calls to malloc, made some stylistic changes, added init function for new struct 2017-06-19 14:02:27 -07:00
Paul Cruz
8f4fa43fe8 got files to decompress 2017-06-19 13:21:50 -07:00
Yann Collet
688952062b minor declaration statement warning fix 2017-06-19 11:53:42 -07:00
Yann Collet
d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Paul Cruz
da4ddc1f76 fixed decompression so that it uses compressed version 2017-06-19 11:40:45 -07:00
Paul Cruz
33467add60 made changes so that HUF_writeCTable no longer fails 2017-06-19 11:27:50 -07:00
Yann Collet
ed1d039127 newapi fuzzer tests : random flush orders in main loop 2017-06-19 11:07:33 -07:00
Paul Cruz
09ac51b56b small style changes 2017-06-19 09:24:38 -07:00
Yann Collet
f5deae8a67 new api fuzzer tests : frame parameters are randonly set 2017-06-18 23:41:38 -07:00
Yann Collet
ea3630a889 new api fuzzer set : dictionary is randomly set 2017-06-18 23:31:55 -07:00
Yann Collet
8dee0ec99e new api fuzzer tests : compression parameters are randomly set 2017-06-18 23:25:15 -07:00
Yann Collet
bd05b19713 removed macro CLAMP from decodecorpus 
never used,
and would duplicate with CLAMP from zstd_compress.c
 2017-06-16 18:16:27 -07:00
Yann Collet
01743a36e7 fuzzer tests for new API 2017-06-16 17:56:41 -07:00
Paul Cruz
2f30433cd7 fixing general style 2017-06-15 18:06:32 -07:00
Paul Cruz
ddd1ab710c added tests in playTest.sh 2017-06-15 16:53:03 -07:00
Paul Cruz
623b8fce85 add print statements for debugging and change malloc with updated size 2017-06-15 11:37:24 -07:00
Paul Cruz
52934fbfc8 added print statements for debugging, fixed first segfault 2017-06-15 09:47:29 -07:00
Paul Cruz
03a208e27c changed samples to geometric distribution so that it is huffman compressible 2017-06-14 18:06:16 -07:00
Paul Cruz
d93207a79f changed dictionary from using fixed amount of bytes for the header / entropy tables 2017-06-14 17:23:56 -07:00
Paul Cruz
664ed05ff6 changed randomly generated samples to randomly selected from the dictionary content 2017-06-14 16:42:51 -07:00
Paul Cruz
3b0e24c06b added error message to catch case where size not given 2017-06-14 12:04:22 -07:00
Paul Cruz
aa95bc0a5f code refactoring and adding some print statements for debugging 2017-06-14 10:54:47 -07:00
Paul Cruz
07cfc975bb debugging error with finalizeDictionary() not writing dictionary properly 2017-06-13 17:27:02 -07:00
Paul Cruz
8dd621f788 changed makefile to include zdict functions, wrote out dictionary to path 2017-06-13 14:19:35 -07:00
Paul Cruz
f35f252e36 added code to generate dictionary using finalizeDictionary 2017-06-13 11:54:43 -07:00
Paul Cruz
37e1b1488b removed previous double free crash, used new crash function 2017-06-09 13:57:18 -07:00
Paul Cruz
be9b0ae627 added error function for different compilation modes 2017-06-09 13:41:34 -07:00
Paul Cruz
11c3987baf added code to extend the offset when a dictionary is detected 2017-06-09 09:48:56 -07:00
Paul Cruz
125ed59968 made sure dictionary ID was being written in the frame header of each file 2017-06-08 17:27:15 -07:00
Paul Cruz
233ee5334e set the lower bits of frame header descriptor if dictionary is needed 2017-06-08 17:11:33 -07:00
Paul Cruz
c2d909e396 added code for generating dictionary/test files randomly. Still need to make sure dictionary ID matches 2017-06-08 17:06:30 -07:00
Yann Collet
8c910d2097 updated ZSTDMT streaming API 
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
 2017-06-03 01:15:02 -07:00
Yann Collet
cdf7e82222 Added ZSTD_initStaticCDict() 2017-05-25 18:05:49 -07:00
Yann Collet
57827f906f added ZSTD_initStaticDDict() 2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference 
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
 2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet
c7fe262dc9 added ZSTD_initStaticCCtx() 
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
 2017-05-23 13:20:41 -07:00
Yann Collet
60a557e7fd fixing symbol.c test 
I believe it would be better to rely on fuzzer tests compiled with dll
 2017-05-19 11:27:43 -07:00
Yann Collet
fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX 
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
 2017-05-19 10:51:30 -07:00
Yann Collet
009d604e00 ZSTD_compress_generic() supports multiple successive frames 
also : clarified streaming API implementation
 2017-05-19 10:17:59 -07:00
Yann Collet
0bdb575c31 Merge branch 'dev' into advancedAPI2 2017-05-16 16:32:29 -07:00
Yann Collet
2d4d31c18a removed gcc compilation flag -Wbad-function-cast 
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
 2017-05-16 11:34:38 -07:00
Yann Collet
83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Yann Collet
a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1) 
this makes it possible to use `0` to mean:
"do not change strategy"
 2017-05-12 16:29:19 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt 
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
 2017-05-10 16:49:58 -07:00
Yann Collet
30ab64e21d added test for ZSTD_estimateCStreamSize() 2017-05-10 11:30:19 -07:00
Yann Collet
461a9cc7c6 fixed symbols test 2017-05-09 16:53:09 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader 
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
 2017-05-09 15:46:07 -07:00
Yann Collet
fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() 
for clarity
 2017-05-08 18:24:16 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() 
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
 2017-05-08 17:51:49 -07:00
Yann Collet
fc5145955a updated ZSTD_estimateCCtxSize() 
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
 2017-05-08 17:07:59 -07:00
Yann Collet
0be6fd3429 merged CCtx and CStream as a single same object 
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
 2017-05-08 16:08:01 -07:00
Yann Collet
01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Yann Collet
a51cab6e68 Merge pull request #678 from facebook/apiChange 
Breaking API Change around CDict
 2017-04-28 10:02:45 -07:00
Yann Collet
36e79a1a1f tests : -g is part of DEBUGFLAGS 2017-04-27 17:29:40 -07:00
Yann Collet
c6915429f2 shortened Appveyor release tests 
fuzzer supports time suffix `s` for "seconds"
 2017-04-27 16:24:53 -07:00
Yann Collet
a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet
7d283cdfa2 added test for ZSTD_initCStream_usingCDict_advanced() 2017-04-27 14:48:34 -07:00
Yann Collet
2f73427d35 added test for ZSTD_compress_usingCDict_advanced() 2017-04-27 14:39:39 -07:00
Yann Collet
7321345fd2 fixed another VS2008 init error 2017-04-27 14:19:34 -07:00
Yann Collet
f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Sean Purcell
4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Yann Collet
230d7acc7d cli : add support for --threads=# command 
updated documentation
add relevant test case
 2017-04-21 11:38:13 -07:00
Yann Collet
c17e020c9a disable assert when compiling paramgrill 
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
 2017-04-20 12:50:02 -07:00
Yann Collet
2c5514c759 fixed ZSTDMT_initCStream_advanced() 
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
 2017-04-18 22:52:41 -07:00
Yann Collet
0bb381dad8 added test for ZSTD_initCStream_advanced() 
when params.fParams.contentSizeFlags = 1
and pledgedSrcSize = 0
then srcSize should be considered 0 (empty), and not "unknown"
 2017-04-18 15:08:52 -07:00
Yann Collet
4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict() 
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
 2017-04-17 18:29:06 -07:00
Yann Collet
9c4c970399 Merge branch 'dev' into compressionFlow 2017-04-17 16:33:09 -07:00
Sean Purcell
d845dab69c Fix input size too small to trigger zstdmt 2017-04-17 12:10:58 -07:00
Sean Purcell
c424ec2eae Add multithreading tests to playTests.sh 2017-04-17 11:41:46 -07:00
Yann Collet
2afb075382 Merge branch 'dev' into compressionFlow 2017-04-14 15:08:31 -07:00
Yann Collet
f913cbed33 fixed : memory leak in fuzzer test 2017-04-13 22:46:51 -07:00
Yann Collet
20d5e03893 content size is controlled at bufferless level 
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
 2017-04-11 18:34:02 -07:00
Sean Purcell
d37e1df2ab Fix message 2017-04-11 17:33:26 -07:00
Sean Purcell
eb70d219fd Add test of file > 4GB to playTests 2017-04-11 17:15:13 -07:00
Yann Collet
4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() 
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
 2017-04-11 11:59:44 -07:00
Yann Collet
4b987ad8ce Introduce ZSTD_initCStream_internal() 
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
 2017-04-10 17:50:44 -07:00
Yann Collet
1fa3b75369 Merge pull request #648 from facebook/zstdmt 
improved zstreamtest --mt to trap bug #644
 2017-04-07 06:09:52 -07:00
Sean Purcell
1652172b2d Add refuse to overwrite test and fix format 2017-04-06 17:06:30 -07:00
Sean Purcell
9da11c6aae Add test for overwriting read-only 2017-04-06 12:58:49 -07:00
Yann Collet
ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Yann Collet
9631e603ae paramgrill : results from optimizer use cli format 
Best found configuration is displayed
using documented cli advanced parameter syntax.
 2017-04-04 16:54:33 -07:00
Yann Collet
89b32f3db0 fix paramgrill -O# 
ensure proposed config does not require more memory than necessary
 2017-04-04 16:41:11 -07:00
Yann Collet
2724f25567 fixed paramgrill -O# 
find optimal settings for a minimum speed
 2017-04-04 16:31:17 -07:00
Yann Collet
c2007388a5 fixed bench.c : optional advanced parameters applied 
before creating cdict
 2017-04-04 15:35:06 -07:00
Nick Terrell
39a6cc5172 Make ZSTD_compress_usingCDict() respect contentSizeFlag 2017-04-03 21:09:55 -07:00
Nick Terrell
62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet
30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet
6476c51b86 Merge pull request #637 from facebook/zstdmt 
Zstdmt
 2017-03-30 14:18:37 -07:00
Nick Terrell
5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet
1e1e26f2ac fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header 2017-03-29 17:09:59 -07:00
Yann Collet
0ef6803707 added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer 
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
 2017-03-29 16:58:57 -07:00
Yann Collet
41fefd573a Improved speed tests 
Better compatibility with Mac OS-X
Force attribution to a selected core # (Linux)
 2017-03-26 23:52:19 -07:00
Yann Collet
4c41d37fcc changed test for new syntax 
--dictID= and --maxdict=
 2017-03-24 18:36:56 -07:00
Yann Collet
f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00
Sean Purcell
680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Sean Purcell
042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Sean Purcell
1b418bd3da Merge remote-tracking branch 'origin/dev' into appveyor 2017-03-21 10:59:54 -07:00
Sean Purcell
5a2b857029 Update appveyor.yml to have short and long tests 2017-03-20 17:04:44 -07:00
Sean Purcell
c771977489 Fix decodecorpus clang compile errors 2017-03-20 16:02:24 -07:00
Sean Purcell
2f6c7e6a53 Fix windows compiler warnings for decodecorpus 2017-03-20 13:45:12 -07:00
Yann Collet
170861da70 Merge pull request #610 from iburinoc/slowfuzzer 
Strongly limit compression level when source is large for tests running on QEMU
 2017-03-20 13:36:01 -07:00
Sean Purcell
0b0b10ec83 Add --no-big-tests flag for qemu fuzz-tests 2017-03-20 11:32:24 -07:00
Sean Purcell
7ebf2de02d Add ability to strongly limit fuzzer test size with flag 2017-03-20 11:25:00 -07:00
Yann Collet
35c147ab6a Merge pull request #605 from iburinoc/term 
Fix zstd not erroring on compressing to terminal without forceStdout
 2017-03-17 14:02:48 -07:00
Sean Purcell
9a38dfa794 Only run IS_CONSOLE tests with a TTY 2017-03-17 12:32:18 -07:00
Sean Purcell
d973071e90 Add tests for compressed data on console 2017-03-16 16:25:19 -07:00
Sean Purcell
f5e50519e0 Prevent fuzz testers from combining large dicts with high clevel 2017-03-15 15:04:54 -07:00
Yann Collet
3773bd318f Merge pull request #601 from terrelln/nxz 
Add xz and lzma support.
 2017-03-13 22:00:03 -07:00
Nick Terrell
aa8bcf360f Add xz and lzma support. 
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
 2017-03-13 18:45:41 -07:00
Sean Purcell
120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell
7c8f5d5bc7 Make test times overwritable 2017-03-09 16:05:23 -08:00
Sean Purcell
daec40db24 Update .travis.yml and Makefile for medium tests 2017-03-09 16:05:22 -08:00
Nick Terrell
eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Yann Collet
8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00