Commit Graph

2587 Commits

Author SHA1 Message Date
Yann Collet
3dc85bae66 minor : fixed zstd-frugal
fixed a minor unused variable warning when compiling zstd-frugal target
2017-02-02 16:19:09 -08:00
Yann Collet
4dfabc45d9 removed unused chart 2017-02-02 13:50:51 -08:00
Yann Collet
c13cd3aa62 updated dictionary compression paragraph 2017-02-02 13:50:07 -08:00
Yann Collet
48bed91606 Merge pull request #527 from facebook/zstdmt
zstdmt refinements
2017-01-31 16:36:46 -08:00
Yann Collet
b2e1b3d670 fixed overlapLog==0 => no overlap 2017-01-30 14:54:46 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
Yann Collet
6ccd37c8d4 cli : added advanced parameter overlapLog
as a hidden (undocumented) parameter for now
2017-01-30 13:07:24 -08:00
Yann Collet
92c98a5b21 zstreamtest uses random section sizes for fuzzing 2017-01-30 12:50:31 -08:00
Yann Collet
cd23dd24af zstreamtest uses random overlapLog for fuzzing 2017-01-30 12:46:35 -08:00
Yann Collet
6be2337c26 added command --block-size=
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
Yann Collet
88df1aed61 changed advanced parameter overlapLog
Follows a positive logic (increasing value => increasing overlap)
which is easier to use
2017-01-30 11:00:00 -08:00
Yann Collet
b5fd15ccb2 fixed : legacy decoders v04 and v05 2017-01-30 10:45:58 -08:00
Yann Collet
cc3d1bc262 Merge pull request #525 from terrelln/covermt
Multithreaded COVER dictionary training
2017-01-30 10:15:33 -08:00
Yann Collet
4fd0fffdea Merge pull request #523 from terrelln/buck
Add BUCK files for Nuclide integration
2017-01-28 19:17:58 -08:00
Nick Terrell
43474313f8 Fix documentation about memory usage 2017-01-27 18:43:05 -08:00
Nick Terrell
90db3afdee Fix typos in VS2010 build config 2017-01-27 17:32:16 -08:00
Yann Collet
64bf8ffce6 report @terrelln patch to ST fuzzer tests 2017-01-27 17:25:07 -08:00
Yann Collet
716ea3b361 Merge pull request #524 from terrelln/asan-failure
Fix segfault in zstreamtest MT
2017-01-27 16:50:16 -08:00
Nick Terrell
b42dd27ef5 Add include guards and extern C 2017-01-27 16:00:19 -08:00
Yann Collet
f6d4a786fc reduced zstdmt latency when using small custom section sizes with high compression levels
Previous version was requiring a fairly large initial amount of input data
before starting to create compression jobs.
This new version starts the process much sooner.
2017-01-27 15:55:30 -08:00
Nick Terrell
d98bf49224 Fix segfault in zstreamtest MT
It was reading beyond the end of the input buffer because no errors were
detected. Once that was fixed, it wasn't making forward progress because
no errors were detected and it was waiting for input.
2017-01-27 15:42:36 -08:00
Nick Terrell
6570167f8b Fix VS fuzzer build configs 2017-01-27 13:56:41 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
Nick Terrell
5cf84a05e7 Revert unnecessary change to Logging.h 2017-01-27 13:26:44 -08:00
Nick Terrell
8959910412 Add threading and pool to build configs 2017-01-27 12:29:27 -08:00
Yann Collet
20bed4210c changed format specification version number 2017-01-27 12:16:16 -08:00
Nick Terrell
c43c27127f Merge branch 'dev' into buck
* dev:
  updated NEWS
  fixed MSAN warnings in legacy decoders
  Fix cmake build
  updated NEWS
  Edits as per comments, and change wildcard 'X' to '?'
  Fix Visual Studios project
  Fix pool.c threading.h import
  Fix zstdmt_compress.h include
  Fixed commented issues
  Updated format specification to be easier to understand
  improved #232 fix
  Fixed https://github.com/facebook/zstd/issues/232
  .travis.yml: different tests for "master" branch
  .travis.yml: optimized order of short tests
  .travis.yml: test jobs 12-15
  JOB_NUMBER -eq 9
  improved ZSTD_compressBlock_opt_extDict_generic
2017-01-27 12:05:48 -08:00
Nick Terrell
2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Yann Collet
609c123a01 Merge pull request #522 from terrelln/benchmt
Fix some includes
2017-01-27 11:40:25 -08:00
Yann Collet
bbfcc24309 updated NEWS 2017-01-27 11:27:34 -08:00
Yann Collet
b9fd442d30 Merge pull request #520 from iburinoc/spec
Updated format specification to be easier to understand
2017-01-27 11:26:38 -08:00
Yann Collet
cafdd31a38 fixed MSAN warnings in legacy decoders
In some extraordinary circumstances,
*Length field can be generated from reading a partially uninitialized memory segment.
Data is correctly identified as corrupted later on,
but the read taints some later pointer arithmetic operation.
2017-01-27 10:44:03 -08:00
Nick Terrell
58f499c41e Clean imports and shorten tests 2017-01-27 10:43:18 -08:00
Nick Terrell
9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski
eb2d23a90c improved #232 fix 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
92a4dbf2e4 Fixed https://github.com/facebook/zstd/issues/232 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
4b66ddea7e .travis.yml: different tests for "master" branch 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
e1ccaa7957 .travis.yml: optimized order of short tests 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
89f74fc0a0 .travis.yml: test jobs 12-15 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
a1cc179669 JOB_NUMBER -eq 9 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
29157320fb improved ZSTD_compressBlock_opt_extDict_generic 2017-01-27 10:43:02 -08:00
Nick Terrell
1f1a336241 Fix cmake build 2017-01-27 10:27:29 -08:00
Yann Collet
122b6aa657 updated NEWS 2017-01-27 03:22:37 -08:00
Sean Purcell
d86153d903 Edits as per comments, and change wildcard 'X' to '?' 2017-01-26 16:58:25 -08:00
Nick Terrell
9aa1aa13c1 Fix Visual Studios project 2017-01-26 16:52:47 -08:00
Nick Terrell
e628eaf87a Fix pool.c threading.h import 2017-01-26 15:29:10 -08:00
Nick Terrell
83c387eb8e Fix zstdmt_compress.h include 2017-01-26 15:25:32 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet
c8a27d9303 Merge pull request #518 from facebook/zstdmt
zstd with multi-threading
2017-01-26 14:22:27 -08:00