Yann Collet
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
Yann Collet
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
Yann Collet
ed0243a63c
removed zbufftest from list of all tests
2017-07-07 16:16:14 -07:00
Yann Collet
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
Yann Collet
842644e42e
target gpptest uses CXX environment variable
2017-07-07 15:55:41 -07:00
Yann Collet
d4dcace34b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-07 15:53:20 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Yann Collet
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
Yann Collet
49d3d4f484
Merge pull request #751 from ivuk/fix-typo
...
Fix typos in README.md
2017-07-06 12:06:43 -07:00
Igor Vuk
e6e25c9507
Fix typos in README.md
2017-07-06 20:43:14 +02:00
Yann Collet
7758ed8458
fixed fullbench, part 2
2017-07-06 02:48:00 -07:00
Yann Collet
9b2c1acfc0
fixed fullbench
2017-07-06 02:22:57 -07:00
Yann Collet
f04deff4fc
fixed #718 , reported by @GregSlazinski, solution suggested by @mcmilk
2017-07-06 01:42:46 -07:00
Yann Collet
d75c0e71c4
minor code refactoring
2017-07-05 18:10:07 -07:00
Yann Collet
3345a91964
cli : use new advanced API by default
2017-07-05 17:34:15 -07:00
Yann Collet
49af41820d
clarified status of zstdmt_compress.h API
2017-07-05 17:21:13 -07:00
Yann Collet
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
Yann Collet
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
Yann Collet
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
Yann Collet
9ceea19dc1
Merge pull request #747 from facebook/noExit
...
No exit() on decoding errors
2017-07-03 18:15:13 -07:00
Yann Collet
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4
fixed comments
2017-07-03 15:52:19 -07:00
Yann Collet
5a77361595
fixed wrong function name in comment
2017-07-03 15:21:24 -07:00
Yann Collet
95c4a6e2c8
Merge pull request #745 from terrelln/libfuzzer
...
[fuzz] Add libFuzzer targets
2017-07-03 15:15:20 -07:00
Yann Collet
b8280fe929
minor code refactoring
2017-07-03 15:14:55 -07:00
Yann Collet
368b974387
fixed gz error reporting
2017-07-03 13:47:46 -07:00
Yann Collet
c9f21c8a71
gzip : only display inflateEnd error when there is no error before
2017-07-03 13:45:09 -07:00
Yann Collet
7b68d7eb63
Merge pull request #746 from terrelln/dev
...
[libzstd] Fix memcpy() on potential NULL source
2017-07-03 13:31:27 -07:00
Yann Collet
6c35112d08
gzip decoding does no longer exit() on invalid input
2017-07-03 13:24:50 -07:00
Nick Terrell
bea0f0cfa0
[fuzz] Move from fuzz/ to tests/fuzz/
2017-07-03 12:40:12 -07:00
Nick Terrell
c80fc50a8d
[libzstd] Fix memcpy() on potential NULL source
...
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
the source should not be `NULL`.
2017-07-03 12:31:55 -07:00
Yann Collet
e97ff3b54a
decoding malformed lz4 frames does no longer exit()
2017-07-03 11:27:29 -07:00
Yann Collet
c34185df13
fileio : decoding malformed lzma frame does no longer exit()
...
makes it possible to continue decoding file list
2017-07-03 10:27:16 -07:00
Yann Collet
2485f88bf8
fixed legacy version init bug
2017-07-01 09:09:34 -07:00
cyan4973
4b26306cb8
blindfix : fullbench's one-time leak, detected by valgrind
2017-07-01 08:03:59 -07:00
cyan4973
c07e43c2b5
added --show-leak-kind=all to valgrind tests
2017-07-01 07:05:11 -07:00
cyan4973
21fdf97e00
Merge branch 'dev' into fullbench
2017-07-01 07:01:08 -07:00
cyan4973
1bafe393e4
fix : ZSTDMT_compressStream_generic() can accept NULL input
...
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
2017-07-01 06:59:24 -07:00
