Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Paul Cruz
01fc7c4244
changed how the detection of the last job works
2017-07-10 16:27:58 -07:00
Paul Cruz
c36552ef8a
dst buffer should use ZSTD_compressBound to determine how much space it needs
2017-07-10 16:10:19 -07:00
Paul Cruz
7aa36df6df
fixed memory leak that was happening when creating jobs
2017-07-10 16:03:09 -07:00
Stella Lau
ef2b728316
Clean up and refactor compress function
2017-07-10 15:48:47 -07:00
Paul Cruz
e410d63d45
made input buffer an internal part of the compression context
2017-07-10 15:37:14 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
Stella Lau
e4155b11d7
Add warning flags to makefile and clean up code to remove warnings
2017-07-10 13:08:19 -07:00
Stella Lau
10a71d9f1c
Add compression context
2017-07-10 12:38:27 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Paul Cruz
89190ef07d
renamed pool.c to poolTests.c
2017-07-10 11:32:30 -07:00
Paul Cruz
cc7f8e4d71
small changes
2017-07-10 11:10:11 -07:00
Paul Cruz
7e09b508ff
changed name
2017-07-10 11:05:37 -07:00
Paul Cruz
ed72ea5438
removed single from Makefile
2017-07-10 10:58:03 -07:00
Paul Cruz
ced3ec5714
removed scripts
2017-07-10 10:53:02 -07:00
Paul Cruz
82f0d64bee
removed single.c
2017-07-10 10:51:50 -07:00
Paul Cruz
62ebbabd32
updated error checking in each thread
2017-07-10 09:36:22 -07:00
Stella Lau
ae9cf235d6
Add LDM_DCtx
2017-07-10 07:38:09 -07:00
Stella Lau
5432214ee3
Minor refactoring
2017-07-10 06:50:49 -07:00
Stella Lau
b94b468e84
Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm
2017-07-10 06:32:46 -07:00
Stella Lau
474e06ac5b
Minor refactoring
2017-07-10 06:32:29 -07:00
Stella Lau
eb280cd568
Add folder for old versions
2017-07-10 06:32:05 -07:00
Stella Lau
719ccdc5a5
Update mainfile
2017-07-09 22:45:54 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
Stella Lau
acdeb9f302
Add compression statistics
2017-07-07 17:09:28 -07:00
Paul Cruz
c3ae23d459
added ability to compress without specifying out filename
2017-07-07 17:07:05 -07:00
Yann Collet
ed0243a63c
removed zbufftest from list of all tests
2017-07-07 16:16:14 -07:00
Yann Collet
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
Paul Cruz
7163ffafde
playing around with adapt param
2017-07-07 15:56:00 -07:00
Yann Collet
842644e42e
target gpptest uses CXX environment variable
2017-07-07 15:55:41 -07:00
Yann Collet
d4dcace34b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-07 15:53:20 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Paul Cruz
1c9d6b2c6b
rewrote time elapsed with UTIL
2017-07-07 15:42:20 -07:00
Yann Collet
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
Paul Cruz
c0c236a28b
changed to using compressCCtx
2017-07-07 15:13:40 -07:00
Stella Lau
4076be09ec
[ldm] Update to hash every position
2017-07-07 14:52:40 -07:00
Yann Collet
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
Stella Lau
7945f9ee47
Fix offset overflow bug
2017-07-07 14:14:01 -07:00
Paul Cruz
11fc0f4119
changed completed -> compressed
2017-07-07 13:55:38 -07:00
Paul Cruz
09d7c6a994
changed completed variables to compressed for clarity
2017-07-07 13:18:55 -07:00
Stella Lau
f791fc27e3
Add header with compress and decompress size
2017-07-07 12:44:29 -07:00
Paul Cruz
8c0eb62920
removed unnecessary comments, uncommented DEBUGLOG for later use
2017-07-07 11:47:16 -07:00
Paul Cruz
70a4153bd3
added ability to force output to stdout, wrote an additional test for this functionality
2017-07-07 11:32:14 -07:00
Paul Cruz
532f439961
cleaned up code for arguments a bit
2017-07-07 10:58:43 -07:00
