Yann Collet
8977224b9b
Merge pull request #859 from terrelln/31
...
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284
added control from frame content size at end of decompression
...
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
Yann Collet
645563583e
Merge branch 'dev' into newFormats
2017-09-21 16:08:06 -07:00
Yann Collet
f97c2dbd39
created ZSTD_format declaration
2017-09-21 16:07:29 -07:00
Yann Collet
da74aabc00
Merge pull request #850 from terrelln/fse-optimal
...
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804
Merge pull request #849 from terrelln/30
...
[bitstream] Allow adding 31 bits at a time
2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43
[bitstream] Allow adding 31 bits at a time
2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676
[ldm] Fix corner case where minMatch < 8
...
There is a potential read buffer overflow when minMatch < 8.
fix-fuzz-failure
2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff
[libzstd] Fix bad window size assert
...
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.
fix-fuzz-failure
2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b
fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream()
...
previous result was over-estimated
by counting streaming buffers twice
2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec
Merge pull request #848 from terrelln/fparams
...
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-18 13:48:31 -07:00
Yann Collet
539b91ee9b
minor : added assert in bt
2017-09-16 23:41:58 -07:00
Nick Terrell
5f22479517
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-15 17:37:20 -07:00
Yann Collet
77c137b3ae
minor comment refactor
2017-09-14 15:12:57 -07:00
Yann Collet
335780c427
fixed too strong alignment assert in ZSTD_initStaticCCtx()
...
64-bits fields are only 32-bits aligned on 32-bits CPU
2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
4120a7fd5a
Merge pull request #837 from facebook/libzstd-nomt
...
makes it possible to compile libzstd in single-thread mode without zs…
2017-09-12 17:13:17 -07:00
Yann Collet
3306bcb0e6
fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic
...
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676
Merge branch 'ldm-integrate' into ldm-mergeDev
2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84
fixed pass-through warning
2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61
Explicitly cast raw numerals when left-shifting
2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3
makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c ( #819 )
2017-09-11 14:09:34 -07:00
Yann Collet
ce31004f20
fix following suggestions by @terrelln
2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9
Move ldm functions to their own file
2017-09-06 18:09:26 -07:00
Yann Collet
baa37c3362
programs/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
Yann Collet
3a12531a3d
lib/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
2017-09-06 16:35:49 -07:00
Yann Collet
1c7b914cdf
update README on BUCK file
2017-09-06 16:23:39 -07:00
Yann Collet
36374cc3b4
update and clarify lib/README
2017-09-06 16:15:18 -07:00
Stella Lau
2b99d696de
Remove debug code
2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f
Merge pull request #831 from terrelln/split-compress
...
Split parsers out of zstd_compress.c
2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395
Rename and add short ldm parameters in cli
2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1
Fix setting of nextToUpdate at end of ldm matcher
2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688
Split parsers out of zstd_compress.c
2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9
Fix parameter handling in copyCCtx with cdict
2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29
Fix parameter handling with ZSTD_copyCCtx
2017-09-05 15:34:17 -07:00
Stella Lau
643d28c701
Add ldm options to 'man zstd'
2017-09-05 11:27:15 -07:00
Nick Terrell
423b133568
[POOL] Allow free on NULL when multithreading is disabled
2017-09-05 11:18:13 -07:00
Stella Lau
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
Yann Collet
8a5c0c98ae
restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+)
2017-09-01 11:56:57 -07:00
Yann Collet
36aa8b5999
improved decoding speed
2017-09-01 11:40:59 -07:00
Stella Lau
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
Yann Collet
d963daa6a9
fixed minor warning (empty translation unit)
2017-09-01 00:12:07 -07:00
Yann Collet
3704507774
fixed decompression bug reported by @Etsukata ( #828 )
2017-09-01 00:05:37 -07:00
Yann Collet
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
Stella Lau
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
Stella Lau
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau
ee65701720
Minor fixes; remove formatting only changes
2017-08-29 20:27:35 -07:00
Stella Lau
a6e20e1bd7
Add test for raw content starting with dict header
2017-08-29 18:36:18 -07:00
Stella Lau
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
Stella Lau
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
Stella Lau
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
Yann Collet
d6ddb879da
Merge pull request #817 from terrelln/pool-custom-alloc
...
[pool] Accept custom allocators
2017-08-29 13:05:39 -07:00
Stella Lau
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
Nick Terrell
9822f97721
[error] Don't guard undef X with ifdef X
2017-08-29 11:54:38 -07:00
Stella Lau
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
Stella Lau
0e56a84a1e
Fix getting cParams from CCtxParams
2017-08-28 19:25:17 -07:00
Nick Terrell
02033be08c
[pool] Visual Studios disallows empty structs
2017-08-28 17:19:01 -07:00
Nick Terrell
7c365eb02c
[threading] Fix ERROR macro after including windows.h
2017-08-28 16:25:02 -07:00
Bernhard M. Wiedemann
cf689b84f9
Sort input file list
...
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
2017-08-26 17:08:00 +02:00
Stella Lau
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
Stella Lau
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
Nick Terrell
de6c6bce85
Fix zstd_internal.h for C++ mode
2017-08-24 18:09:50 -07:00
Nick Terrell
26dc040a7b
[pool] Accept custom allocators
2017-08-24 17:01:41 -07:00
Nick Terrell
89dc856cae
[pool] Fix formatting
2017-08-24 16:48:32 -07:00
Stella Lau
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
Nick Terrell
376f435914
[dictBuilder] Set default compression level to 3
2017-08-24 16:21:05 -07:00
Stella Lau
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
Stella Lau
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
Stella Lau
64ce49426b
Fix cstream compression level
2017-08-23 12:30:47 -07:00
Stella Lau
5bc2c1e982
Add prototype support for customMem with cctxParams
2017-08-23 12:03:30 -07:00
Yann Collet
e9ce1208a1
Merge pull request #812 from facebook/longRangeFix
...
fixed extraordinary scenario where all fields use maximum nbBits
2017-08-23 11:35:28 -07:00
Yann Collet
74cde5a4d8
Merge pull request #813 from stellamplau/highbit32fix
...
Fix undefined behavior when srcSize==1
2017-08-23 11:31:06 -07:00
Stella Lau
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
Dmitriy Titarenko
20f715d709
Fix displayLevel overflow
2017-08-23 15:56:15 +05:00
Stella Lau
11303778d0
Add function to make cctxParams from ZSTD_parameters
2017-08-22 14:53:13 -07:00
Yann Collet
bd9c8ca146
Merge pull request #811 from terrelln/segmentSize
...
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
Stella Lau
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
Yann Collet
6b2b6a9bd5
fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously
...
can only happen if windowLog>=27 (level 22 --ultra)
2017-08-22 12:09:21 -07:00
Stella Lau
e50ed1fa3a
Fix undefined behavior when srcSize==1
2017-08-22 11:55:42 -07:00
Stella Lau
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
Nick Terrell
29c2d9a4d0
[cover] Turn down notification for ZDICT subroutines
2017-08-21 14:28:31 -07:00
Nick Terrell
98de3f6847
[cover] Add dictionary size to compressed size
2017-08-21 14:23:17 -07:00
Yann Collet
78c3d16bf4
Merge pull request #809 from terrelln/dev
...
[cover] Fix divide by zero
2017-08-21 13:33:19 -07:00
Nick Terrell
9a54a315aa
[cover] Convert score to U32 and check for zero
2017-08-21 13:30:07 -07:00
Stella Lau
fd8a25786e
Check parameters are valid in initCCtxParams
2017-08-21 13:23:35 -07:00
Stella Lau
1c0dbe81b1
Add documentation for CCtx_params
2017-08-21 13:18:00 -07:00
Nick Terrell
d49eb40c03
[cover] Stop when segmentSize is less than d
2017-08-21 13:10:03 -07:00
Stella Lau
939f954285
Pass ZSTD_CCtx_params as const ptr when possible
2017-08-21 12:57:18 -07:00
Stella Lau
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
Stella Lau
560b34f6d2
Return error code when initializing NULL cctxParams
2017-08-21 11:52:26 -07:00
Stella Lau
25be09c6b4
Set some parameters to zero before initializing cdict
2017-08-21 11:35:46 -07:00
Yann Collet
232d62b637
fixed a few headers that were too hastily copy/pasted during last license change
2017-08-21 11:24:32 -07:00
Nick Terrell
f306d400c0
[cover] Fix divide by zero
2017-08-21 11:12:11 -07:00
Stella Lau
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
Stella Lau
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
Yann Collet
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
Stella Lau
6cee6e07e5
Add internal createCDict function
2017-08-18 22:48:31 -07:00
Stella Lau
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
Yann Collet
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
Stella Lau
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
Stella Lau
399ae013d4
Add function to apply cctx params
2017-08-18 13:01:55 -07:00
Stella Lau
81d89d82a6
Move nbThreads to cctx params
2017-08-18 12:08:57 -07:00
Stella Lau
2300c58a6f
Move dictContentByRef to cctx params
2017-08-18 12:03:16 -07:00
Stella Lau
b6cb2ed8cb
Move dictMode to cctxParams
2017-08-18 11:43:31 -07:00
Stella Lau
97e27affcb
Move compression level to cctx params
2017-08-18 11:20:08 -07:00
Stella Lau
c0221124d5
Add function to set opaque parameters
2017-08-17 19:30:22 -07:00
Stella Lau
4169f49171
Add initialization/allocation functions for opaque params
2017-08-17 18:45:04 -07:00
Stella Lau
ade95b8bed
Add opaque interfaces for static initialization
2017-08-17 18:13:08 -07:00
Stella Lau
699f11b4f7
Create opaque parameter structure
2017-08-17 17:33:46 -07:00
Yann Collet
f9e6590715
Merge pull request #796 from terrelln/is-error
...
[FSE][HUF] Inline error checks
2017-08-15 12:37:28 -07:00
Yann Collet
2dbcfc6994
Merge pull request #794 from terrelln/force-inline
...
[libzstd] Fix FORCE_INLINE macro
2017-08-15 12:03:44 -07:00
Nick Terrell
07c6ff588e
[FSE][HUF] Inline error checks
...
Caught by Clang's optimization remarks.
2017-08-15 11:23:28 -07:00
Nick Terrell
565e925eb7
[libzstd] Fix FORCE_INLINE macro
2017-08-14 21:12:05 -07:00
Roman Gershman
b9d4f4fb74
Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged
2017-08-13 13:29:42 +03:00
Nick Terrell
9ba97182d1
[CI] Add gcc7build test
2017-08-08 13:28:56 -07:00
Yann Collet
d9f2893eb9
Merge pull request #782 from terrelln/dstSizeTooSmall
...
Fix compression failure on incompressible data
2017-08-07 14:52:02 -07:00
Yann Collet
8049556928
Merge pull request #778 from terrelln/bad-huff
...
[libzstd] Fix bug in Huffman decompresser
2017-08-07 14:05:58 -07:00
Nick Terrell
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
Nick Terrell
308047eb5d
Fix compression failure on incompressible data
...
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.
Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
2017-08-07 11:45:24 -07:00
Stella Lau
73ba58955f
Signal after finishing job when queueSize=0
2017-08-01 20:12:06 -07:00
Stella Lau
1d76da1d87
Replace marker with queueEmpty variable and update pool.h comment
2017-08-01 12:30:16 -07:00
Stella Lau
5adceeed01
Allow queueSize=0 in pool.c and update poolTests
2017-07-31 10:10:16 -07:00
Yann Collet
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
Nick Terrell
ae20d413da
[libzstd] Fix CHECK_V_F macros
2017-07-25 12:52:01 -07:00
Yann Collet
a90b16e150
Visual blind fix 2
2017-07-20 15:57:55 -07:00
Yann Collet
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
Yann Collet
3974d2b38a
blind fix for Windows Multithreading module
...
adds a fake 0 return value for mutex/cond init
2017-07-19 13:33:21 -07:00
Paul Cruz
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
Yann Collet
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
Nick Terrell
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
Yann Collet
77d67fb167
Merge pull request #766 from terrelln/real-block-split
...
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-18 08:26:24 -07:00
Yann Collet
14c83b05c7
Merge pull request #765 from terrelln/real-block-split
...
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 19:25:55 -07:00
Nick Terrell
7a28b9e4a3
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-17 15:29:11 -07:00
Yann Collet
3381bf4b84
Merge pull request #764 from terrelln/real-block-split
...
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 14:46:01 -07:00
Nick Terrell
e198230645
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 12:27:24 -07:00
Nick Terrell
634f012420
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 11:36:11 -07:00
Paul Cruz
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
Yann Collet
3b0cff3c33
fixed clang's -Wdocumentation
2017-07-13 18:58:30 -07:00
Yann Collet
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
Nick Terrell
830ef4152a
[libzstd] Increase granularity of FSECTable repeat mode
2017-07-13 12:45:39 -07:00
Yann Collet
d985319337
Merge pull request #759 from terrelln/real-block-split
...
[libzstd] Pull CTables into sub-structure
2017-07-13 10:24:19 -07:00
Yann Collet
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
Yann Collet
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
Yann Collet
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
Nick Terrell
de0414b736
[libzstd] Pull CTables into sub-structure
2017-07-12 19:49:19 -07:00
Yann Collet
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
Yann Collet
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Yann Collet
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
Yann Collet
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
Yann Collet
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
Yann Collet
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
Yann Collet
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
Yann Collet
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Yann Collet
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
Yann Collet
f04deff4fc
fixed #718 , reported by @GregSlazinski, solution suggested by @mcmilk
2017-07-06 01:42:46 -07:00
Yann Collet
d75c0e71c4
minor code refactoring
2017-07-05 18:10:07 -07:00
Yann Collet
49af41820d
clarified status of zstdmt_compress.h API
2017-07-05 17:21:13 -07:00
Yann Collet
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
Yann Collet
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
Yann Collet
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
Yann Collet
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4
fixed comments
2017-07-03 15:52:19 -07:00
Yann Collet
5a77361595
fixed wrong function name in comment
2017-07-03 15:21:24 -07:00
Nick Terrell
c80fc50a8d
[libzstd] Fix memcpy() on potential NULL source
...
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
the source should not be `NULL`.
2017-07-03 12:31:55 -07:00
Yann Collet
2485f88bf8
fixed legacy version init bug
2017-07-01 09:09:34 -07:00
cyan4973
21fdf97e00
Merge branch 'dev' into fullbench
2017-07-01 07:01:08 -07:00
cyan4973
1bafe393e4
fix : ZSTDMT_compressStream_generic() can accept NULL input
...
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
2017-07-01 06:59:24 -07:00
Yann Collet
58bd0e70fc
fixed : dictionary compression with new advanced API in Multi-threading mode
2017-06-30 16:01:02 -07:00
Yann Collet
d8b33a598d
Optimized ZSTDMT single-pass mode speed on large sources
...
by ensuring job sizes remain "not too large"
2017-06-30 15:44:57 -07:00
Yann Collet
d5c046c609
implemented shortcut for zstd_compress_generic() in MT mode
...
added ZSTDMT_compress_advanced() API
2017-06-30 14:51:01 -07:00
Yann Collet
7f40bb1c39
Merge pull request #742 from stellamplau/stack-space
...
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-30 14:50:23 -07:00
Stella Lau
32df49e9f8
Fix typo
2017-06-30 12:56:24 -07:00
Stella Lau
b0513b519c
Add comment to HUF_DECOMPRESS_WORKSPACE_SIZE
2017-06-30 12:53:56 -07:00
Stella Lau
4c71f59c77
Clarify typedef of rankVal_t and rankValCol_t
2017-06-30 09:52:20 -07:00
Stella Lau
28f711ef95
Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK
2017-06-30 09:38:11 -07:00
Stella Lau
70ad6829e7
Delegate HUF_decompress4X_hufOnly to workspace version
2017-06-29 16:22:32 -07:00
Stella Lau
104c4d57c1
Fix bitshift error
2017-06-29 15:40:49 -07:00
Yann Collet
a3d9926c40
compression optimization opportunity
...
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
2017-06-29 14:44:49 -07:00
Stella Lau
fedc94de8c
Fix pointer casting warning
2017-06-29 13:04:15 -07:00
Stella Lau
c6a5275a28
Fix alignment warnings with pointer casting
2017-06-29 12:39:34 -07:00
Stella Lau
99e315999c
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-29 11:49:59 -07:00
Yann Collet
97f2bf66da
minor : fix typo
2017-06-29 11:31:40 -07:00
Yann Collet
acbef3decd
ZSTD_getFrameContentSize() is promoted to "stable" status
2017-06-29 05:19:51 -07:00
Yann Collet
590937df20
Merge pull request #739 from facebook/refPrefix
...
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
Yann Collet
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
Yann Collet
037466245f
refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget()
...
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.
Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
2017-06-28 20:24:08 -07:00
Yann Collet
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
Yann Collet
1ca76039af
fixed -Wdeclaration-after-statement
2017-06-28 15:40:21 -07:00
Yann Collet
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
Yann Collet
adbe74a8ac
adjusted compression levels to guarantee a monotonically increasing memory budget
2017-06-28 13:22:37 -07:00
Yann Collet
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
Yann Collet
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
Yann Collet
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
Yann Collet
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
Yann Collet
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
Nick Terrell
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
Yann Collet
c7fb884eea
fixed minor conversion warning
2017-06-26 18:02:23 -07:00
Yann Collet
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
Yann Collet
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
Yann Collet
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
Yann Collet
ecb0f46866
add controls over streaming buffers
2017-06-21 17:25:01 -07:00
Yann Collet
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
Yann Collet
204b6b7ef6
fixed streaming buffered allocation with CDict compression
2017-06-21 15:13:00 -07:00
Yann Collet
1e4129b27b
fixed dangling pointer risk, detected by @terrelln
2017-06-21 13:26:10 -07:00
Yann Collet
83095970e6
free cdictLocal faster, suggested by @terrelln
2017-06-21 12:26:40 -07:00
Yann Collet
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
Yann Collet
9c56b12938
Merge pull request #723 from paulcruz74/dev
...
Adding zstd -l
2017-06-21 09:41:55 -07:00
Yann Collet
e51d51bdf7
fixed memcpy() overlap
2017-06-20 17:44:55 -07:00
Yann Collet
466f92eaa6
removed one useless streaming compression stage, detected by @terrelln
2017-06-20 16:25:29 -07:00
Yann Collet
c3bce24ef4
fixed potential dangling pointer, detected by @terrelln
2017-06-20 16:09:11 -07:00
Yann Collet
78b8234554
fixed comments, following suggestion by @terrelln
2017-06-20 14:26:48 -07:00
Yann Collet
b44ab82f7a
ensure new ZSTD_strategy starts at value 1
2017-06-20 14:11:49 -07:00
Yann Collet
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
Yann Collet
695a0a3449
fixed IA64 compilation error, by @mcmilk
2017-06-19 15:27:30 -07:00
Yann Collet
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
Nick Terrell
55f9cd4942
[libzstd] Fix UBSAN failure
2017-06-19 15:12:28 -07:00
Yann Collet
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
Yann Collet
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
Yuri
92bafda406
INSTALL_DATA instead of INSTALL_LIB for libzstd.a
...
INSTALL_LIB can be passed -s flag to strip symbols. Static libraries should not be stripped, only dynamic ones should be stripped.
2017-06-17 00:23:41 -07:00
Yann Collet
381e66cfbd
added ZSTD_clampCParams()
...
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
Yann Collet
aee916e37c
fixed +/-1 error for pledgedSrcSizePlusOne
2017-06-16 17:02:35 -07:00
Yann Collet
d3de3d51a3
fix attempt 2 : Visual sign conversion warning
2017-06-16 16:51:33 -07:00
Yann Collet
944be54774
fixed attempt : minor Visual sign conversion warning
2017-06-16 14:05:01 -07:00
Yann Collet
b26728c9c8
added ZSTD_startNewCompression()
2017-06-16 14:00:46 -07:00
Yann Collet
a0ba849fe6
changed frameContentSize field to pledgedSrcSizePlusOne
...
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.
PlusOne is meant to have 0 (default) == unknown
2017-06-16 13:29:17 -07:00
Yann Collet
2cf7755da7
fix : pledgedSrcSize correctly reset to unknown in "continue" mode
2017-06-16 12:34:41 -07:00
Yann Collet
9e73f2f320
fix : correctly reset pledgedSrcSize to unknown status
...
when starting a new compression with an existing context
2017-06-16 12:24:01 -07:00
Yann Collet
33873f0e74
fixed : new advanced AIP : setting nbThreads to the same value > 1
2017-06-16 12:04:21 -07:00
Yann Collet
559ee82e90
fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps
2017-06-16 11:58:21 -07:00
Yann Collet
bd18c885a3
added ZSTD_CCtx_reset
2017-06-16 10:17:50 -07:00
Yann Collet
cc9f9b7f4c
protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag
2017-06-15 18:17:34 -07:00
Yann Collet
05ae4b2190
added protection : MT incompatible with Static allocation
2017-06-15 18:03:34 -07:00
Paul Cruz
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
Yann Collet
f129fd3970
disabled MT code path when ZSTD_MULTITHREAD is not defined
2017-06-11 18:46:09 -07:00
Yann Collet
23aace9778
added control stage to MT mode
2017-06-11 18:32:36 -07:00
Yann Collet
f35e2de61c
linked newAPI to ZSTDMT
2017-06-05 18:32:48 -07:00
cyan4973
c59162e053
minor fix for -Wdocumentation
2017-06-05 00:12:13 -07:00
cyan4973
8bcbf42617
fixed g++ prototype mismatch
2017-06-04 23:52:00 -07:00
Yann Collet
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1
made debug definitions common within zstd_internal.h
2017-06-02 18:20:48 -07:00
Yann Collet
8ddf4c22d5
fixed missing initialization
2017-06-02 17:16:49 -07:00
Yann Collet
33a7e679e5
significant zlib wrapper code refactoring
...
code indentation
variable scope and names
constify
Only coding style changes.
The logic should remain the same.
2017-06-02 17:10:49 -07:00
Yann Collet
4effccbf56
zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing
...
more generic and safer than using own routing for magic number comparison
2017-06-02 14:27:11 -07:00
Yann Collet
dcb7535352
ensure zlibwrapper uses ZSTD_malloc() and ZSTD_free()
...
which is compatible with { NULL, NULL, NULL }
2017-06-02 14:01:21 -07:00
Yann Collet
b877e834b1
minor indent
2017-06-02 13:47:11 -07:00
Yann Collet
6056e4c3eb
added POOL_sizeof() for single-thread
2017-06-02 11:36:47 -07:00
Yann Collet
c35e535002
added support for multithreading parameters
2017-06-01 18:44:06 -07:00
Yann Collet
c4a5a21c5c
created ZSTDMT_sizeof_CCtx() and POOL_sizeof()
...
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
2017-06-01 17:56:14 -07:00
Yann Collet
cd2892fd1e
protected impossible switch(){default:} with assert(0)
...
can be converted into assume(0) in some future
2017-06-01 09:44:54 -07:00
Yann Collet
06589fe516
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-05-31 10:03:20 -07:00
Yann Collet
18ab5affa5
fixed visual warning
2017-05-31 09:59:22 -07:00
Yann Collet
9a691e0f55
fixed visual warnings
2017-05-31 01:17:44 -07:00
Yann Collet
01b1549f83
finally converted ZSTD_compressStream_generic() to use {in,ou}Buffer
...
replacing the older read/write variables from ZBUFF_* era.
Mostly to help code readability.
Fixed relevant callers.
2017-05-30 18:10:26 -07:00
Yann Collet
c4f46b94ce
ZSTD_createCCtx_advanced() now uses ZSTD_calloc()
...
initially uses calloc() instead of memset().
Performance improvement is unlikely measurable,
since ZSTD_CCtx is now very small,
with all tables transferred into workSpace.
2017-05-30 17:45:37 -07:00
Yann Collet
deee6e523f
expose ZSTD_compress_generic_simpleArgs()
...
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
2017-05-30 17:42:00 -07:00
Yann Collet
ae728a43b8
removed defaultCustomMem
...
now ZSTD_customCMem is promoted as new default.
Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.
ZSTD_customCMem is public constant
(defaultCustomMem was private only).
Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.
Fixed zlibwrapper which depended on defaultCustomMem.
2017-05-30 17:11:39 -07:00
Yann Collet
5bcef1ada2
removed mtctx->cstream
...
use the first cctx in pool when ZSTDMT is used in single-thread mode
now that cctx and cstream are the same object.
2017-05-30 16:37:19 -07:00
Yann Collet
beb62b15a8
Merge branch 'dev' into advancedAPI2
...
Fixed conflic in zstd_decompress.c
2017-05-30 16:18:57 -07:00
Yann Collet
44e45e8423
added ZSTDMT_createCCtx_advanced()
...
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
Yann Collet
f45ca527a1
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-05-30 10:02:03 -07:00
Yann Collet
b6dec4c3ae
fixed minor cast warning
2017-05-27 17:09:06 -07:00
Yann Collet
e071159101
mtctx->jobs allocate its own memory space
...
to make ZSTDMT_CCtx_s size predictable
so that it can be included in CCtx
2017-05-27 00:21:33 -07:00
Yann Collet
b8136f019a
static dctx is incompatible with legacy support
...
documented, and runtime tested
2017-05-27 00:03:08 -07:00
Yann Collet
7028cbd7fd
fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader
2017-05-25 18:29:08 -07:00
Yann Collet
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
Dmitry V. Levin
1ea655c765
Fix typo in libzstd.a-mt make rules
...
The macro name is ZSTD_MULTITHREAD, not ZSTD_MULTHREAD.
Fixes: ca6fae7808
2017-05-25 23:43:05 +00:00
Yann Collet
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
Yann Collet
ba183005d3
merged DStream's inBuff and outBuff into a single buffer
...
Saves one malloc().
Also : makes it easier to implement static allocation
2017-05-24 15:42:24 -07:00
Nick Terrell
55fc1f91fd
[zstd] Fix up formatting edge cases for clang-format
2017-05-24 13:50:10 -07:00
Yann Collet
2e4db3e531
fixed performance regression with ZSTD_decompress() on small files
...
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.
Fixed by only initialization relevant pointers and size fields.
2017-05-24 13:15:19 -07:00
Yann Collet
11ea2f7fda
Merged ZSTD_DCtx and ZSTD_DStream objects
...
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
2017-05-23 16:19:43 -07:00
Yann Collet
b81f19ffce
move MEM_readMINMATCH() into zstd_opt.h
...
which is its only user.
Use case too narrow to belong to mem.h.
renamed to ZSTD_readMINMATCH()
2017-05-23 15:41:55 -07:00
Yann Collet
c7fe262dc9
added ZSTD_initStaticCCtx()
...
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
Yann Collet
5ac72b417c
Buffered are now allocated inside workSpace
2017-05-23 11:18:24 -07:00
Yann Collet
1880337c30
Simplifier compression call graph
...
Everything converge towards ZSTD_compressBegin_internal
which delegated to ZSTD_copyCCtx_internal if cdict!=NULL.
This simplifies routing which was previously depending on cdict.
2017-05-22 18:21:51 -07:00
Yann Collet
b0739bcf8f
simplified reset by removing full-reset policy
...
this was meant to be applied prior to dictionary loading.
But effectively, it seems redundant with later loading stage,
so it can be skipped safely.
2017-05-22 17:45:15 -07:00
Yann Collet
1ad7c82eb5
Implemented separation between requested and applied parameters
...
first version to pass cli tests with -DZSTD_NEWAPI
2017-05-22 17:06:04 -07:00
Yann Collet
24de7b0346
Implemented ZSTD_CCtx_refCDict()
2017-05-22 13:05:45 -07:00
Yann Collet
ee970398b2
Merge branch 'dev' into advancedAPI2
2017-05-22 12:33:56 -07:00
Yann Collet
8b21ec42a9
ZSTD_compress_generic() can handle dictionary compression
2017-05-19 19:46:15 -07:00
Nick Terrell
a1280406b0
[libzstd] Allow users to define custom visibility
2017-05-19 18:01:59 -07:00
Yann Collet
334a288d0d
ZSTD_CCtx_setParameter() only works during initialization stage
...
and generate a stage_wrong error otherwise.
2017-05-19 11:04:41 -07:00
Yann Collet
48855fa0d2
fixed declaration-after-statement warning
2017-05-19 10:56:11 -07:00
Yann Collet
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
Yann Collet
009d604e00
ZSTD_compress_generic() supports multiple successive frames
...
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
Yann Collet
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
Yann Collet
9f95e445ab
minor comment clarifications
2017-05-16 17:26:43 -07:00
Yann Collet
0bdb575c31
Merge branch 'dev' into advancedAPI2
2017-05-16 16:32:29 -07:00
Yann Collet
7101434ec9
pedantic : added one error check
...
on a function which (today) never fails.
But who knows, maybe tomorrow ...
2017-05-16 16:28:24 -07:00
Yann Collet
bfff8999c5
added prototype ZSTD_versionString()
2017-05-16 16:12:23 -07:00
Yann Collet
4eff8136aa
added prototype ZSTD_decompressBegin_usingDDict ( #700 )
2017-05-16 16:05:27 -07:00
Yann Collet
2d4d31c18a
removed gcc compilation flag -Wbad-function-cast
...
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.
also : fixed minor Visual conversion warnings in datagencli.c
2017-05-16 11:34:38 -07:00
Yann Collet
133f0aee54
fixed redundant declarations in legacy v0.5 and v0.7 decoders
...
triggered by new flag -Wredundant-decls
2017-05-15 17:44:04 -07:00
Yann Collet
83d0c764dc
added several compilation flags
2017-05-15 17:15:46 -07:00
Yann Collet
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
Yann Collet
add66f816d
changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams()
...
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
2017-05-12 16:01:15 -07:00
Yann Collet
b0edb7fb0e
added ZSTD_CCtx_setParameter()
2017-05-12 15:31:53 -07:00
Yann Collet
ef738c1b23
better error code when compressing using NULL CDict
...
which is not allowed (but detected, and generates an error).
2017-05-12 13:55:25 -07:00
Yann Collet
db8e21d5a0
made ZSTD_compress_generic() definition accessible
...
note that the implementation is not done yet.
2017-05-12 13:46:49 -07:00
Yann Collet
33eb7ac6b6
updated Advanced API proposal
...
only declarations in zstd.h
2017-05-12 12:36:11 -07:00
Yann Collet
bd1964a988
Merge pull request #696 from joscollin/wip-lib-legacy-fallthrough-warn
...
lib/legacy: warning: this statement may fall through
2017-05-11 10:45:01 -07:00
Yann Collet
4c1cfc0bb6
Merge pull request #695 from joscollin/wip-lib-compress-fallthrough-warn
...
lib/compress: warning: this statement may fall through
2017-05-11 10:44:27 -07:00
Jos Collin
280510f2d5
lib/legacy: warning: this statement may fall through
...
The following warning appears during build at sevaral places.
../lib/legacy/zstd_v04.c:819:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16);
../lib/legacy/zstd_v05.c:821:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16);
../lib/legacy/zstd_v06.c:913:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);
../lib/legacy/zstd_v07.c:583:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) <<
(sizeof(bitD->bitContainer)*8 - 16);
Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 14:27:40 +05:30
Jos Collin
7cd7a7564b
lib/compress: warning: this statement may fall through
...
The following warning appears during build.
../lib/compress/huf_compress.c: In function ‘HUF_compress1X_usingCTable’:
../lib/compress/huf_compress.c:444:8: warning: this statement may fall through [-Wimplicit-fallthrough=]
if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream)
^
../lib/compress/huf_compress.c:465:18: note: in expansion of macro ‘HUF_FLUSHBITS_2’
HUF_FLUSHBITS_2(&bitC);
^~~~~~~~~~~~~~~
../lib/compress/huf_compress.c:466:9: note: here
case 2 : HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);
../lib/compress/zstd_compress.c: In function ‘ZSTD_compressStream_generic’:
../lib/compress/zstd_compress.c:3366:34: warning: this statement may fall through [-Wimplicit-fallthrough=]
zcs->streamStage = zcss_flush; /* pass-through to flush stage */
~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
../lib/compress/zstd_compress.c:3369:9: note: here
case zcss_flush:
Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 13:17:26 +05:30
Jos Collin
05286fdd5a
lib/common: warning: this statement may fall through
...
The following warning appears during the build. Fixed the review comments too.
zstd/lib/common/bitstream.h: In function ‘BIT_initDStream’:
zstd/lib/common/bitstream.h:277:33: warning: this statement may fall through [-Wimplicit-fallthrough=]
case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) <<
(sizeof(bitD->bitContainer)*8 - 16);
Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 09:10:02 +05:30
Nick Terrell
374f868354
Update whitespace
2017-05-10 17:48:42 -07:00
Nick Terrell
5f2c7213c7
Merge remote-tracking branch 'upstream/dev' into btopt
...
* upstream/dev: (305 commits)
added test for ZSTD_estimateCStreamSize()
changed variable name, for clarity
fixed ZSTD_estimateCStreamSize()
shortened ZSTD_createCStream_Advanced()
fixed symbols test
added ZSTD_estimateDStreamSize()
changed name frameParams into frameHeader
regroup memory usage function declarations
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
bumped version number
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
Updated ZSTD_freeCCtx()
updated ZSTD_estimateCCtxSize()
Updated ZSTD_sizeof_CCtx()
merged CCtx and CStream as a single same object
cli : -d and -t do not stop after a failed decompression
added dev branch CircleCI badge
added dev branch Appveyor badge
keep dev branch status only
creates a binary archive without the `programs` directory
...
2017-05-10 16:49:58 -07:00
Yann Collet
ba41b26405
Merge pull request #689 from facebook/cctxMerge
...
Cctx merge
2017-05-10 14:53:54 -07:00
Yann Collet
cef02d9317
changed variable name, for clarity
...
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
Yann Collet
669346fe8b
fixed ZSTD_estimateCStreamSize()
...
https://github.com/facebook/zstd/pull/689#discussion_r115637721
2017-05-10 11:08:00 -07:00
Yann Collet
6fb2f24132
shortened ZSTD_createCStream_Advanced()
...
https://github.com/facebook/zstd/pull/689#discussion_r115637613
2017-05-10 11:06:06 -07:00
Yann Collet
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7
regroup memory usage function declarations
...
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
...
for clarity
2017-05-08 18:24:16 -07:00
Yann Collet
51652522a2
bumped version number
2017-05-08 17:52:46 -07:00
Yann Collet
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet
7855366598
Updated ZSTD_freeCCtx()
...
which can also contain streaming buffers now.
Redirected ZSTD_freeCStream() towards it.
2017-05-08 17:15:00 -07:00
Yann Collet
fc5145955a
updated ZSTD_estimateCCtxSize()
...
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).
Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
Yann Collet
791d744279
Updated ZSTD_sizeof_CCtx()
...
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
2017-05-08 16:17:30 -07:00
Yann Collet
0be6fd3429
merged CCtx and CStream as a single same object
...
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet
d47709b6ea
Merge pull request #654 from iburinoc/splittable
...
[RFC] Splittable Format and API
2017-05-08 13:41:56 -07:00
Yann Collet
a00e9599f1
removed -g from DEBUGFLAGS
...
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
2017-05-04 17:24:29 -07:00
Yann Collet
606c04c228
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-05-02 12:13:52 -07:00
Yann Collet
072484a3bf
Merge pull request #683 from terrelln/odev
...
[CLI] Make cover the default dictionary builder
2017-05-02 12:13:23 -07:00
Nick Terrell
f376d47c11
[CLI] Switch dictionary builder on CLI to cover
2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13
[cover] Make optimization faster
2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0
[cover] Optimize case where d <= 8
2017-05-02 11:02:43 -07:00
Nick Terrell
865918dd04
Fix typo in zdict.h
2017-05-02 11:02:37 -07:00
Yann Collet
b184589c4c
minor code refactoring for clarity
2017-05-01 11:35:47 -07:00
Yann Collet
33c38b0925
fixed const in prototype, that Visual doesn't accept
2017-05-01 11:12:30 -07:00
Yann Collet
f39a6731ec
sync bitstream.h from fse library
2017-05-01 09:56:03 -07:00
Yann Collet
202082f285
sync bitstream from FSE project
...
add assert into unsafe *_fast() variants
2017-04-28 17:00:31 -07:00
Yann Collet
89f50deec7
minor code refactoring
...
clearer tables
2017-04-28 16:52:36 -07:00
Yann Collet
68a7d3d49a
added HUF_PUBLIC_API macro to huf.h
...
to make it possible to control symbol visibility.
Also : better separation and comments between "public" and "static" sections
2017-04-28 12:46:48 -07:00
Yann Collet
a51cab6e68
Merge pull request #678 from facebook/apiChange
...
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
Yann Collet
29297c6751
Changed default level 18 (large input)
...
Previous -18 : 4.7 MB/s, R:3.833
New -18 : 5.1 MB/s. R:3.825
It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s)
The new level 18 also uses significantly less memory.
And, it makes a good transition between level 17 (mml5)
and level 19 (mml3).
Up to now, there was no level with mml4.
(note : minmatch setting can have a large impact on some (specific) datasets)
2017-04-27 17:44:01 -07:00
Yann Collet
a92cbb7004
Added a secondary test, checking dictID presence after setting noDictIdFLag=1
2017-04-27 15:08:56 -07:00
Yann Collet
d3694e6c70
removed C4204
2017-04-27 14:29:35 -07:00
Yann Collet
1c3ab0c77f
fixed init error on Visual 2008
2017-04-27 12:57:11 -07:00
Yann Collet
8b669535f8
bumped version number to v1.2.0
2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50
added ZSTD_initCStream_usingCDict_advanced()
2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81
created ZSTD_compress_usingCDict_advanced()
2017-04-27 11:31:55 -07:00
Yann Collet
69a54d138a
fixed compilation warning : declaration-after-statement
2017-04-27 01:11:26 -07:00
Yann Collet
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Yann Collet
768df129d2
changed ZSTD_compressBegin_usingCDict()
...
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
2017-04-26 15:42:10 -07:00
Yann Collet
e42afbc6fa
Comply with suggested comments by @terrelln
...
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
2017-04-26 11:39:35 -07:00
Sean Purcell
7d37ca1d5b
Merge remote-tracking branch 'origin/dev' into splittable
2017-04-21 14:18:39 -07:00
Yann Collet
7271203bdb
transferred entropy scratch space from CCtx into workSpace
...
Saved 6 KB
2017-04-20 23:21:19 -07:00
Yann Collet
a408645f50
made some room for entropy scratch space
2017-04-20 23:09:39 -07:00
Yann Collet
71aaa32c3c
transferred FSE tables from CCtx into workspace
...
Saved 5 KB from CCtx
2017-04-20 23:03:38 -07:00
Yann Collet
71ddeb67b1
made room in workspace for FSE tables
...
still need to be transferred from CCtx into workspace
2017-04-20 22:54:54 -07:00
Yann Collet
a34a39c183
changed size evaluation of entropy tables
...
so that memcpy() does no longer depends on fse pointer being a static table
2017-04-20 18:26:25 -07:00
Yann Collet
7bb60b17d8
init entropy table pointers only once
...
per workSpace resize
2017-04-20 17:38:56 -07:00
