Commit Graph

1961 Commits

Author SHA1 Message Date
Yann Collet
377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error
wait for all jobs to be completed, so that freeing can happen safely
2017-09-28 02:23:44 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
Yann Collet
e45a2aea9b Merge pull request #869 from terrelln/dev
[libzstd] pthread function prefixed with ZSTD_
2017-09-27 16:35:08 -07:00
Nick Terrell
b555b7ef41 [libzstd][opt] Simplify repcode logic 2017-09-27 15:30:12 -07:00
Yann Collet
ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
Yann Collet
c994932788 fixed ZSTD_format_e value validation 2017-09-27 12:22:22 -07:00
Nick Terrell
6c41adfb28 [libzstd] pthread function prefixed with ZSTD_
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
Yann Collet
ecf1778e23 updated ZSTD_format_e value validation
also updated manual
2017-09-27 11:19:21 -07:00
Yann Collet
9416195221 changed error code when pos<=size condition is not respected
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
2017-09-27 10:35:56 -07:00
Yann Collet
d56a350402 removed unsupported formats 2017-09-27 10:29:31 -07:00
Yann Collet
ca306c1c84 fixed a bug in zstreamtest
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
Yann Collet
cd53ac831b fixed DCtx initialization error
now relying on initialization of dctx->format first
2017-09-26 18:26:09 -07:00
Yann Collet
4791561c4a silence minor gcc warning -Wempty-body
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
Yann Collet
c0dd960363 switch assert() position 2017-09-26 15:36:57 -07:00
Yann Collet
319c699991 created ZSTD_startingInputLength()
as suggested by @terrelln
2017-09-26 15:36:14 -07:00
Yann Collet
8d1e97ea9c minor fixes following @terrelln comments 2017-09-26 15:06:30 -07:00
Yann Collet
df4e9bba25 fixed constant errors for gcc in c99 mode
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
Yann Collet
9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78 Merge pull request #862 from terrelln/static
[zstd] Backport kernel patch from @ColinIanKing
2017-09-25 17:02:40 -07:00
Yann Collet
52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
2017-09-25 16:18:23 -07:00
Yann Collet
f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet
6ee05a02b8 added ZSTD_decompress_generic()
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
Yann Collet
b8d4a3887f introduced constant ZSTD_frameIdSize
within zstd_internal.h
This is the size of magic number.

Avoids using `4` directly in source code, which is a bit less meaningful.
2017-09-25 15:26:18 -07:00
Yann Collet
044fb4c057 implemented magic-less frame decoder 2017-09-25 15:12:09 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames
decoder not implemented yet
2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Yann Collet
96f0cde31a minor function rename
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce added prototypes for advanced parameters for decompression API
required to decode custom formats
2017-09-24 15:57:29 -07:00
Yann Collet
e60f48c549 Merge branch 'dev' into newFormats 2017-09-24 14:33:37 -07:00
Yann Collet
8977224b9b Merge pull request #859 from terrelln/31
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
Yann Collet
645563583e Merge branch 'dev' into newFormats 2017-09-21 16:08:06 -07:00
Yann Collet
f97c2dbd39 created ZSTD_format declaration 2017-09-21 16:07:29 -07:00
Yann Collet
da74aabc00 Merge pull request #850 from terrelln/fse-optimal
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804 Merge pull request #849 from terrelln/30
[bitstream] Allow adding 31 bits at a time
2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream()
previous result was over-estimated
by counting streaming buffers twice
2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec Merge pull request #848 from terrelln/fparams
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-18 13:48:31 -07:00
Yann Collet
539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Nick Terrell
5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Yann Collet
77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet
335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx()
64-bits fields are only 32-bits aligned on 32-bits CPU
2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
4120a7fd5a Merge pull request #837 from facebook/libzstd-nomt
makes it possible to compile libzstd in single-thread mode without zs…
2017-09-12 17:13:17 -07:00
Yann Collet
3306bcb0e6 fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676 Merge branch 'ldm-integrate' into ldm-mergeDev 2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84 fixed pass-through warning 2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61 Explicitly cast raw numerals when left-shifting 2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3 makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819) 2017-09-11 14:09:34 -07:00
Yann Collet
ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min()
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9 Move ldm functions to their own file 2017-09-06 18:09:26 -07:00
Yann Collet
baa37c3362 programs/Makefile : better support for GNU conventions
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
Yann Collet
3a12531a3d lib/Makefile : better support for GNU conventions
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
2017-09-06 16:35:49 -07:00
Yann Collet
1c7b914cdf update README on BUCK file 2017-09-06 16:23:39 -07:00
Yann Collet
36374cc3b4 update and clarify lib/README 2017-09-06 16:15:18 -07:00
Stella Lau
2b99d696de Remove debug code 2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f Merge pull request #831 from terrelln/split-compress
Split parsers out of zstd_compress.c
2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1 Fix setting of nextToUpdate at end of ldm matcher 2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9 Fix parameter handling in copyCCtx with cdict 2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29 Fix parameter handling with ZSTD_copyCCtx 2017-09-05 15:34:17 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Nick Terrell
423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
Yann Collet
8a5c0c98ae restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+) 2017-09-01 11:56:57 -07:00
Yann Collet
36aa8b5999 improved decoding speed 2017-09-01 11:40:59 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet
d963daa6a9 fixed minor warning (empty translation unit) 2017-09-01 00:12:07 -07:00
Yann Collet
3704507774 fixed decompression bug reported by @Etsukata (#828) 2017-09-01 00:05:37 -07:00
Yann Collet
369c29dd1a fixed impact of merge conflict for longRange 2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Stella Lau
6a546efb8c Add long distance matcher
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825)
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Stella Lau
90a31bfa16 Pass dictMode to ZSTDMT_initCStream; fix nits
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau
ee65701720 Minor fixes; remove formatting only changes 2017-08-29 20:27:35 -07:00
Stella Lau
a6e20e1bd7 Add test for raw content starting with dict header 2017-08-29 18:36:18 -07:00
Stella Lau
623e3cd40b Use ZSTD_dm_rawContent in zstdmt_compress 2017-08-29 18:04:32 -07:00
Stella Lau
82d636b76a Rename applyCCtxParams() 2017-08-29 18:03:06 -07:00
Stella Lau
4e835720bf Delay creation of ZSTDMT_CCtx 2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21 Localize 'dictMode' from cctx to function param 2017-08-29 15:52:24 -07:00
Yann Collet
d6ddb879da Merge pull request #817 from terrelln/pool-custom-alloc
[pool] Accept custom allocators
2017-08-29 13:05:39 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Nick Terrell
9822f97721 [error] Don't guard undef X with ifdef X 2017-08-29 11:54:38 -07:00
Stella Lau
b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
Stella Lau
0e56a84a1e Fix getting cParams from CCtxParams 2017-08-28 19:25:17 -07:00
Nick Terrell
02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell
7c365eb02c [threading] Fix ERROR macro after including windows.h 2017-08-28 16:25:02 -07:00
Bernhard M. Wiedemann
cf689b84f9 Sort input file list
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
2017-08-26 17:08:00 +02:00
Stella Lau
024098a47d Fix parameter retrieval from cdict 2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8 Fix typo with ZSTDMT_parameter 2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff Remove ZSTD_setCCtxParameter() 2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38 Add function initializing cctxParams from clevel 2017-08-25 13:36:47 -07:00
Stella Lau
9911153723 Move jobSize and overlapLog in zstdmt to cctxParams 2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d Distinguish between jobParams and cctxParams in zstdmt 2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df [zstdmt] Use POOL_create_advanced() 2017-08-24 18:12:28 -07:00
Nick Terrell
de6c6bce85 Fix zstd_internal.h for C++ mode 2017-08-24 18:09:50 -07:00
Nick Terrell
26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell
89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Stella Lau
15fdeb9e41 Enforce nbThreads<=1 for estimateCCtxSize 2017-08-24 16:28:49 -07:00
Nick Terrell
376f435914 [dictBuilder] Set default compression level to 3 2017-08-24 16:21:05 -07:00
Stella Lau
2fbf0285b2 Fix interaction with ZSTD_setCCtxParameter() and cleanup 2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516 Fix forceWindow and dictMode setting for zstdmt jobs 2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50 Ensure zstdmt uses 'job version' of cctx parameters 2017-08-23 17:03:31 -07:00
Stella Lau
1c81f725ff Remove duplicated testing code 2017-08-23 15:47:15 -07:00
Stella Lau
64ce49426b Fix cstream compression level 2017-08-23 12:30:47 -07:00
Stella Lau
5bc2c1e982 Add prototype support for customMem with cctxParams 2017-08-23 12:03:30 -07:00
Yann Collet
e9ce1208a1 Merge pull request #812 from facebook/longRangeFix
fixed extraordinary scenario where all fields use maximum nbBits
2017-08-23 11:35:28 -07:00
Yann Collet
74cde5a4d8 Merge pull request #813 from stellamplau/highbit32fix
Fix undefined behavior when srcSize==1
2017-08-23 11:31:06 -07:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Dmitriy Titarenko
20f715d709 Fix displayLevel overflow 2017-08-23 15:56:15 +05:00
Stella Lau
11303778d0 Add function to make cctxParams from ZSTD_parameters 2017-08-22 14:53:13 -07:00
Yann Collet
bd9c8ca146 Merge pull request #811 from terrelln/segmentSize
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
Stella Lau
23fc0e41fa Remove 'opaque' naming from internal functions 2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776 Remove unused functions 2017-08-22 13:33:58 -07:00
Yann Collet
6b2b6a9bd5 fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously
can only happen if windowLog>=27  (level 22 --ultra)
2017-08-22 12:09:21 -07:00
Stella Lau
e50ed1fa3a Fix undefined behavior when srcSize==1 2017-08-22 11:55:42 -07:00
Stella Lau
60e1bc617c Explicitly create a job cctxParam for multithreading 2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753 Comment out CCtx_param versions of CDict functions 2017-08-21 14:49:16 -07:00
Nick Terrell
29c2d9a4d0 [cover] Turn down notification for ZDICT subroutines 2017-08-21 14:28:31 -07:00
Nick Terrell
98de3f6847 [cover] Add dictionary size to compressed size 2017-08-21 14:23:17 -07:00
Yann Collet
78c3d16bf4 Merge pull request #809 from terrelln/dev
[cover] Fix divide by zero
2017-08-21 13:33:19 -07:00
Nick Terrell
9a54a315aa [cover] Convert score to U32 and check for zero 2017-08-21 13:30:07 -07:00
Stella Lau
fd8a25786e Check parameters are valid in initCCtxParams 2017-08-21 13:23:35 -07:00
Stella Lau
1c0dbe81b1 Add documentation for CCtx_params 2017-08-21 13:18:00 -07:00
Nick Terrell
d49eb40c03 [cover] Stop when segmentSize is less than d 2017-08-21 13:10:03 -07:00
Stella Lau
939f954285 Pass ZSTD_CCtx_params as const ptr when possible 2017-08-21 12:57:18 -07:00
Stella Lau
73c73bf16a Reduce code duplication in zstreamtest 2017-08-21 12:41:19 -07:00