Yann Collet
377abcc02c
zstdmt : better behavior when freeing a context right after a memory allocation error
...
wait for all jobs to be completed, so that freeing can happen safely
2017-09-28 02:23:44 -07:00
Yann Collet
d6770f80af
minor : rewrite unit tests using CHECK_Z macro
2017-09-28 02:14:48 -07:00
Yann Collet
9b5b47ac93
ensure adjustCParams adjust hLog and cLog even without srcSize
...
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.
Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
Yann Collet
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
Yann Collet
e45a2aea9b
Merge pull request #869 from terrelln/dev
...
[libzstd] pthread function prefixed with ZSTD_
2017-09-27 16:35:08 -07:00
Nick Terrell
b555b7ef41
[libzstd][opt] Simplify repcode logic
2017-09-27 15:30:12 -07:00
Yann Collet
ea1f50bf73
removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
...
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.
Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.
A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
Yann Collet
c994932788
fixed ZSTD_format_e value validation
2017-09-27 12:22:22 -07:00
Nick Terrell
6c41adfb28
[libzstd] pthread function prefixed with ZSTD_
...
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
Yann Collet
ecf1778e23
updated ZSTD_format_e value validation
...
also updated manual
2017-09-27 11:19:21 -07:00
Yann Collet
9416195221
changed error code when pos<=size condition is not respected
...
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
2017-09-27 10:35:56 -07:00
Yann Collet
d56a350402
removed unsupported formats
2017-09-27 10:29:31 -07:00
Yann Collet
ca306c1c84
fixed a bug in zstreamtest
...
decoder output buffer would receive a wrong size.
In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.
This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
Yann Collet
cd53ac831b
fixed DCtx initialization error
...
now relying on initialization of dctx->format first
2017-09-26 18:26:09 -07:00
Yann Collet
4791561c4a
silence minor gcc warning -Wempty-body
...
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
Yann Collet
c0dd960363
switch assert() position
2017-09-26 15:36:57 -07:00
Yann Collet
319c699991
created ZSTD_startingInputLength()
...
as suggested by @terrelln
2017-09-26 15:36:14 -07:00
Yann Collet
8d1e97ea9c
minor fixes following @terrelln comments
2017-09-26 15:06:30 -07:00
Yann Collet
df4e9bba25
fixed constant errors for gcc in c99 mode
...
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
Yann Collet
9f0b8dfbe9
Merge branch 'dev' into newFormats
2017-09-26 14:22:39 -07:00
Nick Terrell
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78
Merge pull request #862 from terrelln/static
...
[zstd] Backport kernel patch from @ColinIanKing
2017-09-25 17:02:40 -07:00
Yann Collet
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
Yann Collet
5d8fdd1641
Merge pull request #855 from terrelln/maxoff
...
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085
[zstd] Backport kernel patch from @ColinIanKing
...
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com .
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED ).
* Benchmarks showed no difference in speed.
2017-09-25 16:18:23 -07:00
Yann Collet
f2a913862c
added ZSTD_decompress_generic_simpleArgs()
2017-09-25 15:46:34 -07:00
Yann Collet
6ee05a02b8
added ZSTD_decompress_generic()
...
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
Yann Collet
b8d4a3887f
introduced constant ZSTD_frameIdSize
...
within zstd_internal.h
This is the size of magic number.
Avoids using `4` directly in source code, which is a bit less meaningful.
2017-09-25 15:26:18 -07:00
Yann Collet
044fb4c057
implemented magic-less frame decoder
2017-09-25 15:12:09 -07:00
Yann Collet
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef
[libzstd] Increase MaxOff
2017-09-25 13:36:18 -07:00
Yann Collet
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce
added prototypes for advanced parameters for decompression API
...
required to decode custom formats
2017-09-24 15:57:29 -07:00
Yann Collet
e60f48c549
Merge branch 'dev' into newFormats
2017-09-24 14:33:37 -07:00
Yann Collet
8977224b9b
Merge pull request #859 from terrelln/31
...
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284
added control from frame content size at end of decompression
...
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
Yann Collet
645563583e
Merge branch 'dev' into newFormats
2017-09-21 16:08:06 -07:00
Yann Collet
f97c2dbd39
created ZSTD_format declaration
2017-09-21 16:07:29 -07:00
Yann Collet
da74aabc00
Merge pull request #850 from terrelln/fse-optimal
...
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804
Merge pull request #849 from terrelln/30
...
[bitstream] Allow adding 31 bits at a time
2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43
[bitstream] Allow adding 31 bits at a time
2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676
[ldm] Fix corner case where minMatch < 8
...
There is a potential read buffer overflow when minMatch < 8.
fix-fuzz-failure
2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff
[libzstd] Fix bad window size assert
...
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.
fix-fuzz-failure
2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b
fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream()
...
previous result was over-estimated
by counting streaming buffers twice
2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec
Merge pull request #848 from terrelln/fparams
...
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-18 13:48:31 -07:00
Yann Collet
539b91ee9b
minor : added assert in bt
2017-09-16 23:41:58 -07:00
Nick Terrell
5f22479517
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-15 17:37:20 -07:00
Yann Collet
77c137b3ae
minor comment refactor
2017-09-14 15:12:57 -07:00
Yann Collet
335780c427
fixed too strong alignment assert in ZSTD_initStaticCCtx()
...
64-bits fields are only 32-bits aligned on 32-bits CPU
2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
4120a7fd5a
Merge pull request #837 from facebook/libzstd-nomt
...
makes it possible to compile libzstd in single-thread mode without zs…
2017-09-12 17:13:17 -07:00
Yann Collet
3306bcb0e6
fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic
...
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676
Merge branch 'ldm-integrate' into ldm-mergeDev
2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84
fixed pass-through warning
2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61
Explicitly cast raw numerals when left-shifting
2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3
makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c ( #819 )
2017-09-11 14:09:34 -07:00
Yann Collet
ce31004f20
fix following suggestions by @terrelln
2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9
Move ldm functions to their own file
2017-09-06 18:09:26 -07:00
Yann Collet
baa37c3362
programs/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
Yann Collet
3a12531a3d
lib/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
2017-09-06 16:35:49 -07:00
Yann Collet
1c7b914cdf
update README on BUCK file
2017-09-06 16:23:39 -07:00
Yann Collet
36374cc3b4
update and clarify lib/README
2017-09-06 16:15:18 -07:00
Stella Lau
2b99d696de
Remove debug code
2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f
Merge pull request #831 from terrelln/split-compress
...
Split parsers out of zstd_compress.c
2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395
Rename and add short ldm parameters in cli
2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1
Fix setting of nextToUpdate at end of ldm matcher
2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688
Split parsers out of zstd_compress.c
2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9
Fix parameter handling in copyCCtx with cdict
2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29
Fix parameter handling with ZSTD_copyCCtx
2017-09-05 15:34:17 -07:00
Stella Lau
643d28c701
Add ldm options to 'man zstd'
2017-09-05 11:27:15 -07:00
Nick Terrell
423b133568
[POOL] Allow free on NULL when multithreading is disabled
2017-09-05 11:18:13 -07:00
Stella Lau
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
Yann Collet
8a5c0c98ae
restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+)
2017-09-01 11:56:57 -07:00
Yann Collet
36aa8b5999
improved decoding speed
2017-09-01 11:40:59 -07:00
Stella Lau
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
Yann Collet
d963daa6a9
fixed minor warning (empty translation unit)
2017-09-01 00:12:07 -07:00
Yann Collet
3704507774
fixed decompression bug reported by @Etsukata ( #828 )
2017-09-01 00:05:37 -07:00
Yann Collet
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
Stella Lau
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
Stella Lau
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau
ee65701720
Minor fixes; remove formatting only changes
2017-08-29 20:27:35 -07:00
Stella Lau
a6e20e1bd7
Add test for raw content starting with dict header
2017-08-29 18:36:18 -07:00
Stella Lau
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
Stella Lau
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
Stella Lau
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
Yann Collet
d6ddb879da
Merge pull request #817 from terrelln/pool-custom-alloc
...
[pool] Accept custom allocators
2017-08-29 13:05:39 -07:00
Stella Lau
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
Nick Terrell
9822f97721
[error] Don't guard undef X with ifdef X
2017-08-29 11:54:38 -07:00
Stella Lau
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
Stella Lau
0e56a84a1e
Fix getting cParams from CCtxParams
2017-08-28 19:25:17 -07:00
Nick Terrell
02033be08c
[pool] Visual Studios disallows empty structs
2017-08-28 17:19:01 -07:00
Nick Terrell
7c365eb02c
[threading] Fix ERROR macro after including windows.h
2017-08-28 16:25:02 -07:00
Bernhard M. Wiedemann
cf689b84f9
Sort input file list
...
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
2017-08-26 17:08:00 +02:00
Stella Lau
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
Stella Lau
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
Nick Terrell
de6c6bce85
Fix zstd_internal.h for C++ mode
2017-08-24 18:09:50 -07:00
Nick Terrell
26dc040a7b
[pool] Accept custom allocators
2017-08-24 17:01:41 -07:00
Nick Terrell
89dc856cae
[pool] Fix formatting
2017-08-24 16:48:32 -07:00
Stella Lau
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
Nick Terrell
376f435914
[dictBuilder] Set default compression level to 3
2017-08-24 16:21:05 -07:00
Stella Lau
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
Stella Lau
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
Stella Lau
64ce49426b
Fix cstream compression level
2017-08-23 12:30:47 -07:00
Stella Lau
5bc2c1e982
Add prototype support for customMem with cctxParams
2017-08-23 12:03:30 -07:00
Yann Collet
e9ce1208a1
Merge pull request #812 from facebook/longRangeFix
...
fixed extraordinary scenario where all fields use maximum nbBits
2017-08-23 11:35:28 -07:00
Yann Collet
74cde5a4d8
Merge pull request #813 from stellamplau/highbit32fix
...
Fix undefined behavior when srcSize==1
2017-08-23 11:31:06 -07:00
Stella Lau
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
Dmitriy Titarenko
20f715d709
Fix displayLevel overflow
2017-08-23 15:56:15 +05:00
Stella Lau
11303778d0
Add function to make cctxParams from ZSTD_parameters
2017-08-22 14:53:13 -07:00
Yann Collet
bd9c8ca146
Merge pull request #811 from terrelln/segmentSize
...
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
Stella Lau
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
Yann Collet
6b2b6a9bd5
fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously
...
can only happen if windowLog>=27 (level 22 --ultra)
2017-08-22 12:09:21 -07:00
Stella Lau
e50ed1fa3a
Fix undefined behavior when srcSize==1
2017-08-22 11:55:42 -07:00
Stella Lau
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
Nick Terrell
29c2d9a4d0
[cover] Turn down notification for ZDICT subroutines
2017-08-21 14:28:31 -07:00
Nick Terrell
98de3f6847
[cover] Add dictionary size to compressed size
2017-08-21 14:23:17 -07:00
Yann Collet
78c3d16bf4
Merge pull request #809 from terrelln/dev
...
[cover] Fix divide by zero
2017-08-21 13:33:19 -07:00
Nick Terrell
9a54a315aa
[cover] Convert score to U32 and check for zero
2017-08-21 13:30:07 -07:00
Stella Lau
fd8a25786e
Check parameters are valid in initCCtxParams
2017-08-21 13:23:35 -07:00
Stella Lau
1c0dbe81b1
Add documentation for CCtx_params
2017-08-21 13:18:00 -07:00
Nick Terrell
d49eb40c03
[cover] Stop when segmentSize is less than d
2017-08-21 13:10:03 -07:00
Stella Lau
939f954285
Pass ZSTD_CCtx_params as const ptr when possible
2017-08-21 12:57:18 -07:00
Stella Lau
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00