AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,494 Commits 1 Branch 0 Tags 27 MiB
d100670045
Commit Graph

666 Commits

Author SHA1 Message Date
Yann Collet
1416bc0f07 erase existence of a buffer when it's sent out of the pool 
In some complex scenario,
the buffer would be freed because it's too large,
another buffer would be allocated, but fail,
trigger an error,
and the general buffer pool would then be freed,
where the definition of the already freed buffer would be found
(beyond total index, but still), and freed again, resulting in double-free error.
 2017-09-29 16:27:47 -07:00
Yann Collet
e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage 
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
 2017-09-28 23:01:31 -07:00
Yann Collet
754ae5cc0b removed ZSTDMT_waitForAllJobsCompleted() from ZSTDMT_freeCCtx() 
as per @terrelln comment
 2017-09-28 20:45:31 -07:00
Yann Collet
86b4fe5b45 adjustCParams : restored previous behavior 
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
 2017-09-28 18:14:28 -07:00
Yann Collet
b93598d6a4 zstdmt : reduced maximum nb of threads 
to avoid memory address space issues on 32-bits systems
(see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876416#17)
 2017-09-28 13:49:12 -07:00
Yann Collet
e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet
8074261d00 zstdmt : move on when not enough memory for a new input buffer 
just continue operations without input forward progress,
instead of an error that stops current compression session.
 2017-09-28 11:46:19 -07:00
Yann Collet
2cd15dd9a4 fixed minor Visual conversion warning 2017-09-28 02:33:41 -07:00
Yann Collet
377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error 
wait for all jobs to be completed, so that freeing can happen safely
 2017-09-28 02:23:44 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
e45a2aea9b Merge pull request #869 from terrelln/dev 
[libzstd] pthread function prefixed with ZSTD_
 2017-09-27 16:35:08 -07:00
Nick Terrell
b555b7ef41 [libzstd][opt] Simplify repcode logic 2017-09-27 15:30:12 -07:00
Yann Collet
c994932788 fixed ZSTD_format_e value validation 2017-09-27 12:22:22 -07:00
Nick Terrell
6c41adfb28 [libzstd] pthread function prefixed with ZSTD_ 
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
 2017-09-27 11:48:48 -07:00
Yann Collet
ecf1778e23 updated ZSTD_format_e value validation 
also updated manual
 2017-09-27 11:19:21 -07:00
Yann Collet
4791561c4a silence minor gcc warning -Wempty-body 
also silence fuzz test artefacts
 2017-09-26 17:57:38 -07:00
Yann Collet
9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78 Merge pull request #862 from terrelln/static 
[zstd] Backport kernel patch from @ColinIanKing
 2017-09-25 17:02:40 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff 
[libzstd] Increase MaxOff
 2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Yann Collet
6ee05a02b8 added ZSTD_decompress_generic() 
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
 2017-09-25 15:41:48 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Yann Collet
96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce added prototypes for advanced parameters for decompression API 
required to decode custom formats
 2017-09-24 15:57:29 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
da74aabc00 Merge pull request #850 from terrelln/fse-optimal 
[fse] Fix FSE_optimalTableLog() for srcSize==1
 2017-09-19 14:59:21 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Yann Collet
539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Yann Collet
335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx() 
64-bits fields are only 32-bits aligned on 32-bits CPU
 2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet
3306bcb0e6 fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic 
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
 2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676 Merge branch 'ldm-integrate' into ldm-mergeDev 2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84 fixed pass-through warning 2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61 Explicitly cast raw numerals when left-shifting 2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3 makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819) 2017-09-11 14:09:34 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9 Move ldm functions to their own file 2017-09-06 18:09:26 -07:00
Stella Lau
2b99d696de Remove debug code 2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f Merge pull request #831 from terrelln/split-compress 
Split parsers out of zstd_compress.c
 2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1 Fix setting of nextToUpdate at end of ldm matcher 2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9 Fix parameter handling in copyCCtx with cdict 2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29 Fix parameter handling with ZSTD_copyCCtx 2017-09-05 15:34:17 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet
369c29dd1a fixed impact of merge conflict for longRange 2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Stella Lau
6a546efb8c Add long distance matcher 
Move last literals section to ZSTD_block_internal
 2017-08-31 12:53:19 -07:00
Stella Lau
90a31bfa16 Pass dictMode to ZSTDMT_initCStream; fix nits 
- Return error code in estimate{CCtx,CStream}Size functions
 2017-08-30 16:19:07 -07:00
Stella Lau
ee65701720 Minor fixes; remove formatting only changes 2017-08-29 20:27:35 -07:00
Stella Lau
a6e20e1bd7 Add test for raw content starting with dict header 2017-08-29 18:36:18 -07:00
Stella Lau
623e3cd40b Use ZSTD_dm_rawContent in zstdmt_compress 2017-08-29 18:04:32 -07:00
Stella Lau
82d636b76a Rename applyCCtxParams() 2017-08-29 18:03:06 -07:00
Stella Lau
4e835720bf Delay creation of ZSTDMT_CCtx 2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21 Localize 'dictMode' from cctx to function param 2017-08-29 15:52:24 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Stella Lau
b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
Stella Lau
0e56a84a1e Fix getting cParams from CCtxParams 2017-08-28 19:25:17 -07:00
Stella Lau
024098a47d Fix parameter retrieval from cdict 2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8 Fix typo with ZSTDMT_parameter 2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff Remove ZSTD_setCCtxParameter() 2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38 Add function initializing cctxParams from clevel 2017-08-25 13:36:47 -07:00
Stella Lau
9911153723 Move jobSize and overlapLog in zstdmt to cctxParams 2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d Distinguish between jobParams and cctxParams in zstdmt 2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df [zstdmt] Use POOL_create_advanced() 2017-08-24 18:12:28 -07:00
Stella Lau
15fdeb9e41 Enforce nbThreads<=1 for estimateCCtxSize 2017-08-24 16:28:49 -07:00
Stella Lau
2fbf0285b2 Fix interaction with ZSTD_setCCtxParameter() and cleanup 2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516 Fix forceWindow and dictMode setting for zstdmt jobs 2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50 Ensure zstdmt uses 'job version' of cctx parameters 2017-08-23 17:03:31 -07:00
Stella Lau
1c81f725ff Remove duplicated testing code 2017-08-23 15:47:15 -07:00
Stella Lau
64ce49426b Fix cstream compression level 2017-08-23 12:30:47 -07:00
Stella Lau
5bc2c1e982 Add prototype support for customMem with cctxParams 2017-08-23 12:03:30 -07:00
Yann Collet
e9ce1208a1 Merge pull request #812 from facebook/longRangeFix 
fixed extraordinary scenario where all fields use maximum nbBits
 2017-08-23 11:35:28 -07:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Stella Lau
11303778d0 Add function to make cctxParams from ZSTD_parameters 2017-08-22 14:53:13 -07:00
Stella Lau
23fc0e41fa Remove 'opaque' naming from internal functions 2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776 Remove unused functions 2017-08-22 13:33:58 -07:00
Yann Collet
6b2b6a9bd5 fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously 
can only happen if windowLog>=27  (level 22 --ultra)
 2017-08-22 12:09:21 -07:00
Stella Lau
e50ed1fa3a Fix undefined behavior when srcSize==1 2017-08-22 11:55:42 -07:00
Stella Lau
60e1bc617c Explicitly create a job cctxParam for multithreading 2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753 Comment out CCtx_param versions of CDict functions 2017-08-21 14:49:16 -07:00
Stella Lau
fd8a25786e Check parameters are valid in initCCtxParams 2017-08-21 13:23:35 -07:00
Stella Lau
1c0dbe81b1 Add documentation for CCtx_params 2017-08-21 13:18:00 -07:00
Stella Lau
939f954285 Pass ZSTD_CCtx_params as const ptr when possible 2017-08-21 12:57:18 -07:00
Stella Lau
560b34f6d2 Return error code when initializing NULL cctxParams 2017-08-21 11:52:26 -07:00
Stella Lau
25be09c6b4 Set some parameters to zero before initializing cdict 2017-08-21 11:35:46 -07:00