AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,444 Commits 1 Branch 0 Tags 27 MiB
86b8134cad
Commit Graph

4444 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
4946993f87 removed isRegularFile parameter 
no longer useful : size of src is determined for each file.
 2017-10-02 12:29:25 -07:00
Yann Collet
7f580f9ee8 interruption handler and variable are static 2017-10-02 11:39:05 -07:00
Yann Collet
0d58aaf6f0 /contrib: fixed license header 
removed last reference to PATENTS file
 2017-10-02 02:07:17 -07:00
Yann Collet
fe5444bc66 removed the statement for all versions of Visual Studio 2017-10-02 02:02:16 -07:00
Yann Collet
51d82d5516 same error in Visual Studio 2012 ... 2017-10-02 01:12:40 -07:00
Yann Collet
ed7ae4c9bd The issue also impacts Visual Studio 2010 2017-10-02 00:45:28 -07:00
Yann Collet
6e7ba3df2f added (void)sig to avoid compilers complaining that sig is not used. 2017-10-02 00:19:47 -07:00
Yann Collet
82bc200f82 conditionnally removed invocation that generates a buggy warning with Visual Studio 2008 2017-10-02 00:02:24 -07:00
Yann Collet
5e4ad557bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-01 15:33:57 -07:00
Yann Collet
bd18095edc blindfix for Visual : minor casting issue 
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
 2017-10-01 15:32:48 -07:00
Yann Collet
b600b5bafd Merge pull request #877 from facebook/compressBound 
added ZSTD_COMPRESSBOUND() as a macro
 2017-10-01 14:59:38 -07:00
Yann Collet
00fc1ba8dd cli: add Ctrl-C support, requested by @mike155 in #854 
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
 2017-10-01 12:10:26 -07:00
Yann Collet
5db19b8685 added comment on ZSTD_COMPRESSBOUND() 
as requested by @terrelln
 2017-10-01 11:32:38 -07:00
Yann Collet
6e930c13d1 Merge branch 'dev' into compressBound 2017-10-01 11:24:02 -07:00
Yann Collet
76ac0b2d99 macro compatible with scenario where windowSize = 1024 (minimum) 2017-09-30 15:34:44 -07:00
Yann Collet
dc404119e5 ZSTD_adjustCParams_internal : minor optimization 2017-09-30 15:02:40 -07:00
Yann Collet
96bb29aa14 Merge pull request #878 from terrelln/adjust 
Don't `size -= 1` in ZSTD_adjustCParams()
 2017-09-30 14:52:54 -07:00
Nick Terrell
c5d6dde502 Don't size -= 1 in ZSTD_adjustCParams() 
The window size could end up too small if the source size is 2^n + 1.

Credit to OSS-Fuzz
 2017-09-30 14:20:06 -07:00
Yann Collet
ee1ed78fcb fix proper naming on FSE_createCTable() arguments in fse.h 2017-09-30 11:08:50 -07:00
Yann Collet
5b10345b26 added ZSTD_COMPRESSBOUND() as a macro 
ZSTD_compressBound() works fine, but is only useful for dynamic allocation.
For static allocation, only a macro can provide the amount during compilation time.
 2017-09-29 23:17:41 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests 
Add Default LZ4 Support When Available
 2017-09-29 22:32:54 -07:00
Yann Collet
8afb151c9b cli: fixed wrong initialization in MT mode 
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.

Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.

This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.

At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
 2017-09-29 22:14:37 -07:00
Yann Collet
657946a913 Merge pull request #875 from facebook/zstdcat 
decode more data before triggering error
 2017-09-29 20:12:25 -07:00
Yann Collet
fbd5ab7027 minor fix : no longer use fake srcSize during resource creation 
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.

At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
 2017-09-29 19:40:27 -07:00
Yann Collet
db1668a43b fix : srcSize written in frame header when multiple files compressed 
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
 2017-09-29 18:05:18 -07:00
Yann Collet
7c9669f272 Merge pull request #873 from facebook/shorterTests 
Leaner tests
 2017-09-29 17:26:46 -07:00
Yann Collet
1416bc0f07 erase existence of a buffer when it's sent out of the pool 
In some complex scenario,
the buffer would be freed because it's too large,
another buffer would be allocated, but fail,
trigger an error,
and the general buffer pool would then be freed,
where the definition of the already freed buffer would be found
(beyond total index, but still), and freed again, resulting in double-free error.
 2017-09-29 16:27:47 -07:00
Yann Collet
8afcc80e07 decode more data before triggering error 
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
 2017-09-29 15:54:09 -07:00
Yann Collet
e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage 
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
 2017-09-28 23:01:31 -07:00
Yann Collet
754ae5cc0b removed ZSTDMT_waitForAllJobsCompleted() from ZSTDMT_freeCCtx() 
as per @terrelln comment
 2017-09-28 20:45:31 -07:00
Yann Collet
e0065cf660 make test : removed zstreamtest unit tests for variants 
slightly reduced time to create dictionary
at beginning of unit tests
 2017-09-28 18:34:38 -07:00
Yann Collet
47c6a95d07 zstreamtest : run unit tests only during "normal" session 
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
 2017-09-28 18:27:22 -07:00
Yann Collet
86b4fe5b45 adjustCParams : restored previous behavior 
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
 2017-09-28 18:14:28 -07:00
W. Felix Handte
dc27c36495 Update documentation to reflect other format support 2017-09-28 19:43:12 -04:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
W. Felix Handte
5705d9f25a Add basic tests for the lz4 integration 2017-09-28 19:16:43 -04:00
Yann Collet
b93598d6a4 zstdmt : reduced maximum nb of threads 
to avoid memory address space issues on 32-bits systems
(see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876416#17)
 2017-09-28 13:49:12 -07:00
Yann Collet
e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet
bbef058ae6 zstreamtest --newapi : reduced maximum allocated memory 2017-09-28 11:48:45 -07:00
Yann Collet
8074261d00 zstdmt : move on when not enough memory for a new input buffer 
just continue operations without input forward progress,
instead of an error that stops current compression session.
 2017-09-28 11:46:19 -07:00
Yann Collet
2cd15dd9a4 fixed minor Visual conversion warning 2017-09-28 02:33:41 -07:00
Yann Collet
377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error 
wait for all jobs to be completed, so that freeing can happen safely
 2017-09-28 02:23:44 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet
824f75ea7c Merge pull request #863 from facebook/newFormats 
magicless frames (#591)
 2017-09-28 00:32:16 -07:00
Yann Collet
fac8c70859 Merge pull request #872 from terrelln/fuzz-minor 
[fuzz] Small changes for oss-fuzz integration
 2017-09-27 19:12:18 -07:00
Nick Terrell
d9c1e9125f [fuzz] Small changes for oss-fuzz integration 2017-09-27 18:23:06 -07:00
Yann Collet
aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet
bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00