AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,357 Commits 1 Branch 0 Tags 27 MiB
86b4fe5b45
Commit Graph

1865 Commits

Author SHA1 Message Date
Yann Collet
86b4fe5b45 adjustCParams : restored previous behavior 
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
 2017-09-28 18:14:28 -07:00
Yann Collet
e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet
8074261d00 zstdmt : move on when not enough memory for a new input buffer 
just continue operations without input forward progress,
instead of an error that stops current compression session.
 2017-09-28 11:46:19 -07:00
Yann Collet
2cd15dd9a4 fixed minor Visual conversion warning 2017-09-28 02:33:41 -07:00
Yann Collet
377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error 
wait for all jobs to be completed, so that freeing can happen safely
 2017-09-28 02:23:44 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
e45a2aea9b Merge pull request #869 from terrelln/dev 
[libzstd] pthread function prefixed with ZSTD_
 2017-09-27 16:35:08 -07:00
Nick Terrell
b555b7ef41 [libzstd][opt] Simplify repcode logic 2017-09-27 15:30:12 -07:00
Yann Collet
ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal() 
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
 2017-09-27 13:51:05 -07:00
Yann Collet
c994932788 fixed ZSTD_format_e value validation 2017-09-27 12:22:22 -07:00
Nick Terrell
6c41adfb28 [libzstd] pthread function prefixed with ZSTD_ 
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
 2017-09-27 11:48:48 -07:00
Yann Collet
ecf1778e23 updated ZSTD_format_e value validation 
also updated manual
 2017-09-27 11:19:21 -07:00
Yann Collet
9416195221 changed error code when pos<=size condition is not respected 
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
 2017-09-27 10:35:56 -07:00
Yann Collet
d56a350402 removed unsupported formats 2017-09-27 10:29:31 -07:00
Yann Collet
ca306c1c84 fixed a bug in zstreamtest 
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
 2017-09-27 00:39:41 -07:00
Yann Collet
cd53ac831b fixed DCtx initialization error 
now relying on initialization of dctx->format first
 2017-09-26 18:26:09 -07:00
Yann Collet
4791561c4a silence minor gcc warning -Wempty-body 
also silence fuzz test artefacts
 2017-09-26 17:57:38 -07:00
Yann Collet
c0dd960363 switch assert() position 2017-09-26 15:36:57 -07:00
Yann Collet
319c699991 created ZSTD_startingInputLength() 
as suggested by @terrelln
 2017-09-26 15:36:14 -07:00
Yann Collet
8d1e97ea9c minor fixes following @terrelln comments 2017-09-26 15:06:30 -07:00
Yann Collet
df4e9bba25 fixed constant errors for gcc in c99 mode 
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
 2017-09-26 14:31:06 -07:00
Yann Collet
9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78 Merge pull request #862 from terrelln/static 
[zstd] Backport kernel patch from @ColinIanKing
 2017-09-25 17:02:40 -07:00
Yann Collet
52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff 
[libzstd] Increase MaxOff
 2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Yann Collet
f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet
6ee05a02b8 added ZSTD_decompress_generic() 
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
 2017-09-25 15:41:48 -07:00
Yann Collet
b8d4a3887f introduced constant ZSTD_frameIdSize 
within zstd_internal.h
This is the size of magic number.

Avoids using `4` directly in source code, which is a bit less meaningful.
 2017-09-25 15:26:18 -07:00
Yann Collet
044fb4c057 implemented magic-less frame decoder 2017-09-25 15:12:09 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Yann Collet
96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce added prototypes for advanced parameters for decompression API 
required to decode custom formats
 2017-09-24 15:57:29 -07:00
Yann Collet
e60f48c549 Merge branch 'dev' into newFormats 2017-09-24 14:33:37 -07:00
Yann Collet
8977224b9b Merge pull request #859 from terrelln/31 
Prepare for ZSTD_WINDOWLOG_MAX == 31
 2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Yann Collet
645563583e Merge branch 'dev' into newFormats 2017-09-21 16:08:06 -07:00
Yann Collet
f97c2dbd39 created ZSTD_format declaration 2017-09-21 16:07:29 -07:00
Yann Collet
da74aabc00 Merge pull request #850 from terrelln/fse-optimal 
[fse] Fix FSE_optimalTableLog() for srcSize==1
 2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804 Merge pull request #849 from terrelln/30 
[bitstream] Allow adding 31 bits at a time
 2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00