AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,422 Commits 1 Branch 0 Tags 27 MiB
13bfe885aa
Commit Graph

4422 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
13bfe885aa edited ZSTD_initCStream_advanced() comment 2017-10-16 14:06:22 -07:00
Yann Collet
241c57adff playtest : minor cosmetic changes 2017-10-16 14:01:42 -07:00
Yann Collet
fc8d293460 dictionary compression use correct file size estimation 
when determining compression parameters
to compress one file only.

For multiple files, it still "bets" that files are going to be small.

There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced()
making it incapable to use pledgedSrcSize to determine compression parameters.
 2017-10-14 01:21:43 -07:00
Yann Collet
5eed8e7a55 changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN 
to mean "pledgedSrcSize is not known at init time" instead of `0`.
Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown,
to avoid breaking 3rd party applications which depend on this behavior.
But this value is no longer recommended to mean "unknown".

In some future version, it might be possible to switch "0" to mean "empty",
as is already the case for several prototypes.
The advantage is that pledgedSrcSize field would have same behavior accross entire API,
making it easier to reason about.

Note that all concerned prototypes belong to the "experimental" API section.

srcSize is controlled at end of compression,
so if someone uses "0" to mean "unknown" while it effectively means "empty",
this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong
 2017-10-14 00:32:06 -07:00
Yann Collet
9ef32b3cf1 minor : zstd -l -v display each file name 2017-10-14 00:02:32 -07:00
Yann Collet
43914f6a20 add test for presence of decompressed size field in header 
previous test was "display" only, it wouldn't trigger an error in case of missing header.
 2017-10-13 23:47:01 -07:00
Yann Collet
beb9b4b398 fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default 
and a wrong test in zstreamtest --mt
 2017-10-13 19:09:30 -07:00
Yann Collet
213ef3b510 fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag, 
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
 2017-10-13 19:01:58 -07:00
Yann Collet
3c1e3f8ec9 contentSizeFlag enabled by default would also fail for streaming and MT operations 
fixed
 2017-10-13 18:32:06 -07:00
Yann Collet
fb44516641 ensure fParams.contentSizeFlag starts at 1 
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
 2017-10-13 17:39:13 -07:00
Yann Collet
dd18d73e7e fileio: content size is enabled by default 2017-10-13 16:32:18 -07:00
Yann Collet
df6b844b4a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-13 02:36:40 -07:00
Yann Collet
a9e5705077 minor code formatting 
added a trace during sequence encoding
 2017-10-13 02:36:16 -07:00
Yann Collet
5da6693fb6 Merge pull request #885 from terrelln/dev 
[doc] Add images for release notes
 2017-10-09 16:50:09 -07:00
Nick Terrell
6d8778d8e7 [doc] Add images for release notes 2017-10-09 16:44:31 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus 
a compressed block must be strictly smaller than its decompressed size.
 2017-10-07 15:19:52 -07:00
Yann Collet
4252621e26 playtests: do not use cat on large files 
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
 2017-10-05 20:21:59 -07:00
Yann Collet
ba55dc8b5e Merge pull request #882 from terrelln/list 
[zstdcli] Add window size to verbose list
 2017-10-04 13:39:30 -07:00
Nick Terrell
6dd958eea2 [zstdcli] Add window size to verbose list 
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None

```
 2017-10-04 12:26:28 -07:00
Yann Collet
c85141caed Merge pull request #881 from terrelln/dev 
Ensure dictionary Huff table can encode any symbol
 2017-10-03 14:49:07 -07:00
Nick Terrell
a86a7097ec Ensure dictionary Huff table can encode any symbol 
* Ensure that the dictionary Huffman CTable has maxSymbolValue 255.
* Fix a stack buffer overflow during compression dictionary loading.
 2017-10-03 13:22:13 -07:00
Yann Collet
67478f4cb0 fixed minor conversion warnings for printf 
in debug mode
 2017-10-02 17:28:57 -07:00
Yann Collet
9b166d2291 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-02 16:34:26 -07:00
Yann Collet
3b27ed41fd Merge branch 'srcSize' into dev 2017-10-02 16:34:14 -07:00
Yann Collet
7e00df4a49 bumped version number 
and updated NEWS in anticipation for release
 2017-10-02 16:27:25 -07:00
Yann Collet
004fd34fd9 Merge pull request #876 from facebook/srcSize 
CLI Fix : srcSize written in frame headers when compressing multiple files
 2017-10-02 15:02:05 -07:00
Yann Collet
7304a63c12 Merge pull request #879 from terrelln/block-size 
[libzstd] Set CLEVEL_CUSTOM correctly
 2017-10-02 14:51:47 -07:00
Nick Terrell
86e83e926f [libzstd] Set CLEVEL_CUSTOM correctly 
In `ZSTD_compressBegin_advanced()`, `ZSTD_parameters` are used to set the
compression parameters, but the level didn't get set to `CLEVEL_CUSTOM`, so
`ZSTD_compressBlock()` used the wrong parameters when checking the source
size.
 2017-10-02 13:43:30 -07:00
Yann Collet
4946993f87 removed isRegularFile parameter 
no longer useful : size of src is determined for each file.
 2017-10-02 12:29:25 -07:00
Yann Collet
7f580f9ee8 interruption handler and variable are static 2017-10-02 11:39:05 -07:00
Yann Collet
0d58aaf6f0 /contrib: fixed license header 
removed last reference to PATENTS file
 2017-10-02 02:07:17 -07:00
Yann Collet
fe5444bc66 removed the statement for all versions of Visual Studio 2017-10-02 02:02:16 -07:00
Yann Collet
51d82d5516 same error in Visual Studio 2012 ... 2017-10-02 01:12:40 -07:00
Yann Collet
ed7ae4c9bd The issue also impacts Visual Studio 2010 2017-10-02 00:45:28 -07:00
Yann Collet
6e7ba3df2f added (void)sig to avoid compilers complaining that sig is not used. 2017-10-02 00:19:47 -07:00
Yann Collet
82bc200f82 conditionnally removed invocation that generates a buggy warning with Visual Studio 2008 2017-10-02 00:02:24 -07:00
Yann Collet
5e4ad557bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-01 15:33:57 -07:00
Yann Collet
bd18095edc blindfix for Visual : minor casting issue 
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
 2017-10-01 15:32:48 -07:00
Yann Collet
b600b5bafd Merge pull request #877 from facebook/compressBound 
added ZSTD_COMPRESSBOUND() as a macro
 2017-10-01 14:59:38 -07:00
Yann Collet
00fc1ba8dd cli: add Ctrl-C support, requested by @mike155 in #854 
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
 2017-10-01 12:10:26 -07:00
Yann Collet
5db19b8685 added comment on ZSTD_COMPRESSBOUND() 
as requested by @terrelln
 2017-10-01 11:32:38 -07:00
Yann Collet
6e930c13d1 Merge branch 'dev' into compressBound 2017-10-01 11:24:02 -07:00
Yann Collet
76ac0b2d99 macro compatible with scenario where windowSize = 1024 (minimum) 2017-09-30 15:34:44 -07:00
Yann Collet
dc404119e5 ZSTD_adjustCParams_internal : minor optimization 2017-09-30 15:02:40 -07:00
Yann Collet
96bb29aa14 Merge pull request #878 from terrelln/adjust 
Don't `size -= 1` in ZSTD_adjustCParams()
 2017-09-30 14:52:54 -07:00
Nick Terrell
c5d6dde502 Don't size -= 1 in ZSTD_adjustCParams() 
The window size could end up too small if the source size is 2^n + 1.

Credit to OSS-Fuzz
 2017-09-30 14:20:06 -07:00
Yann Collet
ee1ed78fcb fix proper naming on FSE_createCTable() arguments in fse.h 2017-09-30 11:08:50 -07:00
Yann Collet
5b10345b26 added ZSTD_COMPRESSBOUND() as a macro 
ZSTD_compressBound() works fine, but is only useful for dynamic allocation.
For static allocation, only a macro can provide the amount during compilation time.
 2017-09-29 23:17:41 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests 
Add Default LZ4 Support When Available
 2017-09-29 22:32:54 -07:00
Yann Collet
8afb151c9b cli: fixed wrong initialization in MT mode 
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.

Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.

This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.

At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
 2017-09-29 22:14:37 -07:00