Commit Graph

4752 Commits

Author SHA1 Message Date
Yann Collet
6d3987b93a
Merge pull request #905 from terrelln/skip
Allow skippable frames of any size
2017-11-01 14:36:48 -07:00
Yann Collet
428e8b3bf4 fix : ZSTD_compress_generic(,,,ZSTD_e_end) automatically sets pledgedSrcSize
as per documentation, on ZSTD_setPledgedSrcSize() :
> If all data is provided and consumed in a single round,
> this value (pledgedSrcSize) is overriden by srcSize instead.

This wasn't applied before compression level is transformed into compression parameters.
As a consequence, small input missed compression parameters adaptation.

It seems to work fine now : compression was compared with ZSTD_compress_advanced(),
results were the same.
2017-11-01 13:15:23 -07:00
Nick Terrell
e19b0822bc Test large skippable frames 2017-11-01 13:10:03 -07:00
Nick Terrell
1fc4f593da Allow skippable frames of any size 2017-11-01 13:07:26 -07:00
Yann Collet
61e5a1adfc removed direct call to malloc() from pool.c 2017-10-31 17:43:24 -07:00
Yann Collet
f73e15de33
Merge pull request #903 from terrelln/empty-input
[libzstd] Fix parameter selection for empty input
2017-10-28 17:28:08 -07:00
Yann Collet
28a40ee74c
Merge pull request #902 from terrelln/buck
Update BUCK files
2017-10-28 17:26:45 -07:00
Nick Terrell
86b8134cad [libzstd] Fix parameter selection for empty input
ZSTD_compress() and friends would treat an empty input as an unknown size
when selecting parameters. Thus, they would drastically overallocate the
context. Tell ZSTD_getParams() that the source size is 1 when it is empty.
2017-10-25 17:24:15 -07:00
Nick Terrell
b495140f67 Update BUCK files
* Correct XXH namespace (Fixes #901)
* Multithreading always enabled
* GZIP/LZ4/LZMA always enabled
* Legacy support always fully enabled
2017-10-25 12:47:57 -07:00
Yann Collet
97dccbbb2b fixed zbufftest
preserve "pledgedSrcSize=0" means "unknown" in init_advanced()
2017-10-19 14:06:02 -07:00
Yann Collet
ca78554028 Merge pull request #896 from facebook/simplifyio
fileio and bench : ZSTD_NEWAPI as the only code path
2017-10-19 13:59:59 -07:00
Yann Collet
91535d71ec fixed missing zstdmt_compress.h dependency
we lose a warning message :
when a job size is chosen < minimum job size for multithreading,
it is automatically resized to minimum size.

If this information is really useful, it should be present in zstd.h now.
2017-10-19 12:09:34 -07:00
Yann Collet
eac42534fe bench: fixed Visual warning regarding struct initialization
also :
removed dependency on zstdmt_compress.h
removed several unused macros
fileio : small code refactoring to reduce some variable scope
2017-10-19 11:56:14 -07:00
Yann Collet
d3b9547aa4 IO and bench : ZSTD_NEWAPI is the only remaining code path
removed the other 2 code paths (single thread, and ZSTDMT ones)
keeping only the new advanced API, for easier code coverage.

It shall also fix identified issue with Visual Studio
which doesn't have ZSTD_NEWAPI defined.
2017-10-18 17:01:53 -07:00
Yann Collet
9d377c6e88 Merge pull request #895 from facebook/fileSize_unknown
Distinguish 0-size from size-unavailable
2017-10-18 16:50:17 -07:00
Yann Collet
300e1df0a3 fixed wrong test to display compression status 2017-10-18 11:41:52 -07:00
Yann Collet
ca1a9ebac5 fixed zlib wrapper
it was invoking ZSTD_initCStream_advanced() with pledgedSrcSize==0 and contentSizeFlag=1
which means "empty"
while the intention was to mean "unknown".

The contentSizeFlag==1 is new, it is a consequence of setting this value to 1 by default.

The solution selected here is to pass ZSTD_CONTENTSIZE_UNKNOWN to mean "unknown".
So contentSizeFlag remains set (it wasn't in previous versions).
2017-10-18 11:22:23 -07:00
Yann Collet
1ff8a8c109 Merge pull request #891 from facebook/contentSize
Content size
2017-10-17 17:24:51 -07:00
Yann Collet
4122af1d1b Merge pull request #890 from terrelln/dev
Don't allow default tables to repeat
2017-10-17 16:34:53 -07:00
Yann Collet
5891f1d952 added a check to ensure 0-size is present in frame header 2017-10-17 16:23:20 -07:00
Yann Collet
18b795374a UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.

Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
Yann Collet
31dc6a56b2 Merge pull request #892 from terrelln/seq-test
[test] Exercise all codes in dictionary tables
2017-10-17 14:11:38 -07:00
Yann Collet
32c9f715ae fixed : Visual build compressing stdin with multi-threading enabled fails
It was multiple reasons stacked :
- Visual use a different code path, because ZSTD_NEWAPI is not defined
- fileio.c sends `0` as `pledgedSrcSize` to mean `ZSTD_CONTENTSIZE_UNKNOWN`  (fixed)
- ZSTDMT_resetCCtx() interpreted `0` as "empty" instead of "unknown" (fixed)
2017-10-17 14:07:43 -07:00
Yann Collet
c99bb00955 Merge pull request #894 from felixhandte/add-check-target
Add `check` Target to Makefile
2017-10-17 13:51:27 -07:00
W. Felix Handte
e2bf70ece1 Add check Target to Makefile 2017-10-17 16:02:39 -04:00
Nick Terrell
e600b5d0da [test] Exercise all codes in dictionary tables 2017-10-16 18:05:36 -07:00
Yann Collet
13bfe885aa edited ZSTD_initCStream_advanced() comment 2017-10-16 14:06:22 -07:00
Yann Collet
241c57adff playtest : minor cosmetic changes 2017-10-16 14:01:42 -07:00
Nick Terrell
7f961ba6cd Don't allow default tables to repeat
It isn't useful in any case to repeat default tables.
Saves a few bytes on Silesia, since we don't trigger the dictionary
heuristic.

Before: 211988480 => 73651998 bytes
After:  211988480 => 73651721 bytes
2017-10-16 11:37:56 -07:00
Yann Collet
b7977e348d Merge pull request #889 from terrelln/bug-fix
Fix invalid use of dictionary offcode table
2017-10-14 01:29:42 -07:00
Yann Collet
fc8d293460 dictionary compression use correct file size estimation
when determining compression parameters
to compress one file only.

For multiple files, it still "bets" that files are going to be small.

There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced()
making it incapable to use pledgedSrcSize to determine compression parameters.
2017-10-14 01:21:43 -07:00
Yann Collet
5eed8e7a55 changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN
to mean "pledgedSrcSize is not known at init time" instead of `0`.
Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown,
to avoid breaking 3rd party applications which depend on this behavior.
But this value is no longer recommended to mean "unknown".

In some future version, it might be possible to switch "0" to mean "empty",
as is already the case for several prototypes.
The advantage is that pledgedSrcSize field would have same behavior accross entire API,
making it easier to reason about.

Note that all concerned prototypes belong to the "experimental" API section.

srcSize is controlled at end of compression,
so if someone uses "0" to mean "unknown" while it effectively means "empty",
this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong
2017-10-14 00:32:06 -07:00
Yann Collet
9ef32b3cf1 minor : zstd -l -v display each file name 2017-10-14 00:02:32 -07:00
Yann Collet
43914f6a20 add test for presence of decompressed size field in header
previous test was "display" only, it wouldn't trigger an error in case of missing header.
2017-10-13 23:47:01 -07:00
Yann Collet
beb9b4b398 fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
Yann Collet
213ef3b510 fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
Yann Collet
3c1e3f8ec9 contentSizeFlag enabled by default would also fail for streaming and MT operations
fixed
2017-10-13 18:32:06 -07:00
Yann Collet
fb44516641 ensure fParams.contentSizeFlag starts at 1
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
Yann Collet
dd18d73e7e fileio: content size is enabled by default 2017-10-13 16:32:18 -07:00
Nick Terrell
ced6e6189c Add DEBUGLOG() that prints FSE encoding types 2017-10-13 14:55:23 -07:00
Nick Terrell
24ac2dbd2a Fix invalid use of dictionary offcode table
Fixes #888.
2017-10-13 12:47:03 -07:00
Yann Collet
df6b844b4a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-13 02:36:40 -07:00
Yann Collet
a9e5705077 minor code formatting
added a trace during sequence encoding
2017-10-13 02:36:16 -07:00
Yann Collet
5da6693fb6 Merge pull request #885 from terrelln/dev
[doc] Add images for release notes
2017-10-09 16:50:09 -07:00
Nick Terrell
6d8778d8e7 [doc] Add images for release notes 2017-10-09 16:44:31 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
Yann Collet
4252621e26 playtests: do not use cat on large files
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
2017-10-05 20:21:59 -07:00
Yann Collet
ba55dc8b5e Merge pull request #882 from terrelln/list
[zstdcli] Add window size to verbose list
2017-10-04 13:39:30 -07:00
Nick Terrell
6dd958eea2 [zstdcli] Add window size to verbose list
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None

```
2017-10-04 12:26:28 -07:00
Yann Collet
c85141caed Merge pull request #881 from terrelln/dev
Ensure dictionary Huff table can encode any symbol
2017-10-03 14:49:07 -07:00