AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,682 Commits 1 Branch 0 Tags 27 MiB
58dd7de640
Commit Graph

1157 Commits

Author SHA1 Message Date
Yann Collet
300e1df0a3 fixed wrong test to display compression status 2017-10-18 11:41:52 -07:00
Yann Collet
18b795374a UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure 
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.

Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
 2017-10-17 16:14:25 -07:00
Yann Collet
32c9f715ae fixed : Visual build compressing stdin with multi-threading enabled fails 
It was multiple reasons stacked :
- Visual use a different code path, because ZSTD_NEWAPI is not defined
- fileio.c sends `0` as `pledgedSrcSize` to mean `ZSTD_CONTENTSIZE_UNKNOWN`  (fixed)
- ZSTDMT_resetCCtx() interpreted `0` as "empty" instead of "unknown" (fixed)
 2017-10-17 14:07:43 -07:00
Yann Collet
fc8d293460 dictionary compression use correct file size estimation 
when determining compression parameters
to compress one file only.

For multiple files, it still "bets" that files are going to be small.

There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced()
making it incapable to use pledgedSrcSize to determine compression parameters.
 2017-10-14 01:21:43 -07:00
Yann Collet
9ef32b3cf1 minor : zstd -l -v display each file name 2017-10-14 00:02:32 -07:00
Yann Collet
dd18d73e7e fileio: content size is enabled by default 2017-10-13 16:32:18 -07:00
Nick Terrell
6dd958eea2 [zstdcli] Add window size to verbose list 
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None

```
 2017-10-04 12:26:28 -07:00
Yann Collet
3b27ed41fd Merge branch 'srcSize' into dev 2017-10-02 16:34:14 -07:00
Yann Collet
4946993f87 removed isRegularFile parameter 
no longer useful : size of src is determined for each file.
 2017-10-02 12:29:25 -07:00
Yann Collet
7f580f9ee8 interruption handler and variable are static 2017-10-02 11:39:05 -07:00
Yann Collet
fe5444bc66 removed the statement for all versions of Visual Studio 2017-10-02 02:02:16 -07:00
Yann Collet
51d82d5516 same error in Visual Studio 2012 ... 2017-10-02 01:12:40 -07:00
Yann Collet
ed7ae4c9bd The issue also impacts Visual Studio 2010 2017-10-02 00:45:28 -07:00
Yann Collet
6e7ba3df2f added (void)sig to avoid compilers complaining that sig is not used. 2017-10-02 00:19:47 -07:00
Yann Collet
82bc200f82 conditionnally removed invocation that generates a buggy warning with Visual Studio 2008 2017-10-02 00:02:24 -07:00
Yann Collet
bd18095edc blindfix for Visual : minor casting issue 
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
 2017-10-01 15:32:48 -07:00
Yann Collet
00fc1ba8dd cli: add Ctrl-C support, requested by @mike155 in #854 
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
 2017-10-01 12:10:26 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests 
Add Default LZ4 Support When Available
 2017-09-29 22:32:54 -07:00
Yann Collet
8afb151c9b cli: fixed wrong initialization in MT mode 
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.

Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.

This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.

At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
 2017-09-29 22:14:37 -07:00
Yann Collet
fbd5ab7027 minor fix : no longer use fake srcSize during resource creation 
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.

At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
 2017-09-29 19:40:27 -07:00
Yann Collet
db1668a43b fix : srcSize written in frame header when multiple files compressed 
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
 2017-09-29 18:05:18 -07:00
Yann Collet
8afcc80e07 decode more data before triggering error 
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
 2017-09-29 15:54:09 -07:00
W. Felix Handte
dc27c36495 Update documentation to reflect other format support 2017-09-28 19:43:12 -04:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
3182ea2e64 Merge pull request #866 from facebook/list 
improved --list display
 2017-09-27 16:34:29 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
3095ca8c56 fixed minor conversion warnings for g++ on Linux 
U64 is not considered equivalent to unsigned long long
 2017-09-26 13:53:50 -07:00
Yann Collet
56f1f0e3dd write summary for --list on multiple files 2017-09-26 11:21:36 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
W. Felix Handte
360238733a Adds LZ4 support by default if LZ4 is available 
Simple makefile change + quick typename change

Test:

  make clean
  make
  # successfully produces binary without lz4 support

  make clean
  # with flags to pick up my lz4 build
  make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib"
  # successfully produces binary with lz4 support

  echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \
    ./lz4/lz4 | \
    LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d
  # successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU
 2017-09-22 13:28:56 -07:00
Yann Collet
b0c0e3a3fb Merge pull request #853 from terrelln/blog 
[zstdcli] Fix LDM advanced options parsing
 2017-09-18 15:21:23 -07:00
Nick Terrell
1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet
92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Yann Collet
1722055799 add comment on using -B# to split input file for dictionary training 2017-09-15 16:23:50 -07:00
Yann Collet
c68d17f2da ensures that sampleSizes table is large enough 
as recommended by @terrelln
 2017-09-15 15:31:31 -07:00
Yann Collet
25a60488dd fixed 64-to-32 conversion warnings 2017-09-15 11:55:13 -07:00
Yann Collet
a9694231ca fixed minor conversion warning 2017-09-15 10:16:26 -07:00
Yann Collet
086b9597d9 added ability to split input files for dictionary training 
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
 2017-09-14 16:45:10 -07:00
Yann Collet
77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet
8f26dc3f9c blindfix for Visual 
LARGE_INTEGER is not an integer :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx

Do not take any risk with the structure definition :
use int init = 0; like Mac code
 2017-09-12 21:21:17 -07:00
Yann Collet
bc41c7f0eb fixed minor prototype warning 2017-09-12 19:32:26 -07:00
Yann Collet
c95c0c9725 modified util::time API 
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
 2017-09-12 18:12:46 -07:00
Nick Terrell
6ab4d5e904 [bench] Use higher resolution timer on POSIX 
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
 2017-09-12 16:46:44 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Yann Collet
baa37c3362 programs/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
 2017-09-06 16:53:59 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau
8c33cfe0bc Add ldm documentation in README 2017-09-06 15:21:01 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Yann Collet
0558850735 bench stops immediately on decoding error 2017-09-01 11:46:15 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Yann Collet
c7818fc676 Merge branch 'modTests' into dev 
fixed conflict
 2017-08-31 17:00:16 -07:00
Yann Collet
4299c27132 improved console log of utils.h 
removed a warning when compiling on Windows
 2017-08-31 16:58:47 -07:00
Yann Collet
8e298382a8 changed target allarch into allzstd 
allzstd contains only zstd-related tests.
allmost = allzstd + zwrapper tests (which require zlib)
 2017-08-31 14:30:52 -07:00
Stella Lau
6a546efb8c Add long distance matcher 
Move last literals section to ZSTD_block_internal
 2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Bernhard M. Wiedemann
cf689b84f9 Sort input file list 
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
 2017-08-26 17:08:00 +02:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Yann Collet
232d62b637 fixed a few headers that were too hastily copy/pasted during last license change 2017-08-21 11:24:32 -07:00
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
1c108c811e cli : Display supported formats on -vV command 
Requested and inspired by patch from @ib (#771)
 2017-08-19 13:33:50 -07:00
Yann Collet
2ecd34ee5e fixed unused variables warnings 2017-08-19 01:23:49 -07:00
Yann Collet
23706fb743 updated doc on compilation variables 2017-08-19 01:14:36 -07:00
Yann Collet
9203003d5f fixed zstd-nolegacy 
and added it to allVariants for CI testings
 2017-08-19 01:01:53 -07:00
Yann Collet
4b387729b6 fixed zstd-small 
and added it to shortest for CI tests
 2017-08-19 00:48:29 -07:00
Yann Collet
7729ab83bb Merge branch 'dev' into variants 2017-08-19 00:37:06 -07:00
Yann Collet
32e943b3ef Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-19 00:36:37 -07:00
Yann Collet
582e19b01c Merge pull request #801 from facebook/newLicense 
New license
 2017-08-19 00:34:34 -07:00
Yann Collet
288e263ec4 added zstd-decompress to the list of variants tested 2017-08-18 18:39:39 -07:00
Yann Collet
166645e7b3 fixed zstd-compress 
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.

also : added zstd-compress compilation test into `make shortest`
 2017-08-18 18:30:41 -07:00
Yann Collet
f207b39f55 blindfix for Windows conversion warning 
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
 2017-08-18 17:06:12 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet
c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
Yann Collet
3f54d788e9 removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) 
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
 2017-08-18 15:15:31 -07:00
codicodi
0fb7b23fcc fix typo in lz4 support code 2017-08-14 14:04:24 +02:00
Nick Terrell
d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet
3345a91964 cli : use new advanced API by default 2017-07-05 17:34:15 -07:00
Yann Collet
b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet
368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet
c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet
6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet
e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet
c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Yann Collet
1d39550471 Merge pull request #744 from terrelln/dev 
[pzstd] Remove appveyor tests
 2017-07-01 05:58:33 -07:00
Nick Terrell
56e3964d85 [man] Specify that strategies start at 1 2017-06-30 16:29:37 -07:00
Yann Collet
811deaea6f Merge pull request #736 from terrelln/cover-default-api 
[zdict] Make COVER the default algorithm
 2017-06-28 20:25:36 -07:00
Nick Terrell
5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Nick Terrell
d88a441406 [zstdcli] Document HAVE_LZMA and zstd-noxz 2017-06-26 11:25:00 -07:00