AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,562 Commits 1 Branch 0 Tags 27 MiB
82bc8fe0cc
Commit Graph

1136 Commits

Author SHA1 Message Date
Yann Collet
8afcc80e07 decode more data before triggering error 
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
 2017-09-29 15:54:09 -07:00
W. Felix Handte
dc27c36495 Update documentation to reflect other format support 2017-09-28 19:43:12 -04:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
3182ea2e64 Merge pull request #866 from facebook/list 
improved --list display
 2017-09-27 16:34:29 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
3095ca8c56 fixed minor conversion warnings for g++ on Linux 
U64 is not considered equivalent to unsigned long long
 2017-09-26 13:53:50 -07:00
Yann Collet
56f1f0e3dd write summary for --list on multiple files 2017-09-26 11:21:36 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
W. Felix Handte
360238733a Adds LZ4 support by default if LZ4 is available 
Simple makefile change + quick typename change

Test:

  make clean
  make
  # successfully produces binary without lz4 support

  make clean
  # with flags to pick up my lz4 build
  make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib"
  # successfully produces binary with lz4 support

  echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \
    ./lz4/lz4 | \
    LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d
  # successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU
 2017-09-22 13:28:56 -07:00
Yann Collet
b0c0e3a3fb Merge pull request #853 from terrelln/blog 
[zstdcli] Fix LDM advanced options parsing
 2017-09-18 15:21:23 -07:00
Nick Terrell
1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet
92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Yann Collet
1722055799 add comment on using -B# to split input file for dictionary training 2017-09-15 16:23:50 -07:00
Yann Collet
c68d17f2da ensures that sampleSizes table is large enough 
as recommended by @terrelln
 2017-09-15 15:31:31 -07:00
Yann Collet
25a60488dd fixed 64-to-32 conversion warnings 2017-09-15 11:55:13 -07:00
Yann Collet
a9694231ca fixed minor conversion warning 2017-09-15 10:16:26 -07:00
Yann Collet
086b9597d9 added ability to split input files for dictionary training 
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
 2017-09-14 16:45:10 -07:00
Yann Collet
77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet
8f26dc3f9c blindfix for Visual 
LARGE_INTEGER is not an integer :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx

Do not take any risk with the structure definition :
use int init = 0; like Mac code
 2017-09-12 21:21:17 -07:00
Yann Collet
bc41c7f0eb fixed minor prototype warning 2017-09-12 19:32:26 -07:00
Yann Collet
c95c0c9725 modified util::time API 
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
 2017-09-12 18:12:46 -07:00
Nick Terrell
6ab4d5e904 [bench] Use higher resolution timer on POSIX 
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
 2017-09-12 16:46:44 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Yann Collet
baa37c3362 programs/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
 2017-09-06 16:53:59 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau
8c33cfe0bc Add ldm documentation in README 2017-09-06 15:21:01 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Yann Collet
0558850735 bench stops immediately on decoding error 2017-09-01 11:46:15 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Yann Collet
c7818fc676 Merge branch 'modTests' into dev 
fixed conflict
 2017-08-31 17:00:16 -07:00
Yann Collet
4299c27132 improved console log of utils.h 
removed a warning when compiling on Windows
 2017-08-31 16:58:47 -07:00
Yann Collet
8e298382a8 changed target allarch into allzstd 
allzstd contains only zstd-related tests.
allmost = allzstd + zwrapper tests (which require zlib)
 2017-08-31 14:30:52 -07:00
Stella Lau
6a546efb8c Add long distance matcher 
Move last literals section to ZSTD_block_internal
 2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Bernhard M. Wiedemann
cf689b84f9 Sort input file list 
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
 2017-08-26 17:08:00 +02:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Yann Collet
232d62b637 fixed a few headers that were too hastily copy/pasted during last license change 2017-08-21 11:24:32 -07:00
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
1c108c811e cli : Display supported formats on -vV command 
Requested and inspired by patch from @ib (#771)
 2017-08-19 13:33:50 -07:00
Yann Collet
2ecd34ee5e fixed unused variables warnings 2017-08-19 01:23:49 -07:00
Yann Collet
23706fb743 updated doc on compilation variables 2017-08-19 01:14:36 -07:00
Yann Collet
9203003d5f fixed zstd-nolegacy 
and added it to allVariants for CI testings
 2017-08-19 01:01:53 -07:00
Yann Collet
4b387729b6 fixed zstd-small 
and added it to shortest for CI tests
 2017-08-19 00:48:29 -07:00
Yann Collet
7729ab83bb Merge branch 'dev' into variants 2017-08-19 00:37:06 -07:00
Yann Collet
32e943b3ef Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-19 00:36:37 -07:00
Yann Collet
582e19b01c Merge pull request #801 from facebook/newLicense 
New license
 2017-08-19 00:34:34 -07:00
Yann Collet
288e263ec4 added zstd-decompress to the list of variants tested 2017-08-18 18:39:39 -07:00
Yann Collet
166645e7b3 fixed zstd-compress 
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.

also : added zstd-compress compilation test into `make shortest`
 2017-08-18 18:30:41 -07:00
Yann Collet
f207b39f55 blindfix for Windows conversion warning 
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
 2017-08-18 17:06:12 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet
c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
Yann Collet
3f54d788e9 removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) 
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
 2017-08-18 15:15:31 -07:00
codicodi
0fb7b23fcc fix typo in lz4 support code 2017-08-14 14:04:24 +02:00
Nick Terrell
d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet
3345a91964 cli : use new advanced API by default 2017-07-05 17:34:15 -07:00
Yann Collet
b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet
368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet
c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet
6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet
e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet
c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Yann Collet
1d39550471 Merge pull request #744 from terrelln/dev 
[pzstd] Remove appveyor tests
 2017-07-01 05:58:33 -07:00
Nick Terrell
56e3964d85 [man] Specify that strategies start at 1 2017-06-30 16:29:37 -07:00
Yann Collet
811deaea6f Merge pull request #736 from terrelln/cover-default-api 
[zdict] Make COVER the default algorithm
 2017-06-28 20:25:36 -07:00
Nick Terrell
5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Nick Terrell
d88a441406 [zstdcli] Document HAVE_LZMA and zstd-noxz 2017-06-26 11:25:00 -07:00
Nick Terrell
6aeb50ea0d [zstdcli] Add unlzma and unxz symlinks 2017-06-26 11:24:36 -07:00
Nick Terrell
849ecf3510 [zstdcli] Support xz by default when liblzma is available 2017-06-23 17:11:38 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
991f9dfcde switched fileio.c to ZSTD_DEBUG trigger macro 2017-06-21 15:16:13 -07:00
Paul Cruz
5be0f5544e added newline to align output 2017-06-21 12:41:10 -07:00
Paul Cruz
6f5fe71041 added error check for when file could not be opened 2017-06-21 12:37:23 -07:00
Paul Cruz
74a725da69 reversed calculation of ratio 2017-06-21 12:27:20 -07:00
Yann Collet
7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Paul Cruz
a73c2a444a added tests with null files, changed condition to check that the file is more than 0 bytes 2017-06-20 14:33:08 -07:00
Yann Collet
b44ab82f7a ensure new ZSTD_strategy starts at value 1 2017-06-20 14:11:49 -07:00
Paul Cruz
0765602903 fixed error where extremely small files were not being detected as not compressed with zstd 2017-06-20 14:04:46 -07:00
Paul Cruz
3701dde3ba changed to use LONG_SEEK 2017-06-20 13:44:05 -07:00
Paul Cruz
93a5510962 added progress metric to display 2017-06-20 13:26:25 -07:00
Paul Cruz
bda57754a2 moved signifcant operations to fileio.c 2017-06-20 12:43:10 -07:00
Paul Cruz
c594507871 removed double if statements for same condition 2017-06-20 11:59:45 -07:00
Paul Cruz
72e0ac9929 minor update to logic 2017-06-20 11:58:27 -07:00
Paul Cruz
cea55a9217 size_t const value 2017-06-20 11:56:11 -07:00
Paul Cruz
bed0dc95bc changed displayInfo alignment and const 2017-06-20 11:47:24 -07:00
Paul Cruz
cb2dbe6434 changed fileInfo initialization, updated error code 2017-06-20 09:35:21 -07:00
Yann Collet
c08e649e95 first implementation of bench.c with new API ZSTD_compress_generic() 
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.

Also : fixed a long standing bug in ZSTDMT streaming API.
 2017-06-19 18:25:35 -07:00
Yann Collet
fe234bf48b fix attempts : fullbench for VS2008 2017-06-19 15:23:19 -07:00