AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,216 Commits 1 Branch 0 Tags 27 MiB
170f948eee
Commit Graph

373 Commits

Author SHA1 Message Date
Yann Collet
7f580f9ee8 interruption handler and variable are static 2017-10-02 11:39:05 -07:00
Yann Collet
fe5444bc66 removed the statement for all versions of Visual Studio 2017-10-02 02:02:16 -07:00
Yann Collet
51d82d5516 same error in Visual Studio 2012 ... 2017-10-02 01:12:40 -07:00
Yann Collet
ed7ae4c9bd The issue also impacts Visual Studio 2010 2017-10-02 00:45:28 -07:00
Yann Collet
6e7ba3df2f added (void)sig to avoid compilers complaining that sig is not used. 2017-10-02 00:19:47 -07:00
Yann Collet
82bc200f82 conditionnally removed invocation that generates a buggy warning with Visual Studio 2008 2017-10-02 00:02:24 -07:00
Yann Collet
bd18095edc blindfix for Visual : minor casting issue 
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
 2017-10-01 15:32:48 -07:00
Yann Collet
00fc1ba8dd cli: add Ctrl-C support, requested by @mike155 in #854 
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
 2017-10-01 12:10:26 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests 
Add Default LZ4 Support When Available
 2017-09-29 22:32:54 -07:00
Yann Collet
8afb151c9b cli: fixed wrong initialization in MT mode 
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.

Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.

This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.

At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
 2017-09-29 22:14:37 -07:00
Yann Collet
fbd5ab7027 minor fix : no longer use fake srcSize during resource creation 
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.

At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
 2017-09-29 19:40:27 -07:00
Yann Collet
db1668a43b fix : srcSize written in frame header when multiple files compressed 
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
 2017-09-29 18:05:18 -07:00
Yann Collet
8afcc80e07 decode more data before triggering error 
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
 2017-09-29 15:54:09 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
3182ea2e64 Merge pull request #866 from facebook/list 
improved --list display
 2017-09-27 16:34:29 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
3095ca8c56 fixed minor conversion warnings for g++ on Linux 
U64 is not considered equivalent to unsigned long long
 2017-09-26 13:53:50 -07:00
Yann Collet
56f1f0e3dd write summary for --list on multiple files 2017-09-26 11:21:36 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
W. Felix Handte
360238733a Adds LZ4 support by default if LZ4 is available 
Simple makefile change + quick typename change

Test:

  make clean
  make
  # successfully produces binary without lz4 support

  make clean
  # with flags to pick up my lz4 build
  make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib"
  # successfully produces binary with lz4 support

  echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \
    ./lz4/lz4 | \
    LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d
  # successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU
 2017-09-22 13:28:56 -07:00
Yann Collet
92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Stella Lau
6a546efb8c Add long distance matcher 
Move last literals section to ZSTD_block_internal
 2017-08-31 12:53:19 -07:00
Yann Collet
32e943b3ef Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-19 00:36:37 -07:00
Yann Collet
582e19b01c Merge pull request #801 from facebook/newLicense 
New license
 2017-08-19 00:34:34 -07:00
Yann Collet
166645e7b3 fixed zstd-compress 
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.

also : added zstd-compress compilation test into `make shortest`
 2017-08-18 18:30:41 -07:00
Yann Collet
f207b39f55 blindfix for Windows conversion warning 
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
 2017-08-18 17:06:12 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet
c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
codicodi
0fb7b23fcc fix typo in lz4 support code 2017-08-14 14:04:24 +02:00
Yann Collet
b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet
368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet
c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet
6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet
e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet
c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Nick Terrell
849ecf3510 [zstdcli] Support xz by default when liblzma is available 2017-06-23 17:11:38 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
991f9dfcde switched fileio.c to ZSTD_DEBUG trigger macro 2017-06-21 15:16:13 -07:00
Paul Cruz
5be0f5544e added newline to align output 2017-06-21 12:41:10 -07:00
Paul Cruz
6f5fe71041 added error check for when file could not be opened 2017-06-21 12:37:23 -07:00
Paul Cruz
74a725da69 reversed calculation of ratio 2017-06-21 12:27:20 -07:00
Paul Cruz
a73c2a444a added tests with null files, changed condition to check that the file is more than 0 bytes 2017-06-20 14:33:08 -07:00
Paul Cruz
0765602903 fixed error where extremely small files were not being detected as not compressed with zstd 2017-06-20 14:04:46 -07:00
Paul Cruz
3701dde3ba changed to use LONG_SEEK 2017-06-20 13:44:05 -07:00
Paul Cruz
93a5510962 added progress metric to display 2017-06-20 13:26:25 -07:00
Paul Cruz
bda57754a2 moved signifcant operations to fileio.c 2017-06-20 12:43:10 -07:00
Paul Cruz
c594507871 removed double if statements for same condition 2017-06-20 11:59:45 -07:00
Paul Cruz
72e0ac9929 minor update to logic 2017-06-20 11:58:27 -07:00
Paul Cruz
cea55a9217 size_t const value 2017-06-20 11:56:11 -07:00
Paul Cruz
bed0dc95bc changed displayInfo alignment and const 2017-06-20 11:47:24 -07:00
Paul Cruz
cb2dbe6434 changed fileInfo initialization, updated error code 2017-06-20 09:35:21 -07:00
Paul Cruz
55b9b28dad made style changes, added description in man file 2017-06-19 09:55:55 -07:00
Paul Cruz
4a0e4c856c fixing more spacing on loops 2017-06-15 18:08:22 -07:00
Paul Cruz
b3e62446c6 added in correct error return from main in zstdcli.c 2017-06-15 17:46:49 -07:00
Paul Cruz
0d7c4d766a initialized info manually to avoid error 2017-06-15 17:30:18 -07:00
Paul Cruz
d3b34e4a25 changed spacing on if statements 2017-06-15 17:12:41 -07:00
Paul Cruz
b316691f91 added const 2017-06-15 17:03:59 -07:00
Paul Cruz
60a2e55e2e added case for when file is not compressed with zstd (incorrect magic number) 2017-06-15 17:00:59 -07:00
Paul Cruz
acaefb531b refactoring brackets to allow for const values 2017-06-15 16:27:38 -07:00
Paul Cruz
0757eae6ff rearranging code to allow for consts 2017-06-15 16:12:04 -07:00
Paul Cruz
e49afae2ab changed implicit precision loss again 2017-06-15 16:03:40 -07:00
Paul Cruz
54e0b6c66b got rid of imlpicit precision loss 2017-06-15 16:01:52 -07:00
Paul Cruz
b766211e10 made another function static 2017-06-15 15:55:49 -07:00
Paul Cruz
e7f02fc58a closed file at end of function, created another variable to keep exit points simpler 2017-06-15 15:53:02 -07:00
Paul Cruz
8b3ff7c9bc changed info to static allocation 2017-06-15 15:35:37 -07:00
Paul Cruz
96cabf7233 cleaned up if statements 2017-06-15 15:25:20 -07:00
Paul Cruz
4495e9a826 allow analysis even if file does not have zst extension 2017-06-15 15:02:54 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Paul Cruz
e208992529 cleaning up code 2017-06-15 12:27:32 -07:00
Paul Cruz
fc428ab350 changing formatting again 2017-06-14 13:26:19 -07:00
Paul Cruz
ebd60349f5 changed U64 to unsigned long to get rid of build test errors 2017-06-14 13:11:18 -07:00
Paul Cruz
81fa33b55a cleaning up code 2017-06-13 10:06:49 -07:00
Paul Cruz
a56dcbfcf8 changed print format for non-verbose version 2017-06-12 17:47:33 -07:00
Paul Cruz
c828b52111 yet more refactoring 2017-06-12 17:19:53 -07:00
Paul Cruz
28dbf8ee6c more refactoring for C90 2017-06-12 17:09:08 -07:00
Paul Cruz
618a7b66e2 refactoring for C90 syntax 2017-06-12 16:53:50 -07:00
Paul Cruz
6e33c74133 formatting issues resolved 2017-06-12 16:29:00 -07:00
Paul Cruz
173a739717 code refactorings with explicit conversion 2017-06-12 16:14:04 -07:00
Paul Cruz
f3d6ab28a2 formated information for verbose output 2017-06-12 15:59:28 -07:00
Paul Cruz
6e0204470a displayed decompressed size 2017-06-12 15:51:59 -07:00
Paul Cruz
dcd6ba6dc6 incremented decompressedSize instead of setting value 2017-06-12 15:40:47 -07:00
Paul Cruz
6996bd2598 removed useless lines 2017-06-12 15:24:50 -07:00
Paul Cruz
9cb602ee25 added in logic for parsing through blocks/frames 2017-06-12 15:22:48 -07:00
Paul Cruz
786b7cac27 added code to analyze the first frame header 2017-06-12 13:46:39 -07:00
Paul Cruz
a3d54cf73d added line spacing for clarity 2017-06-12 10:58:34 -07:00
Yann Collet
9e6a2eaab6 added MT support to NEWAPI 2017-06-11 18:39:46 -07:00
Paul Cruz
0f06f4f266 added display for compressed size 2017-06-06 09:21:42 -07:00
Paul Cruz
4128f67a4a added in check suffix check to ensure file was compressed with zstd 2017-06-05 15:00:06 -07:00
Paul Cruz
901435e9ef setup basic functions for adding --list functionality 2017-06-05 14:45:31 -07:00
Yann Collet
6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet
e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00
Yann Collet
01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00