AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,636 Commits 1 Branch 0 Tags 27 MiB
79106d8de9
Commit Graph

390 Commits

Author SHA1 Message Date
Yann Collet
88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet
c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
codicodi
0fb7b23fcc fix typo in lz4 support code 2017-08-14 14:04:24 +02:00
Yann Collet
b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet
368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet
c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet
6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet
e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet
c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Nick Terrell
849ecf3510 [zstdcli] Support xz by default when liblzma is available 2017-06-23 17:11:38 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
991f9dfcde switched fileio.c to ZSTD_DEBUG trigger macro 2017-06-21 15:16:13 -07:00
Paul Cruz
5be0f5544e added newline to align output 2017-06-21 12:41:10 -07:00
Paul Cruz
6f5fe71041 added error check for when file could not be opened 2017-06-21 12:37:23 -07:00
Paul Cruz
74a725da69 reversed calculation of ratio 2017-06-21 12:27:20 -07:00
Paul Cruz
a73c2a444a added tests with null files, changed condition to check that the file is more than 0 bytes 2017-06-20 14:33:08 -07:00
Paul Cruz
0765602903 fixed error where extremely small files were not being detected as not compressed with zstd 2017-06-20 14:04:46 -07:00
Paul Cruz
3701dde3ba changed to use LONG_SEEK 2017-06-20 13:44:05 -07:00
Paul Cruz
93a5510962 added progress metric to display 2017-06-20 13:26:25 -07:00
Paul Cruz
bda57754a2 moved signifcant operations to fileio.c 2017-06-20 12:43:10 -07:00
Paul Cruz
c594507871 removed double if statements for same condition 2017-06-20 11:59:45 -07:00
Paul Cruz
72e0ac9929 minor update to logic 2017-06-20 11:58:27 -07:00
Paul Cruz
cea55a9217 size_t const value 2017-06-20 11:56:11 -07:00
Paul Cruz
bed0dc95bc changed displayInfo alignment and const 2017-06-20 11:47:24 -07:00
Paul Cruz
cb2dbe6434 changed fileInfo initialization, updated error code 2017-06-20 09:35:21 -07:00
Paul Cruz
55b9b28dad made style changes, added description in man file 2017-06-19 09:55:55 -07:00
Paul Cruz
4a0e4c856c fixing more spacing on loops 2017-06-15 18:08:22 -07:00
Paul Cruz
b3e62446c6 added in correct error return from main in zstdcli.c 2017-06-15 17:46:49 -07:00
Paul Cruz
0d7c4d766a initialized info manually to avoid error 2017-06-15 17:30:18 -07:00
Paul Cruz
d3b34e4a25 changed spacing on if statements 2017-06-15 17:12:41 -07:00
Paul Cruz
b316691f91 added const 2017-06-15 17:03:59 -07:00
Paul Cruz
60a2e55e2e added case for when file is not compressed with zstd (incorrect magic number) 2017-06-15 17:00:59 -07:00
Paul Cruz
acaefb531b refactoring brackets to allow for const values 2017-06-15 16:27:38 -07:00
Paul Cruz
0757eae6ff rearranging code to allow for consts 2017-06-15 16:12:04 -07:00
Paul Cruz
e49afae2ab changed implicit precision loss again 2017-06-15 16:03:40 -07:00
Paul Cruz
54e0b6c66b got rid of imlpicit precision loss 2017-06-15 16:01:52 -07:00
Paul Cruz
b766211e10 made another function static 2017-06-15 15:55:49 -07:00
Paul Cruz
e7f02fc58a closed file at end of function, created another variable to keep exit points simpler 2017-06-15 15:53:02 -07:00
Paul Cruz
8b3ff7c9bc changed info to static allocation 2017-06-15 15:35:37 -07:00
Paul Cruz
96cabf7233 cleaned up if statements 2017-06-15 15:25:20 -07:00
Paul Cruz
4495e9a826 allow analysis even if file does not have zst extension 2017-06-15 15:02:54 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Paul Cruz
e208992529 cleaning up code 2017-06-15 12:27:32 -07:00
Paul Cruz
fc428ab350 changing formatting again 2017-06-14 13:26:19 -07:00
Paul Cruz
ebd60349f5 changed U64 to unsigned long to get rid of build test errors 2017-06-14 13:11:18 -07:00
Paul Cruz
81fa33b55a cleaning up code 2017-06-13 10:06:49 -07:00
Paul Cruz
a56dcbfcf8 changed print format for non-verbose version 2017-06-12 17:47:33 -07:00
Paul Cruz
c828b52111 yet more refactoring 2017-06-12 17:19:53 -07:00
Paul Cruz
28dbf8ee6c more refactoring for C90 2017-06-12 17:09:08 -07:00
Paul Cruz
618a7b66e2 refactoring for C90 syntax 2017-06-12 16:53:50 -07:00
Paul Cruz
6e33c74133 formatting issues resolved 2017-06-12 16:29:00 -07:00
Paul Cruz
173a739717 code refactorings with explicit conversion 2017-06-12 16:14:04 -07:00
Paul Cruz
f3d6ab28a2 formated information for verbose output 2017-06-12 15:59:28 -07:00
Paul Cruz
6e0204470a displayed decompressed size 2017-06-12 15:51:59 -07:00
Paul Cruz
dcd6ba6dc6 incremented decompressedSize instead of setting value 2017-06-12 15:40:47 -07:00
Paul Cruz
6996bd2598 removed useless lines 2017-06-12 15:24:50 -07:00
Paul Cruz
9cb602ee25 added in logic for parsing through blocks/frames 2017-06-12 15:22:48 -07:00
Paul Cruz
786b7cac27 added code to analyze the first frame header 2017-06-12 13:46:39 -07:00
Paul Cruz
a3d54cf73d added line spacing for clarity 2017-06-12 10:58:34 -07:00
Yann Collet
9e6a2eaab6 added MT support to NEWAPI 2017-06-11 18:39:46 -07:00
Paul Cruz
0f06f4f266 added display for compressed size 2017-06-06 09:21:42 -07:00
Paul Cruz
4128f67a4a added in check suffix check to ensure file was compressed with zstd 2017-06-05 15:00:06 -07:00
Paul Cruz
901435e9ef setup basic functions for adding --list functionality 2017-06-05 14:45:31 -07:00
Yann Collet
6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet
e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00
Yann Collet
01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00
Yann Collet
2e63a877f3 fixed xzstd 
--format=xz was missing a break, making the execution continue into lz4 error message
 2017-05-02 15:40:42 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Sean Purcell
eab41c1872 Fix LZ4 wrapper deprecation warnings 2017-04-26 10:17:38 -07:00
Sean Purcell
2c4b6fe6b3 Make lz4 compression/decompression compatible with library r123 2017-04-25 11:00:54 -07:00
Sean Purcell
4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Sean Purcell
42bac7fa84 Change ifndef's to undef's 2017-04-13 15:35:05 -07:00
Sean Purcell
f876f1200c Fix compilation on macOS 2017-04-13 12:33:45 -07:00
Yann Collet
5c42d0edc8 cli : better status display for zstdmt in 1-thread mode 2017-04-11 16:57:32 -07:00
Yann Collet
0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile() 
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
 2017-04-11 14:41:02 -07:00
Sean Purcell
279be2015b Let zstd overwrite read-only files 2017-04-06 12:56:40 -07:00
Nick Terrell
eb38617175 Clean up default sparse logic 2017-03-31 15:20:50 -07:00
Nick Terrell
96fe545a18 Turn off sparse mode for OS X by default 2017-03-31 15:16:43 -07:00
Sean Purcell
042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Nick Terrell
aa8bcf360f Add xz and lzma support. 
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
 2017-03-13 18:45:41 -07:00
Przemyslaw Skibinski
8e5032a965 cli : fix : --rm is silent when input is stdin (decompression) 2017-02-28 09:42:37 +01:00
Yann Collet
a81c2e7e44 Merge pull request #573 from facebook/ddict 
Improved DDict memory usage
 2017-02-27 20:54:42 -08:00
Yann Collet
ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet
dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski
862698f479 minor tweaks in FIO_decompressGzFrame 2017-02-27 13:21:05 +01:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev' 
# Conflicts:
#	programs/fileio.c
 2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe 
Simple API multiframe decoding
 2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Przemyslaw Skibinski
4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski
cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski
02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Nick Terrell
2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet
b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet
c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Przemyslaw Skibinski
7a8a03c20d util.h: restore BSD license for Facebook Open-Source 2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d updated comments 2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski
e679741b18 _CRT_SECURE_NO_WARNINGS moved to util.h 2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
a35b9448ac improved MinGW support 2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski
20b089e53d simplified zstdcli.c 2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski
16ae6563a2 executables use new util.h and platform.h 2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Przemyslaw Skibinski
b866e72826 tools use platform.h 2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski
c71e552b2e fixed "strategy" in advanced compression parameters 2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski
8349d675e0 fileio.c: support advanced compression parameters 2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski
7c6bbc3298 updated FIO_passThrough 2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski
6b508b1770 updated test-gzstd 2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski
b493e3b3d3 introduced srcBufferLoaded 2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski
3c69760275 improved FIO_decompressGzFrame 2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski
8489f184f6 improved detection of -lz 2016-12-05 13:47:00 +01:00
Yann Collet
743b33f57e fix zstdcat 2016-12-02 15:18:57 -08:00
Yann Collet
5bd4237beb minor refactor 2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski
690753ea1a improved formatting in FIO_decompressSrcFile 2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski
4e49580407 removed testing artifacts 2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski
c5eebca128 rewritten FIO_decompressGzFile 2016-12-02 15:01:31 +01:00
Przemyslaw Skibinski
b0f2ef2119 improved gzip_* functions 2016-12-02 13:50:29 +01:00
Przemyslaw Skibinski
4b504f131a added gzip_open, gzip_close, gzip_read 2016-12-02 13:11:39 +01:00
Przemyslaw Skibinski
daaf754539 detect stream with ungetc 2016-12-01 13:29:19 +01:00
Przemyslaw Skibinski
19aad42ee1 added FIO_decompressGzFile 2016-12-01 11:56:31 +01:00
Przemyslaw Skibinski
abfb51f5f2 gzstd: decompresses .gz files 2016-11-30 15:05:54 +01:00
Przemyslaw Skibinski
0e14675df2 fileio.c: detect .gz files 2016-11-30 13:34:21 +01:00
Yann Collet
cdff19c4b3 minor comment change 2016-11-11 17:26:54 -08:00
Yann Collet
0018ca28dc zstd cli : displays total decoded size, even when a stream consists of multiple frames 2016-11-07 14:42:37 -08:00
Przemyslaw Skibinski
1f60eb6b17 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-11-03 11:38:04 +01:00
Przemyslaw Skibinski
26306fcacf BMK_SetNbIterations renamed to BMK_SetNbSeconds 2016-11-03 11:38:01 +01:00
Yann Collet
179b19776f fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h 
Added : ZSTD_isFrame() in experimental section
 2016-11-02 17:30:49 -07:00
Przemyslaw Skibinski
fcf22e3473 set permissions, access and modification times 2016-11-02 14:08:07 +01:00