AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,449 Commits 1 Branch 0 Tags 27 MiB
428e8b3bf4
Commit Graph

335 Commits

Author SHA1 Message Date
Paul Cruz
6996bd2598 removed useless lines 2017-06-12 15:24:50 -07:00
Paul Cruz
9cb602ee25 added in logic for parsing through blocks/frames 2017-06-12 15:22:48 -07:00
Paul Cruz
786b7cac27 added code to analyze the first frame header 2017-06-12 13:46:39 -07:00
Paul Cruz
a3d54cf73d added line spacing for clarity 2017-06-12 10:58:34 -07:00
Yann Collet
9e6a2eaab6 added MT support to NEWAPI 2017-06-11 18:39:46 -07:00
Paul Cruz
0f06f4f266 added display for compressed size 2017-06-06 09:21:42 -07:00
Paul Cruz
4128f67a4a added in check suffix check to ensure file was compressed with zstd 2017-06-05 15:00:06 -07:00
Paul Cruz
901435e9ef setup basic functions for adding --list functionality 2017-06-05 14:45:31 -07:00
Yann Collet
6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet
e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00
Yann Collet
01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00
Yann Collet
2e63a877f3 fixed xzstd 
--format=xz was missing a break, making the execution continue into lz4 error message
 2017-05-02 15:40:42 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Sean Purcell
eab41c1872 Fix LZ4 wrapper deprecation warnings 2017-04-26 10:17:38 -07:00
Sean Purcell
2c4b6fe6b3 Make lz4 compression/decompression compatible with library r123 2017-04-25 11:00:54 -07:00
Sean Purcell
4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Sean Purcell
42bac7fa84 Change ifndef's to undef's 2017-04-13 15:35:05 -07:00
Sean Purcell
f876f1200c Fix compilation on macOS 2017-04-13 12:33:45 -07:00
Yann Collet
5c42d0edc8 cli : better status display for zstdmt in 1-thread mode 2017-04-11 16:57:32 -07:00
Yann Collet
0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile() 
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
 2017-04-11 14:41:02 -07:00
Sean Purcell
279be2015b Let zstd overwrite read-only files 2017-04-06 12:56:40 -07:00
Nick Terrell
eb38617175 Clean up default sparse logic 2017-03-31 15:20:50 -07:00
Nick Terrell
96fe545a18 Turn off sparse mode for OS X by default 2017-03-31 15:16:43 -07:00
Sean Purcell
042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Nick Terrell
aa8bcf360f Add xz and lzma support. 
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
 2017-03-13 18:45:41 -07:00
Przemyslaw Skibinski
8e5032a965 cli : fix : --rm is silent when input is stdin (decompression) 2017-02-28 09:42:37 +01:00
Yann Collet
a81c2e7e44 Merge pull request #573 from facebook/ddict 
Improved DDict memory usage
 2017-02-27 20:54:42 -08:00
Yann Collet
ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet
dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski
862698f479 minor tweaks in FIO_decompressGzFrame 2017-02-27 13:21:05 +01:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev' 
# Conflicts:
#	programs/fileio.c
 2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe 
Simple API multiframe decoding
 2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Przemyslaw Skibinski
4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski
cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski
02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Nick Terrell
2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet
b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet
c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Przemyslaw Skibinski
7a8a03c20d util.h: restore BSD license for Facebook Open-Source 2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d updated comments 2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski
e679741b18 _CRT_SECURE_NO_WARNINGS moved to util.h 2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
a35b9448ac improved MinGW support 2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski
20b089e53d simplified zstdcli.c 2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski
16ae6563a2 executables use new util.h and platform.h 2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Przemyslaw Skibinski
b866e72826 tools use platform.h 2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski
c71e552b2e fixed "strategy" in advanced compression parameters 2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski
8349d675e0 fileio.c: support advanced compression parameters 2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski
7c6bbc3298 updated FIO_passThrough 2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski
6b508b1770 updated test-gzstd 2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski
b493e3b3d3 introduced srcBufferLoaded 2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski
3c69760275 improved FIO_decompressGzFrame 2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski
8489f184f6 improved detection of -lz 2016-12-05 13:47:00 +01:00
Yann Collet
743b33f57e fix zstdcat 2016-12-02 15:18:57 -08:00
Yann Collet
5bd4237beb minor refactor 2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski
690753ea1a improved formatting in FIO_decompressSrcFile 2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski
4e49580407 removed testing artifacts 2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski
c5eebca128 rewritten FIO_decompressGzFile 2016-12-02 15:01:31 +01:00
Przemyslaw Skibinski
b0f2ef2119 improved gzip_* functions 2016-12-02 13:50:29 +01:00
Przemyslaw Skibinski
4b504f131a added gzip_open, gzip_close, gzip_read 2016-12-02 13:11:39 +01:00
Przemyslaw Skibinski
daaf754539 detect stream with ungetc 2016-12-01 13:29:19 +01:00
Przemyslaw Skibinski
19aad42ee1 added FIO_decompressGzFile 2016-12-01 11:56:31 +01:00
Przemyslaw Skibinski
abfb51f5f2 gzstd: decompresses .gz files 2016-11-30 15:05:54 +01:00
Przemyslaw Skibinski
0e14675df2 fileio.c: detect .gz files 2016-11-30 13:34:21 +01:00
Yann Collet
cdff19c4b3 minor comment change 2016-11-11 17:26:54 -08:00
Yann Collet
0018ca28dc zstd cli : displays total decoded size, even when a stream consists of multiple frames 2016-11-07 14:42:37 -08:00
Przemyslaw Skibinski
1f60eb6b17 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-11-03 11:38:04 +01:00
Przemyslaw Skibinski
26306fcacf BMK_SetNbIterations renamed to BMK_SetNbSeconds 2016-11-03 11:38:01 +01:00
Yann Collet
179b19776f fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h 
Added : ZSTD_isFrame() in experimental section
 2016-11-02 17:30:49 -07:00
Przemyslaw Skibinski
fcf22e3473 set permissions, access and modification times 2016-11-02 14:08:07 +01:00
Przemyslaw Skibinski
a42794df61 preserve file modification time 2016-11-02 13:08:39 +01:00
Yann Collet
d4cda27b63 new command -M#, to limit memory usage during decompression (#403) 2016-10-14 13:32:20 -07:00
Nick Terrell
3e82f2b61a Remove references to GPLv2 license 
License headers added to Makefiles were taken from `zstd/Makefile`.
 2016-10-11 15:28:40 -07:00
Yann Collet
993060e0f2 cli : better adaptation to small files 2016-09-21 16:46:08 +02:00
Yann Collet
43eeea4725 fileio uses ZSTD_resetCStream() 2016-09-15 15:38:44 +02:00
Yann Collet
3ecbe6a37c fileio uses ZSTD_resetDStream() 2016-09-14 17:26:59 +02:00
Yann Collet
d77f2ceee6 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-09-12 03:14:03 +02:00
Yann Collet
5c6d244973 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-09-07 14:54:54 +02:00
Yann Collet
ac8bace6b1 support large skippable frames 2016-09-07 14:54:23 +02:00
Yann Collet
03d3f238de minor comment 2016-09-07 07:01:33 +02:00
Yann Collet
aad9fe5470 don't remove() /dev/null (#316) 2016-09-07 07:00:08 +02:00
Yann Collet
4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet
1c69baa1a9 CLI supports Legacy formats v0.4+ 2016-08-28 12:47:17 -07:00
Yann Collet
5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet
ab267e772c removed fileio specific legacy support; 
now depends on ZSTD_decompressStream() transparent legacy support
 2016-08-28 08:46:25 -07:00
Yann Collet
e80d15304a Merge pull request #296 from inikep/Other 
Other
 2016-08-18 11:48:48 +02:00
inikep
5a54870047 fixed Intel Compiler warnings with Visual Studio 
http://encode.ru/threads/2119-Zstandard?p=49504&viewfull=1#post49504
 2016-08-18 09:00:25 +02:00
Yann Collet
bfdc34e5a3 Merge pull request #294 from inikep/Other 
improved Makefiles
 2016-08-17 19:41:56 +02:00
inikep
78f3e0630b disabled %llu warnings with MinGW on Windows 2016-08-17 14:52:11 +02:00
Yann Collet
3d9211e961 Merge pull request #290 from david-y-lam/fix_file_name 
Remove file name truncation in zstd CLI output
 2016-08-17 14:05:29 +02:00
Yann Collet
53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet
94ca85d01b fixed Visual warning 2016-08-16 15:11:28 +02:00
Yann Collet
6263ba5451 fileio uses new streaming API 2016-08-16 15:11:28 +02:00
Yann Collet
fb5e385164 minor changes 2016-08-16 15:11:28 +02:00
David Lam
0f2704581e Remove truncation of filename in zstd CLI log lines 2016-08-13 11:26:21 -07:00
Yann Collet
60ba31c570 zbuff uses ZSTD_compressEnd() 2016-07-28 19:55:09 +02:00
Yann Collet
fbd557d5c2 multi-files -t doesn't stop after detecting magic number read failure 2016-07-26 17:13:58 +02:00
Yann Collet
a1dd6b97d2 fixed null-length round trip 2016-07-26 16:44:09 +02:00
Yann Collet
7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
Yann Collet
24a3d90bf0 strengthened integrity tests 2016-07-26 01:33:20 +02:00
Yann Collet
b402490546 fixed #260, reported by @amnilsson 2016-07-26 00:49:47 +02:00
Yann Collet
17508f1a16 fixed a few minor coverity warnings 2016-07-14 17:18:20 +02:00
Yann Collet
44f684ded3 fixed minor coverity warning 2016-07-13 20:12:33 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
3c242e79d3 updated compression levels table 2016-07-13 14:56:24 +02:00
Yann Collet
2cac5b30b9 changed default compression level to 3 (can be modified with macro ZSTDCLI_DEFAULT_CLEVEL) 2016-07-13 14:15:08 +02:00
Yann Collet
19c27d27f1 simplified legacy functions, no longer need magic number 2016-07-07 14:40:13 +02:00
Yann Collet
c62cda9def fileio.c no longer depends on zstd_internal.h 2016-07-03 01:36:57 +02:00
Yann Collet
92d75667e4 fix for fast mode 2016-07-03 01:10:53 +02:00
Yann Collet
b2b5309c6f removes Visual warning on strerror() 2016-07-03 00:20:03 +02:00
Yann Collet
ed7fb8413c extended use of strerror(errno) 2016-07-02 21:01:54 +02:00
Yann Collet
b71adf45c1 extended use of strerror(errno) 2016-07-02 01:51:12 +02:00
Yann Collet
b7f197633a use strerror(errno) to return system error message 2016-07-02 00:59:25 +02:00
Yann Collet
cebab1d322 Explicit error message when dst file cannot be created / opened 2016-07-02 00:12:14 +02:00
Yann Collet
6c6e1751f6 use ZSTD_getParams() to simplify code 2016-06-27 15:28:45 +02:00
Yann Collet
391a128794 fix : segfault in command line during automatic overwrite protection mode 2016-06-21 17:06:25 +02:00
Yann Collet
c98f8e7b57 Modified : zstd CLI enables checksum by default 2016-06-20 16:31:24 +02:00
Yann Collet
b09b12ce10 Added command --rm : remove source file after successful de/compression 2016-06-09 23:01:19 +02:00
Yann Collet
2cc72f1fd3 fixed initialization issue in bench 2016-06-06 17:50:07 +02:00
Yann Collet
673f0d7cdc new frame format, allowing custom window size 2016-06-06 00:26:38 +02:00
Yann Collet
d3b7f8d21f Merged zstd_static.h into zstd.h . Now requires ZSTD_STATIC_LINKING_ONLY macro 2016-06-04 19:47:02 +02:00
Yann Collet
5347aee8f7 merged zbuff_static.h into zbuff.h . Now requires ZBUFF_STATIC_LINKING_ONLY macro 2016-06-04 19:12:48 +02:00
Yann Collet
0d31160b4e better dynamic memory adaptation when using CLI 2016-06-04 00:09:02 +02:00
Yann Collet
87cfbe3e5f checksum generation can be selected from command line (--checksum) 2016-06-01 19:22:15 +02:00
inikep
a6b942018d Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070 
# Conflicts:
#	.travis.yml
#	Makefile
#	lib/common/zstd_static.h
#	programs/Makefile
#	projects/VS2008/zstd/zstd.vcproj
#	projects/VS2008/zstdlib/zstdlib.vcproj
#	projects/cmake/lib/CMakeLists.txt
#	projects/cmake/programs/CMakeLists.txt
 2016-06-01 00:07:09 +02:00