AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,300 Commits 1 Branch 0 Tags 27 MiB
c233bdbaee
Commit Graph

304 Commits

Author SHA1 Message Date
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev' 
# Conflicts:
#	programs/fileio.c
 2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe 
Simple API multiframe decoding
 2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Przemyslaw Skibinski
4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski
cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski
02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Nick Terrell
2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet
b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet
c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Przemyslaw Skibinski
7a8a03c20d util.h: restore BSD license for Facebook Open-Source 2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d updated comments 2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski
e679741b18 _CRT_SECURE_NO_WARNINGS moved to util.h 2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
a35b9448ac improved MinGW support 2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski
20b089e53d simplified zstdcli.c 2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski
16ae6563a2 executables use new util.h and platform.h 2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Przemyslaw Skibinski
b866e72826 tools use platform.h 2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski
c71e552b2e fixed "strategy" in advanced compression parameters 2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski
8349d675e0 fileio.c: support advanced compression parameters 2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski
7c6bbc3298 updated FIO_passThrough 2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski
6b508b1770 updated test-gzstd 2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski
b493e3b3d3 introduced srcBufferLoaded 2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski
3c69760275 improved FIO_decompressGzFrame 2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski
8489f184f6 improved detection of -lz 2016-12-05 13:47:00 +01:00
Yann Collet
743b33f57e fix zstdcat 2016-12-02 15:18:57 -08:00
Yann Collet
5bd4237beb minor refactor 2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski
690753ea1a improved formatting in FIO_decompressSrcFile 2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski
4e49580407 removed testing artifacts 2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski
c5eebca128 rewritten FIO_decompressGzFile 2016-12-02 15:01:31 +01:00
Przemyslaw Skibinski
b0f2ef2119 improved gzip_* functions 2016-12-02 13:50:29 +01:00
Przemyslaw Skibinski
4b504f131a added gzip_open, gzip_close, gzip_read 2016-12-02 13:11:39 +01:00
Przemyslaw Skibinski
daaf754539 detect stream with ungetc 2016-12-01 13:29:19 +01:00
Przemyslaw Skibinski
19aad42ee1 added FIO_decompressGzFile 2016-12-01 11:56:31 +01:00
Przemyslaw Skibinski
abfb51f5f2 gzstd: decompresses .gz files 2016-11-30 15:05:54 +01:00
Przemyslaw Skibinski
0e14675df2 fileio.c: detect .gz files 2016-11-30 13:34:21 +01:00
Yann Collet
cdff19c4b3 minor comment change 2016-11-11 17:26:54 -08:00
Yann Collet
0018ca28dc zstd cli : displays total decoded size, even when a stream consists of multiple frames 2016-11-07 14:42:37 -08:00
Przemyslaw Skibinski
1f60eb6b17 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-11-03 11:38:04 +01:00
Przemyslaw Skibinski
26306fcacf BMK_SetNbIterations renamed to BMK_SetNbSeconds 2016-11-03 11:38:01 +01:00
Yann Collet
179b19776f fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h 
Added : ZSTD_isFrame() in experimental section
 2016-11-02 17:30:49 -07:00
Przemyslaw Skibinski
fcf22e3473 set permissions, access and modification times 2016-11-02 14:08:07 +01:00
Przemyslaw Skibinski
a42794df61 preserve file modification time 2016-11-02 13:08:39 +01:00
Yann Collet
d4cda27b63 new command -M#, to limit memory usage during decompression (#403) 2016-10-14 13:32:20 -07:00
Nick Terrell
3e82f2b61a Remove references to GPLv2 license 
License headers added to Makefiles were taken from `zstd/Makefile`.
 2016-10-11 15:28:40 -07:00
Yann Collet
993060e0f2 cli : better adaptation to small files 2016-09-21 16:46:08 +02:00
Yann Collet
43eeea4725 fileio uses ZSTD_resetCStream() 2016-09-15 15:38:44 +02:00
Yann Collet
3ecbe6a37c fileio uses ZSTD_resetDStream() 2016-09-14 17:26:59 +02:00
Yann Collet
d77f2ceee6 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-09-12 03:14:03 +02:00
Yann Collet
5c6d244973 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-09-07 14:54:54 +02:00
Yann Collet
ac8bace6b1 support large skippable frames 2016-09-07 14:54:23 +02:00
Yann Collet
03d3f238de minor comment 2016-09-07 07:01:33 +02:00
Yann Collet
aad9fe5470 don't remove() /dev/null (#316) 2016-09-07 07:00:08 +02:00
Yann Collet
4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet
1c69baa1a9 CLI supports Legacy formats v0.4+ 2016-08-28 12:47:17 -07:00
Yann Collet
5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet
ab267e772c removed fileio specific legacy support; 
now depends on ZSTD_decompressStream() transparent legacy support
 2016-08-28 08:46:25 -07:00
Yann Collet
e80d15304a Merge pull request #296 from inikep/Other 
Other
 2016-08-18 11:48:48 +02:00
inikep
5a54870047 fixed Intel Compiler warnings with Visual Studio 
http://encode.ru/threads/2119-Zstandard?p=49504&viewfull=1#post49504
 2016-08-18 09:00:25 +02:00
Yann Collet
bfdc34e5a3 Merge pull request #294 from inikep/Other 
improved Makefiles
 2016-08-17 19:41:56 +02:00
inikep
78f3e0630b disabled %llu warnings with MinGW on Windows 2016-08-17 14:52:11 +02:00
Yann Collet
3d9211e961 Merge pull request #290 from david-y-lam/fix_file_name 
Remove file name truncation in zstd CLI output
 2016-08-17 14:05:29 +02:00
Yann Collet
53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet
94ca85d01b fixed Visual warning 2016-08-16 15:11:28 +02:00
Yann Collet
6263ba5451 fileio uses new streaming API 2016-08-16 15:11:28 +02:00
Yann Collet
fb5e385164 minor changes 2016-08-16 15:11:28 +02:00
David Lam
0f2704581e Remove truncation of filename in zstd CLI log lines 2016-08-13 11:26:21 -07:00
Yann Collet
60ba31c570 zbuff uses ZSTD_compressEnd() 2016-07-28 19:55:09 +02:00
Yann Collet
fbd557d5c2 multi-files -t doesn't stop after detecting magic number read failure 2016-07-26 17:13:58 +02:00
Yann Collet
a1dd6b97d2 fixed null-length round trip 2016-07-26 16:44:09 +02:00
Yann Collet
7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
Yann Collet
24a3d90bf0 strengthened integrity tests 2016-07-26 01:33:20 +02:00
Yann Collet
b402490546 fixed #260, reported by @amnilsson 2016-07-26 00:49:47 +02:00
Yann Collet
17508f1a16 fixed a few minor coverity warnings 2016-07-14 17:18:20 +02:00
Yann Collet
44f684ded3 fixed minor coverity warning 2016-07-13 20:12:33 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
3c242e79d3 updated compression levels table 2016-07-13 14:56:24 +02:00
Yann Collet
2cac5b30b9 changed default compression level to 3 (can be modified with macro ZSTDCLI_DEFAULT_CLEVEL) 2016-07-13 14:15:08 +02:00
Yann Collet
19c27d27f1 simplified legacy functions, no longer need magic number 2016-07-07 14:40:13 +02:00
Yann Collet
c62cda9def fileio.c no longer depends on zstd_internal.h 2016-07-03 01:36:57 +02:00
Yann Collet
92d75667e4 fix for fast mode 2016-07-03 01:10:53 +02:00
Yann Collet
b2b5309c6f removes Visual warning on strerror() 2016-07-03 00:20:03 +02:00
Yann Collet
ed7fb8413c extended use of strerror(errno) 2016-07-02 21:01:54 +02:00
Yann Collet
b71adf45c1 extended use of strerror(errno) 2016-07-02 01:51:12 +02:00
Yann Collet
b7f197633a use strerror(errno) to return system error message 2016-07-02 00:59:25 +02:00
Yann Collet
cebab1d322 Explicit error message when dst file cannot be created / opened 2016-07-02 00:12:14 +02:00
Yann Collet
6c6e1751f6 use ZSTD_getParams() to simplify code 2016-06-27 15:28:45 +02:00
Yann Collet
391a128794 fix : segfault in command line during automatic overwrite protection mode 2016-06-21 17:06:25 +02:00
Yann Collet
c98f8e7b57 Modified : zstd CLI enables checksum by default 2016-06-20 16:31:24 +02:00
Yann Collet
b09b12ce10 Added command --rm : remove source file after successful de/compression 2016-06-09 23:01:19 +02:00
Yann Collet
2cc72f1fd3 fixed initialization issue in bench 2016-06-06 17:50:07 +02:00
Yann Collet
673f0d7cdc new frame format, allowing custom window size 2016-06-06 00:26:38 +02:00
Yann Collet
d3b7f8d21f Merged zstd_static.h into zstd.h . Now requires ZSTD_STATIC_LINKING_ONLY macro 2016-06-04 19:47:02 +02:00
Yann Collet
5347aee8f7 merged zbuff_static.h into zbuff.h . Now requires ZBUFF_STATIC_LINKING_ONLY macro 2016-06-04 19:12:48 +02:00
Yann Collet
0d31160b4e better dynamic memory adaptation when using CLI 2016-06-04 00:09:02 +02:00
Yann Collet
87cfbe3e5f checksum generation can be selected from command line (--checksum) 2016-06-01 19:22:15 +02:00
inikep
a6b942018d Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070 
# Conflicts:
#	.travis.yml
#	Makefile
#	lib/common/zstd_static.h
#	programs/Makefile
#	projects/VS2008/zstd/zstd.vcproj
#	projects/VS2008/zstdlib/zstdlib.vcproj
#	projects/cmake/lib/CMakeLists.txt
#	projects/cmake/programs/CMakeLists.txt
 2016-06-01 00:07:09 +02:00
inikep
f772bf54a5 support for skippable frames 2016-05-31 12:43:46 +02:00
Yann Collet
6fca9e7545 Changed : max dictionary file size increased to 8 MB 2016-05-31 02:40:42 +02:00
Yann Collet
6381e99fb2 Added : ability to create compressed frames without dictID 2016-05-31 02:29:45 +02:00
Yann Collet
c46fb924df added dictionary ID (incomplete) 2016-05-29 05:01:04 +02:00
inikep
e75909ef93 ZBUFF_createCCtx_advanced moved to zbuff_static.h 2016-05-25 11:31:16 +02:00
inikep
02c244bf78 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 2016-05-24 17:15:04 +02:00
Yann Collet
ddbb8e27bf fixed minor conversion warning 2016-05-24 00:52:14 +02:00
Yann Collet
de95f96462 Pass-Through mode support (using -df), for compatibility with gzip 2016-05-23 19:46:47 +02:00
Yann Collet
75424d1139 Added : Sparse write support 
--[no-]sparse command
 2016-05-23 16:56:56 +02:00
inikep
60af95d971 fixed compression with -c with multiple files on Windows 2016-05-19 10:29:49 +02:00
Yann Collet
b3a9938d0c Merge pull request #182 from inikep/dev 
support for Visual Studio 2012, 2013, 2015
 2016-05-10 17:18:12 +02:00
Yann Collet
bd39d54576 zbuff supports byte-by-byte decompression scenarios 2016-05-10 14:14:19 +02:00
inikep
ed9a08538c Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev 
# Conflicts:
#	lib/common/util.h
#	programs/paramgrill.c
#	visual/2013/fullbench/fullbench.vcxproj.filters
#	visual/2013/fuzzer/fuzzer.vcxproj.filters
 2016-05-10 13:20:01 +02:00
Yann Collet
d693117113 fixed : null-length string roundtrip 2016-05-10 05:56:09 +02:00
Yann Collet
8b23eea694 minor refactor : more accurate variables scope 
Added null-length string test
 2016-05-10 05:37:43 +02:00
Yann Collet
f6ca09b5ff Reduced console display on loading lots of files with zstd --train. Reported by @KrzysFR, see #177 2016-05-09 04:44:45 +02:00
Yann Collet
95af06f6ef fixed legacy dictionary decompression 2016-05-08 08:23:51 +02:00
Yann Collet
0c0f314550 CLI supports dictionary decompression in legacy mode v0.5.x 2016-05-08 08:11:24 +02:00
Yann Collet
f849462d4c minor refactor : more accurate variable scope 2016-05-07 22:43:40 +02:00
inikep
13c8424ea0 code cleaning 2016-05-05 13:58:56 +02:00
inikep
0bd0faec32 fixed compatibility issues 2016-05-05 13:10:57 +02:00
inikep
9c22e57bfb Compiler Options moved to util.h 2016-05-05 11:53:42 +02:00
inikep
bab4317961 util.h must the the first include to #define _POSIX_C_SOURCE 2016-04-29 15:19:40 +02:00
inikep
d5ff2c3d9a ordering of #include 2016-04-28 14:40:45 +02:00
inikep
83c76b4594 bench.c: time functions moved to common/util.h 2016-04-28 13:16:01 +02:00
inikep
69fcd7c0ae getFileSize moved to common/util.h 2016-04-28 12:23:33 +02:00
inikep
db3964382c introduced ZSTD_NODECOMPRESS to link only compressor 2016-04-22 18:22:30 +02:00
inikep
3c7c3527d0 introduced ZSTD_NOCOMPRESS to generate decompressor only 2016-04-22 13:59:05 +02:00
inikep
23a0889301 separation of lib/ into common/, compress/, decompress/, dictBuilder/, legacy/ 2016-04-22 12:43:18 +02:00
Yann Collet
9e8b09a7bd Fixed memory initialization issue, reported by Maciej Adamczyk 2016-04-07 19:35:23 +02:00
Yann Collet
3b71925c2d separate params into compressionParams and frameParams 2016-03-30 19:48:05 +02:00