AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,267 Commits 1 Branch 0 Tags 27 MiB
92889709f9
Commit Graph

1074 Commits

Author SHA1 Message Date
Paul Cruz
e7f02fc58a closed file at end of function, created another variable to keep exit points simpler 2017-06-15 15:53:02 -07:00
Paul Cruz
8b3ff7c9bc changed info to static allocation 2017-06-15 15:35:37 -07:00
Paul Cruz
96cabf7233 cleaned up if statements 2017-06-15 15:25:20 -07:00
Paul Cruz
9437cc7498 added --list command 2017-06-15 15:07:54 -07:00
Paul Cruz
4495e9a826 allow analysis even if file does not have zst extension 2017-06-15 15:02:54 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Paul Cruz
e208992529 cleaning up code 2017-06-15 12:27:32 -07:00
Paul Cruz
fc428ab350 changing formatting again 2017-06-14 13:26:19 -07:00
Paul Cruz
607ee9f4bc print message when no files are given, rearrange code for readability 2017-06-14 13:23:36 -07:00
Paul Cruz
ebd60349f5 changed U64 to unsigned long to get rid of build test errors 2017-06-14 13:11:18 -07:00
Paul Cruz
81fa33b55a cleaning up code 2017-06-13 10:06:49 -07:00
Paul Cruz
a56dcbfcf8 changed print format for non-verbose version 2017-06-12 17:47:33 -07:00
Paul Cruz
c828b52111 yet more refactoring 2017-06-12 17:19:53 -07:00
Paul Cruz
28dbf8ee6c more refactoring for C90 2017-06-12 17:09:08 -07:00
Paul Cruz
618a7b66e2 refactoring for C90 syntax 2017-06-12 16:53:50 -07:00
Paul Cruz
6e33c74133 formatting issues resolved 2017-06-12 16:29:00 -07:00
Paul Cruz
173a739717 code refactorings with explicit conversion 2017-06-12 16:14:04 -07:00
Paul Cruz
f3d6ab28a2 formated information for verbose output 2017-06-12 15:59:28 -07:00
Paul Cruz
6e0204470a displayed decompressed size 2017-06-12 15:51:59 -07:00
Paul Cruz
dcd6ba6dc6 incremented decompressedSize instead of setting value 2017-06-12 15:40:47 -07:00
Paul Cruz
6996bd2598 removed useless lines 2017-06-12 15:24:50 -07:00
Paul Cruz
9cb602ee25 added in logic for parsing through blocks/frames 2017-06-12 15:22:48 -07:00
Paul Cruz
786b7cac27 added code to analyze the first frame header 2017-06-12 13:46:39 -07:00
Paul Cruz
a3d54cf73d added line spacing for clarity 2017-06-12 10:58:34 -07:00
Yann Collet
bb0aaf9579 minor man update on -B# option in benchmark mode 2017-06-12 05:19:15 -07:00
Yann Collet
9e6a2eaab6 added MT support to NEWAPI 2017-06-11 18:39:46 -07:00
Paul Cruz
0f06f4f266 added display for compressed size 2017-06-06 09:21:42 -07:00
Paul Cruz
4128f67a4a added in check suffix check to ensure file was compressed with zstd 2017-06-05 15:00:06 -07:00
Paul Cruz
901435e9ef setup basic functions for adding --list functionality 2017-06-05 14:45:31 -07:00
cyan4973
8bcbf42617 fixed g++ prototype mismatch 2017-06-04 23:52:00 -07:00
Yann Collet
6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet
e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00
Yann Collet
2d4d31c18a removed gcc compilation flag -Wbad-function-cast 
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
 2017-05-16 11:34:38 -07:00
Yann Collet
46ac9ade9b fixed vla warning on linux 2017-05-15 18:15:08 -07:00
Yann Collet
83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Nick Terrell
a99ece9e79 Update zstd.1.md to use btultra 2017-05-10 16:51:18 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt 
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
 2017-05-10 16:49:58 -07:00
Yann Collet
01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00
Yann Collet
a00e9599f1 removed -g from DEBUGFLAGS 
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
 2017-05-04 17:24:29 -07:00
Yann Collet
c08e56861e updated dict graphs to 2D mode 2017-05-03 14:33:28 -07:00
Yann Collet
710497d8ea updated programs/README.md, to introduce compilation variables 
make it possible to enable/disable features individually
 2017-05-02 17:18:24 -07:00
Yann Collet
f47284fec0 reorganized Makefile for multiple targets 2017-05-02 16:55:57 -07:00
Yann Collet
3791d2105d added xzstd4 target 
support for all formats, xz/lzma/lz4 included
 2017-05-02 16:38:37 -07:00
Yann Collet
2e63a877f3 fixed xzstd 
--format=xz was missing a break, making the execution continue into lz4 error message
 2017-05-02 15:40:42 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell
feba969a69 Fix LZ4_MSG in xzstd 2017-05-02 11:02:16 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Sean Purcell
eab41c1872 Fix LZ4 wrapper deprecation warnings 2017-04-26 10:17:38 -07:00
Sean Purcell
2c4b6fe6b3 Make lz4 compression/decompression compatible with library r123 2017-04-25 11:00:54 -07:00
Sean Purcell
4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Yann Collet
230d7acc7d cli : add support for --threads=# command 
updated documentation
add relevant test case
 2017-04-21 11:38:13 -07:00
Sean Purcell
9606256a8d Fix no thread message 2017-04-18 13:52:00 -07:00
Yann Collet
1c286950af Merge pull request #663 from iburinoc/zstdmt-default 
Compile CLI using files instead of objs
 2017-04-18 02:59:54 -07:00
Sean Purcell
c8b2df7d62 Compile CLI using files instead of objs 
This avoids conflicts between how the library was
configured and how the CLI was configured.
 2017-04-17 17:13:47 -07:00
Sean Purcell
e6c504dbe6 Update -T0 comment in man page 2017-04-17 17:12:02 -07:00
Sean Purcell
5935c990a0 Add zstdmt and -T0 to man page 2017-04-17 16:05:20 -07:00
Sean Purcell
f6ef4db20e Install zstdmt even without threading support 2017-04-17 12:21:11 -07:00
Sean Purcell
5a61f36474 Make zstd compile with mt by default 2017-04-17 12:06:02 -07:00
Baptiste Daroussin
7dd14d03b0 Enable multithreading on BSD 2017-04-15 16:25:08 +02:00
Sean Purcell
e4f3235c85 Add 0 initializers to static variables 2017-04-13 16:34:28 -07:00
Sean Purcell
42bac7fa84 Change ifndef's to undef's 2017-04-13 15:35:05 -07:00
Sean Purcell
9227aae001 Fix clang linux compilation 2017-04-13 14:06:40 -07:00
Sean Purcell
3b6207d4bd Fix compilation on windows 2017-04-13 14:04:10 -07:00
Sean Purcell
f876f1200c Fix compilation on macOS 2017-04-13 12:33:45 -07:00
Sean Purcell
afa48518e2 -T0 detects number of physical cores 2017-04-13 12:28:28 -07:00
Yann Collet
5c42d0edc8 cli : better status display for zstdmt in 1-thread mode 2017-04-11 16:57:32 -07:00
Yann Collet
0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile() 
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
 2017-04-11 14:41:02 -07:00
Sean Purcell
279be2015b Let zstd overwrite read-only files 2017-04-06 12:56:40 -07:00
Yann Collet
c2007388a5 fixed bench.c : optional advanced parameters applied 
before creating cdict
 2017-04-04 15:35:06 -07:00
Yann Collet
81d6380139 minor bench.c adjustments 
shorter debug messages
no need to check decompressedLength==0 twice
 2017-04-04 15:21:09 -07:00
Yann Collet
a36330a27d minor man update, for overlapSize 2017-04-03 14:57:24 -07:00
Yann Collet
805c5a3efb updated documentation on multithreading modes 2017-04-01 00:36:31 -07:00
Nick Terrell
eb38617175 Clean up default sparse logic 2017-03-31 15:20:50 -07:00
Nick Terrell
96fe545a18 Turn off sparse mode for OS X by default 2017-03-31 15:16:43 -07:00
Nick Terrell
5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet
5bde4be5a1 fix : bench automatically adapts parameters to srcSize 2017-03-29 12:10:38 -07:00
Sean Purcell
efdaf8bb7c Fix inline compile errors 2017-03-27 12:26:40 -07:00
Sean Purcell
894bf49713 Fix IS_CONSOLE returning 1 for NUL on windows 2017-03-27 12:19:30 -07:00
Yann Collet
0246d5c531 Merge pull request #630 from facebook/advancedCliCommands 
changed advanced commands --maxdict= and --dictID=
 2017-03-26 00:13:35 -07:00
Sean Purcell
ef30af843f Ignore extension in command name matching 2017-03-24 17:06:09 -07:00
Yann Collet
96aa3019b2 changed advanced commands --maxdict= and --dictID= 
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
 2017-03-24 16:04:29 -07:00
Yann Collet
f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00
Sean Purcell
dee08caabb Fix displayLevel redefinition issues 2017-03-23 12:09:35 -07:00
Sean Purcell
680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Sean Purcell
042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Nick Terrell
dfc75f74da Add $(EXT) to binary to fix zlib detection for some MinGW versions 2017-03-22 13:24:52 -07:00
Nick Terrell
741e3e8314 Add $(FLAGS) when testing for zlib/lzma support. 
* If zlib/lzma isn't in the usual spot, it won't be used,
  even if `$CFLAGS` and `$LDFLAGS` add the location it is in.
* Update the test code snippets to not trigger any warnings.
 2017-03-22 12:33:08 -07:00
Yann Collet
35c147ab6a Merge pull request #605 from iburinoc/term 
Fix zstd not erroring on compressing to terminal without forceStdout
 2017-03-17 14:02:48 -07:00
Yann Collet
0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Sean Purcell
b8e52d3c83 Fix zstd not erroring on compressing to terminal without forceStdout 2017-03-16 16:06:19 -07:00
Sean Purcell
72a243e3bd Convert manpage to Markdown, converted with ronn 2017-03-16 15:37:32 -07:00
Yann Collet
3773bd318f Merge pull request #601 from terrelln/nxz 
Add xz and lzma support.
 2017-03-13 22:00:03 -07:00
Nick Terrell
aa8bcf360f Add xz and lzma support. 
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
 2017-03-13 18:45:41 -07:00
Sean Purcell
120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell
334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Dmitry V. Levin
201e8c8157 programs/Makefile: remove zstd-internal target 
zstd-internal was intended to be a helper target, but it doesn't help
at all, what it does in practice is a useless rebuild of zstd every time
"make zstd" is invoked.

Fixes: 030ac243a0 ("Changed Makefile to generate zstd with .gz support by default")
 2017-03-09 22:47:14 +00:00
Nick Terrell
eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Yann Collet
8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet
78208bd8be fixed : build zstd cli after libzstd 2017-03-01 21:02:06 -08:00
Yann Collet
c1c040eae1 added gzip tests 
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
 2017-03-01 16:49:20 -08:00
Przemyslaw Skibinski
8e5032a965 cli : fix : --rm is silent when input is stdin (decompression) 2017-02-28 09:42:37 +01:00
Yann Collet
a81c2e7e44 Merge pull request #573 from facebook/ddict 
Improved DDict memory usage
 2017-02-27 20:54:42 -08:00
Yann Collet
ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet
dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski
862698f479 minor tweaks in FIO_decompressGzFrame 2017-02-27 13:21:05 +01:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Przemyslaw Skibinski
337ec875b6 minor tweaks 2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski
684858e7b7 fix memory leaks 2017-02-21 18:17:24 +01:00
Przemyslaw Skibinski
74dcd8d15f bench.c: use a single ticksPerSecond 2017-02-21 12:22:05 +01:00
Przemyslaw Skibinski
1b59333c82 util.h: restore times() 2017-02-21 07:33:45 +01:00
Przemyslaw Skibinski
da4a0f30af util.h: use mach_absolute_time for macOS 2017-02-20 12:18:15 +01:00
Przemyslaw Skibinski
e052c60540 introduce UTIL_freq_t 2017-02-20 11:27:11 +01:00
Przemyslaw Skibinski
83775d9e05 replace times() with clock_gettime(CLOCK_MONOTONIC, x) 2017-02-20 11:11:50 +01:00
Przemyslaw Skibinski
acb6e57ad2 use FindFirstFileA instead of FindFirstFile 2017-02-15 17:13:35 +01:00
Przemyslaw Skibinski
6e59b3ce01 added UTIL_fseek 2017-02-15 17:03:16 +01:00
Przemyslaw Skibinski
3aaa1dae4e simplified zlib detection 2017-02-15 09:17:39 +01:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Yann Collet
c09d16ba8c preset behavior for gzip, gunzip and gzcat 
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
 2017-02-14 10:45:19 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski
970419535f fixed function name (2) 2017-02-14 09:47:29 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
abd6302423 Windows resources updated to v1.1.4 2017-02-14 09:39:09 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
b876b96ce1 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-14 09:26:03 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev' 
# Conflicts:
#	programs/fileio.c
 2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe 
Simple API multiframe decoding
 2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
Przemyslaw Skibinski
5a3bb05bb2 Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress 
# Conflicts:
#	programs/Makefile
 2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski
35bf23c086 MinGW-w64 requires _FILE_OFFSET_BITS 64 2017-02-13 13:57:29 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Yann Collet
acfa151622 Merge pull request #540 from ds77/dev-stat64-fix 
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
 2017-02-11 21:08:08 -08:00
Przemyslaw Skibinski
eb132530cd revert last commit 2017-02-10 21:15:49 +01:00
Przemyslaw Skibinski
645f5b9856 fix for original MinGW 2017-02-10 20:09:28 +01:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
ds77
45f0c207ab use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks. 2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski
456330948a Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw 
# Conflicts:
#	programs/platform.h
 2017-02-10 13:38:54 +01:00
Yann Collet
bbba42acd1 Merge pull request #537 from terrelln/small-bugs 
Fix small bugs
 2017-02-10 04:35:43 -08:00
-
7ec315df0d fix previous commit 
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
 2017-02-10 13:27:43 +01:00
-
19f61b534e use _stat64 only when targetting Win2k or later 2017-02-10 10:56:45 +01:00
Przemyslaw Skibinski
429e13099a fix 64-bit file support for MinGW 2017-02-10 10:36:44 +01:00
Nick Terrell
d08019813b Improvement from @inikep 2017-02-09 14:20:52 -08:00
ds77
9cde3f8b2e use _stat64() on MinGW 
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
 2017-02-09 22:49:31 +01:00
Przemyslaw Skibinski
896638a8a2 echo replaced with printf 2017-02-09 17:01:17 +01:00
Przemyslaw Skibinski
13127fd05b don't use "echo -e" 2017-02-09 11:32:21 +01:00