AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
3,266 Commits 1 Branch 0 Tags 27 MiB
fc5145955a
Commit Graph

935 Commits

Author SHA1 Message Date
Nick Terrell
dfc75f74da Add $(EXT) to binary to fix zlib detection for some MinGW versions 2017-03-22 13:24:52 -07:00
Nick Terrell
741e3e8314 Add $(FLAGS) when testing for zlib/lzma support. 
* If zlib/lzma isn't in the usual spot, it won't be used,
  even if `$CFLAGS` and `$LDFLAGS` add the location it is in.
* Update the test code snippets to not trigger any warnings.
 2017-03-22 12:33:08 -07:00
Yann Collet
35c147ab6a Merge pull request #605 from iburinoc/term 
Fix zstd not erroring on compressing to terminal without forceStdout
 2017-03-17 14:02:48 -07:00
Yann Collet
0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Sean Purcell
b8e52d3c83 Fix zstd not erroring on compressing to terminal without forceStdout 2017-03-16 16:06:19 -07:00
Sean Purcell
72a243e3bd Convert manpage to Markdown, converted with ronn 2017-03-16 15:37:32 -07:00
Yann Collet
3773bd318f Merge pull request #601 from terrelln/nxz 
Add xz and lzma support.
 2017-03-13 22:00:03 -07:00
Nick Terrell
aa8bcf360f Add xz and lzma support. 
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
 2017-03-13 18:45:41 -07:00
Sean Purcell
120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell
334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Dmitry V. Levin
201e8c8157 programs/Makefile: remove zstd-internal target 
zstd-internal was intended to be a helper target, but it doesn't help
at all, what it does in practice is a useless rebuild of zstd every time
"make zstd" is invoked.

Fixes: 030ac243a0 ("Changed Makefile to generate zstd with .gz support by default")
 2017-03-09 22:47:14 +00:00
Yann Collet
8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet
78208bd8be fixed : build zstd cli after libzstd 2017-03-01 21:02:06 -08:00
Yann Collet
c1c040eae1 added gzip tests 
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
 2017-03-01 16:49:20 -08:00
Przemyslaw Skibinski
8e5032a965 cli : fix : --rm is silent when input is stdin (decompression) 2017-02-28 09:42:37 +01:00
Yann Collet
a81c2e7e44 Merge pull request #573 from facebook/ddict 
Improved DDict memory usage
 2017-02-27 20:54:42 -08:00
Yann Collet
ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet
dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski
862698f479 minor tweaks in FIO_decompressGzFrame 2017-02-27 13:21:05 +01:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Przemyslaw Skibinski
337ec875b6 minor tweaks 2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski
684858e7b7 fix memory leaks 2017-02-21 18:17:24 +01:00
Przemyslaw Skibinski
74dcd8d15f bench.c: use a single ticksPerSecond 2017-02-21 12:22:05 +01:00
Przemyslaw Skibinski
1b59333c82 util.h: restore times() 2017-02-21 07:33:45 +01:00
Przemyslaw Skibinski
da4a0f30af util.h: use mach_absolute_time for macOS 2017-02-20 12:18:15 +01:00
Przemyslaw Skibinski
e052c60540 introduce UTIL_freq_t 2017-02-20 11:27:11 +01:00
Przemyslaw Skibinski
83775d9e05 replace times() with clock_gettime(CLOCK_MONOTONIC, x) 2017-02-20 11:11:50 +01:00
Przemyslaw Skibinski
acb6e57ad2 use FindFirstFileA instead of FindFirstFile 2017-02-15 17:13:35 +01:00
Przemyslaw Skibinski
6e59b3ce01 added UTIL_fseek 2017-02-15 17:03:16 +01:00
Przemyslaw Skibinski
3aaa1dae4e simplified zlib detection 2017-02-15 09:17:39 +01:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11 
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
 2017-02-14 12:29:00 -08:00
Yann Collet
c09d16ba8c preset behavior for gzip, gunzip and gzcat 
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
 2017-02-14 10:45:19 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski
970419535f fixed function name (2) 2017-02-14 09:47:29 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
abd6302423 Windows resources updated to v1.1.4 2017-02-14 09:39:09 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
b876b96ce1 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-14 09:26:03 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev' 
# Conflicts:
#	programs/fileio.c
 2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe 
Simple API multiframe decoding
 2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
Przemyslaw Skibinski
5a3bb05bb2 Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress 
# Conflicts:
#	programs/Makefile
 2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski
35bf23c086 MinGW-w64 requires _FILE_OFFSET_BITS 64 2017-02-13 13:57:29 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Yann Collet
acfa151622 Merge pull request #540 from ds77/dev-stat64-fix 
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
 2017-02-11 21:08:08 -08:00
Przemyslaw Skibinski
eb132530cd revert last commit 2017-02-10 21:15:49 +01:00
Przemyslaw Skibinski
645f5b9856 fix for original MinGW 2017-02-10 20:09:28 +01:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
ds77
45f0c207ab use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks. 2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski
456330948a Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw 
# Conflicts:
#	programs/platform.h
 2017-02-10 13:38:54 +01:00
Yann Collet
bbba42acd1 Merge pull request #537 from terrelln/small-bugs 
Fix small bugs
 2017-02-10 04:35:43 -08:00
-
7ec315df0d fix previous commit 
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
 2017-02-10 13:27:43 +01:00
-
19f61b534e use _stat64 only when targetting Win2k or later 2017-02-10 10:56:45 +01:00
Przemyslaw Skibinski
429e13099a fix 64-bit file support for MinGW 2017-02-10 10:36:44 +01:00
Nick Terrell
d08019813b Improvement from @inikep 2017-02-09 14:20:52 -08:00
ds77
9cde3f8b2e use _stat64() on MinGW 
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
 2017-02-09 22:49:31 +01:00
Przemyslaw Skibinski
896638a8a2 echo replaced with printf 2017-02-09 17:01:17 +01:00
Przemyslaw Skibinski
13127fd05b don't use "echo -e" 2017-02-09 11:32:21 +01:00
Nick Terrell
a0f9006e5a #undef _POSIX_C_SOURCE if already defined 2017-02-08 17:25:01 -08:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski
93901fe85c remove redundant "ifeq ($(HAVE_ZLIB), 1)" 2017-02-08 21:11:18 +01:00
Przemyslaw Skibinski
4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski
cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski
02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Przemyslaw Skibinski
cfd4dc299a add "--format=gzip" option 2017-02-08 15:17:55 +01:00
Przemyslaw Skibinski
ca20edd960 fixed zlib detection with MinGW 2017-02-08 14:32:49 +01:00
Przemyslaw Skibinski
cdf5a7bd9f Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-08 13:49:35 +01:00
Przemyslaw Skibinski
40580ff669 added description of "--priority=rt" 2017-02-08 13:49:06 +01:00
Przemyslaw Skibinski
0665a359aa "--rt-prio" renamed to "--priority=rt" 2017-02-07 20:12:59 +01:00
Przemyslaw Skibinski
d05014c739 added the "--rt-prio" option 2017-02-07 16:48:01 +01:00
Przemyslaw Skibinski
94abd6a26c SET_REALTIME_PRIORITY 2017-02-07 16:36:19 +01:00
Nick Terrell
7e3fc73795 Ensure <zlib.h> can be included in HAVE_ZLIB test 2017-02-06 11:54:31 -08:00
Nick Terrell
2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet
f6e9c4f1e1 Merge pull request #529 from facebook/gzstd_default 
Gzstd default
 2017-02-05 18:03:36 -08:00
cyan4973
613087c02b Silence zlib detection routine 
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
 2017-02-04 23:36:12 -08:00
Nick Terrell
b4016ff02f Add cover dictionary training to zstd.1 
Tested with `make install && man zstd` and visual inspection.
 2017-02-03 16:42:07 -08:00
Yann Collet
762ddeeb9e fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep 2017-02-03 14:35:42 -08:00
Yann Collet
21eb80d485 remove zlib detection artefact 
result of compilation test is sent to /dev/null
 2017-02-03 14:34:41 -08:00
Yann Collet
b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet
c2a4632789 release builds use less debug symbols and warnings 
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
 2017-02-02 20:54:41 -08:00
Yann Collet
c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet
030ac243a0 Changed Makefile to generate zstd with .gz support by default 
.gz support is detected by a runtime test.
 2017-02-02 16:49:34 -08:00
Yann Collet
3dc85bae66 minor : fixed zstd-frugal 
fixed a minor unused variable warning when compiling zstd-frugal target
 2017-02-02 16:19:09 -08:00
Yann Collet
48bed91606 Merge pull request #527 from facebook/zstdmt 
zstdmt refinements
 2017-01-31 16:36:46 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
6ccd37c8d4 cli : added advanced parameter overlapLog 
as a hidden (undocumented) parameter for now
 2017-01-30 13:07:24 -08:00
Yann Collet
6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann Collet
cc3d1bc262 Merge pull request #525 from terrelln/covermt 
Multithreaded COVER dictionary training
 2017-01-30 10:15:33 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Nick Terrell
2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Nick Terrell
9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski
eb2d23a90c improved #232 fix 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
92a4dbf2e4 Fixed https://github.com/facebook/zstd/issues/232 2017-01-27 10:43:03 -08:00
Nick Terrell
83c387eb8e Fix zstdmt_compress.h include 2017-01-26 15:25:32 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00