AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
3,362 Commits 1 Branch 0 Tags 27 MiB
3b0e24c06b
Commit Graph

942 Commits

Author SHA1 Message Date
Przemyslaw Skibinski
5a3bb05bb2 Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress 
# Conflicts:
#	programs/Makefile
 2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski
35bf23c086 MinGW-w64 requires _FILE_OFFSET_BITS 64 2017-02-13 13:57:29 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows 
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
 2017-02-12 16:05:19 +01:00
Yann Collet
acfa151622 Merge pull request #540 from ds77/dev-stat64-fix 
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
 2017-02-11 21:08:08 -08:00
Przemyslaw Skibinski
eb132530cd revert last commit 2017-02-10 21:15:49 +01:00
Przemyslaw Skibinski
645f5b9856 fix for original MinGW 2017-02-10 20:09:28 +01:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
ds77
45f0c207ab use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks. 2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski
456330948a Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw 
# Conflicts:
#	programs/platform.h
 2017-02-10 13:38:54 +01:00
Yann Collet
bbba42acd1 Merge pull request #537 from terrelln/small-bugs 
Fix small bugs
 2017-02-10 04:35:43 -08:00
-
7ec315df0d fix previous commit 
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
 2017-02-10 13:27:43 +01:00
-
19f61b534e use _stat64 only when targetting Win2k or later 2017-02-10 10:56:45 +01:00
Przemyslaw Skibinski
429e13099a fix 64-bit file support for MinGW 2017-02-10 10:36:44 +01:00
Nick Terrell
d08019813b Improvement from @inikep 2017-02-09 14:20:52 -08:00
ds77
9cde3f8b2e use _stat64() on MinGW 
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
 2017-02-09 22:49:31 +01:00
Przemyslaw Skibinski
896638a8a2 echo replaced with printf 2017-02-09 17:01:17 +01:00
Przemyslaw Skibinski
13127fd05b don't use "echo -e" 2017-02-09 11:32:21 +01:00
Nick Terrell
a0f9006e5a #undef _POSIX_C_SOURCE if already defined 2017-02-08 17:25:01 -08:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski
93901fe85c remove redundant "ifeq ($(HAVE_ZLIB), 1)" 2017-02-08 21:11:18 +01:00
Przemyslaw Skibinski
4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski
cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski
02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Przemyslaw Skibinski
cfd4dc299a add "--format=gzip" option 2017-02-08 15:17:55 +01:00
Przemyslaw Skibinski
ca20edd960 fixed zlib detection with MinGW 2017-02-08 14:32:49 +01:00
Przemyslaw Skibinski
cdf5a7bd9f Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-08 13:49:35 +01:00
Przemyslaw Skibinski
40580ff669 added description of "--priority=rt" 2017-02-08 13:49:06 +01:00
Przemyslaw Skibinski
0665a359aa "--rt-prio" renamed to "--priority=rt" 2017-02-07 20:12:59 +01:00
Przemyslaw Skibinski
d05014c739 added the "--rt-prio" option 2017-02-07 16:48:01 +01:00
Przemyslaw Skibinski
94abd6a26c SET_REALTIME_PRIORITY 2017-02-07 16:36:19 +01:00
Nick Terrell
7e3fc73795 Ensure <zlib.h> can be included in HAVE_ZLIB test 2017-02-06 11:54:31 -08:00
Nick Terrell
2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet
f6e9c4f1e1 Merge pull request #529 from facebook/gzstd_default 
Gzstd default
 2017-02-05 18:03:36 -08:00
cyan4973
613087c02b Silence zlib detection routine 
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
 2017-02-04 23:36:12 -08:00
Nick Terrell
b4016ff02f Add cover dictionary training to zstd.1 
Tested with `make install && man zstd` and visual inspection.
 2017-02-03 16:42:07 -08:00
Yann Collet
762ddeeb9e fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep 2017-02-03 14:35:42 -08:00
Yann Collet
21eb80d485 remove zlib detection artefact 
result of compilation test is sent to /dev/null
 2017-02-03 14:34:41 -08:00
Yann Collet
b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet
c2a4632789 release builds use less debug symbols and warnings 
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
 2017-02-02 20:54:41 -08:00
Yann Collet
c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet
030ac243a0 Changed Makefile to generate zstd with .gz support by default 
.gz support is detected by a runtime test.
 2017-02-02 16:49:34 -08:00
Yann Collet
3dc85bae66 minor : fixed zstd-frugal 
fixed a minor unused variable warning when compiling zstd-frugal target
 2017-02-02 16:19:09 -08:00
Yann Collet
48bed91606 Merge pull request #527 from facebook/zstdmt 
zstdmt refinements
 2017-01-31 16:36:46 -08:00
Yann Collet
8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet
6ccd37c8d4 cli : added advanced parameter overlapLog 
as a hidden (undocumented) parameter for now
 2017-01-30 13:07:24 -08:00
Yann Collet
6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann Collet
cc3d1bc262 Merge pull request #525 from terrelln/covermt 
Multithreaded COVER dictionary training
 2017-01-30 10:15:33 -08:00
Yann Collet
5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Nick Terrell
2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Nick Terrell
9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski
eb2d23a90c improved #232 fix 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
92a4dbf2e4 Fixed https://github.com/facebook/zstd/issues/232 2017-01-27 10:43:03 -08:00
Nick Terrell
83c387eb8e Fix zstdmt_compress.h include 2017-01-26 15:25:32 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet
f8804d1014 convert tabs to space 
joys of using multiple editors from multiple environments ...
 2017-01-20 17:23:19 -08:00
Yann Collet
d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
cyan4973
5fba09fa41 updated util's time for Windows compatibility 
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
 2017-01-20 12:57:31 -08:00
Yann Collet
458c8a94b4 minor refactoring : cleaner MT integration within bench 2017-01-19 17:44:15 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Yann Collet
0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Yann Collet
f22adae984 fixed minor warning (unused variable) in fuzzer 2017-01-19 13:46:30 -08:00
Yann Collet
736788f8e8 added streaming fuzzer tests for MT API 
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
 2017-01-19 12:15:29 -08:00
Przemyslaw Skibinski
8a0bc30a2d Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski
d72f4b6b7a added "Makefile is validated" 2017-01-17 12:40:06 +01:00
Yann Collet
5eb749e734 ZSTDMT_compress() creates a single frame 
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
 2017-01-11 18:21:25 +01:00
Nick Terrell
c220d4c74d Use COVER_MEMMULT when training with COVER. 2017-01-09 16:49:04 -08:00
Yann Collet
56958500fc minor man page update 2017-01-09 19:47:09 +01:00
Nick Terrell
3a1fefcf00 Simplify COVER parameters 2017-01-02 17:51:38 -08:00
Nick Terrell
df8415c502 Add COVER to the zstd cli 2017-01-02 14:43:08 -08:00
Yann Collet
6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet
747452677d fixed cmake tests 2017-01-02 02:05:45 +01:00
Yann Collet
cdb2763f4a new Makefile target zstdmt 2017-01-02 01:43:56 +01:00
Yann Collet
f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet
0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet
c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann Collet
f765a375a5 Merge pull request #504 from terrelln/thread-pool 
[zstdmt] Add thread pool
 2016-12-30 15:31:49 +01:00
Yann Collet
e70912c72b Changed : input divided into roughly equal parts. 
Debug : can measure time waiting for mutexes to unlock.
 2016-12-29 01:24:01 +01:00
Yann Collet
8d7432914f Merge pull request #503 from inikep/dev11 
Dev11
 2016-12-28 16:50:39 +01:00
Yann Collet
ab7a579180 added -T command , to set nb of threads 2016-12-28 16:11:09 +01:00
Przemyslaw Skibinski
75f3a3a335 changed default PREFIX and MANDIR 2016-12-28 12:32:41 +01:00
Yann Collet
3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet
c07d2e3a31 Merge pull request #499 from inikep/dev11 
improved *BSD and Solaris compatibility
 2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski
63b0014b96 BSD: improved "make install" 2016-12-23 10:05:49 +01:00
Andrew Janke
fce374a100 zstdless: add shebang and quote $@ 2016-12-22 17:40:10 -05:00
Przemyslaw Skibinski
b999170311 Solaris: working "make -C lib install" 2016-12-22 20:14:37 +01:00
Przemyslaw Skibinski
aab442133d Solaris: working "make -C programs install" 2016-12-22 19:26:01 +01:00
Yann Collet
8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann Collet
0d7e84899f Merge pull request #489 from inikep/v112 
improved detection of POSIX
 2016-12-21 16:42:46 +01:00
Yann Collet
9518744025 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:20:41 +01:00
Yann Collet
1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski
7a8a03c20d util.h: restore BSD license for Facebook Open-Source 2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d updated comments 2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski
e679741b18 _CRT_SECURE_NO_WARNINGS moved to util.h 2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
101df4f636 fixed Visual Studio compilation 2016-12-21 11:43:11 +01:00
Przemyslaw Skibinski
a35b9448ac improved MinGW support 2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski
5736db219e fix basic types redefinition 2016-12-21 09:26:00 +01:00
Przemyslaw Skibinski
20b089e53d simplified zstdcli.c 2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski
16ae6563a2 executables use new util.h and platform.h 2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski
ead350bdc0 improved util.h and platform.h 2016-12-21 09:04:59 +01:00
Przemyslaw Skibinski
1c1db6b845 windres updated to v1.1.3 2016-12-20 11:21:26 +01:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Yann Collet
1496c3dc47 Fix : size estimation when some samples are very large 2016-12-18 11:58:23 +01:00
Yann Collet
d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Przemyslaw Skibinski
0b37205098 util.h: minor improvement 2016-12-16 17:12:23 +01:00
Przemyslaw Skibinski
b0f3663edc imporved support for POSIX-type OSes 2016-12-16 15:41:18 +01:00
Przemyslaw Skibinski
3cdfe266cf use PLATFORM_POSIX_VERSION 2016-12-16 15:00:50 +01:00
Przemyslaw Skibinski
b0e670a054 util.h uses platform.h 2016-12-16 14:25:12 +01:00
Przemyslaw Skibinski
b866e72826 tools use platform.h 2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski
b3843afcf5 introduced platform.h 2016-12-16 14:13:15 +01:00
Przemyslaw Skibinski
25314428c9 zstd.1: added advanced compression options 2016-12-14 16:10:13 +01:00
Przemyslaw Skibinski
c71e552b2e fixed "strategy" in advanced compression parameters 2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski
98ef0f98df fixed conversion warning 2016-12-13 14:52:21 +01:00
Przemyslaw Skibinski
8349d675e0 fileio.c: support advanced compression parameters 2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski
897b8bb5eb bench.c: support advanced compression parameters 2016-12-13 13:03:41 +01:00
Przemyslaw Skibinski
334b8962b0 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-12-13 12:19:19 +01:00
Przemyslaw Skibinski
671f28d1e5 added parseCompressionParameters 2016-12-13 12:18:07 +01:00
Dimitry Andric
83cc2fb083 Enable using isatty() and nanosleep() on *BSD 2016-12-12 19:24:51 +01:00
Yann Collet
4c6ce5a57e Merge pull request #476 from inikep/dev11 
Dev11
 2016-12-12 16:44:55 +01:00
Przemyslaw Skibinski
c855da7533 allow all operationg systems with _POSIX_C_SOURCE >= 200112L 2016-12-12 15:37:43 +01:00
Przemyslaw Skibinski
242c036875 turn on the '-r' option for HP-UX and AIX 2016-12-12 11:59:17 +01:00
Przemyslaw Skibinski
c01ac6c30a turn on the '-r' option for *BSD and Solaris 2016-12-12 11:23:21 +01:00
Yann Collet
fda539f50b minor coding style changes 2016-12-12 01:03:23 +01:00
Przemyslaw Skibinski
128acb35db improved MSYS support 2016-12-08 10:49:59 +01:00
Przemyslaw Skibinski
13de72e0cc programs\Makefile: use Linux paths 2016-12-08 10:43:55 +01:00
Yann Collet
2a826e592b compatibility with zstd-frugal (noBench mode) 2016-12-06 17:56:20 -08:00
Yann Collet
e63c631aaf decode benchmark, multi-files 2016-12-06 17:46:49 -08:00
Yann Collet
d946501d2c decode benchmark - single file (hidden option) 2016-12-06 16:49:23 -08:00
Yann Collet
6b9a983261 changed gzstd build messages 2016-12-06 11:23:25 -08:00
Przemyslaw Skibinski
7c6bbc3298 updated FIO_passThrough 2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski
6b508b1770 updated test-gzstd 2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski
b493e3b3d3 introduced srcBufferLoaded 2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski
3c69760275 improved FIO_decompressGzFrame 2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski
8489f184f6 improved detection of -lz 2016-12-05 13:47:00 +01:00
Yann Collet
db85a6e09a added zstdgrep 2016-12-02 15:57:07 -08:00
Yann Collet
efaf104b2d added zstdless 2016-12-02 15:24:40 -08:00
Yann Collet
743b33f57e fix zstdcat 2016-12-02 15:18:57 -08:00
Yann Collet
5bd4237beb minor refactor 2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski
690753ea1a improved formatting in FIO_decompressSrcFile 2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski
4e49580407 removed testing artifacts 2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski
c5eebca128 rewritten FIO_decompressGzFile 2016-12-02 15:01:31 +01:00