AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,752 Commits 1 Branch 0 Tags 27 MiB
5d89771529
Commit Graph

1545 Commits

Author SHA1 Message Date
Yann Collet
a29e3d1d01 fix debug trace 2018-11-13 13:27:22 -08:00
Yann Collet
b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet
9867cdb847 benchfn can provided faulty return value 
with BMK_extract_returnValue()
 2018-11-13 12:01:17 -08:00
Yann Collet
d38063f8ae separated bench module into benchfn and benchzstd 
it shall be possible to use benchfn
without any dependency on zstd.
 2018-11-13 11:01:59 -08:00
Yann Collet
9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Yann Collet
8bed4012bd fixed decompression-only benchmark 2018-11-08 12:36:39 -08:00
Bernhard M. Wiedemann
572279e73e pgo: also optimize compression 
and clean *.o so that linking does not fail from code
that is partially augmented with coverage generation instructions
 2018-10-29 17:20:03 +01:00
Bernhard M. Wiedemann
a38ad53ace zstd-pgo: do not make clean and zstd in parallel 
otherwise, there might not be a ./zstd to run
 2018-10-29 15:11:28 +01:00
Yann Collet
fc20b3c441 added flag -Wc++-compat 
for library and cli
 2018-10-26 16:38:23 -07:00
Yann Collet
fab150ce9f fix : removed zstdmt symlink on uninstall 2018-10-22 12:51:56 -07:00
Yann Collet
f181799082 fix decodecorpus incorrect frame generation 
fix #1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.

The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
 2018-10-20 18:56:21 -07:00
Yann Collet
f2cff22804
Merge pull request #1368 from rkjain89/test-branch 
Moving Code To util.c
 2018-10-19 16:31:10 -07:00
Sam Russell
622699706c fix up zgrep/zless references 2018-10-16 08:42:02 +13:00
Sam Russell
490f652974 Add man pages for zstdgrep and zstdless 2018-10-15 22:25:03 +13:00
Rohit Jain
c7251e5151 Address most comments 2018-10-11 18:05:15 -07:00
Rohit Jain
91b2fed8f8 Moving more function to the new C file 2018-10-11 17:34:47 -07:00
Rohit Jain
a47f6e6f2d Removing static and making extern 2018-10-11 16:51:29 -07:00
Yann Collet
5d419071be
Merge pull request #1369 from facebook/v137 
bumped version number to v1.3.7
 2018-10-11 15:59:57 -07:00
Rohit Jain
705e0b18ab Making changes to make it compile on my laptop 2018-10-11 15:51:57 -07:00
Rohit Jain
b19140bc13 Fixed makefile and removed multiple definitions from util.h (UTIL_getTime) 2018-10-11 15:17:40 -07:00
Rohit Jain
d6d240f016 Removed UTIL_setFileStat from util.h and move it to util.c 2018-10-11 15:07:12 -07:00
Yann Collet
1e6208e75e bumped version number to v1.3.7 
updated documentation
 2018-10-11 14:40:12 -07:00
Rohit Jain
f881ee8c48 Moving code to util.c 2018-10-11 14:38:41 -07:00
Yann Collet
433059bbb2 fixed minor gcc warning 
gcc-8 on Linux doesn't like usage of strncat :
`warning: ‘strncat’ output truncated before terminating nul copying as many bytes from a string as its length`.

Not sure what was wrong, it might be a false positive,
but the logic is simple enough to replaced by a simple `memcpy()`,
thus avoiding the shenanigans of null-terminated strings.
 2018-10-10 17:06:25 -07:00
Yann Collet
b304b679e5 use #if BACKTRACE_ENABLE directly 
as suggested by @terrelln
 2018-10-09 17:56:59 -07:00
Julian Fessard
0ea286f740 Merge branch 'backtrace' of github.com:facebook/zstd into backtrace 2018-10-09 17:24:48 -07:00
Julian Fessard
70d8c2a031 fileio.c: Disable backtrace when built with address sanitizer 
Covers clang and gcc's sanitizer flags.
Can still be overridden through CFLAGS on commandline.
 2018-10-09 17:14:57 -07:00
Yann Collet
e0ab6b61b7 fixed explicit BACKTRACE order 
and automatic linux backtrace detection :
__GLIBC__ must be tested after #include <features.h>
 2018-10-09 17:12:21 -07:00
Yann Collet
1e0c5466c5 fixed BACKTRACE_ENABLE macro test 2018-10-09 16:36:57 -07:00
Yann Collet
f17c1df1ac backtrace support compiled with more conditions 
following #1356,
only enable backtrace compilation on linux+glibc.

Also, disable backtrace by default from "release" compilation,
so that less platforms get impacted by the new requirements.
Can be manually enabled/disabled using BACKTRACE=1/0.
 2018-10-08 17:03:06 -07:00
maxice8
1d75697d9b
don't assume __linux__ means __GLIBC__ on BACKTRACES_ENABLE 
check for __GLIBC__ instead of __linux__ as musl libc doesn't provide
execinfo.h

fixes compilation on Alpine Linux and Void Linux musl arches.
 2018-10-05 21:23:45 -03:00
Yann Collet
0854b06d1b
Merge pull request #1351 from facebook/haikubuild 
portability macro flags updates, for Haiku
 2018-10-04 14:59:35 -07:00
Yann Collet
efbc3e823d fixed paramgrill wrong assert() conditions 
and slightly refactored affected function.

Honestly, the formula calculating variance should get a second reviewing round,
it's not clear if it's correct.
 2018-10-04 14:27:13 -07:00
Yann Collet
4a85b126d9 changed ZSTD_NANOSLEEP_SUPPORT definition 
to please `-Wexpansion-to-defined`
 2018-10-03 15:34:41 -07:00
Yann Collet
549c19b42e portability macro flags updates, for Haiku 
some non-trivial changes to platform.h and util.h,
initially related to compilation for Haiku,
but I used this opportunity to make them cleaner
and add some documentation.

Noticed several tests that could be improved
(too harsh conditions, useless exception, etc.)
but I did not dare modifying too many tests just before release.
 2018-10-03 14:56:46 -07:00
Yann Collet
bc93b801f0
Merge pull request #1330 from korli/haiku 
Enable building zstd on Haiku.
 2018-10-03 13:36:00 -07:00
Yann Collet
b1407f9acd fixed wrong assert() position 
could fire on invalid input.
blocking for afl tests.
 2018-10-03 12:43:59 -07:00
Jerome Duval
87c10e2f58 Enable building zstd on Haiku. 2018-10-03 09:51:56 +02:00
Yann Collet
c9843ec232
Merge pull request #1348 from facebook/donotdelete 
Fix #1082
 2018-10-02 16:37:58 -07:00
Yann Collet
3ca6261223 fixed static analyzer warnings 
note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.

They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
 2018-10-02 15:59:11 -07:00
Yann Collet
d98733b37e restored backtrace on failure 
for Linux and Mac OS-X.

Note : the backtraces fires up through a trap
before the sanitizer get a chance to report.
There are situations where the sanitizer report is actually preferable.

It might be good to consider a kind of build macro
which can disable backtrace
when sanitizer is enabled.
 2018-10-01 17:50:16 -07:00
Yann Collet
9012b6cba0 ./zstd -f do no longer overwrite destination file 
if source file does not exist (#1082)
 2018-10-01 17:16:34 -07:00
Yann Collet
c7bd6a41ab zstd -d -f do no longer erase destination file 
when source file does not exist (#1082)
 2018-10-01 14:04:00 -07:00
Yann Collet
1ab71a8e72 regroup name creation logic into its own function 
for a cleaner main file decompression loop
 2018-09-28 18:19:23 -07:00
Yann Collet
ec1cb8e996 changed macro name 
from EXIT_IF() to RETURN_IF()

EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
 2018-09-28 16:04:00 -07:00
Yann Collet
05c0a072b7 minor improvement in the multi-format suffix selection 2018-09-28 15:57:35 -07:00
Nick Terrell
09231dad4d [util] Fix lstat feature test macro 2018-09-28 15:23:09 -07:00
Yann Collet
d987ab5983 fixed unreachable section warning on Visual 2018-09-28 09:34:16 -07:00
Yann Collet
913a0365b6 Merge branch 'dev' into donotdelete 2018-09-27 19:23:55 -07:00
Yann Collet
ef1272737b fixed minor Visual conversion warnings 2018-09-27 18:29:15 -07:00
Yann Collet
9b45db7fa6 minor refactoring of --list 
trying to reduce recurrent patterns.
 2018-09-27 16:49:08 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors 
fixed usage of grep in Makefile
 2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt 
Adaptive compression
 2018-09-26 14:39:08 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout 
Changed default legacy support to v0.5+
 2018-09-26 13:39:05 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile 
when terminal uses colors
as suggested by @danielshir (#1294)
 2018-09-25 16:56:53 -07:00
Yann Collet
6c51bf420c bounds for --adapt mode 
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
 2018-09-25 16:03:28 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel 
defined a minimum negative level
 2018-09-24 16:01:13 -07:00
Yann Collet
0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet
0fc07eb1fd fixed zstd-decompress 
which cannot support ZSTD_minCLevel()
 2018-09-22 17:21:39 -07:00
Yann Collet
364041c6dd enforce minimum compression level limit 
using ZSTD_minCLevel()
 2018-09-22 16:10:10 -07:00
Yann Collet
00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
Yann Collet
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet
bfff4f4809 ensure all writes to job->cSize are mutex protected 
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
 2018-09-21 16:00:39 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests 
involving ZSTD_TARGETLENGTH_MIN (== 0)
 2018-09-21 15:04:43 -07:00
Yann Collet
54001f3dbc fix mingw compatibility 
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
 2018-09-21 14:46:09 -07:00
Yann Collet
b2939163e1 Changed default legacy support to v0.5+ 
thus dropping read support for v0.4.

It's always possible to re-enable it, by changing build macro ZSTD_LEGACY_SUPPORT to 4.
 2018-09-20 14:30:20 -07:00
Yann Collet
15519479ba fixed minor gcc warning on a unused variable 2018-09-20 13:00:11 -07:00
Yann Collet
45010da074 updated man page 
and added `--adapt` test in `playTests.sh`
 2018-09-19 17:37:22 -07:00
Yann Collet
ca02ebee07 removed static variables 
so that --adapt can work on multiple input files too
 2018-09-19 15:25:50 -07:00
Yann Collet
89bc309d90 error out when --adapt is associated with --single-thread 
since they are not compatible
 2018-09-19 14:49:13 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Casey McGinty
d019d15f2b
Add build define to manually enable -rdynamic LD flag 2018-09-12 14:00:08 -07:00
Casey McGinty
b703181f93
Disable -rdynamic LD option on Windows 2018-09-11 16:19:34 -07:00
Casey McGinty
b9118ecdab
Update comments, and LD flag usage in Make 2018-09-11 14:49:47 -07:00
Casey McGinty
2224ecd718
Remove dead code and method name typo 2018-09-11 11:56:50 -07:00
Casey McGinty
d4337b6f1d
Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
Casey McGinty
2a3967b7c4
Refactor abort signal handling 
- Print signal name to term
- Add -rdynamic option to generate Linux symbol names in backtrace
- Raise default signal after handler to ensure program termination
 2018-09-11 10:40:45 -07:00
Casey McGinty
a06574fc97
Print a stack trace on unexpected term signal (e.g. SIGABRT) 
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
 2018-09-06 18:46:52 -07:00
Yann Collet
deb03d8a26 fixed UTIL_isLink() for Mac OS-X 2018-09-06 17:32:16 -07:00
Yann Collet
50b216146f
Merge pull request #1304 from facebook/largeNbDicts 
contrib/largeNbDicts
 2018-09-06 09:50:56 -07:00
Jennifer Liu
21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Yann Collet
39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Jennifer Liu
f87383507d Update comment about default dictionary builder 2018-08-30 15:46:39 -07:00
Yann Collet
4086b2871b largeNbDicts compatible with multiple source files 
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
 2018-08-30 14:38:49 -07:00
Yann Collet
0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet
b37a0a6bde
Merge pull request #1298 from facebook/bench 
Refactored bench.c
 2018-08-28 12:25:02 -07:00
Yann Collet
0491037db9 Merge branch 'bench' into largeNbDicts 2018-08-28 11:26:46 -07:00
Yann Collet
55affc09de timedFn : measurement delay is programmable 
instead of hard-coded 1 second per measurement
 2018-08-28 11:26:27 -07:00
Yann Collet
d97e92dfad Merge branch 'bench' into largeNbDicts 2018-08-27 12:12:51 -07:00
Yann Collet
01dcd0fd17 bench: minor api update, for consistency 
BMK_benchTimedFn()
BMK_isCompleted_TimedFn() uses TimedFnState
 2018-08-26 21:30:18 -07:00
Yann Collet
6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann Collet
c3a4baaf6e fixed minor warnings 
valgrind: memory leak of a few bytes in fullbench
static analyzer: uninitialized data passed as result
 2018-08-24 23:25:35 -07:00
Yann Collet
2279f3d127 bench: reduce nb of return type 
runOutcome is enough
removed timedFnOutcome
 2018-08-24 17:28:38 -07:00
Yann Collet
6ce7b08f17 fix minor warnings 
gcc : prototype with 0 parameter must be labelled (void)
visual : const property must be identical in both declaration and implementation
 2018-08-24 15:59:57 -07:00
Yann Collet
4da5bdf482 fixed zstd -b speed result 
the benchmark was displaying the speed of last run
instead of the best of all previous runs.
 2018-08-23 18:13:49 -07:00
Yann Collet
1f9ec13621 introduced MB_UNIT 
so that all benchmarking programs use the same speed scale
 2018-08-23 16:03:30 -07:00
Yann Collet
d39a25c5ed update fullbench.c to work with new bench.h 2018-08-23 15:00:09 -07:00
Yann Collet
2e45badff4 refactored bench.c 
for clarity and safety, especially at interface level
 2018-08-23 14:21:18 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274) 
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
 2018-08-23 12:06:20 -07:00
Yann Collet
77e805e3db bench: changed creation/reset function to timedFnState 
for consistency
 2018-08-21 18:19:27 -07:00
Yann Collet
801e3bcd97
Merge pull request #1290 from edenzik/ezik/1119-safe-strcpy-in-fileio 
Fixed unsafe string copy and concat in `fileio.c`.
 2018-08-21 13:18:44 -07:00
Eden Zik
78af534f82 Fixed unsafe string copy and concat in fileio.c. 
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c
 2018-08-20 22:15:24 -04:00
Yann Collet
105677c6db created ZSTDMT_toFlushNow() 
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
 2018-08-17 18:11:54 -07:00
Yann Collet
09e63c58ac fix : no longer slow down on input saturation 
only slows down when all buffers are full
 2018-08-17 16:27:43 -07:00
Yann Collet
8b674d7dc7 ensured compression level is maxed at ZSTD_maxCLevel() 2018-08-17 16:01:56 -07:00
Yann Collet
b4e7f71055 Merge branch 'dev' into adapt 2018-08-17 15:54:13 -07:00
Yann Collet
42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
George Lu
e89f1fb45c Fix scan-build warnings in bench.c 2018-08-14 14:44:47 -07:00
Yann Collet
3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
Yann Collet
973a8d42c7
Merge pull request #1236 from GeorgeLu97/paramgrillconstraints 
ParamgrillConstraints
 2018-08-13 15:44:50 -07:00
Yann Collet
0853f86044 adaptive mode uses default window size of 8 MB 2018-08-13 13:13:22 -07:00
Yann Collet
33f7709c71 fileio: changed parameter type from ptr to plain structure 
safer : this parameter is read-only,
we don't want original structure to be modified
 2018-08-13 13:02:03 -07:00
Yann Collet
f3aa510738 rateLimiter does not "catch up" when input speed is slow 2018-08-13 11:38:55 -07:00
Yann Collet
e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
Yann Collet
9d26cb6a75 slow down faster when output speed is limited 2018-08-09 17:44:30 -07:00
Yann Collet
3d7b533f68 Merge branch 'dev' into adapt 2018-08-09 15:57:36 -07:00
Yann Collet
754942cb79 fixed assert() condition 2018-08-09 15:57:19 -07:00
Yann Collet
2dd76037be zstd cli can increase level when input is too slow 2018-08-09 15:51:30 -07:00
Yann Collet
79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
Yann Collet
51e71a5ec7 added zstdgrep documentation 
presenting `zstdgrep` limit regarding dictionary compression
with workaround recommended by @tobwen (#1268)
 2018-08-09 12:28:25 -07:00
George Lu
bfe8392e23 Remove ctx from benchMem 2018-08-09 12:07:57 -07:00
George Lu
8278a49cb6 const srcPtrs 2018-08-09 10:42:58 -07:00
George Lu
3d230db853 Change speed representation from floating point to integral 2018-08-09 10:42:58 -07:00
George Lu
dd270b2f75 Renaming / Style fixes 2018-08-09 10:42:58 -07:00
George Lu
e148db366e Separate capacity vs size 
Also:
Make suggested fixes
-varInds_t
-reorder some arguments
-remove code duplication
-update README / -h
-Fix memory leaks
 2018-08-09 10:42:58 -07:00
George Lu
df026e159f Fix windows implicit casting bugs 2018-08-09 10:42:58 -07:00
George Lu
7b5b3d7ae3 BenchMem with block compressed sizes passed back up 2018-08-09 10:42:58 -07:00
George Lu
3adc217ea4 Total Changes: 
Add different constraint types (decompression speed, compression memory, parameter constraints)
Separate search space by strategy + strategy selection
Memoize results
Real random restarts
Support multiple files
Support Dictionary inputs
Debug Macro for extra printing
 2018-08-09 10:42:58 -07:00
George Lu
eb21b7f482 Not crashing 2018-08-09 10:42:58 -07:00
George Lu
5f49034520 Working V1 2018-08-09 10:42:58 -07:00
George Lu
cffb6da339 Parses additional parameters 
Additional constraint checking

Minor fixes

more param parsing

Add Memory

Change paramVariation

work on feasibility

reformat bench

Changed Paramgrill to use bench.c benchmarking

customlevel macro

Printing Flag

Minor changes

Explicit casting

Makefile fix

casting, type fix

Printing Flag

Minor Changes

comments, helper fn's
 2018-08-09 10:42:58 -07:00
Yann Collet
5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Yann Collet
2fdab1629b fix unused variable warning 2018-08-03 08:30:01 -07:00
Yann Collet
5203f01774 fix : zstd cli can be built with build macro ZSTD_NOBENCH 
which disables bench.c module
 2018-08-03 07:54:29 -07:00
cyan4973
3f535007e4 fix %zu support under minGW 
and relevant test on Appveyor
 2018-07-30 16:56:18 +02:00
George Lu
09ccd977c3 no zero 2018-07-26 15:17:58 -07:00
Yann Collet
effa84c8d1
Merge pull request #1230 from terrelln/train-out 
zstdcli: Allow -o before --train
 2018-07-18 16:34:10 +02:00
Nick Terrell
4e706d7f2c fileio: Error in compression on read errors 
We can write a corrupted file if the input file errors during a read.
We should return a non-zero error code in this case.
 2018-07-17 15:26:30 -07:00
Nick Terrell
58b8219475 zstdcli: Allow -o before --train 
Only set the default value if `outFileName` is unset.

Fixes #1227.
 2018-07-16 12:45:34 -07:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev 
Split samples when building dictionary for COVER
 2018-07-13 13:26:15 -07:00
Jennifer Liu
612b346ed5 Add explanation for split=100 2018-07-11 15:50:28 -07:00
Jennifer Liu
5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu
bfad1af031 Update doc for split==100 2018-07-05 11:05:31 -07:00
Jennifer Liu
0881184c89 Some edits based on pull request comments 2018-07-03 17:53:27 -07:00
Yann Collet
689bfecd48
Merge pull request #1188 from GeorgeLu97/BenchModule 
Bench module
 2018-07-02 13:33:27 -07:00
Jennifer Liu
8afcb8eea7 Update documentation 2018-07-01 19:59:37 -07:00
Jennifer Liu
84e8b2a305 Fix another declaration issue 2018-06-29 18:02:02 -07:00
Jennifer Liu
348e5f77a9 Add split=# to cli 2018-06-29 17:54:41 -07:00
Yann Collet
b5207aadfa make build tests more unforgiving 
`-Werror` will ensure they fail if there is the slightest warning.

fix a minor warning specific to `zstd_decompress` variant.
 2018-06-29 17:10:56 -07:00
W. Felix Handte
712a9fd972 Allow Invoking zstd --list When stdin is not a tty 
Also now returns an error when no inputs are given.

New proposed behavior:

```
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l; echo $?
No files given
1
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst; echo $?
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0     3.08 KB      10.92 KB  3.544  XXH64  Makefile.zst
0
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l <Makefile.zst; echo $?
zstd: --list does not support reading from standard input
No files given
1
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst <Makefile.zst; echo $?
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0     3.08 KB      10.92 KB  3.544  XXH64  Makefile.zst
0
felix@odin:~/prog/zstd (list-stdin-check)$
```
 2018-06-29 15:33:44 -04:00
Yann Collet
a2c3a4cd0e
Merge pull request #1214 from jennifermliu/dev 
Make --fast=0 fail
 2018-06-27 18:53:39 -07:00
Yann Collet
1fd621ff6d minor man page update 
regarding advanced parameter `tlen`
which was recently changed.
`0` in association with `ZSTD_fast` now means "normal fast mode".
 2018-06-27 18:49:02 -07:00
Jennifer Liu
1ab57a7ce1 Redirect failed test result to INTOVOID and update comment about parsing fast command 2018-06-27 16:27:45 -07:00
Jennifer Liu
aef8486fee Make fast=0 fail 2018-06-27 14:27:27 -07:00
cyan4973
f741fb8fcd minor fixes for MSYS2 compilation 2018-06-26 01:22:45 -07:00
George Lu
50d612f4f0 Interleave compression/decompression 
Fix Bugs
 2018-06-25 15:01:03 -07:00
George Lu
d6121ad0e1 Opaque State 
And minor fixups (comments/alignment/checks/fix memory leak)
 2018-06-25 08:07:43 -07:00
George Lu
ab26f24c9c benchFunction Timed Wrappers 
Add BMK_benchFunctionTimed
Add BMK_init_customResultCont..
Change benchMem to use benchFunctionTimed
Minor Fixes/Adjustments
 2018-06-21 16:23:55 -07:00
George Lu
a8eea99ebe Incremental Display + Fn Separations 
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
 2018-06-21 16:23:18 -07:00
Yann Collet
93702a7a62
Merge pull request #1198 from facebook/msdebug 
made Visual Studio compatible with DEBUGLEVEL >= 2
 2018-06-20 12:26:31 -07:00
cyan4973
ae0b7ffa0a made Visual Studio compatible with DEBUGLEVEL >= 2 2018-06-20 09:45:02 -07:00
Yann Collet
6768cf53fd
Merge pull request #1190 from terrelln/ldm-adjust 
Adjust advanced parameters to source size
 2018-06-19 14:40:56 -07:00
Yann Collet
c0b6ce95b1
Merge pull request #1179 from supertopher/dev 
Improves UX for --list command's lack of support for pipes
 2018-06-19 14:36:30 -07:00
Nick Terrell
1d0fcde45d Use debug.h in fileio.c 2018-06-18 15:51:21 -07:00
Nick Terrell
3841dbac84 Adjust advanced parameters to source size 
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
 2018-06-18 15:49:31 -07:00
George Lu
a3c8b59990 Fix cli no print 
Change looping behavior to match old
 2018-06-18 15:38:14 -07:00
George Lu
e482e328cd Reorder Arguments 
make initFn nullable
 2018-06-18 13:21:42 -07:00
George Lu
0d1ee22990 Requested Changes 
Add Comment
Simplify Interface (Remove resultSet)
Reorder Arguments
Remove customBench displayLevel
Reorder bench.h
Change benchFiles return type to match advanced
Rename stuff
 2018-06-18 12:01:12 -07:00
George Lu
8522346322 Make Fullbench use new function 
Rearrange Args
Add nothing function
Use new function, change locals to match
New Display
Comment cleanup
Change builds
 2018-06-15 11:37:49 -04:00
George Lu
20f4f32379 Add to bench 
-Remove global variables
-Remove gv setting functions
-Add advancedParams struct
-Add defaultAdvancedParams();
-Change return type of bench Files
-Change cli to use new interface
-Changed error returns to own struct value
-Change default compression benchmark to use decompress_generic
-Add CustomBench function
-Add Documentation for new functions
 2018-06-14 14:23:24 -04:00
Topher Lubaway
6bca3fb4bf Reduce noise in diff 
putting the code block back on the exact line it came from
 2018-06-13 14:32:59 -07:00
Topher Lubaway
ec24f98cca Removes duplicate IS_CONSOLE from PR 
I misunderstood that this function was included already
 2018-06-13 13:39:23 -07:00
Yann Collet
c986dbf241
Merge pull request #1168 from GeorgeLu97/paramgrillfeatures 
Have paramgrill share bench.c benchmarking function
 2018-06-13 11:38:29 -04:00
George Lu
01d940b670 Requested changes 
-Remove g_displaylevel/setNotificationLevel function
-Add extern "C"
-Remove averaging
-Reorder arguments

More fixes

-Added BMK_return_t (result + possible error)
-Correct comment'
-Nullcheck ctx, dctx when allocated
-Remove extra assert
 2018-06-12 17:02:44 -04:00
Topher Lubaway
b024e1e1f4 Keep windows specific headers 
Accidentially deleted this existing windows only header
 2018-06-12 10:16:27 -07:00
Topher Lubaway
88ae51acb3 Multi-OS support for --list detecting stream input 
IS_CONSOLE stolen wholesale from Options.cpp
not sure if i should have extracted that code for DRY-ness
tested in OSX and functionality seems appropriate
unstested in a windows environment
 2018-06-12 07:59:17 -07:00
Topher Lubaway
881defaeb3 Only check for tty in non-windows environments 
unistd.h is for unix standard tools.
There does not appear to be a simple isatty for windows
this we only run the logic and header include in
non-windows environments
 2018-06-11 15:26:35 -07:00
Topher Lubaway
5ca1d5c6f4 Properly brackets isatty if statement 
¯\_(ツ)_/¯ this is my first commit in c
 2018-06-11 12:19:15 -07:00
Topher Lubaway
4c16608e3c Improves UX for --list command's lack of support for pipes 
--list does not support piped input
This checks for a terminal and exits 1 with a well formatted
error message if the STDIN is not from a terminal
 2018-06-11 10:13:00 -07:00
Ryan Schmidt
b567ce9d68 Fix name of macOS 2018-06-09 14:31:17 -05:00
George Lu
0e808d608b Make paramgrill use bench.c benchmarking 2018-06-08 12:01:05 -07:00
Yann Collet
d3615c28db
Merge pull request #1159 from GeorgeLu97/suffixlist 
Unknown Suffix Error
 2018-06-01 14:00:10 -07:00
George Lu
8984cc93d6 update display 2018-05-31 18:04:05 -07:00
George Lu
547096d672 update man 2018-05-31 18:03:52 -07:00
George Lu
c9b1068298 removed strcats 2018-05-31 17:47:29 -07:00
George Lu
5ff30fe2e5 Unknown Suffix Error 
Changed so only compiled formats are printed in list of supported extensions
 2018-05-31 16:13:36 -07:00
George Lu
140f59d38e Added --format=zstd 
title
 2018-05-31 15:29:35 -07:00
Yann Collet
174bd3d4a7
Merge pull request #1131 from facebook/zstdcli 
minor: control numeric argument overflow
 2018-05-14 11:53:58 -07:00
Yann Collet
9cd5c63771 cli: control numeric argument overflow 
exit on overflow
backported from paramgrill
added associated test case
 2018-05-12 14:29:33 -07:00
Yann Collet
b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
cyan4973
62487b5e76 fixed decoding bogus lz4 frame 
FIO would keep presenting data after an LZ4F decoding error
resulting in a NULL pointer dereference
when associated with older liblz4 version (< v1.8.1.2)
 2018-04-23 18:50:16 -07:00
Yann Collet
1da629f2ad
Merge pull request #1104 from terrelln/fast-train 
Allow negative compression levels in training
 2018-04-09 14:16:20 -07:00
Nick Terrell
569e2abccd Allow negative compression levels in training 
* Set `dictCLevel` in `zstdcli.c`.
* Only set to default level if the compression level `== 0`, not `<= 0`.
 2018-04-09 12:12:03 -07:00
Björn Ketelaars
e5ea8d272a fix typo in programs/zstd.{1,1.md} 
s/nodictID/no-dictID/g
 2018-04-05 06:44:46 +02:00
Yann Collet
7188862d32
Merge pull request #1086 from hagemt/hagemt-patch-1 
Correct small typo in manual (man file and markdown)
 2018-03-30 20:45:10 -06:00
Tor E Hagemann
c7a5e60bc6
Update zstd.1.md 2018-03-30 15:25:32 -07:00
Tor E Hagemann
292d370ab4
Update zstd.1 2018-03-30 14:53:57 -07:00
Yann Collet
525f3fab33 restored ability to manually set overlapLog 2018-03-28 11:33:41 -06:00