Commit Graph

1343 Commits

Author SHA1 Message Date
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors
fixed usage of grep in Makefile
2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt
Adaptive compression
2018-09-26 14:39:08 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout
Changed default legacy support to v0.5+
2018-09-26 13:39:05 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile
when terminal uses colors
as suggested by @danielshir (#1294)
2018-09-25 16:56:53 -07:00
Yann Collet
6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel
defined a minimum negative level
2018-09-24 16:01:13 -07:00
Yann Collet
0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet
0fc07eb1fd fixed zstd-decompress
which cannot support ZSTD_minCLevel()
2018-09-22 17:21:39 -07:00
Yann Collet
364041c6dd enforce minimum compression level limit
using ZSTD_minCLevel()
2018-09-22 16:10:10 -07:00
Yann Collet
00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
Yann Collet
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet
bfff4f4809 ensure all writes to job->cSize are mutex protected
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
2018-09-21 16:00:39 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests
involving ZSTD_TARGETLENGTH_MIN (== 0)
2018-09-21 15:04:43 -07:00
Yann Collet
54001f3dbc fix mingw compatibility
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
2018-09-21 14:46:09 -07:00
Yann Collet
b2939163e1 Changed default legacy support to v0.5+
thus dropping read support for v0.4.

It's always possible to re-enable it, by changing build macro ZSTD_LEGACY_SUPPORT to 4.
2018-09-20 14:30:20 -07:00
Yann Collet
15519479ba fixed minor gcc warning on a unused variable 2018-09-20 13:00:11 -07:00
Yann Collet
45010da074 updated man page
and added `--adapt` test in `playTests.sh`
2018-09-19 17:37:22 -07:00
Yann Collet
ca02ebee07 removed static variables
so that --adapt can work on multiple input files too
2018-09-19 15:25:50 -07:00
Yann Collet
89bc309d90 error out when --adapt is associated with --single-thread
since they are not compatible
2018-09-19 14:49:13 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Casey McGinty
d019d15f2b
Add build define to manually enable -rdynamic LD flag 2018-09-12 14:00:08 -07:00
Casey McGinty
b703181f93
Disable -rdynamic LD option on Windows 2018-09-11 16:19:34 -07:00
Casey McGinty
b9118ecdab
Update comments, and LD flag usage in Make 2018-09-11 14:49:47 -07:00
Casey McGinty
2224ecd718
Remove dead code and method name typo 2018-09-11 11:56:50 -07:00
Casey McGinty
d4337b6f1d
Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
Casey McGinty
2a3967b7c4
Refactor abort signal handling
- Print signal name to term
- Add -rdynamic option to generate Linux symbol names in backtrace
- Raise default signal after handler to ensure program termination
2018-09-11 10:40:45 -07:00
Casey McGinty
a06574fc97
Print a stack trace on unexpected term signal (e.g. SIGABRT)
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
2018-09-06 18:46:52 -07:00
Yann Collet
deb03d8a26 fixed UTIL_isLink() for Mac OS-X 2018-09-06 17:32:16 -07:00
Yann Collet
50b216146f
Merge pull request #1304 from facebook/largeNbDicts
contrib/largeNbDicts
2018-09-06 09:50:56 -07:00
Jennifer Liu
21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Yann Collet
39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Jennifer Liu
f87383507d Update comment about default dictionary builder 2018-08-30 15:46:39 -07:00
Yann Collet
4086b2871b largeNbDicts compatible with multiple source files
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
2018-08-30 14:38:49 -07:00
Yann Collet
0c66a44d1b first working test program
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
2018-08-28 15:47:07 -07:00
Yann Collet
b37a0a6bde
Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
Yann Collet
0491037db9 Merge branch 'bench' into largeNbDicts 2018-08-28 11:26:46 -07:00
Yann Collet
55affc09de timedFn : measurement delay is programmable
instead of hard-coded 1 second per measurement
2018-08-28 11:26:27 -07:00
Yann Collet
d97e92dfad Merge branch 'bench' into largeNbDicts 2018-08-27 12:12:51 -07:00
Yann Collet
01dcd0fd17 bench: minor api update, for consistency
BMK_benchTimedFn()
BMK_isCompleted_TimedFn() uses TimedFnState
2018-08-26 21:30:18 -07:00
Yann Collet
6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann Collet
c3a4baaf6e fixed minor warnings
valgrind: memory leak of a few bytes in fullbench
static analyzer: uninitialized data passed as result
2018-08-24 23:25:35 -07:00
Yann Collet
2279f3d127 bench: reduce nb of return type
runOutcome is enough
removed timedFnOutcome
2018-08-24 17:28:38 -07:00
Yann Collet
6ce7b08f17 fix minor warnings
gcc : prototype with 0 parameter must be labelled (void)
visual : const property must be identical in both declaration and implementation
2018-08-24 15:59:57 -07:00
Yann Collet
4da5bdf482 fixed zstd -b speed result
the benchmark was displaying the speed of last run
instead of the best of all previous runs.
2018-08-23 18:13:49 -07:00
Yann Collet
1f9ec13621 introduced MB_UNIT
so that all benchmarking programs use the same speed scale
2018-08-23 16:03:30 -07:00
Yann Collet
d39a25c5ed update fullbench.c to work with new bench.h 2018-08-23 15:00:09 -07:00
Yann Collet
2e45badff4 refactored bench.c
for clarity and safety, especially at interface level
2018-08-23 14:21:18 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet
77e805e3db bench: changed creation/reset function to timedFnState
for consistency
2018-08-21 18:19:27 -07:00