Commit Graph

68 Commits

Author SHA1 Message Date
Yann Collet
989e0f0cee Added --output-dir-flat=
which is a preferred construction when applicable.

Also, added --output-dir-flat into man page.
2019-10-21 14:39:18 -07:00
Yann Collet
ca73c218be added mode documentation on ZSTD_CLEVEL
providing range restriction [1-19] explicitly
partially answering #1829
2019-10-18 17:22:45 -07:00
Nick Magerko
493f95c7df Fix merge conflicts 2019-08-22 11:51:41 -07:00
Nick Magerko
fd486a846a Differentiate --stream-size from --size-hint 2019-08-22 09:37:47 -07:00
Nick Magerko
2cdda8b3c4 Minor documentation update 2019-08-22 09:13:28 -07:00
Nick Magerko
05d7479a50 Document --size-hint 2019-08-20 14:08:26 -07:00
Nick Magerko
bbd83c2ab3 Update man page 2019-08-19 09:11:22 -07:00
Tyler Tran
f8c1d7979c modifying minor nit 2019-07-22 16:36:44 -07:00
Tyler Tran
02da4497f0 Adding documentation for shrink flag PR #1656 2019-07-22 16:33:22 -07:00
Ori Peleg
bdeb4786b5 Fixed -r typo 2019-04-10 13:37:41 +03:00
Yann Collet
517d8c984c
Merge pull request #1449 from facebook/ovlog_def
overlapLog default values
2018-12-18 09:45:53 -08:00
Nick Terrell
bdfcaecc0a [zstdcli] Add --no-progress flag
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
2018-12-14 11:50:25 -08:00
Yann Collet
1993f5d412 fixed ovlog tests
and updated man page
2018-12-12 21:09:14 -08:00
Yann Collet
34aa401afd updated documentation
introducing ZSTD_btultra2
2018-12-06 17:22:19 -08:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
Nick Terrell
f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
Yann Collet
6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet
45010da074 updated man page
and added `--adapt` test in `playTests.sh`
2018-09-19 17:37:22 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Jennifer Liu
21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Jennifer Liu
f87383507d Update comment about default dictionary builder 2018-08-30 15:46:39 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet
0853f86044 adaptive mode uses default window size of 8 MB 2018-08-13 13:13:22 -07:00
Yann Collet
f3aa510738 rateLimiter does not "catch up" when input speed is slow 2018-08-13 11:38:55 -07:00
Yann Collet
e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev
Split samples when building dictionary for COVER
2018-07-13 13:26:15 -07:00
Jennifer Liu
612b346ed5 Add explanation for split=100 2018-07-11 15:50:28 -07:00
Jennifer Liu
5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu
bfad1af031 Update doc for split==100 2018-07-05 11:05:31 -07:00
Jennifer Liu
348e5f77a9 Add split=# to cli 2018-06-29 17:54:41 -07:00
Yann Collet
1fd621ff6d minor man page update
regarding advanced parameter `tlen`
which was recently changed.
`0` in association with `ZSTD_fast` now means "normal fast mode".
2018-06-27 18:49:02 -07:00
George Lu
547096d672 update man 2018-05-31 18:03:52 -07:00
Yann Collet
b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
Björn Ketelaars
e5ea8d272a fix typo in programs/zstd.{1,1.md}
s/nodictID/no-dictID/g
2018-04-05 06:44:46 +02:00
Tor E Hagemann
c7a5e60bc6
Update zstd.1.md 2018-03-30 15:25:32 -07:00
Yann Collet
a57d43d4d4 updated documentation of targetLength 2018-03-12 11:35:01 -07:00
Yann Collet
6a9b41b731 create command --fast[=#]
access negative compression levels from command line
for both compression and benchmark modes.

also : ensure proper propagation of parameters
through ZSTD_compress_generic() interface.

added relevant cli tests.
2018-03-11 20:01:23 -07:00
Yann Collet
75689838e4 specify new command --single-thread 2018-02-09 15:55:41 -08:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
2018-01-27 22:23:26 -08:00
ne-sted
50aea2f293 cli: fix align of defaults 2018-01-24 15:07:22 +02:00
Scott Baker
31a191b178 Include information about the benchmark output/methodology
Addresses #930
2017-11-22 20:34:25 -08:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
W. Felix Handte
dc27c36495 Update documentation to reflect other format support 2017-09-28 19:43:12 -04:00
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
086b9597d9 added ability to split input files for dictionary training
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
2017-09-14 16:45:10 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Yann Collet
1c108c811e cli : Display supported formats on -vV command
Requested and inspired by patch from @ib (#771)
2017-08-19 13:33:50 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760)
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00