Commit Graph

70 Commits

Author SHA1 Message Date
Yann Collet
1993f5d412 fixed ovlog tests
and updated man page
2018-12-12 21:09:14 -08:00
Yann Collet
34aa401afd updated documentation
introducing ZSTD_btultra2
2018-12-06 17:22:19 -08:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
Yann Collet
092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet
1e6208e75e bumped version number to v1.3.7
updated documentation
2018-10-11 14:40:12 -07:00
Yann Collet
45010da074 updated man page
and added `--adapt` test in `playTests.sh`
2018-09-19 17:37:22 -07:00
Jennifer Liu
21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Jennifer Liu
f87383507d Update comment about default dictionary builder 2018-08-30 15:46:39 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev
Split samples when building dictionary for COVER
2018-07-13 13:26:15 -07:00
Jennifer Liu
612b346ed5 Add explanation for split=100 2018-07-11 15:50:28 -07:00
Jennifer Liu
5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu
bfad1af031 Update doc for split==100 2018-07-05 11:05:31 -07:00
Jennifer Liu
8afcb8eea7 Update documentation 2018-07-01 19:59:37 -07:00
Yann Collet
1fd621ff6d minor man page update
regarding advanced parameter `tlen`
which was recently changed.
`0` in association with `ZSTD_fast` now means "normal fast mode".
2018-06-27 18:49:02 -07:00
Yann Collet
b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
Björn Ketelaars
e5ea8d272a fix typo in programs/zstd.{1,1.md}
s/nodictID/no-dictID/g
2018-04-05 06:44:46 +02:00
Tor E Hagemann
292d370ab4
Update zstd.1 2018-03-30 14:53:57 -07:00
systemcrash
d13a75c969
Update zstd.1 2018-01-29 18:38:02 +01:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
2018-01-27 22:23:26 -08:00
ne-sted
50aea2f293 cli: fix align of defaults 2018-01-24 15:07:22 +02:00
Yann Collet
2cff66b62f version bump to v1.3.3 2017-12-14 16:11:20 -08:00
Yann Collet
2fd765498a updated man page
following patch #931 by @scottchiefbaker
2017-11-24 17:20:54 -08:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
W. Felix Handte
dc27c36495 Update documentation to reflect other format support 2017-09-28 19:43:12 -04:00
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Yann Collet
1c108c811e cli : Display supported formats on -vV command
Requested and inspired by patch from @ib (#771)
2017-08-19 13:33:50 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760)
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
Nick Terrell
56e3964d85 [man] Specify that strategies start at 1 2017-06-30 16:29:37 -07:00
Yann Collet
bb0aaf9579 minor man update on -B# option in benchmark mode 2017-06-12 05:19:15 -07:00
Nick Terrell
a99ece9e79 Update zstd.1.md to use btultra 2017-05-10 16:51:18 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
2017-05-10 16:49:58 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Yann Collet
230d7acc7d cli : add support for --threads=# command
updated documentation
add relevant test case
2017-04-21 11:38:13 -07:00
Sean Purcell
e6c504dbe6 Update -T0 comment in man page 2017-04-17 17:12:02 -07:00
Sean Purcell
5935c990a0 Add zstdmt and -T0 to man page 2017-04-17 16:05:20 -07:00
Yann Collet
a36330a27d minor man update, for overlapSize 2017-04-03 14:57:24 -07:00
Yann Collet
805c5a3efb updated documentation on multithreading modes 2017-04-01 00:36:31 -07:00
Yann Collet
96aa3019b2 changed advanced commands --maxdict= and --dictID=
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
2017-03-24 16:04:29 -07:00
Sean Purcell
680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Yann Collet
0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Sean Purcell
72a243e3bd Convert manpage to Markdown, converted with ronn 2017-03-16 15:37:32 -07:00
Nick Terrell
eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Przemyslaw Skibinski
40580ff669 added description of "--priority=rt" 2017-02-08 13:49:06 +01:00
Nick Terrell
b4016ff02f Add cover dictionary training to zstd.1
Tested with `make install && man zstd` and visual inspection.
2017-02-03 16:42:07 -08:00