AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

[cover] Make optimization faster

Browse Source
This commit is contained in:
Nick Terrell 2017-05-01 21:26:33 -07:00
parent f2d9ef1dc0
commit 020b960e13
3 changed files with 8 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
lib/dictBuilder/cover.c
View File

@ -939,8 +939,8 @@ ZDICTLIB_API size_t COVER_optimizeTrainFromBuffer(void *dictBuffer,
/* constants */
const unsigned nbThreads = parameters->nbThreads;
const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
const unsigned kMaxD = parameters->d == 0 ? 16 : parameters->d;
const unsigned kMinK = parameters->k == 0 ? kMaxD : parameters->k;
const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
const unsigned kMinK = parameters->k == 0 ? 40 + kMaxD : parameters->k;
const unsigned kMaxK = parameters->k == 0 ? 2048 : parameters->k;
const unsigned kSteps = parameters->steps == 0 ? 32 : parameters->steps;
const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1);

6
programs/zstd.1
View File

@ -195,13 +195,13 @@ Example: \fB\-\-train \-\-cover=k=64,d=8 FILEs\fR\.
.
.TP
\fB\-\-optimize\-cover[=steps=#,k=#,d=#]\fR
If \fIsteps\fR is not specified, the default value of 32 is used\. If \fIk\fR is not specified, the \fIk\fR values in [16, 2048] are checked for each value of \fId\fR\. If \fId\fR is not specified, the values checked are [6, 8, \.\.\., 16]\.
If \fIsteps\fR is not specified, the default value of 32 is used\. If \fIk\fR is not specified, the \fIk\fR values in [48, 2048] are checked for each value of \fId\fR\. If \fId\fR is not specified, the values checked are [6, 8]\.
.
.IP
Runs the cover dictionary builder for each parameter set and saves the optimal parameters and dictionary\. Prints optimal parameters and writes optimal dictionary into output file\. Supports multithreading if \fBzstd\fR is compiled with threading support\.
.
.IP
The parameter \fIk\fR is more sensitive than \fId\fR, and is faster to optimize over\. Suggested use is to run with a \fIsteps\fR <= 32 with neither \fIk\fR nor \fId\fR set\. Once it completes, use the value of \fId\fR it selects with a higher \fIsteps\fR (in the range [256, 1024])\.
The parameter \fIk\fR is more sensitive than \fId\fR, and is faster to optimize over\.
.
.IP
Examples :
@ -210,7 +210,7 @@ Examples :
\fBzstd \-\-train \-\-optimize\-cover FILEs\fR
.
.IP
\fBzstd \-\-train \-\-optimize\-cover=d=d,steps=512 FILEs\fR
\fBzstd \-\-train \-\-optimize\-cover=d=8,steps=512 FILEs\fR
.
.SH "BENCHMARK"
.

9
programs/zstd.1.md
View File

@ -194,9 +194,9 @@ Typical gains range from 10% (at 64KB) to x5 better (at <1KB).
* `--optimize-cover[=steps=#,k=#,d=#]`:
If _steps_ is not specified, the default value of 32 is used.
If _k_ is not specified, the _k_ values in [16, 2048] are checked for each
If _k_ is not specified, the _k_ values in [48, 2048] are checked for each
value of _d_.
If _d_ is not specified, the values checked are [6, 8, ..., 16].
If _d_ is not specified, the values checked are [6, 8].
Runs the cover dictionary builder for each parameter set
and saves the optimal parameters and dictionary.
@ -204,15 +204,12 @@ Typical gains range from 10% (at 64KB) to x5 better (at <1KB).
Supports multithreading if `zstd` is compiled with threading support.
The parameter _k_ is more sensitive than _d_, and is faster to optimize over.
Suggested use is to run with a _steps_ <= 32 with neither _k_ nor _d_ set.
Once it completes, use the value of _d_ it selects with a higher _steps_
(in the range [256, 1024]).
Examples :
`zstd --train --optimize-cover FILEs`
`zstd --train --optimize-cover=d=d,steps=512 FILEs`
`zstd --train --optimize-cover=d=8,steps=512 FILEs`
BENCHMARK