b5033d0e1e
According to the format specification regarding complex prefix codes: > If there are at least two non-zero code lengths, any trailing zero > code lengths are omitted, i.e., the last code length in the > sequence must be non-zero. In this case, the sum of (32 >> code > length) over all the non-zero code lengths must equal to 32. > If the lengths have been read for the entire code length alphabet > and there was only one non-zero code length, then the prefix code > has one symbol whose code has zero length. The script does not handle a case where there is just 1 non-zero code length where the sum rule doesn't apply, which causes a StopIteration exception when it attempts to read past the list boundaries. An example of such file is tests/testdata/mapsdatazrh.compressed. I made sure this change doesn't break anything by processing all *.compressed files from the testdata folder with no thrown exceptions. |
||
|---|---|---|
| .. | ||
| esaxx@ca7cb33201 | ||
| img | ||
| brotlidump.py | ||
| BUILD | ||
| deorummolae.cc | ||
| deorummolae.h | ||
| dictionary_generator.cc | ||
| draw_diff.cc | ||
| draw_histogram.cc | ||
| find_opt_references.cc | ||
| Makefile | ||
| read_dist.h | ||
| README.md | ||
| sieve.cc | ||
| sieve.h | ||
Introduction
In this directory we publish simple tools to analyze backward reference distance distributions in LZ77 compression. We developed these tools to be able to make more efficient encoding of distances in large-window brotli. In large-window compression the average cost of a backward reference distance is higher, and this may allow for more advanced encoding strategies, such as delta coding or an increase in context size, to bring significant compression density improvements. Our tools visualize the backward references as histogram images, i.e., one pixel in the image shows how many distances of a certain range exist at a certain locality in the data. The human visual system is excellent at pattern detection, so we tried to roughly identify patterns visually before going into more quantitative analysis. These tools can turn out to be useful in development of other LZ77-based compressors and we hope you try them out.
Tools
find_opt_references
This tool generates optimal (match-length-wise) backward references for every position in the input files and stores them in *.dist file described below.
Example usage:
find_opt_references input.txt output.dist
draw_histogram
This tool generates a visualization of the distribution of backward references stored in *.dist file. The original file size has to be specified as a second parameter. The output is a grayscale PGM (binary) image.
Example usage:
draw_histogram input.dist 65536 output.pgm
Here's an example of resulting image:
draw_diff
This tool generates a diff PPM (binary) image between two input 8-bit PGM (binary) images. Input images must be of same size. Useful for comparing different backward references distributions for same input file. Normally used for comparison of output images from draw_histogram tool.
Example usage:
draw_diff image1.pgm image2.pgm diff.ppm
For example the diff of this image
and this image
looks like this:
Backward distance file format
The format of *.dist files is as follows:
[[ 0| match length][ 1|position|distance]...]
[1 byte| 4 bytes][1 byte| 4 bytes| 4 bytes]
More verbose explanation: for each backward reference there is a position-distance pair, also a copy length may be specified. Copy length is prefixed with flag byte 0, position-distance pair is prefixed with flag byte 1. Each number is a 32-bit integer. Copy length always comes before position-distance pair. Standalone copy length is allowed, in this case it is ignored.
Here's an example of how to read from *.dist file:
#include "read_dist.h"
FILE* f;
int copy, pos, dist;
while (ReadBackwardReference(fin, ©, &pos, &dist)) {
...
}