Commit Graph

16 Commits

Author SHA1 Message Date
Zoltan Szabadka
ea48ce5a6f Fix --Wconversion and --pedantic-erros for the encoder. 2015-10-28 17:44:47 +01:00
Zoltan Szabadka
2726b8a4f6 Encoder fixes.
* Remove default constructors.
* Initialize bit_cost in histogram.Clear().
* Check fseek result in FileSize.
* Replace malloc in BrotliFileIn constructor with "new".
* Catch bad_alloc in bro tool.
2015-10-06 11:23:44 +02:00
Zoltan Szabadka
b39eec8810 Remove C++11 vector::data() calls from encoder. 2015-10-05 11:43:49 +02:00
Zoltan Szabadka
4c37566f4b Move literal cost computation to where it's used.
Move utf8 heuristics functions to their own file.
2015-10-01 15:10:42 +02:00
Zoltan Szabadka
4a7024dcde Make the brotli encoder C++98 compatible. 2015-10-01 12:08:14 +02:00
Lode Vandevenne
17ed258993 msan bugfixes to the brotli encoder 2015-08-10 13:25:45 +02:00
Zoltan Szabadka
618287b373 Deprecate greedy_block_split and enable_context_modeling brotli params.
These affected only quality 11, and now it does not make sense
to disable block splitting or context modeling because most of
the time is spent in zopfli anyway.

Now all speed vs size compromises are controlled by the quality param.
2015-06-12 16:50:49 +02:00
Zoltan Szabadka
66098830a2 Use a static hash table to look up dictionary words and transforms.
This is used for quality 11, for qualities <= 9 we already
have a simpler hash table.

The static data size is 252 kB, and this removes the
need to initialize a huge hash map at startup, which was
the reason why transforms had to be disabled by default.
In comparison, the static dictionary itself is 120 kB.
This supports every transform, except the kOmitFirstN.
2015-06-12 16:45:17 +02:00
Zoltan Szabadka
b3d3723f62 Add "zopfli"-style backward reference search to brotli.
This commit adopts the backward reference search algorithm
from the zopfli project (see https://github.com/google/zopfli)
to brotli.

This slower backward reference search is run only in quality 11
and it runs two iterations of entropy cost modeling and
shortest path search.

As a result, the original backward reference search function can
be simplified a bit, since we can remove some heuristics that were
replaced with the zopfli-style search.
2015-06-12 16:25:41 +02:00
Zoltan Szabadka
65f3fc55f5 Bug fixes for the brotli encoder.
* Fix an out-of-bounds access to depth_histo in the
    bit cost calculation function.

  * Change type of distance symbol to uint16_t in block
    splitter, because if all postfix bits are used, there
    can be 520 distance symbols.

  * Save the distance cache between meta-blocks at the
    correct place. This fixes a roundtrip failure that
    can occur when there is an uncompressed metablock
    between two compressed metablocks.

  * Fix a bug when setting lgwin to 24 in the encoder parameters
    It ended up making metablocks larger than 24 bits in size.

  * Fix out-of-bounds memory accesses in parallel encoder.
    CreateBackwardReferences can read up to 4 bytes past end of
    input if the end of input is before mask.

  * Add missing header for memcpy() in port.h
2015-06-12 16:11:50 +02:00
Zoltan Szabadka
54f69c9ef7 Support window bits 10 - 15 in the decoder.
The previous window bit value 17 is used to
extend the range, since it has not been used
in any previous encoders.
2015-05-07 17:44:33 +02:00
Zoltan Szabadka
0f726df1f1 Don't do any block splitting for quality 1. 2015-04-28 10:12:47 +02:00
Zoltan Szabadka
98539223f5 Remove quality parameter from bitstream writing functions.
Fix a few crashes related to some quality and param combinations.
2015-04-23 16:20:29 +02:00
Zoltan Szabadka
2fd80cdc9a Encoder support for new empty meta-block format.
Changed the parallel implementation to sync meta-blocks
to byte boundary by emitting empty meta-blocks.
2015-04-23 15:43:37 +02:00
Zoltan Szabadka
89a6fb85fb Add params to disable static dictionary and context modeling.
Disable all slow features for quality <= 9 (literal cost modeling,
dictionary, context modeling, advanced block splitting).

Change vector<Command> arguments of internal functions
to Command* and size_t.
2015-04-23 13:15:42 +02:00
Zoltan Szabadka
1428d54178 Proof-of-concept encoder for parallel compression.
Add a version of the brotli encoder that compresses
each meta-block independently, only using the
original input data from previous meta-blocks
and nothing from the compressor state.
This is a proof-of-concept to show that the
current format is flexible enough to support
parallel multi-threaded compression.
2015-04-01 16:35:52 +02:00