These affected only quality 11, and now it does not make sense
to disable block splitting or context modeling because most of
the time is spent in zopfli anyway.
Now all speed vs size compromises are controlled by the quality param.
This is used for quality 11, for qualities <= 9 we already
have a simpler hash table.
The static data size is 252 kB, and this removes the
need to initialize a huge hash map at startup, which was
the reason why transforms had to be disabled by default.
In comparison, the static dictionary itself is 120 kB.
This supports every transform, except the kOmitFirstN.
This commit adopts the backward reference search algorithm
from the zopfli project (see https://github.com/google/zopfli)
to brotli.
This slower backward reference search is run only in quality 11
and it runs two iterations of entropy cost modeling and
shortest path search.
As a result, the original backward reference search function can
be simplified a bit, since we can remove some heuristics that were
replaced with the zopfli-style search.
* Fix an out-of-bounds access to depth_histo in the
bit cost calculation function.
* Change type of distance symbol to uint16_t in block
splitter, because if all postfix bits are used, there
can be 520 distance symbols.
* Save the distance cache between meta-blocks at the
correct place. This fixes a roundtrip failure that
can occur when there is an uncompressed metablock
between two compressed metablocks.
* Fix a bug when setting lgwin to 24 in the encoder parameters
It ended up making metablocks larger than 24 bits in size.
* Fix out-of-bounds memory accesses in parallel encoder.
CreateBackwardReferences can read up to 4 bytes past end of
input if the end of input is before mask.
* Add missing header for memcpy() in port.h
Adds functions to prepend such dictionary to the
encoder and decoder, and twiddles their internal
parameters to do as if that was a previous part of
the input. This dictionary is just a prefilled LZ77
window, it is not related to the built in transformable
brotli dictionary.
* Cluster at most 64 histograms at a time in the first
round of clustering.
* Use a faster histogram cost estimation function.
* Don't compute the log2(total) multiple times in the
block splitter.
* Read data by 4-byte runs.
This resolves unaligned read (Bus error) on arm-android.
* Get rid of malloc/free in BrotliBuildHuffmanTable.
* Tweak order of instructions when reading Huffman codes.
With this the users can distinguish between not knowing
what the input is (ddefault) and knowing that it is text,
and thus can be relied on to force some UTF-8 specific settings.
Add some more test cases that decompress to the empty
file or a one byte long file. These test cases have
examples for the updated stream header and meta-block
header formats.
