Commit Graph

46 Commits

Author SHA1 Message Date
Eugene Kliuchnikov
5db62dcc9d Fixes: (#468)
* fix slow-down after a long copy (q10-11)
 * more thorough hashing for long ranges (q10-11)
 * minor documentation fixes
 * bazel.io -> bazel.build
2016-11-09 14:04:09 +01:00
Eugene Kliuchnikov
e9b278ac6e Update docs and add more java tests (#463)
* doxygenize and update API documentation
 * fix spelling
 * add "fuzz" corpus for java decoder to improve coverage
 * use upper-case-snake names for dictionary constant definitions
 * use `LDFLAGS` in conventional `Makefile`
2016-10-31 14:33:59 +01:00
Eugene Kliuchnikov
69982c25f1 Build shared libraries by default
* Declare `BUILD_SHARED_LIBS` option for CMake
* Define `${LIB}_SHARED_COMPILATION` when compiling shared library
* Define and use BROTLI_xxx_API
* Fix remaining unprefixed defines in port.h
2016-10-18 16:45:32 +02:00
Eugene Kliuchnikov
9521d968f3 Eliminate more magic constants.
Author: Ivan Nikulin
2016-10-17 17:33:12 +02:00
Eugene Kliuchnikov
0a63f99db9 Update encoder
* move `common/port.h` to `includes/port.h`
 * replace magic more magic numbers with constants
 * artificially limit window size to 2^18 for quality 0 and 1
 * use fixed shifts for quality 0 and 1 hashes
 * removed `BrotliEncoderWriteMetadata`
 * added `BROTLI_OPERATION_EMIT_METADATA` instead
 * deprecated low-level API
 * fixed MSVC warnings
2016-09-21 17:20:36 +02:00
Eugene Kliuchnikov
8148001158 Move "public" to "include/brotli" 2016-08-23 14:40:33 +02:00
Eugene Kliuchnikov
801f5f37ee * rename macros with preceding underscore
* add Brotli*TakeOutput methods
* * flushing now doesn't require additional call
* add Brotli*Version methods
* moved public headers to 'public' directory
* removed C++ API
* do not assume STDC_VERSION is defined
2016-08-22 13:28:22 +02:00
Eugene Kliuchnikov
2048189048 Update encoder:
* booleanification
 * integer BR scores, may improve performance if FPU is slow
 * condense speed-quality constants in quality.h
 * code massage to calm down CoverityScan
 * hashers refactoring
 * new hasher - improved speed, compression and reduced memory usage for q:5-9 w:10-16
 * reduced static recources -> binary size
2016-07-26 14:41:59 +02:00
Eugene Kliuchnikov
b972c67780 Convert encoder to plain C. 2016-06-13 11:01:04 +02:00
Eugene Kliuchnikov
352b0b2836 Transform most of C++ comments to C-style. 2016-06-03 11:19:23 +02:00
Eugene Kliuchnikov
028291865d Extract common parts: constants, dictionary, etc. 2016-06-03 10:51:04 +02:00
Zoltan Szabadka
b820c39bd9 Reduce memory usage of brotli encoder at quality 10 and 11. 2016-03-15 10:50:16 +01:00
Eugene Kliuchnikov
25e3796f83 Fix most of ()->(void) and some missing includes. 2016-02-29 14:41:24 +01:00
Zoltan Szabadka
dbb53e6434 Use a hash-to-binary-tree data structure for quality 11 as suggested by Issue #180. 2016-01-27 09:50:39 +01:00
Zoltan Szabadka
417107b3dd Add two more fast modes to the brotli compressor.
The new modes process the input data in independent blocks,
using backward references only from within an input block.

The new modes can be used by specifying quality 0 or quality 1,
the old quality 1 and quality 2 modes are renamed quality 2 and
quality 3, respectively, and the old quality 3 mode is removed.
2016-01-11 11:21:42 +01:00
Zoltan Szabadka
4dd9114c97 Partial Hasher initialization for small input data.
This increases compression speed of very small files (< 1KB) for quality <= 3.
2016-01-07 17:10:34 +01:00
Zoltan Szabadka
8844b7f0d7 Fix more conversion warnings. 2016-01-07 16:27:49 +01:00
Eugene Klyuchnikov
24ffa78414 Fix headers 2015-12-11 11:11:51 +01:00
Eugene Klyuchnikov
771eb10798 Update license statement in source files. 2015-11-27 11:27:11 +01:00
Eugene Klyuchnikov
bb26d1919f Fix sign-comparison warnings
+ add more debug runtime checks
+ minor cleanup
2015-11-23 11:05:12 +01:00
Eugene Klyuchnikov
152e33c3a0 Add more explicit type conversions.
Remove dead code.
Fix includes.
2015-11-17 13:45:41 +01:00
Zoltan Szabadka
ea48ce5a6f Fix --Wconversion and --pedantic-erros for the encoder. 2015-10-28 17:44:47 +01:00
Zoltan Szabadka
a89b57b90c Use uint32_t positions in the hasher and compute distances modulo 2^32. 2015-10-26 17:08:57 +01:00
Zoltan Szabadka
754deaed2f Reduce command buffer memory usage. 2015-10-01 17:08:59 +02:00
Zoltan Szabadka
4a7024dcde Make the brotli encoder C++98 compatible. 2015-10-01 12:08:14 +02:00
Zoltan Szabadka
dfdf2dd4c4 Encoder bug fixes.
* Fix forward declaration mismatch.
* Fix division by zero in 64X test.
* Avoid shadowing of variables in encoder.
2015-10-01 11:40:05 +02:00
Lode Vandevenne
6511d6b016 update brotli encoder with latest improvements 2015-08-28 16:09:23 +02:00
Lode Vandevenne
17ed258993 msan bugfixes to the brotli encoder 2015-08-10 13:25:45 +02:00
Zoltan Szabadka
66098830a2 Use a static hash table to look up dictionary words and transforms.
This is used for quality 11, for qualities <= 9 we already
have a simpler hash table.

The static data size is 252 kB, and this removes the
need to initialize a huge hash map at startup, which was
the reason why transforms had to be disabled by default.
In comparison, the static dictionary itself is 120 kB.
This supports every transform, except the kOmitFirstN.
2015-06-12 16:45:17 +02:00
Zoltan Szabadka
b3d3723f62 Add "zopfli"-style backward reference search to brotli.
This commit adopts the backward reference search algorithm
from the zopfli project (see https://github.com/google/zopfli)
to brotli.

This slower backward reference search is run only in quality 11
and it runs two iterations of entropy cost modeling and
shortest path search.

As a result, the original backward reference search function can
be simplified a bit, since we can remove some heuristics that were
replaced with the zopfli-style search.
2015-06-12 16:25:41 +02:00
Zoltan Szabadka
b43df8f699 Brotli custom LZ77 dictionary support.
Adds functions to prepend such dictionary to the
encoder and decoder, and twiddles their internal
parameters to do as if that was a previous part of
the input. This dictionary is just a prefilled LZ77
window, it is not related to the built in transformable
brotli dictionary.
2015-06-12 15:43:54 +02:00
Zoltan Szabadka
6622355a9a Use the same hasher for text and font mode.
We use 4-byte hashing in both and look for length 3
matches separately.
2015-05-11 14:11:07 +02:00
Zoltan Szabadka
7cde616c9e Faster encoding for low quality settings.
With this commit, the encoder will skip some
compression optimization steps for quality <= 4,
which results in faster compression but higher
compressed sizes.
2015-05-07 17:30:10 +02:00
Zoltan Szabadka
98539223f5 Remove quality parameter from bitstream writing functions.
Fix a few crashes related to some quality and param combinations.
2015-04-23 16:20:29 +02:00
Zoltan Szabadka
fdfb19806b Add a static hash table based dictionary lookup to fast brotli. 2015-04-23 15:52:32 +02:00
Zoltan Szabadka
b4f39bf540 New version of the backward reference search code.
The new	interface of the backward reference search
function makes it possible to use it in	a streaming
manner.

Using the advanced cost model and static dictionary
can be turned on/off by	template parameters.

The distance short codes are now computed as part of
the backward reference search.

Added a	faster version of the Hasher.
2014-10-28 13:25:22 +01:00
Zoltan Szabadka
03b20347e1 Minor tuning of encoder heuristics. 2014-10-28 12:50:33 +01:00
Zoltan Szabadka
347781947a Update the dictionary and the transforms. 2014-03-25 16:48:25 +01:00
Zoltan Szabadka
e7650080a8 Updates to Brotli compression format, decoder and encoder
This commit contains a batch of changes that were made to the Brotli
compression algorithm in the last month. Most important changes:

   * Format change: don't push distances representing static dictionary words to the distance cache.
   * Fix decoder invalid memory access bug caused by building a non-complete Huffman tree.
   * Add a mode parameter to the encoder interface.
   * Use different hashers for text and font mode.
   * Add a heuristics to the hasher for skipping non-compressible data.
   * Exhaustive search of static dictionary during backward reference search.
2014-03-20 14:32:35 +01:00
Zoltan Szabadka
2f268ad158 Add the initial version of the static dictionary and transforms to Brotli. 2014-02-17 14:25:36 +01:00
Zoltan Szabadka
0454ab4ec0 Updates to Brotli compression format, decoder and encoder
This commit contains a batch of changes that were made to the Brotli
compression algorithm in the last month. Most important changes:

   * Fixes to the spec.
   * Change of code length code order.
   * Use a 2-level Huffman lookup table in the decoder.
   * Faster uncompressed meta-block decoding.
   * Optimized encoding of the Huffman code.
   * Detection of UTF-8 input encoding.
   * UTF-8 based literal cost modeling for improved
     backward reference selection.
2014-02-14 15:04:23 +01:00
Zoltan Szabadka
d762bc6845 Bug fixes for the brotli encoder and decoder. 2014-01-06 16:01:57 +01:00
Roderick Sheeter
c23cb1e83d Support for OSX build; tested using OSX 10.9/clang-500.2.79 2013-12-12 10:43:05 -08:00
Roderick Sheeter
1cdcbd851f Added Brotli compress/decompress utilities and makefiles 2013-11-19 14:32:56 -08:00
Zoltan Szabadka
c6b9c7c5c8 Updates to Brotli compression format, decoder and encoder
This commit contains a batch of changes that were made to the Brotli
compression algorithm in the last three weeks. Most important changes:

  * Added UTF8 context model for good text compression.
  * Simplified context modeling by having only 4 context modes.
  * Per-block context mode selection.
  * Faster backward copying and bit reading functions.
  * More efficient histogram coding.
  * Streaming support for the decoder and encoder.
2013-11-15 19:02:17 +01:00
Zoltan Szabadka
c66e4e3e4f Add brotli compressor
This commit is for the encoder for brotli compression format.
Brotli is a generic byte-level compression algorithm.
2013-10-23 13:06:13 +02:00