Commit Graph

49 Commits

Author SHA1 Message Date
Evgenii Kliuchnikov
04f294b18a Fix emitting 1-byte long metadata block
PiperOrigin-RevId: 505484299
2023-01-30 09:10:28 +00:00
Brotli
1e61e972fb speed up encoding by ~5 %
PiperOrigin-RevId: 505061835
2023-01-30 09:10:14 +00:00
Aron Parker
71fe6cac06
Fix BrotliEncoderEstimatePeakMemoryUsage (#1002)
Fixes https://github.com/google/brotli/issues/1001
2023-01-07 22:01:47 +01:00
Eugene Kliuchnikov
0ea4603880
Fix MSVC warning (#998)
Fix #875
2023-01-04 12:10:29 +01:00
Evgenii Kliuchnikov
a8f5813b84 Update
Documentation:
  - add note that brotli is a "stream" format, not an archive-like
  - regenerate .1 with Pandoc
Build:
  - drop legacy "BROTLI_BUILD_PORTABLE" option
  - drop "BROTLI_SANITIZED" definition
Code:
  - c: comb includes
  - c/enc: extract encoder state into separate header
  - c/enc: drop designated q10 codepath
  - c/enc: dealing better with flushing of empty stream
  - fix MSVC compilation
API:
  - py: use library version instead of one in version.h
  - c: add plugable API to report consumed input / produced output
  - c/java: support "lean" prepared dictionaries (without copy of source)
2022-11-17 13:03:09 +00:00
Jyrki Alakuijala
4ec67035c0
Merge pull request #929 from jbms/fix-vla-parameter
Fix -Werror=vla-parameter errors with GCC 11.2.0
2021-12-07 01:47:07 +01:00
Eugene Kliuchnikov
8376f72ed6
Prepare for copybara (#939)
Co-authored-by: Eugene Kliuchnikov <eustas@chromium.org>
2021-11-10 10:34:39 +01:00
Jeremy Maitin-Shepard
27dd726540 Fix -Werror=vla-parameter errors with GCC 11.2.0 2021-09-14 12:27:45 -07:00
Eugene Kliuchnikov
62662f87cd
Strip "./" in includes (#925)
Co-authored-by: Eugene Kliuchnikov <eustas@chromium.org>
2021-09-08 09:18:45 +02:00
Eugene Kliuchnikov
68f1b90ad0
Update (#918)
Prepare to use copybara worklow.
2021-08-18 19:15:07 +02:00
Eugene Kliuchnikov
19d86fb9a6
Merge-in SharedDictionary feature (#916)
Co-authored-by: Eugene Kliuchnikov <eustas@chromium.org>
2021-08-04 14:42:02 +02:00
Eugene Kliuchnikov
630b5084ee
Update (#914)
* slimmer stack frames in encoder
 * fix MSAN problem in hasher_composite
   (not dangerous, only in large_window mode)
 * fix JNI decoder wrapper - power-of-two payloads fail to decode sometimes
 * reformat polyfil.js and decode_test.js
2021-07-29 22:29:43 +02:00
Adrián Herrera Arcila
0a3944c8c9
Fix VLA parameter warning (#893)
Make VLA buffer types consistent in declarations and definitions.
Resolves build crash when using -Werror due to "vla-parameter" warning.

Signed-off-by: Adrian Herrera <adr.her.arc.95@gmail.com>
2021-06-23 09:53:59 +02:00
Ikko Ashimine
bdcfb123c8
Fix typo in hash_composite_inc.h (#903)
defered -> deferred
2021-06-23 09:42:28 +02:00
Eugene Kliuchnikov
5692e422da
Update (#852)
* Update

 * comments and clarifications in block_splitter
 * power-of-2 aligned allocations for Hasher
 * refresh decode.js from Java sources
 * disable JS build
2021-01-18 10:56:39 +01:00
Dmitry Rozhkov
7e8e207ce2
Fix clang-10 compilation issue (#839)
clang-10 throws the following error:
In file included from external/org_brotli/c/enc/bit_cost.c:9:
external/org_brotli/c/enc/./bit_cost.h:48:16: error: implicit conversion
from 'size_t' (aka 'unsigned long') to 'double' may lose precision
[-Werror,-Wimplicit-int-float-conversion]
  if (retval < sum) {
             ~ ^~~
1 error generated.

Make the conversion explicit.
2020-09-07 09:40:03 +02:00
Eugene Kliuchnikov
223d80cfbe
Update (#826)
* IMPORTANT: decoder: fix potential overflow when input chunk is >2GiB
 * simplify max Huffman table size calculation
 * eliminate symbol duplicates (static arrays in .h files)
 * minor combing in research/ code
2020-08-26 12:32:27 +02:00
Eugene Kliuchnikov
7f740f1308
Update (#807)
- fix formatting
 - fix type conversion
 - fix no-op arithmetic with null-pointer
 - improve performance of hash_longest_match64
 - go: detect read after close
 - java decoder: support compound dictionary
 - remove executable flag on non-scripts
2020-05-15 11:06:21 +02:00
Clinton Ingram
924b2b2b9d
Move TZCNT and BSR intrinsics to platform.h, add MSVC versions (#636) 2020-03-19 11:57:56 +01:00
Cristi Vîjdea
f503cb709c
Add HAVE_LOG2 build macro (#783)
* Add HAVE_LOG2 build macro

Fixes #781

* Rename macro to BROTLI_HAVE_LOG2 and move comment for visibility
2020-03-19 10:46:52 +01:00
Eugene Kliuchnikov
78e7bbc3c3
Update (#753)
* fix executable mode of decode.js
 * explain clang-analyser about non-nullability
 * fix "dead assignment"
 * rename proguard.cfg -> proguard.pgcfg
2019-05-03 11:51:11 +02:00
Eugene Kliuchnikov
4b2b2d4f83
Update (#749)
Update:

 * Bazel: fix MSVC configuration
 * C: common: extended documentation and helpers around distance codes
 * C: common: enable BROTLI_DCHECK in "debug" builds
 * C: common: fix implicit trailing zero in `kPrefixSuffix`
 * C: dec: fix possible bit reader discharge for "large-window" mode
 * C: dec: simplify distance decoding via lookup table
 * C: dec: reuse decoder state members memory via union with lookup table
 * C: dec: add decoder state diagram
 * C: enc: clarify access to static dictionary
 * C: enc: improve static dictionary hash
 * C: enc: add "stream offset" parameter for parallel encoding
 * C: enc: reorganize hasher; now Q2-Q3 require exactly 256KiB
           to avoid global TCMalloc lock
 * C: enc: fix rare access to uninitialized data in ring-buffer
 * C: enc: reorganize logging / checks in `write_bits.h`
 * Java: dec: add "large-window" support
 * Java: dec: improve speed
 * Java: dec: debug and 32-bit mode are now activated via system properties
 * Java: dec: demystify some state variables (use better names)
 * Dictionary generator: add single input mode
 * Java: dec: modernize tests
 * Bazel: js: pick working commit for closure rules
2019-04-12 13:57:42 +02:00
Eugene Kliuchnikov
d0ffe60b87
Verbose CLI + start pulling "Shared-Brotli" (#722)
* Verbose CLI + start pulling "Shared-Brotli"

 * vesbose CLI output; fix #666
 * pull `SHIFT` transforms; currently this is semantically dead code;
   later it will be used by "Shared-Brotli"
2018-10-24 16:06:09 +02:00
Eugene Kliuchnikov
f7cbc97c96
Fix typo / minor formatting (#716)
* Fix typo / minor formatting / pull computable constant to the place of use.
2018-10-16 16:46:54 +02:00
Eugene Kliuchnikov
2216a0dd63
Update (#706)
Update
 * add ASAN/MSAN unaligned read specializations
 * add "brotli" prefix to u_uint64 type
 * increment version to 1.0.06
 * fix CoverityScan "unused assignment" warning
 * fix JDK 8<->9 incompatibility
 * add encoder optimization for empty input
 * regenerate JS decoder
 * unbreak Travis builds
2018-09-13 08:09:32 -04:00
Eugene Kliuchnikov
eb12ec04eb
Update (#688)
* add rolling-composite-hasher for large-window mode
* make API methods explicitly public
2018-06-20 15:14:10 +02:00
Eugene Kliuchnikov
8544ae858d
Update (#680)
* fix MSVC warnings
 * cleanups
2018-06-09 11:17:13 +02:00
Eugene Kliuchnikov
48a25b3fa4
Fix #671 (#672) 2018-05-18 22:07:52 +02:00
Eugene Kliuchnikov
f5ed35d065
Update (#664)
* Update
 * fix ifdef style
 * get back to fine-compiler-version-based-macros (use Hedley)
 * fix q=0 histogram collection for very long copy/insert commands
2018-05-03 11:16:21 +02:00
Eugene Kliuchnikov
6000396155
Remove unprefixed macros from public headers (#662) 2018-04-20 14:10:55 +02:00
Eugene Kliuchnikov
68db5c0272
Update (#660)
* Update
 * improve q=1 compression on small files
 * fix "left shift before promotion"
 * fix osx Travis builds
2018-04-13 11:44:34 +02:00
Eugene Kliuchnikov
c6333e1e79
Fix MSVC compilation (#657)
* tell bazel not to pass strict options to a fancy compiler
 * fix signed-unsigned comparison warning found by MSVC
2018-03-29 10:37:07 +02:00
Eugene Kliuchnikov
0f3c84e745
Update (#656)
* proper fix for the "fall through" warning"
 * automatic NDIRECT/NPOSTFIX tuning (better compression)
 * fix unaligned access for `aarch64`-cross-`armhf` build
 * fix `aarch64` detection (10% decoder speedup)
 * expose `large_window` CLI option
 * make default window size 16MiB
 * ramp up version to 1.0.4
2018-03-27 22:29:22 +02:00
Eugene Kliuchnikov
631fe194a1
Update (#651)
* fix `bazel` build (ignore switch case fall-through)
* add `NPOSTFIX` / `NDIRECT` encoder parameters
* fix source file lists (add `params.h`)
* fix bug in `durchschlag`
* print clarifying messages wheb CLI argument parsing fails
2018-03-20 17:37:41 +06:00
Eugene Kliuchnikov
533843e354
Update (#643)
Update
 * make the zopflification aware of `NDIRECT`, `NPOSTFIX`
   (better compression in `font` mode)
 * add small and simple decoder tool
 * fix typo
 * Java: wrapper: make decoder channel more async-friendly

Ramp up version to 1.0.3 / 1.0.3
2018-03-02 15:49:58 +01:00
Eugene Kliuchnikov
35e69fc7cf
New feature: "Large Window Brotli" (#640)
* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.
2018-02-26 09:04:36 -05:00
Eugene Kliuchnikov
da254cffdb
Update (#630)
* merge {dec|enc}/port.h into common/platform.h
 * fix one-shot q=10 1-byte input compression
 * fix some unprefixed definitions
 * make hashers host-endianness-independent
 * extract enc/params.h from enc/quality.h
 * fix API documentation / typos
 * improve `BrotliEncoderMaxCompressedSize`
2017-12-12 14:33:12 +01:00
Eugene Kliuchnikov
0ad94eed00
Update (#620)
* add autotools build
* separate semantic and ABI version
* extract sources.lst (used by CMake and Automake)
* share pkgconfig templates (used by CMake and Automake)
* decoder: always set `total_out`
* encoder: fix `BROTLI_ENSURE_CAPACITY` macro (no-op after preprocessor)
* decoder/encoder: refine `free_func` contract
2017-11-28 15:37:28 +01:00
Eugene Kliuchnikov
3e58ea5f90 Update (#617)
* remove `const` on `BrotliDictionary` members
 * extend `ZofliNode` distance range to 128MiB
 * add missing `port.h` include to `quality.h`
 * fix typo in encoder API-doc
 * regenerate `decode.min.js`
2017-10-13 14:50:51 +02:00
Tomáš Popela
a0c7dafe28 Fix permissions of various files in project (#613)
Move from 755 to 644.
2017-10-10 11:24:13 +02:00
Eugene Kliuchnikov
37fb83ec0d Update: (#600)
* encoder: relax backward references candidates asserts
 * encoder: make RNG more platform-independent
 * encoder: remove "unused" param (context mode)
 * CLI: improve first-encounter experience
 * Java: update SynthTest
 * Java: refine proguard config
 * Java/JNI: fix one-shot compression workflow
2017-09-19 15:57:15 +02:00
Eugene Kliuchnikov
d7bce1e092 Update (#593)
* Update:
 * fix CLI error messages
 * fix CLI console IO on Windows
2017-09-07 20:27:49 +02:00
Eugene Kliuchnikov
a629289e32 Update (#590)
* add transpiled JS decoder
 * make PY wrapper accept memview
 * fix dictionary generator
 * speedup compression of RLEish data
2017-08-28 11:31:29 +02:00
Eugene Kliuchnikov
6535435413 Update (#589)
* cleanup
 * fix `unbrotli` CLI
 * Java retouch for faster JS decoder
2017-08-24 13:29:48 +02:00
Eugene Kliuchnikov
d63e8f75f5 Update API, and more (#581)
Update API, and more:
 * remove "custom dictionary" support
 * c/encoder: fix #580: big-endian build
 * Java: reduce jar size
 * Java: speedup decoding
 * Java: add 32-bit CPU support
 * Java: make source code JS transpiler-ready
2017-08-04 10:02:56 +02:00
Eugene Kliuchnikov
52441069ef Update (#574)
* Update
 * decoder: better behavior after failure
 * encoder: replace "len_x_code" with delta
 * research: add experimental dictionary generator
 * python: test combing
2017-07-21 10:07:24 +02:00
Elouan Martinet
00cacfdff6 Fix compilation issue with BROTLI_ALLOC macro using GCC 7 (-Wint-in-bool-context) (#562) 2017-06-17 15:22:07 +02:00
Eugene Kliuchnikov
05d5f3d77a Update (#560)
Update:
 * add decoder API to avoid ringbuffer reallocation
 * fix MSVC warnings
 * remove dead code
2017-06-13 12:52:56 +02:00
Eugene Kliuchnikov
6ece1d8791 Move files & update paths (#541)
* Move files & update paths

* Rename build to scripts.

* Fix paths

* Fix script.
2017-04-23 14:07:08 +02:00