Commit Graph

59 Commits

Author SHA1 Message Date
Eugene Kliuchnikov
5519352661
Add workaround for lying feof. (#814)
Should fix #812
2020-07-02 17:57:40 +02:00
fisherwky
d2ea198232
Update platform.h (#813)
fix compile error (platform.h:362: error: cast discards qualifiers from pointer target type)
2020-06-30 11:23:07 +02:00
Eugene Kliuchnikov
7f740f1308
Update (#807)
- fix formatting
 - fix type conversion
 - fix no-op arithmetic with null-pointer
 - improve performance of hash_longest_match64
 - go: detect read after close
 - java decoder: support compound dictionary
 - remove executable flag on non-scripts
2020-05-15 11:06:21 +02:00
Clinton Ingram
924b2b2b9d
Move TZCNT and BSR intrinsics to platform.h, add MSVC versions (#636) 2020-03-19 11:57:56 +01:00
Cristi Vîjdea
f503cb709c
Add HAVE_LOG2 build macro (#783)
* Add HAVE_LOG2 build macro

Fixes #781

* Rename macro to BROTLI_HAVE_LOG2 and move comment for visibility
2020-03-19 10:46:52 +01:00
shenglei10
666c3280cc
Make types of variable match (#796) 2020-02-14 10:40:02 +01:00
agrieve
4b5771bee7 Add missing "const" to a couple of kConstants (#780)
These showed up in a Chromium audit:
https://bugs.chromium.org/p/chromium/issues/detail?id=747064#c8

Although already effectively const, adding "const" causes the symbols to
be moved into the read-only section of the binary.
2019-12-20 00:15:58 +01:00
Eugene Kliuchnikov
3d1767186d
Fix include for EMCC build (#765) 2019-07-30 10:01:21 +02:00
Eugene Kliuchnikov
f1124c8524
More careful sanitizer detection (#764) 2019-07-22 14:29:51 +02:00
Eugene Kliuchnikov
c8b37e8fd1
Update (#762)
* put LICENSE file into .jar
 * fix typo
 * add clarification comment in PY wrapper
2019-07-17 14:39:56 +02:00
Eugene Kliuchnikov
78e7bbc3c3
Update (#753)
* fix executable mode of decode.js
 * explain clang-analyser about non-nullability
 * fix "dead assignment"
 * rename proguard.cfg -> proguard.pgcfg
2019-05-03 11:51:11 +02:00
Eugene Kliuchnikov
4b2b2d4f83
Update (#749)
Update:

 * Bazel: fix MSVC configuration
 * C: common: extended documentation and helpers around distance codes
 * C: common: enable BROTLI_DCHECK in "debug" builds
 * C: common: fix implicit trailing zero in `kPrefixSuffix`
 * C: dec: fix possible bit reader discharge for "large-window" mode
 * C: dec: simplify distance decoding via lookup table
 * C: dec: reuse decoder state members memory via union with lookup table
 * C: dec: add decoder state diagram
 * C: enc: clarify access to static dictionary
 * C: enc: improve static dictionary hash
 * C: enc: add "stream offset" parameter for parallel encoding
 * C: enc: reorganize hasher; now Q2-Q3 require exactly 256KiB
           to avoid global TCMalloc lock
 * C: enc: fix rare access to uninitialized data in ring-buffer
 * C: enc: reorganize logging / checks in `write_bits.h`
 * Java: dec: add "large-window" support
 * Java: dec: improve speed
 * Java: dec: debug and 32-bit mode are now activated via system properties
 * Java: dec: demystify some state variables (use better names)
 * Dictionary generator: add single input mode
 * Java: dec: modernize tests
 * Bazel: js: pick working commit for closure rules
2019-04-12 13:57:42 +02:00
Eugene Kliuchnikov
d0ffe60b87
Verbose CLI + start pulling "Shared-Brotli" (#722)
* Verbose CLI + start pulling "Shared-Brotli"

 * vesbose CLI output; fix #666
 * pull `SHIFT` transforms; currently this is semantically dead code;
   later it will be used by "Shared-Brotli"
2018-10-24 16:06:09 +02:00
Eugene Kliuchnikov
d6d98957ca
Ramp up version to 1.0.7 2018-10-23 12:24:40 +02:00
Eugene Kliuchnikov
a1e44975a7
Fix #698 2018-10-19 17:01:54 +02:00
Eugene Kliuchnikov
ce8951c3e9
Fix <arm_neon.h> inclusion guard. (#717) 2018-10-16 17:19:37 +02:00
Eugene Kliuchnikov
f7cbc97c96
Fix typo / minor formatting (#716)
* Fix typo / minor formatting / pull computable constant to the place of use.
2018-10-16 16:46:54 +02:00
Stephen Kyle
cc7a74f15f decode: fix NEON inclusion (#714)
The macro that checks for NEON support should be __ARM_NEON, not
__ARM_NEON__. [1]

AArch64 compilers define __ARM_NEON but not __ARM_NEON__.
AArch32 compilers currently seem to define both, but could be within their
rights to drop __ARM_NEON__ in future versions.

This change moves the check into the common/platform.h file, checks for
both forms, and sets BROTLI_TARGET_NEON if NEON support is available.

[1] Section 6.5.4 of the ARM C Language Extensions.
    (At the time of writing, the latest version was Release 2.1.)
2018-10-08 15:40:11 +02:00
Alexey Ivanov
c94c6f805c tools/brotli: improve window size autodetect (#710)
Window size is defined as:
    `(1 << BROTLI_PARAM_LGWIN) - 16`
in `c/include/brotli/encode.h`

Therefore we should probably take these 16 bytes into account.

Done basic manual testing:
$ python3 -c 'print ("A"*2046)' > t
$ bazel run -- //:brotli -w 0 -f -o $(realpath t).br $(realpath ./t)
$ python3 research/brotlidump.py t.br |& fgrep WSIZE
0000  c1                1000001 WSIZE   windowsize=(1<<12)-16=4080

New version properly detects window size of `4080`, while previous one used `2032`:
$ python3 research/brotlidump.py t.br |& fgrep WSIZE
0000  b1                0110001 WSIZE   windowsize=(1<<11)-16=2032
2018-10-02 16:28:37 +02:00
Stephen Kyle
9402ac5c08 decode: faster huffman code loading on 32-bit Arm (#703)
* platform: add macro for using the 'aligned' attribute

* decode: add accessor macros for HuffmanCode fields

Adds a constructor function for building HuffmanCode values
so they can be accessed quickly on different architectures.

Also adds macros for marking a HuffmanCode table pointer
that can be accessed quickly (BROTLI_HC_MARK_TABLE_FOR_FAST_LOAD),
adjusting the index into that table (BROTLI_HC_ADJUST_TABLE_INDEX),
and getting the .bits or .value fields out of the table at the
current index (BROTLI_HC_GET_BITS/VALUE).

For example, assuming |table| contains a HuffmanCode pointer:

  BROTLI_HC_MARK_TABLE_FOR_FAST_LOAD(table);
  BROTLI_HC_ADJUST_TABLE_INDEX(table, index_into_table);
  *bits = BROTLI_HC_GET_BITS(table);
  *value = BROTLI_HC_GET_VALUE(table);
  BROTLI_HC_ADJUST_TABLE_INDEX(table, offset);
  *bits2 = BROTLI_HC_GET_BITS(table);
  *value2 = BROTLI_HC_GET_VALUE(table);

All uses of the HuffmanCode have been updated appropriately.

* decode: add alternative accessors for HuffmanCode on Arm AArch32
2018-09-27 13:15:46 +02:00
Eugene Kliuchnikov
2216a0dd63
Update (#706)
Update
 * add ASAN/MSAN unaligned read specializations
 * add "brotli" prefix to u_uint64 type
 * increment version to 1.0.06
 * fix CoverityScan "unused assignment" warning
 * fix JDK 8<->9 incompatibility
 * add encoder optimization for empty input
 * regenerate JS decoder
 * unbreak Travis builds
2018-09-13 08:09:32 -04:00
Stephen Kyle
d4cd6cdf1c platform: fix unaligned 64-bit accesses on AArch32 (#702)
Ensures that Aarch32 Arm builds with an Armv8 compiler do not set
BROTLI_64_BITS.

This scenario is possible with ChromeOS builds, as they may use a
toolchain with the target armv7-cros-gnueabi, but with -march=armv8.
This will set __ARM_ARCH to 8 (defining BROTLI_TARGET_ARMV8), but will
also set __ARM_32BIT_STATE and not __ARM_64BIT_STATE. Without this,
illegal 64-bit non-word-aligned reads (LDRD) may be emitted.

Also fix unaligned 64-bit reads on AArch32 - STRD was still possible to
emit.
2018-07-25 11:43:06 +02:00
Eugene Kliuchnikov
8a073bd9e2
Revert "platform: fix unaligned 64-bit accesses on AArch32 (#699)" (#701)
This reverts commit 6d027d1648.
2018-07-24 17:32:13 +02:00
Stephen Kyle
6d027d1648 platform: fix unaligned 64-bit accesses on AArch32 (#699)
Ensures that Aarch32 Arm builds with an Armv8 compiler do not set
BROTLI_64_BITS.

This scenario is possible with ChromeOS builds, as they may use a
toolchain with the target armv7-cros-gnueabi, but with -march=armv8.
This will set __ARM_ARCH to 8 (defining BROTLI_TARGET_ARMV8), but will
also set __ARM_32BIT_STATE and not __ARM_64BIT_STATE. Without this,
illegal 64-bit non-word-aligned reads (LDRD) may be emitted.

Also fix unaligned 64-bit reads on AArch32 - STRD was still possible to
emit.
2018-07-24 17:29:50 +02:00
Eugene Kliuchnikov
b601fe817b Ramp up version to 1.0.5 2018-06-27 17:03:45 +02:00
Eugene Kliuchnikov
eb12ec04eb
Update (#688)
* add rolling-composite-hasher for large-window mode
* make API methods explicitly public
2018-06-20 15:14:10 +02:00
Eugene Kliuchnikov
7505290ef9
Convert fuzzer to C99. (#686) 2018-06-18 14:39:38 +02:00
Eugene Kliuchnikov
09cd3e877f
Update 2018-06-11 15:17:26 +02:00
Eugene Kliuchnikov
8544ae858d
Update (#680)
* fix MSVC warnings
 * cleanups
2018-06-09 11:17:13 +02:00
Eugene Kliuchnikov
1e7ea1d8e6
Inverse bazel project/workspace tree (#677)
* Inverse bazel workspace tree.

Now each subproject directly depends on root (c) project.

This helps to mitigate Bazel bug bazelbuild/bazel#2391; short summary:
Bazel does not work if referenced subproject `WORKSPACE` uses any
repositories that embedding project does not.

Bright side: building C project is much faster;
no need to download closure, go and JDK...
2018-06-04 17:53:16 +02:00
davidlt
f9b8c02673 Add RISC-V 64-bit (riscv64) platform configuration (#669)
Signed-off-by: David Abdurachmanov <david.abdurachmanov@gmail.com>
2018-05-22 14:35:04 +02:00
Eugene Kliuchnikov
48a25b3fa4
Fix #671 (#672) 2018-05-18 22:07:52 +02:00
Eugene Kliuchnikov
f5ed35d065
Update (#664)
* Update
 * fix ifdef style
 * get back to fine-compiler-version-based-macros (use Hedley)
 * fix q=0 histogram collection for very long copy/insert commands
2018-05-03 11:16:21 +02:00
Eugene Kliuchnikov
6000396155
Remove unprefixed macros from public headers (#662) 2018-04-20 14:10:55 +02:00
Eugene Kliuchnikov
68db5c0272
Update (#660)
* Update
 * improve q=1 compression on small files
 * fix "left shift before promotion"
 * fix osx Travis builds
2018-04-13 11:44:34 +02:00
Eugene Kliuchnikov
c6333e1e79
Fix MSVC compilation (#657)
* tell bazel not to pass strict options to a fancy compiler
 * fix signed-unsigned comparison warning found by MSVC
2018-03-29 10:37:07 +02:00
Eugene Kliuchnikov
0f3c84e745
Update (#656)
* proper fix for the "fall through" warning"
 * automatic NDIRECT/NPOSTFIX tuning (better compression)
 * fix unaligned access for `aarch64`-cross-`armhf` build
 * fix `aarch64` detection (10% decoder speedup)
 * expose `large_window` CLI option
 * make default window size 16MiB
 * ramp up version to 1.0.4
2018-03-27 22:29:22 +02:00
Eugene Kliuchnikov
631fe194a1
Update (#651)
* fix `bazel` build (ignore switch case fall-through)
* add `NPOSTFIX` / `NDIRECT` encoder parameters
* fix source file lists (add `params.h`)
* fix bug in `durchschlag`
* print clarifying messages wheb CLI argument parsing fails
2018-03-20 17:37:41 +06:00
Eugene Kliuchnikov
533843e354
Update (#643)
Update
 * make the zopflification aware of `NDIRECT`, `NPOSTFIX`
   (better compression in `font` mode)
 * add small and simple decoder tool
 * fix typo
 * Java: wrapper: make decoder channel more async-friendly

Ramp up version to 1.0.3 / 1.0.3
2018-03-02 15:49:58 +01:00
Eugene Kliuchnikov
35e69fc7cf
New feature: "Large Window Brotli" (#640)
* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.
2018-02-26 09:04:36 -05:00
Eugene Kliuchnikov
da254cffdb
Update (#630)
* merge {dec|enc}/port.h into common/platform.h
 * fix one-shot q=10 1-byte input compression
 * fix some unprefixed definitions
 * make hashers host-endianness-independent
 * extract enc/params.h from enc/quality.h
 * fix API documentation / typos
 * improve `BrotliEncoderMaxCompressedSize`
2017-12-12 14:33:12 +01:00
Eugene Kliuchnikov
0ad94eed00
Update (#620)
* add autotools build
* separate semantic and ABI version
* extract sources.lst (used by CMake and Automake)
* share pkgconfig templates (used by CMake and Automake)
* decoder: always set `total_out`
* encoder: fix `BROTLI_ENSURE_CAPACITY` macro (no-op after preprocessor)
* decoder/encoder: refine `free_func` contract
2017-11-28 15:37:28 +01:00
Eugene Kliuchnikov
3e58ea5f90 Update (#617)
* remove `const` on `BrotliDictionary` members
 * extend `ZofliNode` distance range to 128MiB
 * add missing `port.h` include to `quality.h`
 * fix typo in encoder API-doc
 * regenerate `decode.min.js`
2017-10-13 14:50:51 +02:00
Tomáš Popela
a0c7dafe28 Fix permissions of various files in project (#613)
Move from 755 to 644.
2017-10-10 11:24:13 +02:00
Eugene Kliuchnikov
4f8cd4c0f4 Fix fuzzer test script and add it to travis matrix (#606) 2017-09-26 13:49:30 +02:00
Eugene Kliuchnikov
5b4769990d Ramp up to version to 1.0.1 2017-09-22 14:05:06 +02:00
Eugene Kliuchnikov
c60563591a Fix API documentation + theoretical NPEs (#602) 2017-09-20 15:02:01 +02:00
Eugene Kliuchnikov
37fb83ec0d Update: (#600)
* encoder: relax backward references candidates asserts
 * encoder: make RNG more platform-independent
 * encoder: remove "unused" param (context mode)
 * CLI: improve first-encounter experience
 * Java: update SynthTest
 * Java: refine proguard config
 * Java/JNI: fix one-shot compression workflow
2017-09-19 15:57:15 +02:00
Eugene Kliuchnikov
d7bce1e092 Update (#593)
* Update:
 * fix CLI error messages
 * fix CLI console IO on Windows
2017-09-07 20:27:49 +02:00
Eugene Kliuchnikov
a629289e32 Update (#590)
* add transpiled JS decoder
 * make PY wrapper accept memview
 * fix dictionary generator
 * speedup compression of RLEish data
2017-08-28 11:31:29 +02:00