Commit Graph

845 Commits

Author SHA1 Message Date
Eugene Kliuchnikov
527db7af8c Release org.brotli.* 0.1.0 (#514) 2017-02-20 15:51:48 +01:00
Eugene Kliuchnikov
56a7fda830 Java: fix typos and adjust visibility. (#513) 2017-02-20 14:04:55 +01:00
Evan Nemerson
d03c38da7f Blacklist PGI from using conformant array parameters. (#511)
* Blacklist PGI from using conformant array parameters.

There is a bug in pgcc with conformant array parameters where the
length argument is a pointer which triggers a compiler error
(PGC-S-0094, to be specific).  The issue has been reported to PGI and
is being tracked internally as TPR 23778.  For more information, see

  https://www.pgroup.com/userforum/viewtopic.php?t=5501

* travis: Add PGI Community Edition build.

For details on the installation script, see
https://github.com/nemequ/pgi-travis
2017-02-19 10:06:13 +01:00
Eugene Kliuchnikov
53366083e0 Prepare org.brotli.dec for deployment. (#512) 2017-02-17 15:39:34 +01:00
Eugene Kliuchnikov
9fa1ad5a91 Fix "zero-distance-code", take 2 (#506) 2017-02-08 21:14:01 +01:00
Eugene Kliuchnikov
0749d9ca8b Fix #502 decoder bug (#505)
Decoder may have produced invalid output if:
 * at offset 0..3 dictionary word with index 3..0 for some length N is used
   and distance is encoded with direct distance code 0, and
 * at least one of next 4 commands use value from distance ringbuffer
2017-02-07 15:35:03 +01:00
Eugene Kliuchnikov
11df843cf0 Update encoder (#504)
* pull `BROTLI_MAX_BACKWARD_LIMIT` to constants
 * split generic and Zopfli backward references code
 * pull hashers init and stitch invocation to encoder
 * make `dictionary_hash` a compilation unit
 * add `size hint` parameter
 * add new hasher
 * use `size hint` to pick new hasher for q4
 * modernize clz guard (fix #495)
 * move `hash to binary tree` to separate file
 * add `Initialize` and `Cleanup` to all hashers
 * do not raise OOM if malloc(0) == NULL (fix #500)
2017-02-06 14:20:43 +01:00
Eugene Kliuchnikov
8d3fdc1dfe Update encoder (#497)
* pad dictionary LUTs to length 32, etc. (#493)
 * avoid using INFINITY constant (#496)
 * make dictionary_hash.h more compact
 * add "disable literal context modelling" parameter
2017-01-26 11:32:18 +01:00
Eugene Kliuchnikov
7e347a7c84 Update encoder (#492)
* fix comment position in `context.h`
 * fix typo in internal quality constant name
 * deduplicate `BuildMetaBlockGreedy` code
 * simplify aggregation in `ChooseContextMap`
2016-12-22 15:55:05 +01:00
Eugene Kliuchnikov
27d94590a2 Research (#491)
* add advanced mode for optimal references generator
 * fix #489

Thanks to Ivan Nikulin for working on it.
2016-12-22 13:03:28 +01:00
Eugene Kliuchnikov
fe9f9a9182 Split brotli common/dec/enc .pc files (#490)
Add URL, and use DEPENS_PRIVATE generator params
2016-12-22 08:57:44 +01:00
Alex Nicksay
6ab0a5cee7 Python: Create Makefile for development shortcuts (#488) 2016-12-21 10:17:11 +01:00
Eugene Kliuchnikov
fd96151b2a Move brotlidump.py to research/ (#487) 2016-12-20 18:00:51 +01:00
Eugene Kliuchnikov
5814438791 Add configure-cmake (#474) (#486)
* Add configure-cmake

 * `curl https://raw.githubusercontent.com/nemequ/configure-cmake/7b0464af79bbaca535f0279316558e1d84e5c124/configure > configure-cmake`

* Add `--disable-shared-libs` parameter.

* Unix-friendly script prologue.

* Update README.md
2016-12-20 17:45:40 +01:00
jneb
f62cd2bcd2 brotlidump.py: disassemble brotli file (revisited) (#314)
* Create brotlidump.py

Sorry, I am a newbie. I couldn't find my file anymore when I wanted to edit it. Hope I don't waste your time.

* Fixed a bug where it couldn't read its own compression.

The problem was that a prefix code ending with a 16 "repeat" didn't realize the table was full already.
Also minor bug fixes, comments and stuff.

* Major refactoring

Rewrote almost everything.
Now can dump its own compression.

* Now more or less complete

Appears to handle all files completely (including metablock data).
Used as inspiration for the the hex example (see makehexexample.py)
2016-12-20 14:41:47 +01:00
Alex Nicksay
89a5b6e625 Python: Simplify test suite generation by using unittest discovery (#485) 2016-12-20 14:40:47 +01:00
Alex Nicksay
6f227228ce Python: Use a temporary directory for generated files in tests (#481) 2016-12-12 10:28:44 +01:00
Alex Nicksay
4651f7c000 Python: Format bro.py with yapf (#480) 2016-12-12 10:28:15 +01:00
Eugene Kliuchnikov
0ee416139f Update python brotli wrapper (#479)
* Update python brotli wrapper
 * release GIL on CPU intensive blocks, fixes #476
 * use BrotliDecoderTakeOutput (less memory, less memcpy)

* Python: Convert bro.py tests to unittest style (#478)

* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
  of iterables using `itertools.product`

* Update python brotli wrapper
 * release GIL on CPU intensive blocks, fixes #476
 * use BrotliDecoderTakeOutput (less memory, less memcpy)
2016-12-12 10:27:13 +01:00
Alex Nicksay
4a60128c13 Python: Convert bro.py tests to unittest style (#478)
* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
  of iterables using `itertools.product`
2016-12-09 13:44:05 +01:00
Frank Denis
50bc3a7145 Do not assume that bash is installed in /bin (#477)
This is required in order to run the tests on *BSD.
2016-12-09 08:58:13 +01:00
Eugene Kliuchnikov
ccabf811ff Added fuzzer and updated decoder (#475)
* log dictionary usage
 * remove dead assignment
 * added fuzzer for https://github.com/google/oss-fuzz
 * added standalone test for fuzzer
2016-12-08 12:55:18 +01:00
Eugene Kliuchnikov
222564a95d Fix encoder (#472)
* fix undefined behavior introduced with PR #468
2016-12-02 13:32:20 +01:00
Piotr Sikora
6a4bf43968 Fix build with -Wconditional-uninitialized. (#471)
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2016-12-02 09:52:54 +01:00
Eugene Kliuchnikov
396309a529 Update (#470)
* condense generated `static_dict_lut.h`
 * implement BrotliInputStream.close()
2016-11-30 13:36:20 +01:00
Eugene Kliuchnikov
5db62dcc9d Fixes: (#468)
* fix slow-down after a long copy (q10-11)
 * more thorough hashing for long ranges (q10-11)
 * minor documentation fixes
 * bazel.io -> bazel.build
2016-11-09 14:04:09 +01:00
Alex Nicksay
1e5ea6aedd Python: Add unit tests for brotli.compress and brotli.decompress (#467)
Also
  - rename `test_utils` to `_test_utils`
  - refactor shared code into `_test_utils`
2016-11-09 12:21:13 +01:00
Evan Nemerson
12750768c2 bro: check return values of chown and chmod (#465)
Apparently some libc versions declare chown with the warn_unused_result
attribute, which is enabled by default.
2016-11-02 14:03:06 +01:00
Evan Nemerson
6c47009892 FInishing touches for installing libbrotli with CMake (#464)
* build: fix bundled mode + BUILD_SHARED_LIBS

* cmake: add soversion information

* cmake: generate pkg-config file
2016-11-01 10:03:29 +01:00
Eugene Kliuchnikov
e9b278ac6e Update docs and add more java tests (#463)
* doxygenize and update API documentation
 * fix spelling
 * add "fuzz" corpus for java decoder to improve coverage
 * use upper-case-snake names for dictionary constant definitions
 * use `LDFLAGS` in conventional `Makefile`
2016-10-31 14:33:59 +01:00
Alex Nicksay
a260b6ba73 Python: Add tests for streamed compression (#458)
Progress on #191
2016-10-31 13:24:01 +01:00
Alex Nicksay
9203765492 Python: Use "build" instead of "build_ext" in scripts (#460)
Previously, the Python package consisted of a single extension
module, so `build_ext` was sufficient.  Now, the package
contains a native module and an extension module, so both
`build_py` and `build_ext` are required.  Instead, run `build`,
which calls both `build_py` and `build_ext` automatically.
2016-10-31 12:58:45 +01:00
Alex Nicksay
1a8ee40de9 Python: Run Appveyor tests in CMD mode (#461)
Any command executed in PowerShell mode that writes to `stderr`
is treated as failing.  To avoid this problem, run tests in CMD
mode instead.
2016-10-27 17:25:05 +02:00
Mo DeJong
3b9d4a227d enable rbit instruction for arm64 (#459) 2016-10-27 17:21:12 +02:00
Eugene Kliuchnikov
4e157c409a Update API (#457)
* explicitly define `BROTLI_BOOL` to be `int`
 * add `BROTLI_` prefix to `MAKE_UINT64_T` macros
 * replace `true`/`false`/`1`/`0` mentions with `BROTLI_TRUE`/`FALSE`
 * add `BrotliEncoderSetParameter` documentation
 * add explicit caution to `BrotliEncoderMaxCompressedSize`
 * fix formatting in `port.h`
2016-10-25 16:02:05 +02:00
Alex Nicksay
afb1272792 Python: Publicly expose the Compressor object in the Python API (#456)
Progress on #191
2016-10-25 10:19:29 +02:00
Alex Nicksay
5632315d35 Python: Support streamed compression with the Compressor object (#448)
This adds `flush` and `finish` methods to the `Compressor`
object in the extension module, renames the `compress` method to
`process`, and updates that method to only process data.  Now,
one or more `process` calls followed by a `finish` call will be
equivalent to a module-level `compress` call.

Note: To maximize the compression efficiency (and match
underlying Brotli behavior, the `Compressor` object `process`
method does not guarantee all input is immediately written to
output. To ensure immediate output, call `flush` to manually
flush the compression buffer.  Extraneous flushing can increase
the size, but may be required when processing streaming data.

Progress on #191
2016-10-24 13:28:56 +02:00
Eugene Kliuchnikov
678f8627d3 Fix OSX gcc-4.x compilation (#455)
Fix OSX gcc-4.x compilation
2016-10-20 14:16:00 +02:00
Eugene Kliuchnikov
b1db6f149a Fix -Wcast-align warnings 2016-10-20 10:28:44 +02:00
Eugene Kliuchnikov
74147a1a41 Merge pull request #454 from fred-wang/brotli-readme
Fix build instructions for cmake
2016-10-19 22:27:45 +02:00
Frédéric Wang
82c297f356 Fix build instructions for cmake 2016-10-19 21:44:42 +02:00
Eugene Kliuchnikov
058a113dd9 Merge pull request #451 from eustas/flush
Fix "take output" flush workflow.
2016-10-19 20:03:43 +02:00
Eugene Kliuchnikov
8bcaabb0d1 Fix "take output" flush workflow. 2016-10-19 16:19:26 +02:00
Eugene Kliuchnikov
1b364aeb42 Merge pull request #450 from eustas/master
Build shared libraries by default
2016-10-18 20:52:04 +02:00
Eugene Kliuchnikov
b93cb69831 * leave static compilation declaration intouch (e.g. Python build) 2016-10-18 17:14:49 +02:00
Eugene Kliuchnikov
f5ba0b6c17 (compress_fragment_two_pass) 2016-10-18 16:56:39 +02:00
Eugene Kliuchnikov
69982c25f1 Build shared libraries by default
* Declare `BUILD_SHARED_LIBS` option for CMake
* Define `${LIB}_SHARED_COMPILATION` when compiling shared library
* Define and use BROTLI_xxx_API
* Fix remaining unprefixed defines in port.h
2016-10-18 16:45:32 +02:00
Eugene Kliuchnikov
0781cb10ab Merge pull request #449 from eustas/master
Fix POM files sources paths
2016-10-18 15:29:39 +02:00
Eugene Kliuchnikov
d18c7369d9 Fix POM files sources paths
* also add javadocs and sources generation
2016-10-18 15:28:43 +02:00
Eugene Kliuchnikov
2d441179bb Merge pull request #446 from nicksay/py-3-compressor-object
Python: Create an extension Compressor object
2016-10-18 10:31:05 +02:00