Documentation:
- add note that brotli is a "stream" format, not an archive-like
- regenerate .1 with Pandoc
Build:
- drop legacy "BROTLI_BUILD_PORTABLE" option
- drop "BROTLI_SANITIZED" definition
Code:
- c: comb includes
- c/enc: extract encoder state into separate header
- c/enc: drop designated q10 codepath
- c/enc: dealing better with flushing of empty stream
- fix MSVC compilation
API:
- py: use library version instead of one in version.h
- c: add plugable API to report consumed input / produced output
- c/java: support "lean" prepared dictionaries (without copy of source)
Not all combinations are migrated to the initial configuration; corresponding TODOs added.
Drive-by: additional combinations uncovered minor portability problems -> fixed
Drive-by: remove no-longer used "script" files.
Co-authored-by: Eugene Kliuchnikov <eustas@chromium.org>
* re-enable Js build/test
* improve decoder performance
* rewrite dictionary data in Java/Js to a shorter uncompressed form
* improve dictionary generation tool
* IMPORTANT: decoder: fix potential overflow when input chunk is >2GiB
* simplify max Huffman table size calculation
* eliminate symbol duplicates (static arrays in .h files)
* minor combing in research/ code
Otherwise libraries will not be found at runtime when installing to a
path not included in the default runtime linker's path with programs
linking brotli configured via pkg-config.
* build: add cross-compilation support to make
Set CROSS_COMPILE when running make to use the selected cross
compilation toolchain, such as arm-linux-gnueabihf, or
aarch64-linux-gnu.
Testing requires the presence of qemu - 'qemu-$(ARCH)' will be executed,
where ARCH is the first part of the toolchain triplet.
* build: add cross-compilation support to cmake
If C_COMPILER/CXX_COMPILER/CC/CXX are found to have cross-compilation
triplets in front of the compiler, then qemu will be used to execute the
tests.
* CI: add arm-linux-gnueabihf-gcc builder to Travis
The version of qemu available in Ubuntu trusty (as provided by Travis)
appears to have a bug in qemu-aarch64, which leads to the compatibility
tests failing on some inputs, erroneously rejecting the input as
corrupt.
Once Travis supports xenial, we could add an aarch64-gnu-linux-gcc
builder as well.
* CI: propagate cmake errors out of .travis.sh
Seems like even if cmake fails, the error isn't picked up by Travis.
Our dist tarball is missing hash_rolling_inc.h and
hash_composite_inc.h, which causes subsequent autotools
builds to fail. Fix this by adding it to the sources list.
Signed-off-by: William A. Kennington III <william@wkennington.com>
* Inverse bazel workspace tree.
Now each subproject directly depends on root (c) project.
This helps to mitigate Bazel bug bazelbuild/bazel#2391; short summary:
Bazel does not work if referenced subproject `WORKSPACE` uses any
repositories that embedding project does not.
Bright side: building C project is much faster;
no need to download closure, go and JDK...
* New feature: "Large Window Brotli"
By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.
Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).
Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.
Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.
Other C changes:
- combined / moved `context.h` to `common`
- moved transforms to `common`
- unified some aspects of code formatting
- added an abstraction for encoder (static) dictionary
- moved default allocator/deallocator functions to `common`
brotli CLI:
- window size is auto-adjusted if not specified explicitly
Java:
- added "eager" decoding both to JNI wrapper and pure decoder
- huge speed-up of `DictionaryData` initialization
* Add dictionaryless compressed dictionary
* Fix `sources.lst`
* Fix `sources.lst` and add a note that `libtool` is also required.
* Update setup.py
* Fix `EagerStreamTest`
* Fix BUILD file
* Add missing `libdivsufsort` dependency
* Fix "unused parameter" warning.
* Update go and closure bazel rules
* Follow the new bazel go rules guide
* Swap go & closure rules initialization
* Update bazel to 0.10.0 in appveyor build
* Use Clang-5.0
* Disable unholy ASAN leak detector (to unbreak build)
* Reduce build matrix and use faster env, where compiler version is not important
* Add autotools build to Travis matrix
* add autotools build
* separate semantic and ABI version
* extract sources.lst (used by CMake and Automake)
* share pkgconfig templates (used by CMake and Automake)
* decoder: always set `total_out`
* encoder: fix `BROTLI_ENSURE_CAPACITY` macro (no-op after preprocessor)
* decoder/encoder: refine `free_func` contract
* [appveyor] remove 'deploy' stage; only test python 2.7 and 3.6
all the other python versions are being built and tested on
https://github.com/google/brotli-wheels/blob/d571d63/appveyor.yml
* remove terrify submodule as not needed any more
* [travis] just test py2.7 and 3.6 on linux; remove extra osx python builds
All the other python versions for OSX are being built/tested on:
https://github.com/google/brotli-wheels/blob/d571d63/.travis.yml
Also, there's no need to build and deploy wheels here, as that's done
in the separate repository.
* [setup.py] only rebuild if dependency are newer; fix typo in list of 'depends'
https://github.com/python/cpython/blob/v3.6.2/Lib/distutils/command/build_ext.py#L485-L500
* [ci] only run 'python setup.py test'
if we run 'python setup.py built test', the setuptools 'test' command will
forcibly re-run the build_ext subcommand because it wants to pass the --inplace
option (it ignores whether it's up to date, just re-runs it all the time).
with this we go from running built_ext twice, to running it only once per build
* [Makefile] run 'build_ext --inplace' instead of 'develop' as default target
The 'develop' command is like 'install' in the sense that it
modifies the user's python environment.
The default make target should be less intrusive, i.e. just building
the extension module in-place without modify anything in the user's
environment.
We don't need to tell make about the dependency between 'test' and
'build' target as that is baked in the `python setup.py test` command.
* [Makefile] add 'develop' target; remove unnecessary 'tests' target
`make test` is good enough
* [Makefile] `setup.py test` requires setuptools; run `python -m unittest`
This will work even if setuptools is not installed, which is unlikely
nowadays but still our `setup.py` works with plain distutils, so
we may well have our tests work without setuptools.
* [python/README.md] add ref to 'develop' target; remove 'tests', just 'make test'
* [setup.py] import modules as per nicksay's comment
https://github.com/google/brotli/pull/583#discussion_r131981049
* [Makefile] add 'develop' to .PHONY targets
remove 'tests' from .PHONY
* [appveyor] remove unused setup scripts
We don't need to install custom python versions, we are
using the pre-installed ones on Appveyor.
* [appveyor] remove unneeded setup code
