Currently, the output buffer is a std::vector<uint8_t>.
When the buffer grows, resizing will cause unnecessary memcpy().
This change uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing.
In addition, C++ code can be removed, it's a pure C extension.
Documentation:
- add note that brotli is a "stream" format, not an archive-like
- regenerate .1 with Pandoc
Build:
- drop legacy "BROTLI_BUILD_PORTABLE" option
- drop "BROTLI_SANITIZED" definition
Code:
- c: comb includes
- c/enc: extract encoder state into separate header
- c/enc: drop designated q10 codepath
- c/enc: dealing better with flushing of empty stream
- fix MSVC compilation
API:
- py: use library version instead of one in version.h
- c: add plugable API to report consumed input / produced output
- c/java: support "lean" prepared dictionaries (without copy of source)
- fix formatting
- fix type conversion
- fix no-op arithmetic with null-pointer
- improve performance of hash_longest_match64
- go: detect read after close
- java decoder: support compound dictionary
- remove executable flag on non-scripts
* [appveyor] remove 'deploy' stage; only test python 2.7 and 3.6
all the other python versions are being built and tested on
https://github.com/google/brotli-wheels/blob/d571d63/appveyor.yml
* remove terrify submodule as not needed any more
* [travis] just test py2.7 and 3.6 on linux; remove extra osx python builds
All the other python versions for OSX are being built/tested on:
https://github.com/google/brotli-wheels/blob/d571d63/.travis.yml
Also, there's no need to build and deploy wheels here, as that's done
in the separate repository.
* [setup.py] only rebuild if dependency are newer; fix typo in list of 'depends'
https://github.com/python/cpython/blob/v3.6.2/Lib/distutils/command/build_ext.py#L485-L500
* [ci] only run 'python setup.py test'
if we run 'python setup.py built test', the setuptools 'test' command will
forcibly re-run the build_ext subcommand because it wants to pass the --inplace
option (it ignores whether it's up to date, just re-runs it all the time).
with this we go from running built_ext twice, to running it only once per build
* [Makefile] run 'build_ext --inplace' instead of 'develop' as default target
The 'develop' command is like 'install' in the sense that it
modifies the user's python environment.
The default make target should be less intrusive, i.e. just building
the extension module in-place without modify anything in the user's
environment.
We don't need to tell make about the dependency between 'test' and
'build' target as that is baked in the `python setup.py test` command.
* [Makefile] add 'develop' target; remove unnecessary 'tests' target
`make test` is good enough
* [Makefile] `setup.py test` requires setuptools; run `python -m unittest`
This will work even if setuptools is not installed, which is unlikely
nowadays but still our `setup.py` works with plain distutils, so
we may well have our tests work without setuptools.
* [python/README.md] add ref to 'develop' target; remove 'tests', just 'make test'
* [setup.py] import modules as per nicksay's comment
https://github.com/google/brotli/pull/583#discussion_r131981049
* [Makefile] add 'develop' to .PHONY targets
remove 'tests' from .PHONY
* [appveyor] remove unused setup scripts
We don't need to install custom python versions, we are
using the pre-installed ones on Appveyor.
* [appveyor] remove unneeded setup code
python-brotli has Compressor for streaming compression but nothing for
streaming decompression.
This is a straight-forward copy of the Compressor code into the new
class Decompressor.
* Create brotlidump.py
Sorry, I am a newbie. I couldn't find my file anymore when I wanted to edit it. Hope I don't waste your time.
* Fixed a bug where it couldn't read its own compression.
The problem was that a prefix code ending with a 16 "repeat" didn't realize the table was full already.
Also minor bug fixes, comments and stuff.
* Major refactoring
Rewrote almost everything.
Now can dump its own compression.
* Now more or less complete
Appears to handle all files completely (including metablock data).
Used as inspiration for the the hex example (see makehexexample.py)
* Update python brotli wrapper
* release GIL on CPU intensive blocks, fixes#476
* use BrotliDecoderTakeOutput (less memory, less memcpy)
* Python: Convert bro.py tests to unittest style (#478)
* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
of iterables using `itertools.product`
* Update python brotli wrapper
* release GIL on CPU intensive blocks, fixes#476
* use BrotliDecoderTakeOutput (less memory, less memcpy)
* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
of iterables using `itertools.product`
Previously, the Python package consisted of a single extension
module, so `build_ext` was sufficient. Now, the package
contains a native module and an extension module, so both
`build_py` and `build_ext` are required. Instead, run `build`,
which calls both `build_py` and `build_ext` automatically.
This adds `flush` and `finish` methods to the `Compressor`
object in the extension module, renames the `compress` method to
`process`, and updates that method to only process data. Now,
one or more `process` calls followed by a `finish` call will be
equivalent to a module-level `compress` call.
Note: To maximize the compression efficiency (and match
underlying Brotli behavior, the `Compressor` object `process`
method does not guarantee all input is immediately written to
output. To ensure immediate output, call `flush` to manually
flush the compression buffer. Extraneous flushing can increase
the size, but may be required when processing streaming data.
Progress on #191
- Create a `Compressor` object in the extension module
- Move the `compress` method into the native module and use
the new `Compressor` object to do the compression
Note: This does not change the module-level Python API. The
`Compressor` object will not be publicly exposed until its
methods have stabilized.
