Commit Graph

113 Commits

Author SHA1 Message Date
Evgenii Kliuchnikov
2dfaadcef3 (PY) clarify compressor mode parameter values
PiperOrigin-RevId: 698023020
2024-11-19 08:15:20 -08:00
Thomas Fischbacher
acc265655d Small Python modernization of Brotli code.
PiperOrigin-RevId: 549289787
2023-07-19 05:44:36 -07:00
Evgenii Kliuchnikov
bc32ae12d5 add tests with UTF8/UTF16 non-ASCII text
PiperOrigin-RevId: 545424981
2023-07-05 19:15:11 +00:00
Evgenii Kliuchnikov
38e9add9d2 Fix permissions
PiperOrigin-RevId: 506096478
2023-02-01 09:20:14 +00:00
Brotli
36533a866e Internal change
PiperOrigin-RevId: 502401179
2023-01-17 13:51:00 +00:00
Ma Lin
c8df4b3049
Python: use a new output buffer code (#902)
Currently, the output buffer is a std::vector<uint8_t>.
When the buffer grows, resizing will cause unnecessary memcpy().

This change uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing.
In addition, C++ code can be removed, it's a pure C extension.
2022-12-29 14:07:16 +01:00
Evgenii Kliuchnikov
a8f5813b84 Update
Documentation:
  - add note that brotli is a "stream" format, not an archive-like
  - regenerate .1 with Pandoc
Build:
  - drop legacy "BROTLI_BUILD_PORTABLE" option
  - drop "BROTLI_SANITIZED" definition
Code:
  - c: comb includes
  - c/enc: extract encoder state into separate header
  - c/enc: drop designated q10 codepath
  - c/enc: dealing better with flushing of empty stream
  - fix MSVC compilation
API:
  - py: use library version instead of one in version.h
  - c: add plugable API to report consumed input / produced output
  - c/java: support "lean" prepared dictionaries (without copy of source)
2022-11-17 13:03:09 +00:00
Eugene Kliuchnikov
8376f72ed6
Prepare for copybara (#939)
Co-authored-by: Eugene Kliuchnikov <eustas@chromium.org>
2021-11-10 10:34:39 +01:00
Eugene Kliuchnikov
68f1b90ad0
Update (#918)
Prepare to use copybara worklow.
2021-08-18 19:15:07 +02:00
Eugene Kliuchnikov
7f740f1308
Update (#807)
- fix formatting
 - fix type conversion
 - fix no-op arithmetic with null-pointer
 - improve performance of hash_longest_match64
 - go: detect read after close
 - java decoder: support compound dictionary
 - remove executable flag on non-scripts
2020-05-15 11:06:21 +02:00
Eugene Kliuchnikov
c8b37e8fd1
Update (#762)
* put LICENSE file into .jar
 * fix typo
 * add clarification comment in PY wrapper
2019-07-17 14:39:56 +02:00
Justin Ridgewell
5805f99a53 Ensure decompression consumes all input (#730)
* Ensure decompression consumes all input

If not, it's a corrupt stream.

* Use byte strings
2018-11-12 10:36:00 +01:00
Eugene Kliuchnikov
2c03482569
Fix "memory leak" in python tests (#652)
OOMs on RPi (1GB)
2018-03-23 02:09:00 +06:00
Eugene Kliuchnikov
a629289e32 Update (#590)
* add transpiled JS decoder
 * make PY wrapper accept memview
 * fix dictionary generator
 * speedup compression of RLEish data
2017-08-28 11:31:29 +02:00
Cosimo Lupo
4f455cac32 disable buidling/deployment of python wheels (#583)
* [appveyor] remove 'deploy' stage; only test python 2.7 and 3.6

all the other python versions are being built and tested on
https://github.com/google/brotli-wheels/blob/d571d63/appveyor.yml

* remove terrify submodule as not needed any more

* [travis] just test py2.7 and 3.6 on linux; remove extra osx python builds

All the other python versions for OSX are being built/tested on:
https://github.com/google/brotli-wheels/blob/d571d63/.travis.yml

Also, there's no need to build and deploy wheels here, as that's done
in the separate repository.

* [setup.py] only rebuild if dependency are newer; fix typo in list of 'depends'

https://github.com/python/cpython/blob/v3.6.2/Lib/distutils/command/build_ext.py#L485-L500

* [ci] only run 'python setup.py test'

if we run 'python setup.py built test', the setuptools 'test' command will
forcibly re-run the build_ext subcommand because it wants to pass the --inplace
option (it ignores whether it's up to date, just re-runs it all the time).

with this we go from running built_ext twice, to running it only once per build

* [Makefile] run 'build_ext --inplace' instead of 'develop' as default target

The 'develop' command is like 'install' in the sense that it
modifies the user's python environment.
The default make target should be less intrusive, i.e. just building
the extension module in-place without modify anything in the user's
environment.

We don't need to tell make about the dependency between 'test' and
'build' target as that is baked in the `python setup.py test` command.

* [Makefile] add 'develop' target; remove unnecessary 'tests' target

`make test` is good enough

* [Makefile] `setup.py test` requires setuptools; run `python -m unittest`

This will work even if setuptools is not installed, which is unlikely
nowadays but still our `setup.py` works with plain distutils, so
we may well have our tests work without setuptools.

* [python/README.md] add ref to 'develop' target; remove 'tests', just 'make test'

* [setup.py] import modules as per nicksay's comment

https://github.com/google/brotli/pull/583#discussion_r131981049

* [Makefile] add 'develop' to .PHONY targets

remove 'tests' from .PHONY

* [appveyor] remove unused setup scripts

We don't need to install custom python versions, we are
using the pre-installed ones on Appveyor.

* [appveyor] remove unneeded setup code
2017-08-23 20:45:13 +02:00
Alex Nicksay
019091f994 Python: Update bro_test to reference script directly (#582) 2017-08-08 10:25:39 +02:00
Eugene Kliuchnikov
d63e8f75f5 Update API, and more (#581)
Update API, and more:
 * remove "custom dictionary" support
 * c/encoder: fix #580: big-endian build
 * Java: reduce jar size
 * Java: speedup decoding
 * Java: add 32-bit CPU support
 * Java: make source code JS transpiler-ready
2017-08-04 10:02:56 +02:00
Alex Nicksay
0608253110 Python: Add a "make install" command and clarify installation docs (#578)
Closes #576
2017-08-02 16:59:46 +02:00
Eugene Kliuchnikov
52441069ef Update (#574)
* Update
 * decoder: better behavior after failure
 * encoder: replace "len_x_code" with delta
 * research: add experimental dictionary generator
 * python: test combing
2017-07-21 10:07:24 +02:00
Eugene Kliuchnikov
1becbbf231 Update (#569)
* add misssing fclose in `brotli.c`
 * add basic tests for python `Decompressor` type
 * minor lint fixes in `_brotli.cc`
2017-06-30 13:09:50 +02:00
Janek
58f5c37f3b Python: Decompressor: Streaming decompression support (#546)
python-brotli has Compressor for streaming compression but nothing for
streaming decompression.
This is a straight-forward copy of the Compressor code into the new
class Decompressor.
2017-06-28 16:32:28 +02:00
Alex Nicksay
6ab0a5cee7 Python: Create Makefile for development shortcuts (#488) 2016-12-21 10:17:11 +01:00
Eugene Kliuchnikov
fd96151b2a Move brotlidump.py to research/ (#487) 2016-12-20 18:00:51 +01:00
jneb
f62cd2bcd2 brotlidump.py: disassemble brotli file (revisited) (#314)
* Create brotlidump.py

Sorry, I am a newbie. I couldn't find my file anymore when I wanted to edit it. Hope I don't waste your time.

* Fixed a bug where it couldn't read its own compression.

The problem was that a prefix code ending with a 16 "repeat" didn't realize the table was full already.
Also minor bug fixes, comments and stuff.

* Major refactoring

Rewrote almost everything.
Now can dump its own compression.

* Now more or less complete

Appears to handle all files completely (including metablock data).
Used as inspiration for the the hex example (see makehexexample.py)
2016-12-20 14:41:47 +01:00
Alex Nicksay
89a5b6e625 Python: Simplify test suite generation by using unittest discovery (#485) 2016-12-20 14:40:47 +01:00
Alex Nicksay
6f227228ce Python: Use a temporary directory for generated files in tests (#481) 2016-12-12 10:28:44 +01:00
Alex Nicksay
4651f7c000 Python: Format bro.py with yapf (#480) 2016-12-12 10:28:15 +01:00
Eugene Kliuchnikov
0ee416139f Update python brotli wrapper (#479)
* Update python brotli wrapper
 * release GIL on CPU intensive blocks, fixes #476
 * use BrotliDecoderTakeOutput (less memory, less memcpy)

* Python: Convert bro.py tests to unittest style (#478)

* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
  of iterables using `itertools.product`

* Update python brotli wrapper
 * release GIL on CPU intensive blocks, fixes #476
 * use BrotliDecoderTakeOutput (less memory, less memcpy)
2016-12-12 10:27:13 +01:00
Alex Nicksay
4a60128c13 Python: Convert bro.py tests to unittest style (#478)
* Create unittest-style tests for `bro.py` decompression and compression
* Delete old tests for `bro.py`
* Update test method generation to properly create a Cartesian product
  of iterables using `itertools.product`
2016-12-09 13:44:05 +01:00
Alex Nicksay
1e5ea6aedd Python: Add unit tests for brotli.compress and brotli.decompress (#467)
Also
  - rename `test_utils` to `_test_utils`
  - refactor shared code into `_test_utils`
2016-11-09 12:21:13 +01:00
Alex Nicksay
a260b6ba73 Python: Add tests for streamed compression (#458)
Progress on #191
2016-10-31 13:24:01 +01:00
Alex Nicksay
9203765492 Python: Use "build" instead of "build_ext" in scripts (#460)
Previously, the Python package consisted of a single extension
module, so `build_ext` was sufficient.  Now, the package
contains a native module and an extension module, so both
`build_py` and `build_ext` are required.  Instead, run `build`,
which calls both `build_py` and `build_ext` automatically.
2016-10-31 12:58:45 +01:00
Alex Nicksay
afb1272792 Python: Publicly expose the Compressor object in the Python API (#456)
Progress on #191
2016-10-25 10:19:29 +02:00
Alex Nicksay
5632315d35 Python: Support streamed compression with the Compressor object (#448)
This adds `flush` and `finish` methods to the `Compressor`
object in the extension module, renames the `compress` method to
`process`, and updates that method to only process data.  Now,
one or more `process` calls followed by a `finish` call will be
equivalent to a module-level `compress` call.

Note: To maximize the compression efficiency (and match
underlying Brotli behavior, the `Compressor` object `process`
method does not guarantee all input is immediately written to
output. To ensure immediate output, call `flush` to manually
flush the compression buffer.  Extraneous flushing can increase
the size, but may be required when processing streaming data.

Progress on #191
2016-10-24 13:28:56 +02:00
Eugene Kliuchnikov
2d441179bb Merge pull request #446 from nicksay/py-3-compressor-object
Python: Create an extension Compressor object
2016-10-18 10:31:05 +02:00
Alex Nicksay
b04f4ea185 Python: Update README with information about code formatting
Also, add a `yapf` section to `setup.cfg` to ensure YAPF runs
format code with the Google style.
2016-10-17 13:57:56 -04:00
Alex Nicksay
595a5246b4 Python: Create an extension Compressor object
- Create a `Compressor` object in the extension module
- Move the `compress` method into the native module and use
  the new `Compressor` object to do the compression

Note: This does not change the module-level Python API.  The
`Compressor` object will not be publicly exposed until its
methods have stabilized.
2016-10-17 13:03:58 -04:00
Alex Nicksay
f7b5b3dc2c Python: Create native brotli module and move extension to _brotli 2016-10-17 09:35:27 -04:00
Eugene Kliuchnikov
86fdb68373 Update brotlimodule.cc 2016-09-21 16:02:32 +02:00
Eugene Kliuchnikov
7cbdb4aa0c Update brotlimodule.cc 2016-09-21 15:51:54 +02:00
Eugene Kliuchnikov
b754f607aa Update python module
* use new decoder API
2016-09-21 15:37:45 +02:00
Eugene Kliuchnikov
8148001158 Move "public" to "include/brotli" 2016-08-23 14:40:33 +02:00
Eugene Kliuchnikov
2c2d5578a6 Use version from common/version.h 2016-08-22 15:44:12 +02:00
Eugene Kliuchnikov
dae9c7ffd0 Fix brotlimodule 2016-08-22 14:00:42 +02:00
eustas
27f9d00efc Fix issue #383 2016-06-23 11:02:53 +02:00
eustas
81054a4667 Restore platform suffix 2016-06-17 20:24:24 +02:00
eustas
f0c7ece315 Remove platform suffix 2016-06-16 17:45:05 +02:00
eustas
bac060f1a1 Fix library path 2016-06-16 17:25:26 +02:00
Eugene Kliuchnikov
db3a11625d Fix CI build. 2016-06-13 15:22:13 +02:00
Eugene Kliuchnikov
b972c67780 Convert encoder to plain C. 2016-06-13 11:01:04 +02:00