2014-04-15 13:03:17 +00:00
LZ4 - Extremely fast compression
================================
2016-02-12 22:34:07 +00:00
LZ4 is lossless compression algorithm,
2018-09-05 21:05:08 +00:00
providing compression speed > 500 MB/s per core,
2016-02-12 22:34:07 +00:00
scalable with multi-cores CPU.
It features an extremely fast decoder,
with speed in multiple GB/s per core,
2015-03-31 08:44:56 +00:00
typically reaching RAM speed limits on multi-core systems.
2015-05-04 13:43:37 +00:00
Speed can be tuned dynamically, selecting an "acceleration" factor
2018-09-05 21:05:08 +00:00
which trades compression ratio for faster speed.
2015-05-04 13:43:37 +00:00
On the other end, a high compression derivative, LZ4_HC, is also provided,
trading CPU time for improved compression ratio.
2015-10-20 08:46:18 +00:00
All versions feature the same decompression speed.
2018-09-05 21:05:08 +00:00
LZ4 is also compatible with [dictionary compression ](https://github.com/facebook/zstd#the-case-for-small-data-compression ),
2019-04-11 22:19:48 +00:00
both at [API ](https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4frame.h#L481 ) and [CLI ](https://github.com/lz4/lz4/blob/v1.8.3/programs/lz4.1.md#operation-modifiers ) levels.
It can ingest any input file as dictionary, though only the final 64KB are used.
This capability can be combined with the [Zstandard Dictionary Builder ](https://github.com/facebook/zstd/blob/v1.3.5/programs/zstd.1.md#dictionary-builder ),
in order to drastically improve compression performance on small files.
2018-09-05 21:05:08 +00:00
2016-11-02 02:14:04 +00:00
LZ4 library is provided as open-source software using BSD 2-Clause license.
2015-05-04 13:43:37 +00:00
2014-04-15 13:03:17 +00:00
2014-05-03 01:49:14 +00:00
|Branch |Status |
|------------|---------|
2015-05-29 17:50:12 +00:00
|master | [![Build Status][travisMasterBadge]][travisLink] [![Build status][AppveyorMasterBadge]][AppveyorLink] [![coverity][coverBadge]][coverlink] |
|dev | [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] |
2016-11-03 17:52:03 +00:00
[travisMasterBadge]: https://travis-ci.org/lz4/lz4.svg?branch=master "Continuous Integration test suite"
[travisDevBadge]: https://travis-ci.org/lz4/lz4.svg?branch=dev "Continuous Integration test suite"
[travisLink]: https://travis-ci.org/lz4/lz4
2017-06-19 06:14:52 +00:00
[AppveyorMasterBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=master& svg=true "Windows test suite"
[AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=dev& svg=true "Windows test suite"
2016-11-15 20:58:53 +00:00
[AppveyorLink]: https://ci.appveyor.com/project/YannCollet/lz4-1lndh
2015-05-29 17:50:12 +00:00
[coverBadge]: https://scan.coverity.com/projects/4735/badge.svg "Static code analysis of Master branch"
[coverlink]: https://scan.coverity.com/projects/4735
2014-04-15 13:03:17 +00:00
2015-03-16 10:26:49 +00:00
> **Branch Policy:**
> - The "master" branch is considered stable, at all times.
2015-04-13 19:43:54 +00:00
> - The "dev" branch is the one where all contributions must be merged
2015-03-31 08:44:56 +00:00
before being promoted to master.
2015-04-13 19:43:54 +00:00
> + If you plan to propose a patch, please commit into the "dev" branch,
or its own feature branch.
2015-03-31 08:44:56 +00:00
Direct commit to "master" are not permitted.
2014-04-15 13:03:17 +00:00
Benchmarks
-------------------------
2016-11-16 19:36:45 +00:00
The benchmark uses [lzbench], from @inikep
2019-04-13 01:00:43 +00:00
compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17).
The reference system uses a Core i7-9700K CPU @ 4.9GHz.
2015-03-31 08:44:56 +00:00
Benchmark evaluates the compression of reference [Silesia Corpus]
in single-thread mode.
2014-04-15 13:03:17 +00:00
2016-11-16 19:36:45 +00:00
[lzbench]: https://github.com/inikep/lzbench
[Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
2018-05-02 23:41:15 +00:00
| Compressor | Ratio | Compression | Decompression |
| ---------- | ----- | ----------- | ------------- |
2019-04-13 01:00:43 +00:00
| memcpy | 1.000 | 13700 MB/s | 13700 MB/s |
|**LZ4 default (v1.9.0)** |**2.101**| **780 MB/s** | **4900 MB/s** |
| LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
| QuickLZ 1.5.0 | 2.238 | 575 MB/s | 780 MB/s |
| Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
| [Zstandard] 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
| LZF v3.6 | 2.073 | 415 MB/s | 910 MB/s |
| [zlib] deflate 1.2.11 -1| 2.730 | 100 MB/s | 415 MB/s |
|**LZ4 HC -9 (v1.8.2)** |**2.721**| 39 MB/s | **4850 MB/s** |
| [zlib] deflate 1.2.11 -6| 3.099 | 36 MB/s | 445 MB/s |
2016-02-25 13:49:40 +00:00
[zlib]: http://www.zlib.net/
[Zstandard]: http://www.zstd.net/
2015-05-04 13:43:37 +00:00
2018-09-05 21:05:08 +00:00
LZ4 is also compatible and optimized for x32 mode,
for which it provides additional speed performance.
2017-08-14 22:13:23 +00:00
Installation
-------------------------
2017-08-14 22:18:47 +00:00
```
make
2018-09-05 21:05:08 +00:00
make install # this command may require root permissions
2017-08-14 22:18:47 +00:00
```
2017-08-14 22:13:23 +00:00
LZ4's `Makefile` supports standard [Makefile conventions],
2017-08-14 22:18:47 +00:00
including [staged installs], [redirection], or [command redefinition].
2017-08-30 22:02:04 +00:00
It is compatible with parallel builds (`-j#`).
2017-08-14 22:13:23 +00:00
[Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
[staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
2017-08-14 22:18:47 +00:00
[redirection]: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
2017-08-14 22:13:23 +00:00
[command redefinition]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html
2016-11-19 01:54:26 +00:00
2015-05-04 13:43:37 +00:00
Documentation
-------------------------
2015-03-30 17:32:21 +00:00
2015-04-06 00:03:49 +00:00
The raw LZ4 block compression format is detailed within [lz4_Block_format].
2015-03-26 18:49:15 +00:00
2018-09-05 21:05:08 +00:00
Arbitrarily long files or data streams are compressed using multiple blocks,
for streaming requirements. These blocks are organized into a frame,
defined into [lz4_Frame_format].
Interoperable versions of LZ4 must also respect the frame format.
2015-05-04 13:43:37 +00:00
2016-11-16 19:36:45 +00:00
[lz4_Block_format]: doc/lz4_Block_format.md
[lz4_Frame_format]: doc/lz4_Frame_format.md
2015-05-04 13:43:37 +00:00
Other source versions
-------------------------
2016-02-12 22:34:07 +00:00
Beyond the C reference source,
2015-05-29 19:36:59 +00:00
many contributors have created versions of lz4 in multiple languages
(Java, C#, Python, Perl, Ruby, etc.).
A list of known source ports is maintained on the [LZ4 Homepage].
2015-05-04 13:43:37 +00:00
2015-10-20 08:46:18 +00:00
[LZ4 Homepage]: http://www.lz4.org