365 lines
19 KiB
HTML
365 lines
19 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
|
|
<title>LZ4 - Extremely fast compression</title>
|
|
|
|
<!-- Bootstrap -->
|
|
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
|
|
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
|
|
|
|
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
|
|
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
|
|
<!--[if lt IE 9]>
|
|
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
|
|
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
|
|
<![endif]-->
|
|
|
|
<link rel="stylesheet" href="js/bootstrap-patch.css">
|
|
<style>
|
|
.list-inline li a img {
|
|
height: 50px;
|
|
}
|
|
</style>
|
|
</head>
|
|
<body style="padding-top:0; padding-bottom:0; ">
|
|
<nav class="navbar navbar-default">
|
|
<div class="container-fluid">
|
|
<!-- Brand and toggle get grouped for better mobile display -->
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
|
|
<span class="sr-only">Toggle navigation</span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand" href="/">LZ4</a>
|
|
</div>
|
|
|
|
<!-- Collect the nav links, forms, and other content for toggling -->
|
|
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
|
|
<ul class="nav navbar-nav navbar-nav">
|
|
<li><a href="https://github.com/Cyan4973/lz4/releases/latest">Latest Release</a></li>
|
|
<li><a href="#interoperable-lz4">Versions</a></li>
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Index<span class="caret"></span></a>
|
|
<ul class="dropdown-menu" role="menu">
|
|
<li><a href="#benchmarks">Benchmarks</a></li>
|
|
<li><a href="#interoperable-lz4">Interoperable LZ4 versions</a></li>
|
|
<li><a href="#other-languages">Other Source versions</a></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li><a href="https://groups.google.com/forum/#!forum/lz4c">Public discussion forum</a></li>
|
|
<li><a href="https://github.com/Cyan4973/lz4/issues">Issue Tracker</a></li>
|
|
<li><a href="https://github.com/Cyan4973/lz4"><i class="fa fa-lg fa-github"></i> GitHub</a></li>
|
|
</ul>
|
|
</div><!-- /.navbar-collapse -->
|
|
</div><!-- /.container-fluid -->
|
|
</nav>
|
|
|
|
<a name="summary"></a>
|
|
<div class="container">
|
|
<div class="page-header jumbotron text-justify" style="padding-top:1em;padding-bottom:2em;">
|
|
<h1>LZ4</h1>
|
|
<iframe src="https://ghbtns.com/github-btn.html?user=Cyan4973&repo=lz4&type=star&count=true&size=large&v=2" frameborder="0" scrolling="0" width="160px" height="30px"></iframe>
|
|
<p>
|
|
LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core (0.16 Bytes/cycle).
|
|
It also features an extremely fast decoder, with speed in multiple GB/s per core (0.71 Bytes/cycle).
|
|
A high compression derivative, called LZ4_HC, is also provided.
|
|
It trades CPU time for compression ratio.
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
<xmp theme="sandstone" style="display:none;">
|
|
<a name="benchmarks"></a>
|
|
Benchmarks
|
|
----------
|
|
|
|
The benchmark uses the
|
|
[Open-Source Benchmark program by m^2 (v0.14.2)](http://encode.ru/threads/1371-Filesystem-benchmark?p=33548&viewfull=1#post33548)
|
|
compiled with GCC v4.6.1 on Linux Ubuntu 64-bits v11.10,
|
|
The reference system uses a Core i5-3340M @2.7GHz.
|
|
Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia) in single-thread mode.
|
|
|
|
<div>
|
|
<ul id="tabs" class="nav nav-tabs" data-tabs="tabs">
|
|
<li class="active"><a href="#tab-1" data-toggle="tab">Transfer + Decompression Time<br/>@1000Mbit/s</a></li>
|
|
<li ><a href="#tab-2" data-toggle="tab">Compression + Transfer + Decompression Time<br/>@1000Mbit/s</a></li>
|
|
</ul>
|
|
<div id="my-tab-content" class="tab-content">
|
|
<div id="tab-1" class="tab-pane container-fluid active">
|
|
<div class="row">
|
|
<div class="col-md-4 text-justify">
|
|
<h2>Transfer + Decompression Time @1000Mbit/s <small>(without I/O overwrapping, multithreading)</small></h2>
|
|
<p>
|
|
This benchmark simulates simple "static content transfer" scenario such as OS Kernel compression
|
|
or video game's static assets (text/images/tables/scripts/etc) which loading from Flash Memory / HDD / SSD.
|
|
</p>
|
|
<p>
|
|
In this case, compression time is completely ignored.
|
|
Because only content developers compress the data at once and usually they don't care about its computational cost.
|
|
But they always care enduser's experience a.k.a. "loading time" and bandwidth.
|
|
</p>
|
|
<p>
|
|
Please pay attention to "LZ4HC -9" which is quite faster than other methods.
|
|
</p>
|
|
</div>
|
|
<div class="col-md-8"
|
|
style="height:34em;"
|
|
data-csv-props='{
|
|
"csvSrc": "csv/benchmark-silesia-1000mbps-offline-compression.csv",
|
|
"csvGvType": "google.visualization.ColumnChart",
|
|
"csvGvPackage": "corechart",
|
|
"chartArea": {"width": "80%", "height": "30em"},
|
|
"legend": {"position": "top"},
|
|
"isStacked": "true",
|
|
"vAxis": {"title": "Seconds" },
|
|
"title": "Transfer + Decompression Time\nSMALLER IS BETTER"
|
|
}'
|
|
></div>
|
|
</div>
|
|
</div>
|
|
<div id="tab-2" class="tab-pane container-fluid">
|
|
<div class="row">
|
|
<div class="col-md-4 text-justify">
|
|
<h2>Compression + Transfer + Decompression Time @1000Mbit/s <small>(without I/O overwrapping, multithreading)</small></h2>
|
|
<p>
|
|
This benchmark simulates "dynamic content transfer" scenario such as filesystems, caches, network packets, IPC/RPCs, and databases.
|
|
</p>
|
|
<p>
|
|
In this case, both compression and decompression times are important.
|
|
You can observe "Fast compression algorithms" are better than traditional algorithms such as DEFLATE (zlib).
|
|
</p>
|
|
</div>
|
|
<div class="col-md-8"
|
|
style="height:34em;"
|
|
data-csv-props='{
|
|
"csvSrc": "csv/benchmark-silesia-1000mbps.csv",
|
|
"csvGvType": "google.visualization.ColumnChart",
|
|
"csvGvPackage": "corechart",
|
|
"chartArea": {"width": "80%", "height": "30em"},
|
|
"legend": {"position": "top"},
|
|
"isStacked": "true",
|
|
"vAxis": {"title": "Seconds" },
|
|
"title": "Compression + Transfer + Decompression Time\nSMALLER IS BETTER"
|
|
}'
|
|
></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
## Documentation
|
|
|
|
The LZ4 block compression format is detailed within [lz4_Block_format](lz4_Block_format.md).
|
|
|
|
For streaming arbitrarily large amount of data, or compress files of any size,
|
|
a frame format has been established, detailed within the file [lz4_Frame_format](lz4_Frame_format.md).
|
|
|
|
|
|
|
|
<br/>
|
|
<a name="interoperable-lz4"></a>
|
|
## Interoperable LZ4 versions
|
|
The following versions are provided for languages beyond the C reference version.
|
|
They are in conformance with the LZ4 [block] and [frame] specifications, and are therefore interoperable.
|
|
|
|
|Language |Author |URL |
|
|
|-- |-- |-- |
|
|
|__Javascript__ |Pierre Curto |https://github.com/pierrec/node-lz4 |
|
|
|__C++11 multi-threads__ |Takayuki Matsuoka |https://github.com/t-mat/lz4mt |
|
|
|__Python__ |Christopher Jackson |https://github.com/darkdragn/lz4tools |
|
|
|__Delphi__ |Hanno Hugenberg |https://github.com/Hugie/lz4-delphi |
|
|
|__Go__ |Pierre Curto |https://github.com/pierrec/lz4 |
|
|
|__Rust__ |Artem Navrotskiy |https://github.com/bozaro/lz4-rs |
|
|
|
|
[block]:lz4_Block_format.md
|
|
[frame]:lz4_Frame_format.md
|
|
|
|
|
|
<br/>
|
|
<a name="other-languages"></a>
|
|
## Multiple languages
|
|
The following versions compress data blocks with LZ4 compression algorithm using various programming languages.
|
|
Each version encapsulates the compressed result into its own header/frame format convention.
|
|
|
|
|Language |Author |URL |
|
|
|-- |-- |-- |
|
|
|__Java__ |Adrien Grand |https://github.com/jpountz/lz4-java |
|
|
|__Python__ |Steeve Morin |http://pypi.python.org/pypi/lz4 |
|
|
|__Perl__ |Gray |http://search.cpan.org/dist/Compress-LZ4/ |
|
|
|__C#__ |Milosz Krajewski |http://lz4net.codeplex.com |
|
|
|__C#__ streaming |Phill Djonov |https://github.com/pdjonov/Lz4Stream |
|
|
|__Go__ |Branimir Karadzic |https://github.com/bkaradzic/go-lz4 |
|
|
|__Ruby__ |Komiya Atsushi |http://rubygems.org/gems/lz4-ruby |
|
|
|__PHP__ |Kamijo |https://github.com/kjdev/php-ext-lz4 |
|
|
|__Lua__ |Christophe Delord |http://cdsoft.fr/bl/bonaluna.html |
|
|
|__D__ |Jude Young |http://code.dlang.org/packages/lz4-d |
|
|
|__Haskell__ |Mark Wotton |http://hackage.haskell.org/package/lz4 |
|
|
|__Erlang__ |Tetsuya Suzuki |https://github.com/szktty/erlang-lz4 |
|
|
|__Smalltalk__ (Pharo) |Mariano Martinez Peck |http://smalltalkhub.com/#!/~marianopeck/LZ4/ |
|
|
|__OCaml__ |Peter Zotov |https://github.com/whitequark/ocaml-lz4 |
|
|
|__Rust__ |Alex Crichton |http://alexcrichton.com/rust-compress/compress/lz4/index.html |
|
|
|__8088 assembly__ decoder |Jim Leonard |http://www.oldskool.org/pc/lz4_8088 |
|
|
|__6502 & 65C02 assembly__ decoder |Peter Ferrie |http://pferrie.host22.com/misc/appleii.htm |
|
|
|__Atari XL/XE assembly__ decoder |xxl |http://xxl.atari.pl/?p=1524 |
|
|
|__65c816 assembly__ decoder |Olivier Zardini |http://www.brutaldeluxe.fr/products/crossdevtools/lz4/index.html |
|
|
|
|
|
|
|
|
<br/>
|
|
<a name="lz4-is-used-by"></a>
|
|
## LZ4 is used by
|
|
|
|
#### Operating Systems
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.linux.org" ><img src="images/logo50/linux.png" /><span> Linux</span></a></li>
|
|
<li><a href="https://www.freebsd.org/"><img src="images/logo50/freebsd.png"/><span> FreeBSD</span></a></li>
|
|
<li><a href="https://www.illumos.org/"><img src="images/logo50/illumos.png"/><span> Illumos</span></a></li>
|
|
<li><a href="https://smartos.org/" ><img src="images/logo50/smartos.png"/><span> SmartOS</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### File Systems
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.open-zfs.org/wiki/Main_Page"><img src="images/logo50/openzfs.png" /><span> OpenZFS</span></a></li>
|
|
<li><a href="https://www.dragonflybsd.org/hammer/" ><img src="images/logo50/hammer.png" /><span> HAMMER2</span></a></li>
|
|
<li><a href="https://squashfs.sourceforge.net/" ><img src="images/logo50/squashfs.png"/><span> SquashFS</span></a></li>
|
|
<li><a href="http://www.lessfs.com/wordpress/" ><img src="images/logo50/lessfs.png" /><span> LessFS</span></a></li>
|
|
<li><a href="https://leo-project.net/leofs/" ><img src="images/logo50/leofs.png" /><span> LeoFS</span></a></li>
|
|
<li><a href="https://www.gnu.org/software/grub/" ><img src="images/logo50/grub.png" /><span> GNU GRUB</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Big Data
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://hadoop.apache.org/" ><img src="images/logo50/hadoop.png" /><span> Hadoop</span></a></li>
|
|
<li><a href="https://cassandra.apache.org/" ><img src="images/logo50/cassandra.png" /><span> Cassandra</span></a></li>
|
|
<li><a href="https://hbase.apache.org/" ><img src="images/logo50/hbaseW90.png" /><span> Hbase</span></a></li>
|
|
<li><a href="https://www.rarelogic.com/" ><img src="images/logo50/rarelogic.png" /><span> Rarelogic</span></a></li>
|
|
<li><a href="https://spark.apache.org/" ><img src="images/logo50/spark.png" /><span> Spark</span></a></li>
|
|
<li><a href="https://github.com/chango/hustle" ><img src="images/logo50/hustleW90.png" /><span> Hustle</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Search Engine
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://lucene.apache.org/" ><img src="images/logo50/lucene.png" /><span> Lucene</span></a></li>
|
|
<li><a href="https://lucene.apache.org/solr/" ><img src="images/logo50/solr.png" /><span> solr</span></a></li>
|
|
<li><a href="https://www.scalyr.com/" ><img src="images/logo50/scalyr.png" /><span> Scalyr</span></a></li>
|
|
<li><a href="https://kafka.apache.org/" ><img src="images/logo50/kafka.png" /><span> Kafka</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Database
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.mysql.com/" ><img src="images/logo50/mysql.png" /><span> MySQL</span></a></li>
|
|
<li><a href="https://www.tokutek.com/tokudb-for-mysql/" ><img src="images/logo50/tokudb.png" /><span> Tokudb</span></a></li>
|
|
<li><a href="https://www.delphix.com/" ><img src="images/logo50/delphix.png" /><span> Delphix</span></a></li>
|
|
<li><a href="http://www.infinisql.org/" ><img src="images/logo50/infinisql.png" /><span> infiniSQL</span></a></li>
|
|
<li><a href="https://rocksdb.org/" ><img src="images/logo50/rocksdb.png" /><span> RocksDB</span></a></li>
|
|
<li><a href="https://olegdb.org/" ><img src="images/logo50/olegdb.png" /><span> OlegDB</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Games
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.battlefield.com/battlefield-4" ><img src="images/logo50/battlefield4.png" /><span> Battlefield 4</span></a></li>
|
|
<li><a href="https://www.guildwars2.com/" ><img src="images/logo50/guildwars2.png" /><span> Guild Wars 2</span></a></li>
|
|
<li><a href="http://www.crytek.com/" ><img src="images/logo50/crytek.png" /><span> Crytek</span></a></li>
|
|
<li><a href="https://www.sinemoragame.com/" ><img src="images/logo50/sinemora.png" /><span> Sine Mora</span></a></li>
|
|
<li><a href="https://www.mediatonic.co.uk/blog/1000-tiny-claws-out-today"><img src="images/logo50/1000tinycraws.png" /><span> 1000 Tiny Claws</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Graphics
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.nvidia.com/" ><img src="images/logo50/nvidia.png" /><span> nVidia</span></a></li>
|
|
<li><a href="https://www.enlightenment.org/" ><img src="images/logo50/enlightment.png" /><span> Enlightenment</span></a></li>
|
|
<li><a href="https://www.xpra.org/" ><img src="images/logo50/xpra.png" /><span> Xpra</span></a></li>
|
|
<li><a href="https://www.openvdb.org/" ><img src="images/logo50/openvdb.png" /><span> OpenVDB</span></a></li>
|
|
<li><a href="https://scap.codeplex.com/" ><img src="images/logo50/placeholder.png" /><span> Scaplib</span></a></li>
|
|
<li><a href="https://github.com/flanglet/kanzi" ><img src="images/logo50/placeholder.png" /><span> kanzi</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Network
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.freenas.org/" ><img src="images/logo50/freenas.png" /><span> freeNAS</span></a></li>
|
|
<li><a href="https://openvpn.net/" ><img src="images/logo50/openvpn.png" /><span> openVPN</span></a></li>
|
|
<li><a href="https://netty.io/" ><img src="images/logo50/netty.png" /><span> Netty</span></a></li>
|
|
<li><a href="https://dovecot.org/" ><img src="images/logo50/dovecot.png" /><span> dovecot</span></a></li>
|
|
<li><a href="https://www.virtualhere.com/" ><img src="images/logo50/virtualhere.png" /><span> virtualHere</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Storage
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.datastax.com/" ><img src="images/logo50/datastax.png" /><span> DataStax</span></a></li>
|
|
<li><a href="https://www.hybrid-san.co.uk/" ><img src="images/logo50/nimblestorage.png" /><span> Nimble Storage</span></a></li>
|
|
<li><a href="https://www.nexenta.com/" ><img src="images/logo50/nexenta.png" /><span> Nexenta</span></a></li>
|
|
<li><a href="https://kaminario.com/flash-array/" ><img src="images/logo50/k2spear.png" /><span> K2 Spear</span></a></li>
|
|
<li><a href="https://syneto.net/" ><img src="images/logo50/syneto.png" /><span> Syneto OS</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Caching
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://en.wikipedia.org/wiki/ZRam" ><img src="images/logo50/linux.png" /><span> Zram</span></a></li>
|
|
<li><a href="https://hhvm.com/" ><img src="images/logo50/hhvm.png" /><span> HHVM</span></a></li>
|
|
<li><a href="https://www.zend.com/" ><img src="images/logo50/zend.png" /><span> PHP Zend Optimizer</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
#### Other
|
|
<div id="lz4usedby" class="container">
|
|
<ul id="icons" class="list-inline">
|
|
<li><a href="https://www.mozilla.org/" ><img src="images/logo50/firefox.png" /><span> Mozilla Firefox</span></a></li>
|
|
<li><a href="https://www.bareos.org/en/" ><img src="images/logo50/bareos.png" /><span> Bareos</span></a></li>
|
|
<li><a href="https://gtkwave.sourceforge.net/" ><img src="images/logo50/gtkwave.png" /><span> GTKWave</span></a></li>
|
|
<li><a href="https://www.blosc.org/" ><img src="images/logo50/placeholder.png" /><span> Blosc</span></a></li>
|
|
<li><a href="https://bitbucket.org/facebook/lz4revlog" ><img src="images/logo50/fb-mercurial.png" /><span> Facebook's Mercurial</span></a></li>
|
|
<li><a href="https://github.com/ptaoussanis/nippy" ><img src="images/logo50/placeholder.png" /><span> Nippy</span></a></li>
|
|
<li><a href="https://www.freedesktop.org/wiki/Software/systemd" ><img src="images/logo50/freedesktop.png" /><span> systemd</span></a></li>
|
|
</ul>
|
|
</div>
|
|
|
|
|
|
### Special Thanks to
|
|
|
|
[Takatuki Matsuoka](https://github.com/t-mat/)
|
|
who greatly contributed to the creation of this LZ4 webpage
|
|
|
|
---
|
|
</xmp>
|
|
|
|
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
|
|
<!-- Include all compiled plugins (below), or include individual files as needed -->
|
|
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
|
|
<script src="strapdown/v/0.2/strapdown.js"></script>
|
|
|
|
<!-- Chart -->
|
|
<script src="https://www.google.com/jsapi"></script>
|
|
<script src="js/rendercharts.js"></script>
|
|
</body>
|
|
</html>
|