AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge branch 'dev' of github.com:facebook/zstd into dev

Browse Source
This commit is contained in:
Yann Collet 2017-05-03 14:33:39 -07:00
commit b668badf00
2 changed files with 79 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
contrib/linux-kernel/README.md
View File

@ -1,6 +1,6 @@
# Linux Kernel Patch
There are two pieces, the `zstd_compress` and `zstd_decompress` kernel modules, and the BtrFS patch.
There are three pieces, the `zstd_compress` and `zstd_decompress` kernel modules, the BtrFS patch, and the SquashFS patch.
The patches are based off of the linux kernel master branch (version 4.10).
## Zstd Kernel modules
@ -22,14 +22,15 @@ The patches are based off of the linux kernel master branch (version 4.10).
* The patch is located in `btrfs.diff`.
* Additionally `fs/btrfs/zstd.c` is provided as a source for convenience.
* The patch seems to be working, it doesn't crash the kernel, and compresses at speeds and ratios athat are expected.
It can still use some more testing for fringe features, like printing options.
* The patch seems to be working, it doesn't crash the kernel, and compresses at speeds and ratios that are expected.
It could still use some more testing for fringe features, like printing options.
### Benchmarks
Benchmarks run on a Ubuntu 14.04 with 2 cores and 4 GiB of RAM.
The VM is running on a Macbook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of ram, and a SSD.
The kernel running was built from the master branch with the patch (version 4.10).
The compression benchmark is copying 10 copies of the
unzipped [silesia corpus](http://mattmahoney.net/dc/silesia.html) into a BtrFS
@ -50,3 +51,39 @@ See `btrfs-benchmark.sh` for details.
| zstd 9 | 2.92 | 43 MB/s | 406 MB/s |
| zstd 12 | 2.93 | 21 MB/s | 408 MB/s |
| zstd 15 | 3.01 | 11 MB/s | 354 MB/s |
## SquashFS
* The patch is located in `squashfs.diff`
* Additionally `fs/squashfs/zstd_wrapper.c` is provided as a source for convenience.
* The patch has been tested on a 4.10 kernel.
### Benchmarks
Benchmarks run on a Ubuntu 14.04 with 2 cores and 4 GiB of RAM.
The VM is running on a Macbook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of ram, and a SSD.
The kernel running was built from the master branch with the patch (version 4.10).
The compression benchmark is the file tree from the SquashFS archive found in the
Ubuntu 16.10 desktop image (ubuntu-16.10-desktop-amd64.iso).
The compression benchmark uses mksquashfs with the default block size (128 KB)
and various compression algorithms/compression levels.
`xz` and `zstd` are also benchmarked with 256 KB blocks.
The decompression benchmark is timing how long it takes to `tar` the file tree
into `/dev/null`.
See `squashfs-benchmark.sh` for details.
| Algorithm | Compression ratio | Compression speed | Decompression speed |
|----------------|-------------------|-------------------|---------------------|
| gzip | 2.92 | 15 MB/s | 128 MB/s |
| lzo | 2.64 | 9.5 MB/s | 217 MB/s |
| lz4 | 2.12 | 94 MB/s | 218 MB/s |
| xz | 3.43 | 5.5 MB/s | 35 MB/s |
| xz 256 KB | 3.53 | 5.4 MB/s | 40 MB/s |
| zstd 1 | 2.71 | 96 MB/s | 210 MB/s |
| zstd 5 | 2.93 | 69 MB/s | 198 MB/s |
| zstd 10 | 3.01 | 41 MB/s | 225 MB/s |
| zstd 15 | 3.13 | 11.4 MB/s | 224 MB/s |
| zstd 16 256 KB | 3.24 | 8.1 MB/s | 210 MB/s |

39
contrib/linux-kernel/squashfs-benchmark.sh Executable file
View File

@ -0,0 +1,39 @@
# !/bin/sh
set -e
# Benchmarks run on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
# The VM is running on a Macbook Pro with a 3.1 GHz Intel Core i7 processor and
# 16 GB of RAM and an SSD.
# $BENCHMARK_DIR is generated with the following commands, from the Ubuntu image
# ubuntu-16.10-desktop-amd64.iso.
# > mkdir mnt
# > sudo mount -o loop ubuntu-16.10-desktop-amd64.iso mnt
# > cp mnt/casper/filesystem.squashfs .
# > sudo unsquashfs filesystem.squashfs
# $HOME is on a ext4 filesystem
BENCHMARK_DIR="$HOME/squashfs-root/"
BENCHMARK_FS="$HOME/filesystem.squashfs"
# Normalize the environment
sudo rm -f $BENCHMARK_FS 2> /dev/null > /dev/null || true
sudo umount /mnt/squashfs 2> /dev/null > /dev/null || true
# Run the benchmark
echo "Compression"
echo "sudo mksquashfs $BENCHMARK_DIR $BENCHMARK_FS $@"
time sudo mksquashfs $BENCHMARK_DIR $BENCHMARK_FS $@ 2> /dev/null > /dev/null
echo "Approximate compression ratio"
printf "%d / %d\n" \
$(sudo du -sx --block-size=1 $BENCHMARK_DIR | cut -f1) \
$(sudo du -sx --block-size=1 $BENCHMARK_FS | cut -f1);
# Mount the filesystem
sudo mount -t squashfs $BENCHMARK_FS /mnt/squashfs
echo "Decompression"
time sudo tar -c /mnt/squashfs 2> /dev/null | wc -c > /dev/null
sudo umount /mnt/squashfs