Yann Collet
commit
fdc4a01fb6
@ -16,8 +16,9 @@ env:
|
||||
- ZSTD_TRAVIS_CI_ENV=armtest
|
||||
- ZSTD_TRAVIS_CI_ENV=test
|
||||
- ZSTD_TRAVIS_CI_ENV="-C programs test32"
|
||||
- ZSTD_TRAVIS_CI_ENV=sanitize
|
||||
- ZSTD_TRAVIS_CI_ENV="-C programs memtest"
|
||||
- ZSTD_TRAVIS_CI_ENV=usan
|
||||
- ZSTD_TRAVIS_CI_ENV=asan
|
||||
- ZSTD_TRAVIS_CI_ENV="-C programs valgrindTest"
|
||||
|
||||
matrix:
|
||||
fast_finish: true
|
||||
|
||||
10
Makefile
10
Makefile
@ -32,7 +32,7 @@
|
||||
# ################################################################
|
||||
|
||||
# Version number
|
||||
export VERSION := 0.2.2
|
||||
export VERSION := 0.3.0
|
||||
|
||||
PRGDIR = programs
|
||||
ZSTDDIR = lib
|
||||
@ -83,7 +83,13 @@ armtest: clean
|
||||
$(MAKE) -C $(ZSTDDIR) -e all CC=arm-linux-gnueabi-gcc MOREFLAGS="-Werror"
|
||||
$(MAKE) -C $(PRGDIR) -e CC=arm-linux-gnueabi-gcc MOREFLAGS="-Werror"
|
||||
|
||||
sanitize: clean
|
||||
usan: clean
|
||||
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=undefined"
|
||||
|
||||
asan: clean
|
||||
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address"
|
||||
|
||||
uasan: clean
|
||||
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address -fsanitize=undefined"
|
||||
|
||||
endif
|
||||
|
||||
3
NEWS
3
NEWS
@ -1,3 +1,6 @@
|
||||
v0.3
|
||||
HC mode : compression levels 2-26
|
||||
|
||||
v0.2.2
|
||||
Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier
|
||||
|
||||
|
||||
50
README.md
50
README.md
@ -1,4 +1,4 @@
|
||||
**Zstd**, short for Zstandard, is a new lossless compression algorithm, which provides both good compression ratio _and_ speed for your standard compression needs. "Standard" translates into everyday situations which neither look for highest possible ratio nor extreme speed.
|
||||
**Zstd**, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level compression ratio.
|
||||
|
||||
It is provided as a BSD-license package, hosted on Github.
|
||||
|
||||
@ -7,42 +7,40 @@ It is provided as a BSD-license package, hosted on Github.
|
||||
|master | [](https://travis-ci.org/Cyan4973/zstd) |
|
||||
|dev | [](https://travis-ci.org/Cyan4973/zstd) |
|
||||
|
||||
For a taste of its performance, here are a few benchmark numbers, completed on a Core i7-5600U @ 2.6 GHz, using [fsbench 0.14.3](http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029), an open-source benchmark program by m^2.
|
||||
For a taste of its performance, here are a few benchmark numbers from a number of compression codecs suitable for real-time. The test was completed on a Core i7-5600U @ 2.6 GHz, using [fsbench 0.14.3](http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029), an open-source benchmark program by m^2.
|
||||
|
||||
|Name | Ratio | C.speed | D.speed |
|
||||
|----------------|-------|--------:|--------:|
|
||||
| | | MB/s | MB/s |
|
||||
| [zlib 1.2.8] -6| 3.099 | 21 | 320 |
|
||||
| **zstd 0.2** |**2.871**|**255**| **670** |
|
||||
| [zlib 1.2.8] -1| 2.730 | 70 | 300 |
|
||||
| [LZ4] HC r131 | 2.720 | 25 | 2100 |
|
||||
| QuickLZ 1.5.1b6| 2.237 | 370 | 415 |
|
||||
| LZO 2.06 | 2.106 | 400 | 580 |
|
||||
| Snappy 1.1.0 | 2.091 | 330 | 1100 |
|
||||
| [LZ4] r131 | 2.101 | 450 | 2100 |
|
||||
| LZF 3.6 | 2.077 | 200 | 560 |
|
||||
|Name | Ratio | C.speed | D.speed |
|
||||
|-----------------|-------|--------:|--------:|
|
||||
| | | MB/s | MB/s |
|
||||
| **zstd 0.3** |**2.858**|**280**| **670** |
|
||||
| [zlib 1.2.8] -1 | 2.730 | 70 | 300 |
|
||||
| QuickLZ 1.5.1b6 | 2.237 | 370 | 415 |
|
||||
| LZO 2.06 | 2.106 | 400 | 580 |
|
||||
| [LZ4] r131 | 2.101 | 450 | 2100 |
|
||||
| Snappy 1.1.0 | 2.091 | 330 | 1100 |
|
||||
| LZF 3.6 | 2.077 | 200 | 560 |
|
||||
|
||||
[zlib 1.2.8]:http://www.zlib.net/
|
||||
[LZ4]:http://www.lz4.org/
|
||||
|
||||
An interesting feature of zstd is that it can qualify as both a reasonably strong compressor and a fast one.
|
||||
Zstd can also offer stronger compression ratio at the cost of compression speed, but preserving its decompression speed. In the following test, a few compressors suitable for this scenario are selected (they offer very asymetric performance, useful when compression time has little importance). The test was completed on a Core i7-5600U @ 2.6 GHz, using [benchmark 0.6.1](http://encode.ru/threads/1266-In-memory-benchmark-with-fastest-LZSS-(QuickLZ-Snappy)-compressors?p=45217&viewfull=1#post45217), an open-source benchmark program by inikep.
|
||||
|
||||
Zstd delivers very high decompression speed, at more than >600 MB/s per core.
|
||||
Obviously, your exact mileage will vary depending on your target system.
|
||||
|Name | Ratio | C.speed | D.speed |
|
||||
|-----------------|-------|--------:|--------:|
|
||||
| | | MB/s | MB/s |
|
||||
| brotli -9 | 3.729 | 4 | 340 |
|
||||
| **zstd 0.3 -9** |**3.447**|**30** | **640** |
|
||||
| [zlib 1.2.8] -9 | 3.133 | 10 | 300 |
|
||||
| LZO 2.06 -999 | 2.790 | 1 | 560 |
|
||||
| [LZ4] r131 -9 | 2.720 | 25 | 2100 |
|
||||
|
||||
Zstd compression speed will be configurable to fit different situations.
|
||||
The first available version is the fast one, at ~250 MB/s per core, which is suitable for a few real-time scenarios.
|
||||
But similar to [LZ4], zstd can offer derivatives trading compression time for compression ratio, keeping decompression properties intact. "Offline compression", where compression time is of little importance because the content is only compressed once and decompressed many times, will likely prefer this setup.
|
||||
[lzma]:http://www.7-zip.org/
|
||||
|
||||
Note that high compression derivatives still have to be developed.
|
||||
It's a complex area which will require time and benefit from contributions.
|
||||
|
||||
|
||||
Another property zstd is developed for is configurable memory requirement, with the objective to fit into low-memory configurations, or servers handling many connections in parallel.
|
||||
Zstd compression speed is highly configurable, by small increment, to fit different situations. Its memory requirement can also be configured to fit into low-memory hardware configurations, or servers handling multiple connections/contexts in parallel.
|
||||
|
||||
Zstd entropy stage is provided by [Huff0 and FSE, from Finite State Entrop library](https://github.com/Cyan4973/FiniteStateEntropy).
|
||||
|
||||
Zstd has not yet reached "stable" status. Specifically, it doesn't guarantee yet that its current compressed format will remain stable and supported in future versions. It may still change to adapt further optimizations still being investigated. However, the library starts to be pretty robust, able to withstand hazards situations, including invalid input. The library reliability has been tested using [Fuzz Testing](https://en.wikipedia.org/wiki/Fuzz_testing), using both [internal tools](programs/fuzzer.c) and [external ones](http://lcamtuf.coredump.cx/afl). Therefore, you can now safely test zstd, even within production environments.
|
||||
Zstd has not yet reached "stable" status. Specifically, it doesn't guarantee yet that its current compressed format will remain stable and supported in future versions. It may still change to adapt further optimizations still being investigated. That being said, the library is now pretty robust, able to withstand hazards situations, including invalid input. The library reliability has been tested using [Fuzz Testing](https://en.wikipedia.org/wiki/Fuzz_testing), with both [internal tools](programs/fuzzer.c) and [external ones](http://lcamtuf.coredump.cx/afl). Therefore, it's now safe to test Zstandard even within production environments.
|
||||
|
||||
"Stable Format" is projected sometimes early 2016.
|
||||
|
||||
|
||||
@ -40,7 +40,7 @@ VERSION?= $(LIBVER)
|
||||
|
||||
DESTDIR?=
|
||||
PREFIX ?= /usr/local
|
||||
CPPFLAGS= -I./legacy -DZSTD_LEGACY_SUPPORT=1
|
||||
CPPFLAGS= -I. -I./legacy -DZSTD_LEGACY_SUPPORT=1
|
||||
CFLAGS ?= -O3
|
||||
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-qual -Wcast-align -Wstrict-prototypes
|
||||
FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MOREFLAGS)
|
||||
|
||||
@ -150,7 +150,7 @@ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
|
||||
MEM_STATIC unsigned BIT_highbit32 (register U32 val)
|
||||
{
|
||||
# if defined(_MSC_VER) /* Visual */
|
||||
unsigned long r;
|
||||
unsigned long r=0;
|
||||
_BitScanReverse ( &r, val );
|
||||
return (unsigned) r;
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */
|
||||
|
||||
@ -39,6 +39,12 @@ extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
/******************************************
|
||||
* Includes
|
||||
******************************************/
|
||||
#include <stddef.h> /* size_t, ptrdiff_t */
|
||||
|
||||
|
||||
/******************************************
|
||||
* Compiler-specific
|
||||
******************************************/
|
||||
@ -62,6 +68,7 @@ extern "C" {
|
||||
|
||||
#define ERROR_LIST(ITEM) \
|
||||
ITEM(PREFIX(No_Error)) ITEM(PREFIX(GENERIC)) \
|
||||
ITEM(PREFIX(memory_allocation)) \
|
||||
ITEM(PREFIX(dstSize_tooSmall)) ITEM(PREFIX(srcSize_wrong)) \
|
||||
ITEM(PREFIX(prefix_unknown)) ITEM(PREFIX(corruption_detected)) \
|
||||
ITEM(PREFIX(tableLog_tooLarge)) ITEM(PREFIX(maxSymbolValue_tooLarge)) ITEM(PREFIX(maxSymbolValue_tooSmall)) \
|
||||
|
||||
@ -32,7 +32,8 @@
|
||||
- Source repository : https://github.com/Cyan4973/FiniteStateEntropy
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
****************************************************************** */
|
||||
#pragma once
|
||||
#ifndef FSE_H
|
||||
#define FSE_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
@ -290,3 +291,5 @@ If there is an error, the function will return an error code, which can be teste
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* FSE_H */
|
||||
|
||||
@ -32,7 +32,8 @@
|
||||
- Source repository : https://github.com/Cyan4973/FiniteStateEntropy
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
****************************************************************** */
|
||||
#pragma once
|
||||
#ifndef FSE_STATIC_H
|
||||
#define FSE_STATIC_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
@ -305,3 +306,5 @@ MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* FSE_STATIC_H */
|
||||
|
||||
59
lib/huff0.c
59
lib/huff0.c
@ -68,6 +68,18 @@
|
||||
#include "fse.h" /* header compression */
|
||||
|
||||
|
||||
/****************************************************************
|
||||
* Constants
|
||||
****************************************************************/
|
||||
#define HUF_ABSOLUTEMAX_TABLELOG 16 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
|
||||
#define HUF_MAX_TABLELOG 12 /* max configured tableLog (for static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
|
||||
#define HUF_DEFAULT_TABLELOG HUF_MAX_TABLELOG /* tableLog by default, when not specified */
|
||||
#define HUF_MAX_SYMBOL_VALUE 255
|
||||
#if (HUF_MAX_TABLELOG > HUF_ABSOLUTEMAX_TABLELOG)
|
||||
# error "HUF_MAX_TABLELOG is too large !"
|
||||
#endif
|
||||
|
||||
|
||||
/****************************************************************
|
||||
* Error Management
|
||||
****************************************************************/
|
||||
@ -85,14 +97,6 @@ const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }
|
||||
/*********************************************************
|
||||
* Huff0 : Huffman block compression
|
||||
*********************************************************/
|
||||
#define HUF_ABSOLUTEMAX_TABLELOG 16 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
|
||||
#define HUF_MAX_TABLELOG 12 /* max configured tableLog (for static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
|
||||
#define HUF_DEFAULT_TABLELOG HUF_MAX_TABLELOG /* tableLog by default, when not specified */
|
||||
#define HUF_MAX_SYMBOL_VALUE 255
|
||||
#if (HUF_MAX_TABLELOG > HUF_ABSOLUTEMAX_TABLELOG)
|
||||
# error "HUF_MAX_TABLELOG is too large !"
|
||||
#endif
|
||||
|
||||
typedef struct HUF_CElt_s {
|
||||
U16 val;
|
||||
BYTE nbBits;
|
||||
@ -1017,29 +1021,24 @@ size_t HUF_readDTableX4 (U32* DTable, const void* src, size_t srcSize)
|
||||
rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
|
||||
}
|
||||
|
||||
/* Build rankVal */
|
||||
/* Build rankVal */
|
||||
{
|
||||
const U32 minBits = tableLog+1 - maxW;
|
||||
U32 nextRankVal = 0;
|
||||
U32 w, consumed;
|
||||
const int rescale = (memLog-tableLog) - 1; /* tableLog <= memLog */
|
||||
U32* rankVal0 = rankVal[0];
|
||||
for (w=1; w<=maxW; w++)
|
||||
{
|
||||
U32 current = nextRankVal;
|
||||
nextRankVal += rankStats[w] << (w+rescale);
|
||||
rankVal0[w] = current;
|
||||
}
|
||||
for (consumed = minBits; consumed <= memLog - minBits; consumed++)
|
||||
{
|
||||
U32* rankValPtr = rankVal[consumed];
|
||||
for (w = 1; w <= maxW; w++)
|
||||
{
|
||||
rankValPtr[w] = rankVal0[w] >> consumed;
|
||||
}
|
||||
rankVal[0][w] = current;
|
||||
}
|
||||
for (consumed=minBits; consumed <= memLog-minBits; consumed++)
|
||||
for (w=1; w<=maxW; w++)
|
||||
rankVal[consumed][w] = rankVal[0][w] >> consumed;
|
||||
}
|
||||
|
||||
|
||||
HUF_fillDTableX4(dt, memLog,
|
||||
sortedSymbol, sizeOfSort,
|
||||
rankStart0, rankVal, maxW,
|
||||
@ -1389,34 +1388,28 @@ size_t HUF_readDTableX6 (U32* DTable, const void* src, size_t srcSize)
|
||||
rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
|
||||
}
|
||||
|
||||
/* Build rankVal */
|
||||
/* Build rankVal */
|
||||
{
|
||||
const U32 minBits = tableLog+1 - maxW;
|
||||
U32 nextRankVal = 0;
|
||||
U32 w, consumed;
|
||||
const int rescale = (memLog-tableLog) - 1; /* tableLog <= memLog */
|
||||
U32* rankVal0 = rankVal[0];
|
||||
for (w=1; w<=maxW; w++)
|
||||
{
|
||||
U32 current = nextRankVal;
|
||||
nextRankVal += rankStats[w] << (w+rescale);
|
||||
rankVal0[w] = current;
|
||||
}
|
||||
for (consumed = minBits; consumed <= memLog - minBits; consumed++)
|
||||
{
|
||||
U32* rankValPtr = rankVal[consumed];
|
||||
for (w = 1; w <= maxW; w++)
|
||||
{
|
||||
rankValPtr[w] = rankVal0[w] >> consumed;
|
||||
}
|
||||
rankVal[0][w] = current;
|
||||
}
|
||||
for (consumed=minBits; consumed <= memLog-minBits; consumed++)
|
||||
for (w=1; w<=maxW; w++)
|
||||
rankVal[consumed][w] = rankVal[0][w] >> consumed;
|
||||
}
|
||||
|
||||
|
||||
/* fill tables */
|
||||
{
|
||||
HUF_DDescX6* DDescription = (HUF_DDescX6*)(DTable+1);
|
||||
HUF_DSeqX6* DSequence = (HUF_DSeqX6*)(DTable + 1 + (1<<(memLog-1)));
|
||||
HUF_DSeqX6* DSequence = (HUF_DSeqX6*)(DTable + 1 + ((size_t)1<<(memLog-1)));
|
||||
HUF_DSeqX6 DSeq;
|
||||
HUF_DDescX6 DDesc;
|
||||
DSeq.sequence = 0;
|
||||
@ -1476,7 +1469,7 @@ static U32 HUF_decodeLastSymbolsX6(void* op, const U32 maxL, BIT_DStream_t* DStr
|
||||
static inline size_t HUF_decodeStreamX6(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd, const U32* DTable, const U32 dtLog)
|
||||
{
|
||||
const HUF_DDescX6* dd = (const HUF_DDescX6*)(DTable+1);
|
||||
const HUF_DSeqX6* ds = (const HUF_DSeqX6*)(DTable + 1 + (1<<(dtLog-1)));
|
||||
const HUF_DSeqX6* ds = (const HUF_DSeqX6*)(DTable + 1 + ((size_t)1<<(dtLog-1)));
|
||||
BYTE* const pStart = p;
|
||||
|
||||
/* up to 16 symbols at a time */
|
||||
@ -1558,7 +1551,7 @@ size_t HUF_decompress4X6_usingDTable(
|
||||
|
||||
const U32 dtLog = DTable[0];
|
||||
const HUF_DDescX6* dd = (const HUF_DDescX6*)(DTable+1);
|
||||
const HUF_DSeqX6* ds = (const HUF_DSeqX6*)(DTable + 1 + (1<<(dtLog-1)));
|
||||
const HUF_DSeqX6* ds = (const HUF_DSeqX6*)(DTable + 1 + ((size_t)1<<(dtLog-1)));
|
||||
size_t errorCode;
|
||||
|
||||
/* Init */
|
||||
|
||||
@ -32,7 +32,8 @@
|
||||
- Source repository : https://github.com/Cyan4973/FiniteStateEntropy
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
****************************************************************** */
|
||||
#pragma once
|
||||
#ifndef HUFF0_H
|
||||
#define HUFF0_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
@ -91,3 +92,5 @@ size_t HUF_compress2 (void* dst, size_t dstSize, const void* src, size_t srcSize
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* HUFF0_H */
|
||||
|
||||
@ -32,7 +32,8 @@
|
||||
- Source repository : https://github.com/Cyan4973/FiniteStateEntropy
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
****************************************************************** */
|
||||
#pragma once
|
||||
#ifndef HUFF0_STATIC_H
|
||||
#define HUFF0_STATIC_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
@ -74,3 +75,5 @@ size_t HUF_decompress4X6 (void* dst, size_t dstSize, const void* cSrc, size_t cS
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* HUFF0_STATIC_H */
|
||||
|
||||
81
lib/legacy/zstd_legacy.h
Normal file
81
lib/legacy/zstd_legacy.h
Normal file
@ -0,0 +1,81 @@
|
||||
/*
|
||||
zstd_v02 - decoder for 0.2 format
|
||||
Header File
|
||||
Copyright (C) 2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#ifndef ZSTD_LEGACY_H
|
||||
#define ZSTD_LEGACY_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include "mem.h" /* MEM_STATIC */
|
||||
#include "error.h" /* ERROR */
|
||||
#include "zstd_v01.h"
|
||||
#include "zstd_v02.h"
|
||||
|
||||
MEM_STATIC unsigned ZSTD_isLegacy (U32 magicNumberLE)
|
||||
{
|
||||
switch(magicNumberLE)
|
||||
{
|
||||
case ZSTDv01_magicNumberLE :
|
||||
case ZSTDv02_magicNumber : return 1;
|
||||
default : return 0;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
MEM_STATIC size_t ZSTD_decompressLegacy(
|
||||
void* dst, size_t maxOriginalSize,
|
||||
const void* src, size_t compressedSize,
|
||||
U32 magicNumberLE)
|
||||
{
|
||||
switch(magicNumberLE)
|
||||
{
|
||||
case ZSTDv01_magicNumberLE :
|
||||
return ZSTDv01_decompress(dst, maxOriginalSize, src, compressedSize);
|
||||
case ZSTDv02_magicNumber :
|
||||
return ZSTDv02_decompress(dst, maxOriginalSize, src, compressedSize);
|
||||
default :
|
||||
return ERROR(prefix_unknown);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_LEGACY_H */
|
||||
@ -988,7 +988,7 @@ static size_t HUF_readDTable (U16* DTable, const void* src, size_t srcSize)
|
||||
{
|
||||
static int l[14] = { 1, 2, 3, 4, 7, 8, 15, 16, 31, 32, 63, 64, 127, 128 };
|
||||
oSize = l[iSize-242];
|
||||
memset(huffWeight, 1, oSize);
|
||||
memset(huffWeight, 1, sizeof(huffWeight));
|
||||
iSize = 0;
|
||||
}
|
||||
else /* Incompressible */
|
||||
|
||||
3729
lib/legacy/zstd_v02.c
Normal file
3729
lib/legacy/zstd_v02.c
Normal file
File diff suppressed because it is too large
Load Diff
99
lib/legacy/zstd_v02.h
Normal file
99
lib/legacy/zstd_v02.h
Normal file
@ -0,0 +1,99 @@
|
||||
/*
|
||||
zstd_v02 - decoder for 0.2 format
|
||||
Header File
|
||||
Copyright (C) 2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include <stddef.h> /* size_t */
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Simple one-step function
|
||||
***************************************/
|
||||
/**
|
||||
ZSTDv02_decompress() : decompress ZSTD frames compliant with v0.2.x format
|
||||
compressedSize : is the exact source size
|
||||
maxOriginalSize : is the size of the 'dst' buffer, which must be already allocated.
|
||||
It must be equal or larger than originalSize, otherwise decompression will fail.
|
||||
return : the number of bytes decompressed into destination buffer (originalSize)
|
||||
or an errorCode if it fails (which can be tested using ZSTDv01_isError())
|
||||
*/
|
||||
size_t ZSTDv02_decompress( void* dst, size_t maxOriginalSize,
|
||||
const void* src, size_t compressedSize);
|
||||
|
||||
/**
|
||||
ZSTDv02_isError() : tells if the result of ZSTDv02_decompress() is an error
|
||||
*/
|
||||
unsigned ZSTDv02_isError(size_t code);
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Advanced functions
|
||||
***************************************/
|
||||
typedef struct ZSTDv02_Dctx_s ZSTDv02_Dctx;
|
||||
ZSTDv02_Dctx* ZSTDv02_createDCtx(void);
|
||||
size_t ZSTDv02_freeDCtx(ZSTDv02_Dctx* dctx);
|
||||
|
||||
size_t ZSTDv02_decompressDCtx(void* ctx,
|
||||
void* dst, size_t maxOriginalSize,
|
||||
const void* src, size_t compressedSize);
|
||||
|
||||
/* *************************************
|
||||
* Streaming functions
|
||||
***************************************/
|
||||
size_t ZSTDv02_resetDCtx(ZSTDv02_Dctx* dctx);
|
||||
|
||||
size_t ZSTDv02_nextSrcSizeToDecompress(ZSTDv02_Dctx* dctx);
|
||||
size_t ZSTDv02_decompressContinue(ZSTDv02_Dctx* dctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
|
||||
/**
|
||||
Use above functions alternatively.
|
||||
ZSTD_nextSrcSizeToDecompress() tells how much bytes to provide as 'srcSize' to ZSTD_decompressContinue().
|
||||
ZSTD_decompressContinue() will use previous data blocks to improve compression if they are located prior to current block.
|
||||
Result is the number of bytes regenerated within 'dst'.
|
||||
It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some header.
|
||||
*/
|
||||
|
||||
/* *************************************
|
||||
* Prefix - version detection
|
||||
***************************************/
|
||||
#define ZSTDv02_magicNumber 0xFD2FB522 /* v0.2 */
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
335
lib/zstd.c
335
lib/zstd.c
@ -39,7 +39,7 @@
|
||||
* Increasing memory usage improves compression ratio
|
||||
* Reduced memory usage can improve speed, due to cache effect
|
||||
*/
|
||||
#define ZSTD_MEMORY_USAGE 17
|
||||
#define ZSTD_MEMORY_USAGE 16
|
||||
|
||||
/*!
|
||||
* HEAPMODE :
|
||||
@ -68,11 +68,12 @@
|
||||
#include <stdio.h> /* debug : printf */
|
||||
#include "mem.h" /* low level memory routines */
|
||||
#include "zstd_static.h"
|
||||
#include "zstd_internal.h"
|
||||
#include "fse_static.h"
|
||||
#include "huff0.h"
|
||||
|
||||
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT==1)
|
||||
# include "zstd_v01.h"
|
||||
# include "zstd_legacy.h"
|
||||
#endif
|
||||
|
||||
|
||||
@ -130,12 +131,6 @@ static const U32 g_searchStrength = 8;
|
||||
|
||||
#define WORKPLACESIZE (BLOCKSIZE*3)
|
||||
#define MINMATCH 4
|
||||
#define MLbits 7
|
||||
#define LLbits 6
|
||||
#define Offbits 5
|
||||
#define MaxML ((1<<MLbits )-1)
|
||||
#define MaxLL ((1<<LLbits )-1)
|
||||
#define MaxOff 31
|
||||
#define LitFSELog 11
|
||||
#define MLFSELog 10
|
||||
#define LLFSELog 10
|
||||
@ -153,49 +148,12 @@ static const size_t ZSTD_frameHeaderSize = 4;
|
||||
/* *******************************************************
|
||||
* Memory operations
|
||||
**********************************************************/
|
||||
static void ZSTD_copy4(void* dst, const void* src) { memcpy(dst, src, 4); }
|
||||
|
||||
static void ZSTD_copy8(void* dst, const void* src) { memcpy(dst, src, 8); }
|
||||
|
||||
#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
|
||||
|
||||
/*! ZSTD_wildcopy : custom version of memcpy(), can copy up to 7-8 bytes too many */
|
||||
static void ZSTD_wildcopy(void* dst, const void* src, size_t length)
|
||||
{
|
||||
const BYTE* ip = (const BYTE*)src;
|
||||
BYTE* op = (BYTE*)dst;
|
||||
BYTE* const oend = op + length;
|
||||
do COPY8(op, ip) while (op < oend);
|
||||
}
|
||||
static void ZSTD_copy4(void* dst, const void* src) { memcpy(dst, src, 4); }
|
||||
|
||||
|
||||
/* **************************************
|
||||
* Local structures
|
||||
****************************************/
|
||||
typedef enum { bt_compressed, bt_raw, bt_rle, bt_end } blockType_t;
|
||||
|
||||
typedef struct
|
||||
{
|
||||
blockType_t blockType;
|
||||
U32 origSize;
|
||||
} blockProperties_t;
|
||||
|
||||
typedef struct {
|
||||
void* buffer;
|
||||
U32* offsetStart;
|
||||
U32* offset;
|
||||
BYTE* offCodeStart;
|
||||
BYTE* offCode;
|
||||
BYTE* litStart;
|
||||
BYTE* lit;
|
||||
BYTE* litLengthStart;
|
||||
BYTE* litLength;
|
||||
BYTE* matchLengthStart;
|
||||
BYTE* matchLength;
|
||||
BYTE* dumpsStart;
|
||||
BYTE* dumps;
|
||||
} seqStore_t;
|
||||
|
||||
void ZSTD_resetSeqStore(seqStore_t* ssPtr)
|
||||
{
|
||||
ssPtr->offset = ssPtr->offsetStart;
|
||||
@ -205,7 +163,6 @@ void ZSTD_resetSeqStore(seqStore_t* ssPtr)
|
||||
ssPtr->dumps = ssPtr->dumpsStart;
|
||||
}
|
||||
|
||||
|
||||
struct ZSTD_CCtx_s
|
||||
{
|
||||
const BYTE* base;
|
||||
@ -217,21 +174,21 @@ struct ZSTD_CCtx_s
|
||||
#else
|
||||
U32 hashTable[HASH_TABLESIZE];
|
||||
#endif
|
||||
BYTE buffer[WORKPLACESIZE];
|
||||
BYTE buffer[WORKPLACESIZE];
|
||||
};
|
||||
|
||||
|
||||
void ZSTD_resetCCtx(ZSTD_CCtx* ctx)
|
||||
{
|
||||
ctx->base = NULL;
|
||||
ctx->seqStore.buffer = ctx->buffer;
|
||||
ctx->seqStore.buffer = ctx->buffer;
|
||||
ctx->seqStore.offsetStart = (U32*) (ctx->seqStore.buffer);
|
||||
ctx->seqStore.offCodeStart = (BYTE*) (ctx->seqStore.offsetStart + (BLOCKSIZE>>2));
|
||||
ctx->seqStore.litStart = ctx->seqStore.offCodeStart + (BLOCKSIZE>>2);
|
||||
ctx->seqStore.litLengthStart = ctx->seqStore.litStart + BLOCKSIZE;
|
||||
ctx->seqStore.matchLengthStart = ctx->seqStore.litLengthStart + (BLOCKSIZE>>2);
|
||||
ctx->seqStore.dumpsStart = ctx->seqStore.matchLengthStart + (BLOCKSIZE>>2);
|
||||
memset(ctx->hashTable, 0, HASH_TABLESIZE*4);
|
||||
memset(ctx->hashTable, 0, sizeof(ctx->hashTable));
|
||||
}
|
||||
|
||||
ZSTD_CCtx* ZSTD_createCCtx(void)
|
||||
@ -266,122 +223,6 @@ const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); }
|
||||
***************************************/
|
||||
unsigned ZSTD_versionNumber (void) { return ZSTD_VERSION_NUMBER; }
|
||||
|
||||
static unsigned ZSTD_highbit(U32 val)
|
||||
{
|
||||
# if defined(_MSC_VER) /* Visual */
|
||||
unsigned long r;
|
||||
_BitScanReverse(&r, val);
|
||||
return (unsigned)r;
|
||||
# elif defined(__GNUC__) && (GCC_VERSION >= 304) /* GCC Intrinsic */
|
||||
return 31 - __builtin_clz(val);
|
||||
# else /* Software version */
|
||||
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
int r;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Function body to include
|
||||
***************************************/
|
||||
#include "mem.h"
|
||||
static size_t ZSTD_read_ARCH(const void* p) { size_t r; memcpy(&r, p, sizeof(r)); return r; }
|
||||
|
||||
MEM_STATIC unsigned ZSTD_NbCommonBytes (register size_t val)
|
||||
{
|
||||
if (MEM_isLittleEndian())
|
||||
{
|
||||
if (MEM_64bits())
|
||||
{
|
||||
# if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
unsigned long r = 0;
|
||||
_BitScanForward64( &r, (U64)val );
|
||||
return (int)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
return (__builtin_ctzll((U64)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 };
|
||||
return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
|
||||
# endif
|
||||
}
|
||||
else /* 32 bits */
|
||||
{
|
||||
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
unsigned long r;
|
||||
_BitScanForward( &r, (U32)val );
|
||||
return (int)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
return (__builtin_ctz((U32)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 };
|
||||
return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
|
||||
# endif
|
||||
}
|
||||
}
|
||||
else /* Big Endian CPU */
|
||||
{
|
||||
if (MEM_32bits())
|
||||
{
|
||||
# if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse64( &r, val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
return (__builtin_clzll(val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
const unsigned n32 = sizeof(size_t)*4; /* calculate this way due to compiler complaining in 32-bits mode */
|
||||
if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
|
||||
if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
else /* 32 bits */
|
||||
{
|
||||
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse( &r, (unsigned long)val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) && !defined(LZ4_FORCE_SW_BITCOUNT)
|
||||
return (__builtin_clz((U32)val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit)
|
||||
{
|
||||
const BYTE* const pStart = pIn;
|
||||
|
||||
while ((pIn<pInLimit-(sizeof(size_t)-1)))
|
||||
{
|
||||
size_t diff = ZSTD_read_ARCH(pMatch) ^ ZSTD_read_ARCH(pIn);
|
||||
if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; }
|
||||
pIn += ZSTD_NbCommonBytes(diff);
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
|
||||
if (MEM_32bits()) if ((pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; }
|
||||
if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; }
|
||||
if ((pIn<pInLimit) && (*pMatch == *pIn)) pIn++;
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
|
||||
|
||||
/* *******************************************************
|
||||
* Compression
|
||||
@ -392,7 +233,7 @@ size_t ZSTD_compressBound(size_t srcSize) /* maximum compressed size */
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_noCompressBlock (void* dst, size_t maxDstSize, const void* src, size_t srcSize)
|
||||
size_t ZSTD_noCompressBlock (void* dst, size_t maxDstSize, const void* src, size_t srcSize)
|
||||
{
|
||||
BYTE* const ostart = (BYTE* const)dst;
|
||||
|
||||
@ -433,7 +274,7 @@ static size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t maxDstSize, const
|
||||
size_t ZSTD_minGain(size_t srcSize) { return (srcSize >> 6) + 1; }
|
||||
|
||||
static size_t ZSTD_compressLiterals (void* dst, size_t maxDstSize,
|
||||
const void* src, size_t srcSize)
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
const size_t minGain = ZSTD_minGain(srcSize);
|
||||
BYTE* const ostart = (BYTE*)dst;
|
||||
@ -461,9 +302,9 @@ static size_t ZSTD_compressLiterals (void* dst, size_t maxDstSize,
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize,
|
||||
const seqStore_t* seqStorePtr,
|
||||
size_t srcSize)
|
||||
size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize,
|
||||
const seqStore_t* seqStorePtr,
|
||||
size_t srcSize)
|
||||
{
|
||||
U32 count[MaxSeq+1];
|
||||
S16 norm[MaxSeq+1];
|
||||
@ -475,7 +316,6 @@ static size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize,
|
||||
U32 CTable_MatchLength[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML )];
|
||||
U32 LLtype, Offtype, MLtype; /* compressed, raw or rle */
|
||||
const BYTE* const op_lit_start = seqStorePtr->litStart;
|
||||
const BYTE* op_lit = seqStorePtr->lit;
|
||||
const BYTE* const llTable = seqStorePtr->litLengthStart;
|
||||
const BYTE* const llPtr = seqStorePtr->litLength;
|
||||
const BYTE* const mlTable = seqStorePtr->matchLengthStart;
|
||||
@ -492,7 +332,7 @@ static size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize,
|
||||
/* Compress literals */
|
||||
{
|
||||
size_t cSize;
|
||||
size_t litSize = op_lit - op_lit_start;
|
||||
size_t litSize = seqStorePtr->lit - op_lit_start;
|
||||
|
||||
if (litSize <= LITERAL_NOENTROPY)
|
||||
cSize = ZSTD_compressRawLiteralsBlock(op, maxDstSize, op_lit_start, litSize);
|
||||
@ -665,63 +505,25 @@ static size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize,
|
||||
}
|
||||
|
||||
|
||||
static void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, size_t offset, size_t matchLength)
|
||||
{
|
||||
BYTE* op_lit = seqStorePtr->lit;
|
||||
BYTE* const l_end = op_lit + litLength;
|
||||
|
||||
/* copy Literals */
|
||||
while (op_lit<l_end) COPY8(op_lit, literals);
|
||||
seqStorePtr->lit += litLength;
|
||||
|
||||
/* literal Length */
|
||||
if (litLength >= MaxLL)
|
||||
{
|
||||
*(seqStorePtr->litLength++) = MaxLL;
|
||||
if (litLength<255 + MaxLL)
|
||||
*(seqStorePtr->dumps++) = (BYTE)(litLength - MaxLL);
|
||||
else
|
||||
{
|
||||
*(seqStorePtr->dumps++) = 255;
|
||||
MEM_writeLE32(seqStorePtr->dumps, (U32)litLength); seqStorePtr->dumps += 3;
|
||||
}
|
||||
}
|
||||
else *(seqStorePtr->litLength++) = (BYTE)litLength;
|
||||
|
||||
/* match offset */
|
||||
*(seqStorePtr->offset++) = (U32)offset;
|
||||
|
||||
/* match Length */
|
||||
if (matchLength >= MaxML)
|
||||
{
|
||||
*(seqStorePtr->matchLength++) = MaxML;
|
||||
if (matchLength < 255+MaxML)
|
||||
*(seqStorePtr->dumps++) = (BYTE)(matchLength - MaxML);
|
||||
else
|
||||
{
|
||||
*(seqStorePtr->dumps++) = 255;
|
||||
MEM_writeLE32(seqStorePtr->dumps, (U32)matchLength); seqStorePtr->dumps+=3;
|
||||
}
|
||||
}
|
||||
else *(seqStorePtr->matchLength++) = (BYTE)matchLength;
|
||||
}
|
||||
|
||||
|
||||
//static const U32 hashMask = (1<<HASH_LOG)-1;
|
||||
//static const U64 prime5bytes = 889523592379ULL;
|
||||
//static const U64 prime6bytes = 227718039650203ULL;
|
||||
static const U64 prime7bytes = 58295818150454627ULL;
|
||||
//static const U64 prime8bytes = 14923729446516375013ULL;
|
||||
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return (U32) _bextr_u64(*(U64*)p * prime7bytes, (56-HASH_LOG), HASH_LOG); }
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U64*)p * prime7bytes) << 8 >> (64-HASH_LOG)); }
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U64*)p * prime7bytes) >> (56-HASH_LOG)) & ((1<<HASH_LOG)-1); }
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( ((*(U64*)p & 0xFFFFFFFFFFFFFF) * prime7bytes) >> (64-HASH_LOG)); }
|
||||
|
||||
//static const U64 prime8bytes = 14923729446516375013ULL;
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U64*)p * prime8bytes) >> (64-HASH_LOG)); }
|
||||
|
||||
static const U64 prime7bytes = 58295818150454627ULL;
|
||||
static U32 ZSTD_hashPtr(const void* p) { return ( (MEM_read64(p) * prime7bytes) >> (56-HASH_LOG)) & HASH_MASK; }
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U64*)p * prime6bytes) >> (48-HASH_LOG)) & HASH_MASK; }
|
||||
|
||||
//static const U64 prime6bytes = 227718039650203ULL;
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (MEM_read64(p) * prime6bytes) >> (48-HASH_LOG)) & HASH_MASK; }
|
||||
|
||||
//static const U64 prime5bytes = 889523592379ULL;
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U64*)p * prime5bytes) >> (40-HASH_LOG)) & HASH_MASK; }
|
||||
|
||||
//static U32 ZSTD_hashPtr(const void* p) { return ( (*(U32*)p * KNUTH) >> (32-HASH_LOG)); }
|
||||
|
||||
static void ZSTD_addPtr(U32* table, const BYTE* p, const BYTE* start) { table[ZSTD_hashPtr(p)] = (U32)(p-start); }
|
||||
@ -731,7 +533,7 @@ static const BYTE* ZSTD_updateMatch(U32* table, const BYTE* p, const BYTE* start
|
||||
U32 h = ZSTD_hashPtr(p);
|
||||
const BYTE* r;
|
||||
r = table[h] + start;
|
||||
ZSTD_addPtr(table, p, start);
|
||||
table[h] = (U32)(p-start);
|
||||
return r;
|
||||
}
|
||||
|
||||
@ -751,39 +553,45 @@ static size_t ZSTD_compressBlock(ZSTD_CCtx* ctx, void* dst, size_t maxDstSize, c
|
||||
const BYTE* ip = istart + 1;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 16;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
|
||||
size_t prevOffset=0, offset=0;
|
||||
size_t offset_2=4, offset_1=4;
|
||||
|
||||
|
||||
/* init */
|
||||
if (ip-base < 4)
|
||||
{
|
||||
ZSTD_addPtr(HashTable, ip+0, base);
|
||||
ZSTD_addPtr(HashTable, ip+1, base);
|
||||
ZSTD_addPtr(HashTable, ip+2, base);
|
||||
ZSTD_addPtr(HashTable, ip+3, base);
|
||||
ip += 4;
|
||||
}
|
||||
ZSTD_resetSeqStore(seqStorePtr);
|
||||
|
||||
/* Main Search Loop */
|
||||
while (ip < ilimit)
|
||||
while (ip <= ilimit)
|
||||
{
|
||||
const BYTE* match = (const BYTE*) ZSTD_updateMatch(HashTable, ip, base);
|
||||
const BYTE* match = ZSTD_updateMatch(HashTable, ip, base);
|
||||
|
||||
if (!ZSTD_checkMatch(match,ip)) { ip += ((ip-anchor) >> g_searchStrength) + 1; continue; }
|
||||
/* catch up */
|
||||
while ((ip>anchor) && (match>base) && (ip[-1] == match[-1])) { ip--; match--; }
|
||||
if (ZSTD_checkMatch(ip-offset_2,ip)) match = ip-offset_2;
|
||||
if (!ZSTD_checkMatch(match,ip)) { ip += ((ip-anchor) >> g_searchStrength) + 1; offset_2 = offset_1; continue; }
|
||||
while ((ip>anchor) && (match>base) && (ip[-1] == match[-1])) { ip--; match--; } /* catch up */
|
||||
|
||||
{
|
||||
size_t litLength = ip-anchor;
|
||||
size_t matchLength = ZSTD_count(ip+MINMATCH, match+MINMATCH, iend);
|
||||
size_t offsetCode;
|
||||
if (litLength) prevOffset = offset;
|
||||
offsetCode = ip-match;
|
||||
if (offsetCode == prevOffset) offsetCode = 0;
|
||||
prevOffset = offset;
|
||||
offset = ip-match;
|
||||
size_t offsetCode = ip-match;
|
||||
if (offsetCode == offset_2) offsetCode = 0;
|
||||
offset_2 = offset_1;
|
||||
offset_1 = ip-match;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offsetCode, matchLength);
|
||||
|
||||
/* Fill Table */
|
||||
ZSTD_addPtr(HashTable, ip+1, base);
|
||||
ip += matchLength + MINMATCH;
|
||||
if (ip<=iend-8) ZSTD_addPtr(HashTable, ip-2, base);
|
||||
anchor = ip;
|
||||
if (ip <= ilimit) ZSTD_addPtr(HashTable, ip-2, base);
|
||||
}
|
||||
}
|
||||
|
||||
@ -1085,23 +893,23 @@ static size_t ZSTD_decompressLiterals(void* dst, size_t* maxDstSizePtr,
|
||||
/** ZSTD_decodeLiteralsBlock
|
||||
@return : nb of bytes read from src (< srcSize )*/
|
||||
size_t ZSTD_decodeLiteralsBlock(void* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
|
||||
{
|
||||
ZSTD_DCtx* dctx = (ZSTD_DCtx*)ctx;
|
||||
const BYTE* const istart = (const BYTE* const)src;
|
||||
const BYTE* const istart = (const BYTE*) src;
|
||||
|
||||
/* any compressed block with literals segment must be at least this size */
|
||||
if (srcSize < MIN_CBLOCK_SIZE) return ERROR(corruption_detected);
|
||||
|
||||
switch(*istart & 3)
|
||||
{
|
||||
default:
|
||||
/* compressed */
|
||||
case 0:
|
||||
{
|
||||
size_t litSize = BLOCKSIZE;
|
||||
const size_t readSize = ZSTD_decompressLiterals(dctx->litBuffer, &litSize, src, srcSize);
|
||||
dctx->litPtr = dctx->litBuffer;
|
||||
dctx->litBufSize = BLOCKSIZE;
|
||||
dctx->litBufSize = BLOCKSIZE+8;
|
||||
dctx->litSize = litSize;
|
||||
return readSize; /* works if it's an error too */
|
||||
}
|
||||
@ -1110,28 +918,29 @@ size_t ZSTD_decodeLiteralsBlock(void* ctx,
|
||||
const size_t litSize = (MEM_readLE32(istart) & 0xFFFFFF) >> 2; /* no buffer issue : srcSize >= MIN_CBLOCK_SIZE */
|
||||
if (litSize > srcSize-11) /* risk of reading too far with wildcopy */
|
||||
{
|
||||
if (litSize > srcSize-3) return ERROR(corruption_detected);
|
||||
memcpy(dctx->litBuffer, istart, litSize);
|
||||
dctx->litBufSize = BLOCKSIZE;
|
||||
dctx->litSize = litSize;
|
||||
return litSize+3;
|
||||
}
|
||||
/* direct reference into compressed stream */
|
||||
if (litSize > srcSize-3) return ERROR(corruption_detected);
|
||||
memcpy(dctx->litBuffer, istart, litSize);
|
||||
dctx->litBufSize = BLOCKSIZE+8;
|
||||
dctx->litSize = litSize;
|
||||
return litSize+3;
|
||||
}
|
||||
/* direct reference into compressed stream */
|
||||
dctx->litPtr = istart+3;
|
||||
dctx->litBufSize = srcSize-3;
|
||||
dctx->litSize = litSize;
|
||||
return litSize+3;
|
||||
}
|
||||
return litSize+3; }
|
||||
case IS_RLE:
|
||||
{
|
||||
const size_t litSize = (MEM_readLE32(istart) & 0xFFFFFF) >> 2; /* no buffer issue : srcSize >= MIN_CBLOCK_SIZE */
|
||||
if (litSize > BLOCKSIZE) return ERROR(corruption_detected);
|
||||
memset(dctx->litBuffer, istart[3], litSize);
|
||||
dctx->litPtr = dctx->litBuffer;
|
||||
dctx->litBufSize = BLOCKSIZE;
|
||||
dctx->litBufSize = BLOCKSIZE+8;
|
||||
dctx->litSize = litSize;
|
||||
return 4;
|
||||
}
|
||||
default:
|
||||
return ERROR(corruption_detected); /* forbidden nominal case */
|
||||
}
|
||||
}
|
||||
|
||||
@ -1286,7 +1095,7 @@ static void ZSTD_decodeSequence(seq_t* seq, seqState_t* seqState)
|
||||
|
||||
/* Offset */
|
||||
{
|
||||
static const size_t offsetPrefix[MaxOff+1] = { /* note : size_t faster than U32 */
|
||||
static const U32 offsetPrefix[MaxOff+1] = {
|
||||
1 /*fake*/, 1, 2, 4, 8, 16, 32, 64, 128, 256,
|
||||
512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144,
|
||||
524288, 1048576, 2097152, 4194304, 8388608, 16777216, 33554432, /*fake*/ 1, 1, 1, 1, 1 };
|
||||
@ -1325,7 +1134,7 @@ static void ZSTD_decodeSequence(seq_t* seq, seqState_t* seqState)
|
||||
|
||||
static size_t ZSTD_execSequence(BYTE* op,
|
||||
seq_t sequence,
|
||||
const BYTE** litPtr, const BYTE* const litLimit,
|
||||
const BYTE** litPtr, const BYTE* const litLimit_8,
|
||||
BYTE* const base, BYTE* const oend)
|
||||
{
|
||||
static const int dec32table[] = {0, 1, 2, 1, 4, 4, 4, 4}; /* added */
|
||||
@ -1336,10 +1145,10 @@ static size_t ZSTD_execSequence(BYTE* op,
|
||||
BYTE* const oend_8 = oend-8;
|
||||
const BYTE* const litEnd = *litPtr + sequence.litLength;
|
||||
|
||||
/* checks */
|
||||
/* check */
|
||||
if (oLitEnd > oend_8) return ERROR(dstSize_tooSmall); /* last match must start at a minimum distance of 8 from oend */
|
||||
if (oMatchEnd > oend) return ERROR(dstSize_tooSmall); /* overwrite beyond dst buffer */
|
||||
if (litEnd > litLimit-8) return ERROR(corruption_detected); /* overRead beyond lit buffer */
|
||||
if (litEnd > litLimit_8) return ERROR(corruption_detected); /* risk read beyond lit buffer */
|
||||
|
||||
/* copy Literals */
|
||||
ZSTD_wildcopy(op, *litPtr, sequence.litLength); /* note : oLitEnd <= oend-8 : no risk of overwrite beyond oend */
|
||||
@ -1351,8 +1160,8 @@ static size_t ZSTD_execSequence(BYTE* op,
|
||||
const BYTE* match = op - sequence.offset;
|
||||
|
||||
/* check */
|
||||
if (sequence.offset > (size_t)op) return ERROR(corruption_detected); /* address space overflow test (this test seems kept by clang optimizer) */
|
||||
//if (match > op) return ERROR(corruption_detected); /* address space overflow test (is clang optimizer removing this test ?) */
|
||||
if (sequence.offset > (size_t)op) return ERROR(corruption_detected); /* address space overflow test (this test seems kept by clang optimizer) */
|
||||
if (match < base) return ERROR(corruption_detected);
|
||||
|
||||
/* close range match, overlap */
|
||||
@ -1405,7 +1214,7 @@ static size_t ZSTD_decompressSequences(
|
||||
BYTE* const oend = ostart + maxDstSize;
|
||||
size_t errorCode, dumpsLength;
|
||||
const BYTE* litPtr = dctx->litPtr;
|
||||
const BYTE* const litMax = litPtr + dctx->litBufSize;
|
||||
const BYTE* const litLimit_8 = litPtr + dctx->litBufSize - 8;
|
||||
const BYTE* const litEnd = litPtr + dctx->litSize;
|
||||
int nbSeq;
|
||||
const BYTE* dumps;
|
||||
@ -1427,9 +1236,10 @@ static size_t ZSTD_decompressSequences(
|
||||
seqState_t seqState;
|
||||
|
||||
memset(&sequence, 0, sizeof(sequence));
|
||||
sequence.offset = 4;
|
||||
seqState.dumps = dumps;
|
||||
seqState.dumpsEnd = dumps + dumpsLength;
|
||||
seqState.prevOffset = 1;
|
||||
seqState.prevOffset = 4;
|
||||
errorCode = BIT_initDStream(&(seqState.DStream), ip, iend-ip);
|
||||
if (ERR_isError(errorCode)) return ERROR(corruption_detected);
|
||||
FSE_initDState(&(seqState.stateLL), &(seqState.DStream), DTableLL);
|
||||
@ -1441,7 +1251,7 @@ static size_t ZSTD_decompressSequences(
|
||||
size_t oneSeqSize;
|
||||
nbSeq--;
|
||||
ZSTD_decodeSequence(&sequence, &seqState);
|
||||
oneSeqSize = ZSTD_execSequence(op, sequence, &litPtr, litMax, base, oend);
|
||||
oneSeqSize = ZSTD_execSequence(op, sequence, &litPtr, litLimit_8, base, oend);
|
||||
if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
|
||||
op += oneSeqSize;
|
||||
}
|
||||
@ -1455,7 +1265,7 @@ static size_t ZSTD_decompressSequences(
|
||||
size_t lastLLSize = litEnd - litPtr;
|
||||
if (litPtr > litEnd) return ERROR(corruption_detected);
|
||||
if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
|
||||
if (op != litPtr) memmove(op, litPtr, lastLLSize);
|
||||
if (op != litPtr) memcpy(op, litPtr, lastLLSize);
|
||||
op += lastLLSize;
|
||||
}
|
||||
}
|
||||
@ -1497,8 +1307,9 @@ static size_t ZSTD_decompressDCtx(void* ctx, void* dst, size_t maxDstSize, const
|
||||
if (srcSize < ZSTD_frameHeaderSize+ZSTD_blockHeaderSize) return ERROR(srcSize_wrong);
|
||||
magicNumber = MEM_readLE32(src);
|
||||
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT==1)
|
||||
if (magicNumber == ZSTDv01_magicNumberLE) return ZSTDv01_decompressDCtx(ctx, dst, maxDstSize, src, srcSize);
|
||||
#endif /* ZSTD_LEGACY_SUPPORT */
|
||||
if (ZSTD_isLegacy(magicNumber))
|
||||
return ZSTD_decompressLegacy(dst, maxDstSize, src, srcSize, magicNumber);
|
||||
#endif
|
||||
if (magicNumber != ZSTD_magicNumber) return ERROR(prefix_unknown);
|
||||
ip += ZSTD_frameHeaderSize; remainingSize -= ZSTD_frameHeaderSize;
|
||||
|
||||
@ -1550,9 +1361,9 @@ size_t ZSTD_decompress(void* dst, size_t maxDstSize, const void* src, size_t src
|
||||
}
|
||||
|
||||
|
||||
/*******************************
|
||||
/* ******************************
|
||||
* Streaming Decompression API
|
||||
*******************************/
|
||||
********************************/
|
||||
|
||||
size_t ZSTD_resetDCtx(ZSTD_DCtx* dctx)
|
||||
{
|
||||
|
||||
@ -30,7 +30,8 @@
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#pragma once
|
||||
#ifndef ZSTD_H
|
||||
#define ZSTD_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
@ -46,8 +47,8 @@ extern "C" {
|
||||
* Version
|
||||
***************************************/
|
||||
#define ZSTD_VERSION_MAJOR 0 /* for breaking interface changes */
|
||||
#define ZSTD_VERSION_MINOR 2 /* for new (non-breaking) interface capabilities */
|
||||
#define ZSTD_VERSION_RELEASE 2 /* for tweaks, bug-fixes, or development */
|
||||
#define ZSTD_VERSION_MINOR 3 /* for new (non-breaking) interface capabilities */
|
||||
#define ZSTD_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */
|
||||
#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
|
||||
unsigned ZSTD_versionNumber (void);
|
||||
|
||||
@ -105,3 +106,5 @@ size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx, void* dst, size_t maxDstSize, const voi
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_H */
|
||||
|
||||
265
lib/zstd_internal.h
Normal file
265
lib/zstd_internal.h
Normal file
@ -0,0 +1,265 @@
|
||||
/*
|
||||
zstd_internal - common functions to include
|
||||
Header File for include
|
||||
Copyright (C) 2014-2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#ifndef ZSTD_CCOMMON_H_MODULE
|
||||
#define ZSTD_CCOMMON_H_MODULE
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include "mem.h"
|
||||
#include "error.h"
|
||||
|
||||
|
||||
/* **************************************
|
||||
* Function body to include for inlining
|
||||
****************************************/
|
||||
static size_t ZSTD_read_ARCH(const void* p) { size_t r; memcpy(&r, p, sizeof(r)); return r; }
|
||||
|
||||
static unsigned ZSTD_highbit(U32 val)
|
||||
{
|
||||
# if defined(_MSC_VER) /* Visual */
|
||||
unsigned long r=0;
|
||||
_BitScanReverse(&r, val);
|
||||
return (unsigned)r;
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */
|
||||
return 31 - __builtin_clz(val);
|
||||
# else /* Software version */
|
||||
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
int r;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
|
||||
MEM_STATIC unsigned ZSTD_NbCommonBytes (register size_t val)
|
||||
{
|
||||
if (MEM_isLittleEndian())
|
||||
{
|
||||
if (MEM_64bits())
|
||||
{
|
||||
# if defined(_MSC_VER) && defined(_WIN64)
|
||||
unsigned long r = 0;
|
||||
_BitScanForward64( &r, (U64)val );
|
||||
return (int)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_ctzll((U64)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 };
|
||||
return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
|
||||
# endif
|
||||
}
|
||||
else /* 32 bits */
|
||||
{
|
||||
# if defined(_MSC_VER)
|
||||
unsigned long r=0;
|
||||
_BitScanForward( &r, (U32)val );
|
||||
return (int)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_ctz((U32)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 };
|
||||
return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
|
||||
# endif
|
||||
}
|
||||
}
|
||||
else /* Big Endian CPU */
|
||||
{
|
||||
if (MEM_32bits())
|
||||
{
|
||||
# if defined(_MSC_VER) && defined(_WIN64)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse64( &r, val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_clzll(val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
const unsigned n32 = sizeof(size_t)*4; /* calculate this way due to compiler complaining in 32-bits mode */
|
||||
if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
|
||||
if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
else /* 32 bits */
|
||||
{
|
||||
# if defined(_MSC_VER)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse( &r, (unsigned long)val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_clz((U32)val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit)
|
||||
{
|
||||
const BYTE* const pStart = pIn;
|
||||
|
||||
while ((pIn<pInLimit-(sizeof(size_t)-1)))
|
||||
{
|
||||
size_t diff = ZSTD_read_ARCH(pMatch) ^ ZSTD_read_ARCH(pIn);
|
||||
if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; }
|
||||
pIn += ZSTD_NbCommonBytes(diff);
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
|
||||
if (MEM_32bits()) if ((pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; }
|
||||
if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; }
|
||||
if ((pIn<pInLimit) && (*pMatch == *pIn)) pIn++;
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
|
||||
|
||||
static void ZSTD_copy8(void* dst, const void* src) { memcpy(dst, src, 8); }
|
||||
|
||||
#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
|
||||
|
||||
/*! ZSTD_wildcopy : custom version of memcpy(), can copy up to 7-8 bytes too many */
|
||||
static void ZSTD_wildcopy(void* dst, const void* src, size_t length)
|
||||
{
|
||||
const BYTE* ip = (const BYTE*)src;
|
||||
BYTE* op = (BYTE*)dst;
|
||||
BYTE* const oend = op + length;
|
||||
do COPY8(op, ip) while (op < oend);
|
||||
}
|
||||
|
||||
|
||||
typedef enum { bt_compressed, bt_raw, bt_rle, bt_end } blockType_t;
|
||||
|
||||
typedef struct
|
||||
{
|
||||
blockType_t blockType;
|
||||
U32 origSize;
|
||||
} blockProperties_t;
|
||||
|
||||
size_t ZSTD_noCompressBlock(void* op, size_t maxDstSize, const void* ip, size_t blockSize);
|
||||
|
||||
|
||||
typedef struct {
|
||||
void* buffer;
|
||||
U32* offsetStart;
|
||||
U32* offset;
|
||||
BYTE* offCodeStart;
|
||||
BYTE* offCode;
|
||||
BYTE* litStart;
|
||||
BYTE* lit;
|
||||
BYTE* litLengthStart;
|
||||
BYTE* litLength;
|
||||
BYTE* matchLengthStart;
|
||||
BYTE* matchLength;
|
||||
BYTE* dumpsStart;
|
||||
BYTE* dumps;
|
||||
} seqStore_t;
|
||||
|
||||
void ZSTD_resetSeqStore(seqStore_t* ssPtr);
|
||||
|
||||
#define REPCODE_STARTVALUE 4
|
||||
#define MLbits 7
|
||||
#define LLbits 6
|
||||
#define Offbits 5
|
||||
#define MaxML ((1<<MLbits) - 1)
|
||||
#define MaxLL ((1<<LLbits) - 1)
|
||||
#define MaxOff 31
|
||||
|
||||
/** ZSTD_storeSeq
|
||||
Store a sequence (literal length, literals, offset code and match length) into seqStore_t
|
||||
@offsetCode : distance to match, or 0 == repCode
|
||||
@matchCode : matchLength - MINMATCH
|
||||
*/
|
||||
MEM_STATIC void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, size_t offsetCode, size_t matchCode)
|
||||
{
|
||||
/* copy Literals */
|
||||
ZSTD_wildcopy(seqStorePtr->lit, literals, litLength);
|
||||
seqStorePtr->lit += litLength;
|
||||
|
||||
/* literal Length */
|
||||
if (litLength >= MaxLL)
|
||||
{
|
||||
*(seqStorePtr->litLength++) = MaxLL;
|
||||
if (litLength<255 + MaxLL)
|
||||
*(seqStorePtr->dumps++) = (BYTE)(litLength - MaxLL);
|
||||
else
|
||||
{
|
||||
*(seqStorePtr->dumps++) = 255;
|
||||
MEM_writeLE32(seqStorePtr->dumps, (U32)litLength); seqStorePtr->dumps += 3;
|
||||
}
|
||||
}
|
||||
else *(seqStorePtr->litLength++) = (BYTE)litLength;
|
||||
|
||||
/* match offset */
|
||||
*(seqStorePtr->offset++) = (U32)offsetCode;
|
||||
|
||||
/* match Length */
|
||||
if (matchCode >= MaxML)
|
||||
{
|
||||
*(seqStorePtr->matchLength++) = MaxML;
|
||||
if (matchCode < 255+MaxML)
|
||||
*(seqStorePtr->dumps++) = (BYTE)(matchCode - MaxML);
|
||||
else
|
||||
{
|
||||
*(seqStorePtr->dumps++) = 255;
|
||||
MEM_writeLE32(seqStorePtr->dumps, (U32)matchCode); seqStorePtr->dumps += 3;
|
||||
}
|
||||
}
|
||||
else *(seqStorePtr->matchLength++) = (BYTE)matchCode;
|
||||
}
|
||||
|
||||
|
||||
/* prototype, body into zstd.c */
|
||||
size_t ZSTD_compressSequences(BYTE* dst, size_t maxDstSize, const seqStore_t* seqStorePtr, size_t srcSize);
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_CCOMMON_H_MODULE */
|
||||
@ -30,7 +30,8 @@
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#pragma once
|
||||
#ifndef ZSTD_STATIC_H
|
||||
#define ZSTD_STATIC_H
|
||||
|
||||
/* The objects defined into this file should be considered experimental.
|
||||
* They are not labelled stable, as their prototype may change in the future.
|
||||
@ -73,7 +74,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, co
|
||||
/* *************************************
|
||||
* Prefix - version detection
|
||||
***************************************/
|
||||
#define ZSTD_magicNumber 0xFD2FB522 /* v0.2 (current)*/
|
||||
#define ZSTD_magicNumber 0xFD2FB523 /* v0.3 (current)*/
|
||||
|
||||
|
||||
/* *************************************
|
||||
@ -85,3 +86,5 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize, co
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_STATIC_H */
|
||||
|
||||
535
lib/zstdhc.c
Normal file
535
lib/zstdhc.c
Normal file
@ -0,0 +1,535 @@
|
||||
/*
|
||||
ZSTD HC - High Compression Mode of Zstandard
|
||||
Copyright (C) 2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- Zstd source repository : https://www.zstd.net
|
||||
*/
|
||||
|
||||
|
||||
/* *******************************************************
|
||||
* Compiler specifics
|
||||
*********************************************************/
|
||||
#ifdef _MSC_VER /* Visual Studio */
|
||||
# define FORCE_INLINE static __forceinline
|
||||
# include <intrin.h> /* For Visual 2005 */
|
||||
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
|
||||
# pragma warning(disable : 4324) /* disable: C4324: padded structure */
|
||||
#else
|
||||
# define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
|
||||
# ifdef __GNUC__
|
||||
# define FORCE_INLINE static inline __attribute__((always_inline))
|
||||
# else
|
||||
# define FORCE_INLINE static inline
|
||||
# endif
|
||||
#endif
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include <stdlib.h> /* malloc */
|
||||
#include <string.h> /* memset */
|
||||
#include "zstdhc_static.h"
|
||||
#include "zstd_static.h"
|
||||
#include "zstd_internal.h"
|
||||
#include "mem.h"
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Local Constants
|
||||
***************************************/
|
||||
#define MINMATCH 4
|
||||
#define MAXD_LOG 26
|
||||
|
||||
#define KB *1024
|
||||
#define MB *1024*1024
|
||||
#define GB *(1ULL << 30)
|
||||
|
||||
/* *************************************
|
||||
* Local Types
|
||||
***************************************/
|
||||
#define BLOCKSIZE (128 KB) /* define, for static allocation */
|
||||
#define WORKPLACESIZE (BLOCKSIZE*3)
|
||||
|
||||
struct ZSTD_HC_CCtx_s
|
||||
{
|
||||
const BYTE* end; /* next block here to continue on current prefix */
|
||||
const BYTE* base; /* All regular indexes relative to this position */
|
||||
const BYTE* dictBase; /* extDict indexes relative to this position */
|
||||
U32 dictLimit; /* below that point, need extDict */
|
||||
U32 lowLimit; /* below that point, no more data */
|
||||
U32 nextToUpdate; /* index from which to continue dictionary update */
|
||||
ZSTD_HC_parameters params;
|
||||
void* workSpace;
|
||||
size_t workSpaceSize;
|
||||
|
||||
seqStore_t seqStore; /* sequences storage ptrs */
|
||||
U32* hashTable;
|
||||
U32* chainTable;
|
||||
};
|
||||
|
||||
|
||||
ZSTD_HC_CCtx* ZSTD_HC_createCCtx(void)
|
||||
{
|
||||
return (ZSTD_HC_CCtx*) calloc(1, sizeof(ZSTD_HC_CCtx));
|
||||
}
|
||||
|
||||
size_t ZSTD_HC_freeCCtx(ZSTD_HC_CCtx* cctx)
|
||||
{
|
||||
free(cctx->workSpace);
|
||||
free(cctx);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static size_t ZSTD_HC_resetCCtx_advanced (ZSTD_HC_CCtx* zc,
|
||||
ZSTD_HC_parameters params)
|
||||
{
|
||||
/* validate params */
|
||||
if (params.windowLog > ZSTD_HC_WINDOWLOG_MAX) params.windowLog = ZSTD_HC_WINDOWLOG_MAX;
|
||||
if (params.windowLog < ZSTD_HC_WINDOWLOG_MIN) params.windowLog = ZSTD_HC_WINDOWLOG_MIN;
|
||||
if (params.chainLog > params.windowLog) params.chainLog = params.windowLog; /* <= ZSTD_HC_CHAINLOG_MAX */
|
||||
if (params.chainLog < ZSTD_HC_CHAINLOG_MIN) params.chainLog = ZSTD_HC_CHAINLOG_MIN;
|
||||
if (params.hashLog > ZSTD_HC_HASHLOG_MAX) params.hashLog = ZSTD_HC_HASHLOG_MAX;
|
||||
if (params.hashLog < ZSTD_HC_HASHLOG_MIN) params.hashLog = ZSTD_HC_HASHLOG_MIN;
|
||||
if (params.searchLog > ZSTD_HC_SEARCHLOG_MAX) params.searchLog = ZSTD_HC_SEARCHLOG_MAX;
|
||||
if (params.searchLog < ZSTD_HC_SEARCHLOG_MIN) params.searchLog = ZSTD_HC_SEARCHLOG_MIN;
|
||||
if (params.searchLength> ZSTD_HC_SEARCHLENGTH_MAX) params.searchLength = ZSTD_HC_SEARCHLENGTH_MAX;
|
||||
if (params.searchLength< ZSTD_HC_SEARCHLENGTH_MIN) params.searchLength = ZSTD_HC_SEARCHLENGTH_MIN;
|
||||
|
||||
/* reserve table memory */
|
||||
{
|
||||
const size_t tableSpace = ((1 << params.chainLog) + (1 << params.hashLog)) * sizeof(U32);
|
||||
const size_t neededSpace = tableSpace + WORKPLACESIZE;
|
||||
if (zc->workSpaceSize < neededSpace)
|
||||
{
|
||||
free(zc->workSpace);
|
||||
zc->workSpaceSize = neededSpace;
|
||||
zc->workSpace = malloc(neededSpace);
|
||||
if (zc->workSpace == NULL) return ERROR(memory_allocation);
|
||||
}
|
||||
zc->hashTable = (U32*)zc->workSpace;
|
||||
zc->chainTable = zc->hashTable + ((size_t)1 << params.hashLog);
|
||||
zc->seqStore.buffer = (void*) (zc->chainTable + ((size_t)1 << params.chainLog));
|
||||
memset(zc->hashTable, 0, tableSpace );
|
||||
}
|
||||
|
||||
zc->nextToUpdate = 0;
|
||||
zc->end = NULL;
|
||||
zc->base = NULL;
|
||||
zc->dictBase = NULL;
|
||||
zc->dictLimit = 0;
|
||||
zc->lowLimit = 0;
|
||||
zc->params = params;
|
||||
zc->seqStore.offsetStart = (U32*) (zc->seqStore.buffer);
|
||||
zc->seqStore.offCodeStart = (BYTE*) (zc->seqStore.offsetStart + (BLOCKSIZE>>2));
|
||||
zc->seqStore.litStart = zc->seqStore.offCodeStart + (BLOCKSIZE>>2);
|
||||
zc->seqStore.litLengthStart = zc->seqStore.litStart + BLOCKSIZE;
|
||||
zc->seqStore.matchLengthStart = zc->seqStore.litLengthStart + (BLOCKSIZE>>2);
|
||||
zc->seqStore.dumpsStart = zc->seqStore.matchLengthStart + (BLOCKSIZE>>2);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Inline functions and Macros
|
||||
***************************************/
|
||||
|
||||
static const U32 prime4bytes = 2654435761U;
|
||||
static U32 ZSTD_HC_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32-h) ; }
|
||||
static size_t ZSTD_HC_hash4Ptr(const void* ptr, U32 h) { return ZSTD_HC_hash4(MEM_read32(ptr), h); }
|
||||
|
||||
static const U64 prime5bytes = 889523592379ULL;
|
||||
static size_t ZSTD_HC_hash5(U64 u, U32 h) { return (size_t)((u * prime5bytes) << (64-40) >> (64-h)) ; }
|
||||
static size_t ZSTD_HC_hash5Ptr(const void* p, U32 h) { return ZSTD_HC_hash5(MEM_read64(p), h); }
|
||||
|
||||
static const U64 prime6bytes = 227718039650203ULL;
|
||||
static size_t ZSTD_HC_hash6(U64 u, U32 h) { return (size_t)((u * prime6bytes) << (64-48) >> (64-h)) ; }
|
||||
static size_t ZSTD_HC_hash6Ptr(const void* p, U32 h) { return ZSTD_HC_hash6(MEM_read64(p), h); }
|
||||
|
||||
static size_t ZSTD_HC_hashPtr(const void* p, U32 h, U32 mls)
|
||||
{
|
||||
switch(mls)
|
||||
{
|
||||
default:
|
||||
case 4: return ZSTD_HC_hash4Ptr(p,h);
|
||||
case 5: return ZSTD_HC_hash5Ptr(p,h);
|
||||
case 6: return ZSTD_HC_hash6Ptr(p,h);
|
||||
}
|
||||
}
|
||||
|
||||
#define NEXT_IN_CHAIN(d) chainTable[(d) & chainMask] /* flexible, CHAINSIZE dependent */
|
||||
|
||||
|
||||
/* *************************************
|
||||
* HC Compression
|
||||
***************************************/
|
||||
/* Update chains up to ip (excluded) */
|
||||
static void ZSTD_HC_insert (ZSTD_HC_CCtx* zc, const BYTE* ip, U32 mls)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
const U32 hashLog = zc->params.hashLog;
|
||||
U32* const chainTable = zc->chainTable;
|
||||
const U32 chainMask = (1 << zc->params.chainLog) - 1;
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 target = (U32)(ip - base);
|
||||
U32 idx = zc->nextToUpdate;
|
||||
|
||||
while(idx < target)
|
||||
{
|
||||
size_t h = ZSTD_HC_hashPtr(base+idx, hashLog, mls);
|
||||
NEXT_IN_CHAIN(idx) = hashTable[h];
|
||||
hashTable[h] = idx;
|
||||
idx++;
|
||||
}
|
||||
|
||||
zc->nextToUpdate = target;
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE /* inlining is important to hardwire a hot branch (template emulation) */
|
||||
size_t ZSTD_HC_insertAndFindBestMatch (
|
||||
ZSTD_HC_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
const BYTE** matchpos,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
const U32 hashLog = zc->params.hashLog;
|
||||
U32* const chainTable = zc->chainTable;
|
||||
const U32 chainSize = (1 << zc->params.chainLog);
|
||||
const U32 chainMask = chainSize-1;
|
||||
const BYTE* const base = zc->base;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const U32 maxDistance = (1 << zc->params.windowLog);
|
||||
const U32 lowLimit = (zc->lowLimit + maxDistance > (U32)(ip-base)) ? zc->lowLimit : (U32)(ip - base) - (maxDistance - 1);
|
||||
U32 matchIndex;
|
||||
const BYTE* match;
|
||||
int nbAttempts=maxNbAttempts;
|
||||
size_t ml=0;
|
||||
|
||||
/* HC4 match finder */
|
||||
ZSTD_HC_insert(zc, ip, matchLengthSearch);
|
||||
matchIndex = hashTable[ZSTD_HC_hashPtr(ip, hashLog, matchLengthSearch)];
|
||||
|
||||
while ((matchIndex>=lowLimit) && (nbAttempts))
|
||||
{
|
||||
nbAttempts--;
|
||||
if (matchIndex >= dictLimit)
|
||||
{
|
||||
match = base + matchIndex;
|
||||
if (*(match+ml) == *(ip+ml)
|
||||
&& (MEM_read32(match) == MEM_read32(ip)))
|
||||
{
|
||||
const size_t mlt = ZSTD_count(ip+MINMATCH, match+MINMATCH, iLimit) + MINMATCH;
|
||||
if (mlt > ml) { ml = mlt; *matchpos = match; if (ip+ml >= iLimit) break; }
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
match = dictBase + matchIndex;
|
||||
if (MEM_read32(match) == MEM_read32(ip))
|
||||
{
|
||||
size_t mlt;
|
||||
const BYTE* vLimit = ip + (dictLimit - matchIndex);
|
||||
if (vLimit > iLimit) vLimit = iLimit;
|
||||
mlt = ZSTD_count(ip+MINMATCH, match+MINMATCH, vLimit) + MINMATCH;
|
||||
if ((ip+mlt == vLimit) && (vLimit < iLimit))
|
||||
mlt += ZSTD_count(ip+mlt, base+dictLimit, iLimit);
|
||||
if (mlt > ml) { ml = mlt; *matchpos = base + matchIndex; } /* virtual matchpos */
|
||||
}
|
||||
}
|
||||
|
||||
if (base + matchIndex <= ip - chainSize) break;
|
||||
matchIndex = NEXT_IN_CHAIN(matchIndex);
|
||||
}
|
||||
|
||||
return ml;
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_HC_insertAndFindBestMatch_selectMLS (
|
||||
ZSTD_HC_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
const BYTE** matchpos,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
default :
|
||||
case 4 : return ZSTD_HC_insertAndFindBestMatch(zc, ip, iLimit, matchpos, maxNbAttempts, 4);
|
||||
case 5 : return ZSTD_HC_insertAndFindBestMatch(zc, ip, iLimit, matchpos, maxNbAttempts, 5);
|
||||
case 6 : return ZSTD_HC_insertAndFindBestMatch(zc, ip, iLimit, matchpos, maxNbAttempts, 6);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_HC_compressBlock(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* match = istart;
|
||||
|
||||
size_t offset_2=REPCODE_STARTVALUE, offset_1=REPCODE_STARTVALUE;
|
||||
const U32 maxSearches = 1 << ctx->params.searchLog;
|
||||
const U32 mls = ctx->params.searchLength;
|
||||
|
||||
/* init */
|
||||
ZSTD_resetSeqStore(seqStorePtr);
|
||||
if (((ip-ctx->base) - ctx->dictLimit) < REPCODE_STARTVALUE) ip += REPCODE_STARTVALUE;
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit)
|
||||
{
|
||||
/* repcode */
|
||||
if (MEM_read32(ip) == MEM_read32(ip - offset_2))
|
||||
{
|
||||
/* store sequence */
|
||||
size_t matchLength = ZSTD_count(ip+MINMATCH, ip+MINMATCH-offset_2, iend);
|
||||
size_t litLength = ip-anchor;
|
||||
size_t offset = offset_2;
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, 0, matchLength);
|
||||
ip += matchLength+MINMATCH;
|
||||
anchor = ip;
|
||||
continue;
|
||||
}
|
||||
|
||||
offset_2 = offset_1; /* failed once : necessarily offset_1 now */
|
||||
|
||||
/* repcode at ip+1 */
|
||||
if (MEM_read32(ip+1) == MEM_read32(ip+1 - offset_1))
|
||||
{
|
||||
size_t matchLength = ZSTD_count(ip+1+MINMATCH, ip+1+MINMATCH-offset_1, iend);
|
||||
size_t litLength = ip+1-anchor;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, 0, matchLength);
|
||||
ip += 1+matchLength+MINMATCH;
|
||||
anchor = ip;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* search */
|
||||
{
|
||||
size_t matchLength = ZSTD_HC_insertAndFindBestMatch_selectMLS(ctx, ip, iend, &match, maxSearches, mls);
|
||||
if (!matchLength) { ip++; continue; }
|
||||
/* store sequence */
|
||||
{
|
||||
size_t litLength = ip-anchor;
|
||||
offset_1 = ip-match;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset_1, matchLength-MINMATCH);
|
||||
ip += matchLength;
|
||||
anchor = ip;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Last Literals */
|
||||
{
|
||||
size_t lastLLSize = iend - anchor;
|
||||
memcpy(seqStorePtr->lit, anchor, lastLLSize);
|
||||
seqStorePtr->lit += lastLLSize;
|
||||
}
|
||||
|
||||
/* Final compression stage */
|
||||
return ZSTD_compressSequences((BYTE*)dst, maxDstSize,
|
||||
seqStorePtr, srcSize);
|
||||
}
|
||||
|
||||
static size_t ZSTD_HC_compress_generic (ZSTD_HC_CCtx* ctxPtr,
|
||||
void* dst, size_t maxDstSize,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
static const size_t blockSize = 128 KB;
|
||||
size_t remaining = srcSize;
|
||||
const BYTE* ip = (const BYTE*)src;
|
||||
BYTE* const ostart = (BYTE*)dst;
|
||||
BYTE* op = ostart;
|
||||
BYTE* const oend = op + maxDstSize;
|
||||
|
||||
while (remaining > blockSize)
|
||||
{
|
||||
size_t cSize = ZSTD_HC_compressBlock(ctxPtr, op+3, oend-op, ip, blockSize);
|
||||
|
||||
if (cSize == 0)
|
||||
{
|
||||
cSize = ZSTD_noCompressBlock(op, maxDstSize, ip, blockSize); /* block is not compressible */
|
||||
}
|
||||
else
|
||||
{
|
||||
op[0] = (BYTE)(cSize>>16);
|
||||
op[1] = (BYTE)(cSize>>8);
|
||||
op[2] = (BYTE)cSize;
|
||||
op[0] += (BYTE)(bt_compressed << 6); /* is a compressed block */
|
||||
cSize += 3;
|
||||
}
|
||||
|
||||
remaining -= blockSize;
|
||||
ip += blockSize;
|
||||
op += cSize;
|
||||
if (ZSTD_isError(cSize)) return cSize;
|
||||
}
|
||||
|
||||
/* last block */
|
||||
{
|
||||
size_t cSize = ZSTD_HC_compressBlock(ctxPtr, op+3, oend-op, ip, remaining);
|
||||
|
||||
if (cSize == 0)
|
||||
{
|
||||
cSize = ZSTD_noCompressBlock(op, maxDstSize, ip, remaining); /* block is not compressible */
|
||||
}
|
||||
else
|
||||
{
|
||||
op[0] = (BYTE)(cSize>>16);
|
||||
op[1] = (BYTE)(cSize>>8);
|
||||
op[2] = (BYTE)cSize;
|
||||
op[0] += (BYTE)(bt_compressed << 6); /* is a compressed block */
|
||||
cSize += 3;
|
||||
}
|
||||
|
||||
op += cSize;
|
||||
if (ZSTD_isError(cSize)) return cSize;
|
||||
}
|
||||
|
||||
return op-ostart;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_HC_compressContinue (ZSTD_HC_CCtx* ctxPtr,
|
||||
void* dst, size_t dstSize,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
const BYTE* const ip = (const BYTE*) src;
|
||||
|
||||
/* Check if blocks follow each other */
|
||||
if (ip != ctxPtr->end)
|
||||
{
|
||||
if (ctxPtr->end != NULL)
|
||||
ZSTD_HC_resetCCtx_advanced(ctxPtr, ctxPtr->params); /* just reset, but no need to re-alloc */
|
||||
ctxPtr->base = ip;
|
||||
}
|
||||
|
||||
ctxPtr->end = ip + srcSize;
|
||||
return ZSTD_HC_compress_generic (ctxPtr, dst, dstSize, src, srcSize);
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_HC_compressBegin_advanced(ZSTD_HC_CCtx* ctx,
|
||||
void* dst, size_t maxDstSize,
|
||||
const ZSTD_HC_parameters params)
|
||||
{
|
||||
size_t errorCode;
|
||||
if (maxDstSize < 4) return ERROR(dstSize_tooSmall);
|
||||
errorCode = ZSTD_HC_resetCCtx_advanced(ctx, params);
|
||||
if (ZSTD_isError(errorCode)) return errorCode;
|
||||
MEM_writeLE32(dst, ZSTD_magicNumber); /* Write Header */
|
||||
return 4;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_HC_compressBegin(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, int compressionLevel)
|
||||
{
|
||||
if (compressionLevel<=0) compressionLevel = 1;
|
||||
if (compressionLevel > ZSTD_HC_MAX_CLEVEL) compressionLevel = ZSTD_HC_MAX_CLEVEL;
|
||||
return ZSTD_HC_compressBegin_advanced(ctx, dst, maxDstSize, ZSTD_HC_defaultParameters[compressionLevel]);
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_HC_compressEnd(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize)
|
||||
{
|
||||
BYTE* op = (BYTE*)dst;
|
||||
|
||||
/* Sanity check */
|
||||
(void)ctx;
|
||||
if (maxDstSize < 3) return ERROR(dstSize_tooSmall);
|
||||
|
||||
/* End of frame */
|
||||
op[0] = (BYTE)(bt_end << 6);
|
||||
op[1] = 0;
|
||||
op[2] = 0;
|
||||
|
||||
return 3;
|
||||
}
|
||||
|
||||
size_t ZSTD_HC_compress_advanced (ZSTD_HC_CCtx* ctx,
|
||||
void* dst, size_t maxDstSize,
|
||||
const void* src, size_t srcSize,
|
||||
ZSTD_HC_parameters params)
|
||||
{
|
||||
BYTE* const ostart = (BYTE*)dst;
|
||||
BYTE* op = ostart;
|
||||
size_t oSize;
|
||||
|
||||
/* correct params, to use less memory */
|
||||
U32 srcLog = ZSTD_highbit((U32)srcSize-1) + 1;
|
||||
if (params.windowLog > srcLog) params.windowLog = srcLog;
|
||||
if (params.chainLog > srcLog) params.chainLog = srcLog;
|
||||
|
||||
/* Header */
|
||||
oSize = ZSTD_HC_compressBegin_advanced(ctx, dst, maxDstSize, params);
|
||||
if(ZSTD_isError(oSize)) return oSize;
|
||||
op += oSize;
|
||||
maxDstSize -= oSize;
|
||||
|
||||
/* body (compression) */
|
||||
ctx->base = (const BYTE*)src;
|
||||
op += ZSTD_HC_compress_generic (ctx, op, maxDstSize, src, srcSize);
|
||||
if(ZSTD_isError(oSize)) return oSize;
|
||||
op += oSize;
|
||||
maxDstSize -= oSize;
|
||||
|
||||
/* Close frame */
|
||||
oSize = ZSTD_HC_compressEnd(ctx, op, maxDstSize);
|
||||
if(ZSTD_isError(oSize)) return oSize;
|
||||
op += oSize;
|
||||
|
||||
return (op - ostart);
|
||||
}
|
||||
|
||||
size_t ZSTD_HC_compressCCtx (ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize, int compressionLevel)
|
||||
{
|
||||
if (compressionLevel<=1) return ZSTD_compress(dst, maxDstSize, src, srcSize); /* fast mode */
|
||||
if (compressionLevel > ZSTD_HC_MAX_CLEVEL) compressionLevel = ZSTD_HC_MAX_CLEVEL;
|
||||
return ZSTD_HC_compress_advanced(ctx, dst, maxDstSize, src, srcSize, ZSTD_HC_defaultParameters[compressionLevel]);
|
||||
}
|
||||
|
||||
size_t ZSTD_HC_compress(void* dst, size_t maxDstSize, const void* src, size_t srcSize, int compressionLevel)
|
||||
{
|
||||
size_t result;
|
||||
ZSTD_HC_CCtx ctxBody;
|
||||
memset(&ctxBody, 0, sizeof(ctxBody));
|
||||
result = ZSTD_HC_compressCCtx(&ctxBody, dst, maxDstSize, src, srcSize, compressionLevel);
|
||||
free(ctxBody.workSpace);
|
||||
return result;
|
||||
}
|
||||
76
lib/zstdhc.h
Normal file
76
lib/zstdhc.h
Normal file
@ -0,0 +1,76 @@
|
||||
/*
|
||||
zstdhc - high compression variant
|
||||
Header File
|
||||
Copyright (C) 2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : http://www.zstd.net
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include <stddef.h> /* size_t */
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Simple function
|
||||
***************************************/
|
||||
/**
|
||||
ZSTD_HC_compress() :
|
||||
Compresses 'srcSize' bytes from buffer 'src' into buffer 'dst', of maximum size 'dstSize'.
|
||||
Destination buffer must be already allocated.
|
||||
Compression runs faster if maxDstSize >= ZSTD_compressBound(srcSize).
|
||||
@return : the number of bytes written into buffer 'dst'
|
||||
or an error code if it fails (which can be tested using ZSTD_isError())
|
||||
*/
|
||||
size_t ZSTD_HC_compress(void* dst, size_t maxDstSize,
|
||||
const void* src, size_t srcSize,
|
||||
int compressionLevel);
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Advanced functions
|
||||
***************************************/
|
||||
typedef struct ZSTD_HC_CCtx_s ZSTD_HC_CCtx; /* incomplete type */
|
||||
ZSTD_HC_CCtx* ZSTD_HC_createCCtx(void);
|
||||
size_t ZSTD_HC_freeCCtx(ZSTD_HC_CCtx* cctx);
|
||||
|
||||
/**
|
||||
ZSTD_HC_compressCCtx() :
|
||||
Same as ZSTD_compress(), but requires a ZSTD_HC_CCtx working space already allocated
|
||||
*/
|
||||
size_t ZSTD_HC_compressCCtx(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize, int compressionLevel);
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
126
lib/zstdhc_static.h
Normal file
126
lib/zstdhc_static.h
Normal file
@ -0,0 +1,126 @@
|
||||
/*
|
||||
zstdhc - high compression variant
|
||||
Header File - Experimental API, static linking only
|
||||
Copyright (C) 2015, Yann Collet.
|
||||
|
||||
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : http://www.zstd.net
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include "mem.h"
|
||||
#include "zstdhc.h"
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Types
|
||||
***************************************/
|
||||
typedef struct
|
||||
{
|
||||
U32 windowLog; /* largest match distance : impact decompression buffer size */
|
||||
U32 chainLog; /* full search distance : larger == more compression, slower, more memory*/
|
||||
U32 hashLog; /* dispatch table : larger == more memory, faster*/
|
||||
U32 searchLog; /* nb of searches : larger == more compression, slower*/
|
||||
U32 searchLength; /* size of matches : larger == faster decompression */
|
||||
} ZSTD_HC_parameters;
|
||||
|
||||
/* parameters boundaries */
|
||||
#define ZSTD_HC_WINDOWLOG_MAX 26
|
||||
#define ZSTD_HC_WINDOWLOG_MIN 18
|
||||
#define ZSTD_HC_CHAINLOG_MAX ZSTD_HC_WINDOWLOG_MAX
|
||||
#define ZSTD_HC_CHAINLOG_MIN 4
|
||||
#define ZSTD_HC_HASHLOG_MAX 28
|
||||
#define ZSTD_HC_HASHLOG_MIN 4
|
||||
#define ZSTD_HC_SEARCHLOG_MAX (ZSTD_HC_CHAINLOG_MAX-1)
|
||||
#define ZSTD_HC_SEARCHLOG_MIN 1
|
||||
#define ZSTD_HC_SEARCHLENGTH_MAX 6
|
||||
#define ZSTD_HC_SEARCHLENGTH_MIN 4
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Advanced function
|
||||
***************************************/
|
||||
/** ZSTD_HC_compress_advanced
|
||||
* Same as ZSTD_HC_compressCCtx(), but can fine-tune each compression parameter */
|
||||
size_t ZSTD_HC_compress_advanced (ZSTD_HC_CCtx* ctx,
|
||||
void* dst, size_t maxDstSize,
|
||||
const void* src, size_t srcSize,
|
||||
ZSTD_HC_parameters params);
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Streaming functions
|
||||
***************************************/
|
||||
size_t ZSTD_HC_compressBegin(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, int compressionLevel);
|
||||
size_t ZSTD_HC_compressContinue(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
|
||||
size_t ZSTD_HC_compressEnd(ZSTD_HC_CCtx* ctx, void* dst, size_t maxDstSize);
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Pre-defined compression levels
|
||||
***************************************/
|
||||
#define ZSTD_HC_MAX_CLEVEL 26
|
||||
static const ZSTD_HC_parameters ZSTD_HC_defaultParameters[ZSTD_HC_MAX_CLEVEL+1] = {
|
||||
/* W, C, H, S */
|
||||
{ 18, 12, 14, 1, 4 }, /* level 0 - never used */
|
||||
{ 18, 12, 14, 1, 4 }, /* level 1 - in fact redirected towards zstd fast */
|
||||
{ 18, 12, 15, 2, 4 }, /* level 2 */
|
||||
{ 19, 14, 16, 3, 4 }, /* level 3 */
|
||||
{ 20, 15, 17, 4, 5 }, /* level 4 */
|
||||
{ 20, 17, 19, 4, 5 }, /* level 5 */
|
||||
{ 20, 19, 19, 4, 5 }, /* level 6 */
|
||||
{ 20, 19, 19, 5, 5 }, /* level 7 */
|
||||
{ 20, 20, 20, 5, 5 }, /* level 8 */
|
||||
{ 20, 20, 20, 6, 5 }, /* level 9 */
|
||||
{ 21, 21, 20, 5, 5 }, /* level 10 */
|
||||
{ 22, 21, 22, 6, 5 }, /* level 11 */
|
||||
{ 23, 21, 22, 6, 5 }, /* level 12 */
|
||||
{ 23, 21, 22, 7, 5 }, /* level 13 */
|
||||
{ 22, 22, 23, 7, 5 }, /* level 14 */
|
||||
{ 22, 22, 23, 7, 5 }, /* level 15 */
|
||||
{ 22, 22, 23, 8, 5 }, /* level 16 */
|
||||
{ 22, 22, 23, 8, 5 }, /* level 17 */
|
||||
{ 22, 22, 23, 9, 5 }, /* level 18 */
|
||||
{ 22, 22, 23, 9, 5 }, /* level 19 */
|
||||
{ 23, 23, 23, 9, 5 }, /* level 20 */
|
||||
{ 23, 23, 23, 9, 5 }, /* level 21 */
|
||||
{ 23, 23, 23, 10, 5 }, /* level 22 */
|
||||
{ 23, 23, 23, 10, 5 }, /* level 23 */
|
||||
{ 23, 23, 23, 11, 5 }, /* level 24 */
|
||||
{ 23, 23, 23, 12, 5 }, /* level 25 */
|
||||
{ 23, 23, 23, 13, 5 }, /* level 26 */ /* ZSTD_HC_MAX_CLEVEL */
|
||||
};
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
@ -30,11 +30,11 @@
|
||||
# fullbench32: Same as fullbench, but forced to compile in 32-bits mode
|
||||
# ##########################################################################
|
||||
|
||||
VERSION?= v0.2.0
|
||||
VERSION?= 0.3.0
|
||||
|
||||
DESTDIR?=
|
||||
PREFIX ?= /usr/local
|
||||
CPPFLAGS= -I../lib -I../lib/legacy -DZSTD_VERSION=\"$(VERSION)\" -DZSTD_LEGACY_SUPPORT=1
|
||||
CPPFLAGS= -I../lib -I../lib/legacy -I./legacy -DZSTD_VERSION=\"$(VERSION)\" -DZSTD_LEGACY_SUPPORT=1
|
||||
CFLAGS ?= -O3 # -falign-loops=32 # not always positive
|
||||
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-qual -Wcast-align -Wstrict-prototypes
|
||||
FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MOREFLAGS)
|
||||
@ -56,26 +56,43 @@ endif
|
||||
|
||||
default: zstd
|
||||
|
||||
all: zstd zstd32 fullbench fullbench32 fuzzer fuzzer32 datagen
|
||||
all: zstd zstd32 fullbench fullbench32 fuzzer fuzzer32 paramgrill datagen
|
||||
|
||||
zstd : $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c xxhash.c bench.c fileio.c zstdcli.c
|
||||
zstd: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/zstdhc.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
xxhash.c bench.c fileio.c zstdcli.c legacy/fileio_legacy.c
|
||||
$(CC) $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
zstd32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c xxhash.c bench.c fileio.c zstdcli.c
|
||||
zstd32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/zstdhc.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
xxhash.c bench.c fileio.c zstdcli.c legacy/fileio_legacy.c
|
||||
$(CC) -m32 $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
fullbench : $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c datagen.c fullbench.c
|
||||
fullbench : $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
datagen.c fullbench.c
|
||||
$(CC) $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
fullbench32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c datagen.c fullbench.c
|
||||
fullbench32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
datagen.c fullbench.c
|
||||
$(CC) -m32 $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
fuzzer : $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c datagen.c xxhash.c fuzzer.c
|
||||
fuzzer : $(ZSTDDIR)/zstd.c $(ZSTDDIR)/zstdhc.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
datagen.c xxhash.c fuzzer.c
|
||||
$(CC) $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
fuzzer32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c $(ZSTDDIR)/legacy/zstd_v01.c datagen.c xxhash.c fuzzer.c
|
||||
fuzzer32: $(ZSTDDIR)/zstd.c $(ZSTDDIR)/zstdhc.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
datagen.c xxhash.c fuzzer.c
|
||||
$(CC) -m32 $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
paramgrill : $(ZSTDDIR)/zstdhc.c $(ZSTDDIR)/zstd.c $(ZSTDDIR)/fse.c $(ZSTDDIR)/huff0.c \
|
||||
$(ZSTDDIR)/legacy/zstd_v01.c $(ZSTDDIR)/legacy/zstd_v02.c \
|
||||
datagen.c xxhash.c paramgrill.c
|
||||
$(CC) $(FLAGS) $^ -lm -o $@$(EXT)
|
||||
|
||||
datagen : datagen.c datagencli.c
|
||||
$(CC) $(FLAGS) $^ -o $@$(EXT)
|
||||
|
||||
@ -84,7 +101,7 @@ clean:
|
||||
zstd$(EXT) zstd32$(EXT) \
|
||||
fullbench$(EXT) fullbench32$(EXT) \
|
||||
fuzzer$(EXT) fuzzer32$(EXT) \
|
||||
datagen$(EXT)
|
||||
datagen$(EXT) paramgrill$(EXT)
|
||||
@echo Cleaning completed
|
||||
|
||||
|
||||
@ -117,7 +134,7 @@ test: test-zstd test-fullbench test-fuzzer
|
||||
|
||||
test32: test-zstd32 test-fullbench32 test-fuzzer32
|
||||
|
||||
test-all: test test32 memtest
|
||||
test-all: test test32 valgrindTest
|
||||
|
||||
test-zstd: zstd datagen
|
||||
@echo "\n**** frame concatenation **** "
|
||||
@ -137,7 +154,9 @@ test-zstd: zstd datagen
|
||||
echo foo | ./zstd | ./zstd -d > /dev/full; if [ $$? -eq 0 ] ; then echo "write error not detected!"; false; fi
|
||||
@echo "**** zstd round-trip tests **** "
|
||||
./datagen | ./zstd -v | ./zstd -d > $(VOID)
|
||||
./datagen | ./zstd -6 -v | ./zstd -d > $(VOID)
|
||||
./datagen -g256MB | ./zstd -v | ./zstd -d > $(VOID)
|
||||
./datagen -g256MB | ./zstd -3 -v | ./zstd -d > $(VOID)
|
||||
./datagen -g6GB -P99 | ./zstd -vq | ./zstd -d > $(VOID)
|
||||
|
||||
test-zstd32: zstd32 datagen
|
||||
@ -159,11 +178,13 @@ test-fuzzer: fuzzer
|
||||
test-fuzzer32: fuzzer32
|
||||
./fuzzer32
|
||||
|
||||
memtest: zstd datagen fuzzer fullbench
|
||||
valgrindTest: zstd datagen fuzzer fullbench
|
||||
@echo "\n ---- valgrind tests : memory analyzer ----"
|
||||
valgrind --leak-check=yes --error-exitcode=1 ./datagen -g50M > $(VOID)
|
||||
./datagen -g16KB > tmp
|
||||
valgrind --leak-check=yes --error-exitcode=1 ./zstd -vf tmp $(VOID)
|
||||
./datagen -g2930KB > tmp
|
||||
valgrind --leak-check=yes --error-exitcode=1 ./zstd -4 -vf tmp $(VOID)
|
||||
./datagen -g64MB > tmp
|
||||
valgrind --leak-check=yes --error-exitcode=1 ./zstd -vf tmp $(VOID)
|
||||
@rm tmp
|
||||
|
||||
152
programs/bench.c
152
programs/bench.c
@ -23,76 +23,59 @@
|
||||
- ztsd public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
|
||||
/***************************************
|
||||
/* **************************************
|
||||
* Compiler Options
|
||||
***************************************/
|
||||
****************************************/
|
||||
/* Disable some Visual warning messages */
|
||||
#define _CRT_SECURE_NO_WARNINGS /* fopen */
|
||||
|
||||
// Unix Large Files support (>4GB)
|
||||
/* Unix Large Files support (>4GB) */
|
||||
#define _FILE_OFFSET_BITS 64
|
||||
#if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits requires specific definitions
|
||||
#if (defined(__sun__) && (!defined(__LP64__))) /* Sun Solaris 32-bits requires specific definitions */
|
||||
# define _LARGEFILE_SOURCE
|
||||
#elif ! defined(__LP64__) // No point defining Large file for 64 bit
|
||||
#elif ! defined(__LP64__) /* No point defining Large file for 64 bit */
|
||||
# define _LARGEFILE64_SOURCE
|
||||
#endif
|
||||
|
||||
// S_ISREG & gettimeofday() are not supported by MSVC
|
||||
/* S_ISREG & gettimeofday() are not supported by MSVC */
|
||||
#if defined(_MSC_VER) || defined(_WIN32)
|
||||
# define BMK_LEGACY_TIMER 1
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Includes
|
||||
**************************************/
|
||||
***************************************/
|
||||
#include <stdlib.h> /* malloc, free */
|
||||
#include <string.h> /* memset */
|
||||
#include <stdio.h> // fprintf, fopen, ftello64
|
||||
#include <sys/types.h> // stat64
|
||||
#include <sys/stat.h> // stat64
|
||||
#include <stdio.h> /* fprintf, fopen, ftello64 */
|
||||
#include <sys/types.h> /* stat64 */
|
||||
#include <sys/stat.h> /* stat64 */
|
||||
|
||||
// Use ftime() if gettimeofday() is not available on your target
|
||||
/* Use ftime() if gettimeofday() is not available */
|
||||
#if defined(BMK_LEGACY_TIMER)
|
||||
# include <sys/timeb.h> // timeb, ftime
|
||||
# include <sys/timeb.h> /* timeb, ftime */
|
||||
#else
|
||||
# include <sys/time.h> // gettimeofday
|
||||
# include <sys/time.h> /* gettimeofday */
|
||||
#endif
|
||||
|
||||
#include "mem.h"
|
||||
#include "zstd.h"
|
||||
#include "zstdhc.h"
|
||||
#include "xxhash.h"
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Compiler specifics
|
||||
**************************************/
|
||||
***************************************/
|
||||
#if !defined(S_ISREG)
|
||||
# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG)
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
* Basic Types
|
||||
**************************************/
|
||||
#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
|
||||
# include <stdint.h>
|
||||
typedef uint8_t BYTE;
|
||||
typedef uint16_t U16;
|
||||
typedef uint32_t U32;
|
||||
typedef int32_t S32;
|
||||
typedef uint64_t U64;
|
||||
#else
|
||||
typedef unsigned char BYTE;
|
||||
typedef unsigned short U16;
|
||||
typedef unsigned int U32;
|
||||
typedef signed int S32;
|
||||
typedef unsigned long long U64;
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Constants
|
||||
**************************************/
|
||||
***************************************/
|
||||
#define NBLOOPS 3
|
||||
#define TIMELOOP 2500
|
||||
|
||||
@ -108,15 +91,15 @@ static U32 prime1 = 2654435761U;
|
||||
static U32 prime2 = 2246822519U;
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Macros
|
||||
**************************************/
|
||||
***************************************/
|
||||
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Benchmark Parameters
|
||||
**************************************/
|
||||
***************************************/
|
||||
static int nbIterations = NBLOOPS;
|
||||
static size_t g_blockSize = 0;
|
||||
|
||||
@ -133,9 +116,9 @@ void BMK_SetBlockSize(size_t blockSize)
|
||||
}
|
||||
|
||||
|
||||
/*********************************************************
|
||||
/* ********************************************************
|
||||
* Private functions
|
||||
*********************************************************/
|
||||
**********************************************************/
|
||||
|
||||
#if defined(BMK_LEGACY_TIMER)
|
||||
|
||||
@ -176,10 +159,9 @@ static int BMK_GetMilliSpan( int nTimeStart )
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*********************************************************
|
||||
/* ********************************************************
|
||||
* Data generator
|
||||
*********************************************************/
|
||||
**********************************************************/
|
||||
/* will hopefully be converted into ROL instruction by compiler */
|
||||
static U32 BMK_rotl32(unsigned val32, unsigned nbBits) { return((val32 << nbBits) | (val32 >> (32 - nbBits))); }
|
||||
|
||||
@ -193,7 +175,6 @@ static U32 BMK_rand(U32* src)
|
||||
return rand32 >> 9;
|
||||
}
|
||||
|
||||
|
||||
#define BMK_RAND15BITS ( BMK_rand(&seed) & 0x7FFF)
|
||||
#define BMK_RANDLENGTH ((BMK_rand(&seed) & 3) ? (BMK_rand(&seed) % 15) : (BMK_rand(&seed) % 510) + 15)
|
||||
#define BMK_RANDCHAR (BYTE)((BMK_rand(&seed) & 63) + '0')
|
||||
@ -234,9 +215,9 @@ static void BMK_datagen(void* buffer, size_t bufferSize, double proba, U32 seed)
|
||||
}
|
||||
|
||||
|
||||
/*********************************************************
|
||||
/* ********************************************************
|
||||
* Bench functions
|
||||
*********************************************************/
|
||||
**********************************************************/
|
||||
typedef struct
|
||||
{
|
||||
char* srcPtr;
|
||||
@ -248,6 +229,13 @@ typedef struct
|
||||
size_t resSize;
|
||||
} blockParam_t;
|
||||
|
||||
typedef size_t (*compressor_t) (void* dst, size_t maxDstSize, const void* src, size_t srcSize, int compressionLevel);
|
||||
|
||||
static size_t local_compress_fast (void* dst, size_t maxDstSize, const void* src, size_t srcSize, int compressionLevel)
|
||||
{
|
||||
(void)compressionLevel;
|
||||
return ZSTD_compress(dst, maxDstSize, src, srcSize);
|
||||
}
|
||||
|
||||
#define MIN(a,b) (a<b ? a : b)
|
||||
|
||||
@ -259,10 +247,12 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
const size_t maxCompressedSize = (size_t)nbBlocks * ZSTD_compressBound(blockSize);
|
||||
void* const compressedBuffer = malloc(maxCompressedSize);
|
||||
void* const resultBuffer = malloc(srcSize);
|
||||
const compressor_t compressor = (cLevel <= 1) ? local_compress_fast : ZSTD_HC_compress;
|
||||
U64 crcOrig;
|
||||
|
||||
/* Init */
|
||||
(void)cLevel;
|
||||
/* init */
|
||||
if (strlen(fileName)>16)
|
||||
fileName += strlen(fileName)-16;
|
||||
|
||||
/* Memory allocation & restrictions */
|
||||
if (!compressedBuffer || !resultBuffer || !blockTable)
|
||||
@ -318,7 +308,7 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
U32 blockNb;
|
||||
|
||||
/* Compression */
|
||||
DISPLAY("%1i-%-14.14s : %9u ->\r", loopNb, fileName, (U32)srcSize);
|
||||
DISPLAY("%2i-%-17.17s :%10u ->\r", loopNb, fileName, (U32)srcSize);
|
||||
memset(compressedBuffer, 0xE5, maxCompressedSize);
|
||||
|
||||
nbLoops = 0;
|
||||
@ -328,8 +318,7 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
while (BMK_GetMilliSpan(milliTime) < TIMELOOP)
|
||||
{
|
||||
for (blockNb=0; blockNb<nbBlocks; blockNb++)
|
||||
blockTable[blockNb].cSize = ZSTD_compress(blockTable[blockNb].cPtr, blockTable[blockNb].cRoom,
|
||||
blockTable[blockNb].srcPtr,blockTable[blockNb].srcSize);
|
||||
blockTable[blockNb].cSize = compressor(blockTable[blockNb].cPtr, blockTable[blockNb].cRoom, blockTable[blockNb].srcPtr,blockTable[blockNb].srcSize, cLevel);
|
||||
nbLoops++;
|
||||
}
|
||||
milliTime = BMK_GetMilliSpan(milliTime);
|
||||
@ -338,8 +327,8 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
for (blockNb=0; blockNb<nbBlocks; blockNb++)
|
||||
cSize += blockTable[blockNb].cSize;
|
||||
if ((double)milliTime < fastestC*nbLoops) fastestC = (double)milliTime / nbLoops;
|
||||
ratio = (double)cSize / (double)srcSize*100.;
|
||||
DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s\r", loopNb, fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000.);
|
||||
ratio = (double)srcSize / (double)cSize;
|
||||
DISPLAY("%2i-%-17.17s :%10i ->%10i (%5.3f),%6.1f MB/s\r", loopNb, fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000.);
|
||||
|
||||
#if 1
|
||||
/* Decompression */
|
||||
@ -358,19 +347,20 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
milliTime = BMK_GetMilliSpan(milliTime);
|
||||
|
||||
if ((double)milliTime < fastestD*nbLoops) fastestD = (double)milliTime / nbLoops;
|
||||
DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\r", loopNb, fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000., (double)srcSize / fastestD / 1000.);
|
||||
DISPLAY("%2i-%-17.17s :%10i ->%10i (%5.3f),%6.1f MB/s ,%6.1f MB/s\r", loopNb, fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000., (double)srcSize / fastestD / 1000.);
|
||||
|
||||
/* CRC Checking */
|
||||
crcCheck = XXH64(resultBuffer, srcSize, 0);
|
||||
if (crcOrig!=crcCheck)
|
||||
{
|
||||
unsigned i;
|
||||
unsigned u;
|
||||
unsigned eBlockSize = (unsigned)(MIN(65536*2, blockSize));
|
||||
DISPLAY("\n!!! WARNING !!! %14s : Invalid Checksum : %x != %x\n", fileName, (unsigned)crcOrig, (unsigned)crcCheck);
|
||||
for (i=0; i<srcSize; i++)
|
||||
for (u=0; u<srcSize; u++)
|
||||
{
|
||||
if (((BYTE*)srcBuffer)[i] != ((BYTE*)resultBuffer)[i])
|
||||
if (((BYTE*)srcBuffer)[u] != ((BYTE*)resultBuffer)[u])
|
||||
{
|
||||
printf("\nDecoding error at pos %u \n", i);
|
||||
printf("Decoding error at pos %u (block %u, pos %u) \n", u, u / eBlockSize, u % eBlockSize);
|
||||
break;
|
||||
}
|
||||
}
|
||||
@ -380,12 +370,7 @@ static int BMK_benchMem(void* srcBuffer, size_t srcSize, const char* fileName, i
|
||||
}
|
||||
|
||||
if (crcOrig == crcCheck)
|
||||
{
|
||||
if (ratio<100.)
|
||||
DISPLAY("%-16.16s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\n", fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000., (double)srcSize / fastestD / 1000.);
|
||||
else
|
||||
DISPLAY("%-16.16s : %9i -> %9i (%5.1f%%),%7.1f MB/s ,%7.1f MB/s \n", fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000., (double)srcSize / fastestD / 1000.);
|
||||
}
|
||||
DISPLAY("%2i-%-17.17s :%10i ->%10i (%5.3f),%6.1f MB/s ,%6.1f MB/s \n", cLevel, fileName, (int)srcSize, (int)cSize, ratio, (double)srcSize / fastestC / 1000., (double)srcSize / fastestD / 1000.);
|
||||
}
|
||||
|
||||
/* End cleaning */
|
||||
@ -434,10 +419,7 @@ static int BMK_benchOneFile(char* inFileName, int cLevel)
|
||||
U64 inFileSize;
|
||||
size_t benchedSize, readSize;
|
||||
void* srcBuffer;
|
||||
int result;
|
||||
|
||||
/* Init */
|
||||
(void)cLevel;
|
||||
int result=0;
|
||||
|
||||
/* Check file existence */
|
||||
inFile = fopen(inFileName, "rb");
|
||||
@ -473,10 +455,17 @@ static int BMK_benchOneFile(char* inFileName, int cLevel)
|
||||
return 13;
|
||||
}
|
||||
|
||||
// Bench
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, inFileName, cLevel);
|
||||
/* Bench */
|
||||
if (cLevel<0)
|
||||
{
|
||||
int l;
|
||||
for (l=1; l <= -cLevel; l++)
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, inFileName, l);
|
||||
}
|
||||
else
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, inFileName, cLevel);
|
||||
|
||||
// End
|
||||
/* clean up */
|
||||
free(srcBuffer);
|
||||
DISPLAY("\n");
|
||||
return result;
|
||||
@ -487,12 +476,9 @@ static int BMK_syntheticTest(int cLevel, double compressibility)
|
||||
{
|
||||
size_t benchedSize = 10000000;
|
||||
void* srcBuffer = malloc(benchedSize);
|
||||
int result;
|
||||
int result=0;
|
||||
char name[20] = {0};
|
||||
|
||||
/* Init */
|
||||
(void)cLevel;
|
||||
|
||||
/* Memory allocation */
|
||||
if (!srcBuffer)
|
||||
{
|
||||
@ -510,7 +496,15 @@ static int BMK_syntheticTest(int cLevel, double compressibility)
|
||||
#else
|
||||
snprintf (name, 20, "Synthetic %2u%%", (unsigned)(compressibility*100));
|
||||
#endif
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, name, cLevel);
|
||||
/* Bench */
|
||||
if (cLevel<0)
|
||||
{
|
||||
int l;
|
||||
for (l=1; l <= -cLevel; l++)
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, name, l);
|
||||
}
|
||||
else
|
||||
result = BMK_benchMem(srcBuffer, benchedSize, name, cLevel);
|
||||
|
||||
/* End */
|
||||
free(srcBuffer);
|
||||
|
||||
@ -29,9 +29,9 @@
|
||||
The license of this file is GPLv2.
|
||||
*/
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Tuning options
|
||||
**************************************/
|
||||
***************************************/
|
||||
#ifndef ZSTD_LEGACY_SUPPORT
|
||||
/**LEGACY_SUPPORT :
|
||||
* decompressor can decode older formats (starting from Zstd 0.1+) */
|
||||
@ -39,9 +39,9 @@
|
||||
#endif // ZSTD_LEGACY_SUPPORT
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Compiler Options
|
||||
**************************************/
|
||||
***************************************/
|
||||
/* Disable some Visual warning messages */
|
||||
#ifdef _MSC_VER
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
@ -55,9 +55,9 @@
|
||||
#define _POSIX_SOURCE 1 /* enable fileno() within <stdio.h> on unix */
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Includes
|
||||
**************************************/
|
||||
***************************************/
|
||||
#include <stdio.h> /* fprintf, fopen, fread, _fileno, stdin, stdout */
|
||||
#include <stdlib.h> /* malloc, free */
|
||||
#include <string.h> /* strcmp, strlen */
|
||||
@ -66,15 +66,17 @@
|
||||
#include "mem.h"
|
||||
#include "fileio.h"
|
||||
#include "zstd_static.h"
|
||||
#include "zstdhc_static.h"
|
||||
|
||||
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT==1)
|
||||
# include "zstd_v01.h" /* legacy */
|
||||
#endif // ZSTD_LEGACY_SUPPORT
|
||||
# include "zstd_legacy.h" /* legacy */
|
||||
# include "fileio_legacy.h" /* legacy */
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* OS-specific Includes
|
||||
**************************************/
|
||||
***************************************/
|
||||
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__)
|
||||
# include <fcntl.h> /* _O_BINARY */
|
||||
# include <io.h> /* _setmode, _isatty */
|
||||
@ -90,9 +92,9 @@
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Constants
|
||||
**************************************/
|
||||
***************************************/
|
||||
#define KB *(1U<<10)
|
||||
#define MB *(1U<<20)
|
||||
#define GB *(1U<<30)
|
||||
@ -116,15 +118,9 @@ static const unsigned FIO_blockHeaderSize = 3;
|
||||
#define CACHELINE 64
|
||||
|
||||
|
||||
/**************************************
|
||||
* Complex types
|
||||
**************************************/
|
||||
typedef enum { bt_compressed, bt_raw, bt_rle, bt_crc } bType_t;
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Macros
|
||||
**************************************/
|
||||
***************************************/
|
||||
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
|
||||
#define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
|
||||
static U32 g_displayLevel = 2; /* 0 : no display; 1: errors; 2 : + result + interaction + warnings; 3 : + progression; 4 : + information */
|
||||
@ -137,19 +133,21 @@ static const unsigned refreshRate = 150;
|
||||
static clock_t g_time = 0;
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Local Parameters
|
||||
**************************************/
|
||||
***************************************/
|
||||
static U32 g_overwrite = 0;
|
||||
|
||||
void FIO_overwriteMode(void) { g_overwrite=1; }
|
||||
void FIO_setNotificationLevel(unsigned level) { g_displayLevel=level; }
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Exceptions
|
||||
**************************************/
|
||||
#define DEBUG 0
|
||||
***************************************/
|
||||
#ifndef DEBUG
|
||||
# define DEBUG 0
|
||||
#endif
|
||||
#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
|
||||
#define EXM_THROW(error, ...) \
|
||||
{ \
|
||||
@ -161,9 +159,9 @@ void FIO_setNotificationLevel(unsigned level) { g_displayLevel=level; }
|
||||
}
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Functions
|
||||
**************************************/
|
||||
***************************************/
|
||||
static unsigned FIO_GetMilliSpan(clock_t nPrevious)
|
||||
{
|
||||
clock_t nCurrent = clock();
|
||||
@ -217,8 +215,47 @@ static void FIO_getFileHandles(FILE** pfinput, FILE** pfoutput, const char* inpu
|
||||
if ( *pfoutput==0) EXM_THROW(13, "Pb opening dst : %s", output_filename);
|
||||
}
|
||||
|
||||
typedef void* (*FIO_createC) (void);
|
||||
static void* local_ZSTD_createCCtx(void) { return (void*) ZSTD_createCCtx(); }
|
||||
static void* local_ZSTD_HC_createCCtx(void) { return (void*) ZSTD_HC_createCCtx(); }
|
||||
|
||||
unsigned long long FIO_compressFilename(const char* output_filename, const char* input_filename)
|
||||
typedef size_t (*FIO_initC) (void* ctx, void* dst, size_t maxDstSize, int cLevel);
|
||||
static size_t local_ZSTD_compressBegin (void* ctx, void* dst, size_t maxDstSize, int cLevel)
|
||||
{
|
||||
(void)cLevel;
|
||||
return ZSTD_compressBegin((ZSTD_CCtx*)ctx, dst, maxDstSize);
|
||||
}
|
||||
static size_t local_ZSTD_HC_compressBegin (void* ctx, void* dst, size_t maxDstSize, int cLevel)
|
||||
{
|
||||
return ZSTD_HC_compressBegin((ZSTD_HC_CCtx*)ctx, dst, maxDstSize, cLevel);
|
||||
}
|
||||
|
||||
typedef size_t (*FIO_continueC) (void* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
|
||||
static size_t local_ZSTD_compressContinue (void* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressContinue((ZSTD_CCtx*)ctx, dst, maxDstSize, src, srcSize);
|
||||
}
|
||||
static size_t local_ZSTD_HC_compressContinue (void* ctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_HC_compressContinue((ZSTD_HC_CCtx*)ctx, dst, maxDstSize, src, srcSize);
|
||||
}
|
||||
|
||||
typedef size_t (*FIO_endC) (void* ctx, void* dst, size_t maxDstSize);
|
||||
static size_t local_ZSTD_compressEnd (void* ctx, void* dst, size_t maxDstSize)
|
||||
{
|
||||
return ZSTD_compressEnd((ZSTD_CCtx*)ctx, dst, maxDstSize);
|
||||
}
|
||||
static size_t local_ZSTD_HC_compressEnd (void* ctx, void* dst, size_t maxDstSize)
|
||||
{
|
||||
return ZSTD_HC_compressEnd((ZSTD_HC_CCtx*)ctx, dst, maxDstSize);
|
||||
}
|
||||
|
||||
typedef void (*FIO_freeC) (void* ctx);
|
||||
static void local_ZSTD_freeCCtx(void* ctx) { ZSTD_freeCCtx((ZSTD_CCtx*)ctx); }
|
||||
static void local_ZSTD_HC_freeCCtx(void* ctx) { ZSTD_HC_freeCCtx((ZSTD_HC_CCtx*)ctx); }
|
||||
|
||||
|
||||
unsigned long long FIO_compressFilename(const char* output_filename, const char* input_filename, int cLevel)
|
||||
{
|
||||
U64 filesize = 0;
|
||||
U64 compressedfilesize = 0;
|
||||
@ -232,13 +269,34 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
FILE* finput;
|
||||
FILE* foutput;
|
||||
size_t sizeCheck, cSize;
|
||||
ZSTD_CCtx* ctx = ZSTD_createCCtx();
|
||||
|
||||
void* ctx;
|
||||
FIO_createC createC=NULL;
|
||||
FIO_initC initC=NULL;
|
||||
FIO_continueC continueC = NULL;
|
||||
FIO_endC endC = NULL;
|
||||
FIO_freeC freeC = NULL;
|
||||
|
||||
/* Init */
|
||||
if (cLevel <= 1)
|
||||
{
|
||||
createC = local_ZSTD_createCCtx;
|
||||
initC = local_ZSTD_compressBegin;
|
||||
continueC = local_ZSTD_compressContinue;
|
||||
endC = local_ZSTD_compressEnd;
|
||||
freeC = local_ZSTD_freeCCtx;
|
||||
}
|
||||
else
|
||||
{
|
||||
createC = local_ZSTD_HC_createCCtx;
|
||||
initC = local_ZSTD_HC_compressBegin;
|
||||
continueC = local_ZSTD_HC_compressContinue;
|
||||
endC = local_ZSTD_HC_compressEnd;
|
||||
freeC = local_ZSTD_HC_freeCCtx;
|
||||
}
|
||||
FIO_getFileHandles(&finput, &foutput, input_filename, output_filename);
|
||||
|
||||
/* Allocate Memory */
|
||||
ctx = createC();
|
||||
inBuff = (BYTE*)malloc(inBuffSize);
|
||||
outBuff = (BYTE*)malloc(outBuffSize);
|
||||
if (!inBuff || !outBuff || !ctx) EXM_THROW(21, "Allocation error : not enough memory");
|
||||
@ -246,7 +304,7 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
inEnd = inBuff + inBuffSize;
|
||||
|
||||
/* Write Frame Header */
|
||||
cSize = ZSTD_compressBegin(ctx, outBuff, outBuffSize);
|
||||
cSize = initC(ctx, outBuff, outBuffSize, cLevel);
|
||||
if (ZSTD_isError(cSize)) EXM_THROW(22, "Compression error : cannot create frame header");
|
||||
|
||||
sizeCheck = fwrite(outBuff, 1, cSize, foutput);
|
||||
@ -266,7 +324,7 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
DISPLAYUPDATE(2, "\rRead : %u MB ", (U32)(filesize>>20));
|
||||
|
||||
/* Compress Block */
|
||||
cSize = ZSTD_compressContinue(ctx, outBuff, outBuffSize, inSlot, inSize);
|
||||
cSize = continueC(ctx, outBuff, outBuffSize, inSlot, inSize);
|
||||
if (ZSTD_isError(cSize))
|
||||
EXM_THROW(24, "Compression error : %s ", ZSTD_getErrorName(cSize));
|
||||
|
||||
@ -280,7 +338,7 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
}
|
||||
|
||||
/* End of Frame */
|
||||
cSize = ZSTD_compressEnd(ctx, outBuff, outBuffSize);
|
||||
cSize = endC(ctx, outBuff, outBuffSize);
|
||||
if (ZSTD_isError(cSize)) EXM_THROW(26, "Compression error : cannot create frame end");
|
||||
|
||||
sizeCheck = fwrite(outBuff, 1, cSize, foutput);
|
||||
@ -295,7 +353,7 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
/* clean */
|
||||
free(inBuff);
|
||||
free(outBuff);
|
||||
ZSTD_freeCCtx(ctx);
|
||||
freeC(ctx);
|
||||
fclose(finput);
|
||||
if (fclose(foutput)) EXM_THROW(28, "Write error : cannot properly close %s", output_filename);
|
||||
|
||||
@ -303,70 +361,6 @@ unsigned long long FIO_compressFilename(const char* output_filename, const char*
|
||||
}
|
||||
|
||||
|
||||
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT==1)
|
||||
|
||||
unsigned long long FIOv01_decompressFrame(FILE* foutput, FILE* finput)
|
||||
{
|
||||
size_t outBuffSize = 512 KB;
|
||||
BYTE* outBuff = (BYTE*)malloc(outBuffSize);
|
||||
size_t inBuffSize = 128 KB + 8;
|
||||
BYTE inBuff[128 KB + 8];
|
||||
BYTE* op = outBuff;
|
||||
BYTE* const oend = outBuff + outBuffSize;
|
||||
U64 filesize = 0;
|
||||
size_t toRead;
|
||||
size_t sizeCheck;
|
||||
ZSTDv01_Dctx* dctx = ZSTDv01_createDCtx();
|
||||
|
||||
|
||||
/* init */
|
||||
if (outBuff==NULL) EXM_THROW(41, "Error : not enough memory to decode legacy frame");
|
||||
|
||||
/* restore header, already read from input */
|
||||
MEM_writeLE32(inBuff, ZSTDv01_magicNumberLE);
|
||||
sizeCheck = ZSTDv01_decompressContinue(dctx, NULL, 0, inBuff, sizeof(ZSTDv01_magicNumberLE)); /* Decode frame header */
|
||||
if (ZSTDv01_isError(sizeCheck)) EXM_THROW(42, "Error decoding legacy header");
|
||||
|
||||
/* Main decompression Loop */
|
||||
toRead = ZSTDv01_nextSrcSizeToDecompress(dctx);
|
||||
while (toRead)
|
||||
{
|
||||
size_t readSize, decodedSize;
|
||||
|
||||
/* Fill input buffer */
|
||||
if (toRead > inBuffSize)
|
||||
EXM_THROW(43, "too large block");
|
||||
readSize = fread(inBuff, 1, toRead, finput);
|
||||
if (readSize != toRead)
|
||||
EXM_THROW(44, "Read error");
|
||||
|
||||
/* Decode block */
|
||||
decodedSize = ZSTDv01_decompressContinue(dctx, op, oend-op, inBuff, readSize);
|
||||
if (ZSTDv01_isError(decodedSize)) EXM_THROW(45, "Decoding error : input corrupted");
|
||||
|
||||
if (decodedSize) /* not a header */
|
||||
{
|
||||
/* Write block */
|
||||
sizeCheck = fwrite(op, 1, decodedSize, foutput);
|
||||
if (sizeCheck != decodedSize) EXM_THROW(46, "Write error : unable to write data block to destination file");
|
||||
filesize += decodedSize;
|
||||
op += decodedSize;
|
||||
if (op==oend) op = outBuff;
|
||||
DISPLAYUPDATE(2, "\rDecoded : %u MB... ", (U32)(filesize>>20) );
|
||||
}
|
||||
|
||||
/* prepare for next Block */
|
||||
toRead = ZSTDv01_nextSrcSizeToDecompress(dctx);
|
||||
}
|
||||
|
||||
/* release resources */
|
||||
free(outBuff);
|
||||
free(dctx);
|
||||
return filesize;
|
||||
}
|
||||
#endif /* ZSTD_LEGACY_SUPPORT */
|
||||
|
||||
|
||||
unsigned long long FIO_decompressFrame(FILE* foutput, FILE* finput,
|
||||
BYTE* inBuff, size_t inBuffSize,
|
||||
BYTE* outBuff, size_t outBuffSize,
|
||||
@ -446,24 +440,20 @@ unsigned long long FIO_decompressFilename(const char* output_filename, const cha
|
||||
if (sizeCheck != toRead) EXM_THROW(31, "Read error : cannot read header");
|
||||
|
||||
magicNumber = MEM_readLE32(header);
|
||||
switch(magicNumber)
|
||||
{
|
||||
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT==1)
|
||||
case ZSTDv01_magicNumberLE:
|
||||
filesize += FIOv01_decompressFrame(foutput, finput);
|
||||
if (ZSTD_isLegacy(magicNumber))
|
||||
{
|
||||
filesize += FIO_decompressLegacyFrame(foutput, finput, magicNumber);
|
||||
continue;
|
||||
#endif /* ZSTD_LEGACY_SUPPORT */
|
||||
case ZSTD_magicNumber:
|
||||
break; /* normal case */
|
||||
default :
|
||||
EXM_THROW(32, "Error : unknown frame prefix");
|
||||
}
|
||||
#endif /* ZSTD_LEGACY_SUPPORT */
|
||||
if (magicNumber != ZSTD_magicNumber) EXM_THROW(32, "Error : unknown frame prefix");
|
||||
|
||||
/* prepare frame decompression, by completing header */
|
||||
ZSTD_resetDCtx(dctx);
|
||||
toRead = ZSTD_nextSrcSizeToDecompress(dctx) - sizeof(ZSTD_magicNumber);
|
||||
if (toRead > MAXHEADERSIZE) EXM_THROW(30, "Not enough memory to read header");
|
||||
sizeCheck = fread(header+sizeof(ZSTD_magicNumber), (size_t)1, toRead, finput);
|
||||
sizeCheck = fread(&header[sizeof(ZSTD_magicNumber)], 1, toRead, finput);
|
||||
if (sizeCheck != toRead) EXM_THROW(31, "Read error : cannot read header");
|
||||
sizeCheck = ZSTD_decompressContinue(dctx, NULL, 0, header, sizeof(ZSTD_magicNumber)+toRead); // Decode frame header
|
||||
if (ZSTD_isError(sizeCheck)) EXM_THROW(32, "Error decoding header");
|
||||
|
||||
@ -29,7 +29,7 @@ extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Special i/o constants
|
||||
**************************************/
|
||||
#define nullString "null"
|
||||
@ -42,28 +42,28 @@ extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Parameters
|
||||
**************************************/
|
||||
***************************************/
|
||||
void FIO_overwriteMode(void);
|
||||
void FIO_setNotificationLevel(unsigned level);
|
||||
|
||||
|
||||
/**************************************
|
||||
/* *************************************
|
||||
* Stream/File functions
|
||||
**************************************/
|
||||
unsigned long long FIO_compressFilename (const char* outfilename, const char* infilename);
|
||||
***************************************/
|
||||
unsigned long long FIO_compressFilename (const char* outfilename, const char* infilename, int compressionLevel);
|
||||
unsigned long long FIO_decompressFilename (const char* outfilename, const char* infilename);
|
||||
/*
|
||||
/**
|
||||
FIO_compressFilename :
|
||||
result : size of compressed file
|
||||
@result : size of compressed file
|
||||
|
||||
FIO_decompressFilename :
|
||||
result : size of regenerated file
|
||||
@result : size of regenerated file
|
||||
*/
|
||||
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
#endif
|
||||
|
||||
@ -78,7 +78,7 @@
|
||||
/**************************************
|
||||
* Constants
|
||||
**************************************/
|
||||
#define PROGRAM_DESCRIPTION "zStandard speed analyzer"
|
||||
#define PROGRAM_DESCRIPTION "Zstandard speed analyzer"
|
||||
#ifndef ZSTD_VERSION
|
||||
# define ZSTD_VERSION ""
|
||||
#endif
|
||||
|
||||
@ -47,6 +47,7 @@
|
||||
#include <sys/timeb.h> /* timeb */
|
||||
#include <string.h> /* strcmp */
|
||||
#include "zstd_static.h"
|
||||
#include "zstdhc_static.h"
|
||||
#include "datagen.h" /* RDG_genBuffer */
|
||||
#include "xxhash.h" /* XXH64 */
|
||||
#include "mem.h"
|
||||
@ -263,8 +264,12 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
U32 result = 0;
|
||||
U32 testNb = 0;
|
||||
U32 coreSeed = seed, lseed = 0;
|
||||
ZSTD_CCtx* ctx;
|
||||
ZSTD_HC_CCtx* hcctx;
|
||||
|
||||
/* allocation */
|
||||
ctx = ZSTD_createCCtx();
|
||||
hcctx = ZSTD_HC_createCCtx();
|
||||
cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
|
||||
cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
|
||||
cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
|
||||
@ -272,7 +277,7 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
|
||||
dstBuffer = (BYTE*)malloc (dstBufferSize);
|
||||
cBuffer = (BYTE*)malloc (cBufferSize);
|
||||
CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !dstBuffer || !cBuffer,
|
||||
CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !dstBuffer || !cBuffer || !ctx || !hcctx,
|
||||
"Not enough memory, fuzzer tests cancelled");
|
||||
|
||||
/* Create initial samples */
|
||||
@ -294,6 +299,7 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
size_t cSize, dSize, dSupSize;
|
||||
U32 sampleSizeLog, buffNb;
|
||||
U64 crcOrig, crcDest;
|
||||
int cLevel;
|
||||
|
||||
/* init */
|
||||
DISPLAYUPDATE(2, "\r%6u/%6u ", testNb, nbTests);
|
||||
@ -322,8 +328,13 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
sampleStart = FUZ_rand(&lseed) % (srcBufferSize - sampleSize);
|
||||
crcOrig = XXH64(srcBuffer + sampleStart, sampleSize, 0);
|
||||
|
||||
/* HC compression test */
|
||||
cLevel = (FUZ_rand(&lseed) & 3) + 2;
|
||||
cSize = ZSTD_HC_compressCCtx(hcctx, cBuffer, cBufferSize, srcBuffer + sampleStart, sampleSize, cLevel);
|
||||
CHECK(ZSTD_isError(cSize), "ZSTD_compress failed");
|
||||
|
||||
/* compression test */
|
||||
cSize = ZSTD_compress(cBuffer, cBufferSize, srcBuffer + sampleStart, sampleSize);
|
||||
cSize = ZSTD_compressCCtx(ctx, cBuffer, cBufferSize, srcBuffer + sampleStart, sampleSize);
|
||||
CHECK(ZSTD_isError(cSize), "ZSTD_compress failed");
|
||||
|
||||
/* compression failure test : too small dest buffer */
|
||||
@ -335,7 +346,7 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
static const U32 endMark = 0x4DC2B1A9;
|
||||
U32 endCheck;
|
||||
memcpy(dstBuffer+tooSmallSize, &endMark, 4);
|
||||
errorCode = ZSTD_compress(dstBuffer, tooSmallSize, srcBuffer + sampleStart, sampleSize);
|
||||
errorCode = ZSTD_compressCCtx(ctx, dstBuffer, tooSmallSize, srcBuffer + sampleStart, sampleSize);
|
||||
CHECK(!ZSTD_isError(errorCode), "ZSTD_compress should have failed ! (buffer too small)");
|
||||
memcpy(&endCheck, dstBuffer+tooSmallSize, 4);
|
||||
CHECK(endCheck != endMark, "ZSTD_compress : dst buffer overflow");
|
||||
@ -424,6 +435,8 @@ int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibilit
|
||||
DISPLAY("\rAll fuzzer tests completed \n");
|
||||
|
||||
_cleanup:
|
||||
ZSTD_freeCCtx(ctx);
|
||||
ZSTD_HC_freeCCtx(hcctx);
|
||||
free(cNoiseBuffer[0]);
|
||||
free(cNoiseBuffer[1]);
|
||||
free(cNoiseBuffer[2]);
|
||||
|
||||
284
programs/legacy/fileio_legacy.c
Normal file
284
programs/legacy/fileio_legacy.c
Normal file
@ -0,0 +1,284 @@
|
||||
/*
|
||||
fileio.c - File i/o handler
|
||||
Copyright (C) Yann Collet 2013-2015
|
||||
|
||||
GPL v2 License
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
You can contact the author at :
|
||||
- zstd source repository : https://github.com/Cyan4973/zstd
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
/*
|
||||
Note : this is stand-alone program.
|
||||
It is not part of ZSTD compression library, it is a user program of ZSTD library.
|
||||
The license of ZSTD library is BSD.
|
||||
The license of this file is GPLv2.
|
||||
*/
|
||||
|
||||
/* *************************************
|
||||
* Compiler Options
|
||||
***************************************/
|
||||
/* Disable some Visual warning messages */
|
||||
#ifdef _MSC_VER
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define _CRT_SECURE_NO_DEPRECATE /* VS2005 */
|
||||
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
|
||||
#endif
|
||||
|
||||
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
|
||||
|
||||
#define _FILE_OFFSET_BITS 64 /* Large file support on 32-bits unix */
|
||||
#define _POSIX_SOURCE 1 /* enable fileno() within <stdio.h> on unix */
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Includes
|
||||
***************************************/
|
||||
#include <stdio.h> /* fprintf, fopen, fread, _fileno, stdin, stdout */
|
||||
#include <stdlib.h> /* malloc, free */
|
||||
#include <string.h> /* strcmp, strlen */
|
||||
#include <time.h> /* clock */
|
||||
#include <errno.h> /* errno */
|
||||
#include "mem.h"
|
||||
#include "fileio_legacy.h"
|
||||
#include "zstd_legacy.h" /* legacy support */
|
||||
|
||||
|
||||
/* *************************************
|
||||
* OS-specific Includes
|
||||
***************************************/
|
||||
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__)
|
||||
# include <fcntl.h> /* _O_BINARY */
|
||||
# include <io.h> /* _setmode, _isatty */
|
||||
# ifdef __MINGW32__
|
||||
/* int _fileno(FILE *stream); // seems no longer useful // MINGW somehow forgets to include this windows declaration into <stdio.h> */
|
||||
# endif
|
||||
# define SET_BINARY_MODE(file) { int unused = _setmode(_fileno(file), _O_BINARY); (void)unused; }
|
||||
# define IS_CONSOLE(stdStream) _isatty(_fileno(stdStream))
|
||||
#else
|
||||
# include <unistd.h> /* isatty */
|
||||
# define SET_BINARY_MODE(file)
|
||||
# define IS_CONSOLE(stdStream) isatty(fileno(stdStream))
|
||||
#endif
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Constants
|
||||
***************************************/
|
||||
#define KB *(1U<<10)
|
||||
#define MB *(1U<<20)
|
||||
#define GB *(1U<<30)
|
||||
|
||||
#define _1BIT 0x01
|
||||
#define _2BITS 0x03
|
||||
#define _3BITS 0x07
|
||||
#define _4BITS 0x0F
|
||||
#define _6BITS 0x3F
|
||||
#define _8BITS 0xFF
|
||||
|
||||
#define BIT6 0x40
|
||||
#define BIT7 0x80
|
||||
|
||||
#define FIO_FRAMEHEADERSIZE 5 /* as a define, because needed to allocated table on stack */
|
||||
#define FSE_CHECKSUM_SEED 0
|
||||
|
||||
#define CACHELINE 64
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Macros
|
||||
***************************************/
|
||||
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
|
||||
#define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
|
||||
static U32 g_displayLevel = 1; /* 0 : no display; 1: errors; 2 : + result + interaction + warnings; 3 : + progression; 4 : + information */
|
||||
|
||||
#define DISPLAYUPDATE(l, ...) if (g_displayLevel>=l) { \
|
||||
if ((FIO_GetMilliSpan(g_time) > refreshRate) || (g_displayLevel>=4)) \
|
||||
{ g_time = clock(); DISPLAY(__VA_ARGS__); \
|
||||
if (g_displayLevel>=4) fflush(stdout); } }
|
||||
static const unsigned refreshRate = 150;
|
||||
static clock_t g_time = 0;
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Local Parameters
|
||||
***************************************/
|
||||
void FIO_legacy_setNotificationLevel(unsigned level) { g_displayLevel=level; }
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Exceptions
|
||||
***************************************/
|
||||
#ifndef DEBUG
|
||||
# define DEBUG 0
|
||||
#endif
|
||||
#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
|
||||
#define EXM_THROW(error, ...) \
|
||||
{ \
|
||||
DEBUGOUTPUT("Error defined at %s, line %i : \n", __FILE__, __LINE__); \
|
||||
DISPLAYLEVEL(1, "Error %i : ", error); \
|
||||
DISPLAYLEVEL(1, __VA_ARGS__); \
|
||||
DISPLAYLEVEL(1, "\n"); \
|
||||
exit(error); \
|
||||
}
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Functions
|
||||
***************************************/
|
||||
static unsigned FIO_GetMilliSpan(clock_t nPrevious)
|
||||
{
|
||||
clock_t nCurrent = clock();
|
||||
unsigned nSpan = (unsigned)(((nCurrent - nPrevious) * 1000) / CLOCKS_PER_SEC);
|
||||
return nSpan;
|
||||
}
|
||||
|
||||
|
||||
unsigned long long FIOv01_decompressFrame(FILE* foutput, FILE* finput)
|
||||
{
|
||||
size_t outBuffSize = 512 KB;
|
||||
BYTE* outBuff = (BYTE*)malloc(outBuffSize);
|
||||
size_t inBuffSize = 128 KB + 8;
|
||||
BYTE inBuff[128 KB + 8];
|
||||
BYTE* op = outBuff;
|
||||
BYTE* const oend = outBuff + outBuffSize;
|
||||
U64 filesize = 0;
|
||||
size_t toRead;
|
||||
size_t sizeCheck;
|
||||
ZSTDv01_Dctx* dctx = ZSTDv01_createDCtx();
|
||||
|
||||
|
||||
/* init */
|
||||
if (outBuff==NULL) EXM_THROW(41, "Error : not enough memory to decode legacy frame");
|
||||
|
||||
/* restore header, already read from input */
|
||||
MEM_writeLE32(inBuff, ZSTDv01_magicNumberLE);
|
||||
sizeCheck = ZSTDv01_decompressContinue(dctx, NULL, 0, inBuff, sizeof(ZSTDv01_magicNumberLE)); /* Decode frame header */
|
||||
if (ZSTDv01_isError(sizeCheck)) EXM_THROW(42, "Error decoding legacy header");
|
||||
|
||||
/* Main decompression Loop */
|
||||
toRead = ZSTDv01_nextSrcSizeToDecompress(dctx);
|
||||
while (toRead)
|
||||
{
|
||||
size_t readSize, decodedSize;
|
||||
|
||||
/* Fill input buffer */
|
||||
if (toRead > inBuffSize)
|
||||
EXM_THROW(43, "too large block");
|
||||
readSize = fread(inBuff, 1, toRead, finput);
|
||||
if (readSize != toRead)
|
||||
EXM_THROW(44, "Read error");
|
||||
|
||||
/* Decode block */
|
||||
decodedSize = ZSTDv01_decompressContinue(dctx, op, oend-op, inBuff, readSize);
|
||||
if (ZSTDv01_isError(decodedSize)) EXM_THROW(45, "Decoding error : input corrupted");
|
||||
|
||||
if (decodedSize) /* not a header */
|
||||
{
|
||||
/* Write block */
|
||||
sizeCheck = fwrite(op, 1, decodedSize, foutput);
|
||||
if (sizeCheck != decodedSize) EXM_THROW(46, "Write error : unable to write data block to destination file");
|
||||
filesize += decodedSize;
|
||||
op += decodedSize;
|
||||
if (op==oend) op = outBuff;
|
||||
DISPLAYUPDATE(2, "\rDecoded : %u MB... ", (U32)(filesize>>20) );
|
||||
}
|
||||
|
||||
/* prepare for next Block */
|
||||
toRead = ZSTDv01_nextSrcSizeToDecompress(dctx);
|
||||
}
|
||||
|
||||
/* release resources */
|
||||
free(outBuff);
|
||||
free(dctx);
|
||||
return filesize;
|
||||
}
|
||||
|
||||
|
||||
unsigned long long FIOv02_decompressFrame(FILE* foutput, FILE* finput)
|
||||
{
|
||||
size_t outBuffSize = 512 KB;
|
||||
BYTE* outBuff = (BYTE*)malloc(outBuffSize);
|
||||
size_t inBuffSize = 128 KB + 8;
|
||||
BYTE inBuff[128 KB + 8];
|
||||
BYTE* op = outBuff;
|
||||
BYTE* const oend = outBuff + outBuffSize;
|
||||
U64 filesize = 0;
|
||||
size_t toRead;
|
||||
size_t sizeCheck;
|
||||
ZSTDv02_Dctx* dctx = ZSTDv02_createDCtx();
|
||||
|
||||
|
||||
/* init */
|
||||
if (outBuff==NULL) EXM_THROW(41, "Error : not enough memory to decode legacy frame");
|
||||
|
||||
/* restore header, already read from input */
|
||||
MEM_writeLE32(inBuff, ZSTDv02_magicNumber);
|
||||
sizeCheck = ZSTDv02_decompressContinue(dctx, NULL, 0, inBuff, sizeof(ZSTDv02_magicNumber)); /* Decode frame header */
|
||||
if (ZSTDv02_isError(sizeCheck)) EXM_THROW(42, "Error decoding legacy header");
|
||||
|
||||
/* Main decompression Loop */
|
||||
toRead = ZSTDv02_nextSrcSizeToDecompress(dctx);
|
||||
while (toRead)
|
||||
{
|
||||
size_t readSize, decodedSize;
|
||||
|
||||
/* Fill input buffer */
|
||||
if (toRead > inBuffSize)
|
||||
EXM_THROW(43, "too large block");
|
||||
readSize = fread(inBuff, 1, toRead, finput);
|
||||
if (readSize != toRead)
|
||||
EXM_THROW(44, "Read error");
|
||||
|
||||
/* Decode block */
|
||||
decodedSize = ZSTDv02_decompressContinue(dctx, op, oend-op, inBuff, readSize);
|
||||
if (ZSTDv01_isError(decodedSize)) EXM_THROW(45, "Decoding error : input corrupted");
|
||||
|
||||
if (decodedSize) /* not a header */
|
||||
{
|
||||
/* Write block */
|
||||
sizeCheck = fwrite(op, 1, decodedSize, foutput);
|
||||
if (sizeCheck != decodedSize) EXM_THROW(46, "Write error : unable to write data block to destination file");
|
||||
filesize += decodedSize;
|
||||
op += decodedSize;
|
||||
if (op==oend) op = outBuff;
|
||||
DISPLAYUPDATE(2, "\rDecoded : %u MB... ", (U32)(filesize>>20) );
|
||||
}
|
||||
|
||||
/* prepare for next Block */
|
||||
toRead = ZSTDv02_nextSrcSizeToDecompress(dctx);
|
||||
}
|
||||
|
||||
/* release resources */
|
||||
free(outBuff);
|
||||
free(dctx);
|
||||
return filesize;
|
||||
}
|
||||
|
||||
|
||||
unsigned long long FIO_decompressLegacyFrame(FILE* foutput, FILE* finput, U32 magicNumberLE)
|
||||
{
|
||||
switch(magicNumberLE)
|
||||
{
|
||||
case ZSTDv01_magicNumberLE :
|
||||
return FIOv01_decompressFrame(foutput, finput);
|
||||
case ZSTDv02_magicNumber :
|
||||
return FIOv02_decompressFrame(foutput, finput);
|
||||
default :
|
||||
return ERROR(prefix_unknown);
|
||||
}
|
||||
}
|
||||
46
programs/legacy/fileio_legacy.h
Normal file
46
programs/legacy/fileio_legacy.h
Normal file
@ -0,0 +1,46 @@
|
||||
/*
|
||||
fileio.h - file i/o handler
|
||||
Copyright (C) Yann Collet 2013-2015
|
||||
|
||||
GPL v2 License
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
You can contact the author at :
|
||||
- ZSTD source repository : https://github.com/Cyan4973/zstd
|
||||
- Public forum : https://groups.google.com/forum/#!forum/lz4c
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Parameters
|
||||
***************************************/
|
||||
void FIO_legacy_setNotificationLevel(unsigned level);
|
||||
|
||||
|
||||
/* *************************************
|
||||
* Stream/File functions
|
||||
***************************************/
|
||||
unsigned long long FIO_decompressLegacyFrame(FILE* foutput, FILE* finput, U32 magicNumberLE);
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
1025
programs/paramgrill.c
Normal file
1025
programs/paramgrill.c
Normal file
File diff suppressed because it is too large
Load Diff
@ -118,6 +118,8 @@ static int usage(const char* programName)
|
||||
DISPLAY( "input : a filename\n");
|
||||
DISPLAY( " with no FILE, or when FILE is - , read standard input\n");
|
||||
DISPLAY( "Arguments :\n");
|
||||
DISPLAY( " -1 : Fast compression (default) \n");
|
||||
DISPLAY( " -9 : High compression \n");
|
||||
DISPLAY( " -d : decompression (default for %s extension)\n", ZSTD_EXTENSION);
|
||||
//DISPLAY( " -z : force compression\n");
|
||||
DISPLAY( " -f : overwrite output without prompting \n");
|
||||
@ -137,9 +139,10 @@ static int usage_advanced(const char* programName)
|
||||
DISPLAY( " -c : force write to standard output, even if it is the console\n");
|
||||
//DISPLAY( " -t : test compressed file integrity\n");
|
||||
DISPLAY( "Benchmark arguments :\n");
|
||||
DISPLAY( " -b : benchmark file(s)\n");
|
||||
DISPLAY( " -b# : benchmark file(s), using # compression level (default : 1) \n");
|
||||
DISPLAY( " -B# : cut file into independent blocks of size # (default : no block)\n");
|
||||
DISPLAY( " -i# : iteration loops [1-9](default : 3)\n");
|
||||
DISPLAY( " -r# : test all compression levels from 1 to # (default : disabled)\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -166,9 +169,11 @@ int main(int argc, char** argv)
|
||||
bench=0,
|
||||
decode=0,
|
||||
forceStdout=0,
|
||||
main_pause=0;
|
||||
main_pause=0,
|
||||
rangeBench = 1;
|
||||
unsigned fileNameStart = 0;
|
||||
unsigned nbFiles = 0;
|
||||
unsigned cLevel = 1;
|
||||
const char* programName = argv[0];
|
||||
const char* inFileName = NULL;
|
||||
const char* outFileName = NULL;
|
||||
@ -217,6 +222,19 @@ int main(int argc, char** argv)
|
||||
|
||||
while (argument[0]!=0)
|
||||
{
|
||||
/* compression Level */
|
||||
if ((*argument>='0') && (*argument<='9'))
|
||||
{
|
||||
cLevel = 0;
|
||||
while ((*argument >= '0') && (*argument <= '9'))
|
||||
{
|
||||
cLevel *= 10;
|
||||
cLevel += *argument - '0';
|
||||
argument++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
switch(argument[0])
|
||||
{
|
||||
/* Display help */
|
||||
@ -275,7 +293,14 @@ int main(int argc, char** argv)
|
||||
BMK_SetBlockSize(bSize);
|
||||
}
|
||||
break;
|
||||
/* Pause at the end (hidden option) */
|
||||
|
||||
/* range bench (benchmark only) */
|
||||
case 'r':
|
||||
rangeBench = -1;
|
||||
argument++;
|
||||
break;
|
||||
|
||||
/* Pause at the end (hidden option) */
|
||||
case 'p': main_pause=1; argument++; break;
|
||||
|
||||
/* unknown command */
|
||||
@ -307,7 +332,7 @@ int main(int argc, char** argv)
|
||||
if (!strcmp(inFileName, stdinmark) && IS_CONSOLE(stdin) ) return badusage(programName);
|
||||
|
||||
/* Check if benchmark is selected */
|
||||
if (bench) { BMK_benchFiles(argv+fileNameStart, nbFiles, 0); goto _end; }
|
||||
if (bench) { BMK_benchFiles(argv+fileNameStart, nbFiles, cLevel*rangeBench); goto _end; }
|
||||
|
||||
/* No output filename ==> try to select one automatically (when possible) */
|
||||
while (!outFileName)
|
||||
@ -351,7 +376,7 @@ int main(int argc, char** argv)
|
||||
if (decode)
|
||||
FIO_decompressFilename(outFileName, inFileName);
|
||||
else
|
||||
FIO_compressFilename(outFileName, inFileName);
|
||||
FIO_compressFilename(outFileName, inFileName, cLevel);
|
||||
|
||||
_end:
|
||||
if (main_pause) waitEnter();
|
||||
|
||||
@ -83,7 +83,7 @@
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
|
||||
<LinkIncremental>false</LinkIncremental>
|
||||
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<IncludePath>$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<RunCodeAnalysis>true</RunCodeAnalysis>
|
||||
</PropertyGroup>
|
||||
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
|
||||
@ -162,6 +162,7 @@
|
||||
<ClCompile Include="..\..\..\lib\fse.c" />
|
||||
<ClCompile Include="..\..\..\lib\huff0.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v01.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c" />
|
||||
<ClCompile Include="..\..\..\lib\zstd.c" />
|
||||
<ClCompile Include="..\..\..\programs\datagen.c" />
|
||||
<ClCompile Include="..\..\..\programs\fullbench.c" />
|
||||
@ -171,7 +172,9 @@
|
||||
<ClInclude Include="..\..\..\lib\fse_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v01.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd_static.h" />
|
||||
<ClInclude Include="..\..\..\programs\datagen.h" />
|
||||
|
||||
@ -33,6 +33,9 @@
|
||||
<ClCompile Include="..\..\..\lib\huff0.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ClInclude Include="..\..\..\lib\fse.h">
|
||||
@ -59,5 +62,11 @@
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@ -83,7 +83,7 @@
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
|
||||
<LinkIncremental>false</LinkIncremental>
|
||||
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<IncludePath>$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<RunCodeAnalysis>true</RunCodeAnalysis>
|
||||
</PropertyGroup>
|
||||
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
|
||||
@ -162,7 +162,9 @@
|
||||
<ClCompile Include="..\..\..\lib\fse.c" />
|
||||
<ClCompile Include="..\..\..\lib\huff0.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v01.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c" />
|
||||
<ClCompile Include="..\..\..\lib\zstd.c" />
|
||||
<ClCompile Include="..\..\..\lib\zstdhc.c" />
|
||||
<ClCompile Include="..\..\..\programs\datagen.c" />
|
||||
<ClCompile Include="..\..\..\programs\fuzzer.c" />
|
||||
<ClCompile Include="..\..\..\programs\xxhash.c" />
|
||||
@ -172,8 +174,13 @@
|
||||
<ClInclude Include="..\..\..\lib\fse_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v01.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstdhc.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstdhc_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd_internal.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd_static.h" />
|
||||
<ClInclude Include="..\..\..\programs\datagen.h" />
|
||||
<ClInclude Include="..\..\..\programs\xxhash.h" />
|
||||
|
||||
@ -36,6 +36,12 @@
|
||||
<ClCompile Include="..\..\..\lib\huff0.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\lib\zstdhc.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ClInclude Include="..\..\..\lib\fse.h">
|
||||
@ -65,5 +71,20 @@
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstd_internal.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstdhc.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstdhc_static.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@ -22,9 +22,12 @@
|
||||
<ClCompile Include="..\..\..\lib\fse.c" />
|
||||
<ClCompile Include="..\..\..\lib\huff0.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v01.c" />
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c" />
|
||||
<ClCompile Include="..\..\..\lib\zstd.c" />
|
||||
<ClCompile Include="..\..\..\lib\zstdhc.c" />
|
||||
<ClCompile Include="..\..\..\programs\bench.c" />
|
||||
<ClCompile Include="..\..\..\programs\fileio.c" />
|
||||
<ClCompile Include="..\..\..\programs\legacy\fileio_legacy.c" />
|
||||
<ClCompile Include="..\..\..\programs\xxhash.c" />
|
||||
<ClCompile Include="..\..\..\programs\zstdcli.c" />
|
||||
</ItemGroup>
|
||||
@ -33,11 +36,17 @@
|
||||
<ClInclude Include="..\..\..\lib\fse_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0.h" />
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v01.h" />
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstdhc.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstdhc_static.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd_internal.h" />
|
||||
<ClInclude Include="..\..\..\lib\zstd_static.h" />
|
||||
<ClInclude Include="..\..\..\programs\bench.h" />
|
||||
<ClInclude Include="..\..\..\programs\fileio.h" />
|
||||
<ClInclude Include="..\..\..\programs\legacy\fileio_legacy.h" />
|
||||
<ClInclude Include="..\..\..\programs\xxhash.h" />
|
||||
</ItemGroup>
|
||||
<PropertyGroup Label="Globals">
|
||||
@ -90,7 +99,7 @@
|
||||
<PropertyGroup Label="UserMacros" />
|
||||
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
|
||||
<LinkIncremental>true</LinkIncremental>
|
||||
<IncludePath>$(SolutionDir)..\..\lib;$(SolutionDir)..\..\lib\legacy;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<IncludePath>$(SolutionDir)..\..\programs\legacy;$(SolutionDir)..\..\lib;$(SolutionDir)..\..\lib\legacy;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<RunCodeAnalysis>true</RunCodeAnalysis>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
|
||||
@ -105,7 +114,7 @@
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
|
||||
<LinkIncremental>false</LinkIncremental>
|
||||
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<IncludePath>$(SolutionDir)..\..\programs\legacy;$(SolutionDir)..\..\lib\legacy;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
|
||||
<RunCodeAnalysis>true</RunCodeAnalysis>
|
||||
</PropertyGroup>
|
||||
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
|
||||
@ -172,6 +181,7 @@
|
||||
<TreatWarningAsError>true</TreatWarningAsError>
|
||||
<EnablePREfast>true</EnablePREfast>
|
||||
<AdditionalOptions>/analyze:stacksize19000 %(AdditionalOptions)</AdditionalOptions>
|
||||
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
|
||||
</ClCompile>
|
||||
<Link>
|
||||
<SubSystem>Console</SubSystem>
|
||||
|
||||
@ -39,6 +39,15 @@
|
||||
<ClCompile Include="..\..\..\lib\huff0.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\lib\zstdhc.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\lib\legacy\zstd_v02.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="..\..\..\programs\legacy\fileio_legacy.c">
|
||||
<Filter>Fichiers sources</Filter>
|
||||
</ClCompile>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ClInclude Include="..\..\..\lib\fse.h">
|
||||
@ -71,5 +80,23 @@
|
||||
<ClInclude Include="..\..\..\lib\huff0_static.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstd_internal.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstdhc.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\zstdhc_static.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_legacy.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\lib\legacy\zstd_v02.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="..\..\..\programs\legacy\fileio_legacy.h">
|
||||
<Filter>Fichiers d%27en-tête</Filter>
|
||||
</ClInclude>
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
Loading…
Reference in New Issue
Block a user