Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev
This commit is contained in:
inikep
commit
c70c90e8a3
75
README.md
75
README.md
@ -16,7 +16,7 @@ As a reference, several fast compression algorithms were tested and compared to
|
||||
|Name | Ratio | C.speed | D.speed |
|
||||
|-----------------|-------|--------:|--------:|
|
||||
| | | MB/s | MB/s |
|
||||
|**zstd 0.5.1 -1**|**2.876**|**330**| **890** |
|
||||
|**zstd 0.6.0 -1**|**2.877**|**330**| **915** |
|
||||
| [zlib] 1.2.8 -1 | 2.730 | 95 | 360 |
|
||||
| brotli -0 | 2.708 | 220 | 430 |
|
||||
| QuickLZ 1.5 | 2.237 | 510 | 605 |
|
||||
@ -43,33 +43,18 @@ For a larger picture including very slow modes, [click on this link](images/DCsp
|
||||
|
||||
### The case for Small Data compression
|
||||
|
||||
Above chart provides results applicable to large files or large streams scenarios (200 MB for this case).
|
||||
Small data (< 64 KB) come with different perspectives.
|
||||
The smaller the amount of data to compress, the more difficult it is to achieve any significant compression.
|
||||
On reaching the 1 KB region, it becomes almost impossible to compress anything.
|
||||
This problem is common to any compression algorithms, and throwing CPU power at it achieves little gains.
|
||||
Previous charts provide results applicable to typical files and streams scenarios (several MB). Small data come with different perspectives. The smaller the amount of data to compress, the more difficult it is to achieve any significant compression.
|
||||
|
||||
The reason is, compression algorithms learn from past data how to compress future data.
|
||||
But at the beginning of a new file, there is no "past" to build upon.
|
||||
This problem is common to any compression algorithm. The reason is, compression algorithms learn from past data how to compress future data. But at the beginning of a new file, there is no "past" to build upon.
|
||||
|
||||
To solve this situation, Zstd now offers a __training mode__,
|
||||
which can be used to make the algorithm fit a selected type of data, by providing it with some samples.
|
||||
The result of the training is a file called "dictionary", which can be loaded before compression and decompression.
|
||||
Using this dictionary, the compression ratio achievable on small data improves dramatically :
|
||||
To solve this situation, Zstd offers a __training mode__, which can be used to tune the algorithm for a selected type of data, by providing it with a few samples. The result of the training is stored in a file called "dictionary", which can be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically :
|
||||
|
||||
| Collection Name | Direct compression | Dictionary Compression | Gains | Average unit | Range |
|
||||
| --------------- | ------------------ | ---------------------- | --------- | ------------:| ----- |
|
||||
| Small JSON records | x1.331 - x1.366 | x5.860 - x6.830 | ~ __x4.7__ | 300 | 200 - 400 |
|
||||
| Mercurial events | x2.322 - x2.538 | x3.377 - x4.462 | ~ __x1.5__ | 1.5 KB | 20 - 200 KB |
|
||||
| Large JSON docs | x3.813 - x4.043 | x8.935 - x13.366 | ~ __x2.8__ | 6 KB | 800 - 20 KB |
|
||||
|
||||
|
||||
These compression gains are achieved without any speed loss, and prove in general a bit faster to compress and decompress.
|
||||
These compression gains are achieved while simultaneously providing faster compression and decompression speeds.
|
||||
|
||||
Dictionary work if there is some correlation in a family of small data (there is no _universal dictionary_).
|
||||
Hence, deploying one dictionary per type of data will provide the greater benefits.
|
||||
|
||||
Large documents will benefit proportionally less, since dictionary gains are mostly effective in the first few KB.
|
||||
Then, the compression algorithm will rely more and more on already decoded content to compress the rest of the file.
|
||||
Hence, deploying one dictionary per type of data will provide the greater benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will rely more and more on previously decoded content to compress the rest of the file.
|
||||
|
||||
#### Dictionary compression How To :
|
||||
|
||||
@ -79,7 +64,7 @@ Then, the compression algorithm will rely more and more on already decoded conte
|
||||
|
||||
`zstd --train FullPathToTrainingSet/* -o dictionaryName`
|
||||
|
||||
2) Compression with dictionary
|
||||
2) Compress with dictionary
|
||||
|
||||
`zstd FILE -D dictionaryName`
|
||||
|
||||
@ -87,53 +72,15 @@ Then, the compression algorithm will rely more and more on already decoded conte
|
||||
|
||||
`zstd --decompress FILE.zst -D dictionaryName`
|
||||
|
||||
##### _Using API_ :
|
||||
|
||||
1) Create dictionary
|
||||
|
||||
```
|
||||
#include "zdict.h"
|
||||
(...)
|
||||
/* Train a dictionary from a memory buffer `samplesBuffer`,
|
||||
where `nbSamples` samples have been stored concatenated. */
|
||||
size_t dictSize = ZDICT_trainFromBuffer(dictBuffer, dictBufferCapacity,
|
||||
samplesBuffer, samplesSizes, nbSamples);
|
||||
```
|
||||
|
||||
2) Compression with dictionary
|
||||
|
||||
```
|
||||
#include "zstd.h"
|
||||
(...)
|
||||
ZSTD_CCtx* context = ZSTD_createCCtx();
|
||||
size_t compressedSize = ZSTD_compress_usingDict(context, dst, dstCapacity, src, srcSize, dictBuffer, dictSize, compressionLevel);
|
||||
```
|
||||
|
||||
3) Decompress with dictionary
|
||||
|
||||
```
|
||||
#include "zstd.h"
|
||||
(...)
|
||||
ZSTD_DCtx* context = ZSTD_createDCtx();
|
||||
size_t regeneratedSize = ZSTD_decompress_usingDict(context, dst, dstCapacity, cSrc, cSrcSize, dictBuffer, dictSize);
|
||||
```
|
||||
|
||||
|
||||
### Status
|
||||
|
||||
Zstd has not yet reached "stable format" status. It doesn't guarantee yet that its current compression format will remain stable in future versions. During this period, it can still change to adapt new optimizations still being investigated. "Stable Format" is projected H1 2016, and will be tagged `v1.0`.
|
||||
|
||||
That being said, the library is now fairly robust, able to withstand hazards situations, including invalid inputs. It also features legacy support, so that documents compressed with current and previous version of zstd can still be decoded in the future.
|
||||
Library reliability has been tested using [Fuzz Testing](https://en.wikipedia.org/wiki/Fuzz_testing), with both [internal tools](programs/fuzzer.c) and [external ones](http://lcamtuf.coredump.cx/afl). Therefore, Zstandard is considered safe for testings, even within production environments.
|
||||
Zstd is in development. The internal format evolves to reach better performance. "Final Format" is projected H1 2016, and will be tagged `v1.0`. Zstd offers legacy support, meaning any data compressed by any version >= 0.1 (therefore including current one) remain decodable in the future.
|
||||
The library is also quite robust, able to withstand hazards situations, including invalid inputs. Library reliability has been tested using [Fuzz Testing](https://en.wikipedia.org/wiki/Fuzz_testing), with both [internal tools](programs/fuzzer.c) and [external ones](http://lcamtuf.coredump.cx/afl). Therefore, Zstandard is considered safe for production environments.
|
||||
|
||||
### Branch Policy
|
||||
|
||||
The "dev" branch is the one where all contributions will be merged before reaching "master". If you plan to propose a patch, please commit into the "dev" branch or its own feature branch. Direct commit to "master" are not permitted.
|
||||
|
||||
|
||||
### Trivia
|
||||
### Miscellaneous
|
||||
|
||||
Zstd entropy stage is provided by [Huff0 and FSE, from Finite State Entropy library](https://github.com/Cyan4973/FiniteStateEntropy).
|
||||
|
||||
Its memory requirement can be configured to fit into low-memory hardware configurations, or servers handling multiple connections/contexts in parallel.
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 68 KiB After Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.3 KiB After Width: | Height: | Size: 9.7 KiB |
BIN
images/smallData.png
Normal file
BIN
images/smallData.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
@ -51,7 +51,7 @@ FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MOREFLAGS)
|
||||
LIBDIR ?= $(PREFIX)/lib
|
||||
INCLUDEDIR=$(PREFIX)/include
|
||||
|
||||
ZSTD_FILES := zstd_compress.c zstd_decompress.c fse.c huff0.c zdict.c divsufsort.c
|
||||
ZSTD_FILES := zstd_compress.c zstd_decompress.c fse.c huff0.c zbuff.c zdict.c divsufsort.c
|
||||
ZSTD_LEGACY:= legacy/zstd_v01.c legacy/zstd_v02.c legacy/zstd_v03.c legacy/zstd_v04.c legacy/zstd_v05.c
|
||||
|
||||
ifeq ($(ZSTD_LEGACY_SUPPORT), 0)
|
||||
|
||||
@ -98,13 +98,11 @@ struct ZSTD_CCtx_s
|
||||
U32 nextToUpdate3; /* index from which to continue dictionary update */
|
||||
U32 hashLog3; /* dispatch table : larger == faster, more memory */
|
||||
U32 loadedDictEnd;
|
||||
U32 stage;
|
||||
U32 stage; /* 0: created; 1: init,dictLoad; 2:started */
|
||||
ZSTD_parameters params;
|
||||
void* workSpace;
|
||||
size_t workSpaceSize;
|
||||
size_t blockSize;
|
||||
size_t hbSize;
|
||||
char headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
|
||||
|
||||
seqStore_t seqStore; /* sequences storage ptrs */
|
||||
U32* hashTable;
|
||||
@ -213,7 +211,7 @@ size_t ZSTD_sizeofCCtx(ZSTD_compressionParameters cParams) /* hidden interface
|
||||
/*! ZSTD_resetCCtx_advanced() :
|
||||
note : 'params' is expected to be validated */
|
||||
static size_t ZSTD_resetCCtx_advanced (ZSTD_CCtx* zc,
|
||||
ZSTD_parameters params)
|
||||
ZSTD_parameters params, U32 reset)
|
||||
{ /* note : params considered validated here */
|
||||
const size_t blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << params.cParams.windowLog);
|
||||
const U32 divider = (params.cParams.searchLength==3) ? 3 : 4;
|
||||
@ -236,7 +234,7 @@ static size_t ZSTD_resetCCtx_advanced (ZSTD_CCtx* zc,
|
||||
zc->workSpaceSize = neededSpace;
|
||||
} }
|
||||
|
||||
memset(zc->workSpace, 0, tableSpace ); /* reset only tables */
|
||||
if (reset) memset(zc->workSpace, 0, tableSpace ); /* reset only tables */
|
||||
zc->hashTable3 = (U32*)(zc->workSpace);
|
||||
zc->hashTable = zc->hashTable3 + h3Size;
|
||||
zc->chainTable = zc->hashTable + hSize;
|
||||
@ -272,8 +270,7 @@ static size_t ZSTD_resetCCtx_advanced (ZSTD_CCtx* zc,
|
||||
zc->seqStore.offCodeStart = zc->seqStore.mlCodeStart + maxNbSeq;
|
||||
zc->seqStore.litStart = zc->seqStore.offCodeStart + maxNbSeq;
|
||||
|
||||
zc->hbSize = 0;
|
||||
zc->stage = 0;
|
||||
zc->stage = 1;
|
||||
zc->loadedDictEnd = 0;
|
||||
|
||||
return 0;
|
||||
@ -282,14 +279,15 @@ static size_t ZSTD_resetCCtx_advanced (ZSTD_CCtx* zc,
|
||||
|
||||
/*! ZSTD_copyCCtx() :
|
||||
* Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
|
||||
* Only works during stage 0 (i.e. before first call to ZSTD_compressContinue()).
|
||||
* Only works during stage 1 (i.e. after creation, but before first call to ZSTD_compressContinue()).
|
||||
* @return : 0, or an error code */
|
||||
size_t ZSTD_copyCCtx(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx)
|
||||
{
|
||||
if (srcCCtx->stage!=0) return ERROR(stage_wrong);
|
||||
if (srcCCtx->stage!=1) return ERROR(stage_wrong);
|
||||
|
||||
dstCCtx->hashLog3 = srcCCtx->hashLog3; /* must be before ZSTD_resetCCtx_advanced */
|
||||
ZSTD_resetCCtx_advanced(dstCCtx, srcCCtx->params);
|
||||
ZSTD_resetCCtx_advanced(dstCCtx, srcCCtx->params, 0);
|
||||
dstCCtx->params.fParams.contentSizeFlag = 0; /* content size different from the one set during srcCCtx init */
|
||||
|
||||
/* copy tables */
|
||||
{ const size_t chainSize = (srcCCtx->params.cParams.strategy == ZSTD_fast) ? 0 : (1 << srcCCtx->params.cParams.chainLog);
|
||||
@ -299,10 +297,6 @@ size_t ZSTD_copyCCtx(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx)
|
||||
memcpy(dstCCtx->workSpace, srcCCtx->workSpace, tableSpace);
|
||||
}
|
||||
|
||||
/* copy frame header */
|
||||
dstCCtx->hbSize = srcCCtx->hbSize;
|
||||
memcpy(dstCCtx->headerBuffer , srcCCtx->headerBuffer, srcCCtx->hbSize);
|
||||
|
||||
/* copy dictionary pointers */
|
||||
dstCCtx->nextToUpdate = srcCCtx->nextToUpdate;
|
||||
dstCCtx->nextToUpdate3= srcCCtx->nextToUpdate3;
|
||||
@ -2068,21 +2062,46 @@ static size_t ZSTD_compress_generic (ZSTD_CCtx* zc,
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
|
||||
ZSTD_parameters params, U64 pledgedSrcSize)
|
||||
{ BYTE* const op = (BYTE*)dst;
|
||||
U32 const fcsId = params.fParams.contentSizeFlag ?
|
||||
(pledgedSrcSize>0) + (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) : /* 0-3 */
|
||||
0;
|
||||
BYTE const fdescriptor = (BYTE)((params.cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) /* windowLog : 4 KB - 128 MB */
|
||||
| (fcsId << 6) );
|
||||
size_t const hSize = ZSTD_frameHeaderSize_min + ZSTD_fcs_fieldSize[fcsId];
|
||||
if (hSize > dstCapacity) return ERROR(dstSize_tooSmall);
|
||||
|
||||
MEM_writeLE32(dst, ZSTD_MAGICNUMBER);
|
||||
op[4] = fdescriptor;
|
||||
switch(fcsId)
|
||||
{
|
||||
default: /* impossible */
|
||||
case 0 : break;
|
||||
case 1 : op[5] = (BYTE)(pledgedSrcSize); break;
|
||||
case 2 : MEM_writeLE16(op+5, (U16)(pledgedSrcSize-256)); break;
|
||||
case 3 : MEM_writeLE64(op+5, (U64)(pledgedSrcSize)); break;
|
||||
}
|
||||
return hSize;
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* zc,
|
||||
void* dst, size_t dstCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
U32 frame)
|
||||
{
|
||||
const BYTE* const ip = (const BYTE*) src;
|
||||
size_t hbSize = 0;
|
||||
size_t fhSize = 0;
|
||||
|
||||
if (frame && (zc->stage==0)) {
|
||||
hbSize = zc->hbSize;
|
||||
if (dstCapacity <= hbSize) return ERROR(dstSize_tooSmall);
|
||||
zc->stage = 1;
|
||||
memcpy(dst, zc->headerBuffer, hbSize);
|
||||
dstCapacity -= hbSize;
|
||||
dst = (char*)dst + hbSize;
|
||||
if (zc->stage==0) return ERROR(stage_wrong);
|
||||
if (frame && (zc->stage==1)) { /* copy saved header */
|
||||
fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, zc->params, srcSize);
|
||||
if (ZSTD_isError(fhSize)) return fhSize;
|
||||
dstCapacity -= fhSize;
|
||||
dst = (char*)dst + fhSize;
|
||||
zc->stage = 2;
|
||||
}
|
||||
|
||||
/* Check if blocks follow each other */
|
||||
@ -2123,7 +2142,7 @@ static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* zc,
|
||||
ZSTD_compress_generic (zc, dst, dstCapacity, src, srcSize) :
|
||||
ZSTD_compressBlock_internal (zc, dst, dstCapacity, src, srcSize);
|
||||
if (ZSTD_isError(cSize)) return cSize;
|
||||
return cSize + hbSize;
|
||||
return cSize + fhSize;
|
||||
}
|
||||
}
|
||||
|
||||
@ -2248,38 +2267,19 @@ static size_t ZSTD_compress_insertDictionary(ZSTD_CCtx* zc, const void* dict, si
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*! ZSTD_compressBegin_internal() :
|
||||
* @return : 0, or an error code */
|
||||
static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* zc,
|
||||
const void* dict, size_t dictSize,
|
||||
ZSTD_parameters params, U64 pledgedSrcSize)
|
||||
{
|
||||
U32 hashLog3 = (pledgedSrcSize || pledgedSrcSize >= 8192) ? ZSTD_HASHLOG3_MAX : ((pledgedSrcSize >= 2048) ? ZSTD_HASHLOG3_MIN + 1 : ZSTD_HASHLOG3_MIN);
|
||||
zc->hashLog3 = (params.cParams.searchLength==3) ? hashLog3 : 0;
|
||||
// printf("windowLog=%d hashLog=%d hashLog3=%d \n", params.windowLog, params.hashLog, zc->hashLog3);
|
||||
{ U32 const hashLog3 = (pledgedSrcSize || pledgedSrcSize >= 8192) ? ZSTD_HASHLOG3_MAX : ((pledgedSrcSize >= 2048) ? ZSTD_HASHLOG3_MIN + 1 : ZSTD_HASHLOG3_MIN);
|
||||
zc->hashLog3 = (params.cParams.searchLength==3) ? hashLog3 : 0; }
|
||||
|
||||
{ size_t const errorCode = ZSTD_resetCCtx_advanced(zc, params);
|
||||
if (ZSTD_isError(errorCode)) return errorCode; }
|
||||
{ size_t const resetError = ZSTD_resetCCtx_advanced(zc, params, 1);
|
||||
if (ZSTD_isError(resetError)) return resetError; }
|
||||
|
||||
/* Write Frame Header into ctx headerBuffer */
|
||||
MEM_writeLE32(zc->headerBuffer, ZSTD_MAGICNUMBER);
|
||||
{ BYTE* const op = (BYTE*)zc->headerBuffer;
|
||||
U32 const fcsId = (pledgedSrcSize>0) + (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256); /* 0-3 */
|
||||
BYTE fdescriptor = (BYTE)(params.cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN); /* windowLog : 4 KB - 128 MB */
|
||||
fdescriptor |= (BYTE)(fcsId << 6);
|
||||
op[4] = fdescriptor;
|
||||
switch(fcsId)
|
||||
{
|
||||
default: /* impossible */
|
||||
case 0 : break;
|
||||
case 1 : op[5] = (BYTE)(pledgedSrcSize); break;
|
||||
case 2 : MEM_writeLE16(op+5, (U16)(pledgedSrcSize-256)); break;
|
||||
case 3 : MEM_writeLE64(op+5, (U64)(pledgedSrcSize)); break;
|
||||
}
|
||||
zc->hbSize = ZSTD_frameHeaderSize_min + ZSTD_fcs_fieldSize[fcsId];
|
||||
}
|
||||
|
||||
zc->stage = 0;
|
||||
return ZSTD_compress_insertDictionary(zc, dict, dictSize);
|
||||
}
|
||||
|
||||
@ -2333,16 +2333,18 @@ size_t ZSTD_compressBegin(ZSTD_CCtx* zc, int compressionLevel)
|
||||
size_t ZSTD_compressEnd(ZSTD_CCtx* zc, void* dst, size_t dstCapacity)
|
||||
{
|
||||
BYTE* op = (BYTE*)dst;
|
||||
size_t hbSize = 0;
|
||||
size_t fhSize = 0;
|
||||
|
||||
/* special case : empty frame : header still within internal buffer */
|
||||
if (zc->stage==0) {
|
||||
hbSize = zc->hbSize;
|
||||
if (dstCapacity <= hbSize) return ERROR(dstSize_tooSmall);
|
||||
zc->stage = 1;
|
||||
memcpy(dst, zc->headerBuffer, hbSize);
|
||||
dstCapacity -= hbSize;
|
||||
op += hbSize;
|
||||
/* not even init ! */
|
||||
if (zc->stage==0) return ERROR(stage_wrong);
|
||||
|
||||
/* special case : empty frame */
|
||||
if (zc->stage==1) {
|
||||
fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, zc->params, 0);
|
||||
if (ZSTD_isError(fhSize)) return fhSize;
|
||||
dstCapacity -= fhSize;
|
||||
op += fhSize;
|
||||
zc->stage = 2;
|
||||
}
|
||||
|
||||
/* frame epilogue */
|
||||
@ -2351,7 +2353,8 @@ size_t ZSTD_compressEnd(ZSTD_CCtx* zc, void* dst, size_t dstCapacity)
|
||||
op[1] = 0;
|
||||
op[2] = 0;
|
||||
|
||||
return 3+hbSize;
|
||||
zc->stage = 0; /* return to "created by not init" status */
|
||||
return 3+fhSize;
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -63,7 +63,7 @@
|
||||
#endif
|
||||
|
||||
#define ZSTD_OPT_NUM (1<<12)
|
||||
#define ZSTD_DICT_MAGIC 0xEC30A435
|
||||
#define ZSTD_DICT_MAGIC 0xEC30A436
|
||||
|
||||
#define ZSTD_REP_NUM 3
|
||||
#define ZSTD_REP_INIT ZSTD_REP_NUM
|
||||
|
||||
@ -166,7 +166,7 @@ ZSTDLIB_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapaci
|
||||
Start by initializing a context.
|
||||
Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
|
||||
or ZSTD_compressBegin_advanced(), for finer parameter control.
|
||||
It's also possible to duplicate a reference context which has been initialized, using ZSTD_copyCCtx()
|
||||
It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
|
||||
|
||||
Then, consume your input using ZSTD_compressContinue().
|
||||
The interface is synchronous, so all input will be consumed and produce a compressed output.
|
||||
@ -200,7 +200,7 @@ ZSTDLIB_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t ds
|
||||
Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
|
||||
A ZSTD_DCtx object can be re-used multiple times.
|
||||
|
||||
First optional operation is to retrieve frame parameters, using ZSTD_getFrameParams().
|
||||
First optional operation is to retrieve frame parameters, using ZSTD_getFrameParams(), which doesn't consume the input.
|
||||
It can provide the minimum size of rolling buffer required to properly decompress data,
|
||||
and optionally the final size of uncompressed content.
|
||||
(Note : content size is an optional info that may not be present. 0 means : content size unknown)
|
||||
|
||||
@ -169,18 +169,22 @@ static int basicUnitTests(U32 seed, double compressibility)
|
||||
if (result != (size_t)-ZSTD_error_srcSize_wrong) goto _output_error;
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
/* Dictionary and Duplication tests */
|
||||
/* Dictionary and CCtx Duplication tests */
|
||||
{ ZSTD_CCtx* ctxOrig = ZSTD_createCCtx();
|
||||
ZSTD_CCtx* ctxDuplicated = ZSTD_createCCtx();
|
||||
ZSTD_DCtx* dctx = ZSTD_createDCtx();
|
||||
const size_t dictSize = 500;
|
||||
size_t cSizeOrig;
|
||||
size_t const dictSize = 500;
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : copy context too soon : ", testNb++);
|
||||
{ size_t const copyResult = ZSTD_copyCCtx(ctxDuplicated, ctxOrig);
|
||||
if (!ZSTD_isError(copyResult)) goto _output_error; } /* error should be detected */
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : load dictionary into context : ", testNb++);
|
||||
result = ZSTD_compressBegin_usingDict(ctxOrig, CNBuffer, dictSize, 2);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
result = ZSTD_copyCCtx(ctxDuplicated, ctxOrig);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
{ size_t const initResult = ZSTD_compressBegin_usingDict(ctxOrig, CNBuffer, dictSize, 2);
|
||||
if (ZSTD_isError(initResult)) goto _output_error; }
|
||||
{ size_t const copyResult = ZSTD_copyCCtx(ctxDuplicated, ctxOrig);
|
||||
if (ZSTD_isError(copyResult)) goto _output_error; }
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : compress with dictionary : ", testNb++);
|
||||
@ -200,20 +204,19 @@ static int basicUnitTests(U32 seed, double compressibility)
|
||||
CNBuffer, dictSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
if (result != COMPRESSIBLE_NOISE_LENGTH - dictSize) goto _output_error;
|
||||
ZSTD_freeCCtx(ctxOrig); /* if ctxOrig is read, will produce segfault */
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : compress with duplicated context : ", testNb++);
|
||||
cSizeOrig = cSize;
|
||||
cSize = 0;
|
||||
result = ZSTD_compressContinue(ctxDuplicated, compressedBuffer, ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH), (const char*)CNBuffer + dictSize, COMPRESSIBLE_NOISE_LENGTH - dictSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
cSize += result;
|
||||
result = ZSTD_compressEnd(ctxDuplicated, (char*)compressedBuffer+cSize, ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH)-cSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
cSize += result;
|
||||
if (cSize != cSizeOrig) goto _output_error; /* should be identical == have same size */
|
||||
ZSTD_freeCCtx(ctxDuplicated);
|
||||
{ size_t const cSizeOrig = cSize;
|
||||
cSize = 0;
|
||||
result = ZSTD_compressContinue(ctxDuplicated, compressedBuffer, ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH), (const char*)CNBuffer + dictSize, COMPRESSIBLE_NOISE_LENGTH - dictSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
cSize += result;
|
||||
result = ZSTD_compressEnd(ctxDuplicated, (char*)compressedBuffer+cSize, ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH)-cSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
cSize += result;
|
||||
if (cSize != cSizeOrig) goto _output_error; /* should be identical == have same size */
|
||||
}
|
||||
DISPLAYLEVEL(4, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/COMPRESSIBLE_NOISE_LENGTH*100);
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : frame built with duplicated context should be decompressible : ", testNb++);
|
||||
@ -223,8 +226,30 @@ static int basicUnitTests(U32 seed, double compressibility)
|
||||
CNBuffer, dictSize);
|
||||
if (ZSTD_isError(result)) goto _output_error;
|
||||
if (result != COMPRESSIBLE_NOISE_LENGTH - dictSize) goto _output_error;
|
||||
ZSTD_freeDCtx(dctx);
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
DISPLAYLEVEL(4, "test%3i : check content size on duplicated context : ", testNb++);
|
||||
{ size_t const testSize = COMPRESSIBLE_NOISE_LENGTH / 3;
|
||||
{ ZSTD_parameters p;
|
||||
p.cParams = ZSTD_getCParams(2, testSize, dictSize);
|
||||
p.fParams.contentSizeFlag = 1;
|
||||
{ size_t const initResult = ZSTD_compressBegin_advanced(ctxOrig, CNBuffer, dictSize, p, testSize-1);
|
||||
if (ZSTD_isError(initResult)) goto _output_error;
|
||||
} }
|
||||
{ size_t const copyResult = ZSTD_copyCCtx(ctxDuplicated, ctxOrig);
|
||||
if (ZSTD_isError(copyResult)) goto _output_error; }
|
||||
cSize = ZSTD_compressContinue(ctxDuplicated, compressedBuffer, ZSTD_compressBound(testSize), (const char*)CNBuffer + dictSize, COMPRESSIBLE_NOISE_LENGTH - dictSize);
|
||||
if (ZSTD_isError(cSize)) goto _output_error;
|
||||
{ ZSTD_frameParams fp;
|
||||
size_t const gfpResult = ZSTD_getFrameParams(&fp, compressedBuffer, cSize);
|
||||
if (gfpResult!=0) goto _output_error;
|
||||
if ((fp.frameContentSize != testSize) && (fp.frameContentSize != 0)) goto _output_error;
|
||||
} }
|
||||
DISPLAYLEVEL(4, "OK \n");
|
||||
|
||||
ZSTD_freeCCtx(ctxOrig);
|
||||
ZSTD_freeCCtx(ctxDuplicated);
|
||||
ZSTD_freeDCtx(dctx);
|
||||
}
|
||||
|
||||
/* Decompression defense tests */
|
||||
|
||||
Loading…
Reference in New Issue
Block a user