From b93f629681ad3245a09add28e4d0b2e43bcde58a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 18:31:55 +0100 Subject: [PATCH 001/121] changed file name --- lz4_block_format.txt => lz4_Block_format.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename lz4_block_format.txt => lz4_Block_format.md (100%) diff --git a/lz4_block_format.txt b/lz4_Block_format.md similarity index 100% rename from lz4_block_format.txt rename to lz4_Block_format.md From 44793b8be9f18bb51f524b3a210de11bb0df6654 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 18:32:21 +0100 Subject: [PATCH 002/121] Updated documentation --- README.md | 44 ++++++++++++-------------------------------- lz4_Block_format.md | 25 ++++++++++++++----------- 2 files changed, 26 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index f960e7d..275085e 100644 --- a/README.md +++ b/README.md @@ -20,41 +20,21 @@ A high compression derivative, called LZ4_HC, is also provided. It trades CPU ti Benchmarks ------------------------- -The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.2)](http://encode.ru/threads/1371-Filesystem-benchmark?p=33548&viewfull=1#post33548) compiled with GCC v4.6.1 on Linux Ubuntu 64-bits v11.10, -The reference system uses a Core i5-3340M @2.7GHz. +The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.3)](http://encode.ru/threads/1371-Filesystem-benchmark?p=33548&viewfull=1#post33548) compiled with GCC v4.8.2 on Linux Mint 64-bits v17. +The reference system uses a Core i5-4300U @1.9GHz. Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia) in single-thread mode. - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CompressorRatioCompressionDecompression
LZ4 (r101)2.084422 MB/s1820 MB/s
LZO 2.062.106414 MB/s600 MB/s
QuickLZ 1.5.1b62.237373 MB/s420 MB/s
Snappy 1.1.02.091323 MB/s1070 MB/s
LZF2.077270 MB/s570 MB/s
zlib 1.2.8 -12.73065 MB/s280 MB/s
LZ4 HC (r101)2.72025 MB/s2080 MB/s
zlib 1.2.8 -63.09921 MB/s300 MB/s
+| Compressor | Ratio | Compression | Decompression | +| ---------- | ----- | ----------- | ------------- | +|**LZ4 (r129)** | 2.101 |**385 MB/s** |**1850 MB/s** | +| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | +| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | +| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | +| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | +|**LZ4 HC (r129)** |**2.720**| 22 MB/s |**1830 MB/s** | +| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | -The LZ4 block compression format is detailed within [lz4_block_format.txt](lz4_block_format.txt). +The LZ4 block compression format is detailed within [lz4_Block_format](lz4_Block_format.md). For streaming unknown amount of data and compress files of any size, a frame format has been published, and can be consulted within the file LZ4_Frame_Format.html . diff --git a/lz4_Block_format.md b/lz4_Block_format.md index e248fd9..b933a6a 100644 --- a/lz4_Block_format.md +++ b/lz4_Block_format.md @@ -1,10 +1,9 @@ LZ4 Block Format Description ============================ -Last revised: 2015-03-26; +Last revised: 2015-03-26. Author : Yann Collet - This small specification intents to provide enough information to anyone willing to produce LZ4-compatible compressed data blocks using any programming language. @@ -26,7 +25,8 @@ on implementation details of the compressor, and vice versa. Compressed block format ----------------------- An LZ4 compressed block is composed of sequences. -Schematically, a sequence is a suite of literals, followed by a match copy. +A sequence is a suite of literals (not-compressed bytes), +followed by a match copy. Each sequence starts with a token. The token is a one byte value, separated into two 4-bits fields. @@ -35,14 +35,14 @@ Therefore each field ranges from 0 to 15. The first field uses the 4 high-bits of the token. It provides the length of literals to follow. -(Note : a literal is a not-compressed byte). + If the field value is 0, then there is no literal. If it is 15, then we need to add some more bytes to indicate the full length. -Each additionnal byte then represent a value from 0 to 255, +Each additional byte then represent a value from 0 to 255, which is added to the previous value to produce a total length. When the byte value is 255, another byte is output. There can be any number of bytes following the token. There is no "size limit". -(Sidenote this is why a not-compressible input block is expanded by 0.4%). +(Side note : this is why a not-compressible input block is expanded by 0.4%). Example 1 : A length of 48 will be represented as : - 15 : value for the 4-bits High field @@ -65,7 +65,8 @@ It's possible that there are zero literal. Following the literals is the match copy operation. It starts by the offset. -This is a 2 bytes value, in little endian format. +This is a 2 bytes value, in little endian format +(the 1st byte is the "low" byte, the 2nd one is the "high" byte). The offset represents the position of the match to be copied from. 1 means "current position - 1 byte". @@ -95,9 +96,12 @@ Parsing restrictions ----------------------- There are specific parsing rules to respect in order to remain compatible with assumptions made by the decoder : -1) The last 5 bytes are always literals -2) The last match must start at least 12 bytes before end of block -Consequently, a block with less than 13 bytes cannot be compressed. + +1. The last 5 bytes are always literals +2. The last match must start at least 12 bytes before end of block. + + Consequently, a block with less than 13 bytes cannot be compressed. + These rules are in place to ensure that the decoder will never read beyond the input buffer, nor write beyond the output buffer. @@ -118,4 +122,3 @@ or full optimal parsing. All these trade-off offer distinctive speed/memory/compression advantages. Whatever the method used by the compressor, its result will be decodable by any LZ4 decoder if it follows the format specification described above. - From 6c69dc176c318fba721fa8adade53f3e413cac52 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 18:34:15 +0100 Subject: [PATCH 003/121] faster compression in 64 bits mode --- lib/lz4.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/lib/lz4.c b/lib/lz4.c index 881d1af..2a1ae57 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -464,7 +464,24 @@ static U32 LZ4_hashSequence(U32 sequence, tableType_t const tableType) return (((sequence) * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG)); } -static U32 LZ4_hashPosition(const BYTE* p, tableType_t tableType) { return LZ4_hashSequence(LZ4_read32(p), tableType); } +static const U64 prime5bytes = 889523592379ULL; +static U32 LZ4_hashSequence64(size_t sequence, tableType_t const tableType) +{ + const U32 hashLog = (tableType == byU16) ? LZ4_HASHLOG+1 : LZ4_HASHLOG; + const U32 hashMask = (1<> (40 - hashLog)) & hashMask; +} + +static U32 LZ4_hashSequenceT(size_t sequence, tableType_t const tableType) +{ + if (LZ4_64bits()) + return LZ4_hashSequence64(sequence, tableType); + return LZ4_hashSequence((U32)sequence, tableType); +} + +static U32 LZ4_hashPosition(const void* p, tableType_t tableType) { return LZ4_hashSequenceT(LZ4_read_ARCH(p), tableType); } + +//static U32 LZ4_hashPosition(const BYTE* p, tableType_t tableType) { return LZ4_hashSequence(LZ4_read32(p), tableType); } static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase) { From 2a826193110e5423e7dedcaa78ddb7c1aa1f0950 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 18:39:29 +0100 Subject: [PATCH 004/121] fixed fullbench memory allocation error --- programs/fullbench.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/programs/fullbench.c b/programs/fullbench.c index 0c6e05e..6c42ed0 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -544,7 +544,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) return 10; } - // Loop for each file + /* Loop for each fileName */ while (fileIdx inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) @@ -940,7 +940,7 @@ _exit_blockProperties: // Modify Nb Iterations case 'i': - if ((argument[1] >='1') && (argument[1] <='9')) + if ((argument[1] >='0') && (argument[1] <='9')) { int iters = argument[1] - '0'; BMK_SetNbIterations(iters); From 4c227a487e25c175d98a320386c96dbea6628216 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 21:32:25 +0100 Subject: [PATCH 005/121] Added LZ4_compress_fast() --- lib/lz4.c | 90 +++++++++++++++++++++++++++++++------------- lib/lz4.h | 10 +++++ programs/fullbench.c | 68 +++++++++++++++++---------------- 3 files changed, 109 insertions(+), 59 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 2a1ae57..c8bde3a 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -43,6 +43,12 @@ */ #define HEAPMODE 0 +/* + * ACCELERATION_DEFAULT : + * Select the value of "acceleration" for LZ4_compress_fast() when parameter == 0 + */ +#define ACCELERATION_DEFAULT 17 + /* * CPU_HAS_EFFICIENT_UNALIGNED_MEMORY_ACCESS : * By default, the source code expects the compiler to correctly optimize @@ -513,15 +519,16 @@ static const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, tableType_t t } static int LZ4_compress_generic( - void* ctx, - const char* source, - char* dest, - int inputSize, - int maxOutputSize, - limitedOutput_directive outputLimited, - tableType_t const tableType, - dict_directive dict, - dictIssue_directive dictIssue) + void* const ctx, + const char* const source, + char* const dest, + const int inputSize, + const int maxOutputSize, + const limitedOutput_directive outputLimited, + const tableType_t tableType, + const dict_directive dict, + const dictIssue_directive dictIssue, + const U32 acceleration) { LZ4_stream_t_internal* const dictPtr = (LZ4_stream_t_internal*)ctx; @@ -544,7 +551,7 @@ static int LZ4_compress_generic( size_t refDelta=0; /* Init conditions */ - if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ + if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ switch(dict) { case noDict: @@ -576,14 +583,14 @@ static int LZ4_compress_generic( { const BYTE* forwardIp = ip; unsigned step=1; - unsigned searchMatchNb = (1U << LZ4_skipTrigger); + unsigned searchMatchNb = ((1+acceleration) << LZ4_skipTrigger); /* Find a match */ do { U32 h = forwardH; ip = forwardIp; forwardIp += step; - step = searchMatchNb++ >> LZ4_skipTrigger; + step = (searchMatchNb++ >> LZ4_skipTrigger); if (unlikely(forwardIp > mflimit)) goto _last_literals; @@ -724,7 +731,7 @@ _last_literals: } -int LZ4_compress(const char* source, char* dest, int inputSize) +int LZ4_compress(const char* source, char* dest, const int inputSize) { #if (HEAPMODE) void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ @@ -734,9 +741,9 @@ int LZ4_compress(const char* source, char* dest, int inputSize) int result; if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 0); else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); #if (HEAPMODE) FREEMEM(ctx); @@ -754,9 +761,40 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in int result; if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 0); else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); + +#if (HEAPMODE) + FREEMEM(ctx); +#endif + return result; +} + + +int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) +{ +#if (HEAPMODE) + void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ +#else + U64 ctx[LZ4_STREAMSIZE_U64] = {0}; /* Ensure data is aligned on 8-bytes boundaries */ +#endif + int result; + + if (acceleration == 0) + { + if (inputSize < LZ4_64Klimit) + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); + else + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, ACCELERATION_DEFAULT); + } + else + { + if (inputSize < LZ4_64Klimit) + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); + else + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); + } #if (HEAPMODE) FREEMEM(ctx); @@ -875,9 +913,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall, 0); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue, 0); streamPtr->dictSize += (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; @@ -887,9 +925,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall, 0); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue, 0); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; @@ -919,7 +957,7 @@ int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* if (smallest > (const BYTE*) source) smallest = (const BYTE*) source; LZ4_renormDictT((LZ4_stream_t_internal*)LZ4_dict, smallest); - result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue); + result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue, 0); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; @@ -1352,9 +1390,9 @@ int LZ4_compress_withState (void* state, const char* source, char* dest, int inp MEM_INIT(state, 0, LZ4_STREAMSIZE); if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 0); else - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); } int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) @@ -1363,9 +1401,9 @@ int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* MEM_INIT(state, 0, LZ4_STREAMSIZE); if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue); + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 0); else - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue); + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); } /* Obsolete streaming decompression functions */ diff --git a/lib/lz4.h b/lib/lz4.h index de43fc0..eabc40f 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -129,6 +129,16 @@ LZ4_compress_limitedOutput() : int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); +/* +LZ4_compress_fast() : + Same as LZ4_compress_limitedOutput, but allows to select an "acceleration" factor. + The larger the value, the faster the algorithm, but also the lesser the compression. + So it's a trade-off, which can be fine tuned, selecting whichever value you want. + An acceleration value of "0" means "use Default value", which is typically about 15 (see lz4.c source code). +*/ +int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); + + /* LZ4_compress_withState() : Same compression functions, but using an externally allocated memory space to store compression state. diff --git a/programs/fullbench.c b/programs/fullbench.c index 6c42ed0..cd6e3c8 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -146,9 +146,9 @@ struct chunkParameters -//************************************** -// Benchmark Parameters -//************************************** +/************************************** +* Benchmark Parameters +**************************************/ static int chunkSize = DEFAULT_CHUNKSIZE; static int nbIterations = NBLOOPS; static int BMK_pause = 0; @@ -175,17 +175,17 @@ void BMK_SetPause(void) BMK_pause = 1; } -//********************************************************* -// Private functions -//********************************************************* +/********************************************************* +* Private functions +*********************************************************/ #if defined(BMK_LEGACY_TIMER) static int BMK_GetMilliStart(void) { - // Based on Legacy ftime() - // Rolls over every ~ 12.1 days (0x100000/24/60/60) - // Use GetMilliSpan to correct for rollover + /* Based on Legacy ftime() + * Rolls over every ~ 12.1 days (0x100000/24/60/60) + * Use GetMilliSpan to correct for rollover */ struct timeb tb; int nCount; ftime( &tb ); @@ -197,8 +197,8 @@ static int BMK_GetMilliStart(void) static int BMK_GetMilliStart(void) { - // Based on newer gettimeofday() - // Use GetMilliSpan to correct for rollover + /* Based on newer gettimeofday() + * Use GetMilliSpan to correct for rollover */ struct timeval tv; int nCount; gettimeofday(&tv, NULL); @@ -386,6 +386,11 @@ static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSiz return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)); } +static int local_LZ4_compress_fast(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); +} + static void* stateLZ4; static int local_LZ4_compress_withState(const char* in, char* out, int inSize) { @@ -530,7 +535,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) { int fileIdx=0; char* orig_buff; -# define NB_COMPRESSION_ALGORITHMS 16 +# define NB_COMPRESSION_ALGORITHMS 17 double totalCTime[NB_COMPRESSION_ALGORITHMS+1] = {0}; double totalCSize[NB_COMPRESSION_ALGORITHMS+1] = {0}; # define NB_DECOMPRESSION_ALGORITHMS 9 @@ -568,22 +573,18 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) return 11; } - /* Init */ - stateLZ4 = LZ4_createStream(); - stateLZ4HC = LZ4_createStreamHC(); - - /* Memory allocation & restrictions */ + /* Memory size adjustments */ inFileSize = BMK_GetFileSize(inFileName); if (inFileSize==0) { DISPLAY( "file is empty\n"); return 11; } - benchedSize = (size_t) BMK_findMaxMem(inFileSize); + benchedSize = (size_t) BMK_findMaxMem(inFileSize*2) / 2; /* because 2 buffers */ if (benchedSize==0) { DISPLAY( "not enough memory\n"); return 11; } if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) - { DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); - } - // Alloc + /* Allocation */ + stateLZ4 = LZ4_createStream(); + stateLZ4HC = LZ4_createStreamHC(); chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)chunkSize)+1) * sizeof(struct chunkParameters)); orig_buff = (char*) malloc((size_t)benchedSize); nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); @@ -602,7 +603,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) return 12; } - // Fill input buffer + /* Fill in src buffer */ DISPLAY("Loading %s... \r", inFileName); readSize = fread(orig_buff, 1, benchedSize, inFile); fclose(inFile); @@ -664,20 +665,21 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) case 4 : compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; case 5 : compressionFunction = local_LZ4_compress_continue; initFunction = LZ4_create; compressorName = "LZ4_compress_continue"; break; case 6 : compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = LZ4_create; compressorName = "LZ4_compress_limitedOutput_continue"; break; - case 7 : compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; - case 8 : compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; - case 9 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; - case 10: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; - case 11: compressionFunction = local_LZ4_compressHC_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_continue"; break; - case 12: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; - case 13: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; - case 14: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; + case 7 : compressionFunction = local_LZ4_compress_fast; compressorName = "LZ4_compress_fast"; break; + case 8 : compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; + case 9 : compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; + case 10 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; + case 11: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; + case 12: compressionFunction = local_LZ4_compressHC_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_continue"; break; + case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; + case 14: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; + case 15: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; chunkP[0].origSize = (int)benchedSize; nbChunks=1; break; - case 15: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; + case 16: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; LZ4_loadDict(&LZ4_dict, chunkP[0].origBuffer, chunkP[0].origSize); break; - case 16: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; + case 17: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; LZ4_loadDictHC(&LZ4_dictHC, chunkP[0].origBuffer, chunkP[0].origSize); break; default : DISPLAY("ERROR ! Bad algorithm Id !! \n"); free(chunkP); return 1; @@ -724,7 +726,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) totalCSize[cAlgNb] += cSize; } - // Prepare layout for decompression + /* Prepare layout for decompression */ // Init data chunks { int i; From 4783cb8c57f890211f0c57c78c7e378c4a97075b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 21:38:37 +0100 Subject: [PATCH 006/121] Updated readme --- NEWS | 3 +++ README.md | 20 +++++++++++++------- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/NEWS b/NEWS index 2eeb948..d80b138 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,6 @@ +r129: +Added : LZ4_compress_fast() + r128: New : lz4cli sparse file support New : command -m, to compress multiple files in a single command diff --git a/README.md b/README.md index 275085e..f808010 100644 --- a/README.md +++ b/README.md @@ -26,13 +26,19 @@ Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei | Compressor | Ratio | Compression | Decompression | | ---------- | ----- | ----------- | ------------- | -|**LZ4 (r129)** | 2.101 |**385 MB/s** |**1850 MB/s** | -| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | -| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | -| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | -| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | -|**LZ4 HC (r129)** |**2.720**| 22 MB/s |**1830 MB/s** | -| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | +| memcpy | 1.000 | 4200 MB/s | 4200 MB/s | +| RLE64 v3.0 | 1.029 | 2800 MB/s | 2800 MB/s | +| density -c1 | 1.592 | 700 MB/s | 920 MB/s | +|**LZ4 fast (r129)**| 1.595 |**680 MB/s** | **2220 MB/s** | +|**LZ4 (r129)** |**2.101**|**385 MB/s** | **1850 MB/s** | +| density -c2 | 2.083 | 370 MB/s | 505 MB/s | +| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | +| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | +| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | +| density -c3 | 2.370 | 190 MB/s | 185 MB/s | +| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | +|**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | +| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | The LZ4 block compression format is detailed within [lz4_Block_format](lz4_Block_format.md). From 5b9fb6971522514ffc02366a32533facfd03c528 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 30 Mar 2015 22:39:08 +0100 Subject: [PATCH 007/121] minor tweak --- README.md | 2 +- lib/lz4.c | 28 ++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index f808010..0f2cd40 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei | memcpy | 1.000 | 4200 MB/s | 4200 MB/s | | RLE64 v3.0 | 1.029 | 2800 MB/s | 2800 MB/s | | density -c1 | 1.592 | 700 MB/s | 920 MB/s | -|**LZ4 fast (r129)**| 1.595 |**680 MB/s** | **2220 MB/s** | +|**LZ4 fast (r129)**| 1.607 |**680 MB/s** | **2220 MB/s** | |**LZ4 (r129)** |**2.101**|**385 MB/s** | **1850 MB/s** | | density -c2 | 2.083 | 370 MB/s | 505 MB/s | | LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | diff --git a/lib/lz4.c b/lib/lz4.c index c8bde3a..cd6147f 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -583,7 +583,7 @@ static int LZ4_compress_generic( { const BYTE* forwardIp = ip; unsigned step=1; - unsigned searchMatchNb = ((1+acceleration) << LZ4_skipTrigger); + unsigned searchMatchNb = ((acceleration) << LZ4_skipTrigger); /* Find a match */ do { @@ -741,9 +741,9 @@ int LZ4_compress(const char* source, char* dest, const int inputSize) int result; if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 0); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); #if (HEAPMODE) FREEMEM(ctx); @@ -761,9 +761,9 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in int result; if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 0); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); #if (HEAPMODE) FREEMEM(ctx); @@ -913,9 +913,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall, 0); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall, 1); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue, 0); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue, 1); streamPtr->dictSize += (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; @@ -925,9 +925,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall, 0); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall, 1); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue, 0); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue, 1); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; @@ -957,7 +957,7 @@ int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* if (smallest > (const BYTE*) source) smallest = (const BYTE*) source; LZ4_renormDictT((LZ4_stream_t_internal*)LZ4_dict, smallest); - result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue, 0); + result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue, 1); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; @@ -1390,9 +1390,9 @@ int LZ4_compress_withState (void* state, const char* source, char* dest, int inp MEM_INIT(state, 0, LZ4_STREAMSIZE); if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 0); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); else - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); } int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) @@ -1401,9 +1401,9 @@ int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* MEM_INIT(state, 0, LZ4_STREAMSIZE); if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 0); + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); else - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 0); + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); } /* Obsolete streaming decompression functions */ From 880381c11bc9838b8609643dff64044fca8856e2 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 09:42:44 +0100 Subject: [PATCH 008/121] Removed HTML Frame Format documentation --- LZ4_Frame_Format.html | 1 - images/image00.png | Bin 2908 -> 0 bytes images/image01.png | Bin 5793 -> 0 bytes images/image02.png | Bin 6246 -> 0 bytes images/image03.png | Bin 4089 -> 0 bytes images/image04.png | Bin 3121 -> 0 bytes images/image05.png | Bin 4166 -> 0 bytes images/image06.png | Bin 8028 -> 0 bytes 8 files changed, 1 deletion(-) delete mode 100644 LZ4_Frame_Format.html delete mode 100644 images/image00.png delete mode 100644 images/image01.png delete mode 100644 images/image02.png delete mode 100644 images/image03.png delete mode 100644 images/image04.png delete mode 100644 images/image05.png delete mode 100644 images/image06.png diff --git a/LZ4_Frame_Format.html b/LZ4_Frame_Format.html deleted file mode 100644 index 6622dbb..0000000 --- a/LZ4_Frame_Format.html +++ /dev/null @@ -1 +0,0 @@ -LZ4 Framing format - v1.5.0

LZ4 Framing Format

Notices

Copyright (c) 2013-2015 Yann Collet

Permission is granted to copy and distribute this document for any  purpose and without charge, including translations into other  languages and incorporation into compilations, provided that the copyright notice and this notice are preserved, and that any substantive changes or deletions from the original are clearly marked.

Version

1.5.0

Introduction

The purpose of this document is to define a lossless compressed data format, that is independent of CPU type, operating system, file system and character set, suitable for File compression, Pipe and streaming compression using the LZ4 algorithm : http://code.google.com/p/lz4/

The data can be produced or consumed, even for an arbitrarily long sequentially presented input data stream, using only an a priori bounded amount of intermediate storage, and hence can be used in data communications.  The format uses the LZ4 compression method, and optional xxHash-32 checksum method, for detection of data corruption.

The data format defined by this specification does not attempt to allow random access to compressed data.

This specification is intended for use by implementers of software to compress data into LZ4 format and/or decompress data from LZ4 format. The text of the specification assumes a basic background in programming at the level of bits and other primitive data representations.

Unless otherwise indicated below, a compliant compressor must produce data sets that conform to the specifications presented here. It doesn’t need to support all options though.

A compliant decompressor must be able to decompress at least one working set of parameters that conforms to the specifications presented here. It may also ignore checksums. Whenever it does not support a specific parameter used within the compressed stream, it must produce a non-ambiguous error code and associated error message explaining which parameter is unsupported.

Distribution of this document is unlimited.

Summary :

Introduction

General structure of LZ4 Framing Format

Frame Descriptor

Data Blocks

Skippable Frames

Legacy format

Appendix

General Structure of LZ4 Framing format

Magic Number

4 Bytes, Little endian format.
Value : 0x184D2204

Frame Descriptor

3 to 11 Bytes, to be detailed in the next part.
Most important part of the spec.

Data Blocks

To be detailed later on.
That’s where compressed data is stored.

EndMark

The flow of blocks ends when the last data block has a size of “0”.
The size is expressed as a 32-bits value.

Content Checksum

Content Checksum verify that the full content has been decoded correctly.
The content checksum is the result of xxh32() hash function digesting the original (decoded) data as input, and a seed of zero.
Content checksum is only present when its associated flag is set in the framing descriptor. Content Checksum validates the result, that all blocks were fully transmitted in the correct order and without error, and also that the encoding/decoding process itself generated no distortion. Its usage is recommended.

Frame Concatenation

In some circumstances, it may be preferable to append multiple frames, for example in order to add new data to an existing compressed file without re-framing it.

In such case, each frame has its own set of descriptor flags. Each frame is considered independent. The only relation between frames is their sequential order.

The ability to decode multiple concatenated frames within a single stream or file is left outside of this specification. As an example, the reference lz4 command line utility behavior is to decode all concatenated frames in their sequential order.

Frame Descriptor

The descriptor uses a minimum of 3 bytes, and up to 11 bytes depending on optional parameters.
In the picture, bit 7 is highest bit, while bit 0 is lowest.

Version Number :

2-bits field, must be set to “01”.
Any other value cannot be decoded by this version of the specification.
Other version numbers will use different flag layouts.

Block Independence flag :

If this flag is set to “1”, blocks are independent, and can therefore be decoded independently, in parallel.
If this flag is set to “0”, each block depends on previous ones for decoding (up to LZ4 window size, which is 64 KB). In this case, it’s necessary to decode all blocks in sequence.

Block dependency improves compression ratio, especially for small blocks. On the other hand, it makes jumps or multi-threaded decoding impossible.

Block checksum flag :

If this flag is set, each data block will be followed by a 4-bytes checksum, calculated by using the xxHash-32 algorithm on the raw (compressed) data block.
The intention is to detect data corruption (storage or transmission errors) immediately, before decoding.
Block checksum usage is optional.

Content Size flag :

If this flag is set, the original (uncompressed) size of data included within the frame will be present as an 8 bytes unsigned value, little endian format, after the flags.

Recommended value : “0” (not present)

Content checksum flag :

If this flag is set, a content checksum will be appended after the EoS mark.

Recommended value : “1” (content checksum is present)

Block Maximum Size :

This information is intended to help the decoder allocate the right amount of memory.
Size here refers to the original (uncompressed) data size.
Block Maximum Size is one value among the following table :

The decoder may refuse to allocate block sizes above a (system-specific) size.
Unused values may be used in a future revision of the spec.
A decoder conformant to the current version of the spec is only able to decode blocksizes defined in this spec.

Reserved bits :

Value of reserved bits must be 0 (zero).
Reserved bit might be used in a future version of the specification, to enable any (yet-to-decide) optional feature.
If this happens, a decoder respecting the current version of the specification shall not be able to decode such a frame.

Content Size

This is the original (uncompressed) size.
This information is optional, and only present if the associated flag is set.
Content size is provided using unsigned 8 Bytes, for a maximum of 16 HexaBytes.
Format is Little endian.
This field has no impact on decoding, it just informs the decoder how much data the frame holds (for example, to display it during decoding process, or for verification purpose). It can be safely skipped by a conformant decoder.

Header Checksum :

One-byte checksum of all descriptor fields, including optional ones when present.
The byte is second byte of xxh32() : { (xxh32()>>8) & 0xFF } ,
using zero as a seed,
and the full Frame Descriptor as an input (including optional fields when they are present).
A different checksum indicates an error in the descriptor.

Data Blocks

Block Size

This field uses 4-bytes,  format is little-endian.

The highest bit is “1” if data in the block is uncompressed.

The highest bit is “0” if data in the block is compressed by LZ4.

All other bits give the size, in bytes, of the following data block (the size does not include the checksum if present).

Block Size shall never be larger than Block Maximum Size. Such a thing could happen when the original data is incompressible. In this case, such a data block shall be passed in uncompressed format.

Data

Where the actual data to decode stands. It might be compressed or not, depending on previous field indications.
Uncompressed size of Data can be any size, up to “block maximum size”.
Note that data block is not necessarily full : an arbitrary “flush” may happen anytime. Any block can be “partially filled”.

Block checksum :

Only present if the associated flag is set.
This is a 4-bytes checksum value, in little endian format,
calculated by using the xxHash-32 algorithm on the raw (undecoded) data block,
and a seed of zero.
The intention is to detect data corruption (storage or transmission errors) before decoding.

Block checksum is cumulative with Content checksum.

Skippable Frames

LZ4 Framing Format - Skippable Frame.png

Skippable frames allow the integration of user-defined data into a flow of concatenated frames.
Its design is pretty straightforward, with the sole objective to allow the decoder to quickly skip over user-defined data and continue decoding.

For the purpose of facilitating identification, it is discouraged to start a flow of concatenated frames with a skippable frame. If there is a need to start such a flow with some user data encapsulated into a skippable frame, it’s recommended to start will a zero-byte LZ4 frame followed by a skippable frame. This will make it easier for file type identifiers.

 

Magic Number

4 Bytes, Little endian format.
Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F. All 16 values are valid to identify a skippable frame.

Frame Size 

This is the size, in bytes, of the following User Data (without including the magic number nor the size field itself).
4 Bytes, Little endian format, unsigned 32-bits.
This means User Data can’t be bigger than (2^32-1) Bytes.

User Data

User Data can be anything. Data will just be skipped by the decoder.

Legacy frame

The Legacy frame format was defined into the initial versions of “LZ4Demo”.
Newer compressors should not use this format anymore, since it is too restrictive.
It is recommended that decompressors shall be able to decode this format during the transition period.

Main properties of legacy format :
- Fixed block size : 8 MB.
- All blocks must be completely filled, except the last one.
- All blocks are always compressed, even when compression is detrimental.
- The last block is detected either because it is followed by the “EOF” (End of File) mark, or because it is followed by a known Frame Magic Number.
- No checksum
- Convention is Little endian

Magic Number

4 Bytes, Little endian format.
Value : 0x184C2102

Block Compressed Size

This is the size, in bytes, of the following compressed data block.
4 Bytes, Little endian format.

Data

Where the actual data stands.
Data is always compressed, even when compression is detrimental (i.e. larger than original size).

Appendix  

Version changes

1.4.1 : changed wording from “stream” to “frame”

1.4 : added skippable streams, re-added stream checksum

1.3 : modified header checksum

1.2 : reduced choice of “block size”, to postpone decision on “dynamic size of BlockSize Field”.

1.1 : optional fields are now part of the descriptor

1.0 : changed “block size” specification, adding a compressed/uncompressed flag

0.9 : reduced scale of “block maximum size” table

0.8 : removed : high compression flag

0.7 : removed : stream checksum

0.6 : settled : stream size uses 8 bytes, endian convention is little endian

0.5: added copyright notice

0.4 : changed format to Google Doc compatible OpenDocument

\ No newline at end of file diff --git a/images/image00.png b/images/image00.png deleted file mode 100644 index f4645f2d44217dde99785e74b1776f8591d9f65e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 2908 zcmeHJSv=d@7XG6sEn+NcjJA3WA=E+KQks~h@0f748*f(E601yB!JGvrxJ^*L~ z01V0_06+n47%w;EnGOJe0StqofCCJ`xC4MYpp7}SV6lcg#l#vq05B2^MgqJD0EUL) zzv$dB8b3bVeN;6J2SF24eAlXzkhky80;x5}o zvGSL`Z(QyaKET>@UAlSw<89n%@->H=@wf@w-HlJMZXZ0F)e+K?+9_^NT&v`T_e7JC zhUMj51D63vGhz0}{qh!TWl1x^wc6$<4KvH%t7n5~JAKF#t^o`L!NZGhdrTCz@)ndn zI~os<3(hs%wBXiPpj(;ha+6N3Cg2dGyukS3P}b*Uqz}IEVp*wKd(M%{{972zCr>4E zIMiOH*ryBBQi-*8Lta+h51JW8b!vkV)^c}v*c zhfVKe;~;m+s6hC- z?D>1j7(60_s%0z!b+}aXkcMzh%mX1DqwnIdAK<@o@L#hh;xmwSv|GSl7fBjs& z>{jIBH|*xo!~~C&$!ZR$cFXzhF^b41)F60^0-msL7EYkR7xK1hJWQ}UfpL+a)}dXH zJ7a;LITDVL9P}wJSq25pv}zLaydur93qrjIX`U3uoY$(O7|I?^Cn_ihEwmj;G{DOr zVakl`yh%c6F1X2{jK-b}g?yT&UX0Vcr>R{|kScyqcD`8JIa0TS+xbBjz-LEVs+L@82{}WKtLo7O%`(tIjf*)(e$B zPDFY4j8<7me}boyUBJ-oYaa!ajN&GkFPSHxv&AbZ9G+DLAd>`*J*>Uj|mFtXC2H`3n}~zv#Duc zJ91rLX;?FCd!|kpj(gnhd`Xr3lInIz^%JsLocuVk=V|p6Dk)UcAvma5 zIx%A3Cw^@bcJb@3nNip`#aQ)U%mqR1SjFjfNdEY+C`t>g^7+^f_4?K9i%MH-`vdk_ z6Yh3P1_*ChohWC0by~P=bxY7MCMjuathDEw7(#w||Eoomv(^B9Md#_2c^OcBsr6vM zi8l!c@7+~ar5+9^R_PSlcth&6IH4Vy3Wf{4#R&rq0o~~xZN}X7iL#A@EHOP3)RwvK zezaIfIX}D*5qjZu_^v0v97`v4(CTWXQg~xguw?yplzxETXr$;&_bzJVc8YztrZOjO z8ny&c$W(E@#D)J=13xfvZgr8PBy~nR?hs$_7mBA(M=+Ir#>;VOQp?EvZd{Dk9sQi& zLtmJZm&1zOEw46!IsMRAEzi~bZm`e{*DeR+dmI7o%*LISf8(Y_(N$PREsCOsG@dOM z5@H4hF5sj7z{3YEhQEFct(pM7}^Jwi#ogn8NbDouKj( z@451G>PuO3J*8(dPO|d7r6!6Jv*0cA7Q1-tKeVh>sCpI$<8A6ac-_BzpGn5|F_j-T zj@SO6i3w-t4)ngU?*NDUv4+hPq9klyE`L|u_k72c4BzfB;YiCi0-ht_>&)7K8c%N& zM?TrRnB7xq{Yqo*yi%EcBc_JYb_(2VdT-^zS6s*=q%H(1V&Hgutbljm^qZX6xD;y` zHA;ULCP*A;mZ%SGkg^b?M42Xzp2}Bydh7e2I*}S3lQ(Vh_l(Z;%AHm}r@}Cu_x`6d zRp-xy!3uQzD_~|(otQZr{p9e|V241$TdZ!R F{0BK0O)vle diff --git a/images/image01.png b/images/image01.png deleted file mode 100644 index 661c465b20c3df02aaa68bf65ad805b8feb485be..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5793 zcmeHLc{r5s*MDpYAzK)0WQmf=3}fuu*t4WBWlzSwHnNwL-7q41Sw^;EC`|}ivS%5} zHpo~)MN^2ZzwzB(y??y_|E}wOu6gd6`+m;(Jm;MI+@Eu)lG9xm0H6U(jVyF1>i{4P z0P+9;2UBhU00YusKpsw+$^l>~0Ebi73<1Cq0FZ#3Apms*y;=8}LL--FyRvNW>5ia#z>z2gKKhZbh-0!B_9L#)xaml(TZ;t^=}0z zX7DzqL*#z0&K5X&DkQ|1FgxXd(I8@y+xxLwM%WI#69|eI7%r_|$wVl$kgi)!^A2Ys zbYV8ai{6Id0n+Sej$ozsp_E5j zw|0-RP+FSkLca2dGM@1jQ6AMlRdpqK%-PGC>9n(FZ5LE~{W8|YrLgP(a+2YR(O6Tj+ z>rbDtTsM}#6U8%@!+)WmLkBY*?En&J8J0Xh-dsoAVrjn>CV0QI(e)N5y45{tPX`O3 z!9Oyj<5GG$)b~*uSGu0s!TJz-uu)fDW63G zQF-+Tol3y!CbkG}CIK@!u;)$P(=cvC9tV)uUGGz8QYJuoZz1lUO2uYQL@G3> z)4 zvttd_+$y{zFMx{){4b(>hQ$yWw}-9~P3^85VHJ@~ABznWW%i0*;tLwyW+HCBJJn>I z!DG9v@8P8t90x{!$~+&IBLl8qPaTH}ME$=hB>ElX6qWL^;KfMh%U*BX*< z);8u{{K?G8u#~4n?qW?b!G-RxBZ&6u{3q~W^kb^6HB6w%GSM9fqRGuNS)G8k?og&3vd3OWZ#MPf_;xx z$o8`;G-zccWZ0GW`?X{uAyr^_>6)oWs;J{OQJQwm}wyDZTT3)~(( z?MRu2rIWt)=u0S`~LAM~!$!qEe&tZIGFF{5VB(dS(`@+W#dNrv@Z&z@5!bb=lnNr(qBA`M)6 z?!~c3^Qb7{xDVZf?6V?A((l5uNUQbTB4oVYk*qbWk0t*hT$UyBe?V%38Z|Pu?e|UN z)q{Rko~;fIoyyyGINa&S8yiC5BHu7gBm!57Xlrud zn#V_0==RKFo}Ti#slD$K$+*-TWh=86aI~Os-|rQj3R5QSu?U~=7B$8F@zU;+yKnKD6li2x}AH`WL<- zP&jJgRU!N>v}(jQuI=WsSt%h$_+$UCSVUWJyISb0CUU!PwG3~Ypv@O-w_|)aFlxuk zvK8@+JNJco8o$uzvjfe&MTqa6;;S{s(8v-q6TiT`gQ2YlNgm;E=*b{K`#T@XSW=GV z3Q&zQM=cqoAk5x9x>tbsW_#lH?Y$QPh{Eg(iBrqZg<4M(XGdnUuem| znzG3{puP2}0FU*Sc861m%N2ef8h30)6;Zv?@&tv+y-%Gq=vJ1@0GX*GnhXSFd#kYs zfqgQd7TQ1?TvF+9LbNb1;;cCJOi1alB<5H3Sy)rxuo7KovtbHi?$Qbrw0e<#eWc;X ztp`j~AqL+-zFC57vK5dZxfZ2HQ`z$IZoKvrPB+LKA+!tE!BULC-<_{l2z+x-;GHQN$&5GZ*j#!;kA*+`|ZMr-tH%~$e+Hrv!7?k zA6k0rJZTZs^eA_HkUfR_@ornuoVst}T+KBP3mJ!@pppG4gEv$@&wsLSoR0>r;BZxM zU%B|*i5GhuSpDWE729i}c+zc%h?3D|r_y5M2=@ywqW(Id>T=SlN2*8$ij7pEZZ4;D z<_PDD*=h$#W8NFF9W535+-%c~XP}q0|Ee(R*qTr%UhtRb6RwkOE<$}mtbgo!;t)B& z2RnIw)%?~38Q+{524|6C!U;YWAybkLcTnrLujMrC66=G!Uqwc5A=!>?BpRuPCF*4B z;YB4r}tE!FTb7%eUojk34~|5@*Vde}D7sG}Zdx zj5k(G^Fu45xrPM`3KtR$EplSrAZ^iY^cFz`I>A@PikypvgOY9>>06?^pQ$W-u7f^^ zUkCM7k;qhBcBs3Z75Z2Sr5&fWCDql-ou{#KZk=aB~$y}vH*v^E9d zsY(agj*A5sga-DK%q_|Zpx?z&9xBX=`!O6`dK>dYJLKm@IgWN4%PmBuDV6B%v6?o! zjk5cldJ&8nOHp@@-6g}SYSanI7RN|Wzw%BUS3KO%SNb`o;zMqnC8nU`{Gs($_ZWX| z2bTH|a4x;xtYW&znlk)BH5qwU3=Xl9QIHJ%XeD$ zFbU+{IC`|<=t9-+$gg14bFey8&U#4LVzMsf$2HTyBpTA5iUH0`#NbqZ6K1^vT_D(&W3 zAf)Ak=$VZzzle)$!>-YIlAnZW6+LUS(s?5t8CP>53+f){p5Zp3bSeMXza>n_c!t%& zUnBufPrvtdzPmp@a^vQ>fN7B#sjy!R-o9D!HiJc~rL@W_K{-B~yEG>p^O#>qO|x{| znfbZ?i1pSf?tQJmy-Dv$oKsp{CPJ=;IXSQ^$ssm_98#OGNuBhw2oN1oQ@MF%Yl!)# z@n1$n{_urcuSRZPvYWF$)%!FrZhVK^TwKdfiuTMC`!*3l52;z$=J{ zwgH#0xkm2ie0FQ6QoSvHjc<-F#s*8vdDW%o+6Ud7}5|&Qy-`hLm zDmx^Zo;)A{Dc3(V*)0gZ12djMYi6bgeVC^n5R_1ot*T_S>klWhGxJbv&XkNaP`BVe z%RU;iw6Sd%IC$1wW<>JYNA8)=cPy~k+6E8Dp+=pq^; z#l>m*6@6X*Szhs-;0@U~`W3YK*3Rm=bb%dVW;%vpaB#Y^EKH~7F1>I7in?(dqw2wE zuk$bIgx9MP{5jq5xfA{m=X3#6DPYPuZ zfv^uT6Em0kW0iV^zLw#A4Q(LB@C`c;oVN$gtUMTIC=HLedATVHz*HCVh0{UIZu1;oMs#|NL>uh8% zqDlmP{mg};fr(bje*>3dhm{52fY!fAe#%$rrjH)VoPim1j>!^Pzqn2Ny=0G|pWujt z+;qvrl}fztp! za1}{ScUxfF_!utLqqyIS$-jl~@mzPHo$?@!6sx(eR^7v9ioHkkDHic1Gv4^#lY+kB z@}JNDuIzu73@{ow_ILBa`=zxLCD5a7gysDwyC>?`!L~6MQ(n4T7$5C^CbGTRXi#x_ zN}-e3;ptHRqpc9k%XT}nQtN0p(0?^UrTEKly~EmwSn@LbPip$7Cu)G&v0#;#Px;Os zzVnc8nPXTFGbbZS- YCc+0O%SqcShkpR->s->V)O3ja56n$6Qvd(} diff --git a/images/image02.png b/images/image02.png deleted file mode 100644 index a909353a038fc30a599a9641bfcdfa6639ed0891..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6246 zcmZvBWmr^Q)HWd9A>A!4p&-&A?Z87LDFf2YkOG1t-O^H$Lo-Ns4Bas>0z>Bjip~I% z@A!Rx-XHIubKUzo`?}ZJd#}CjwbqH%2WXHHGZAB9VUcNRsu^HmVasFi8$x`{bDT9| z0t<^lL`zNCC=mP9%F2^vBtzcv>TTgIfv}Q+lqx>8%^}tsTc;RTO)3>?Wn7@L^II9n z+n|^gB~3*q_a_OrRByaW2`CRRHwY9`+^qQ0BcUVtX1~Tx{hc4a&gJBRLAF zk8F%}q=uA~xWSAO^ZPdEdflWS60H7>DX=-a5qZ9hjU2NJgoH{$n^j zI&#)KFezU+WG+wzE}9Ond9e@wK0Q)R%e*y^q6h$mZi>yyU zceF_)Hi?+MKV@(<-Vo&7l_=Q1Rk=Cs@JE^Z`LW}u1g}e1oW-cu7nkNZI9$i=NI;Os zZ9_fJMop}l7K91}TZXQ$aRMZJ3ReA2OCAsNOCf7Lcr3p4th!#yC1{e10frL<%B6d( zEo?UNZ?0k#SsvLQDP}W=nmxH zP3b0Z(~79QRpx@Aq@)O3!L+&+R;_Jv(g{7kiUM^Na-x|zd=`S6q8xO=H zl8ZA|oZ1{-AmnjBq5LUsDwch!*gpbXUkO)wg6Qyeh?%8jdX-?d#D)_Q$fNeM<)_-b zW9YByRpfHR`kkpuH2c=blwN=$w8O{ zTRau^)OM+chj6&Nk+hZdr;$PI=jF8-%?4@@yG9k99Hq<7lCaGCr0ra;ms}D%2u3@t z#SOP^wK6Popy{?s@E%!B)v=n2f+RK=%$Pho%4fKxq~kHnufJdegqq)(Wut`GpKe`C z$H)!)J(uf#zLQ$H)KjMA|B@7QhU9q-G75&xZ3FG#}2;D{NCbXF;h^u6Kg!xgt z$fAq0$Sy4Rw-ou!mXgGd-Cjc1P0oxfK}(#MhBw0ARF~Gr-Po=EGtzTt&)1WiqqMd4 zphCX;57Ar7JC@dO3_0Q_@3|z--1nc(d)Gd<5CgF1QwXM1Y-nnPtWWc}r-B3>4U~`|j?0 zT`Mf5_PZA~*GtK$#lxU(dp?TZq{6#0H;?OFX6ElVyf^vrSiNtwTL2nG`o$mQd%1+~ z#nP+4Og<%n2fnZu`m(0a%qW{_S)MkrWI7{N(!6-ae3_>UXwLAV{rv{KUc{d9=|0!R zGK-qrB>c38*$NvF)?Ia>P5axU1iiu#+@0~j`Wfp_zFBa7NN-i=fnZ*{Ze}9*ZSDFW zfOolpqpX0tZDw;q!AyJIUSPw%7`VW>9HF<=Tgrot&SIE4C1Ksu&fq|5CknJmouomS zEw(0oCo^*g612|^wpGm2JobOoLxbH&?{MHpQn=UIe|JNJ^PjuV+bAnhI^kmn6C+@QkxC_)d;cshd-NaH;QmE=S#W0w$HX7ZJUraTM_ODZf5H3A zpEsA~!VH?a2}jY{2^BmNY3Hd1|E8$m@{}(}r3as~8NX2UL-2VItTz73F;Vh`RQ_a7Ti{$N{YkP9n^Q$YFxU z`HE&N2M3`aCk9)|fF*b^v@se=nqq9s0#q1lqH35pN5XIXnFm9Aj#&j1$jytfHXDV( zpsmvJ6bvl~8^+TE-AdIk);cJ$(F*eSR{H;Id7+oBA>B`5tc*Jpm-&6y*VjphbY=fG zg_M+Z^Tr1}=OWMj;3i1s?HTf}!=VU6--lrz194z%%18B0ryyXgw>Ld{zc|RXjl*zG zm^#7@ukK7Nd0%Y63FZ2r5t41~>+q7pa>Pcy*2nd3M?;#nHwRsAi0r;58tDc`f}#2P zWK6rv51~O=S=O2yIwehXV5S#; zPvmf0JhvzNpjXWA4icU8F0zS!Zg^QiA9{KfKffFaPwZ?C?g)L9-h`SuQ;bm+hXg!a z(s`*P%T~BX=pSYusQ5e=-z{}Y9{iF4aj5&=09p%CJ%4fGKa*@La};`H5*AhlU?!3o zWl5MLSd79-UEZ^B{1pwS(tH1)%+fDA9^0j6vU2{qOG@(7a##)0_*qNhZ+tH)HOpuN z)j%^uRhqwEASjt-?PmcZqU3oxV&nXVC8aR_QQ0tIkNNOvmd)@C1r`N@bNMnq8W44g zH1s>8>AJ+RNQ)mmYg*rm4x9_W#vmUr?mbi{IOB>KxV(iWY2T_G0 z*chHdQ0)H2b3}vHmL}Xkpd|pces{txC+Ca9&&(iU@%HNLJ#k-OL|dGN*#n(dtSeD3 zH*%JH+}$$Y%gVV$rXus|mZ(by)Ek(eEPoeq^IFtr^b|DC9aP0#3@x4zft7qRq=^=&*!pkKZLgXq4sW~8baJ)^O z8RPqf8;jBV6Y?%*`qV8Ui?&~mr|RKaqNpzdzOp<7?yo)&o+Pfy+`&8E%==!>;{TTe zUxT1$0FPSh4Z40ZD0GEH>|aPJSoB6U9hbB7yZVhe0to}@jJulYiW9f}PKaj`pMuFa zXdM&W75jlz(?feu;=g|kN!P^wvx0T9gYPxWHnu~5tZc7c(T2T%uzOnnrA>;g(esHm zH)-E6y3lMNz5I1P{Nn3VQHjeDF>d+4FmJyo-eMZ66@H2<6_Y>5aEeR(;H-)(yv7W@A z_gB>YVyj79zc$%CD6IOB*W_&vCwXcP@3?(RniJV`J9ipsOmb4i3lm3kWq&g!W6><9 zxYi0l;y5L_%4rJO2DtsB%tiZE)^5u`M9(mP`@J2sk+m^!ecsmrG`&2hnPGVw|7X;q zKKQ)9cB=2%UZU&Sr{wxWiR+f0zp$LXS8OjLmV539RRvh1S^kni$-@m>_kKLr`Pu){ z-uZQ_CLh5?ap4&)o3ea#Y9U+y>BG6@+G&XWpPoYdS3fd$@FuEc;wbL@i2`{W1UDqr zn8F=2wc-ig6pJ?j^X1fZKhUg!gSquF+KHp;Mp=e*I8?Y5qj6)C(_K{wLhbnnB!OD# z6ce0}>zky2O)MF^4EPH&?cm0+li1^wa*TA2h{T_R zv{SBD3HMiUy21&wj4#@gcoJ;r1BZWjzP;o;3xCBnZp!~PUb@qhs$y*KmSbEt-u;x2 zoE1f!0c}aPT-#8HuC0%lYv&P46B%g*TDtNn91~RUT_J=DHv?8SqKBH#{;e*<(&kgmE~!;~LEmSbISd6=|RBU(hwqjK)B+-+M#a&feb; zQpCPTp!Ev+njFTbb9rLItvjb`M&N#t+aoUDA|6G1`Rx#$A)aiWPtr~bz*9&IcSyaA znf(f4b3|r>AUK!3aUpxA8A4rcYMIt;&Sjm~HhIGG$ws$j{9%)|;sdKa_p}}lPPp6z zx*EybR};J2*8IzK7eS(}kS^o8hGu}ih-d0qcYsEgrPSr8ixd10t>M+TEh&b@vf+QuoBV_bG4SP4oYzY{MdmMwh%7GrK z_G%j=f~`45CP${F7JYkg?Awv*T*5(64ufYXQw^&TU8R3&dxGWahDgpm1nP8wym3n} zGqnNMJ?<=tKNHCK>SeVHRAkh+$fzE-lcH2kT{C{O5P*|6p7x$x@nMHoETmeBt#}Cf z)e++@L=Sg+&%-n!ySwx6GQhyUt&)U+TCtM+V^L-JpG7T;+N|bWk5`awPyNy)zFWHh z4+k9mZt+hg+WioJ>jJ1|e^mdsj-%D!aUFP3y^>@Rc4Q8VW#C~eZwhetot@=3`f>{WJo2}r zUteW5XNX|2s3u@H*OHQ&69xOhdO`b_SQRaF56w5zX22d|Eq$m z2%qTG8-Ty-*vuRWxZl$rsVW|}dry40RE|O9gQ3oodSAZ0bpPQ@-=h+Is$zipmysz2d9w-Sd>HD=Mv%hh zmAAjCl^4Z9sm!PLo1O^k{WaM*1PeEGC~Hx;v(nY|U)mZPn$-y}q%&b5ZMThvvI|CV zQ;#L+Px|;uTOSq0X)705P})lT7I_-3Um}xX)ql%X;re2~0_c6RFDqJQ6EU=6mkHk7OhZ=BpF@7qStlpUs|45G@}aOw0v=9b?Bt}1lHDReC%I%5=tZjB|@ zzaaO04wNUTM-~2iEhw;+{?DR!?rXZ^HP*`)$%AC4A4K3&`XZnBm#X};Y zOEY#%FTOe|E&=RHDY(-M$zaI;=Iw+F`-f!O!6g+bRyolU7w_AUh7!H8<`wD6 zW^H7vGtM~r`7VPt+W^Mqpl91N?%(=mJ-nP3ZxWFW#c#g|^Mde3B^rs^MVxW9r>#pw zfFz^;qHyBA{BG1?np=ATu`f&5b6$6XGFA6~IBWA0LKK^M9Vqe0yR${ixqgxF&NNee z)|}vA!373bG8R&VoA214w&bpP_RYsTMf}8t=NyV|Gfi16Zh7S|D~+kdLkM^s--{ zjuaEg`{NH~USivsSG{zUPqQC43k#%9cd^Ro7sb|d6g|1u_e>vu5=i}0!@qkhDsZxm zM++tj)Lmo%ucM^ka=Sio58qNZp9O;s0?N&0=<=F&yEgvjrJnJfgU@%aXldT`+7uF9 zi(-}7V`6g-gvqEuceX@@H8^eJFt`d(Znx z7p4>Z`Fl;viC9$Ve=9fIYaQA7@oV#C{^+QDU|Mqz!`sGD;7=|iT&K~RDt7BvuLAeu zrz)y;Ywp!sXusBw^eT0Ahs2s4ms5}IQxjotrDTM#h%VRIRG8J*6-b{^K54!Hze{^LMuuyGu30NjM<2qXZeI9w}w z2SG-1(4pj-fkhdh7Tb2`YU>|!xhn`f_)lGt@#$@oMkRcpZV zyDZ%+j$)8`#59i_nk>%Kz~290L}cU@UWcGvi@H6~ zX!K}ZxrldOXl80&b%18mL1wyKSP3~%iF5XYS|152-(vPY_FyD_C*07Hm2djikZ4{G(h4qfEfhb^`#&kl)z*;qi`TEk4tRx{;#EL4KM2`!+~< zam53C4Lu4Zno?!%nBI8(z~ua~$G_6{{bGIb7{%(W!|E>VrQ*;4ojsGQWdG{3+)uhy zryMv_R>>^n=6wQh(&?>5m*1OS?xVRBNf&Puv|lS1g-|+8=qL?(d9f(kkvW^W(I;8N z9i6I#muX>S--#|_*3L!Ib^2UOTQlP}E4)M8|DdcWjCVa&CZA;_0MRy(S=cH&Q0ayG zOdrb3>G65>=}53*{sc5B|X4O zeU~(4Bhj(G8J)C*ELwzoowYDI#G`kq2GgJbU8Px($LQ>`!E9p+ylh@OM7&z$vN`;& z>3Sc;wRm~G9itLFjmJR{sYK7!x4SpAEZ1t>4t5A5s_cek#tsH|@Szrx?ZxNf6$D1| zO4AFdw&_LFp|p1f`biTx=;q@+2HGnRuy&Q%o%gjy@M({%gOn$T&mPkT`o)xm>FSW< zxG4KilE{Im2g7^%_>y9>A}?}2sisXg&^od4JRKuMe@Q`qz$xRpssJsO*XS&S-1#%p zFz@chz#UV)HU>x3UtdYe;@isE{;i4nuxU!dDG;7ZIAonX{p}}5(y+L62)6gtij^?i zq9HJD4t%^qLDbNYiQaJsScOnEbsxU87&E^kxvYhGKR!!H76y=S;4#U0c!)cFOpIyc z(C7b`Pd?&}Pf!29TU%d3m~Djr^_MU$DSDg-tR|CmBzNjU2u8iO(GM$T4lw z-e#g}dg{jQBjLGP(>ph*3nC!P0U)d;pDl?1y_C*d9SE$Ccz@6f{zvpIy={L)&r+nJ z_aW2v^COy1JU2O-?CXihLo)n7TjqW~NTrMXgZWWP_jU1)J@v$)ok#9#U1fc!r_nFb zc|A$%Ds-_FHhF}85m*>(GGzPsn7ZsXSZ(z6K3Tpc)7mxd^ZV1vb2N2o>=Ir5z6R5t z*PeFj}Bzu;4h1I>$+@$L*^k7OOwZR=uo$>A#+w$E6u7pc-5HE?|u%lrx8otN$8C}`}iIumfPuz#P{$4auy)z z1C5LBzc?T`R}lhiEQ=7TGkrUSm#itv!#56mX|rw?JnBBq%>0*_?zHkH#zxi6>Jv7rpW}RyP;)Aj`1$v%3xa_~g zIlxDqHo%>ZeDKjZaXoaefV#@g)BBpq2bVUwBAx$LMSzWERRja&aTH}dL-K%ZFDUU4 zy@KY(@Xn(@M_wLnJDNny%qR2s|CdW?oFLCo`-SkhCtXVn@~Gg>b=t3_VrITl-RuET z^3?E78GSmOz+IGwn<=f83Ejs})$cmMgWkFJ0CP0X^861`uOJNJNk_*``U3dMp|R9+q!%X2M2XJ|CU04jb$mgDr8m>WEd__liyJvl<`4TOvq$I-pVX^MC5)8t1of~v>Ot+ah zM86mFtT@G>jL3F+-@uKWc;U$_T8NFD_;Yot>&SS_F1Ck{l^R0l7+)KgPwXd8r|#`o zPCeM80^c~BV~n8pUQ?c}Cz21VAi&Y^Q(|MGGC9#rJ9?`lovG+3$}~~iBz z7}bn(W8rAHiL58Om~x@|oM!lCtSGCFvK#$WMAsar^~IsKit;CNbad9}ujTr>nd8Ax zzdVqRCfj)Fy0W9k5=WJ^Pes4YlmvQA_o;UV)0&hyGki@x z0FdQSCm*Xa0weMU`vc2Se}ASTZc5-rzZH;Q!SMA6zWT1*EsJ7`pAldLq7mQ#Pc&4Vkr7}7{xSk=EdO%vbK4mK zMj#r2e<&x7$9ZD}7y(9r5y(bB7|S*Vc*t-IArxk)0TX0M>8gb5vGS9Ymg-5Ej=Y^gLqx`};h#R<1=}#|aZwWr0dS zHDg(q7B$N04u>hKs~W{u64g{ER#J)zt2s5~?}fScA}R~5UjUJEtj13lwgRMlK(6lf zJX(vUVT!tlrVo&p6#4(mGAU6UEJbn~O_!@}3Tncd>-lxYvSxg06Gzhny=foTj;C3h z_-camGOewiv)sVTYMir%Q8fq}E3K9R8NX@v{26>*z`U@*1DTdg|pU<8ZtX1IM4UPQ&pMD z1`=@2-7K=#wmJ8Jx`K3ha@WY{qnfew=O{09jw671ZKsS!d9(s4wMqXgg5)ZaX4Oow z-j7_wtNp{m3+^LY!esk}3!7ZdMk|pv6-=vUEYW`7O>pS}wPc}qrK2IMemUJ`X^L}! z_cRaKR*zD=n`XT#L=U@G26*%;l4>X`2$G3t*d;}_kZ;fK#b6wO487tRz7A-XFA;>r z%?xn^wV!ZGu0$%l^>VW&82~$Vt=h18#dAK$~ec zTrt35J~s*H-VsdaIG2hq%*%}o(Cg(5bx|h5#GOU)G`A97cc# zd%Ojf_lAKyaENjFq)1>?Q_!oU2=$0dc0B{(i1 zZq6a}jcH?M+iGt|s3zIAZ79h(hrFSKCA|ALzTj!~IYp~r?NpjNh`Sz|*EP~uPGCOY zC7(T|nnNdu8i{OWH%ZkDAjmCQFDWqVlYm^b*CwnY$Yf6<_obVk3}wXk_pxLl1;owS zguW4N%W1&*@Bb$Oukm#d&jy zn{x<#3)`5bSwrpct1C&4eRYdQarI$Mtl1qd)n>3EmmG;iYn{4eeXJfR*li0|nPjqM z$$fEJ2nimh{W)HfBS-}x1ss8Sv=bKHGBkUdxLJ6QIOUW@=$qNbP#>MI9iiHb&vijM zm-41h-RDL?Jy;uY7$GT7_S~$l<%}hHm$?4PDrNDMYi?{Mxz9*Lgcl7l+Q=t6dH{&T zGBsPAz*ByY^@+!=vtTM@D=_bSn=}0|TqsV+;yb;DUUWgq7>Bhh@|5dANoY{l^2QQ%q9t#4(`4=D!}ks*)(||C z1ek7CBnw`d2@n^eQ;%4mc;tQ*E7AWUGTG|nzFiDJ&ha&u%sF)hQS=Iy_zF)ZadRG_ zZ)O|g`l$AH#8q36OObq!FbLti8kk=!8taspD%V4Ex}Ib#$-8`;4^$K7D?sd4a1{U~ZngtNLbM?C-N8stG4;M+rsp#e>}qYSw;fL3S2&>imb^sg)j&ty=~z2r zYB*?0S0?{0=$y6@6SPgAJos4GJan2#W6Ybw^)%128GTqPBftnS0J8R~5nu#bi@^T@FOZjIVWRvt00000NkvXXu0mjfB~Wol!q2;U0p z;8anfoQB1=uOezh&gr1`%=7)_dH#dvhv&N9_xrl8_jTQ`*L~kV-RVv^TZMI7)&T$% z?Corv0ayW-=D}-MNtruxE=3wvL^<19gUXJrW70x~V0F|Afcu&9OMa^XSOuK0u7{*L z04M;^06_l)GikOgq0j(j2G9_Iqy~Tk09XJ#0Kfwi2Vm?0P()x7425D^9Zj6s})_z$*=SFaVxw z_9p{HWSjk2hlgnBi)j!|$}EV+g=j)7rO~~FRb&p5tXhLZ&DHjR35-BA8BWe93tp1U@ zSrJ1bq|J)zBoc0?NGK5fEtE)RMN*PN^6z*4KmE_Wz~VWbJJLhRh1+>X0RU@SUMpI{ z?gU6hHPRsul1mt#6yp~e2*Uk>LP>_9fh28HLnA|zjF0(F0Ibckx3O}KJvmu)np#ZBv-T1U{`&blv8uM!Am8E3SKimk ziNp!IgK^Nu_{MAYW`nM0Q@R-!1$W5Z8BY&0vvsMKh~1GZi!59AR;`(wNQ(~MxQ924 z_&9Om_`zLUCf|f|h=dc!zDDm%pUSei!p99GJwr1&g2ezrI6Ki{fK)d8#tAFi5MyJj z6&}{*Iq6ce@T2o%NLR)n)*cBo@Yw=GokDbZ|A!cKW`UrFODt!!}i## z9p{dRFx{2_-P0Q~2g8=CBs+YTY&omehZ|Tm^J%%ew zk6)*L(;A8phfl3RMbUDO$MCula+~dKF?g|dlq>y@dxM6*Qm#h5dOx$HcK66b0lBw> zHAi*2wUa1^a@Km0H&^HpS@9%?nwQR))ZtfBVB?r3970{ra@Q)BzdN06W5=MtA?L|# zx5FX7UL)^@a!L+_IzK%HGZq9f{ZF$CsL1mWf>RzxgITC`m-5rg*FfucXn5LB>a{)0 zjJKNbCClHJ z-r#Uv!-E?4 z$70vTzip2`jH&%{>yz9?oSJa9y7R+67CHS>%gM{02>R$y8QC5g8mFn(HBfndR7XZp zr8Bc$wTfB9g!}5Oo|KXLH|4d08U4+(xGr25b$61q5veovUyY>b`6z*rv0O;Vci2Mt zBr7j2c>6}NspRR(A`7%Eltc}CATJ2=#4Pse^Z%+rt3HhRef7vd^NWcxd@$eVTxtz5 zP{}Rq7d-cVtGxQgisb7H)1p1*k&@};kQmJy{)avq;xjIc-Yw|w=zo`c0TW&v` zh|=5{UrX&2CXaNkfU1@A`~x5i(6R~htEOQrS4imyV9fG z)--Blh%E{&X1^JNiuH*O`Wy23J#6Q{M9&8~X9!zmYwz3Kulx0hH$1lco8qAIn}nUX zNphOak*2r---Fa>1H;M}q(|oRTjs(n1j>_28!7EtnYuVJ_M}JpSa`%5(@iHEQx*D; zxN395-`IGolWq;lW^gzd3S*{8ag$@#RZdmxtYDa+_$V=*9+;o-y_xhr#zt18L;FMs zU8nw=#^cXrYPDfO0D4J#cdGLfA5C<5drNh75ik=tUqaIYx4c9qIvWh<5>6ssSf_p zLB&0o`~w$tlA0leSM$3{Qi{#k!}8Co?|Qnucy1{}lnPIH=DkXSWPJb7XiQ7NZ9OJB zJ!Y|A-_2H2MRn^V=5ag|xZqs*evWjM3{VsLcKG-UaE? z6#ZBgQrMhOexlvBN`1Xsg+Ylf)bPg-;pjX2dl?TM$$?AyOqJ1hB22pJ+M)|pEjM_$ z-{nIN=)K$b?8^vQ`CC#_LgtXur+V^?C!VVmPFHuj+gf3ky+P*0Tk4nmr(sMpQfjH$ z*L3u1yC(|bLG9F%+eEjDNIPxU#+~A8x@tR%+zL%!7^JG!29N{XDvu|c|CBkq1GlsW zS{mQ0=hqE@NDzV1h{4L_?Nfex6*h2|J!-RZ*;A-hF!EsF+Oo`_zQ;Cwa$` zBv_ry`q{2u?P9e9i9CD#l7Ij?_}Ofxu7ZDve5Ph+8zoIgX7dUwWG}3)1Rq(g>*-nT zAEHcR*2m6Uz{jo!h+VtKn@M122PO>`;tkbY_Tt6ez*Rg?VR_m6{AT$0lKhcARU*T^ z>U7RoZ-{%XNAnd%J?KcAg1qO4X{AeKd%tnfOZd7e>(XlhBsMAzLwNXBY&XT6xa+6r zz8@<{^8MP7Dro(ls#zIVkTV&9X^|(*o>>hg;V|7fkKD)Khpmj&+n}i8+u3$cp1yJ0 zYj@jM)_jvPLR)A85y&Mh*9zdRFX{kUb!$WKq)?9x<>c*7JRPh(OU$}2UpH6TXlQQY=L z2yPMaieq9_R=fBmo|ghsZAyFAOHHWjnI>z;?Rj91T&l{BVYIBHN54^xaDhbhQzJvG zweY4HgPi1@WjFVif1RP{Xn4NCc;EhX-gpe_FJ$nkgE#gC@@q+!t4UqxCt8lyDx^?J zH{wv+{LfBy(^-S5S3NEk8~QH5wOKa@+1Ux_obU0Qjw`DC<$`LxlZ#2mH?~R3?5wQ| n5SE!u-qnRB*EOTb7u}m_kN{$&x%KTT)VE z>|-xRl4O|~m8{u9SxU`s)cd~A-@m`!f8P7K_k8bJ?)lz(&bjB_d)?mFQc_$|8~^~x z6IRC^0YHGt@B3{5@ge<1RTKXs5bS7a3Y5Q6p5`M$=Z@JN1AvNDi48Ad0005@=+owW zIsiZc01f~K0D$E84*Un%WIzu925f zoJc}aC~#svl0rvP*l02hO>T54sKZkzUb%GyS|grLx6NMk%4Or}{K^ZcS&VG0BkKYb zB)=lbXduM}$if0;t^gem&c^%a+p=+k1=dTst~@$^sXl73)sI4_kQ-^_Mt&jFHsK~9*Rv^Xde&GSolU78tD~^l zDMO1CHm_u;r;Nic8Cs^!6m)7JM*(DelX@WT_wkDb2kIhUIfC2J|$UX?LC z>Vj3ec%IOd+P-^DfG#U*&%W_!H`P?!fOyv2k4QWz0Hvop3T|^WvHIolewEcRKZ3f; z!i(IC@*0_}9;&aLWQ|0`8tc_YEmW5H?~r^@s+44IRxF6Oa$T$YU5r@L}3# z^Y%y>HN|BJc_)@hLBbHF#FvFJIFtNRPnCtZG)aS~7y7$#^G9W2W~Kj_P?pKJ>$XUA zVS^mjhQ#31I;P*5bz=T?(huYK)J32`v&8qrOHbkCLR!Y&y-~8TtXpmw{~ZCKMQy*F zWS&3Av!oP?In7NzMPf{8z1pXtIrNa&dzxv_pQND0DFHM7BuSF2Uur|LMRta>PYFLq zrTjA!dLIPTX?9I%l^H|dr~u1h&r^~JQ9(~v#(7pUp;NxKx;sG=fljEEhO%Rc4+j*J zpqYk$7e>ycx9AL|7f!t^Qrf!KD%z!}sx;}~ib3}}xrP^C>37zCRc1N-iAUM}nVA&! zKs_(|Ykp}Yw|up{NNl3jrO~_TLd(NK86mwN++~-TfS>m~Ry!@WR!B_gW@H+^J0yss ze-bXr5Z*2yzC2hcvDDt#kW$`U15#P~#Ympkp8*d|OEw+7YK2VW9rX3KS@H3Hy)-iR7 z!b!*FU=5|4Q8jJeS&3iw=RGR*2$>m-KfLjyRkFt4*giz|oyzw$C_)qJs(g8={t9e~ z^*!$kqo`$K>L{@gA*jldxnNW|5_Wgw+aacMB!QJZ%h(`Q4gD}bY-mKc^Z6)@^7UiAD_zoj^2U<%_#@j^)o}XLE&Wu^3AVXT!Kkf(bf;HM&2#QAMgTW@kSekqViQ z>>5aBki<2(YUkimQcn`*T|a|iP2d*$JU?S%?&8HYl6Y8`f1%SBZ2vgvZglTpWl>|p z@|4-P({DtA!}0|B8daqyhDW$>YVh4jdY^-S(jW_PzDwZLp%411Qo%iU5{+<1&Ga^H z)!`o96YKl+Rx@D@?MulrHJdf&!#TT? z6^}*aJiKA4ZJVwdeLLfg;iR8?&V>NtvB$XcA+qhtX@+;shs+T-(j6?t^P(5F{CeY~ zo|n882>gt(lR#S@*1@{PXe4dz>B^u$6!JZT;s@Zm&TV|pI`mHom&U!6Oqdai|7f)I z<};|>BVMaU9d|)K=bujwXf_5YTtchkcE>JtV>ICNCJmYkd@4H{#9`E@KMp1zxvWm+~++?PM%!RI(p{K zowrYEK9BiX$+7F!t$J~Y&5487d=Bo=-scB?r>#E%22|mQc4Xf4P{ixcr&oI)a_x5Z z?9fj&z3Jj#B^Xp4{-89SB|5An{V4aHDs@;(S!lWVt?oC_gQquZ)MEV-J*Epd{=%b| z7EU_pMUd=1+?%g?qs9t2V`3BBQ!fF#mQwKkh)lRRhTdB8V^Na2Eg$O&>C%MxAEWtV`V;+~Z)&P3KI%?DfU*FzD9ID3?;jm#^Rg*D{inLT=wg7`r8Ss=xwek8L~WPD*P%0V8F&tbVFh zHG}P|7q+-JJF?y=`k&Q21*u!R_KJAVk%i*BpJZOJSBL_?!CZ7tEomD3DhFX#h~8m8 z`t3HqJ+&2P3-FknB#=6f@*?i8VlBeEKww|Y%?cc4M5q#`nJt?BZ`plz*_FC_9f8Oo z@Rv(r&&2C%aI|ghUd@-kMQwzuO~fkX@2D>_+8TN^CRjsn+6GoU{I=iuABPI49GqxW zY;M*g-kvA}KbSKuGJu%J^+Jvm(^AhsH?8+p>#z6Qwndp&7>f}wuvyLP-mi5}h7#ft*FFJbClDJRn*ko{h7xX_LK=)Z2xo^!bn-nX<6T3p!;)zu!feG zi$LQf>#w%W{CI`iwg*NvNN>5+zsn3WIYG`z&_@5ta#^yU{lk8>3IUOde#n7Mo&dPn zZW;dWI@vn^N3jU=P$ak=o{aYaxrV0ReIifhFNl9+w=x7^b#w5lVWI-VXq5TlLBz(t zlzDa_V243UKL(_QS2sHWavKAl)#@X z--WF*BqT7YpR^cx6OViU&uLtY zreR@(N`f*tuY2M1&o-S}Q6OvKYxjOwRWIj}cRuue!b|-Mv9W#yg3?BLEygBUcXNwq z?dy65l5{&4o&R#re&_Qy&7@dTF}*`VeQV{O+A&2A2Lt}9fqG1g`ZB)+52j4as4e$* z$L_V)&b}x)3!j(xg4c(4{*G$u1$P{_HxT*hQGpImFI)^wnLyk?L2QK&!d%~2got1| zo|pf)HLN1V`ikSUSNC%?GQC$AGO$9C5{Dx_CjY1zK2tl-fse0fU#&P9tlXv#VS?kX zu1=9=qmhu>v|Ok11~mLvB<%8Jc=D}({>~Bb=E589dowi@1`toOw_^No_Fp=DBa`Ry zJYZWcDebyom@ZAMPK`_^WLNf8dq*EDQFsiQH1#Qo6R_OTViA578~+j9q-$Ed+KJZ8 z#@c+rt6y_B?10%bsfuIc4M1aN)7vvE1u3I7m9JjGSF{DHpf|m(L12S|%c?~p3sS_0 znE=LT^FwwA122t;c#FUq(~feED@K+mEenBdZC&F-JbwtUp(NX5h8CEJ@MN6>!n+CS zqX+^V=S*8e4^2CE&N=@MhCkQSQEcI#3*nCUy5{_wYq%)d{@;m!=A%Q@R5ax>7Gv}E WC$Bu>7Twx>B%Ls~Jzj2#i~Bzg&JR!k diff --git a/images/image06.png b/images/image06.png deleted file mode 100644 index 69b4ee623dacab0d736971b73dcf5a4ddd9fc66b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8028 zcmZvBWmHsc)HaMrDIqD+(jZ;ZASvD5DJ?KVr=Y|T(j9_Bhjf>8cXu<4^w0?3_&(3$ z_x^a-y4SsCtvS~|*S^m_d+&4Ya8+elEX-G!NJvOn@8zV_k&pm_h~N6?sEECi=Dr#U ziIV2Ml!T@y@?qv0m}V}qFNR3$`25{-?n^llg=Q0aWcI=+hF*LIb^OT`sxk8nmmjPz z_P!;C8ndrPZ_$$DbK%4iaVZRqE|TNVPN7Uu*VLqvTUg6IoBcWrp7hkW^LZ5)Ko!to zH`b_Cf1q7dtf>z52za_uTzm&x9&4Qx5!};UnReOxuHYGjgdX_M>Hy$*%H1Y3p6mnM zv5Wx7xJduA-VUKGGGtGzWm+Na3;n&;96vC(dVUMM`Y*mCl^#iYPZ38<5C=eo^k;#U zkRnFSW7xgZcRS;^dp0R-wp@^H_qnJ*R zh*{Ji0W4_7i^{#1MM{wUDO4ya?znmGy)w;`=}nH%3lP zt5gfVwO3+?nzl1rn`g~+HDtc6#diaJSVp75Nuu-5g)h7t!&IAauqg}H%tr=P8*Rgl zhe$z}$5^fiPKuubg*kB9agl=bs8x@N!ndijjOsp0wH-*aWN-66RwdSHlp76rPY(M^ z^^`lkE@k5iK`&!5fXaiFo7D1$Sap97xMNk4-7Lc&ejm<)zxlbOPal#fO0~H|(l1=j zsyNORMC;!<%_ir=)Vvl~w1Q=8;xW@Ri~vW7ptpB z{qV5fE?B89F>(f4QlH9!El#VJE_p3>KdU~T$?*#?YH2FbacnPO)BY}esOA=fJr9pJ zX8BsY$kfHlbNOB?lXYBwajx`uy6TH(nS4267%Q(JdM{KVS+}vOd-zJmD zW$L!+h-NP5?mLX80O$AociXLTtHCCm`bNob<_{k75WGqz^*OoED~~KtNKG&@J|Uur2$@l^nwR1$DvJ4weKG@PMN4a068&#< zW$ALrHbEW=LY$1LRb}Fhwg>%1Lmn$`Y5BYgQ)a(p|XK|zL4?k~?oIG$c~3VmXx zqHpq7!sRWVP2Esk+er$xLswTW0np5crN-(ZZ~w^U@a^u&@BIB|k(CNcwklsr`imUG zp5*;*aWT*-XmV_ZMG*Aa@q8GnoPU|t1gdLFjOoM)g@ zo#Ap&Wu7_GKpt(U$8R2JjzVoOkh>i@S=pN1B$DV(WHI5lhhG~>!$v-x%bw1Blr8A4WVf4dSB%FMsjBg z?BlW}kB=`xx$Wzdr|GWreT)Y|yBb`sF!w-?wGmC-^VQRBCgF9Ap@FL(chfv|FKhd* zb2?KTT+{sQbW?cB z3&t~Fi{*{*Rzv%f9-qtQ_??l6@R*WVq9Nm^kOIyUO#OzuD#mGQ>*Ow?KUvgkEmt`S z9Xr0#K?I7psX5AeQ`$>A9uz=4J=IuXCB^G~=Y`|VvjPS&o2%s0mh|n5Z>WDV;>&0x zL!B-cNv3}jGg1%v5ZXr29CI3uCO6_;T8K=Q#IqFXXge zowJj)zlC(Z#fF>+|HR=G-0=Mh_%~ToaqOYw><`}{z4QIgSkOnrJN$Ej>M6<}ntzy% zVu(+qo+F1`|C8Vm`vw9OQZ0e`w=85J$^!S^6ZBI=!2ARIi2z~P1QYxYv>3NLk5=mzf_P=b~BI&G6g8b!~QEb zRJaI`@>|mH|JX$A!vctKaT5*1rTotWsz0EQd~(SDnN4v6;Dp~7Y1)l8BmYO)loNx@ zLz$@B0{=iO0@?jB=)w8j2&*x7K=$Z(K|o`xCwaPR%){vh3-IybH%4up=ZIm8=Zkap z&L8Fvbb)=+Ru7=&70bjNAT2VdRGBso-Y_?U+qtm!SMtBy z;yOGDGN-Qgw@TzY=pAZ*8lpP>Zq%rWNhK@p%+(y^#8!E9$>OeLX!eGL9x^ zz|J8o*-oR4Vn6TGJE?^w&sFyMhvv4Jc;gl-=D+jr|7eyJcyI7S$Ej7W%px04PP*7F zBcc1$^YJ-q6dr`jO23HI>p<#en%N69xZdQxOSZdDm?^YY;7nnX+ZUQ@np`nn4;2NC zo>jnN0+f4HDDxD$PJHYVv+~g3>zy2*)lgAVkaj8aJEt)Siqvf<>O7uiHED8Fi(99D zI1aooR<3a%FLs(M?3a(0H5)ioZGNZOrcM5%ZGYC96Vx3s%CvI*bi%oOFBAsLT!J6X z98Z?)g*iD^8rZ|43$o&`Fntg2YEEl(BSmwZ^Obbw$QM5vnu(U+vY3>Ehu>@!`J*a7 z$hU8X>y#K%C@00>KzGl_rs_xWSNx(@^eem$?Fx<^PHMBKaI$d06Uc;=d`3pu<2YnG zDPzpt;B44y?xE)lyVD9Qev&J!uRS6SeY-sRGE(y=kE;w|iFoan!s7?9T|*5KHE zlQDS=yV_qex#83j+vLHUo6Ay<2-EFkZc&6Q3=z6J!$ke6PblHF%JBx z4Sjqd{WlT}Nl1(zB`DU=k7zHYlXZ$1|xi-9qi zl}FB}V3~Ng`!(Pf>dppbjE_|FV68@hIjc&M{bgf9nd8zCsanLfY}4VYZM7CtdvO_g zDbkL?o4Ui>q6)?Ni!ppNOpAn`pqCD8T~CsnCEXdHdui z#g`Bx&(mi3q1X)cD*D+kij@G<MQ`WJp`DNFTQT-cIi$e|qfcqT~2b38+6#qR$*R18JWW6D+yibL&z9+EhOEH6!Py7Hsm-OB& zk%4Agx@yqdD|C@wY=8ojfUL=gNv_9Z-y~33bhn?$iDp^fQa{W%--FygSUacCSDSr3^$y--w&R)M zf1O+(X8f^CR&0E53SNozrW3l?H({kY&d^-b?Fc$043-qHlsH2r$c88&B8()eTIjL+ zoyK?lC<|#aNFJpdTlG-%t0lZ?_Nxz*N?@5Ra`W4sq&&6?39mUqqKjPYp(syrsiG)J76Ayf;TGD(l;Mf_VY}tlciYjm7k9gB9NYuY{KRsX+7(N zJNad8sYVurkN*p47qtWLIZ=#V&QZ$tv1DoqV`LlSVe}YswO?s}bZ?Nh`P#CO1E18lzncvzts_?6p;D8|1u(kkwE#mQaWmQiHO%RG5m=5 z9Ihps-Kc1^Qh* z#gvD?5`Q}|l`VVjhO(d_+vaLU=vSS&)mSSVw0A{M#EET7K=eQx{uEt&R<4^Ibj*6Y zV+8vNJ;Iu^bMyXr*-hu&ofY7*c47dGhMsFQ=eBu)g3nK)Z}=DJ#K%;aV@m*-WXL2d{E+=Y7NDojWM;@XWPEkq_$m8G^0QD?f} z-ZrFdH563|krNw=*Y~>HA|5TrS!y}`P#Mtn?)C3pVzKjKoq0g@Z+!1zAegCVmqN@< z)*y0pqQvKm&jD0WPn4i}lGu{$__`{l1 zGtMw7ZPU|JmL#78-UrZj_rCBAUaSB6qtdELCqvKlvEzwwM*h7vMFCNNSSlF)h4HFu33EJYvPDl@0@Icrh~`6;mQ8&e)|hGELIQW&XWljSOJ3XE=2MP@FT|Uhp6_|?d*&2CAZ@6#sMl;2&os^H}xyEK7foL>ePIDJ?k=mpD0-yB^zjAAV`=mI_q-K~8tkXcXQl>!8m9bK66tw;4#)VdV8fRfpSm`! z@rFw!W76Y+D}4}yPfPLAi0eGv8`1EQu9L7wDT(xniRb<>)fn3l80*Q;!V9?{m|=M# z(ckst?K3($^qbZ55>975v1d}_eedx)?$c=ZI{1BcDVF; z-#b?6k2={}O$0qUUt>@`v=4ul4Xs#@Oh#$9#7DaCU?&N(gm6B_oq{!9MQ*8Lj?vub zd(LS!nEwA}7%4~%7ty_AMif5zuPHU9$B5LKaB&aA^2W6CMAF`s%tU46yDU{+ zJKTHq7Ki7h&k*-wVB{4N&N4u3?d_**oc_Rqb$%)ilS|~A8~H8ZFv|T}a+Z>4sGdbA z<+03w@wx99*4eL@$FXDnt1*p9>3uZf{KnF<7%313zV*|oC_jHVPXXOwh;V-qOtgW? z7DU6ujuSdaHj(6S=sCAI&IZ~|%uE4xveOrSbyz0`(C=EEfH5!^tigc+cbGoS5#i-A zSc?@_Wk#z{_(iai)DB4ndux3G5u=!dn0j_MSz{~ndm}5DpBK^?Xkx$cZU|qMIWe15 z2^f8@s-5`NEriEuQ-^n=z{i95f*L^2;X!|{l;ftzs{aiFALosPtfLT+=R&d^ccft~ zlecLVDrX9VJhrJsxKlhj#p)p5j%B);)SCvjYD^RREdfFWLYFnywu8hvJ7K3KngZx# zte=_gkzJxT?=xsf>KZg*d#6e(I9)mZzJ!k1#i%HO&nQAbUF@}?N%ao=2mj6nypYEq z-K$hk;ZBr_x?p|p6K7*^_V1PT^z@W~Szpe*h&5@lL_RzJqhCD}r7u+N<-+H{S~i(R zW$uPdBVLoocfE!&aNl_Q7BB61osN4B`wCW*6Jy8=SmaXhcwRW8AgxdAI0gBK{Hhef zIj>~ub4WmVQ*&3G1@ZNzCie2iNP5K*#m53ejSSkp1*?eFi&ch#MaiBL z&*{rB_e5C9s88B96KF(qPt=Q5Qfc$|QK8c|h~Bwc=9k!3Unm$fu&5iuUECL5FG32# zxJ$BfO7ND-ve#mxt|+YW2Sr#Z#OQ$w0)~mN7nVD5&YiKv+BwB``}ynu)IP>g%J$;M zR~6mOc3Atf;ORFed(NI-16RJw$LU`$6#R#WZo9?cRwC+0j9T?K?BX&pzShQmWVNIF z!V|~wYCuM%B(*+c!5+sMi?jZTXbf3C)~Tuz&n!&G#kAN(0_B3#<2GL&ne6O^Umbme zEjGG-nkKEy$3*q;=qeKZU2*!o<<*X(!o#|QrUBQQf%G-r%SiOCELKS?FBc@*DlCQ{Pn6O6_hZ>}ET;wWzl!Wnh7gdO=VAA4{UZO7*x_#AH6oDY zK;fplT8RXVHV|DX>sj-N^*+CGQ`~RylKMh1x%A;KERCY%GrUig-dmx&EzVe<8au42 zjMLwws7R^VH~dgUs4dgch)#%EsT;GLdqOwRq^j|`|N7qiW5LNy)NjYnk?{=cCirhK zP;c8Pwr&YWX*@GA#*URzQ*tGtbo#bcPrr3`%4bQh63N|MT^5el8Io5)TI%)-dzNTC z(xU-}I;}UtpXuFlOSzFfQp0Uk?^vbdFW<;aQg+?OiL3LP=_Jj6A%dUTPst2_Rd;u| zbeJsNQPN4Xa*SvA0B)R0CoUd~l~)a*Rx5pL(Va*e=d{|=LmW*Bf2e{WXiK~2Ou5jN|T@Ejcc#(#v%s6GG! zEMAJP?VBPu5oHOSqioq{;{1X?WRz4?eHT9Z?;ZFah_PT-DC(f{ccJqv*Tp*=2gIDw62g zx_8>uiGtaCWeu@BkFPhV3TB?Kanf2vNLwAcY^r2<``*y9pmhzhZ@+JwBn_Q5inD=! z51SE$X_dyqST2T`#>n4VYJF_baoGvua$t?qshiA~ERsX>FDW^Tadr_aHSmxtvX@E6 z8`2cx+Kx&k(kjpmi{u>AW)`+4Gb7)6>JPxUpWdUkb25L>TO(16%$@A*X780H)9ZaF zzAV&P@7pifkhjJAWQk(?t~WhdCsPWhlk+wU{K7t&!Q?u8vDNp14?@5{&c|Bq%3({_ z8{^tUS{@Ezl$7G{&FmJ9)?g{S8 z%3O`_@(}5znXJAu?^iwV@c>m!%wpl{&O7DBo6RQI4|fiZ*&fqBpP}@=M>==T0IKn$ z2X^dZMCwjZ>o3kjey(1uKb?Cl>2`BUZ!Ol+Pqf+T&(>wsq}W=v{4uulX5Oonjobc- z63fWP4LKu3IjUS@N#rU|2aVoW)=qipIp0Hec+1|>&wUw^tB&(Z zh(D~zXw)dZpq?$pa6-+8+405BN+K{vt?<-cu6dBWt2udYp4<&lQvBkpY8|UejHoCY(H`EN8Y4%=< z{n6rvQ>cQ*@py}L5dSNa-*LEIwB@TX8{l+S|JWDQj_+9Xb5=vmbvoFu8CZB}dr&$);9GYi|p%hv-_~lfLvM$m)prA?Dt06tOX6J^ZINqljO^kPMUbjrQk7Z;R5l z(p^+JUkR%_+*Yt~vif-!lcURwr<^++GdLR9$9nO}`#);EO)(i0%ZkNgnyGcf;#8gS^z+H1QXBrG= z=pnY_8mYDRjljl{bEx_u9k)s z$*^8Kk!o|e1)6iHDsU(0Eq~IRjT~v9A*#@5m9Z-Rx@Z07#nNdas>7$I{nAfgl$O;- zxS*7znFn?Dr}mVmzQ6!I9(jLzQ*dIip6uAKN)^AcT>o`aI)@+Lz(IO zOCvoG%OaC~WUz1KtqnIA9upj)GpN1hNNi+XA$~{!#zG4WN9`Y~uRd=ctT-mq2Wm^u z^iztvpVxlc%`xtF%wHM;X)_--a4ebAlW8$(YOdy36Yu`v_w8SP@>By?83{YtYa|z+ z1y*Y5LCXSAYc^p)HFxl|?%g&wpnXG@;QosZ16}9;HD_>cA<8_di~T!jK#Xbs3;=$G xP5h^Y2X@q;GtSM_ypnDHI{~2LPzF5WI8f@yxy2nZBBtg@@1>Qc%HNrO`hV%Nc{u<8 From f17496423cb7ca9be9d9d855fefcf78fbb70439a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 09:44:56 +0100 Subject: [PATCH 009/121] Added : Frame documentation in MarkDown format --- NEWS | 3 +- README.md | 38 +++-- lz4_Frame_format.md | 385 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 416 insertions(+), 10 deletions(-) mode change 100644 => 100755 README.md create mode 100755 lz4_Frame_format.md diff --git a/NEWS b/NEWS index d80b138..b692b8d 100644 --- a/NEWS +++ b/NEWS @@ -1,5 +1,6 @@ r129: -Added : LZ4_compress_fast() +Added : LZ4_compress_fast() +Updated: Documentation converted to MarkDown r128: New : lz4cli sparse file support diff --git a/README.md b/README.md old mode 100644 new mode 100755 index 0f2cd40..69f7397 --- a/README.md +++ b/README.md @@ -1,8 +1,15 @@ LZ4 - Extremely fast compression ================================ -LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems. -A high compression derivative, called LZ4_HC, is also provided. It trades CPU time for compression ratio. +LZ4 is lossless compression algorithm, +providing compression speed at 400 MB/s per core, +scalable with multi-cores CPU. +It also features an extremely fast decoder, +with speed in multiple GB/s per core, +typically reaching RAM speed limits on multi-core systems. + +A high compression derivative, called LZ4_HC, is also provided. +It trades CPU time for compression ratio. |Branch |Status | |------------|---------| @@ -13,16 +20,21 @@ A high compression derivative, called LZ4_HC, is also provided. It trades CPU ti > **Branch Policy:** > - The "master" branch is considered stable, at all times. -> - The "dev" branch is the one where all contributions must be merged before being promoted to master. -> - If you plan to propose a patch, please commit into the "dev" branch. Direct commit to "master" are not permitted. -> - Feature branches can also exist, for dedicated testing of larger modifications before merge into "dev" branch. +> - The "dev" branch is the one where all contributions must be merged + before being promoted to master. +> + If you plan to propose a patch, please commit into the "dev" branch. + Direct commit to "master" are not permitted. +> - Feature branches can also exist, + for dedicated tests of larger modifications before merge into "dev" branch. Benchmarks ------------------------- -The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.3)](http://encode.ru/threads/1371-Filesystem-benchmark?p=33548&viewfull=1#post33548) compiled with GCC v4.8.2 on Linux Mint 64-bits v17. +The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.3)] +compiled with GCC v4.8.2 on Linux Mint 64-bits v17. The reference system uses a Core i5-4300U @1.9GHz. -Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia) in single-thread mode. +Benchmark evaluates the compression of reference [Silesia Corpus] +in single-thread mode. | Compressor | Ratio | Compression | Decompression | | ---------- | ----- | ----------- | ------------- | @@ -40,7 +52,15 @@ Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei |**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | | zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | -The LZ4 block compression format is detailed within [lz4_Block_format](lz4_Block_format.md). +The LZ4 block compression format is detailed within [lz4_Block_format]. -For streaming unknown amount of data and compress files of any size, a frame format has been published, and can be consulted within the file LZ4_Frame_Format.html . +Block format doesn't deal with header information, +nor how to handle arbitrarily long files or data streams. +This is the purpose of the Frame format. +Interoperable versions of LZ4 should use the same frame format, +defined into [lz4_Frame_format]. +[Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029 +[Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia +[lz4_Block_format]: lz4_Block_format.md +[lz4_Frame_format]: lz4_Frame_format.md \ No newline at end of file diff --git a/lz4_Frame_format.md b/lz4_Frame_format.md new file mode 100755 index 0000000..55edaa0 --- /dev/null +++ b/lz4_Frame_format.md @@ -0,0 +1,385 @@ +LZ4 Frame Format Description +============================ + +###Notices + +Copyright (c) 2013-2015 Yann Collet + +Permission is granted to copy and distribute this document +for any purpose and without charge, +including translations into other languages +and incorporation into compilations, +provided that the copyright notice and this notice are preserved, +and that any substantive changes or deletions from the original +are clearly marked. +Distribution of this document is unlimited. + +###Version + +1.5.1 (31/03/2015) + + +Introduction +------------ + +The purpose of this document is to define a lossless compressed data format, +that is independent of CPU type, operating system, +file system and character set, suitable for +File compression, Pipe and streaming compression +using the [LZ4 algorithm](http://www.lz4.info). + +The data can be produced or consumed, +even for an arbitrarily long sequentially presented input data stream, +using only an a priori bounded amount of intermediate storage, +and hence can be used in data communications. +The format uses the LZ4 compression method, +and optional [xxHash-32 checksum method](https://github.com/Cyan4973/xxHash), +for detection of data corruption. + +The data format defined by this specification +does not attempt to allow random access to compressed data. + +This specification is intended for use by implementers of software +to compress data into LZ4 format and/or decompress data from LZ4 format. +The text of the specification assumes a basic background in programming +at the level of bits and other primitive data representations. + +Unless otherwise indicated below, +a compliant compressor must produce data sets +that conform to the specifications presented here. +It doesn’t need to support all options though. + +A compliant decompressor must be able to decompress +at least one working set of parameters +that conforms to the specifications presented here. +It may also ignore checksums. +Whenever it does not support a specific parameter within the compressed stream, +it must produce a non-ambiguous error code +and associated error message explaining which parameter is unsupported. + + +General Structure of LZ4 Frame format +------------------------------------- + +| MagicNb | F. Descriptor | Block | (...) | EndMark | C. Checksum | +|:-------:|:-------------:| ----- | ----- | ------- | ----------- | +| 4 bytes | 3-11 bytes | | | 4 bytes | 4 bytes | + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184D2204 + +__Frame Descriptor__ + +3 to 11 Bytes, to be detailed in the next part. +Most important part of the spec. + +__Data Blocks__ + +To be detailed later on. +That’s where compressed data is stored. + +__EndMark__ + +The flow of blocks ends when the last data block has a size of “0”. +The size is expressed as a 32-bits value. + +__Content Checksum__ + +Content Checksum verify that the full content has been decoded correctly. +The content checksum is the result +of [xxh32() hash function](https://github.com/Cyan4973/xxHash) +digesting the original (decoded) data as input, and a seed of zero. +Content checksum is only present when its associated flag +is set in the frame descriptor. +Content Checksum validates the result, +that all blocks were fully transmitted in the correct order and without error, +and also that the encoding/decoding process itself generated no distortion. +Its usage is recommended. + +__Frame Concatenation__ + +In some circumstances, it may be preferable to append multiple frames, +for example in order to add new data to an existing compressed file +without re-framing it. + +In such case, each frame has its own set of descriptor flags. +Each frame is considered independent. +The only relation between frames is their sequential order. + +The ability to decode multiple concatenated frames +within a single stream or file +is left outside of this specification. +As an example, the reference lz4 command line utility behavior is +to decode all concatenated frames in their sequential order. + + +Frame Descriptor +---------------- + +| FLG | BD | (Content Size) | HC | +| ------- | ------- |:--------------:| ------- | +| 1 byte | 1 byte | 0 - 8 bytes | 1 byte | + +The descriptor uses a minimum of 3 bytes, +and up to 11 bytes depending on optional parameters. + +__FLG byte__ + +| BitNb | 7-6 | 5 | 4 | 3 | 2 | 1-0 | +| ------- | ------- | ------- | --------- | ------- | --------- | -------- | +|FieldName| Version | B.Indep | B.Checksum| C.Size | C.Checksum|*Reserved*| + + +__BD byte__ + +| BitNb | 7 | 6-5-4 | 3-2-1-0 | +| ------- | -------- | ------------ | -------- | +|FieldName|*Reserved*| Block MaxSize|*Reserved*| + +In the tables, bit 7 is highest bit, while bit 0 is lowest. + +__Version Number__ + +2-bits field, must be set to “01”. +Any other value cannot be decoded by this version of the specification. +Other version numbers will use different flag layouts. + +__Block Independence flag__ + +If this flag is set to “1”, blocks are independent. +If this flag is set to “0”, each block depends on previous ones +(up to LZ4 window size, which is 64 KB). +In such case, it’s necessary to decode all blocks in sequence. + +Block dependency improves compression ratio, especially for small blocks. +On the other hand, it makes direct jumps or multi-threaded decoding impossible. + +__Block checksum flag__ + +If this flag is set, each data block will be followed by a 4-bytes checksum, +calculated by using the xxHash-32 algorithm on the raw (compressed) data block. +The intention is to detect data corruption (storage or transmission errors) +immediately, before decoding. +Block checksum usage is optional. + +__Content Size flag__ + +If this flag is set, the uncompressed size of data included within the frame +will be present as an 8 bytes unsigned little endian value, after the flags. +Content Size usage is optional. + +__Content checksum flag__ + +If this flag is set, a content checksum will be appended after the EndMark. + +Recommended value : “1” (content checksum is present) + +__Block Maximum Size__ + +This information is intended to help the decoder allocate memory. +Size here refers to the original (uncompressed) data size. +Block Maximum Size is one value among the following table : + +| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | +| --- | --- | --- | --- | ----- | ------ | ---- | ---- | +| N/A | N/A | N/A | N/A | 64 KB | 256 KB | 1 MB | 4 MB | + +The decoder may refuse to allocate block sizes above a (system-specific) size. +Unused values may be used in a future revision of the spec. +A decoder conformant to the current version of the spec +is only able to decode blocksizes defined in this spec. + +__Reserved bits__ + +Value of reserved bits **must** be 0 (zero). +Reserved bit might be used in a future version of the specification, +typically enabling new optional features. +If this happens, a decoder respecting the current version of the specification +shall not be able to decode such a frame. + +__Content Size__ + +This is the original (uncompressed) size. +This information is optional, and only present if the associated flag is set. +Content size is provided using unsigned 8 Bytes, for a maximum of 16 HexaBytes. +Format is Little endian. +This value is informational. +It can be safely skipped by a conformant decoder. + +__Header Checksum__ + +One-byte checksum of combined descriptor fields, including optional ones. +The value is the second byte of xxh32() : ` (xxh32()>>8) & 0xFF } ` +using zero as a seed, +and the full Frame Descriptor as an input +(including optional fields when they are present). +A wrong checksum indicates an error in the descriptor. +Header checksum is informational and can be skipped. + + +Data Blocks +----------- + +| Block Size | data | (Block Checksum) | +|:----------:| ------ |:----------------:| +| 4 bytes | | 0 - 4 bytes | + + +__Block Size__ + +This field uses 4-bytes, format is little-endian. + +The highest bit is “1” if data in the block is uncompressed. + +The highest bit is “0” if data in the block is compressed by LZ4. + +All other bits give the size, in bytes, of the following data block +(the size does not include the block checksum if present). + +Block Size shall never be larger than Block Maximum Size. +Such a thing could happen for incompressible source data. +In such case, such a data block shall be passed in uncompressed format. + +__Data__ + +Where the actual data to decode stands. +It might be compressed or not, depending on previous field indications. +Uncompressed size of Data can be any size, up to “block maximum size”. +Note that data block is not necessarily full : +an arbitrary “flush” may happen anytime. Any block can be “partially filled”. + +__Block checksum__ + +Only present if the associated flag is set. +This is a 4-bytes checksum value, in little endian format, +calculated by using the xxHash-32 algorithm on the raw (undecoded) data block, +and a seed of zero. +The intention is to detect data corruption (storage or transmission errors) +before decoding. + +Block checksum is cumulative with Content checksum. + + +Skippable Frames +---------------- + +| Magic Number | Frame Size | User Data | +|:------------:|:----------:| --------- | +| 4 bytes | 4 bytes | | + +Skippable frames allow the integration of user-defined data +into a flow of concatenated frames. +Its design is pretty straightforward, +with the sole objective to allow the decoder to quickly skip +over user-defined data and continue decoding. + +For the purpose of facilitating identification, +it is discouraged to start a flow of concatenated frames with a skippable frame. +If there is a need to start such a flow with some user data +encapsulated into a skippable frame, +it’s recommended to start with a zero-byte LZ4 frame +followed by a skippable frame. +This will make it easier for file type identifiers. + + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F. +All 16 values are valid to identify a skippable frame. + +__Frame Size__ + +This is the size, in bytes, of the following User Data +(without including the magic number nor the size field itself). +4 Bytes, Little endian format, unsigned 32-bits. +This means User Data can’t be bigger than (2^32-1) Bytes. + +__User Data__ + +User Data can be anything. Data will just be skipped by the decoder. + + +Legacy frame +------------ + +The Legacy frame format was defined into the initial versions of “LZ4Demo”. +Newer compressors should not use this format anymore, as it is too restrictive. + +Main characteristics of the legacy format : + +- Fixed block size : 8 MB. +- All blocks must be completely filled, except the last one. +- All blocks are always compressed, even when compression is detrimental. +- The last block is detected either because + it is followed by the “EOF” (End of File) mark, + or because it is followed by a known Frame Magic Number. +- No checksum +- Convention is Little endian + +| MagicNb | B.CSize | CData | B.CSize | CData | (...) | EndMark | +| ------- | ------- | ----- | ------- | ----- | ------- | ------- | +| 4 bytes | 4 bytes | CSize | 4 bytes | CSize | x times | EOF | + + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184C2102 + +__Block Compressed Size__ + +This is the size, in bytes, of the following compressed data block. +4 Bytes, Little endian format. + +__Data__ + +Where the actual compressed data stands. +Data is always compressed, even when compression is detrimental. + +__EndMark__ + +End of compressed stream is implicit. +It needs to be followed by a standard EOF (End Of File) signal, +wether it is a file or a stream. + +Alternatively, if the frame is followed by a valid Frame Magic Number, +it is considered completed. +It makes legacy frames compatible with frame concatenation. + +Any other value will be interpreted as a block size, +and trigger an error if it does not fit within acceptable range. + + +Version changes +--------------- + +1.5.1 : changed format to MarkDown compatible + +1.5 : removed Dictionary ID from specification + +1.4.1 : changed wording from “stream” to “frame” + +1.4 : added skippable streams, re-added stream checksum + +1.3 : modified header checksum + +1.2 : reduced choice of “block size”, to postpone decision on “dynamic size of BlockSize Field”. + +1.1 : optional fields are now part of the descriptor + +1.0 : changed “block size” specification, adding a compressed/uncompressed flag + +0.9 : reduced scale of “block maximum size” table + +0.8 : removed : high compression flag + +0.7 : removed : stream checksum + +0.6 : settled : stream size uses 8 bytes, endian convention is little endian + +0.5: added copyright notice + +0.4 : changed format to Google Doc compatible OpenDocument \ No newline at end of file From 7c26b038547e3a488a6cd5fe13df260b6e4d6e08 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 10:22:18 +0100 Subject: [PATCH 010/121] Updated make dist --- Makefile | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 59c5e3b..59759c0 100644 --- a/Makefile +++ b/Makefile @@ -31,7 +31,7 @@ # ################################################################ # Version number -export VERSION=128 +export VERSION=129 export RELEASE=r$(VERSION) DESTDIR?= @@ -47,7 +47,7 @@ TEXT = $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4.h $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4hc.h \ $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4frame.h $(LZ4DIR)/lz4frame_static.h \ $(LZ4DIR)/xxhash.c $(LZ4DIR)/xxhash.h \ $(LZ4DIR)/liblz4.pc.in $(LZ4DIR)/Makefile $(LZ4DIR)/LICENSE \ - Makefile lz4_block_format.txt LZ4_Frame_Format.html NEWS README.md \ + Makefile lz4_Block_format.md lz4_Frame_format.md NEWS README.md \ cmake_unofficial/CMakeLists.txt \ $(PRGDIR)/fullbench.c $(PRGDIR)/lz4cli.c \ $(PRGDIR)/datagen.c $(PRGDIR)/datagen.h $(PRGDIR)/datagencli.c $(PRGDIR)/fuzzer.c \ @@ -55,9 +55,7 @@ TEXT = $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4.h $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4hc.h \ $(PRGDIR)/bench.c $(PRGDIR)/bench.h \ $(PRGDIR)/lz4.1 \ $(PRGDIR)/Makefile $(PRGDIR)/COPYING -NONTEXT = images/image00.png images/image01.png images/image02.png \ - images/image03.png images/image04.png images/image05.png \ - images/image06.png +NONTEXT = SOURCES = $(TEXT) $(NONTEXT) @@ -112,7 +110,6 @@ dist: clean @install -dD -m 700 lz4-$(RELEASE)/lib/ @install -dD -m 700 lz4-$(RELEASE)/programs/ @install -dD -m 700 lz4-$(RELEASE)/cmake_unofficial/ - @install -dD -m 700 lz4-$(RELEASE)/images/ @for f in $(TEXT); do \ tr -d '\r' < $$f > .tmp; \ install -m 600 .tmp lz4-$(RELEASE)/$$f; \ From 3bba55c741293dd8162e5afb9eace5e03fcff023 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 14:47:23 +0100 Subject: [PATCH 011/121] Fixed : Windows compilation Added : Appveyor badge --- README.md | 4 +++- lib/lz4frame.c | 6 ++++-- lz4_Frame_format.md | 0 programs/frametest.c | 2 +- programs/lz4io.c | 4 ++++ 5 files changed, 12 insertions(+), 4 deletions(-) mode change 100755 => 100644 README.md mode change 100755 => 100644 lz4_Frame_format.md diff --git a/README.md b/README.md old mode 100755 new mode 100644 index 69f7397..453bb01 --- a/README.md +++ b/README.md @@ -15,6 +15,8 @@ It trades CPU time for compression ratio. |------------|---------| |master | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=master)](https://travis-ci.org/Cyan4973/lz4) | |dev | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=dev)](https://travis-ci.org/Cyan4973/lz4) | +|visual | [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) | + > **Branch Policy:** @@ -63,4 +65,4 @@ defined into [lz4_Frame_format]. [Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029 [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [lz4_Block_format]: lz4_Block_format.md -[lz4_Frame_format]: lz4_Frame_format.md \ No newline at end of file +[lz4_Frame_format]: lz4_Frame_format.md diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 5f69c95..474b196 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -1088,8 +1088,10 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, doAnotherStage = 0; /* not enough src data, ask for some more */ break; } - LZ4F_errorCode_t errorCode = LZ4F_decodeHeader(dctxPtr, dctxPtr->header, dctxPtr->tmpInTarget); - if (LZ4F_isError(errorCode)) return errorCode; + { + LZ4F_errorCode_t errorCode = LZ4F_decodeHeader(dctxPtr, dctxPtr->header, dctxPtr->tmpInTarget); + if (LZ4F_isError(errorCode)) return errorCode; + } break; } diff --git a/lz4_Frame_format.md b/lz4_Frame_format.md old mode 100755 new mode 100644 diff --git a/programs/frametest.c b/programs/frametest.c index 96d5acb..4beee23 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -607,7 +607,7 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi /* create a skippable frame (rare case) */ BYTE* op = (BYTE*)compressedBuffer; FUZ_writeLE32(op, LZ4F_MAGIC_SKIPPABLE_START + (FUZ_rand(&randState) & 15)); - FUZ_writeLE32(op+4, srcSize); + FUZ_writeLE32(op+4, (U32)srcSize); cSize = srcSize+8; } else if ((FUZ_rand(&randState) & 0xF) == 2) diff --git a/programs/lz4io.c b/programs/lz4io.c index 34d24bf..02e03c8 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -75,6 +75,10 @@ # define SET_SPARSE_FILE_MODE(file) #endif +#if !defined(S_ISREG) +# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) +#endif + /***************************** * Constants From 7db667820fbaf8caec7f33806f5895416194f1a7 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 16:14:54 +0100 Subject: [PATCH 012/121] Restored proper credit --- NEWS | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/NEWS b/NEWS index b692b8d..233dc82 100644 --- a/NEWS +++ b/NEWS @@ -3,21 +3,21 @@ Added : LZ4_compress_fast() Updated: Documentation converted to MarkDown r128: -New : lz4cli sparse file support -New : command -m, to compress multiple files in a single command -Fixed : Restored lz4hc compression ratio (was slightly lower since r124) -New : lz4 cli supports long commands -New : lz4frame & lz4cli frame content size support -New : lz4frame supports skippable frames -Changed:Default "make install" directory is /usr/local -New : lz4 cli supports "pass-through" mode -New : datagen can generate sparse files -New : scan-build tests -New : g++ compatibility tests -New : arm cross-compilation test -Fixed : Fuzzer + frametest compatibility with NetBSD (issue #48) -Added : Visual project directory -Updated:Man page & Specification +New : lz4cli sparse file support (Requested by Neil Wilson, and contributed by Takayuki Matsuoka) +New : command -m, to compress multiple files in a single command (suggested by Kyle J. Harper) +Fixed : Restored lz4hc compression ratio (slightly lower since r124) +New : lz4 cli supports long commands (suggested by Takayuki Matsuoka) +New : lz4frame & lz4cli frame content size support +New : lz4frame supports skippable frames, as requested by Sergey Cherepanov +Changed: Default "make install" directory is /usr/local, as notified by Ron Johnson +New : lz4 cli supports "pass-through" mode, requested by Neil Wilson +New : datagen can generate sparse files +New : scan-build tests, thanks to kind help by Takayuki Matsuoka +New : g++ compatibility tests +New : arm cross-compilation test, thanks to kind help by Takayuki Matsuoka +Fixed : Fuzzer + frametest compatibility with NetBSD (issue #48, reported by Thomas Klausner) +Added : Visual project directory +Updated: Man page & Specification r127: N/A : added a file on SVN From bf146ec2476f5a6d4128dfee665e9e0d2db00f4d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 18:38:16 +0100 Subject: [PATCH 013/121] Removed .suo & .user files from Visual solutions --- visual/2012/fuzzer/fuzzer.vcxproj.user | 4 ---- visual/2012/lz4.v11.suo | Bin 20992 -> 0 bytes visual/2012/lz4/lz4.vcxproj.user | 4 ---- 3 files changed, 8 deletions(-) delete mode 100644 visual/2012/fuzzer/fuzzer.vcxproj.user delete mode 100644 visual/2012/lz4.v11.suo delete mode 100644 visual/2012/lz4/lz4.vcxproj.user diff --git a/visual/2012/fuzzer/fuzzer.vcxproj.user b/visual/2012/fuzzer/fuzzer.vcxproj.user deleted file mode 100644 index 7cbb321..0000000 --- a/visual/2012/fuzzer/fuzzer.vcxproj.user +++ /dev/null @@ -1,4 +0,0 @@ - - - - \ No newline at end of file diff --git a/visual/2012/lz4.v11.suo b/visual/2012/lz4.v11.suo deleted file mode 100644 index 832198456bc219222357ef3b84addb3c26064b3e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 20992 zcmeHOU2Ggz6~3D^X`G+7At97d(j{?{CU!Pf5-*5`CnTg4BzTB~l&S(GUZ5|)LzD1*cjjhi z$1}6DyOWrpz1G?Jxp(e4_uPBVJ@=gB_it>y_5F8uejr@slvpe7+-(u-9q(g~d+~%2 z+wshIckbT3%PQXEwpkt`4orvw{!9@ULn4oBQ7m&RLzf+5-Mu;AQ01%mhj0BWefa&@ zQ)u_=ZC2xPU@VBs;w-RdaW^EEs`OG>y+f?Iho5Y(to45S0$yYJ%90XkVTdbYMC5Qa z#d-W{ph>+q>~?;aRH*V&qrFC}_z8<|G9PF)r)!c$yW% zpeZfpabHPMH}Dhh<2B$PLEFs$mYsmx>>#2-UGOCO{0!ie06y=)_0s_I=(B)bfTsa* zz~=zF0eb*@0VHF;edV{!a(_7xrTn*G-h#>#3%~gj<)3TsIc^>(t0Dj6z`L6AKL-3! z%Ktj_$9H*3eY1;%~zkJ?^EBk*GumwPUM}0#cZUuY-Ky}^< zQ2o_=-Dmf4z|mJz57f)e>fe#K3LXK^)mq{^t^-~<|6C7TGtT*Eotl5%Q>3Z$sqd-l zxerk9bC2PEz);@u4L}?U4%^S2xE=v?0geKW0aTk4xbJa1 z_u{?}a0+l5@Eo8YFaSsb1_48WVF3F$gX>wq=K<#cBLLO6^NOYqZBhDH6B$@PDOedL zX$L8LtO%PUjhY-Rlsv}oE*-ExxNg=0$OL!2*E4yi-cNz^@q4>~MTM#hbA|mEi=iRc zS6KX{^^bPvV;HX~bZKeP5}Ol~c;<*TtcM)<5nO2h5w9x2t*zQMxKN+641li%*q`Kb z5!W(ytG?RrQ+3$C0T1SEt8*03^Pq_50HxmxFQxn1!u&>dUR4!+3(Rh#(xN1P`9zD_0Mz10@eXXPWwOXJaGQ0 z_o&;#j-Pw&w>b$Y#LKl-l$O5^-ID_qwDJ{YmP{l;8F``E9I?I@;@LXJ@07nJu<-Cmq%c-qF@CAjspG?r---V`#zU^BamL{}r*hPOtFBDL?gqzy2BV&pxbv8E z9n=hi8eP6Af-BWZqxc6f@(fn{99BR9a>%pbh|P~d)KM1&m3?)LpLX1r?M~G>SDk++ z#U)@b0&hlY#%dhk{Bzz_38K|mr{r%6JWc^4c|L>RCCmb)VG;9^!Ckci{>QeVQ7!m+ z{^p()RZ+NkjjGRD`e_7z3VdFG)RE@~?&$-ejv?=${w%w| zxTl+)RP+MB^1Xx||4HEgJ&gbq_x^W5%6tY===Stj?cKj|;HQmEn>#H2XVI2>7KmY5pFjR(*kRO&I$Haedi1|j{sw`8nx39T+80ha z97i2>g0lX-W`g!V$E}J#rRAD-+|piT3Bs>@u}b=_FDic%z|V6jeG#Ogvex^_IeyNw zE_XWXgkxDhS&Vv4&VSI^hrPr8m3V6zKliH*_5olFb9@<|U+({I%WF#T1RGqq|8YO( zxjXFm$*Y&`Zohbgl2d#tc2mU%mLjDHzZi_)slfXKDPx+q#yZ~#u3mCiL z5$M6mJ}C}KI*y6sxH}5{J%r~&;Oa2eT|e%QVr;8w;k~WV$ zhG#vVeJt6ZhUNVw&_9h?n?&1b)Nluwme!~N>}lwVY0L@N_7U7UhgtcQz&9DJ8{%Mm zi*#17;k-S7Gn5qPpS5QBCiu?z4ewd3Q!B0alC%BAY-u&^6Z9ly5_=tGyw>$NZTlS2-cK7T5xpDn%y2QYlzir?9hj;eudHxrF`QWAf zGxKz|p-hSs+3nZwT;DbF_FJ<*``)g9^wjpfz~oVrkHFJjtGN^fVhqn&`n&_6++fhLq@Oh6KNlI5xK67G0@ zc5!*;C0KLah)mH8lAqu!S|?dhOxu?}3(HoB%U!DkZAc3v4y(bkiIoL1PlVEb zl{bzb+O+Nx_MJr;jV$i}!IUW+?E7_qW!{A!e>g1uG0vpR$;ttLKmFVSJ6 zW$`S;YyU%(k*g-?ahBnnrenu@M68d+`qk1h6H_^(l-4KB(rm7fNNI+7A(x*mTulru znOfe+&E)i)mmiY0+6HT~*L{pbT~Lo)f7bvmvmEp8wR)qsfGsXilKh8zHg9UhY&xSE z?eQtCXygj{zRu1>2mh4(Sp8VPwaBj&KpuQmN1Mqwt?C@Cq@+EbEa_&csP*Nwl37ga z?eXzoN;Y$%7OJ931dYFTtd;u9pXQMSLwgc#gh&ujH910Ghf8HO) zm1iDj{9m=9e~&&!^-O;!cZV?Xe||HJw)7F70jT)D>dSdW(}%uTn*K6&+4|%EoPVwv zG6R>Yg*;S^RxAGhXPba#{J-J&f6e|+`hQb4|I4v1)p8<_1^FMfIB5oGkRxQj)Ud{XuMwNM1V+o4C->>-p zLEz=sfoRnD}1}N9=j>yI>nx)%gEj5XQX^boj;pMQ{ee19Mec$qCbN zD>F&KD?o4OFWi^VJb2_4|6l$1dt>qcX8hehho~8Ucjf?`h6F^Zp%W`KbodBkGyeVp zJm*nmv8|kmX8gUJAyezBjB;Mer^^@Z%-%5BDM9gf?)Bc5(Mj0y{W6N%ft)*LHzx3_ zyvu(0c@C(Izn_%3++pMY_Ay$<|HDP?D&uEBA9?(}miQy_voz!H?#u|k%m8=%U1fK= zGc&?vCMZAP%FK*#___X8KmJ~S{9nnBT!i*dEg7)Q_`6?{N4WUA9hdLVn7Kvm)Rl`D z2hy1<2-M4rlHI+C+%s>aKZ^{Cfl^MNbvJpsHDoLf?DHqKcfMFk7Ut*E`Pr1VWRBzv(>>I> z@^8##WaDU=l8K}g#lu?iZ&qgXc=bprdvr0am$Zu)#Rm4TN(-HbTlvGcTh?we%t=kx zGN!g-sy(N&QTc1~-^G8(P%ZP{DF4fuP<3Vk^;FM4a%dtf@ns#d!!#fcvu(BUBMqrv z*Qw(SMO#hl-=WGPUCQfYG6PR;k1gcXR{qd3-m;N4a8!jkRbG2=*#7^SWc026@=)^M zv6XS;X=^L1YyO@5FK@iaX9_w0jq*SF#&JAe1{kzalE*RRw%la1p4F9S#r AMgRZ+ diff --git a/visual/2012/lz4/lz4.vcxproj.user b/visual/2012/lz4/lz4.vcxproj.user deleted file mode 100644 index 7cbb321..0000000 --- a/visual/2012/lz4/lz4.vcxproj.user +++ /dev/null @@ -1,4 +0,0 @@ - - - - \ No newline at end of file From a761546b1b8499bd52ed84876df03cb00e81c02e Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 18:52:52 +0100 Subject: [PATCH 014/121] Fix : minor warning under Visual --- lib/lz4.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4.c b/lib/lz4.c index cd6147f..3dc44f2 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -731,7 +731,7 @@ _last_literals: } -int LZ4_compress(const char* source, char* dest, const int inputSize) +int LZ4_compress(const char* source, char* dest, int inputSize) { #if (HEAPMODE) void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ From 08b24af5a014b1410f81c363f799fe5bff4cc695 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 19:12:59 +0100 Subject: [PATCH 015/121] Updated Visual 2012 solution : + 3 projects (fullbench, frametest, datagen) --- visual/2012/datagen/datagen.vcxproj | 159 +++++++++++++++++ visual/2012/datagen/datagen.vcxproj.filters | 30 ++++ visual/2012/frametest/frametest.vcxproj | 166 ++++++++++++++++++ .../2012/frametest/frametest.vcxproj.filters | 51 ++++++ visual/2012/fullbench/fullbench.vcxproj | 166 ++++++++++++++++++ .../2012/fullbench/fullbench.vcxproj.filters | 51 ++++++ visual/2012/fuzzer/fuzzer.vcxproj | 2 + visual/2012/fuzzer/fuzzer.vcxproj.filters | 0 visual/2012/lz4.sln | 40 +++-- visual/2012/lz4/lz4.vcxproj | 2 + visual/2012/lz4/lz4.vcxproj.filters | 0 11 files changed, 657 insertions(+), 10 deletions(-) create mode 100755 visual/2012/datagen/datagen.vcxproj create mode 100755 visual/2012/datagen/datagen.vcxproj.filters create mode 100755 visual/2012/frametest/frametest.vcxproj create mode 100755 visual/2012/frametest/frametest.vcxproj.filters create mode 100755 visual/2012/fullbench/fullbench.vcxproj create mode 100755 visual/2012/fullbench/fullbench.vcxproj.filters mode change 100644 => 100755 visual/2012/fuzzer/fuzzer.vcxproj mode change 100644 => 100755 visual/2012/fuzzer/fuzzer.vcxproj.filters mode change 100644 => 100755 visual/2012/lz4.sln mode change 100644 => 100755 visual/2012/lz4/lz4.vcxproj mode change 100644 => 100755 visual/2012/lz4/lz4.vcxproj.filters diff --git a/visual/2012/datagen/datagen.vcxproj b/visual/2012/datagen/datagen.vcxproj new file mode 100755 index 0000000..ea61533 --- /dev/null +++ b/visual/2012/datagen/datagen.vcxproj @@ -0,0 +1,159 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {D745AE2F-596A-403A-9B91-81A8C6779243} + Win32Proj + datagen + + + + Application + true + v110 + Unicode + + + Application + true + v110 + Unicode + + + Application + false + v110 + true + Unicode + + + Application + false + v110 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2012/datagen/datagen.vcxproj.filters b/visual/2012/datagen/datagen.vcxproj.filters new file mode 100755 index 0000000..822d9bc --- /dev/null +++ b/visual/2012/datagen/datagen.vcxproj.filters @@ -0,0 +1,30 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2012/frametest/frametest.vcxproj b/visual/2012/frametest/frametest.vcxproj new file mode 100755 index 0000000..b86f7de --- /dev/null +++ b/visual/2012/frametest/frametest.vcxproj @@ -0,0 +1,166 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7} + Win32Proj + frametest + + + + Application + true + v110 + Unicode + + + Application + true + v110 + Unicode + + + Application + false + v110 + true + Unicode + + + Application + false + v110 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2012/frametest/frametest.vcxproj.filters b/visual/2012/frametest/frametest.vcxproj.filters new file mode 100755 index 0000000..19c3d15 --- /dev/null +++ b/visual/2012/frametest/frametest.vcxproj.filters @@ -0,0 +1,51 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2012/fullbench/fullbench.vcxproj b/visual/2012/fullbench/fullbench.vcxproj new file mode 100755 index 0000000..11f8f45 --- /dev/null +++ b/visual/2012/fullbench/fullbench.vcxproj @@ -0,0 +1,166 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E} + Win32Proj + fullbench + + + + Application + true + v110 + Unicode + + + Application + true + v110 + Unicode + + + Application + false + v110 + true + Unicode + + + Application + false + v110 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2012/fullbench/fullbench.vcxproj.filters b/visual/2012/fullbench/fullbench.vcxproj.filters new file mode 100755 index 0000000..cdcf599 --- /dev/null +++ b/visual/2012/fullbench/fullbench.vcxproj.filters @@ -0,0 +1,51 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2012/fuzzer/fuzzer.vcxproj b/visual/2012/fuzzer/fuzzer.vcxproj old mode 100644 new mode 100755 index 9ba4fab..6287947 --- a/visual/2012/fuzzer/fuzzer.vcxproj +++ b/visual/2012/fuzzer/fuzzer.vcxproj @@ -117,6 +117,7 @@ true true WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console @@ -134,6 +135,7 @@ true true WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console diff --git a/visual/2012/fuzzer/fuzzer.vcxproj.filters b/visual/2012/fuzzer/fuzzer.vcxproj.filters old mode 100644 new mode 100755 diff --git a/visual/2012/lz4.sln b/visual/2012/lz4.sln old mode 100644 new mode 100755 index 2d07075..80d1a54 --- a/visual/2012/lz4.sln +++ b/visual/2012/lz4.sln @@ -5,40 +5,60 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lz4", "lz4\lz4.vcxproj", "{ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fuzzer", "fuzzer\fuzzer.vcxproj", "{18B9F1A7-9C66-4352-898B-30804DADE0FD}" EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench", "fullbench\fullbench.vcxproj", "{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "frametest", "frametest\frametest.vcxproj", "{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "datagen", "datagen\datagen.vcxproj", "{D745AE2F-596A-403A-9B91-81A8C6779243}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|Mixed Platforms = Debug|Mixed Platforms Debug|Win32 = Debug|Win32 Debug|x64 = Debug|x64 - Release|Mixed Platforms = Release|Mixed Platforms Release|Win32 = Release|Win32 Release|x64 = Release|x64 EndGlobalSection GlobalSection(ProjectConfigurationPlatforms) = postSolution - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Mixed Platforms.ActiveCfg = Debug|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Mixed Platforms.Build.0 = Debug|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.ActiveCfg = Debug|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.Build.0 = Debug|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.ActiveCfg = Debug|x64 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.Build.0 = Debug|x64 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Mixed Platforms.ActiveCfg = Release|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Mixed Platforms.Build.0 = Release|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.ActiveCfg = Release|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.Build.0 = Release|Win32 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.ActiveCfg = Release|x64 {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.Build.0 = Release|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Mixed Platforms.ActiveCfg = Debug|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Mixed Platforms.Build.0 = Debug|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.ActiveCfg = Debug|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.Build.0 = Debug|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.ActiveCfg = Debug|x64 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.Build.0 = Debug|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Mixed Platforms.ActiveCfg = Release|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Mixed Platforms.Build.0 = Release|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.ActiveCfg = Release|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.Build.0 = Release|Win32 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.ActiveCfg = Release|x64 {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.Build.0 = Release|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.ActiveCfg = Debug|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.Build.0 = Debug|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.ActiveCfg = Debug|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.Build.0 = Debug|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.ActiveCfg = Release|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.Build.0 = Release|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.ActiveCfg = Release|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.Build.0 = Release|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.ActiveCfg = Debug|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.Build.0 = Debug|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.ActiveCfg = Debug|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.Build.0 = Debug|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.ActiveCfg = Release|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.Build.0 = Release|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.ActiveCfg = Release|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.Build.0 = Release|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.ActiveCfg = Debug|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.Build.0 = Debug|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.ActiveCfg = Debug|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.Build.0 = Debug|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.ActiveCfg = Release|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.Build.0 = Release|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.ActiveCfg = Release|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/visual/2012/lz4/lz4.vcxproj b/visual/2012/lz4/lz4.vcxproj old mode 100644 new mode 100755 index 5540d5d..b7e0061 --- a/visual/2012/lz4/lz4.vcxproj +++ b/visual/2012/lz4/lz4.vcxproj @@ -117,6 +117,7 @@ true true WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console @@ -134,6 +135,7 @@ true true WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console diff --git a/visual/2012/lz4/lz4.vcxproj.filters b/visual/2012/lz4/lz4.vcxproj.filters old mode 100644 new mode 100755 From 117ab8c84196474bda1fd819941bc41ce9a574ee Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 20:08:52 +0100 Subject: [PATCH 016/121] Added : Visual 2013 solution --- visual/2012/fuzzer/fuzzer.vcxproj | 2 + visual/2012/lz4/lz4.vcxproj | 2 + visual/2013/datagen/datagen.vcxproj | 159 ++++++++++++++++ visual/2013/datagen/datagen.vcxproj.filters | 30 ++++ visual/2013/frametest/frametest.vcxproj | 166 +++++++++++++++++ .../2013/frametest/frametest.vcxproj.filters | 51 ++++++ visual/2013/fullbench/fullbench.vcxproj | 166 +++++++++++++++++ .../2013/fullbench/fullbench.vcxproj.filters | 51 ++++++ visual/2013/fuzzer/fuzzer.vcxproj | 163 +++++++++++++++++ visual/2013/fuzzer/fuzzer.vcxproj.filters | 42 +++++ visual/2013/lz4.sln | 66 +++++++ visual/2013/lz4/lz4.vcxproj | 170 ++++++++++++++++++ visual/2013/lz4/lz4.vcxproj.filters | 63 +++++++ 13 files changed, 1131 insertions(+) create mode 100755 visual/2013/datagen/datagen.vcxproj create mode 100755 visual/2013/datagen/datagen.vcxproj.filters create mode 100755 visual/2013/frametest/frametest.vcxproj create mode 100755 visual/2013/frametest/frametest.vcxproj.filters create mode 100755 visual/2013/fullbench/fullbench.vcxproj create mode 100755 visual/2013/fullbench/fullbench.vcxproj.filters create mode 100755 visual/2013/fuzzer/fuzzer.vcxproj create mode 100755 visual/2013/fuzzer/fuzzer.vcxproj.filters create mode 100755 visual/2013/lz4.sln create mode 100755 visual/2013/lz4/lz4.vcxproj create mode 100755 visual/2013/lz4/lz4.vcxproj.filters diff --git a/visual/2012/fuzzer/fuzzer.vcxproj b/visual/2012/fuzzer/fuzzer.vcxproj index 6287947..4da822f 100755 --- a/visual/2012/fuzzer/fuzzer.vcxproj +++ b/visual/2012/fuzzer/fuzzer.vcxproj @@ -89,6 +89,7 @@ Level4 Disabled WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console @@ -102,6 +103,7 @@ Level4 Disabled WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console diff --git a/visual/2012/lz4/lz4.vcxproj b/visual/2012/lz4/lz4.vcxproj index b7e0061..8ac9387 100755 --- a/visual/2012/lz4/lz4.vcxproj +++ b/visual/2012/lz4/lz4.vcxproj @@ -89,6 +89,7 @@ Level4 Disabled WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console @@ -102,6 +103,7 @@ Level4 Disabled WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true Console diff --git a/visual/2013/datagen/datagen.vcxproj b/visual/2013/datagen/datagen.vcxproj new file mode 100755 index 0000000..0d58b59 --- /dev/null +++ b/visual/2013/datagen/datagen.vcxproj @@ -0,0 +1,159 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {D745AE2F-596A-403A-9B91-81A8C6779243} + Win32Proj + datagen + + + + Application + true + v120 + Unicode + + + Application + true + v120 + Unicode + + + Application + false + v120 + true + Unicode + + + Application + false + v120 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2013/datagen/datagen.vcxproj.filters b/visual/2013/datagen/datagen.vcxproj.filters new file mode 100755 index 0000000..822d9bc --- /dev/null +++ b/visual/2013/datagen/datagen.vcxproj.filters @@ -0,0 +1,30 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2013/frametest/frametest.vcxproj b/visual/2013/frametest/frametest.vcxproj new file mode 100755 index 0000000..50f941d --- /dev/null +++ b/visual/2013/frametest/frametest.vcxproj @@ -0,0 +1,166 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7} + Win32Proj + frametest + + + + Application + true + v120 + Unicode + + + Application + true + v120 + Unicode + + + Application + false + v120 + true + Unicode + + + Application + false + v120 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2013/frametest/frametest.vcxproj.filters b/visual/2013/frametest/frametest.vcxproj.filters new file mode 100755 index 0000000..19c3d15 --- /dev/null +++ b/visual/2013/frametest/frametest.vcxproj.filters @@ -0,0 +1,51 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2013/fullbench/fullbench.vcxproj b/visual/2013/fullbench/fullbench.vcxproj new file mode 100755 index 0000000..1fcfb6a --- /dev/null +++ b/visual/2013/fullbench/fullbench.vcxproj @@ -0,0 +1,166 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E} + Win32Proj + fullbench + + + + Application + true + v120 + Unicode + + + Application + true + v120 + Unicode + + + Application + false + v120 + true + Unicode + + + Application + false + v120 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2013/fullbench/fullbench.vcxproj.filters b/visual/2013/fullbench/fullbench.vcxproj.filters new file mode 100755 index 0000000..cdcf599 --- /dev/null +++ b/visual/2013/fullbench/fullbench.vcxproj.filters @@ -0,0 +1,51 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2013/fuzzer/fuzzer.vcxproj b/visual/2013/fuzzer/fuzzer.vcxproj new file mode 100755 index 0000000..eadb17e --- /dev/null +++ b/visual/2013/fuzzer/fuzzer.vcxproj @@ -0,0 +1,163 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {18B9F1A7-9C66-4352-898B-30804DADE0FD} + Win32Proj + fuzzer + + + + Application + true + v120 + Unicode + + + Application + true + v120 + Unicode + + + Application + false + v120 + true + Unicode + + + Application + false + v120 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2013/fuzzer/fuzzer.vcxproj.filters b/visual/2013/fuzzer/fuzzer.vcxproj.filters new file mode 100755 index 0000000..81002e9 --- /dev/null +++ b/visual/2013/fuzzer/fuzzer.vcxproj.filters @@ -0,0 +1,42 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + \ No newline at end of file diff --git a/visual/2013/lz4.sln b/visual/2013/lz4.sln new file mode 100755 index 0000000..80d1a54 --- /dev/null +++ b/visual/2013/lz4.sln @@ -0,0 +1,66 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio Express 2012 for Windows Desktop +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lz4", "lz4\lz4.vcxproj", "{E30329AC-0057-4FE0-8FDA-7F650D398C4C}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fuzzer", "fuzzer\fuzzer.vcxproj", "{18B9F1A7-9C66-4352-898B-30804DADE0FD}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench", "fullbench\fullbench.vcxproj", "{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "frametest", "frametest\frametest.vcxproj", "{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "datagen", "datagen\datagen.vcxproj", "{D745AE2F-596A-403A-9B91-81A8C6779243}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Win32 = Debug|Win32 + Debug|x64 = Debug|x64 + Release|Win32 = Release|Win32 + Release|x64 = Release|x64 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.ActiveCfg = Debug|Win32 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.Build.0 = Debug|Win32 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.ActiveCfg = Debug|x64 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.Build.0 = Debug|x64 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.ActiveCfg = Release|Win32 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.Build.0 = Release|Win32 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.ActiveCfg = Release|x64 + {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.Build.0 = Release|x64 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.ActiveCfg = Debug|Win32 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.Build.0 = Debug|Win32 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.ActiveCfg = Debug|x64 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.Build.0 = Debug|x64 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.ActiveCfg = Release|Win32 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.Build.0 = Release|Win32 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.ActiveCfg = Release|x64 + {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.Build.0 = Release|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.ActiveCfg = Debug|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.Build.0 = Debug|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.ActiveCfg = Debug|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.Build.0 = Debug|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.ActiveCfg = Release|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.Build.0 = Release|Win32 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.ActiveCfg = Release|x64 + {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.Build.0 = Release|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.ActiveCfg = Debug|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.Build.0 = Debug|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.ActiveCfg = Debug|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.Build.0 = Debug|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.ActiveCfg = Release|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.Build.0 = Release|Win32 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.ActiveCfg = Release|x64 + {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.Build.0 = Release|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.ActiveCfg = Debug|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.Build.0 = Debug|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.ActiveCfg = Debug|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.Build.0 = Debug|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.ActiveCfg = Release|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.Build.0 = Release|Win32 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.ActiveCfg = Release|x64 + {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.Build.0 = Release|x64 + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection +EndGlobal diff --git a/visual/2013/lz4/lz4.vcxproj b/visual/2013/lz4/lz4.vcxproj new file mode 100755 index 0000000..3886f3f --- /dev/null +++ b/visual/2013/lz4/lz4.vcxproj @@ -0,0 +1,170 @@ + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {E30329AC-0057-4FE0-8FDA-7F650D398C4C} + Win32Proj + lz4 + + + + Application + true + v120 + Unicode + + + Application + true + v120 + Unicode + + + Application + false + v120 + true + Unicode + + + Application + false + v120 + true + Unicode + + + + + + + + + + + + + + + + + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + true + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + false + $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + + + Level4 + Disabled + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + Level4 + + + MaxSpeed + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + + + Console + true + true + true + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/visual/2013/lz4/lz4.vcxproj.filters b/visual/2013/lz4/lz4.vcxproj.filters new file mode 100755 index 0000000..fbe688b --- /dev/null +++ b/visual/2013/lz4/lz4.vcxproj.filters @@ -0,0 +1,63 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms + + + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + Fichiers d%27en-tête + + + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + Fichiers sources + + + \ No newline at end of file From 7b5e94575a5f587553247b685c6686fcc63c2343 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 20:27:34 +0100 Subject: [PATCH 017/121] Removed Visual 2013 solution, as AppVeyor automated mode only works with a single solution --- visual/2013/datagen/datagen.vcxproj | 159 ---------------- visual/2013/datagen/datagen.vcxproj.filters | 30 ---- visual/2013/frametest/frametest.vcxproj | 166 ----------------- .../2013/frametest/frametest.vcxproj.filters | 51 ------ visual/2013/fullbench/fullbench.vcxproj | 166 ----------------- .../2013/fullbench/fullbench.vcxproj.filters | 51 ------ visual/2013/fuzzer/fuzzer.vcxproj | 163 ----------------- visual/2013/fuzzer/fuzzer.vcxproj.filters | 42 ----- visual/2013/lz4.sln | 66 ------- visual/2013/lz4/lz4.vcxproj | 170 ------------------ visual/2013/lz4/lz4.vcxproj.filters | 63 ------- 11 files changed, 1127 deletions(-) delete mode 100755 visual/2013/datagen/datagen.vcxproj delete mode 100755 visual/2013/datagen/datagen.vcxproj.filters delete mode 100755 visual/2013/frametest/frametest.vcxproj delete mode 100755 visual/2013/frametest/frametest.vcxproj.filters delete mode 100755 visual/2013/fullbench/fullbench.vcxproj delete mode 100755 visual/2013/fullbench/fullbench.vcxproj.filters delete mode 100755 visual/2013/fuzzer/fuzzer.vcxproj delete mode 100755 visual/2013/fuzzer/fuzzer.vcxproj.filters delete mode 100755 visual/2013/lz4.sln delete mode 100755 visual/2013/lz4/lz4.vcxproj delete mode 100755 visual/2013/lz4/lz4.vcxproj.filters diff --git a/visual/2013/datagen/datagen.vcxproj b/visual/2013/datagen/datagen.vcxproj deleted file mode 100755 index 0d58b59..0000000 --- a/visual/2013/datagen/datagen.vcxproj +++ /dev/null @@ -1,159 +0,0 @@ - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {D745AE2F-596A-403A-9B91-81A8C6779243} - Win32Proj - datagen - - - - Application - true - v120 - Unicode - - - Application - true - v120 - Unicode - - - Application - false - v120 - true - Unicode - - - Application - false - v120 - true - Unicode - - - - - - - - - - - - - - - - - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - - - - - - - - - \ No newline at end of file diff --git a/visual/2013/datagen/datagen.vcxproj.filters b/visual/2013/datagen/datagen.vcxproj.filters deleted file mode 100755 index 822d9bc..0000000 --- a/visual/2013/datagen/datagen.vcxproj.filters +++ /dev/null @@ -1,30 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hpp;hxx;hm;inl;inc;xsd - - - {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} - rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms - - - - - Fichiers sources - - - Fichiers sources - - - - - Fichiers d%27en-tête - - - \ No newline at end of file diff --git a/visual/2013/frametest/frametest.vcxproj b/visual/2013/frametest/frametest.vcxproj deleted file mode 100755 index 50f941d..0000000 --- a/visual/2013/frametest/frametest.vcxproj +++ /dev/null @@ -1,166 +0,0 @@ - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7} - Win32Proj - frametest - - - - Application - true - v120 - Unicode - - - Application - true - v120 - Unicode - - - Application - false - v120 - true - Unicode - - - Application - false - v120 - true - Unicode - - - - - - - - - - - - - - - - - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/visual/2013/frametest/frametest.vcxproj.filters b/visual/2013/frametest/frametest.vcxproj.filters deleted file mode 100755 index 19c3d15..0000000 --- a/visual/2013/frametest/frametest.vcxproj.filters +++ /dev/null @@ -1,51 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hpp;hxx;hm;inl;inc;xsd - - - {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} - rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms - - - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - \ No newline at end of file diff --git a/visual/2013/fullbench/fullbench.vcxproj b/visual/2013/fullbench/fullbench.vcxproj deleted file mode 100755 index 1fcfb6a..0000000 --- a/visual/2013/fullbench/fullbench.vcxproj +++ /dev/null @@ -1,166 +0,0 @@ - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E} - Win32Proj - fullbench - - - - Application - true - v120 - Unicode - - - Application - true - v120 - Unicode - - - Application - false - v120 - true - Unicode - - - Application - false - v120 - true - Unicode - - - - - - - - - - - - - - - - - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/visual/2013/fullbench/fullbench.vcxproj.filters b/visual/2013/fullbench/fullbench.vcxproj.filters deleted file mode 100755 index cdcf599..0000000 --- a/visual/2013/fullbench/fullbench.vcxproj.filters +++ /dev/null @@ -1,51 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hpp;hxx;hm;inl;inc;xsd - - - {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} - rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms - - - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - \ No newline at end of file diff --git a/visual/2013/fuzzer/fuzzer.vcxproj b/visual/2013/fuzzer/fuzzer.vcxproj deleted file mode 100755 index eadb17e..0000000 --- a/visual/2013/fuzzer/fuzzer.vcxproj +++ /dev/null @@ -1,163 +0,0 @@ - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {18B9F1A7-9C66-4352-898B-30804DADE0FD} - Win32Proj - fuzzer - - - - Application - true - v120 - Unicode - - - Application - true - v120 - Unicode - - - Application - false - v120 - true - Unicode - - - Application - false - v120 - true - Unicode - - - - - - - - - - - - - - - - - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/visual/2013/fuzzer/fuzzer.vcxproj.filters b/visual/2013/fuzzer/fuzzer.vcxproj.filters deleted file mode 100755 index 81002e9..0000000 --- a/visual/2013/fuzzer/fuzzer.vcxproj.filters +++ /dev/null @@ -1,42 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hpp;hxx;hm;inl;inc;xsd - - - {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} - rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms - - - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - \ No newline at end of file diff --git a/visual/2013/lz4.sln b/visual/2013/lz4.sln deleted file mode 100755 index 80d1a54..0000000 --- a/visual/2013/lz4.sln +++ /dev/null @@ -1,66 +0,0 @@ - -Microsoft Visual Studio Solution File, Format Version 12.00 -# Visual Studio Express 2012 for Windows Desktop -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lz4", "lz4\lz4.vcxproj", "{E30329AC-0057-4FE0-8FDA-7F650D398C4C}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fuzzer", "fuzzer\fuzzer.vcxproj", "{18B9F1A7-9C66-4352-898B-30804DADE0FD}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench", "fullbench\fullbench.vcxproj", "{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "frametest", "frametest\frametest.vcxproj", "{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "datagen", "datagen\datagen.vcxproj", "{D745AE2F-596A-403A-9B91-81A8C6779243}" -EndProject -Global - GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|Win32 = Debug|Win32 - Debug|x64 = Debug|x64 - Release|Win32 = Release|Win32 - Release|x64 = Release|x64 - EndGlobalSection - GlobalSection(ProjectConfigurationPlatforms) = postSolution - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.ActiveCfg = Debug|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.Build.0 = Debug|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.ActiveCfg = Debug|x64 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.Build.0 = Debug|x64 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.ActiveCfg = Release|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.Build.0 = Release|Win32 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.ActiveCfg = Release|x64 - {E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.Build.0 = Release|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.ActiveCfg = Debug|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.Build.0 = Debug|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.ActiveCfg = Debug|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.Build.0 = Debug|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.ActiveCfg = Release|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.Build.0 = Release|Win32 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.ActiveCfg = Release|x64 - {18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.Build.0 = Release|x64 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.ActiveCfg = Debug|Win32 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.Build.0 = Debug|Win32 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.ActiveCfg = Debug|x64 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.Build.0 = Debug|x64 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.ActiveCfg = Release|Win32 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.Build.0 = Release|Win32 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.ActiveCfg = Release|x64 - {6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.Build.0 = Release|x64 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.ActiveCfg = Debug|Win32 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.Build.0 = Debug|Win32 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.ActiveCfg = Debug|x64 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.Build.0 = Debug|x64 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.ActiveCfg = Release|Win32 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.Build.0 = Release|Win32 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.ActiveCfg = Release|x64 - {39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.Build.0 = Release|x64 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.ActiveCfg = Debug|Win32 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.Build.0 = Debug|Win32 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.ActiveCfg = Debug|x64 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.Build.0 = Debug|x64 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.ActiveCfg = Release|Win32 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.Build.0 = Release|Win32 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.ActiveCfg = Release|x64 - {D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.Build.0 = Release|x64 - EndGlobalSection - GlobalSection(SolutionProperties) = preSolution - HideSolutionNode = FALSE - EndGlobalSection -EndGlobal diff --git a/visual/2013/lz4/lz4.vcxproj b/visual/2013/lz4/lz4.vcxproj deleted file mode 100755 index 3886f3f..0000000 --- a/visual/2013/lz4/lz4.vcxproj +++ /dev/null @@ -1,170 +0,0 @@ - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {E30329AC-0057-4FE0-8FDA-7F650D398C4C} - Win32Proj - lz4 - - - - Application - true - v120 - Unicode - - - Application - true - v120 - Unicode - - - Application - false - v120 - true - Unicode - - - Application - false - v120 - true - Unicode - - - - - - - - - - - - - - - - - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - true - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - false - $(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - - - Level4 - Disabled - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - Level4 - - - MaxSpeed - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - - - Console - true - true - true - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/visual/2013/lz4/lz4.vcxproj.filters b/visual/2013/lz4/lz4.vcxproj.filters deleted file mode 100755 index fbe688b..0000000 --- a/visual/2013/lz4/lz4.vcxproj.filters +++ /dev/null @@ -1,63 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hpp;hxx;hm;inl;inc;xsd - - - {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} - rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms - - - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - Fichiers d%27en-tête - - - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - Fichiers sources - - - \ No newline at end of file From 886b19951c08d4155482ae7841cc6773f1f02f46 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 31 Mar 2015 20:29:24 +0100 Subject: [PATCH 018/121] Modified files rights --- NEWS | 1 + visual/2012/datagen/datagen.vcxproj | 0 visual/2012/datagen/datagen.vcxproj.filters | 0 visual/2012/frametest/frametest.vcxproj | 0 visual/2012/frametest/frametest.vcxproj.filters | 0 visual/2012/fullbench/fullbench.vcxproj | 0 visual/2012/fullbench/fullbench.vcxproj.filters | 0 visual/2012/fuzzer/fuzzer.vcxproj | 0 visual/2012/fuzzer/fuzzer.vcxproj.filters | 0 visual/2012/lz4.sln | 0 visual/2012/lz4/lz4.vcxproj | 0 visual/2012/lz4/lz4.vcxproj.filters | 0 12 files changed, 1 insertion(+) mode change 100755 => 100644 visual/2012/datagen/datagen.vcxproj mode change 100755 => 100644 visual/2012/datagen/datagen.vcxproj.filters mode change 100755 => 100644 visual/2012/frametest/frametest.vcxproj mode change 100755 => 100644 visual/2012/frametest/frametest.vcxproj.filters mode change 100755 => 100644 visual/2012/fullbench/fullbench.vcxproj mode change 100755 => 100644 visual/2012/fullbench/fullbench.vcxproj.filters mode change 100755 => 100644 visual/2012/fuzzer/fuzzer.vcxproj mode change 100755 => 100644 visual/2012/fuzzer/fuzzer.vcxproj.filters mode change 100755 => 100644 visual/2012/lz4.sln mode change 100755 => 100644 visual/2012/lz4/lz4.vcxproj mode change 100755 => 100644 visual/2012/lz4/lz4.vcxproj.filters diff --git a/NEWS b/NEWS index 233dc82..a1d452a 100644 --- a/NEWS +++ b/NEWS @@ -1,5 +1,6 @@ r129: Added : LZ4_compress_fast() +Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka Updated: Documentation converted to MarkDown r128: diff --git a/visual/2012/datagen/datagen.vcxproj b/visual/2012/datagen/datagen.vcxproj old mode 100755 new mode 100644 diff --git a/visual/2012/datagen/datagen.vcxproj.filters b/visual/2012/datagen/datagen.vcxproj.filters old mode 100755 new mode 100644 diff --git a/visual/2012/frametest/frametest.vcxproj b/visual/2012/frametest/frametest.vcxproj old mode 100755 new mode 100644 diff --git a/visual/2012/frametest/frametest.vcxproj.filters b/visual/2012/frametest/frametest.vcxproj.filters old mode 100755 new mode 100644 diff --git a/visual/2012/fullbench/fullbench.vcxproj b/visual/2012/fullbench/fullbench.vcxproj old mode 100755 new mode 100644 diff --git a/visual/2012/fullbench/fullbench.vcxproj.filters b/visual/2012/fullbench/fullbench.vcxproj.filters old mode 100755 new mode 100644 diff --git a/visual/2012/fuzzer/fuzzer.vcxproj b/visual/2012/fuzzer/fuzzer.vcxproj old mode 100755 new mode 100644 diff --git a/visual/2012/fuzzer/fuzzer.vcxproj.filters b/visual/2012/fuzzer/fuzzer.vcxproj.filters old mode 100755 new mode 100644 diff --git a/visual/2012/lz4.sln b/visual/2012/lz4.sln old mode 100755 new mode 100644 diff --git a/visual/2012/lz4/lz4.vcxproj b/visual/2012/lz4/lz4.vcxproj old mode 100755 new mode 100644 diff --git a/visual/2012/lz4/lz4.vcxproj.filters b/visual/2012/lz4/lz4.vcxproj.filters old mode 100755 new mode 100644 From 662506890285faa949fd627b8da8394813b37f0b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 14:48:24 +0100 Subject: [PATCH 019/121] simplified LZ4_compress() --- lib/lz4.c | 84 +++++++++++++++++++++++++++++-------------------------- lib/lz4.h | 6 ++-- 2 files changed, 47 insertions(+), 43 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 3dc44f2..a078ced 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -582,8 +582,8 @@ static int LZ4_compress_generic( BYTE* token; { const BYTE* forwardIp = ip; - unsigned step=1; - unsigned searchMatchNb = ((acceleration) << LZ4_skipTrigger); + unsigned step = 1; + unsigned searchMatchNb = acceleration << LZ4_skipTrigger; /* Find a match */ do { @@ -717,13 +717,22 @@ _next_match: _last_literals: /* Encode Last Literals */ { - int lastRun = (int)(iend - anchor); - if ((outputLimited) && (((char*)op - dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize)) + const size_t lastRun = (size_t)(iend - anchor); + if ((outputLimited) && ((op - (BYTE*)dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize)) return 0; /* Check output limit */ - if (lastRun>=(int)RUN_MASK) { *op++=(RUN_MASK<= 255 ; lastRun-=255) *op++ = 255; *op++ = (BYTE) lastRun; } - else *op++ = (BYTE)(lastRun<= RUN_MASK) + { + size_t accumulator = lastRun - RUN_MASK; + *op++ = RUN_MASK << ML_BITS; + for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; + *op++ = (BYTE) accumulator; + } + else + { + *op++ = (BYTE)(lastRun<= LZ4_compressBound(inputSize)) + { + if (inputSize < LZ4_64Klimit) + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); + else + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); + } else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); - + { + if (inputSize < LZ4_64Klimit) + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); + else + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); + } #if (HEAPMODE) FREEMEM(ctx); #endif @@ -772,6 +770,12 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in } +int LZ4_compress(const char* source, char* dest, int inputSize) +{ + return LZ4_compress_limitedOutput(source, dest, inputSize, LZ4_compressBound(inputSize)); +} + + int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) { #if (HEAPMODE) @@ -807,6 +811,14 @@ int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutp * Experimental : Streaming functions *****************************************/ +LZ4_stream_t* LZ4_createStream(void) +{ + LZ4_stream_t* lz4s = (LZ4_stream_t*)ALLOCATOR(8, LZ4_STREAMSIZE_U64); + LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */ + LZ4_resetStream(lz4s); + return lz4s; +} + /* * LZ4_initStream * Use this function once, to init a newly allocated LZ4_stream_t structure @@ -817,14 +829,6 @@ void LZ4_resetStream (LZ4_stream_t* LZ4_stream) MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t)); } -LZ4_stream_t* LZ4_createStream(void) -{ - LZ4_stream_t* lz4s = (LZ4_stream_t*)ALLOCATOR(8, LZ4_STREAMSIZE_U64); - LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */ - LZ4_resetStream(lz4s); - return lz4s; -} - int LZ4_freeStream (LZ4_stream_t* LZ4_stream) { FREEMEM(LZ4_stream); diff --git a/lib/lz4.h b/lib/lz4.h index eabc40f..7072c37 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -99,7 +99,7 @@ LZ4_decompress_safe() : * Advanced Functions **************************************/ #define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ -#define LZ4_COMPRESSBOUND(isize) ((unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16) +#define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16) /* LZ4_compressBound() : @@ -107,11 +107,11 @@ LZ4_compressBound() : This function is primarily useful for memory allocation purposes (output buffer size). Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). - isize : is the input size. Max supported value is LZ4_MAX_INPUT_SIZE + inputSize : max supported value is LZ4_MAX_INPUT_SIZE return : maximum output size in a "worst case" scenario or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE) */ -int LZ4_compressBound(int isize); +int LZ4_compressBound(int inputSize); /* From 76a03c1035829116e6fa9e80ad16364524514f9f Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 14:59:42 +0100 Subject: [PATCH 020/121] simplified LZ4_compress_withState() --- lib/lz4.c | 54 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 28 insertions(+), 26 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index a078ced..31dfc9f 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -740,6 +740,34 @@ _last_literals: } +int LZ4_sizeofState() { return LZ4_STREAMSIZE; } + +int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) +{ + if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */ + MEM_INIT(state, 0, LZ4_STREAMSIZE); + + if (maxOutputSize >= LZ4_compressBound(inputSize)) + { + if (inputSize < LZ4_64Klimit) + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); + else + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); + } + else + { + if (inputSize < LZ4_64Klimit) + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); + else + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); + } +} + +int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize) +{ + return LZ4_compress_limitedOutput_withState(state, source, dest, inputSize, LZ4_compressBound(inputSize)); +} + int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { #if (HEAPMODE) @@ -1384,32 +1412,6 @@ char* LZ4_slideInputBuffer (void* LZ4_Data) return (char*)(ctx->bufferStart + dictSize); } -/* Obsolete compresson functions using User-allocated state */ - -int LZ4_sizeofState() { return LZ4_STREAMSIZE; } - -int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize) -{ - if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */ - MEM_INIT(state, 0, LZ4_STREAMSIZE); - - if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); - else - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); -} - -int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) -{ - if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */ - MEM_INIT(state, 0, LZ4_STREAMSIZE); - - if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); - else - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); -} - /* Obsolete streaming decompression functions */ int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, int compressedSize, int maxOutputSize) From 0615eb4814d68579b26b9c5b388c146d9d553228 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 15:05:27 +0100 Subject: [PATCH 021/121] Stricter tests : treat warnings as errors --- Makefile | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 59759c0..9cd2a29 100644 --- a/Makefile +++ b/Makefile @@ -131,18 +131,18 @@ test-travis: $(TRAVIS_TARGET) cmake: @cd cmake_unofficial; cmake CMakeLists.txt; $(MAKE) +gpptest: clean + export CC=g++; export CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror"; $(MAKE) -e all + clangtest: clean - export CC=clang; $(MAKE) all + export CFLAGS=-Werror; export CC=clang; $(MAKE) all staticAnalyze: clean export CFLAGS=-g; scan-build -v $(MAKE) all -gpptest: clean - export CC=g++; export CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align"; $(MAKE) -e all - armtest: clean - export CC=arm-linux-gnueabi-gcc; cd lib; $(MAKE) -e all - export CC=arm-linux-gnueabi-gcc; cd programs; $(MAKE) -e bins + export CFLAGS=-Werror; export CC=arm-linux-gnueabi-gcc; cd lib; $(MAKE) -e all + export CFLAGS=-Werror; export CC=arm-linux-gnueabi-gcc; cd programs; $(MAKE) -e bins streaming-examples: cd examples; $(MAKE) -e test From 89eee0d28b2b7c5954e1cb0dce60542e4c597130 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 15:09:32 +0100 Subject: [PATCH 022/121] Removed make dist --- .travis.yml | 1 - Makefile | 34 ---------------------------------- 2 files changed, 35 deletions(-) diff --git a/.travis.yml b/.travis.yml index 8fabb1c..8f45f71 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,7 +10,6 @@ before_install: - sudo apt-get install -qq valgrind env: - - LZ4_TRAVIS_CI_ENV=dist - LZ4_TRAVIS_CI_ENV=travis-install - LZ4_TRAVIS_CI_ENV=streaming-examples - LZ4_TRAVIS_CI_ENV=cmake diff --git a/Makefile b/Makefile index 9cd2a29..284972f 100644 --- a/Makefile +++ b/Makefile @@ -41,22 +41,6 @@ LIBDIR ?= $(PREFIX)/lib INCLUDEDIR=$(PREFIX)/include PRGDIR = programs LZ4DIR = lib -DISTRIBNAME=lz4-$(RELEASE).tar.gz - -TEXT = $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4.h $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4hc.h \ - $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4frame.h $(LZ4DIR)/lz4frame_static.h \ - $(LZ4DIR)/xxhash.c $(LZ4DIR)/xxhash.h \ - $(LZ4DIR)/liblz4.pc.in $(LZ4DIR)/Makefile $(LZ4DIR)/LICENSE \ - Makefile lz4_Block_format.md lz4_Frame_format.md NEWS README.md \ - cmake_unofficial/CMakeLists.txt \ - $(PRGDIR)/fullbench.c $(PRGDIR)/lz4cli.c \ - $(PRGDIR)/datagen.c $(PRGDIR)/datagen.h $(PRGDIR)/datagencli.c $(PRGDIR)/fuzzer.c \ - $(PRGDIR)/lz4io.c $(PRGDIR)/lz4io.h \ - $(PRGDIR)/bench.c $(PRGDIR)/bench.h \ - $(PRGDIR)/lz4.1 \ - $(PRGDIR)/Makefile $(PRGDIR)/COPYING -NONTEXT = -SOURCES = $(TEXT) $(NONTEXT) # Select test target for Travis CI's Build Matrix @@ -84,7 +68,6 @@ lz4programs: @cd $(PRGDIR); $(MAKE) -e clean: - @rm -f $(DISTRIBNAME) *.sha1 > $(VOID) @cd $(PRGDIR); $(MAKE) clean > $(VOID) @cd $(LZ4DIR); $(MAKE) clean > $(VOID) @cd examples; $(MAKE) clean > $(VOID) @@ -106,23 +89,6 @@ uninstall: travis-install: sudo $(MAKE) install -dist: clean - @install -dD -m 700 lz4-$(RELEASE)/lib/ - @install -dD -m 700 lz4-$(RELEASE)/programs/ - @install -dD -m 700 lz4-$(RELEASE)/cmake_unofficial/ - @for f in $(TEXT); do \ - tr -d '\r' < $$f > .tmp; \ - install -m 600 .tmp lz4-$(RELEASE)/$$f; \ - done - @rm .tmp - @for f in $(NONTEXT); do \ - install -m 600 $$f lz4-$(RELEASE)/$$f; \ - done - @tar -czf $(DISTRIBNAME) lz4-$(RELEASE)/ - @rm -rf lz4-$(RELEASE) - @sha1sum $(DISTRIBNAME) > $(DISTRIBNAME).sha1 - @echo Distribution $(DISTRIBNAME) built - test: @cd $(PRGDIR); $(MAKE) -e test From 28e237e954098c418f048a871aa53199d92314f4 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 15:53:08 +0100 Subject: [PATCH 023/121] simplified LZ4_compress_limitedOutput() --- lib/lz4.c | 19 +++---------------- lib/lz4.h | 2 +- 2 files changed, 4 insertions(+), 17 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 31dfc9f..b0ce64f 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -771,26 +771,13 @@ int LZ4_compress_withState (void* state, const char* source, char* dest, int inp int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { #if (HEAPMODE) - void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ + void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* malloc-calloc aligned on 8-bytes boundaries */ #else U64 ctx[LZ4_STREAMSIZE_U64] = {0}; /* Ensure data is aligned on 8-bytes boundaries */ #endif - int result; - if (maxOutputSize >= LZ4_compressBound(inputSize)) - { - if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); - else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); - } - else - { - if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); - else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); - } + int result = LZ4_compress_limitedOutput_withState(ctx, source, dest, inputSize, maxOutputSize); + #if (HEAPMODE) FREEMEM(ctx); #endif diff --git a/lib/lz4.h b/lib/lz4.h index 7072c37..92c8738 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -48,7 +48,7 @@ extern "C" { * Version **************************************/ #define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */ -#define LZ4_VERSION_MINOR 6 /* for new (non-breaking) interface capabilities */ +#define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */ #define LZ4_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */ #define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE) int LZ4_versionNumber (void); From ef7cd83271eb6a513dd4fe389ee37a9bfc8e00e6 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 17:30:36 +0100 Subject: [PATCH 024/121] Fixed issue 160, reported by Eric Berge --- lib/lz4frame.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 474b196..6c65d0b 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -204,7 +204,7 @@ static U64 LZ4F_readLE64 (const BYTE* srcPtr) U64 value64 = srcPtr[0]; value64 += (srcPtr[1]<<8); value64 += (srcPtr[2]<<16); - value64 += (srcPtr[3]<<24); + value64 += ((U64)srcPtr[3]<<24); value64 += ((U64)srcPtr[4]<<32); value64 += ((U64)srcPtr[5]<<40); value64 += ((U64)srcPtr[6]<<48); From 78d2dfd42758328c65d8517b8ee8e6124d7f8407 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 1 Apr 2015 18:21:03 +0100 Subject: [PATCH 025/121] fullbench : tests of _limitedOutput variants intentionnally provides less memory space than safe (LZ4_compressBound()) --- lib/lz4.c | 43 ++++++++++++++++--------------------------- lib/lz4.h | 1 + programs/Makefile | 5 ++++- programs/fullbench.c | 25 ++++++++++++------------- 4 files changed, 33 insertions(+), 41 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index b0ce64f..a06f711 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -119,7 +119,7 @@ # pragma warning(disable : 4293) /* disable: C4293: too large shift (32-bits) */ #else # if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ -# ifdef __GNUC__ +# if defined(__GNUC__) || defined(__clang__) # define FORCE_INLINE static inline __attribute__((always_inline)) # else # define FORCE_INLINE static inline @@ -428,13 +428,6 @@ static const int LZ4_64Klimit = ((64 KB) + (MFLIMIT-1)); static const U32 LZ4_skipTrigger = 6; /* Increase this value ==> compression run slower on incompressible data */ -/************************************** -* Local Utils -**************************************/ -int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; } -int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); } - - /************************************** * Local Structures and types **************************************/ @@ -457,6 +450,14 @@ typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive; typedef enum { full = 0, partial = 1 } earlyEnd_directive; +/************************************** +* Local Utils +**************************************/ +int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; } +int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); } +int LZ4_sizeofState() { return LZ4_STREAMSIZE; } + + /******************************** * Compression functions @@ -487,8 +488,6 @@ static U32 LZ4_hashSequenceT(size_t sequence, tableType_t const tableType) static U32 LZ4_hashPosition(const void* p, tableType_t tableType) { return LZ4_hashSequenceT(LZ4_read_ARCH(p), tableType); } -//static U32 LZ4_hashPosition(const BYTE* p, tableType_t tableType) { return LZ4_hashSequence(LZ4_read32(p), tableType); } - static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase) { switch (tableType) @@ -740,11 +739,8 @@ _last_literals: } -int LZ4_sizeofState() { return LZ4_STREAMSIZE; } - int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) { - if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */ MEM_INIT(state, 0, LZ4_STREAMSIZE); if (maxOutputSize >= LZ4_compressBound(inputSize)) @@ -757,7 +753,7 @@ int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* else { if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); else return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); } @@ -784,7 +780,6 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in return result; } - int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_limitedOutput(source, dest, inputSize, LZ4_compressBound(inputSize)); @@ -904,8 +899,7 @@ static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src) } -FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* source, char* dest, int inputSize, - int maxOutputSize, limitedOutput_directive limit) +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) { LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_stream; const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; @@ -932,9 +926,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, dictSmall, 1); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, noDictIssue, 1); streamPtr->dictSize += (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; @@ -944,9 +938,9 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, dictSmall, 1); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, noDictIssue, 1); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; @@ -956,12 +950,7 @@ FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* so int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { - return LZ4_compress_continue_generic(LZ4_stream, source, dest, inputSize, 0, notLimited); -} - -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) -{ - return LZ4_compress_continue_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput); + return LZ4_compress_limitedOutput_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize)); } diff --git a/lib/lz4.h b/lib/lz4.h index 92c8738..81a11c9 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -144,6 +144,7 @@ LZ4_compress_withState() : Same compression functions, but using an externally allocated memory space to store compression state. Use LZ4_sizeofState() to know how much memory must be allocated, and then, provide it as 'void* state' to compression functions. + Note that 'state' must be aligned on 4-bytes boundaries. */ int LZ4_sizeofState(void); int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); diff --git a/programs/Makefile b/programs/Makefile index a324148..0cebdfe 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -27,11 +27,14 @@ # lz4c32: Same as lz4c, but forced to compile in 32-bits mode # fuzzer : Test tool, to check lz4 integrity on target platform # fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode +# frametest : Test tool, to check lz4frame integrity on target platform +# frametest32: Same as frametest, but forced to compile in 32-bits mode # fullbench : Precisely measure speed for each LZ4 function variant # fullbench32: Same as fullbench, but forced to compile in 32-bits mode +# datagen : generates synthetic data samples for tests & benchmarks # ########################################################################## -RELEASE?= r128 +RELEASE?= r129 DESTDIR?= PREFIX ?= /usr/local diff --git a/programs/fullbench.c b/programs/fullbench.c index cd6e3c8..41075bc 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -383,7 +383,7 @@ start: static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSize) { - return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); } static int local_LZ4_compress_fast(const char* in, char* out, int inSize) @@ -399,7 +399,7 @@ static int local_LZ4_compress_withState(const char* in, char* out, int inSize) static int local_LZ4_compress_limitedOutput_withState(const char* in, char* out, int inSize) { - return LZ4_compress_limitedOutput_withState(stateLZ4, in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compress_limitedOutput_withState(stateLZ4, in, out, inSize, LZ4_compressBound(inSize)-1); } static LZ4_stream_t* ctx; @@ -410,7 +410,7 @@ static int local_LZ4_compress_continue(const char* in, char* out, int inSize) static int local_LZ4_compress_limitedOutput_continue(const char* in, char* out, int inSize) { - return LZ4_compress_limitedOutput_continue(ctx, in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compress_limitedOutput_continue(ctx, in, out, inSize, LZ4_compressBound(inSize)-1); } @@ -437,12 +437,12 @@ static int local_LZ4_compressHC_withStateHC(const char* in, char* out, int inSiz static int local_LZ4_compressHC_limitedOutput_withStateHC(const char* in, char* out, int inSize) { - return LZ4_compressHC_limitedOutput_withStateHC(stateLZ4HC, in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compressHC_limitedOutput_withStateHC(stateLZ4HC, in, out, inSize, LZ4_compressBound(inSize)-1); } static int local_LZ4_compressHC_limitedOutput(const char* in, char* out, int inSize) { - return LZ4_compressHC_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compressHC_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); } static int local_LZ4_compressHC_continue(const char* in, char* out, int inSize) @@ -452,7 +452,7 @@ static int local_LZ4_compressHC_continue(const char* in, char* out, int inSize) static int local_LZ4_compressHC_limitedOutput_continue(const char* in, char* out, int inSize) { - return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, in, out, inSize, LZ4_compressBound(inSize)); + return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, in, out, inSize, LZ4_compressBound(inSize)-1); } static int local_LZ4F_compressFrame(const char* in, char* out, int inSize) @@ -477,7 +477,6 @@ static int local_LZ4_saveDictHC(const char* in, char* out, int inSize) static int local_LZ4_decompress_fast(const char* in, char* out, int inSize, int outSize) { (void)inSize; - //lz4_decode_sse((BYTE*)out, (BYTE*)in, inSize); LZ4_decompress_fast(in, out, outSize); return outSize; } @@ -617,11 +616,11 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) return 13; } - // Calculating input Checksum + /* Calculating input Checksum */ crcOriginal = XXH32(orig_buff, (unsigned int)benchedSize,0); - // Bench + /* Bench */ { int loopNb, nb_loops, chunkNb, cAlgNb, dAlgNb; size_t cSize=0; @@ -638,7 +637,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) void* (*initFunction)(const char*) = NULL; double bestTime = 100000000.; - // Init data chunks + /* Init data chunks */ { int i; size_t remaining = benchedSize; @@ -691,7 +690,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) int milliTime; PROGRESS("%1i- %-28.28s :%9i ->\r", loopNb, compressorName, (int)benchedSize); - { size_t i; for (i=0; i Date: Wed, 1 Apr 2015 15:47:24 -0300 Subject: [PATCH 026/121] Cygwin has fileno, so there's no need to use _fileno. --- programs/lz4cli.c | 2 +- programs/lz4io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index da5da71..888f21e 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -65,7 +65,7 @@ /**************************** * OS-specific Includes *****************************/ -#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__) +#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) # include /* _isatty */ # ifdef __MINGW32__ int _fileno(FILE *stream); /* MINGW somehow forgets to include this prototype into */ diff --git a/programs/lz4io.c b/programs/lz4io.c index 02e03c8..0cf6293 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -61,7 +61,7 @@ /****************************** * OS-specific Includes ******************************/ -#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__) +#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) # include /* _O_BINARY */ # include /* _setmode, _fileno, _get_osfhandle */ # define SET_BINARY_MODE(file) _setmode(_fileno(file), _O_BINARY) From 90c0104c4eaf8f06932990aacf07e60dd21685b9 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 2 Apr 2015 13:25:28 +0100 Subject: [PATCH 027/121] Added : progress indicator, in fast and decompression modes --- programs/lz4io.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 02e03c8..0d931fe 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -412,7 +412,6 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena /* Init */ start = clock(); memset(&prefs, 0, sizeof(prefs)); - if ((g_displayLevel==2) && (compressionLevel>=3)) g_displayLevel=3; errorCode = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION); if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); get_fileHandle(input_filename, output_filename, &finput, &foutput); @@ -428,6 +427,8 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena { unsigned long long fileSize = LZ4IO_GetFileSize(input_filename); prefs.frameInfo.contentSize = fileSize; /* == 0 if input == stdin */ + if (fileSize==0) + DISPLAYLEVEL(3, "Warning : cannot determine uncompressed frame content size \n"); } /* Allocate Memory */ @@ -456,7 +457,7 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena outSize = LZ4F_compressUpdate(ctx, out_buff, outBuffSize, in_buff, readSize, NULL); if (LZ4F_isError(outSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(outSize)); compressedfilesize += outSize; - DISPLAYUPDATE(3, "\rRead : %i MB ==> %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); + DISPLAYUPDATE(2, "\rRead : %i MB ==> %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); /* Write Block */ sizeCheck = fwrite(out_buff, 1, outSize, foutput); @@ -639,6 +640,7 @@ static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) { /* Write Block */ filesize += decodedBytes; + DISPLAYUPDATE(2, "\rDecompressed : %u MB ", (unsigned)(filesize>>20)); if (g_sparseFileSupport) { size_t* const oBuffStartT = (size_t*)outBuff; /* since outBuff is malloc'ed, it's aligned on size_t */ From f72761ff44ebeeb5deb19ac588ff4e3905148758 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 2 Apr 2015 15:11:22 +0100 Subject: [PATCH 028/121] new tests for large files with content size support (#70) --- programs/Makefile | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/programs/Makefile b/programs/Makefile index 0cebdfe..9a1bade 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -171,6 +171,11 @@ test-lz4-contentSize: lz4 datagen ./lz4 -v tmp | ./lz4 -t ./lz4 -v --content-size tmp | ./lz4 -d > tmp2 diff -s tmp tmp2 + # test large size [2-4] GB + @./datagen -g3G -P100 | ./lz4 | ./lz4 --decompress --force --sparse - tmp + @ls -ls tmp + ./lz4 --quiet --content-size tmp | ./lz4 --verbose --decompress --force --sparse - tmp2 + @ls -ls tmp2 @rm tmp* test-lz4-frame-concatenation: lz4 datagen From dd699026e6680e1828f5f00c1929861e0cdc471e Mon Sep 17 00:00:00 2001 From: Yongwoon Cho Date: Fri, 3 Apr 2015 11:52:50 +0900 Subject: [PATCH 029/121] Removed unused lines. --- programs/lz4io.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 02e03c8..e526a95 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -587,8 +587,6 @@ static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) unsigned long long filesize = 0; void* inBuff; void* outBuff; -# define HEADERMAX 20 - char headerBuff[HEADERMAX]; size_t sizeCheck; const size_t inBuffSize = 256 KB; const size_t outBuffSize = 256 KB; @@ -599,7 +597,6 @@ static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) /* init */ errorCode = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION); if (LZ4F_isError(errorCode)) EXM_THROW(60, "Can't create context : %s", LZ4F_getErrorName(errorCode)); - LZ4IO_writeLE32(headerBuff, LZ4IO_MAGICNUMBER); /* regenerated here, as it was already read from finput */ /* Allocate Memory */ inBuff = malloc(256 KB); From 979a991badffa8359232a6b7b74756a9821d2053 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 3 Apr 2015 23:58:29 +0100 Subject: [PATCH 030/121] memcpy speed as reference --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 453bb01..6de310a 100644 --- a/README.md +++ b/README.md @@ -41,15 +41,11 @@ in single-thread mode. | Compressor | Ratio | Compression | Decompression | | ---------- | ----- | ----------- | ------------- | | memcpy | 1.000 | 4200 MB/s | 4200 MB/s | -| RLE64 v3.0 | 1.029 | 2800 MB/s | 2800 MB/s | -| density -c1 | 1.592 | 700 MB/s | 920 MB/s | |**LZ4 fast (r129)**| 1.607 |**680 MB/s** | **2220 MB/s** | |**LZ4 (r129)** |**2.101**|**385 MB/s** | **1850 MB/s** | -| density -c2 | 2.083 | 370 MB/s | 505 MB/s | | LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | | QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | | Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | -| density -c3 | 2.370 | 190 MB/s | 185 MB/s | | zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | |**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | | zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | From 17f86149c52a3a3c131c23f2b6452f172d9364a6 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 6 Apr 2015 01:02:24 +0100 Subject: [PATCH 031/121] added : memtest on fullbench --- programs/Makefile | 3 ++- programs/fullbench.c | 20 +++++++++++--------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index 9a1bade..3ca46d7 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -243,13 +243,14 @@ test-frametest: frametest test-frametest32: frametest32 ./frametest32 -test-mem: lz4 datagen fuzzer frametest +test-mem: lz4 datagen fuzzer frametest fullbench valgrind --leak-check=yes ./datagen -g50M > $(VOID) ./datagen -g16KB > tmp valgrind --leak-check=yes ./lz4 -9 -BD -f tmp $(VOID) ./datagen -g16MB > tmp valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp tmp2 valgrind --leak-check=yes ./lz4 -t tmp2 + valgrind --leak-check=yes ./fullbench -i1 tmp ./datagen -g256MB > tmp valgrind --leak-check=yes ./lz4 -B4D -f -vq tmp $(VOID) rm tmp* diff --git a/programs/fullbench.c b/programs/fullbench.c index 41075bc..7c970e0 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -541,12 +541,6 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) double totalDTime[NB_DECOMPRESSION_ALGORITHMS+1] = {0}; size_t errorCode; - errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) - { - DISPLAY("dctx allocation issue \n"); - return 10; - } /* Loop for each fileName */ while (fileIdx Date: Mon, 6 Apr 2015 01:03:49 +0100 Subject: [PATCH 032/121] updated doc --- README.md | 11 +++++------ lz4_Frame_format.md | 4 ++-- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 6de310a..bcb8d7c 100644 --- a/README.md +++ b/README.md @@ -50,13 +50,12 @@ in single-thread mode. |**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | | zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | -The LZ4 block compression format is detailed within [lz4_Block_format]. +The raw LZ4 block compression format is detailed within [lz4_Block_format]. -Block format doesn't deal with header information, -nor how to handle arbitrarily long files or data streams. -This is the purpose of the Frame format. -Interoperable versions of LZ4 should use the same frame format, -defined into [lz4_Frame_format]. +Compressing an arbitrarily long file or data stream requires multiple blocks. +Organizing these blocks and providing a common header format to handle their content +is the purpose of the Frame format, defined into [lz4_Frame_format]. +Interoperable versions of LZ4 must respect this frame format too. [Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029 [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia diff --git a/lz4_Frame_format.md b/lz4_Frame_format.md index 55edaa0..cb41d47 100644 --- a/lz4_Frame_format.md +++ b/lz4_Frame_format.md @@ -341,8 +341,8 @@ Data is always compressed, even when compression is detrimental. __EndMark__ -End of compressed stream is implicit. -It needs to be followed by a standard EOF (End Of File) signal, +End of legacy frame is implicit only. +It must be followed by a standard EOF (End Of File) signal, wether it is a file or a stream. Alternatively, if the frame is followed by a valid Frame Magic Number, From 43e053513bb2386b526daad5621e29a817663691 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 7 Apr 2015 07:14:32 +0100 Subject: [PATCH 033/121] fix g++ typecast --- programs/fullbench.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/fullbench.c b/programs/fullbench.c index 7c970e0..2f246a2 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -391,7 +391,7 @@ static int local_LZ4_compress_fast(const char* in, char* out, int inSize) return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); } -static void* stateLZ4; +static LZ4_stream_t* stateLZ4; static int local_LZ4_compress_withState(const char* in, char* out, int inSize) { return LZ4_compress_withState(stateLZ4, in, out, inSize); @@ -429,7 +429,7 @@ static int local_LZ4_compress_forceDict(const char* in, char* out, int inSize) } -static void* stateLZ4HC; +static LZ4_streamHC_t* stateLZ4HC; static int local_LZ4_compressHC_withStateHC(const char* in, char* out, int inSize) { return LZ4_compressHC_withStateHC(stateLZ4HC, in, out, inSize); From 327cb04f44a650afd3907701513b1c2e851a3b45 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 8 Apr 2015 09:20:21 +0100 Subject: [PATCH 034/121] minor memory leak fix and test --- programs/Makefile | 1 + programs/lz4cli.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/programs/Makefile b/programs/Makefile index 3ca46d7..9585c9f 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -250,6 +250,7 @@ test-mem: lz4 datagen fuzzer frametest fullbench ./datagen -g16MB > tmp valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp tmp2 valgrind --leak-check=yes ./lz4 -t tmp2 + valgrind --leak-check=yes ./lz4 -bi1 tmp valgrind --leak-check=yes ./fullbench -i1 tmp ./datagen -g256MB > tmp valgrind --leak-check=yes ./lz4 -B4D -f -vq tmp $(VOID) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index 888f21e..2870c97 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -470,7 +470,12 @@ int main(int argc, char** argv) if (!strcmp(input_filename, stdinmark) && IS_CONSOLE(stdin) ) badusage(); /* Check if benchmark is selected */ - if (bench) return BMK_benchFiles(inFileNames, ifnIdx, cLevel); + if (bench) + { + int bmkResult = BMK_benchFiles(inFileNames, ifnIdx, cLevel); + free(inFileNames); + return bmkResult; + } /* No output filename ==> try to select one automatically (when possible) */ while (!output_filename) From b41137f42a59e6636a02c61f49b67bde7876d4c4 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 8 Apr 2015 09:33:57 +0100 Subject: [PATCH 035/121] minor Makefile test refactoring --- Makefile | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 284972f..831f0d3 100644 --- a/Makefile +++ b/Makefile @@ -98,17 +98,17 @@ cmake: @cd cmake_unofficial; cmake CMakeLists.txt; $(MAKE) gpptest: clean - export CC=g++; export CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror"; $(MAKE) -e all + $(MAKE) all CC=g++ CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror" clangtest: clean - export CFLAGS=-Werror; export CC=clang; $(MAKE) all + $(MAKE) all CC=clang CFLAGS="-O3 -Werror" staticAnalyze: clean - export CFLAGS=-g; scan-build -v $(MAKE) all + scan-build -v $(MAKE) all CFLAGS=-g armtest: clean - export CFLAGS=-Werror; export CC=arm-linux-gnueabi-gcc; cd lib; $(MAKE) -e all - export CFLAGS=-Werror; export CC=arm-linux-gnueabi-gcc; cd programs; $(MAKE) -e bins + cd lib; $(MAKE) -e all CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" + cd programs; $(MAKE) -e bins CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" streaming-examples: cd examples; $(MAKE) -e test From 1853622360cdd1adf4786f00dc18412e93a352fd Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 8 Apr 2015 10:35:38 +0100 Subject: [PATCH 036/121] fixed over-cautious visual warning --- programs/lz4cli.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index 2870c97..47cf7e0 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -473,7 +473,7 @@ int main(int argc, char** argv) if (bench) { int bmkResult = BMK_benchFiles(inFileNames, ifnIdx, cLevel); - free(inFileNames); + free((void*)inFileNames); return bmkResult; } From 2f8a4c32f998e6440a0b580996ecf8f101df2c74 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 9 Apr 2015 13:34:38 +0100 Subject: [PATCH 037/121] New LZ4_compress_safe() API --- lib/lz4.c | 39 +++++++++++-------------- lib/lz4.h | 85 ++++++++++++++++++++++++------------------------------- 2 files changed, 53 insertions(+), 71 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index a06f711..9d7e5b6 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -739,14 +739,14 @@ _last_literals: } -int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) { MEM_INIT(state, 0, LZ4_STREAMSIZE); if (maxOutputSize >= LZ4_compressBound(inputSize)) { if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); else return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); } @@ -759,12 +759,7 @@ int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* } } -int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize) -{ - return LZ4_compress_limitedOutput_withState(state, source, dest, inputSize, LZ4_compressBound(inputSize)); -} - -int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_safe(const char* source, char* dest, int inputSize, int maxOutputSize) { #if (HEAPMODE) void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* malloc-calloc aligned on 8-bytes boundaries */ @@ -772,7 +767,7 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in U64 ctx[LZ4_STREAMSIZE_U64] = {0}; /* Ensure data is aligned on 8-bytes boundaries */ #endif - int result = LZ4_compress_limitedOutput_withState(ctx, source, dest, inputSize, maxOutputSize); + int result = LZ4_compress_safe_extState(ctx, source, dest, inputSize, maxOutputSize); #if (HEAPMODE) FREEMEM(ctx); @@ -780,11 +775,6 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in return result; } -int LZ4_compress(const char* source, char* dest, int inputSize) -{ - return LZ4_compress_limitedOutput(source, dest, inputSize, LZ4_compressBound(inputSize)); -} - int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) { @@ -798,14 +788,14 @@ int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutp if (acceleration == 0) { if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); else result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, ACCELERATION_DEFAULT); } else { if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); + result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); else result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); } @@ -899,7 +889,7 @@ static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src) } -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) { LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_stream; const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; @@ -948,13 +938,8 @@ int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* s } } -int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) -{ - return LZ4_compress_limitedOutput_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize)); -} - -/* Hidden debug function, to force separate dictionary mode */ +/* Hidden debug function, to force external dictionary mode */ int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize) { LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_dict; @@ -1347,6 +1332,14 @@ int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compres /*************************************************** * Obsolete Functions ***************************************************/ +/* obsolete compression functions */ +int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_safe(source, dest, inputSize, maxOutputSize); } +int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_safe(source, dest, inputSize, LZ4_compressBound(inputSize)); } +int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, dstSize); } +int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize)); } +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_safe_continue(LZ4_stream, source, dest, inputSize, maxOutputSize); } +int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_safe_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize)); } + /* These function names are deprecated and should no longer be used. They are only provided here for compatibility with older user programs. diff --git a/lib/lz4.h b/lib/lz4.h index 81a11c9..659866b 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -70,18 +70,24 @@ int LZ4_versionNumber (void); * Simple Functions **************************************/ -int LZ4_compress (const char* source, char* dest, int sourceSize); +int LZ4_compress_safe (const char* source, char* dest, int sourceSize, int maxOutputSize); int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize); /* -LZ4_compress() : - Compresses 'sourceSize' bytes from 'source' into 'dest'. - Destination buffer must be already allocated, - and must be sized to handle worst cases situations (input data not compressible) - Worst case size evaluation is provided by function LZ4_compressBound() - inputSize : Max supported value is LZ4_MAX_INPUT_SIZE - return : the number of bytes written in buffer dest - or 0 if the compression fails +LZ4_compress_limitedOutput() : + Compresses 'sourceSize' bytes from buffer 'source' + into already allocated 'dest' of size 'maxOutputSize'. + Compression runs faster when 'maxOutputSize' >= LZ4_compressBound(sourceSize). + That's because in such case, it is guaranteed to compress within 'dest' budget, even in worst case scenario. + Compressing into a more limited space budget requires additional checks. + If the function cannot compress 'source' into a limited 'dest' budget, + compression stops *immediately*, and result of the function is zero. + It greatly accelerates behavior on non-compressible input, but as a consequence, 'dest' content is not valid either. + This function never writes outside 'dest' buffer, nor read outside 'source' buffer. + sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE + maxOutputSize : full or partial size of buffer 'dest' (which must be already allocated) + return : the number of bytes written into buffer 'dest' (necessarily <= maxOutputSize) + or 0 if compression fails LZ4_decompress_safe() : compressedSize : is obviously the source size @@ -89,9 +95,8 @@ LZ4_decompress_safe() : return : the number of bytes decompressed into the destination buffer (necessarily <= maxDecompressedSize) If the destination buffer is not large enough, decoding will stop and output an error code (<0). If the source stream is detected malformed, the function will stop decoding and return a negative result. - This function is protected against buffer overflow exploits, - and never writes outside of output buffer, nor reads outside of input buffer. - It is also protected against malicious data packets. + This function is protected against buffer overflow exploits, including malicious data packets. + It never writes outside of output buffer, nor reads outside of input buffer. */ @@ -113,22 +118,6 @@ LZ4_compressBound() : */ int LZ4_compressBound(int inputSize); - -/* -LZ4_compress_limitedOutput() : - Compress 'sourceSize' bytes from 'source' into an output buffer 'dest' of maximum size 'maxOutputSize'. - If it cannot achieve it, compression will stop, and result of the function will be zero. - This saves time and memory on detecting non-compressible (or barely compressible) data. - This function never writes outside of provided output buffer. - - sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE - maxOutputSize : is the size of the destination buffer (which must be already allocated) - return : the number of bytes written in buffer 'dest' - or 0 if compression fails -*/ -int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); - - /* LZ4_compress_fast() : Same as LZ4_compress_limitedOutput, but allows to select an "acceleration" factor. @@ -138,17 +127,15 @@ LZ4_compress_fast() : */ int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); - /* -LZ4_compress_withState() : - Same compression functions, but using an externally allocated memory space to store compression state. +LZ4_compress_safe_withState() : + Same compression function, just using an externally allocated memory space to store compression state. Use LZ4_sizeofState() to know how much memory must be allocated, and then, provide it as 'void* state' to compression functions. - Note that 'state' must be aligned on 4-bytes boundaries. + Note that 'state' ptr must be aligned on 4-bytes boundaries. */ int LZ4_sizeofState(void); -int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); -int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); /* @@ -164,7 +151,6 @@ LZ4_decompress_fast() : */ int LZ4_decompress_fast (const char* source, char* dest, int originalSize); - /* LZ4_decompress_safe_partial() : This function decompress a compressed block of size 'compressedSize' at position 'source' @@ -220,19 +206,14 @@ int LZ4_freeStream (LZ4_stream_t* LZ4_streamPtr); int LZ4_loadDict (LZ4_stream_t* LZ4_streamPtr, const char* dictionary, int dictSize); /* - * LZ4_compress_continue - * Compress data block 'source', using blocks compressed before as dictionary to improve compression ratio - * Previous data blocks are assumed to still be present at their previous location. - * dest buffer must be already allocated, and sized to at least LZ4_compressBound(inputSize) + * LZ4_compress_safe_continue + * Compress data block 'source', using data from previous blocks to improve compression ratio. + * Important : Previous data blocks are assumed to still be present and unmodified ! + * dest buffer must be already allocated. + * if maxOutpuSize >= (inputSize), compression is guaranteed to succeed. + * if not, and if target size objective cannot be met, compression stops, and function returns a zero. */ -int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); - -/* - * LZ4_compress_limitedOutput_continue - * Same as before, but also specify a maximum target compressed size (maxOutputSize) - * If objective cannot be met, compression exits, and returns a zero. - */ -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /* * LZ4_saveDict @@ -299,6 +280,15 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS /************************************** * Obsolete Functions **************************************/ + +/* Obsolete compression functions */ +int LZ4_compress (const char* source, char* dest, int sourceSize); +int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); +int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); +int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); + /* Obsolete decompression functions These function names are deprecated and should no longer be used. @@ -310,7 +300,6 @@ It is highly recommended to stop using these functions and migrate to newer ones /* int LZ4_uncompress (const char* source, char* dest, int outputSize); */ /* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */ - /* Obsolete streaming functions; use new streaming interface whenever possible */ void* LZ4_create (const char* inputBuffer); int LZ4_sizeofStreamState(void); From f344fbd3ca1a0a32668737e79bfdb06f3fadbba7 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 9 Apr 2015 22:59:07 +0100 Subject: [PATCH 038/121] Fixed a few warnings from -fsanitize=undefined --- lib/lz4.c | 132 ++++++++++------------------------------------ lib/lz4frame.c | 17 +++--- lib/xxhash.c | 41 +++++--------- programs/bench.c | 6 +-- programs/lz4cli.c | 10 ++-- programs/lz4io.c | 9 +++- 6 files changed, 67 insertions(+), 148 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 9d7e5b6..6ed6ab3 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -34,7 +34,7 @@ /************************************** - Tuning parameters +* Tuning parameters **************************************/ /* * HEAPMODE : @@ -49,51 +49,10 @@ */ #define ACCELERATION_DEFAULT 17 -/* - * CPU_HAS_EFFICIENT_UNALIGNED_MEMORY_ACCESS : - * By default, the source code expects the compiler to correctly optimize - * 4-bytes and 8-bytes read on architectures able to handle it efficiently. - * This is not always the case. In some circumstances (ARM notably), - * the compiler will issue cautious code even when target is able to correctly handle unaligned memory accesses. - * - * You can force the compiler to use unaligned memory access by uncommenting the line below. - * One of the below scenarios will happen : - * 1 - Your target CPU correctly handle unaligned access, and was not well optimized by compiler (good case). - * You will witness large performance improvements (+50% and up). - * Keep the line uncommented and send a word to upstream (https://groups.google.com/forum/#!forum/lz4c) - * The goal is to automatically detect such situations by adding your target CPU within an exception list. - * 2 - Your target CPU correctly handle unaligned access, and was already already optimized by compiler - * No change will be experienced. - * 3 - Your target CPU inefficiently handle unaligned access. - * You will experience a performance loss. Comment back the line. - * 4 - Your target CPU does not handle unaligned access. - * Program will crash. - * If uncommenting results in better performance (case 1) - * please report your configuration to upstream (https://groups.google.com/forum/#!forum/lz4c) - * This way, an automatic detection macro can be added to match your case within later versions of the library. - */ -/* #define CPU_HAS_EFFICIENT_UNALIGNED_MEMORY_ACCESS 1 */ - /************************************** - CPU Feature Detection +* CPU Feature Detection **************************************/ -/* - * Automated efficient unaligned memory access detection - * Based on known hardware architectures - * This list will be updated thanks to feedbacks - */ -#if defined(CPU_HAS_EFFICIENT_UNALIGNED_MEMORY_ACCESS) \ - || defined(__ARM_FEATURE_UNALIGNED) \ - || defined(__i386__) || defined(__x86_64__) \ - || defined(_M_IX86) || defined(_M_X64) \ - || defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_8__) \ - || (defined(_M_ARM) && (_M_ARM >= 7)) -# define LZ4_UNALIGNED_ACCESS 1 -#else -# define LZ4_UNALIGNED_ACCESS 0 -#endif - /* * LZ4_FORCE_SW_BITCOUNT * Define this parameter if your target system or compiler does not support hardware bit count @@ -142,7 +101,7 @@ /************************************** - Memory routines +* Memory routines **************************************/ #include /* malloc, calloc, free */ #define ALLOCATOR(n,s) calloc(n,s) @@ -152,13 +111,13 @@ /************************************** - Includes +* Includes **************************************/ #include "lz4.h" /************************************** - Basic Types +* Basic Types **************************************/ #if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ # include @@ -177,7 +136,7 @@ /************************************** - Reading and writing into memory +* Reading and writing into memory **************************************/ #define STEPSIZE sizeof(size_t) @@ -190,10 +149,19 @@ static unsigned LZ4_isLittleEndian(void) } +static U16 LZ4_read16(const void* memPtr) +{ + U16 val16; + memcpy(&val16, memPtr, 2); + return val16; +} + static U16 LZ4_readLE16(const void* memPtr) { - if ((LZ4_UNALIGNED_ACCESS) && (LZ4_isLittleEndian())) - return *(U16*)memPtr; + if (LZ4_isLittleEndian()) + { + return LZ4_read16(memPtr); + } else { const BYTE* p = (const BYTE*)memPtr; @@ -203,10 +171,9 @@ static U16 LZ4_readLE16(const void* memPtr) static void LZ4_writeLE16(void* memPtr, U16 value) { - if ((LZ4_UNALIGNED_ACCESS) && (LZ4_isLittleEndian())) + if (LZ4_isLittleEndian()) { - *(U16*)memPtr = value; - return; + memcpy(memPtr, &value, 2); } else { @@ -216,41 +183,18 @@ static void LZ4_writeLE16(void* memPtr, U16 value) } } - -static U16 LZ4_read16(const void* memPtr) -{ - if (LZ4_UNALIGNED_ACCESS) - return *(U16*)memPtr; - else - { - U16 val16; - memcpy(&val16, memPtr, 2); - return val16; - } -} - static U32 LZ4_read32(const void* memPtr) { - if (LZ4_UNALIGNED_ACCESS) - return *(U32*)memPtr; - else - { - U32 val32; - memcpy(&val32, memPtr, 4); - return val32; - } + U32 val32; + memcpy(&val32, memPtr, 4); + return val32; } static U64 LZ4_read64(const void* memPtr) { - if (LZ4_UNALIGNED_ACCESS) - return *(U64*)memPtr; - else - { - U64 val64; - memcpy(&val64, memPtr, 8); - return val64; - } + U64 val64; + memcpy(&val64, memPtr, 8); + return val64; } static size_t LZ4_read_ARCH(const void* p) @@ -262,31 +206,9 @@ static size_t LZ4_read_ARCH(const void* p) } -static void LZ4_copy4(void* dstPtr, const void* srcPtr) -{ - if (LZ4_UNALIGNED_ACCESS) - { - *(U32*)dstPtr = *(U32*)srcPtr; - return; - } - memcpy(dstPtr, srcPtr, 4); -} +static void LZ4_copy4(void* dstPtr, const void* srcPtr) { memcpy(dstPtr, srcPtr, 4); } -static void LZ4_copy8(void* dstPtr, const void* srcPtr) -{ -#if GCC_VERSION!=409 /* disabled on GCC 4.9, as it generates invalid opcode (crash) */ - if (LZ4_UNALIGNED_ACCESS) - { - if (LZ4_64bits()) - *(U64*)dstPtr = *(U64*)srcPtr; - else - ((U32*)dstPtr)[0] = ((U32*)srcPtr)[0], - ((U32*)dstPtr)[1] = ((U32*)srcPtr)[1]; - return; - } -#endif - memcpy(dstPtr, srcPtr, 8); -} +static void LZ4_copy8(void* dstPtr, const void* srcPtr) { memcpy(dstPtr, srcPtr, 8); } /* customized version of memcpy, which may overwrite up to 7 bytes beyond dstEnd */ static void LZ4_wildCopy(void* dstPtr, const void* srcPtr, void* dstEnd) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 6c65d0b..b6dbd20 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -130,6 +130,7 @@ typedef struct LZ4F_frameInfo_t frameInfo; U32 version; U32 dStage; + U64 frameRemainingSize; size_t maxBlockSize; size_t maxBufferSize; const BYTE* srcExpect; @@ -187,7 +188,7 @@ static U32 LZ4F_readLE32 (const BYTE* srcPtr) U32 value32 = srcPtr[0]; value32 += (srcPtr[1]<<8); value32 += (srcPtr[2]<<16); - value32 += (srcPtr[3]<<24); + value32 += ((U32)srcPtr[3])<<24; return value32; } @@ -302,7 +303,7 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf prefs.frameInfo.contentSize = (U64)srcSize; } if (prefs.frameInfo.contentSize != 0) - prefs.frameInfo.contentSize = (U64)srcSize; /* correct content size if selected (!=0) */ + prefs.frameInfo.contentSize = (U64)srcSize; /* auto-correct content size if selected (!=0) */ if (prefs.compressionLevel < minHClevel) { @@ -871,7 +872,7 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo dctxPtr->frameInfo.blockSizeID = (blockSizeID_t)blockSizeID; dctxPtr->maxBlockSize = LZ4F_getBlockSize(blockSizeID); if (contentSizeFlag) - dctxPtr->frameInfo.contentSize = LZ4F_readLE64(srcPtr+6); + dctxPtr->frameRemainingSize = dctxPtr->frameInfo.contentSize = LZ4F_readLE64(srcPtr+6); /* init */ if (contentChecksumFlag) XXH32_reset(&(dctxPtr->xxh), 0); @@ -1158,7 +1159,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, if ((size_t)(dstEnd-dstPtr) < sizeToCopy) sizeToCopy = dstEnd - dstPtr; memcpy(dstPtr, srcPtr, sizeToCopy); if (dctxPtr->frameInfo.contentChecksumFlag) XXH32_update(&(dctxPtr->xxh), srcPtr, sizeToCopy); - if (dctxPtr->frameInfo.contentSize) dctxPtr->frameInfo.contentSize -= sizeToCopy; + if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= sizeToCopy; /* dictionary management */ if (dctxPtr->frameInfo.blockMode==blockLinked) @@ -1231,7 +1232,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, decodedSize = decoder((const char*)selectedIn, (char*)dstPtr, (int)dctxPtr->tmpInTarget, (int)dctxPtr->maxBlockSize, (const char*)dctxPtr->dict, (int)dctxPtr->dictSize); if (decodedSize < 0) return (size_t)-ERROR_GENERIC; /* decompression failed */ if (dctxPtr->frameInfo.contentChecksumFlag) XXH32_update(&(dctxPtr->xxh), dstPtr, decodedSize); - if (dctxPtr->frameInfo.contentSize) dctxPtr->frameInfo.contentSize -= decodedSize; + if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= decodedSize; /* dictionary management */ if (dctxPtr->frameInfo.blockMode==blockLinked) @@ -1277,7 +1278,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, decodedSize = decoder((const char*)selectedIn, (char*)dctxPtr->tmpOut, (int)dctxPtr->tmpInTarget, (int)dctxPtr->maxBlockSize, (const char*)dctxPtr->dict, (int)dctxPtr->dictSize); if (decodedSize < 0) return (size_t)-ERROR_decompressionFailed; /* decompression failed */ if (dctxPtr->frameInfo.contentChecksumFlag) XXH32_update(&(dctxPtr->xxh), dctxPtr->tmpOut, decodedSize); - if (dctxPtr->frameInfo.contentSize) dctxPtr->frameInfo.contentSize -= decodedSize; + if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= decodedSize; dctxPtr->tmpOutSize = decodedSize; dctxPtr->tmpOutStart = 0; dctxPtr->dStage = dstage_flushOut; @@ -1311,7 +1312,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, case dstage_getSuffix: { size_t suffixSize = dctxPtr->frameInfo.contentChecksumFlag * 4; - if (dctxPtr->frameInfo.contentSize) return (size_t)-ERROR_frameSize_wrong; /* incorrect frame size decoded */ + if (dctxPtr->frameRemainingSize) return (size_t)-ERROR_frameSize_wrong; /* incorrect frame size decoded */ if (suffixSize == 0) /* frame completed */ { nextSrcSizeHint = 0; @@ -1392,7 +1393,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, selectedIn = dctxPtr->header + 4; } - /* case dstage_decodeSBlockSize: */ /* no direct access */ + /* case dstage_decodeSFrameSize: */ /* no direct access */ { size_t SFrameSize = LZ4F_readLE32(selectedIn); dctxPtr->frameInfo.contentSize = SFrameSize; diff --git a/lib/xxhash.c b/lib/xxhash.c index aca1e0a..a4a3fbe 100644 --- a/lib/xxhash.c +++ b/lib/xxhash.c @@ -117,35 +117,20 @@ typedef signed int S32; typedef unsigned long long U64; #endif -#if defined(__GNUC__) && !defined(XXH_USE_UNALIGNED_ACCESS) -# define _PACKED __attribute__ ((packed)) -#else -# define _PACKED -#endif - -#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__) -# ifdef __IBMC__ -# pragma pack(1) -# else -# pragma pack(push, 1) -# endif -#endif - -typedef struct _U32_S +static U32 XXH_read32(const void* memPtr) { - U32 v; -} _PACKED U32_S; -typedef struct _U64_S + U32 val32; + memcpy(&val32, memPtr, 4); + return val32; +} + +static U64 XXH_read64(const void* memPtr) { - U64 v; -} _PACKED U64_S; + U64 val64; + memcpy(&val64, memPtr, 8); + return val64; +} -#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__) -# pragma pack(pop) -#endif - -#define A32(x) (((U32_S *)(x))->v) -#define A64(x) (((U64_S *)(x))->v) /***************************************** @@ -230,7 +215,7 @@ typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align) { if (align==XXH_unaligned) - return endian==XXH_littleEndian ? A32(ptr) : XXH_swap32(A32(ptr)); + return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); else return endian==XXH_littleEndian ? *(U32*)ptr : XXH_swap32(*(U32*)ptr); } @@ -243,7 +228,7 @@ FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian) FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align) { if (align==XXH_unaligned) - return endian==XXH_littleEndian ? A64(ptr) : XXH_swap64(A64(ptr)); + return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); else return endian==XXH_littleEndian ? *(U64*)ptr : XXH_swap64(*(U64*)ptr); } diff --git a/programs/bench.c b/programs/bench.c index e1b5357..a5c72d6 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -403,7 +403,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) milliTime = BMK_GetMilliSpan(milliTime); if ((double)milliTime < fastestD*nbLoops) fastestD = (double)milliTime/nbLoops; - DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\r", loopNb, inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); + DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s \r", loopNb, inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); /* CRC Checking */ crcCheck = XXH32(orig_buff, (unsigned int)benchedSize,0); @@ -413,9 +413,9 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) if (crcOrig==crcCheck) { if (ratio<100.) - DISPLAY("%-16.16s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); + DISPLAY("%-16.16s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); else - DISPLAY("%-16.16s : %9i -> %9i (%5.1f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); + DISPLAY("%-16.16s : %9i -> %9i (%5.1f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); } totals += benchedSize; totalz += cSize; diff --git a/programs/lz4cli.c b/programs/lz4cli.c index 47cf7e0..f33ea82 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -420,11 +420,15 @@ int main(int argc, char** argv) /* Modify Nb Iterations (benchmark only) */ case 'i': - if ((argument[1] >='1') && (argument[1] <='9')) { - int iters = argument[1] - '0'; + unsigned iters = 0; + while ((argument[1] >='0') && (argument[1] <='9')) + { + iters *= 10; + iters += argument[1] - '0'; + argument++; + } BMK_setNbIterations(iters); - argument++; } break; diff --git a/programs/lz4io.c b/programs/lz4io.c index 991d9d7..278b766 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -487,8 +487,15 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena /* Final Status */ end = clock(); DISPLAYLEVEL(2, "\r%79s\r", ""); - DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", + if (filesize == 0) + { + DISPLAYLEVEL(2, "Null size input; converted into %u lz4 stream\n", (unsigned)compressedfilesize); + } + else + { + DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); + } { double seconds = (double)(end - start)/CLOCKS_PER_SEC; DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); From 33134fb6c8aff71375f2a629d4d9856a28d930d6 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 9 Apr 2015 23:53:55 +0100 Subject: [PATCH 039/121] Added : sanitize test --- .travis.yml | 1 + Makefile | 3 +++ 2 files changed, 4 insertions(+) diff --git a/.travis.yml b/.travis.yml index 8f45f71..398a0f1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,6 +14,7 @@ env: - LZ4_TRAVIS_CI_ENV=streaming-examples - LZ4_TRAVIS_CI_ENV=cmake - LZ4_TRAVIS_CI_ENV=clangtest + - LZ4_TRAVIS_CI_ENV=sanitize - LZ4_TRAVIS_CI_ENV=staticAnalyze - LZ4_TRAVIS_CI_ENV=gpptest - LZ4_TRAVIS_CI_ENV=armtest diff --git a/Makefile b/Makefile index 831f0d3..ca1f525 100644 --- a/Makefile +++ b/Makefile @@ -103,6 +103,9 @@ gpptest: clean clangtest: clean $(MAKE) all CC=clang CFLAGS="-O3 -Werror" +sanitize: clean + $(MAKE) test CC=clang CFLAGS="-O3 -g -fsanitize=undefined" + staticAnalyze: clean scan-build -v $(MAKE) all CFLAGS=-g From a2864fd49d801fb8a132fada94d319f96ad26e96 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 10 Apr 2015 14:24:13 +0100 Subject: [PATCH 040/121] Fixed a few minor sanitize warnings --- programs/Makefile | 6 +++--- programs/lz4io.c | 5 ++--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index 9585c9f..a394d4b 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -197,11 +197,11 @@ test-lz4: lz4 datagen test-lz4-sparse test-lz4-contentSize test-lz4-frame-concat ./datagen -g16KB | ./lz4 -9 | ./lz4 -t ./datagen | ./lz4 | ./lz4 -t ./datagen -g6M -P99 | ./lz4 -9BD | ./lz4 -t - ./datagen -g17M | ./lz4 -9v | ./lz4 -tq + ./datagen -g17M | ./lz4 -9v | ./lz4 -qt ./datagen -g33M | ./lz4 --no-frame-crc | ./lz4 -t ./datagen -g256MB | ./lz4 -vqB4D | ./lz4 -t - ./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -t - ./datagen -g6GB | ./lz4 -vq9BD | ./lz4 -t + ./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -qt + ./datagen -g6GB | ./lz4 -vq9BD | ./lz4 -qt @echo ---- test multiple input files ---- @./datagen -s1 > file1 @./datagen -s2 > file2 diff --git a/programs/lz4io.c b/programs/lz4io.c index 278b766..4071b1e 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -493,11 +493,9 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena } else { + double seconds = (double)(end - start)/CLOCKS_PER_SEC; DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); - } - { - double seconds = (double)(end - start)/CLOCKS_PER_SEC; DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); } @@ -825,6 +823,7 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file end = clock(); DISPLAYLEVEL(2, "\r%79s\r", ""); DISPLAYLEVEL(2, "Successfully decoded %llu bytes \n", filesize); + if (filesize > 0) { double seconds = (double)(end - start)/CLOCKS_PER_SEC; DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); From c22a0e1e1387a9a70b47be4ff5dbc67bbab7da16 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 10 Apr 2015 23:42:17 +0100 Subject: [PATCH 041/121] Updated : fuzzer tests can be programmed for a timelength --- Makefile | 2 +- programs/Makefile | 32 +++++++++--------- programs/frametest.c | 50 ++++++++++++++++++++++++---- programs/fuzzer.c | 77 +++++++++++++++++++++++++++++++------------- 4 files changed, 114 insertions(+), 47 deletions(-) diff --git a/Makefile b/Makefile index ca1f525..32f38aa 100644 --- a/Makefile +++ b/Makefile @@ -104,7 +104,7 @@ clangtest: clean $(MAKE) all CC=clang CFLAGS="-O3 -Werror" sanitize: clean - $(MAKE) test CC=clang CFLAGS="-O3 -g -fsanitize=undefined" + $(MAKE) test CC=clang CFLAGS="-O3 -g -fsanitize=undefined" FUZZER_TIME="-T5mn" staticAnalyze: clean scan-build -v $(MAKE) all CFLAGS=-g diff --git a/programs/Makefile b/programs/Makefile index a394d4b..1c9e4b5 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -40,14 +40,11 @@ DESTDIR?= PREFIX ?= /usr/local CFLAGS ?= -O3 CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\" -FLAGS = -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) +FLAGS := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) -BINDIR=$(PREFIX)/bin -MANDIR=$(PREFIX)/share/man/man1 -LZ4DIR=../lib - -TEST_FILES = COPYING -TEST_TARGETS=test-native +BINDIR := $(PREFIX)/bin +MANDIR := $(PREFIX)/share/man/man1 +LZ4DIR := ../lib # Define *.exe as extension for Windows systems @@ -61,7 +58,10 @@ endif # Select test target for Travis CI's Build Matrix -TRAVIS_TARGET=$(LZ4_TRAVIS_CI_ENV) +TRAVIS_TARGET:= $(LZ4_TRAVIS_CI_ENV) +TEST_FILES := COPYING +TEST_TARGETS := test-native +FUZZER_TIME ?= -T20mn default: lz4 @@ -220,10 +220,10 @@ test-lz4c32: lz4 lz4c32 datagen ./datagen -g16KB | ./lz4c32 -9 | ./lz4 -t ./datagen | ./lz4c32 | ./lz4c32 -t ./datagen | ./lz4c32 | ./lz4 -t - ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4c32 -t - ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4 -t - ./datagen -g6GB | ./lz4c32 -vqB5D | ./lz4c32 -t - ./datagen -g6GB | ./lz4c32 -vq9BD | ./lz4 -t + ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4c32 -qt + ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4 -qt + ./datagen -g6GB | ./lz4c32 -vqB5D | ./lz4c32 -qt + ./datagen -g6GB | ./lz4c32 -vq9BD | ./lz4 -qt test-fullbench: fullbench ./fullbench --no-prompt $(TEST_FILES) @@ -232,16 +232,16 @@ test-fullbench32: fullbench32 ./fullbench32 --no-prompt $(TEST_FILES) test-fuzzer: fuzzer - ./fuzzer + ./fuzzer $(FUZZER_TIME) test-fuzzer32: fuzzer32 - ./fuzzer32 + ./fuzzer32 $(FUZZER_TIME) test-frametest: frametest - ./frametest + ./frametest $(FUZZER_TIME) test-frametest32: frametest32 - ./frametest32 + ./frametest32 $(FUZZER_TIME) test-mem: lz4 datagen fuzzer frametest fullbench valgrind --leak-check=yes ./datagen -g50M > $(VOID) diff --git a/programs/frametest.c b/programs/frametest.c index 4beee23..5fccd08 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -537,7 +537,7 @@ static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, un static const U32 srcDataLength = 9 MB; /* needs to be > 2x4MB to test large blocks */ -int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility) +int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility, U32 duration) { unsigned testResult = 0; unsigned testNb = 0; @@ -548,10 +548,15 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi LZ4F_decompressionContext_t dCtx = NULL; LZ4F_compressionContext_t cCtx = NULL; size_t result; + const U32 startTime = FUZ_GetMilliStart(); XXH64_state_t xxh64; # define CHECK(cond, ...) if (cond) { DISPLAY("Error => "); DISPLAY(__VA_ARGS__); \ DISPLAY(" (seed %u, test nb %u) \n", seed, testNb); goto _output_error; } + + /* Init */ + duration *= 1000; + /* Create buffers */ result = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result); @@ -566,10 +571,10 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi FUZ_fillCompressibleNoiseBuffer(srcBuffer, srcDataLength, compressibility, &coreRand); /* jump to requested testNb */ - for (testNb =0; testNb < startTest; testNb++) (void)FUZ_rand(&coreRand); // sync randomizer + for (testNb =0; (testNb < startTest); testNb++) (void)FUZ_rand(&coreRand); // sync randomizer /* main fuzzer test loop */ - for ( ; testNb < nbTests; testNb++) + for ( ; (testNb < nbTests) || (duration > FUZ_GetMilliSpan(startTime)) ; testNb++) { U32 randState = coreRand ^ prime1; unsigned BSId = 4 + (FUZ_rand(&randState) & 3); @@ -721,6 +726,7 @@ int FUZ_usage(void) DISPLAY( "\n"); DISPLAY( "Arguments :\n"); DISPLAY( " -i# : Nb of tests (default:%u) \n", nbTestsDefault); + DISPLAY( " -T# : Duration of tests (default: use Nb of tests) \n"); DISPLAY( " -s# : Select seed (default:prompt user)\n"); DISPLAY( " -t# : Select starting test number (default:0)\n"); DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); @@ -739,6 +745,7 @@ int main(int argc, char** argv) int testNb = 0; int proba = FUZ_COMPRESSIBILITY_DEFAULT; int result=0; + U32 duration=0; /* Check command line */ programName = argv[0]; @@ -746,9 +753,9 @@ int main(int argc, char** argv) { char* argument = argv[argNb]; - if(!argument) continue; // Protection if argument empty + if(!argument) continue; /* Protection if argument empty */ - // Decode command (note : aggregated commands are allowed) + /* Decode command (note : aggregated commands are allowed) */ if (argument[0]=='-') { if (!strcmp(argument, "--no-prompt")) @@ -781,7 +788,7 @@ int main(int argc, char** argv) case 'i': argument++; - nbTests=0; + nbTests=0; duration=0; while ((*argument>='0') && (*argument<='9')) { nbTests *= 10; @@ -789,6 +796,35 @@ int main(int argc, char** argv) argument++; } break; + + case 'T': + argument++; + nbTests = 0; duration = 0; + + for (;;) + { + if (argument[0]=='m') + { + duration *= 60; + argument++; + continue; + } + if (argument[0]=='n') + { + argument++; + continue; + } + if ((*argument>='0') && (*argument<='9')) + { + duration *= 10; + duration += *argument - '0'; + argument++; + continue; + } + break; + } + break; + case 's': argument++; seed=0; @@ -841,5 +877,5 @@ int main(int argc, char** argv) if (testNb==0) result = basicTests(seed, ((double)proba) / 100); if (result) return 1; - return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100); + return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100, duration); } diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 3d3cf8e..9e2ce1e 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -19,7 +19,6 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : http://code.google.com/p/lz4 - LZ4 source mirror : https://github.com/Cyan4973/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ @@ -210,7 +209,7 @@ static int FUZ_AddressOverflow(void) printf("Overflow tests : "); - // Only possible in 32-bits + /* Only possible in 32-bits */ if (sizeof(void*)==8) { printf("64 bits mode : no overflow \n"); @@ -300,7 +299,7 @@ static void FUZ_displayUpdate(unsigned testNb) } -static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const double compressibility) +static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double compressibility, U32 duration) { unsigned long long bytes = 0; unsigned long long cbytes = 0; @@ -313,7 +312,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do int ret; unsigned cycleNb; # define FUZ_CHECKTEST(cond, ...) if (cond) { printf("Test %u : ", testNb); printf(__VA_ARGS__); \ - printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; } + printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; } # define FUZ_DISPLAYTEST { testNb++; g_displayLevel<3 ? 0 : printf("%2u\b\b", testNb); if (g_displayLevel==4) fflush(stdout); } void* stateLZ4 = malloc(LZ4_sizeofState()); void* stateLZ4HC = malloc(LZ4_sizeofStateHC()); @@ -324,23 +323,25 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do U32 coreRandState = seed; U32 randState = coreRandState ^ PRIME3; int result = 0; + const U32 startTime = FUZ_GetMilliStart(); - // init + /* init */ memset(&LZ4dict, 0, sizeof(LZ4dict)); + duration *= 1000; - // Create compressible test buffer + /* Create compressible test buffer */ CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); compressedBuffer = (char*)malloc(LZ4_compressBound(FUZ_MAX_BLOCK_SIZE)); decodedBuffer = (char*)malloc(FUZ_MAX_DICT_SIZE + FUZ_MAX_BLOCK_SIZE); - // move to startCycle + /* move to startCycle */ for (cycleNb = 0; cycleNb < startCycle; cycleNb++) { (void)FUZ_rand(&coreRandState); - if (0) // some problems related to dictionary re-use; in this case, enable this loop + if (0) /* some problems are related to dictionary re-use; in this case, enable this loop */ { int dictSize, blockSize, blockStart; char* dict; @@ -362,8 +363,8 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do } } - // Test loop - for (cycleNb = startCycle; cycleNb < nbCycles; cycleNb++) + /* Main test loop */ + for (cycleNb = startCycle; (cycleNb < nbCycles) || (FUZ_GetMilliSpan(startTime) < duration) ; cycleNb++) { U32 testNb = 0; char* dict; @@ -375,7 +376,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do (void)FUZ_rand(&coreRandState); randState = coreRandState ^ PRIME3; - // Select block to test + /* Select block to test */ blockSize = FUZ_rand(&randState) % FUZ_MAX_BLOCK_SIZE; blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize); dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE; @@ -385,7 +386,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do /* Compression tests */ - // Test compression HC + /* Test compression HC */ FUZ_DISPLAYTEST; ret = LZ4_compressHC(block, compressedBuffer, blockSize); FUZ_CHECKTEST(ret==0, "LZ4_compressHC() failed"); @@ -673,21 +674,22 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); - // ***** End of tests *** // - // Fill stats + /* ***** End of tests *** */ + /* Fill stats */ bytes += blockSize; cbytes += compressedSize; hcbytes += HCcompressedSize; ccbytes += blockContinueCompressedSize; } + if (nbCycles<=1) nbCycles = cycleNb; printf("\r%7u /%7u - ", cycleNb, nbCycles); printf("all tests completed successfully \n"); printf("compression ratio: %0.3f%%\n", (double)cbytes/bytes*100); printf("HC compression ratio: %0.3f%%\n", (double)hcbytes/bytes*100); printf("ratio with dict: %0.3f%%\n", (double)ccbytes/bytes*100); - // unalloc + /* release memory */ { _exit: free(CNBuffer); @@ -831,7 +833,7 @@ static void FUZ_unitTests(void) crcNew = XXH64(testVerify, testCompressedSize, 0); FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() simple dictionary decompression test : corruption"); - // multiple HC compression test with dictionary + /* multiple HC compression test with dictionary */ { int result1, result2; int segSize = testCompressedSize / 2; @@ -863,7 +865,7 @@ static void FUZ_unitTests(void) crcNew = XXH64(testVerify, testCompressedSize, 0); FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() decompression corruption"); - // multiple HC compression with ext. dictionary + /* multiple HC compression with ext. dictionary */ { XXH64_state_t crcOrigState; XXH64_state_t crcNewState; @@ -912,7 +914,7 @@ static void FUZ_unitTests(void) } } - // ring buffer test + /* ring buffer test */ { XXH64_state_t xxhOrig; XXH64_state_t xxhNew; @@ -956,7 +958,7 @@ static void FUZ_unitTests(void) } } - // small decoder-side ring buffer test + /* small decoder-side ring buffer test */ { XXH64_state_t xxhOrig; XXH64_state_t xxhNew; @@ -1013,6 +1015,7 @@ static int FUZ_usage(char* programName) DISPLAY( "\n"); DISPLAY( "Arguments :\n"); DISPLAY( " -i# : Nb of tests (default:%i) \n", NB_ATTEMPTS); + DISPLAY( " -T# : Duration of tests (default: use Nb of tests) \n"); DISPLAY( " -s# : Select seed (default:prompt user)\n"); DISPLAY( " -t# : Select starting test number (default:0)\n"); DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); @@ -1033,8 +1036,9 @@ int main(int argc, char** argv) int proba = FUZ_COMPRESSIBILITY_DEFAULT; int pause = 0; char* programName = argv[0]; + U32 duration = 0; - // Check command line + /* Check command line */ for(argNb=1; argNb='0') && (*argument<='9')) { nbTests *= 10; @@ -1075,6 +1079,34 @@ int main(int argc, char** argv) } break; + case 'T': + argument++; + nbTests = 0; duration = 0; + + for (;;) + { + if (argument[0]=='m') + { + duration *= 60; + argument++; + continue; + } + if (argument[0]=='n') + { + argument++; + continue; + } + if ((*argument>='0') && (*argument<='9')) + { + duration *= 10; + duration += *argument - '0'; + argument++; + continue; + } + break; + } + break; + case 's': argument++; seed=0; seedset=1; @@ -1115,7 +1147,6 @@ int main(int argc, char** argv) } } - // Get Seed printf("Starting LZ4 fuzzer (%i-bits, %s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION); if (!seedset) seed = FUZ_GetMilliStart() % 10000; @@ -1127,7 +1158,7 @@ int main(int argc, char** argv) if (nbTests<=0) nbTests=1; { - int result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100); + int result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100, duration); if (pause) { DISPLAY("press enter ... \n"); From 8b8e5efefe2dfdfbeb3ebc7591d2c345c34552f5 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 11 Apr 2015 10:25:08 +0100 Subject: [PATCH 042/121] fixed minor sanitize warning --- programs/Makefile | 3 +-- programs/lz4io.c | 6 +++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index 1c9e4b5..97be7d9 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -61,7 +61,6 @@ endif TRAVIS_TARGET:= $(LZ4_TRAVIS_CI_ENV) TEST_FILES := COPYING TEST_TARGETS := test-native -FUZZER_TIME ?= -T20mn default: lz4 @@ -103,7 +102,7 @@ datagen : datagen.c datagencli.c $(CC) $(FLAGS) $^ -o $@$(EXT) clean: - @rm -f core *.o *.test \ + @rm -f core *.o *.test tmp* \ lz4$(EXT) lz4c$(EXT) lz4c32$(EXT) \ fullbench$(EXT) fullbench32$(EXT) \ fuzzer$(EXT) fuzzer32$(EXT) \ diff --git a/programs/lz4io.c b/programs/lz4io.c index 4071b1e..b6e678e 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -53,8 +53,8 @@ #include /* stat64 */ #include /* stat64 */ #include "lz4io.h" -#include "lz4.h" /* still required for legacy format */ -#include "lz4hc.h" /* still required for legacy format */ +#include "lz4.h" /* still required for legacy format */ +#include "lz4hc.h" /* still required for legacy format */ #include "lz4frame.h" @@ -823,7 +823,7 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file end = clock(); DISPLAYLEVEL(2, "\r%79s\r", ""); DISPLAYLEVEL(2, "Successfully decoded %llu bytes \n", filesize); - if (filesize > 0) + if (end==start) end=start+1; { double seconds = (double)(end - start)/CLOCKS_PER_SEC; DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); From a07db74d24d8fb3f7a3dc64e4c7b1204f52226ad Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 11 Apr 2015 10:46:16 +0100 Subject: [PATCH 043/121] Clarified lz4frame.h inline doc --- lib/lz4frame.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 9d12c7d..e5435cd 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -45,7 +45,7 @@ extern "C" { #endif /************************************** - Includes +* Includes **************************************/ #include /* size_t */ @@ -142,15 +142,17 @@ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* prefsPtr); /* LZ4F_compressBound() : * Provides the minimum size of Dst buffer given srcSize to handle worst case situations. * prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to default. - * Note that different preferences will produce in different results. + * Note that different preferences will produce different results. + * This function doesn't include frame termination cost (4 bytes, or 8 is frame checksum is enabled) */ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* cOptPtr); /* LZ4F_compressUpdate() * LZ4F_compressUpdate() can be called repetitively to compress as much data as necessary. * The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case. - * If this condition is not respected, LZ4F_compress() will fail (result is an errorCode) - * You can get the minimum value of dstMaxSize by using LZ4F_compressBound() + * You can get the minimum value of dstMaxSize by using LZ4F_compressBound(). + * If this condition is not respected, LZ4F_compress() will fail (result is an errorCode). + * LZ4F_compressUpdate() doesn't guarantee error recovery, so you have to reset compression context when an error occurs. * The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. * The result of the function is the number of bytes written into dstBuffer : it can be zero, meaning input data was just buffered. * The function outputs an error code if it fails (can be tested using LZ4F_isError()) @@ -172,10 +174,10 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t * When you want to properly finish the compressed frame, just call LZ4F_compressEnd(). * It will flush whatever data remained within compressionContext (like LZ4_flush()) * but also properly finalize the frame, with an endMark and a checksum. - * The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark size)) + * The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark), or 8 if optional frame checksum is enabled) * The function outputs an error code if it fails (can be tested using LZ4F_isError()) * The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. - * A successful call to LZ4F_compressEnd() makes cctx available again for future compression work. + * A successful call to LZ4F_compressEnd() makes cctx available again for future compression task. */ From be9d248851ecd712a8911526b009c5d0c948ee21 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 11 Apr 2015 12:28:09 +0100 Subject: [PATCH 044/121] Update lz4hc API : LZ4_compressHC_safe() --- NEWS | 2 + lib/lz4hc.c | 86 +++++++++++----------------------- lib/lz4hc.h | 131 +++++++++++++++++++++++----------------------------- 3 files changed, 88 insertions(+), 131 deletions(-) diff --git a/NEWS b/NEWS index a1d452a..0f33388 100644 --- a/NEWS +++ b/NEWS @@ -1,6 +1,8 @@ r129: Added : LZ4_compress_fast() +Changed: New compression functions LZ4_compress_safe() - Suggested by Evan Nemerson & Takayuki Matsuoka Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka +Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Evan Nemerson and Greg Slazinski Updated: Documentation converted to MarkDown r128: diff --git a/lib/lz4hc.c b/lib/lz4hc.c index c7a94a0..234e1af 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -536,55 +536,24 @@ _Search3: } -int LZ4_compressHC2(const char* source, char* dest, int inputSize, int compressionLevel) -{ - LZ4HC_Data_Structure ctx; - LZ4HC_init(&ctx, (const BYTE*)source); - return LZ4HC_compress_generic (&ctx, source, dest, inputSize, 0, compressionLevel, noLimit); -} - -int LZ4_compressHC(const char* source, char* dest, int inputSize) { return LZ4_compressHC2(source, dest, inputSize, 0); } - -int LZ4_compressHC2_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) -{ - LZ4HC_Data_Structure ctx; - LZ4HC_init(&ctx, (const BYTE*)source); - return LZ4HC_compress_generic (&ctx, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput); -} - -int LZ4_compressHC_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) -{ - return LZ4_compressHC2_limitedOutput(source, dest, inputSize, maxOutputSize, 0); -} - - -/***************************** - * Using external allocation - * ***************************/ int LZ4_sizeofStateHC(void) { return sizeof(LZ4HC_Data_Structure); } - -int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel) +int LZ4_compressHC_safe_extStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) { if (((size_t)(state)&(sizeof(void*)-1)) != 0) return 0; /* Error : state is not aligned for pointers (32 or 64 bits) */ LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)source); - return LZ4HC_compress_generic (state, source, dest, inputSize, 0, compressionLevel, noLimit); + if (maxOutputSize < LZ4_compressBound(inputSize)) + return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput); + else + return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, noLimit); } -int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize) -{ return LZ4_compressHC2_withStateHC (state, source, dest, inputSize, 0); } - - -int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) +int LZ4_compressHC_safe(const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) { - if (((size_t)(state)&(sizeof(void*)-1)) != 0) return 0; /* Error : state is not aligned for pointers (32 or 64 bits) */ - LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)source); - return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput); + LZ4HC_Data_Structure state; + return LZ4_compressHC_safe_extStateHC(&state, source, dest, inputSize, maxOutputSize, compressionLevel); } -int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) -{ return LZ4_compressHC2_limitedOutput_withStateHC (state, source, dest, inputSize, maxOutputSize, 0); } - /************************************** @@ -592,7 +561,7 @@ int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, c * ************************************/ /* allocation */ LZ4_streamHC_t* LZ4_createStreamHC(void) { return (LZ4_streamHC_t*)malloc(sizeof(LZ4_streamHC_t)); } -int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { free(LZ4_streamHCPtr); return 0; } +int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { free(LZ4_streamHCPtr); return 0; } /* initialization */ @@ -669,14 +638,12 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr, return LZ4HC_compress_generic (ctxPtr, source, dest, inputSize, maxOutputSize, ctxPtr->compressionLevel, limit); } -int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize) +int LZ4_compressHC_safe_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize) { - return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, 0, noLimit); -} - -int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize) -{ - return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput); + if (maxOutputSize < LZ4_compressBound(inputSize)) + return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput); + else + return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, noLimit); } @@ -705,6 +672,20 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS /*********************************** * Deprecated Functions ***********************************/ +/* Deprecated compression functions */ +int LZ4_compressHC(const char* source, char* dest, int inputSize) { return LZ4_compressHC2(source, dest, inputSize, 0); } +int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, 0); } +int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } +int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, cLevel); } +int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); } +int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe_extStateHC (state, src, dst, srcSize, maxDstSize, 0); } +int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compressHC_safe_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } +int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compressHC_safe_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); } +int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); } +int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe_continue (ctx, src, dst, srcSize, maxDstSize); } + + +/* Deprecated streaming functions */ int LZ4_sizeofStreamStateHC(void) { return LZ4_STREAMHCSIZE; } int LZ4_resetStreamStateHC(void* state, const char* inputBuffer) @@ -727,17 +708,6 @@ int LZ4_freeHC (void* LZ4HC_Data) return (0); } -/* -int LZ4_compressHC_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize) -{ -return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, 0, 0, noLimit); -} -int LZ4_compressHC_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize) -{ -return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, maxOutputSize, 0, limitedOutput); -} -*/ - int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel) { return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, 0, compressionLevel, noLimit); diff --git a/lib/lz4hc.h b/lib/lz4hc.h index 4a05845..c257dcc 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -38,70 +38,6 @@ extern "C" { #endif - -int LZ4_compressHC (const char* source, char* dest, int inputSize); -/* -LZ4_compressHC : - return : the number of bytes in compressed buffer dest - or 0 if compression fails. - note : destination buffer must be already allocated. - To avoid any problem, size it to handle worst cases situations (input data not compressible) - Worst case size evaluation is provided by function LZ4_compressBound() (see "lz4.h") -*/ - -int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize); -/* -LZ4_compress_limitedOutput() : - Compress 'inputSize' bytes from 'source' into an output buffer 'dest' of maximum size 'maxOutputSize'. - If it cannot achieve it, compression will stop, and result of the function will be zero. - This function never writes outside of provided output buffer. - - inputSize : Max supported value is 1 GB - maxOutputSize : is maximum allowed size into the destination buffer (which must be already allocated) - return : the number of output bytes written in buffer 'dest' - or 0 if compression fails. -*/ - - -int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel); -int LZ4_compressHC2_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); -/* - Same functions as above, but with programmable 'compressionLevel'. - Recommended values are between 4 and 9, although any value between 0 and 16 will work. - 'compressionLevel'==0 means use default 'compressionLevel' value. - Values above 16 behave the same as 16. - Equivalent variants exist for all other compression functions below. -*/ - -/* Note : - Decompression functions are provided within LZ4 source code (see "lz4.h") (BSD license) -*/ - - -/************************************** -* Using an external allocation -**************************************/ -int LZ4_sizeofStateHC(void); -int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize); -int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); - -int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel); -int LZ4_compressHC2_limitedOutput_withStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); - -/* -These functions are provided should you prefer to allocate memory for compression tables with your own allocation methods. -To know how much memory must be allocated for the compression tables, use : -int LZ4_sizeofStateHC(); - -Note that tables must be aligned for pointer (32 or 64 bits), otherwise compression will fail (return code 0). - -The allocated memory can be provided to the compression functions using 'void* state' parameter. -LZ4_compress_withStateHC() and LZ4_compress_limitedOutput_withStateHC() are equivalent to previously described functions. -They just use the externally allocated memory for state instead of allocating their own (on stack, or on heap). -*/ - - - /***************************** * Includes *****************************/ @@ -109,7 +45,46 @@ They just use the externally allocated memory for state instead of allocating th /************************************** -* Experimental Streaming Functions +* Block Compression +**************************************/ +int LZ4_compressHC_safe (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +/* +LZ4_compressHC_safe : + return : the number of bytes in compressed buffer dest + or 0 if compression fails. + note : destination buffer must be already allocated. + To guarantee compression completion, size it to handle worst cases situations (data not compressible) + Worst case size evaluation is provided by function LZ4_compressBound() (see "lz4.h") + inputSize : Max supported value is LZ4_MAX_INPUT_SIZE (see "lz4.h") + compressionLevel : Recommended values are between 4 and 9, although any value between 0 and 16 will work. + 0 means use default 'compressionLevel' value. + Values >16 behave the same as 16. +*/ + + +/* Note : + Decompression functions are provided within LZ4 source code (see "lz4.h") (BSD license) +*/ + + +int LZ4_sizeofStateHC(void); +int LZ4_compressHC_safe_extStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); + +/* +This function is provided should you prefer to allocate memory for compression tables with your own allocation methods. +To know how much memory must be allocated for the compression tables, use : +int LZ4_sizeofStateHC(); + +Note that tables must be aligned for pointer (32 or 64 bits), otherwise compression will fail (return code 0). + +The allocated memory can be provided to the compression functions using 'void* state' parameter. +LZ4_compressHC_safe_extStateHC() is equivalent to previously described function. +It just uses externally allocated memory for stateHC instead of allocating their own (on stack, or on heap). +*/ + + +/************************************** +* Streaming Compression **************************************/ #define LZ4_STREAMHCSIZE 262192 #define LZ4_STREAMHCSIZE_SIZET (LZ4_STREAMHCSIZE / sizeof(size_t)) @@ -137,8 +112,7 @@ to avoid size mismatch between different versions. void LZ4_resetStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel); int LZ4_loadDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, const char* dictionary, int dictSize); -int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize); -int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compressHC_safe_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int maxDictSize); @@ -149,8 +123,8 @@ One key assumption is that each previous block will remain read-accessible while Before starting compression, state must be properly initialized, using LZ4_resetStreamHC(). A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional). -Then, use LZ4_compressHC_continue() or LZ4_compressHC_limitedOutput_continue() to compress each successive block. -They work like usual LZ4_compressHC() or LZ4_compressHC_limitedOutput(), but use previous memory blocks to improve compression. +Then, use LZ4_compressHC_safe_continue() to compress each successive block. +It works like LZ4_compressHC_safe(), but use previous memory blocks to improve compression. Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression. If, for any reason, previous data block can't be preserved in memory during next compression block, @@ -161,16 +135,27 @@ using LZ4_saveDictHC(). /************************************** - * Deprecated Streaming Functions + * Deprecated Functions * ************************************/ -/* Note : these streaming functions follows the older model, and should no longer be used */ + +/* compression functions */ +int LZ4_compressHC (const char* source, char* dest, int inputSize); +int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel); +int LZ4_compressHC2_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize); +int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel); +int LZ4_compressHC2_limitedOutput_withStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize); +int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); + +/* Streaming functions following the older model; should no longer be used */ void* LZ4_createHC (const char* inputBuffer); char* LZ4_slideInputBufferHC (void* LZ4HC_Data); int LZ4_freeHC (void* LZ4HC_Data); - int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); - int LZ4_sizeofStreamStateHC(void); int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); From 973e385fa332485f4617cf16406a8c8ef631a555 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 11 Apr 2015 18:59:22 +0100 Subject: [PATCH 045/121] Implemented obsolete warning message --- NEWS | 3 +- lib/lz4.c | 14 ++--- lib/lz4.h | 61 +++++++++++------- programs/fullbench.c | 146 +++++++++++++++++++++---------------------- programs/fuzzer.c | 17 ++--- 5 files changed, 125 insertions(+), 116 deletions(-) diff --git a/NEWS b/NEWS index 0f33388..462a23b 100644 --- a/NEWS +++ b/NEWS @@ -2,7 +2,8 @@ r129: Added : LZ4_compress_fast() Changed: New compression functions LZ4_compress_safe() - Suggested by Evan Nemerson & Takayuki Matsuoka Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka -Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Evan Nemerson and Greg Slazinski +Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Greg Slazinski, Evan Nemerson & Takayuki Matsuoka +Modified:Obsolete functions generate warnings - Suggested by Evan Nemerson & Takayuki Matsuoka Updated: Documentation converted to MarkDown r128: diff --git a/lib/lz4.c b/lib/lz4.c index 6ed6ab3..d6fdcd9 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -62,6 +62,12 @@ #endif +/************************************** +* Includes +**************************************/ +#include "lz4.h" + + /************************************** * Compiler Options **************************************/ @@ -88,8 +94,6 @@ # endif /* __STDC_VERSION__ */ #endif /* _MSC_VER */ -#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) - #if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) #else @@ -110,12 +114,6 @@ #define MEM_INIT memset -/************************************** -* Includes -**************************************/ -#include "lz4.h" - - /************************************** * Basic Types **************************************/ diff --git a/lib/lz4.h b/lib/lz4.h index 659866b..7bd4594 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -39,9 +39,9 @@ extern "C" { #endif /* - * lz4.h provides block compression functions, for optimal performance. + * lz4.h provides block compression functions, and gives full buffer control to programmer. * If you need to generate inter-operable compressed data (respecting LZ4 frame specification), - * please use lz4frame.h instead. + * and can let the library handle its own memory, please use lz4frame.h instead. */ /************************************** @@ -70,7 +70,7 @@ int LZ4_versionNumber (void); * Simple Functions **************************************/ -int LZ4_compress_safe (const char* source, char* dest, int sourceSize, int maxOutputSize); +int LZ4_compress_safe (const char* source, char* dest, int sourceSize, int maxDestSize); int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize); /* @@ -85,7 +85,7 @@ LZ4_compress_limitedOutput() : It greatly accelerates behavior on non-compressible input, but as a consequence, 'dest' content is not valid either. This function never writes outside 'dest' buffer, nor read outside 'source' buffer. sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE - maxOutputSize : full or partial size of buffer 'dest' (which must be already allocated) + maxDestSize : full or partial size of buffer 'dest' (which must be already allocated) return : the number of bytes written into buffer 'dest' (necessarily <= maxOutputSize) or 0 if compression fails @@ -124,11 +124,12 @@ LZ4_compress_fast() : The larger the value, the faster the algorithm, but also the lesser the compression. So it's a trade-off, which can be fine tuned, selecting whichever value you want. An acceleration value of "0" means "use Default value", which is typically about 15 (see lz4.c source code). + Note : this function is "safe", even if its name does not say it. It's just faster and compress less. */ int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); /* -LZ4_compress_safe_withState() : +LZ4_compress_safe_extState() : Same compression function, just using an externally allocated memory space to store compression state. Use LZ4_sizeofState() to know how much memory must be allocated, and then, provide it as 'void* state' to compression functions. @@ -169,7 +170,6 @@ int LZ4_decompress_safe_partial (const char* source, char* dest, int compressedS /*********************************************** * Streaming Compression Functions ***********************************************/ - #define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE-3)) + 4) #define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(long long)) /* @@ -280,35 +280,50 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS /************************************** * Obsolete Functions **************************************/ +/* Warning statements */ +#ifndef _WARNING_STATEMENT_BLOCK +# define _WARNING_STATEMENT_BLOCK +# define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) +# if (GCC_VERSION >= 405) || defined(__clang__) +# define DEPRECATED(message) __attribute__((deprecated(message))) +# elif (GCC_VERSION >= 301) +# define DEPRECATED(message) __attribute__((deprecated)) +# elif defined(_MSC_VER) +# define DEPRECATED(message) __declspec(deprecated(message)) +# else +# pragma message("WARNING: You need to implement DEPRECATED for this compiler") +# define DEPRECATED +# endif +#endif // _WARNING_STATEMENT_BLOCK /* Obsolete compression functions */ -int LZ4_compress (const char* source, char* dest, int sourceSize); +/* These functions are planned to generate warnings by r131 approximately */ +int LZ4_compress (const char* source, char* dest, int sourceSize); int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); -int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); -/* -Obsolete decompression functions -These function names are deprecated and should no longer be used. -They are only provided here for compatibility with older user programs. -- LZ4_uncompress is the same as LZ4_decompress_fast -- LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe -These function prototypes are now disabled; uncomment them if you really need them. -It is highly recommended to stop using these functions and migrate to newer ones */ +/* Obsolete decompression functions */ +/* These function names are completely deprecated and must no longer be used. + They are only provided here for compatibility with older programs. + - LZ4_uncompress is the same as LZ4_decompress_fast + - LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe + These function prototypes are now disabled; uncomment them only if you really need them. + It is highly recommended to stop using these prototypes and migrate to maintained ones */ /* int LZ4_uncompress (const char* source, char* dest, int outputSize); */ /* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */ /* Obsolete streaming functions; use new streaming interface whenever possible */ -void* LZ4_create (const char* inputBuffer); -int LZ4_sizeofStreamState(void); -int LZ4_resetStreamState(void* state, const char* inputBuffer); -char* LZ4_slideInputBuffer (void* state); +DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (const char* inputBuffer); +DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); +DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, const char* inputBuffer); +DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); /* Obsolete streaming decoding functions */ -int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize); -int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize); +DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize); +DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize); #if defined (__cplusplus) diff --git a/programs/fullbench.c b/programs/fullbench.c index 2f246a2..700655b 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -381,6 +381,23 @@ start: #endif // __SSSE3__ +static LZ4_stream_t LZ4_stream; +static void local_LZ4_resetDictT(void) +{ + LZ4_resetStream(&LZ4_stream); +} + +static void local_LZ4_createStream(void) +{ + LZ4_resetStream(&LZ4_stream); +} + +static int local_LZ4_saveDict(const char* in, char* out, int inSize) +{ + (void)in; + return LZ4_saveDict(&LZ4_stream, out, inSize); +} + static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSize) { return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); @@ -391,53 +408,56 @@ static int local_LZ4_compress_fast(const char* in, char* out, int inSize) return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); } -static LZ4_stream_t* stateLZ4; static int local_LZ4_compress_withState(const char* in, char* out, int inSize) { - return LZ4_compress_withState(stateLZ4, in, out, inSize); + return LZ4_compress_withState(&LZ4_stream, in, out, inSize); } static int local_LZ4_compress_limitedOutput_withState(const char* in, char* out, int inSize) { - return LZ4_compress_limitedOutput_withState(stateLZ4, in, out, inSize, LZ4_compressBound(inSize)-1); + return LZ4_compress_limitedOutput_withState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); } -static LZ4_stream_t* ctx; static int local_LZ4_compress_continue(const char* in, char* out, int inSize) { - return LZ4_compress_continue(ctx, in, out, inSize); + return LZ4_compress_continue(&LZ4_stream, in, out, inSize); } static int local_LZ4_compress_limitedOutput_continue(const char* in, char* out, int inSize) { - return LZ4_compress_limitedOutput_continue(ctx, in, out, inSize, LZ4_compressBound(inSize)-1); + return LZ4_compress_limitedOutput_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); } +/* declare hidden function */ +int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize); -LZ4_stream_t LZ4_dict; -static void* local_LZ4_resetDictT(const char* fake) -{ - (void)fake; - memset(&LZ4_dict, 0, sizeof(LZ4_stream_t)); - return NULL; -} - -int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize); static int local_LZ4_compress_forceDict(const char* in, char* out, int inSize) { - return LZ4_compress_forceExtDict(&LZ4_dict, in, out, inSize); + return LZ4_compress_forceExtDict(&LZ4_stream, in, out, inSize); } -static LZ4_streamHC_t* stateLZ4HC; +/* HC compression functions */ +LZ4_streamHC_t LZ4_streamHC; +static void local_LZ4_resetStreamHC(void) +{ + LZ4_resetStreamHC(&LZ4_streamHC, 0); +} + +static int local_LZ4_saveDictHC(const char* in, char* out, int inSize) +{ + (void)in; + return LZ4_saveDictHC(&LZ4_streamHC, out, inSize); +} + static int local_LZ4_compressHC_withStateHC(const char* in, char* out, int inSize) { - return LZ4_compressHC_withStateHC(stateLZ4HC, in, out, inSize); + return LZ4_compressHC_withStateHC(&LZ4_streamHC, in, out, inSize); } static int local_LZ4_compressHC_limitedOutput_withStateHC(const char* in, char* out, int inSize) { - return LZ4_compressHC_limitedOutput_withStateHC(stateLZ4HC, in, out, inSize, LZ4_compressBound(inSize)-1); + return LZ4_compressHC_limitedOutput_withStateHC(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); } static int local_LZ4_compressHC_limitedOutput(const char* in, char* out, int inSize) @@ -447,33 +467,16 @@ static int local_LZ4_compressHC_limitedOutput(const char* in, char* out, int inS static int local_LZ4_compressHC_continue(const char* in, char* out, int inSize) { - return LZ4_compressHC_continue((LZ4_streamHC_t*)ctx, in, out, inSize); + return LZ4_compressHC_continue(&LZ4_streamHC, in, out, inSize); } static int local_LZ4_compressHC_limitedOutput_continue(const char* in, char* out, int inSize) { - return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4F_compressFrame(const char* in, char* out, int inSize) -{ - return (int)LZ4F_compressFrame(out, 2*inSize + 16, in, inSize, NULL); -} - -static int local_LZ4_saveDict(const char* in, char* out, int inSize) -{ - (void)in; - return LZ4_saveDict(&LZ4_dict, out, inSize); -} - -LZ4_streamHC_t LZ4_dictHC; -static int local_LZ4_saveDictHC(const char* in, char* out, int inSize) -{ - (void)in; - return LZ4_saveDictHC(&LZ4_dictHC, out, inSize); + return LZ4_compressHC_limitedOutput_continue(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); } +/* decompression functions */ static int local_LZ4_decompress_fast(const char* in, char* out, int inSize, int outSize) { (void)inSize; @@ -481,13 +484,6 @@ static int local_LZ4_decompress_fast(const char* in, char* out, int inSize, int return outSize; } -static int local_LZ4_decompress_fast_withPrefix64k(const char* in, char* out, int inSize, int outSize) -{ - (void)inSize; - LZ4_decompress_fast_withPrefix64k(in, out, outSize); - return outSize; -} - static int local_LZ4_decompress_fast_usingDict(const char* in, char* out, int inSize, int outSize) { (void)inSize; @@ -516,6 +512,13 @@ static int local_LZ4_decompress_safe_partial(const char* in, char* out, int inSi return LZ4_decompress_safe_partial(in, out, inSize, outSize - 5, outSize); } + +/* frame functions */ +static int local_LZ4F_compressFrame(const char* in, char* out, int inSize) +{ + return (int)LZ4F_compressFrame(out, 2*inSize + 16, in, inSize, NULL); +} + static LZ4F_decompressionContext_t g_dCtx; static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outSize) @@ -534,11 +537,8 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) { int fileIdx=0; char* orig_buff; -# define NB_COMPRESSION_ALGORITHMS 17 - double totalCTime[NB_COMPRESSION_ALGORITHMS+1] = {0}; - double totalCSize[NB_COMPRESSION_ALGORITHMS+1] = {0}; -# define NB_DECOMPRESSION_ALGORITHMS 9 - double totalDTime[NB_DECOMPRESSION_ALGORITHMS+1] = {0}; +# define NB_COMPRESSION_ALGORITHMS 100 +# define NB_DECOMPRESSION_ALGORITHMS 100 size_t errorCode; @@ -576,8 +576,6 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); /* Allocation */ - stateLZ4 = LZ4_createStream(); - stateLZ4HC = LZ4_createStreamHC(); errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); if (LZ4F_isError(errorCode)) { @@ -634,9 +632,12 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) { const char* compressorName; int (*compressionFunction)(const char*, char*, int); - void* (*initFunction)(const char*) = NULL; + void (*initFunction)(void) = NULL; double bestTime = 100000000.; + /* filter compressionAlgo only */ + if ((compressionAlgo != ALL_COMPRESSORS) && (compressionAlgo != cAlgNb)) continue; + /* Init data chunks */ { int i; @@ -654,34 +655,34 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) } } - if ((compressionAlgo != ALL_COMPRESSORS) && (compressionAlgo != cAlgNb)) continue; - switch(cAlgNb) { case 1 : compressionFunction = LZ4_compress; compressorName = "LZ4_compress"; break; case 2 : compressionFunction = local_LZ4_compress_limitedOutput; compressorName = "LZ4_compress_limitedOutput"; break; case 3 : compressionFunction = local_LZ4_compress_withState; compressorName = "LZ4_compress_withState"; break; case 4 : compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; - case 5 : compressionFunction = local_LZ4_compress_continue; initFunction = LZ4_create; compressorName = "LZ4_compress_continue"; break; - case 6 : compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = LZ4_create; compressorName = "LZ4_compress_limitedOutput_continue"; break; + case 5 : compressionFunction = local_LZ4_compress_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_continue"; break; + case 6 : compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_limitedOutput_continue"; break; case 7 : compressionFunction = local_LZ4_compress_fast; compressorName = "LZ4_compress_fast"; break; case 8 : compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; case 9 : compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; case 10 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; case 11: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; - case 12: compressionFunction = local_LZ4_compressHC_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_continue"; break; - case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = LZ4_createHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; + case 12: compressionFunction = local_LZ4_compressHC_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_continue"; break; + case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; case 14: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; case 15: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; chunkP[0].origSize = (int)benchedSize; nbChunks=1; break; case 16: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; - LZ4_loadDict(&LZ4_dict, chunkP[0].origBuffer, chunkP[0].origSize); + LZ4_loadDict(&LZ4_stream, chunkP[0].origBuffer, chunkP[0].origSize); break; case 17: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; - LZ4_loadDictHC(&LZ4_dictHC, chunkP[0].origBuffer, chunkP[0].origSize); + LZ4_loadDictHC(&LZ4_streamHC, chunkP[0].origBuffer, chunkP[0].origSize); break; - default : DISPLAY("ERROR ! Bad algorithm Id !! \n"); free(chunkP); return 1; + default : + continue; /* unknown ID : just skip */ + DISPLAY("ERROR ! Bad algorithm Id !! \n"); free(chunkP); return 1; } for (loopNb = 1; loopNb <= nbIterations; loopNb++) @@ -698,13 +699,12 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) milliTime = BMK_GetMilliStart(); while(BMK_GetMilliSpan(milliTime) < TIMELOOP) { - if (initFunction!=NULL) ctx = (LZ4_stream_t*)initFunction(chunkP[0].origBuffer); + if (initFunction!=NULL) initFunction(); for (chunkNb=0; chunkNb%9i (%5.2f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000.); else DISPLAY("%2i-%-28.28s :%9i ->%9i (%5.1f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000.); - - totalCTime[cAlgNb] += bestTime; - totalCSize[cAlgNb] += cSize; } /* Prepare layout for decompression */ @@ -732,7 +729,8 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) size_t remaining = benchedSize; char* in = orig_buff; char* out = compressed_buff; - nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); + + nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); for (i=0; i %7.1f MB/s\n", dAlgNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000.); - - totalDTime[dAlgNb] += bestTime; } } @@ -822,8 +818,6 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) free(orig_buff); free(compressed_buff); free(chunkP); - LZ4_freeStream(stateLZ4); - LZ4_freeStreamHC(stateLZ4HC); LZ4F_freeDecompressionContext(g_dCtx); } diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 9e2ce1e..d811eb2 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -316,7 +316,6 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c # define FUZ_DISPLAYTEST { testNb++; g_displayLevel<3 ? 0 : printf("%2u\b\b", testNb); if (g_displayLevel==4) fflush(stdout); } void* stateLZ4 = malloc(LZ4_sizeofState()); void* stateLZ4HC = malloc(LZ4_sizeofStateHC()); - void* LZ4continue; LZ4_stream_t LZ4dict; LZ4_streamHC_t LZ4dictHC; U32 crcOrig, crcCheck; @@ -546,16 +545,18 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c /* Compress using dictionary */ FUZ_DISPLAYTEST; - LZ4continue = LZ4_create (dict); - LZ4_compress_continue ((LZ4_stream_t*)LZ4continue, dict, compressedBuffer, dictSize); // Just to fill hash tables - blockContinueCompressedSize = LZ4_compress_continue ((LZ4_stream_t*)LZ4continue, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); - free (LZ4continue); + { + LZ4_stream_t LZ4_stream; + LZ4_resetStream(&LZ4_stream); + LZ4_compress_continue (&LZ4_stream, dict, compressedBuffer, dictSize); /* Just to fill hash tables */ + blockContinueCompressedSize = LZ4_compress_continue (&LZ4_stream, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); + } /* Decompress with dictionary as prefix */ FUZ_DISPLAYTEST; memcpy(decodedBuffer, dict, dictSize); - ret = LZ4_decompress_fast_withPrefix64k(compressedBuffer, decodedBuffer+dictSize, blockSize); + ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer+dictSize, blockSize, decodedBuffer, dictSize); FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_withPrefix64k did not read all compressed block input"); crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); if (crcCheck!=crcOrig) @@ -568,7 +569,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_withPrefix64k corrupted decoded data (dict %i)", dictSize); FUZ_DISPLAYTEST; - ret = LZ4_decompress_safe_withPrefix64k(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize); + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize, decodedBuffer, dictSize); FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_withPrefix64k did not regenerate original data"); crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_withPrefix64k corrupted decoded data"); From 9443f3d9244faedb08527bf684994a0495117833 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 11 Apr 2015 19:12:03 +0100 Subject: [PATCH 046/121] Extended obsolete warning messages to lz4hc --- lib/lz4.h | 13 +++++++++---- lib/lz4hc.h | 35 ++++++++++++++++++++++++++++------- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/lib/lz4.h b/lib/lz4.h index 7bd4594..0ee1a1e 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -280,9 +280,14 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS /************************************** * Obsolete Functions **************************************/ -/* Warning statements */ -#ifndef _WARNING_STATEMENT_BLOCK -# define _WARNING_STATEMENT_BLOCK +/* Deprecate Warnings */ +/* Should these warnings messages be a problem, + it is generally possible to disable them, + with -Wno-deprecated-declarations for gcc + or _CRT_SECURE_NO_WARNINGS in Visual for example. + You can also define _DEPRECATE_WARNING_DEFBLOCK. */ +#ifndef _DEPRECATE_WARNING_DEFBLOCK +# define _DEPRECATE_WARNING_DEFBLOCK # define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) # if (GCC_VERSION >= 405) || defined(__clang__) # define DEPRECATED(message) __attribute__((deprecated(message))) @@ -294,7 +299,7 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS # pragma message("WARNING: You need to implement DEPRECATED for this compiler") # define DEPRECATED # endif -#endif // _WARNING_STATEMENT_BLOCK +#endif // _DEPRECATE_WARNING_DEFBLOCK /* Obsolete compression functions */ /* These functions are planned to generate warnings by r131 approximately */ diff --git a/lib/lz4hc.h b/lib/lz4hc.h index c257dcc..7a5bb02 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -137,8 +137,29 @@ using LZ4_saveDictHC(). /************************************** * Deprecated Functions * ************************************/ +/* Deprecate Warnings */ +/* Should these warnings messages be a problem, + it is generally possible to disable them, + with -Wno-deprecated-declarations for gcc + or _CRT_SECURE_NO_WARNINGS in Visual for example. + You can also define _DEPRECATE_WARNING_DEFBLOCK. */ +#ifndef _DEPRECATE_WARNING_DEFBLOCK +# define _DEPRECATE_WARNING_DEFBLOCK +# define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) +# if (GCC_VERSION >= 405) || defined(__clang__) +# define DEPRECATED(message) __attribute__((deprecated(message))) +# elif (GCC_VERSION >= 301) +# define DEPRECATED(message) __attribute__((deprecated)) +# elif defined(_MSC_VER) +# define DEPRECATED(message) __declspec(deprecated(message)) +# else +# pragma message("WARNING: You need to implement DEPRECATED for this compiler") +# define DEPRECATED +# endif +#endif // _DEPRECATE_WARNING_DEFBLOCK /* compression functions */ +/* these functions are planned to trigger warning messages by r131 approximately */ int LZ4_compressHC (const char* source, char* dest, int inputSize); int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize); int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel); @@ -151,13 +172,13 @@ int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, cons int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /* Streaming functions following the older model; should no longer be used */ -void* LZ4_createHC (const char* inputBuffer); -char* LZ4_slideInputBufferHC (void* LZ4HC_Data); -int LZ4_freeHC (void* LZ4HC_Data); -int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); -int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); -int LZ4_sizeofStreamStateHC(void); -int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); +DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (const char* inputBuffer); +DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); +DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); +DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); +DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); +DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); #if defined (__cplusplus) From 62ed15319570d80690915f0d0dba0dc2e9478631 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 08:21:35 +0100 Subject: [PATCH 047/121] Fixed : a few minor coverity warnings --- lib/lz4frame.c | 18 ++++++++++++------ programs/bench.c | 12 ++++-------- programs/frametest.c | 18 +++++++++--------- programs/fullbench.c | 18 +++++++++++------- programs/fuzzer.c | 9 +++++---- programs/lz4cli.c | 2 +- programs/lz4io.c | 22 ++++++++++++++++------ 7 files changed, 58 insertions(+), 41 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index b6dbd20..7094364 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -372,9 +372,12 @@ LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_comp { LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)LZ4F_compressionContext; - FREEMEM(cctxPtr->lz4CtxPtr); - FREEMEM(cctxPtr->tmpBuff); - FREEMEM(LZ4F_compressionContext); + if (cctxPtr != NULL) /* null pointers can be safely provided to this function, like free() */ + { + FREEMEM(cctxPtr->lz4CtxPtr); + FREEMEM(cctxPtr->tmpBuff); + FREEMEM(LZ4F_compressionContext); + } return OK_NoError; } @@ -768,9 +771,12 @@ LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* LZ LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_decompressionContext) { LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)LZ4F_decompressionContext; - FREEMEM(dctxPtr->tmpIn); - FREEMEM(dctxPtr->tmpOutBuffer); - FREEMEM(dctxPtr); + if (dctxPtr != NULL) /* can accept NULL input, like free() */ + { + FREEMEM(dctxPtr->tmpIn); + FREEMEM(dctxPtr->tmpOutBuffer); + FREEMEM(dctxPtr); + } return OK_NoError; } diff --git a/programs/bench.c b/programs/bench.c index a5c72d6..8921f09 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -281,15 +281,11 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) /* Check file existence */ inFileName = fileNamesTable[fileIdx++]; inFile = fopen( inFileName, "rb" ); - if (inFile==NULL) - { - DISPLAY( "Pb opening %s\n", inFileName); - return 11; - } + if (inFile==NULL) { DISPLAY( "Pb opening %s\n", inFileName); return 11; } /* Memory allocation & restrictions */ inFileSize = BMK_GetFileSize(inFileName); - if (inFileSize==0) { DISPLAY( "file is empty\n"); return 11; } + if (inFileSize==0) { DISPLAY( "file is empty\n"); fclose(inFile); return 11; } benchedSize = (size_t) BMK_findMaxMem(inFileSize * 2) / 2; if (benchedSize==0) { DISPLAY( "not enough memory\n"); return 11; } if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; @@ -306,7 +302,6 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) compressedBuffSize = nbChunks * maxCompressedChunkSize; compressedBuffer = (char*)malloc((size_t)compressedBuffSize); - if (!orig_buff || !compressedBuffer) { DISPLAY("\nError: not enough memory!\n"); @@ -402,6 +397,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) } milliTime = BMK_GetMilliSpan(milliTime); + nbLoops += !nbLoops; /* avoid division by zero */ if ((double)milliTime < fastestD*nbLoops) fastestD = (double)milliTime/nbLoops; DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s \r", loopNb, inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.); @@ -431,7 +427,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) if (nbFiles > 1) DISPLAY("%-16.16s :%10llu ->%10llu (%5.2f%%), %6.1f MB/s , %6.1f MB/s\n", " TOTAL", (long long unsigned int)totals, (long long unsigned int)totalz, (double)totalz/(double)totals*100., (double)totals/totalc/1000., (double)totals/totald/1000.); - if (BMK_pause) { DISPLAY("\npress enter...\n"); getchar(); } + if (BMK_pause) { DISPLAY("\npress enter...\n"); (void)getchar(); } return 0; } diff --git a/programs/frametest.c b/programs/frametest.c index 5fccd08..6204d7a 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -238,10 +238,11 @@ int basicTests(U32 seed, double compressibility) U32 randState = seed; size_t cSize, testSize; LZ4F_preferences_t prefs; - LZ4F_decompressionContext_t dCtx; + LZ4F_decompressionContext_t dCtx = NULL; + LZ4F_compressionContext_t cctx = NULL; U64 crcOrig; - // Create compressible test buffer + /* Create compressible test buffer */ memset(&prefs, 0, sizeof(prefs)); CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); compressedBuffer = malloc(LZ4F_compressFrameBound(COMPRESSIBLE_NOISE_LENGTH, NULL)); @@ -249,7 +250,7 @@ int basicTests(U32 seed, double compressibility) FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - // Trivial tests : one-step frame + /* Trivial tests : one-step frame */ testSize = COMPRESSIBLE_NOISE_LENGTH; DISPLAYLEVEL(3, "Using NULL preferences : \n"); cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL); @@ -385,7 +386,6 @@ int basicTests(U32 seed, double compressibility) size_t errorCode; BYTE* const ostart = (BYTE*)compressedBuffer; BYTE* op = ostart; - LZ4F_compressionContext_t cctx; errorCode = LZ4F_createCompressionContext(&cctx, LZ4F_VERSION); if (LZ4F_isError(errorCode)) goto _output_error; @@ -430,6 +430,7 @@ int basicTests(U32 seed, double compressibility) errorCode = LZ4F_freeCompressionContext(cctx); if (LZ4F_isError(errorCode)) goto _output_error; + cctx = NULL; } DISPLAYLEVEL(3, "Skippable frame test : \n"); @@ -500,10 +501,6 @@ int basicTests(U32 seed, double compressibility) ip += iSize; } DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8)); - - /* release memory */ - errorCode = LZ4F_freeDecompressionContext(dCtx); - if (LZ4F_isError(errorCode)) goto _output_error; } DISPLAY("Basic tests completed \n"); @@ -511,6 +508,8 @@ _end: free(CNBuffer); free(compressedBuffer); free(decodedBuffer); + LZ4F_freeDecompressionContext(dCtx); dCtx = NULL; + LZ4F_freeCompressionContext(cctx); cctx = NULL; return testResult; _output_error: @@ -667,6 +666,7 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi unsigned nonContiguousDst = (FUZ_rand(&randState) & 3) == 1; nonContiguousDst += FUZ_rand(&randState) & nonContiguousDst; /* 0=>0; 1=>1,2 */ XXH64_reset(&xxh64, 1); + if (maxBits < 3) maxBits = 3; while (ip < iend) { unsigned nbBitsI = (FUZ_rand(&randState) % (maxBits-1)) + 1; @@ -709,7 +709,7 @@ _end: if (pause) { DISPLAY("press enter to finish \n"); - getchar(); + (void)getchar(); } return testResult; diff --git a/programs/fullbench.c b/programs/fullbench.c index 700655b..8c95d29 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -533,14 +533,15 @@ static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outS } +#define NB_COMPRESSION_ALGORITHMS 100 +#define NB_DECOMPRESSION_ALGORITHMS 100 +#define CLEANEXIT(c) { benchStatus = c; goto _clean_up; } int fullSpeedBench(char** fileNamesTable, int nbFiles) { int fileIdx=0; char* orig_buff; -# define NB_COMPRESSION_ALGORITHMS 100 -# define NB_DECOMPRESSION_ALGORITHMS 100 size_t errorCode; - + int benchStatus = 0; /* Loop for each fileName */ while (fileIdx %.2f%%\n", (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); { @@ -538,6 +539,7 @@ static unsigned LZ4IO_readLE32 (const void* s) return value32; } +static unsigned g_magicRead = 0; static unsigned long long decodeLegacyStream(FILE* finput, FILE* foutput) { unsigned long long filesize = 0; @@ -562,7 +564,7 @@ static unsigned long long decodeLegacyStream(FILE* finput, FILE* foutput) blockSize = LZ4IO_readLE32(in_buff); /* Convert to Little Endian */ if (blockSize > LZ4_COMPRESSBOUND(LEGACY_BLOCKSIZE)) { /* Cannot read next block : maybe new stream ? */ - fseek(finput, -4, SEEK_CUR); + g_magicRead = blockSize; break; } @@ -761,10 +763,18 @@ static unsigned long long selectDecoder( FILE* finput, FILE* foutput) nbCalls++; /* Check Archive Header */ - nbReadBytes = fread(U32store, 1, MAGICNUMBER_SIZE, finput); - if (nbReadBytes==0) return ENDOFSTREAM; /* EOF */ - if (nbReadBytes != MAGICNUMBER_SIZE) EXM_THROW(40, "Unrecognized header : Magic Number unreadable"); - magicNumber = LZ4IO_readLE32(U32store); /* Little Endian format */ + if (g_magicRead) + { + magicNumber = g_magicRead; + g_magicRead = 0; + } + else + { + nbReadBytes = fread(U32store, 1, MAGICNUMBER_SIZE, finput); + if (nbReadBytes==0) return ENDOFSTREAM; /* EOF */ + if (nbReadBytes != MAGICNUMBER_SIZE) EXM_THROW(40, "Unrecognized header : Magic Number unreadable"); + magicNumber = LZ4IO_readLE32(U32store); /* Little Endian format */ + } if (LZ4IO_isSkippableMagicNumber(magicNumber)) magicNumber = LZ4IO_SKIPPABLE0; /* fold skippable magic numbers */ switch(magicNumber) @@ -809,7 +819,7 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file get_fileHandle(input_filename, output_filename, &finput, &foutput); /* sparse file */ - if (g_sparseFileSupport && foutput) { SET_SPARSE_FILE_MODE(foutput); } + if (g_sparseFileSupport) { SET_SPARSE_FILE_MODE(foutput); } /* Loop over multiple streams */ do From 66b8a4a8afe72ac37a26199249a78a5349201472 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 08:58:00 +0100 Subject: [PATCH 048/121] Fixed : minor Visual warnings --- programs/fullbench.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/programs/fullbench.c b/programs/fullbench.c index 8c95d29..4ffae59 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -684,7 +684,6 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) break; default : continue; /* unknown ID : just skip */ - DISPLAY("ERROR ! Bad algorithm Id !! \n"); free(chunkP); return 1; } for (loopNb = 1; loopNb <= nbIterations; loopNb++) @@ -774,7 +773,6 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) break; default : continue; /* skip if unknown ID */ - DISPLAY("ERROR ! Bad decompression algorithm Id !! \n"); CLEANEXIT(1); } { size_t i; for (i=0; i Date: Sun, 12 Apr 2015 17:28:53 +0900 Subject: [PATCH 049/121] Add LZ4 prefix to deprecation macros Replace the following macros: - "_DEPRECATE_WARNING_DEFBLOCK" to "LZ4_DEPRECATE_WARNING_DEFBLOCK" - "DEPRECATED" to "LZ4_DEPRECATED" --- lib/lz4.h | 30 +++++++++++++++--------------- lib/lz4hc.h | 32 ++++++++++++++++---------------- 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/lib/lz4.h b/lib/lz4.h index 0ee1a1e..20e3c01 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -285,21 +285,21 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS it is generally possible to disable them, with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual for example. - You can also define _DEPRECATE_WARNING_DEFBLOCK. */ -#ifndef _DEPRECATE_WARNING_DEFBLOCK -# define _DEPRECATE_WARNING_DEFBLOCK + You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ +#ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK +# define LZ4_DEPRECATE_WARNING_DEFBLOCK # define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) # if (GCC_VERSION >= 405) || defined(__clang__) -# define DEPRECATED(message) __attribute__((deprecated(message))) +# define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) # elif (GCC_VERSION >= 301) -# define DEPRECATED(message) __attribute__((deprecated)) +# define LZ4_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) -# define DEPRECATED(message) __declspec(deprecated(message)) +# define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # else -# pragma message("WARNING: You need to implement DEPRECATED for this compiler") -# define DEPRECATED +# pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") +# define LZ4_DEPRECATED # endif -#endif // _DEPRECATE_WARNING_DEFBLOCK +#endif // LZ4_DEPRECATE_WARNING_DEFBLOCK /* Obsolete compression functions */ /* These functions are planned to generate warnings by r131 approximately */ @@ -321,14 +321,14 @@ int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const cha /* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */ /* Obsolete streaming functions; use new streaming interface whenever possible */ -DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (const char* inputBuffer); -DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); -DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, const char* inputBuffer); -DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); +LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); +LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); /* Obsolete streaming decoding functions */ -DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize); -DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize); +LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize); +LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize); #if defined (__cplusplus) diff --git a/lib/lz4hc.h b/lib/lz4hc.h index 7a5bb02..95bb6fe 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -142,21 +142,21 @@ using LZ4_saveDictHC(). it is generally possible to disable them, with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual for example. - You can also define _DEPRECATE_WARNING_DEFBLOCK. */ -#ifndef _DEPRECATE_WARNING_DEFBLOCK -# define _DEPRECATE_WARNING_DEFBLOCK + You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ +#ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK +# define LZ4_DEPRECATE_WARNING_DEFBLOCK # define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) # if (GCC_VERSION >= 405) || defined(__clang__) -# define DEPRECATED(message) __attribute__((deprecated(message))) +# define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) # elif (GCC_VERSION >= 301) -# define DEPRECATED(message) __attribute__((deprecated)) +# define LZ4_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) -# define DEPRECATED(message) __declspec(deprecated(message)) +# define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # else -# pragma message("WARNING: You need to implement DEPRECATED for this compiler") -# define DEPRECATED +# pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") +# define LZ4_DEPRECATED # endif -#endif // _DEPRECATE_WARNING_DEFBLOCK +#endif // LZ4_DEPRECATE_WARNING_DEFBLOCK /* compression functions */ /* these functions are planned to trigger warning messages by r131 approximately */ @@ -172,13 +172,13 @@ int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, cons int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /* Streaming functions following the older model; should no longer be used */ -DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (const char* inputBuffer); -DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); -DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); -DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); -DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); -DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); -DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); +LZ4_DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); +LZ4_DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); +LZ4_DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +LZ4_DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); +LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); #if defined (__cplusplus) From 81fdd9df23821d5db8331e66478e2617b470820c Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 09:29:52 +0100 Subject: [PATCH 050/121] Fixed a few Valgrind warnings --- lib/lz4.c | 5 +++-- lib/lz4hc.c | 2 +- programs/fullbench.c | 52 +++++++++++++++----------------------------- 3 files changed, 21 insertions(+), 38 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index d6fdcd9..e57d080 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -756,6 +756,7 @@ int LZ4_freeStream (LZ4_stream_t* LZ4_stream) } +#define HASH_UNIT sizeof(size_t) int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) { LZ4_stream_t_internal* dict = (LZ4_stream_t_internal*) LZ4_dict; @@ -765,7 +766,7 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) if (dict->initCheck) LZ4_resetStream(LZ4_dict); /* Uninitialized structure detected */ - if (dictSize < MINMATCH) + if (dictSize < HASH_UNIT) { dict->dictionary = NULL; dict->dictSize = 0; @@ -778,7 +779,7 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) dict->dictSize = (U32)(dictEnd - p); dict->currentOffset += dict->dictSize; - while (p <= dictEnd-MINMATCH) + while (p <= dictEnd-HASH_UNIT) { LZ4_putPosition(p, dict, byU32, base); p+=3; diff --git a/lib/lz4hc.c b/lib/lz4hc.c index 234e1af..4f1272c 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -656,7 +656,7 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS if (dictSize > 64 KB) dictSize = 64 KB; if (dictSize < 4) dictSize = 0; if (dictSize > prefixSize) dictSize = prefixSize; - memcpy(safeBuffer, streamPtr->end - dictSize, dictSize); + memmove(safeBuffer, streamPtr->end - dictSize, dictSize); { U32 endIndex = (U32)(streamPtr->end - streamPtr->base); streamPtr->end = (const BYTE*)safeBuffer + dictSize; diff --git a/programs/fullbench.c b/programs/fullbench.c index 4ffae59..51e0e9a 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -539,10 +539,16 @@ static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outS int fullSpeedBench(char** fileNamesTable, int nbFiles) { int fileIdx=0; - char* orig_buff; + char* orig_buff = NULL; + struct chunkParameters* chunkP = NULL; + char* compressed_buff=NULL; size_t errorCode; int benchStatus = 0; + /* Init */ + errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) { DISPLAY("dctx allocation issue \n"); CLEANEXIT(10); } + /* Loop for each fileName */ while (fileIdx inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); /* Allocation */ - errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) - { - DISPLAY("dctx allocation issue \n"); - fclose(inFile); - return 10; - } chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)chunkSize)+1) * sizeof(struct chunkParameters)); orig_buff = (char*) malloc((size_t)benchedSize); nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); maxCompressedChunkSize = LZ4_compressBound(chunkSize); compressedBuffSize = nbChunks * maxCompressedChunkSize; compressed_buff = (char*)malloc((size_t)compressedBuffSize); - - - if(!orig_buff || !compressed_buff) - { - DISPLAY("\nError: not enough memory!\n"); - free(orig_buff); - free(compressed_buff); - free(chunkP); - fclose(inFile); - return 12; - } + if(!orig_buff || !compressed_buff) { DISPLAY("\nError: not enough memory!\n"); fclose(inFile); CLEANEXIT(12); } /* Fill in src buffer */ DISPLAY("Loading %s... \r", inFileName); @@ -813,15 +796,14 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) DISPLAY("%2i-%-29.29s :%10i -> %7.1f MB/s\n", dAlgNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000.); } - } + } _clean_up: - free(orig_buff); - free(compressed_buff); - free(chunkP); - LZ4F_freeDecompressionContext(g_dCtx); - } + free(orig_buff); + free(compressed_buff); + free(chunkP); + LZ4F_freeDecompressionContext(g_dCtx); if (BMK_pause) { printf("press enter...\n"); (void)getchar(); } From 138673df5e1f42d098a55d23fe84228726e9739d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 09:37:34 +0100 Subject: [PATCH 051/121] fixed minor g++ warning --- lib/lz4.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4.c b/lib/lz4.c index e57d080..4a6c550 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -766,7 +766,7 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) if (dict->initCheck) LZ4_resetStream(LZ4_dict); /* Uninitialized structure detected */ - if (dictSize < HASH_UNIT) + if (dictSize < (int)HASH_UNIT) { dict->dictionary = NULL; dict->dictSize = 0; From 8a610004f6ac5bf5966b8a16da5a240d289b538b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 09:42:58 +0100 Subject: [PATCH 052/121] Fixed a few coverity warnings --- programs/bench.c | 3 ++- programs/fullbench.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/programs/bench.c b/programs/bench.c index 8921f09..d3b060c 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -287,7 +287,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) inFileSize = BMK_GetFileSize(inFileName); if (inFileSize==0) { DISPLAY( "file is empty\n"); fclose(inFile); return 11; } benchedSize = (size_t) BMK_findMaxMem(inFileSize * 2) / 2; - if (benchedSize==0) { DISPLAY( "not enough memory\n"); return 11; } + if (benchedSize==0) { DISPLAY( "not enough memory\n"); fclose(inFile); return 11; } if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) { @@ -376,6 +376,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) } milliTime = BMK_GetMilliSpan(milliTime); + nbLoops += !nbLoops; /* avoid division by zero */ if ((double)milliTime < fastestC*nbLoops) fastestC = (double)milliTime/nbLoops; cSize=0; for (chunkNb=0; chunkNb Date: Sun, 12 Apr 2015 11:37:55 +0100 Subject: [PATCH 053/121] Fixed : minor coverity warning --- programs/fullbench.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/fullbench.c b/programs/fullbench.c index 40d95e5..658c365 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -569,9 +569,9 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) /* Memory size adjustments */ inFileSize = BMK_GetFileSize(inFileName); - if (inFileSize==0) { DISPLAY( "file is empty\n"); CLEANEXIT(11); } + if (inFileSize==0) { DISPLAY( "file is empty\n"); fclose(inFile); CLEANEXIT(11); } benchedSize = (size_t) BMK_findMaxMem(inFileSize*2) / 2; /* because 2 buffers */ - if (benchedSize==0) { DISPLAY( "not enough memory\n"); CLEANEXIT(11); } + if (benchedSize==0) { DISPLAY( "not enough memory\n"); fclose(inFile); CLEANEXIT(11); } if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); From 2852b9e439d7ad27c09641b1b5d108887a159a01 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 15:17:10 +0100 Subject: [PATCH 054/121] Fixed issue #84 --- Makefile | 4 ++-- lib/lz4.c | 13 +++++++------ lib/lz4frame.c | 6 +++--- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/Makefile b/Makefile index 32f38aa..b4b1c6b 100644 --- a/Makefile +++ b/Makefile @@ -101,10 +101,10 @@ gpptest: clean $(MAKE) all CC=g++ CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror" clangtest: clean - $(MAKE) all CC=clang CFLAGS="-O3 -Werror" + $(MAKE) all CC=clang CPPFLAGS="-Werror -Wconversion -Wno-sign-conversion" sanitize: clean - $(MAKE) test CC=clang CFLAGS="-O3 -g -fsanitize=undefined" FUZZER_TIME="-T5mn" + $(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T5mn" staticAnalyze: clean scan-build -v $(MAKE) all CFLAGS=-g diff --git a/lib/lz4.c b/lib/lz4.c index 4a6c550..d8e4977 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -94,6 +94,7 @@ # endif /* __STDC_VERSION__ */ #endif /* _MSC_VER */ +/* GCC_VERSION is defined into lz4.h */ #if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) #else @@ -260,7 +261,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanForward64( &r, (U64)val ); return (int)(r>>3); -# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctzll((U64)val) >> 3); # else static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 }; @@ -273,7 +274,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r; _BitScanForward( &r, (U32)val ); return (int)(r>>3); -# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctz((U32)val) >> 3); # else static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 }; @@ -289,8 +290,8 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanReverse64( &r, val ); return (unsigned)(r>>3); -# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT) - return (__builtin_clzll(val) >> 3); +# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) + return (__builtin_clzll((U64)val) >> 3); # else unsigned r; if (!(val>>32)) { r=4; } else { r=0; val>>=32; } @@ -305,8 +306,8 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanReverse( &r, (unsigned long)val ); return (unsigned)(r>>3); -# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT) - return (__builtin_clz(val) >> 3); +# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) + return (__builtin_clz((U32)val) >> 3); # else unsigned r; if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; } diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 7094364..31cf9a5 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -447,10 +447,10 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds headerStart = dstPtr; /* FLG Byte */ - *dstPtr++ = ((1 & _2BITS) << 6) /* Version('01') */ + *dstPtr++ = (BYTE)(((1 & _2BITS) << 6) /* Version('01') */ + ((cctxPtr->prefs.frameInfo.blockMode & _1BIT ) << 5) /* Block mode */ - + (BYTE)((cctxPtr->prefs.frameInfo.contentChecksumFlag & _1BIT ) << 2) /* Frame checksum */ - + (BYTE)((cctxPtr->prefs.frameInfo.contentSize > 0) << 3); /* Frame content size */ + + ((cctxPtr->prefs.frameInfo.contentChecksumFlag & _1BIT ) << 2) /* Frame checksum */ + + ((cctxPtr->prefs.frameInfo.contentSize > 0) << 3)); /* Frame content size */ /* BD Byte */ *dstPtr++ = (BYTE)((cctxPtr->prefs.frameInfo.blockSizeID & _3BITS) << 4); /* Optional Frame content size field */ From 42e5bc4a9dab26a9d1fbc9899e107b3fb926233c Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 16:06:12 +0100 Subject: [PATCH 055/121] Updated badges --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bcb8d7c..0945f85 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,9 @@ It trades CPU time for compression ratio. |Branch |Status | |------------|---------| -|master | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=master)](https://travis-ci.org/Cyan4973/lz4) | -|dev | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=dev)](https://travis-ci.org/Cyan4973/lz4) | -|visual | [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) | +|master | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=master)](https://travis-ci.org/Cyan4973/lz4) [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq/branch/master?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) | +|dev | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=dev)](https://travis-ci.org/Cyan4973/lz4) [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq/branch/dev?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) | + From 2c7988761dcd7e5435c519eeffef32a90eed93ad Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 16:40:58 +0100 Subject: [PATCH 056/121] Shortened tests durations --- Makefile | 2 +- programs/Makefile | 1 + programs/frametest.c | 33 +++++++++++++--------------- programs/fullbench.c | 51 +++++++++++++++++++++++++++----------------- programs/fuzzer.c | 33 +++++++++++++--------------- 5 files changed, 63 insertions(+), 57 deletions(-) diff --git a/Makefile b/Makefile index b4b1c6b..b83d920 100644 --- a/Makefile +++ b/Makefile @@ -104,7 +104,7 @@ clangtest: clean $(MAKE) all CC=clang CPPFLAGS="-Werror -Wconversion -Wno-sign-conversion" sanitize: clean - $(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T5mn" + $(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T1mn" staticAnalyze: clean scan-build -v $(MAKE) all CFLAGS=-g diff --git a/programs/Makefile b/programs/Makefile index 97be7d9..5ea0593 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -61,6 +61,7 @@ endif TRAVIS_TARGET:= $(LZ4_TRAVIS_CI_ENV) TEST_FILES := COPYING TEST_TARGETS := test-native +FUZZER_TIME := -T9mn default: lz4 diff --git a/programs/frametest.c b/programs/frametest.c index 6204d7a..60cd765 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -726,7 +726,7 @@ int FUZ_usage(void) DISPLAY( "\n"); DISPLAY( "Arguments :\n"); DISPLAY( " -i# : Nb of tests (default:%u) \n", nbTestsDefault); - DISPLAY( " -T# : Duration of tests (default: use Nb of tests) \n"); + DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); DISPLAY( " -s# : Select seed (default:prompt user)\n"); DISPLAY( " -t# : Select starting test number (default:0)\n"); DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); @@ -800,26 +800,23 @@ int main(int argc, char** argv) case 'T': argument++; nbTests = 0; duration = 0; - for (;;) { - if (argument[0]=='m') + switch(*argument) { - duration *= 60; - argument++; - continue; - } - if (argument[0]=='n') - { - argument++; - continue; - } - if ((*argument>='0') && (*argument<='9')) - { - duration *= 10; - duration += *argument - '0'; - argument++; - continue; + case 'm': duration *= 60; argument++; continue; + case 's': + case 'n': argument++; continue; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': duration *= 10; duration += *argument++ - '0'; continue; } break; } diff --git a/programs/fullbench.c b/programs/fullbench.c index 658c365..ed53819 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -535,24 +535,22 @@ static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outS #define NB_COMPRESSION_ALGORITHMS 100 #define NB_DECOMPRESSION_ALGORITHMS 100 -#define CLEANEXIT(c) { benchStatus = c; goto _clean_up; } int fullSpeedBench(char** fileNamesTable, int nbFiles) { int fileIdx=0; - char* orig_buff = NULL; - struct chunkParameters* chunkP = NULL; - char* compressed_buff=NULL; size_t errorCode; - int benchStatus = 0; /* Init */ errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) { DISPLAY("dctx allocation issue \n"); CLEANEXIT(10); } + if (LZ4F_isError(errorCode)) { DISPLAY("dctx allocation issue \n"); return 10; } /* Loop for each fileName */ while (fileIdx inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); /* Allocation */ chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)chunkSize)+1) * sizeof(struct chunkParameters)); - orig_buff = (char*) malloc((size_t)benchedSize); - nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); + orig_buff = (char*) malloc(benchedSize); + nbChunks = (int) ((benchedSize + (chunkSize-1)) / chunkSize); maxCompressedChunkSize = LZ4_compressBound(chunkSize); compressedBuffSize = nbChunks * maxCompressedChunkSize; compressed_buff = (char*)malloc((size_t)compressedBuffSize); - if(!orig_buff || !compressed_buff) { DISPLAY("\nError: not enough memory!\n"); fclose(inFile); CLEANEXIT(12); } + if(!chunkP || !orig_buff || !compressed_buff) + { + DISPLAY("\nError: not enough memory!\n"); + fclose(inFile); + free(orig_buff); + free(compressed_buff); + free(chunkP); + return(12); + } /* Fill in src buffer */ DISPLAY("Loading %s... \r", inFileName); @@ -600,7 +606,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) } /* Calculating input Checksum */ - crcOriginal = XXH32(orig_buff, (unsigned int)benchedSize,0); + crcOriginal = XXH32(orig_buff, benchedSize,0); /* Bench */ @@ -750,7 +756,14 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) case 8: decompressionFunction = local_LZ4_decompress_safe_forceExtDict; dName = "LZ4_decompress_safe_forceExtDict"; break; case 9: decompressionFunction = local_LZ4F_decompress; dName = "LZ4F_decompress"; errorCode = LZ4F_compressFrame(compressed_buff, compressedBuffSize, orig_buff, benchedSize, NULL); - if (LZ4F_isError(errorCode)) { DISPLAY("Preparation error compressing frame\n"); CLEANEXIT(1); } + if (LZ4F_isError(errorCode)) + { + DISPLAY("Error while preparing compressed frame\n"); + free(orig_buff); + free(compressed_buff); + free(chunkP); + return 1; + } chunkP[0].origSize = (int)benchedSize; chunkP[0].compressedSize = (int)errorCode; nbChunks = 1; @@ -798,17 +811,15 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) DISPLAY("%2i-%-29.29s :%10i -> %7.1f MB/s\n", dAlgNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000.); } } + free(orig_buff); + free(compressed_buff); + free(chunkP); } -_clean_up: - free(orig_buff); - free(compressed_buff); - free(chunkP); LZ4F_freeDecompressionContext(g_dCtx); - if (BMK_pause) { printf("press enter...\n"); (void)getchar(); } - return benchStatus; + return 0; } diff --git a/programs/fuzzer.c b/programs/fuzzer.c index aa91ee8..1704b6e 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -1017,7 +1017,7 @@ static int FUZ_usage(char* programName) DISPLAY( "\n"); DISPLAY( "Arguments :\n"); DISPLAY( " -i# : Nb of tests (default:%i) \n", NB_ATTEMPTS); - DISPLAY( " -T# : Duration of tests (default: use Nb of tests) \n"); + DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); DISPLAY( " -s# : Select seed (default:prompt user)\n"); DISPLAY( " -t# : Select starting test number (default:0)\n"); DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); @@ -1084,26 +1084,23 @@ int main(int argc, char** argv) case 'T': argument++; nbTests = 0; duration = 0; - for (;;) { - if (argument[0]=='m') + switch(*argument) { - duration *= 60; - argument++; - continue; - } - if (argument[0]=='n') - { - argument++; - continue; - } - if ((*argument>='0') && (*argument<='9')) - { - duration *= 10; - duration += *argument - '0'; - argument++; - continue; + case 'm': duration *= 60; argument++; continue; + case 's': + case 'n': argument++; continue; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': duration *= 10; duration += *argument++ - '0'; continue; } break; } From da117257f5946295a379ca7ccf867c2854b0aed9 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 12 Apr 2015 16:49:37 +0100 Subject: [PATCH 057/121] new memory leak test for fullbench using multi-files --- programs/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/Makefile b/programs/Makefile index 5ea0593..2b0b8a9 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -251,7 +251,7 @@ test-mem: lz4 datagen fuzzer frametest fullbench valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp tmp2 valgrind --leak-check=yes ./lz4 -t tmp2 valgrind --leak-check=yes ./lz4 -bi1 tmp - valgrind --leak-check=yes ./fullbench -i1 tmp + valgrind --leak-check=yes ./fullbench -i1 tmp tmp2 ./datagen -g256MB > tmp valgrind --leak-check=yes ./lz4 -B4D -f -vq tmp $(VOID) rm tmp* From 0169502b49dfa5eb8878f5c85be3270012266fc3 Mon Sep 17 00:00:00 2001 From: Kyle J Harper Date: Sun, 12 Apr 2015 17:28:13 -0500 Subject: [PATCH 058/121] Added new LZ4IO_decompressMultipleFilenames to allow decompression of multiple files with the -m switch added in r128 (ref: google code issue 151). Limitation: will only process files matching LZ4_EXTENSION macro, which for now seems reasonable. --- programs/lz4cli.c | 32 +++++++++++++++++++------------- programs/lz4io.c | 28 ++++++++++++++++++++++++++++ programs/lz4io.h | 3 ++- 3 files changed, 49 insertions(+), 14 deletions(-) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index da5da71..f7cf27e 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -515,22 +515,28 @@ int main(int argc, char** argv) /* IO Stream/File */ LZ4IO_setNotificationLevel(displayLevel); - if (decode) DEFAULT_DECOMPRESSOR(input_filename, output_filename); + if (decode) + { + if (multiple_inputs) + LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION); + else + DEFAULT_DECOMPRESSOR(input_filename, output_filename); + } else { - /* compression is default action */ - if (legacy_format) - { - DISPLAYLEVEL(3, "! Generating compressed LZ4 using Legacy format (deprecated) ! \n"); - LZ4IO_compressFilename_Legacy(input_filename, output_filename, cLevel); - } + /* compression is default action */ + if (legacy_format) + { + DISPLAYLEVEL(3, "! Generating compressed LZ4 using Legacy format (deprecated) ! \n"); + LZ4IO_compressFilename_Legacy(input_filename, output_filename, cLevel); + } + else + { + if (multiple_inputs) + LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); else - { - if (multiple_inputs) - LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); - else - DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel); - } + DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel); + } } if (main_pause) waitEnter(); diff --git a/programs/lz4io.c b/programs/lz4io.c index 02e03c8..16a588c 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -517,6 +517,34 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, return 0; } +int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix) +{ + int i; + int skipped_files = 0; + char* outFileName = (char*)malloc(FNSPACE); + size_t ofnSize = FNSPACE; + size_t suffixSize = strlen(suffix); + char* ifnSuffix = (char*)malloc(suffixSize + 1); + + for (i=0; i Date: Sun, 12 Apr 2015 18:41:55 -0500 Subject: [PATCH 059/121] Added support for continuation of file compression and decompression if input files are missing. Should more closely match gzip/bzip2/xz and so forth. Also removed a debug print accidentally left in. --- programs/lz4cli.c | 8 +++++--- programs/lz4io.c | 25 ++++++++++++++++++++++--- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index f7cf27e..f3524b3 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -265,7 +265,8 @@ int main(int argc, char** argv) forceStdout=0, forceCompress=0, main_pause=0, - multiple_inputs=0; + multiple_inputs=0, + multiple_rv=0; const char* input_filename=0; const char* output_filename=0; char* dynNameSpace=0; @@ -518,7 +519,7 @@ int main(int argc, char** argv) if (decode) { if (multiple_inputs) - LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION); + multiple_rv = LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION); else DEFAULT_DECOMPRESSOR(input_filename, output_filename); } @@ -533,7 +534,7 @@ int main(int argc, char** argv) else { if (multiple_inputs) - LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); + multiple_rv = LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); else DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel); } @@ -542,5 +543,6 @@ int main(int argc, char** argv) if (main_pause) waitEnter(); free(dynNameSpace); free((void*)inFileNames); + if (multiple_rv != 0) return multiple_rv; return 0; } diff --git a/programs/lz4io.c b/programs/lz4io.c index 16a588c..d399c90 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -501,12 +501,21 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix, int compressionlevel) { int i; + int missing_files = 0; + FILE *ifp; char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); for (i=0; i 0) return 1; return 0; } @@ -521,6 +531,8 @@ int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSiz { int i; int skipped_files = 0; + int missing_files = 0; + FILE *ifp; char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; size_t suffixSize = strlen(suffix); @@ -528,13 +540,19 @@ int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSiz for (i=0; i 0) return 1; + if (missing_files > 0) return 1; return 0; } From b036eaade6bccc6a2358d22c874acff062762900 Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Mon, 13 Apr 2015 20:05:20 +0900 Subject: [PATCH 060/121] Add snprintf macro for MSVC --- examples/blockStreaming_doubleBuffer.c | 5 ++++- examples/blockStreaming_lineByLine.c | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/examples/blockStreaming_doubleBuffer.c b/examples/blockStreaming_doubleBuffer.c index 0adf6ae..094f80d 100644 --- a/examples/blockStreaming_doubleBuffer.c +++ b/examples/blockStreaming_doubleBuffer.c @@ -2,7 +2,10 @@ // Copyright : Takayuki Matsuoka -#define _CRT_SECURE_NO_WARNINGS // for MSVC +#ifdef _MSC_VER /* Visual Studio */ +# define _CRT_SECURE_NO_WARNINGS +# define snprintf sprintf_s +#endif #include "lz4.h" #include diff --git a/examples/blockStreaming_lineByLine.c b/examples/blockStreaming_lineByLine.c index 6d14801..f0f7d46 100644 --- a/examples/blockStreaming_lineByLine.c +++ b/examples/blockStreaming_lineByLine.c @@ -2,7 +2,10 @@ // Copyright : Takayuki Matsuoka -#define _CRT_SECURE_NO_WARNINGS // for MSVC +#ifdef _MSC_VER /* Visual Studio */ +# define _CRT_SECURE_NO_WARNINGS +# define snprintf sprintf_s +#endif #include "lz4.h" #include From fd77bad52f39106dc1af40405076581426e86877 Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Mon, 13 Apr 2015 20:16:37 +0900 Subject: [PATCH 061/121] Replace obsolete functions --- examples/blockStreaming_doubleBuffer.c | 4 ++-- examples/blockStreaming_lineByLine.c | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/examples/blockStreaming_doubleBuffer.c b/examples/blockStreaming_doubleBuffer.c index 094f80d..58f2bdb 100644 --- a/examples/blockStreaming_doubleBuffer.c +++ b/examples/blockStreaming_doubleBuffer.c @@ -53,8 +53,8 @@ void test_compress(FILE* outFp, FILE* inpFp) { char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)]; - const int cmpBytes = LZ4_compress_continue( - lz4Stream, inpPtr, cmpBuf, inpBytes); + const int cmpBytes = LZ4_compress_safe_continue( + lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf)); if(cmpBytes <= 0) { break; } diff --git a/examples/blockStreaming_lineByLine.c b/examples/blockStreaming_lineByLine.c index f0f7d46..8ae268f 100644 --- a/examples/blockStreaming_lineByLine.c +++ b/examples/blockStreaming_lineByLine.c @@ -41,7 +41,8 @@ static void test_compress( size_t ringBufferBytes) { LZ4_stream_t* const lz4Stream = LZ4_createStream(); - char* const cmpBuf = malloc(LZ4_COMPRESSBOUND(messageMaxBytes)); + const size_t cmpBufBytes = LZ4_COMPRESSBOUND(messageMaxBytes); + char* const cmpBuf = malloc(cmpBufBytes); char* const inpBuf = malloc(ringBufferBytes); int inpOffset = 0; @@ -63,8 +64,8 @@ static void test_compress( #endif { - const int cmpBytes = LZ4_compress_continue( - lz4Stream, inpPtr, cmpBuf, inpBytes); + const int cmpBytes = LZ4_compress_safe_continue( + lz4Stream, inpPtr, cmpBuf, inpBytes, cmpBufBytes); if (cmpBytes <= 0) break; write_uint16(outFp, (uint16_t) cmpBytes); write_bin(outFp, cmpBuf, cmpBytes); From d535214e6101f5bb4697208d439306ca69399c7c Mon Sep 17 00:00:00 2001 From: Charles Allen Date: Mon, 13 Apr 2015 10:57:31 -0700 Subject: [PATCH 062/121] Add more descriptive frame errors --- lib/lz4frame.c | 14 +++++++------- lib/lz4frame_static.h | 5 +++++ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 474b196..2476464 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -854,16 +854,16 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo blockSizeID = (BD>>4) & _3BITS; /* validate */ - if (version != 1) return (size_t)-ERROR_GENERIC; /* Version Number, only supported value */ - if (blockChecksumFlag != 0) return (size_t)-ERROR_GENERIC; /* Only supported value for the time being */ - if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ - if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bit */ - if (blockSizeID < 4) return (size_t)-ERROR_GENERIC; /* 4-7 only supported values for the time being */ - if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ + if (version != 1) return (size_t)-ERROR_version_wrong; /* Version Number, only supported value */ + if (blockChecksumFlag != 0) return (size_t)-ERROR_unsupported_checksum; /* Only supported value for the time being */ + if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ + if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bit */ + if (blockSizeID < 4) return (size_t)-ERROR_unsupported_block_size; /* 4-7 only supported values for the time being */ + if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ /* check */ HC = LZ4F_headerChecksum(srcPtr+4, frameHeaderSize-5); - if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_GENERIC; /* Bad header checksum error */ + if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_header_checksum_invalid; /* Bad header checksum error */ /* save */ dctxPtr->frameInfo.blockMode = (blockMode_t)blockMode; diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index 2e56400..b608d23 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -57,6 +57,11 @@ extern "C" { ITEM(ERROR_wrongSrcPtr) \ ITEM(ERROR_decompressionFailed) \ ITEM(ERROR_checksum_invalid) \ + ITEM(ERROR_version_wrong) \ + ITEM(ERROR_unsupported_checksum) \ + ITEM(ERROR_reserved_flag_set) \ + ITEM(ERROR_unsupported_block_size) \ + ITEM(ERROR_header_checksum_invalid) \ ITEM(ERROR_maxCode) #define LZ4F_GENERATE_ENUM(ENUM) ENUM, From cc2412401400fbacd7530b8dcc4af0b0d7c734ce Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 13 Apr 2015 20:43:54 +0100 Subject: [PATCH 063/121] minor compatibility fixes --- README.md | 7 +++---- programs/lz4.1 | 6 +++--- programs/lz4cli.c | 14 ++++++++------ programs/lz4io.c | 21 ++++++++++++--------- 4 files changed, 26 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 0945f85..5d506d4 100644 --- a/README.md +++ b/README.md @@ -22,12 +22,11 @@ It trades CPU time for compression ratio. > **Branch Policy:** > - The "master" branch is considered stable, at all times. -> - The "dev" branch is the one where all contributions must be merged +> - The "dev" branch is the one where all contributions must be merged before being promoted to master. -> + If you plan to propose a patch, please commit into the "dev" branch. +> + If you plan to propose a patch, please commit into the "dev" branch, + or its own feature branch. Direct commit to "master" are not permitted. -> - Feature branches can also exist, - for dedicated tests of larger modifications before merge into "dev" branch. Benchmarks ------------------------- diff --git a/programs/lz4.1 b/programs/lz4.1 index 1225832..eea248f 100644 --- a/programs/lz4.1 +++ b/programs/lz4.1 @@ -158,13 +158,13 @@ for files that have not been compressed with force write to standard output, even if it is the console .TP -.BR \-m +.BR \-m ", " \--multiple Multiple file names. By default, the second filename is used as the output filename for the compressed file. With .B -m -, you can specify any number of input filenames, each of them will be compressed -with the resulting compressed file named +, you can specify any number of input filenames. Each of them will be compressed +independently, and the resulting name of the compressed file will be .B filename.lz4 . diff --git a/programs/lz4cli.c b/programs/lz4cli.c index ffc54c0..8d7d5ca 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -266,7 +266,7 @@ int main(int argc, char** argv) forceCompress=0, main_pause=0, multiple_inputs=0, - multiple_rv=0; + operationResult=0; const char* input_filename=0; const char* output_filename=0; char* dynNameSpace=0; @@ -295,12 +295,13 @@ int main(int argc, char** argv) /* long commands (--long-word) */ if (!strcmp(argument, "--compress")) { forceCompress = 1; continue; } if ((!strcmp(argument, "--decompress")) - || (!strcmp(argument, "--uncompress"))) { decode = 1; continue; } + || (!strcmp(argument, "--uncompress"))) { decode = 1; continue; } + if (!strcmp(argument, "--multiple")) { multiple_inputs = 1; if (inFileNames==NULL) inFileNames = (const char**)malloc(argc * sizeof(char*)); continue; } if (!strcmp(argument, "--test")) { decode = 1; LZ4IO_setOverwrite(1); output_filename=nulmark; continue; } if (!strcmp(argument, "--force")) { LZ4IO_setOverwrite(1); continue; } if (!strcmp(argument, "--no-force")) { LZ4IO_setOverwrite(0); continue; } if ((!strcmp(argument, "--stdout")) - || (!strcmp(argument, "--to-stdout"))) { forceStdout=1; output_filename=stdoutmark; displayLevel=1; continue; } + || (!strcmp(argument, "--to-stdout"))) { forceStdout=1; output_filename=stdoutmark; displayLevel=1; continue; } if (!strcmp(argument, "--frame-crc")) { LZ4IO_setStreamChecksumMode(1); continue; } if (!strcmp(argument, "--no-frame-crc")) { LZ4IO_setStreamChecksumMode(0); continue; } if (!strcmp(argument, "--content-size")) { LZ4IO_setContentSize(1); continue; } @@ -312,6 +313,7 @@ int main(int argc, char** argv) if (!strcmp(argument, "--version")) { DISPLAY(WELCOME_MESSAGE); return 0; } if (!strcmp(argument, "--keep")) { continue; } /* keep source file (default anyway; just for xz/lzma compatibility) */ + /* Short commands (note : aggregated short commands are allowed) */ if (argument[0]=='-') { @@ -528,7 +530,7 @@ int main(int argc, char** argv) if (decode) { if (multiple_inputs) - multiple_rv = LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION); + operationResult = LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION); else DEFAULT_DECOMPRESSOR(input_filename, output_filename); } @@ -543,7 +545,7 @@ int main(int argc, char** argv) else { if (multiple_inputs) - multiple_rv = LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); + operationResult = LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel); else DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel); } @@ -552,6 +554,6 @@ int main(int argc, char** argv) if (main_pause) waitEnter(); free(dynNameSpace); free((void*)inFileNames); - if (multiple_rv != 0) return multiple_rv; + if (operationResult != 0) return operationResult; return 0; } diff --git a/programs/lz4io.c b/programs/lz4io.c index c510f1d..7a3c10d 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -509,21 +509,21 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, { int i; int missing_files = 0; - FILE *ifp; char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); for (i=0; i 0) return 1; if (missing_files > 0) return 1; return 0; From c64200dd85a85d9cc69a6fda682362d851be863b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 00:07:30 +0100 Subject: [PATCH 064/121] Improved performance when compressing a lot of small files --- programs/lz4io.c | 283 +++++++++++++++++++++++++++++------------------ 1 file changed, 177 insertions(+), 106 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 7a3c10d..44c894c 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -46,7 +46,7 @@ /***************************** * Includes *****************************/ -#include /* fprintf, fopen, fread, stdin, stdout */ +#include /* fprintf, fopen, fread, stdin, stdout, fflush, getchar */ #include /* malloc, free */ #include /* strcmp, strlen */ #include /* clock */ @@ -253,7 +253,7 @@ static int LZ4IO_GetBlockSize_FromBlockId (int id) { return (1 << (8 + (2 * id)) static int LZ4IO_isSkippableMagicNumber(unsigned int magic) { return (magic & LZ4IO_SKIPPABLEMASK) == LZ4IO_SKIPPABLE0; } -static int get_fileHandle(const char* input_filename, const char* output_filename, FILE** pfinput, FILE** pfoutput) +static int LZ4IO_getFiles(const char* input_filename, const char* output_filename, FILE** pfinput, FILE** pfoutput) { if (!strcmp (input_filename, stdinmark)) @@ -267,6 +267,12 @@ static int get_fileHandle(const char* input_filename, const char* output_filenam *pfinput = fopen(input_filename, "rb"); } + if ( *pfinput==0 ) + { + DISPLAYLEVEL(2, "Unable to access file for processing: %s\n", input_filename); + return 1; + } + if (!strcmp (output_filename, stdoutmark)) { DISPLAYLEVEL(4,"Using stdout for output\n"); @@ -285,16 +291,17 @@ static int get_fileHandle(const char* input_filename, const char* output_filenam { char ch; DISPLAYLEVEL(2, "Warning : %s already exists\n", output_filename); - DISPLAYLEVEL(2, "Overwrite ? (Y/N) : "); - if (g_displayLevel <= 1) EXM_THROW(11, "Operation aborted : %s already exists", output_filename); /* No interaction possible */ - ch = (char)getchar(); - if ((ch!='Y') && (ch!='y')) EXM_THROW(11, "Operation aborted : %s already exists", output_filename); + if ((g_displayLevel <= 1) || (*pfinput == stdin)) + EXM_THROW(11, "Operation aborted : %s already exists", output_filename); /* No interaction possible */ + DISPLAYLEVEL(2, "Overwrite ? (Y/n) : "); + ch = 'Y'; + while((ch = (char)getchar()) != '\n' && ch != EOF) /* flush integrated */ + if ((ch!='Y') && (ch!='y')) EXM_THROW(12, "No. Operation aborted : %s already exists", output_filename); } } *pfoutput = fopen( output_filename, "wb" ); } - if ( *pfinput==0 ) EXM_THROW(12, "Pb opening %s", input_filename); if ( *pfoutput==0) EXM_THROW(13, "Pb opening %s", output_filename); return 0; @@ -336,7 +343,8 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output start = clock(); if (compressionlevel < 3) compressionFunction = LZ4_compress; else compressionFunction = LZ4_compressHC; - get_fileHandle(input_filename, output_filename, &finput, &foutput); + if (LZ4IO_getFiles(input_filename, output_filename, &finput, &foutput)) + EXM_THROW(20, "File error"); if ((g_displayLevel==2) && (compressionlevel==1)) g_displayLevel=3; /* Allocate Memory */ @@ -394,59 +402,61 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output * Compression using Frame format *********************************************/ -int LZ4IO_compressFilename(const char* input_filename, const char* output_filename, int compressionLevel) +typedef struct { + const char* srcFileName; + FILE* sFile; + const char* dstFileName; + FILE* dFile; + void* srcBuffer; + size_t srcBufferSize; + void* dstBuffer; + size_t dstBufferSize; + LZ4F_compressionContext_t ctx; + unsigned compressionLevel; +} compressionJob_t; + + +static int LZ4IO_compressJob(compressionJob_t ress) { unsigned long long filesize = 0; unsigned long long compressedfilesize = 0; - char* in_buff; - char* out_buff; - FILE* finput; - FILE* foutput; - clock_t start, end; - int blockSize; - size_t sizeCheck, headerSize, readSize, outBuffSize; - LZ4F_compressionContext_t ctx; - LZ4F_errorCode_t errorCode; + FILE* finput = ress.sFile; + FILE* foutput = ress.dFile; + void* const srcBuffer = ress.srcBuffer; + void* const dstBuffer = ress.dstBuffer; + const size_t dstBufferSize = ress.dstBufferSize; + const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); + size_t sizeCheck, headerSize, readSize; + LZ4F_compressionContext_t ctx = ress.ctx; /* just a pointer */ LZ4F_preferences_t prefs; /* Init */ - start = clock(); memset(&prefs, 0, sizeof(prefs)); - errorCode = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); - get_fileHandle(input_filename, output_filename, &finput, &foutput); - blockSize = LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); /* Set compression parameters */ prefs.autoFlush = 1; - prefs.compressionLevel = compressionLevel; + prefs.compressionLevel = ress.compressionLevel; prefs.frameInfo.blockMode = (blockMode_t)g_blockIndependence; prefs.frameInfo.blockSizeID = (blockSizeID_t)g_blockSizeId; prefs.frameInfo.contentChecksumFlag = (contentChecksum_t)g_streamChecksum; if (g_contentSizeFlag) { - unsigned long long fileSize = LZ4IO_GetFileSize(input_filename); + unsigned long long fileSize = LZ4IO_GetFileSize(ress.srcFileName); prefs.frameInfo.contentSize = fileSize; /* == 0 if input == stdin */ if (fileSize==0) DISPLAYLEVEL(3, "Warning : cannot determine uncompressed frame content size \n"); } - /* Allocate Memory */ - in_buff = (char*)malloc(blockSize); - outBuffSize = LZ4F_compressBound(blockSize, &prefs); - out_buff = (char*)malloc(outBuffSize); - if (!in_buff || !out_buff) EXM_THROW(31, "Allocation error : not enough memory"); - /* Write Archive Header */ - headerSize = LZ4F_compressBegin(ctx, out_buff, outBuffSize, &prefs); + headerSize = LZ4F_compressBegin(ctx, dstBuffer, dstBufferSize, &prefs); if (LZ4F_isError(headerSize)) EXM_THROW(32, "File header generation failed : %s", LZ4F_getErrorName(headerSize)); - sizeCheck = fwrite(out_buff, 1, headerSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, headerSize, foutput); if (sizeCheck!=headerSize) EXM_THROW(33, "Write error : cannot write header"); compressedfilesize += headerSize; /* read first block */ - readSize = fread(in_buff, (size_t)1, (size_t)blockSize, finput); + readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, finput); filesize += readSize; /* Main Loop */ @@ -455,49 +465,83 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena size_t outSize; /* Compress Block */ - outSize = LZ4F_compressUpdate(ctx, out_buff, outBuffSize, in_buff, readSize, NULL); + outSize = LZ4F_compressUpdate(ctx, dstBuffer, dstBufferSize, srcBuffer, readSize, NULL); if (LZ4F_isError(outSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(outSize)); compressedfilesize += outSize; - DISPLAYUPDATE(2, "\rRead : %i MB ==> %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); + DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", (unsigned)(filesize>>20), (double)compressedfilesize/filesize*100); /* Write Block */ - sizeCheck = fwrite(out_buff, 1, outSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, outSize, foutput); if (sizeCheck!=outSize) EXM_THROW(35, "Write error : cannot write compressed block"); /* Read next block */ - readSize = fread(in_buff, (size_t)1, (size_t)blockSize, finput); + readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, finput); filesize += readSize; } /* End of Stream mark */ - headerSize = LZ4F_compressEnd(ctx, out_buff, outBuffSize, NULL); + headerSize = LZ4F_compressEnd(ctx, dstBuffer, dstBufferSize, NULL); if (LZ4F_isError(headerSize)) EXM_THROW(36, "End of file generation failed : %s", LZ4F_getErrorName(headerSize)); - sizeCheck = fwrite(out_buff, 1, headerSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, headerSize, foutput); if (sizeCheck!=headerSize) EXM_THROW(37, "Write error : cannot write end of stream"); compressedfilesize += headerSize; + /* Final Status */ + DISPLAYLEVEL(2, "\r%79s\r", ""); + if (filesize == 0) + { + DISPLAYLEVEL(2, "Empty input\n"); + } + else + { + DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", + (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); + } + + return 0; +} + + +int LZ4IO_compressFilename(const char* input_filename, const char* output_filename, int compressionLevel) +{ + clock_t start, end; + const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); + compressionJob_t ress; + LZ4F_errorCode_t errorCode; + + + /* Init */ + start = clock(); + errorCode = LZ4F_createCompressionContext(&(ress.ctx), LZ4F_VERSION); + if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); + if (LZ4IO_getFiles(input_filename, output_filename, &(ress.sFile), &(ress.dFile))) + EXM_THROW(31, "File error"); + ress.compressionLevel = (unsigned)compressionLevel; + + /* Allocate Memory */ + ress.srcBuffer = malloc(blockSize); + ress.srcBufferSize = blockSize; + ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ + ress.dstBuffer = malloc(ress.dstBufferSize); + if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); + + /* Compress */ + LZ4IO_compressJob(ress); + /* Close & Free */ - free(in_buff); - free(out_buff); - fclose(finput); - fclose(foutput); - errorCode = LZ4F_freeCompressionContext(ctx); + free(ress.srcBuffer); + free(ress.dstBuffer); + fclose(ress.sFile); + fclose(ress.dFile); + errorCode = LZ4F_freeCompressionContext(ress.ctx); if (LZ4F_isError(errorCode)) EXM_THROW(38, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); /* Final Status */ end = clock(); - DISPLAYLEVEL(2, "\r%79s\r", ""); - if (filesize == 0) { - DISPLAYLEVEL(2, "Null size input; converted into %u lz4 stream\n", (unsigned)compressedfilesize); - } - else - { - double seconds = (double)(end - start)/CLOCKS_PER_SEC; - DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", - (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); - DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); + double seconds = (double)(end - start) / CLOCKS_PER_SEC; + DISPLAYLEVEL(4, "Completed in %.2f sec \n", seconds); } return 0; @@ -505,75 +549,60 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena #define FNSPACE 30 -int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix, int compressionlevel) +int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix, int compressionLevel) { int i; int missing_files = 0; char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); + const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); + compressionJob_t ress; + LZ4F_errorCode_t errorCode; + /* init */ + errorCode = LZ4F_createCompressionContext(&(ress.ctx), LZ4F_VERSION); + if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); + ress.compressionLevel = (unsigned)compressionLevel; + + /* Allocate Memory */ + ress.srcBuffer = malloc(blockSize); + ress.srcBufferSize = blockSize; + ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ + ress.dstBuffer = malloc(ress.dstBufferSize); + if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); + + /* loop on each file */ for (i=0; i 0) return 1; return 0; } -int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix) -{ - int i; - int skipped_files = 0; - int missing_files = 0; - char* outFileName = (char*)malloc(FNSPACE); - size_t ofnSize = FNSPACE; - const size_t suffixSize = strlen(suffix); - char* ifnSuffix = (char*)malloc(suffixSize + 1); - - for (i=0; i 0) return 1; - if (missing_files > 0) return 1; - return 0; -} /* ********************************************************************* */ /* ********************** LZ4 file-stream Decompression **************** */ @@ -585,7 +614,7 @@ static unsigned LZ4IO_readLE32 (const void* s) unsigned value32 = srcPtr[0]; value32 += (srcPtr[1]<<8); value32 += (srcPtr[2]<<16); - value32 += (srcPtr[3]<<24); + value32 += ((unsigned)srcPtr[3])<<24; return value32; } @@ -866,7 +895,8 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file /* Init */ start = clock(); - get_fileHandle(input_filename, output_filename, &finput, &foutput); + if (LZ4IO_getFiles(input_filename, output_filename, &finput, &foutput)) + EXM_THROW(50, "File error"); /* sparse file */ if (g_sparseFileSupport) { SET_SPARSE_FILE_MODE(foutput); } @@ -897,3 +927,44 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file return 0; } + +int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix) +{ + int i; + int skipped_files = 0; + int missing_files = 0; + char* outFileName = (char*)malloc(FNSPACE); + size_t ofnSize = FNSPACE; + const size_t suffixSize = strlen(suffix); + char* ifnSuffix = (char*)malloc(suffixSize + 1); + + for (i=0; i 0) return 1; + if (missing_files > 0) return 1; + return 0; +} From 113b150f0aeb3512b2850a65a824ad87f605187e Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 16:05:02 +0100 Subject: [PATCH 065/121] Fix leak issue with compression of multiple files --- programs/lz4io.c | 274 +++++++++++++++++++++++++++++------------------ 1 file changed, 167 insertions(+), 107 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 44c894c..01375be 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -159,7 +159,7 @@ static const int maxBlockSizeID = 7; #define EXTENDED_ARGUMENTS #define EXTENDED_HELP #define EXTENDED_FORMAT -#define DEFAULT_DECOMPRESSOR decodeLZ4S +#define DEFAULT_DECOMPRESSOR LZ4IO_decompressFile /* ************************************************** */ @@ -240,7 +240,7 @@ static unsigned long long LZ4IO_GetFileSize(const char* infilename) struct stat statbuf; r = stat(infilename, &statbuf); #endif - if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ + if (r || !S_ISREG(statbuf.st_mode)) return 0; /* failure, or is not a regular file */ return (unsigned long long)statbuf.st_size; } @@ -289,12 +289,11 @@ static int LZ4IO_getFiles(const char* input_filename, const char* output_filenam fclose(*pfoutput); if (!g_overwrite) { - char ch; + char ch = 'Y'; DISPLAYLEVEL(2, "Warning : %s already exists\n", output_filename); if ((g_displayLevel <= 1) || (*pfinput == stdin)) EXM_THROW(11, "Operation aborted : %s already exists", output_filename); /* No interaction possible */ DISPLAYLEVEL(2, "Overwrite ? (Y/n) : "); - ch = 'Y'; while((ch = (char)getchar()) != '\n' && ch != EOF) /* flush integrated */ if ((ch!='Y') && (ch!='y')) EXM_THROW(12, "No. Operation aborted : %s already exists", output_filename); } @@ -520,6 +519,7 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena ress.compressionLevel = (unsigned)compressionLevel; /* Allocate Memory */ + ress.srcFileName = input_filename; ress.srcBuffer = malloc(blockSize); ress.srcBufferSize = blockSize; ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ @@ -587,14 +587,17 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, issueWithSrcFile = LZ4IO_getFiles(inFileNamesTable[i], outFileName, &(ress.sFile), &(ress.dFile)); missing_files += issueWithSrcFile; - if (!issueWithSrcFile) LZ4IO_compressJob(ress); + if (!issueWithSrcFile) + { + LZ4IO_compressJob(ress); + fclose(ress.sFile); + fclose(ress.dFile); + } } /* Close & Free */ free(ress.srcBuffer); free(ress.dstBuffer); - fclose(ress.sFile); - fclose(ress.dFile); errorCode = LZ4F_freeCompressionContext(ress.ctx); if (LZ4F_isError(errorCode)) EXM_THROW(38, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); free(outFileName); @@ -618,12 +621,101 @@ static unsigned LZ4IO_readLE32 (const void* s) return value32; } +static unsigned LZ4IO_fwriteSparse(FILE* file, const void* buffer, size_t bufferSize, unsigned storedSkips) +{ + size_t* const bufferT = (size_t*)buffer; /* Buffer is supposed malloc'ed, hence aligned on size_t */ + size_t* ptrT = bufferT; + size_t bufferSizeT = bufferSize / sizeT; + size_t* const bufferTEnd = bufferT + bufferSizeT; + static const size_t segmentSizeT = (32 KB) / sizeT; + + if (!g_sparseFileSupport) /* normal write */ + { + size_t sizeCheck = fwrite(buffer, 1, bufferSize, file); + if (sizeCheck != bufferSize) EXM_THROW(68, "Write error : cannot write decoded block"); + return 0; + } + + /* avoid int overflow */ + if (storedSkips > 1 GB) + { + int seekResult = fseek(file, 1 GB, SEEK_CUR); + if (seekResult != 0) EXM_THROW(68, "1 GB skip error (sparse file support)"); + storedSkips -= 1 GB; + } + + while (ptrT < bufferTEnd) + { + size_t seg0SizeT = segmentSizeT; + size_t nb0T; + int seekResult; + + /* count leading zeros */ + if (seg0SizeT > bufferSizeT) seg0SizeT = bufferSizeT; + bufferSizeT -= seg0SizeT; + for (nb0T=0; (nb0T < seg0SizeT) && (ptrT[nb0T] == 0); nb0T++) ; + storedSkips += (unsigned)(nb0T * sizeT); + + if (nb0T != seg0SizeT) /* not all 0s */ + { + size_t sizeCheck; + seekResult = fseek(file, storedSkips, SEEK_CUR); + if (seekResult) EXM_THROW(68, "Skip error (sparse file)"); + storedSkips = 0; + seg0SizeT -= nb0T; + ptrT += nb0T; + sizeCheck = fwrite(ptrT, sizeT, seg0SizeT, file); + if (sizeCheck != seg0SizeT) EXM_THROW(68, "Write error : cannot write decoded block"); + } + ptrT += seg0SizeT; + } + + if (bufferSize & maskT) /* size not multiple of sizeT : implies end of block */ + { + const char* const restStart = (char*)bufferTEnd; + const char* restPtr = restStart; + size_t restSize = bufferSize & maskT; + const char* const restEnd = restStart + restSize; + for (; (restPtr < restEnd) && (*restPtr == 0); restPtr++) ; + storedSkips += (unsigned) (restPtr - restStart); + if (restPtr != restEnd) + { + size_t sizeCheck; + int seekResult = fseek(file, storedSkips, SEEK_CUR); + if (seekResult) EXM_THROW(68, "Skip error (end of block)"); + storedSkips = 0; + sizeCheck = fwrite(restPtr, 1, restEnd - restPtr, file); + if (sizeCheck != (size_t)(restEnd - restPtr)) EXM_THROW(68, "Write error : cannot write decoded end of block"); + } + } + + return storedSkips; +} + +static void LZ4IO_fwriteSparseEnd(FILE* file, unsigned storedSkips) +{ + char lastZeroByte[1] = { 0 }; + + if (storedSkips>0) /* implies g_sparseFileSupport */ + { + int seekResult; + size_t sizeCheck; + storedSkips --; + seekResult = fseek(file, storedSkips, SEEK_CUR); + if (seekResult != 0) EXM_THROW(69, "Final skip error (sparse file)\n"); + sizeCheck = fwrite(lastZeroByte, 1, 1, file); + if (sizeCheck != 1) EXM_THROW(69, "Write error : cannot write last zero\n"); + } +} + + static unsigned g_magicRead = 0; -static unsigned long long decodeLegacyStream(FILE* finput, FILE* foutput) +static unsigned long long LZ4IO_decodeLegacyStream(FILE* finput, FILE* foutput) { unsigned long long filesize = 0; char* in_buff; char* out_buff; + unsigned storedSkips = 0; /* Allocate Memory */ in_buff = (char*)malloc(LZ4_compressBound(LEGACY_BLOCKSIZE)); @@ -657,10 +749,11 @@ static unsigned long long decodeLegacyStream(FILE* finput, FILE* foutput) filesize += decodeSize; /* Write Block */ - sizeCheck = fwrite(out_buff, 1, decodeSize, foutput); - if (sizeCheck != (size_t)decodeSize) EXM_THROW(54, "Write error : cannot write decoded block into output\n"); + storedSkips = LZ4IO_fwriteSparse(foutput, out_buff, decodeSize, storedSkips); } + LZ4IO_fwriteSparseEnd(foutput, storedSkips); + /* Free */ free(in_buff); free(out_buff); @@ -669,37 +762,32 @@ static unsigned long long decodeLegacyStream(FILE* finput, FILE* foutput) } -static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) + +typedef struct { + FILE* sFile; + FILE* dFile; + void* srcBuffer; + size_t srcBufferSize; + void* dstBuffer; + size_t dstBufferSize; + LZ4F_decompressionContext_t dCtx; +} decompressionJob_t; + +static unsigned long long LZ4IO_decompressJob(decompressionJob_t ress) { unsigned long long filesize = 0; - void* inBuff; - void* outBuff; - size_t sizeCheck; - const size_t inBuffSize = 256 KB; - const size_t outBuffSize = 256 KB; - LZ4F_decompressionContext_t ctx; LZ4F_errorCode_t errorCode; unsigned storedSkips = 0; - /* init */ - errorCode = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) EXM_THROW(60, "Can't create context : %s", LZ4F_getErrorName(errorCode)); - - /* Allocate Memory */ - inBuff = malloc(256 KB); - outBuff = malloc(256 KB); - if (!inBuff || !outBuff) EXM_THROW(61, "Allocation error : not enough memory"); - - /* Init feed with magic number (already consumed from FILE) */ + /* Init feed with magic number (already consumed from FILE* sFile) */ { - size_t inSize = 4; - size_t outSize=0; - LZ4IO_writeLE32(inBuff, LZ4IO_MAGICNUMBER); - errorCode = LZ4F_decompress(ctx, outBuff, &outSize, inBuff, &inSize, NULL); + size_t inSize = MAGICNUMBER_SIZE; + size_t outSize= 0; + LZ4IO_writeLE32(ress.srcBuffer, LZ4IO_MAGICNUMBER); + errorCode = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &outSize, ress.srcBuffer, &inSize, NULL); if (LZ4F_isError(errorCode)) EXM_THROW(62, "Header error : %s", LZ4F_getErrorName(errorCode)); } - /* Main Loop */ for (;;) { @@ -707,15 +795,15 @@ static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) size_t pos = 0; /* Read input */ - readSize = fread(inBuff, 1, inBuffSize, finput); + readSize = fread(ress.srcBuffer, 1, ress.srcBufferSize, ress.sFile); if (!readSize) break; /* empty file or stream */ while (pos < readSize) { /* Decode Input (at least partially) */ size_t remaining = readSize - pos; - size_t decodedBytes = outBuffSize; - errorCode = LZ4F_decompress(ctx, outBuff, &decodedBytes, (char*)inBuff+pos, &remaining, NULL); + size_t decodedBytes = ress.dstBufferSize; + errorCode = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &decodedBytes, (char*)(ress.srcBuffer)+pos, &remaining, NULL); if (LZ4F_isError(errorCode)) EXM_THROW(66, "Decompression error : %s", LZ4F_getErrorName(errorCode)); pos += remaining; @@ -724,83 +812,54 @@ static unsigned long long decodeLZ4S(FILE* finput, FILE* foutput) /* Write Block */ filesize += decodedBytes; DISPLAYUPDATE(2, "\rDecompressed : %u MB ", (unsigned)(filesize>>20)); - if (g_sparseFileSupport) - { - size_t* const oBuffStartT = (size_t*)outBuff; /* since outBuff is malloc'ed, it's aligned on size_t */ - size_t* oBuffPosT = oBuffStartT; - size_t oBuffSizeT = decodedBytes / sizeT; - size_t* const oBuffEndT = oBuffStartT + oBuffSizeT; - static const size_t bs0T = (32 KB) / sizeT; - while (oBuffPosT < oBuffEndT) - { - size_t seg0SizeT = bs0T; - size_t nb0T; - int seekResult; - if (seg0SizeT > oBuffSizeT) seg0SizeT = oBuffSizeT; - oBuffSizeT -= seg0SizeT; - for (nb0T=0; (nb0T < seg0SizeT) && (oBuffPosT[nb0T] == 0); nb0T++) ; - storedSkips += (unsigned)(nb0T * sizeT); - if (storedSkips > 1 GB) /* avoid int overflow */ - { - seekResult = fseek(foutput, 1 GB, SEEK_CUR); - if (seekResult != 0) EXM_THROW(68, "1 GB skip error (sparse file)"); - storedSkips -= 1 GB; - } - if (nb0T != seg0SizeT) /* not all 0s */ - { - seekResult = fseek(foutput, storedSkips, SEEK_CUR); - if (seekResult) EXM_THROW(68, "Skip error (sparse file)"); - storedSkips = 0; - seg0SizeT -= nb0T; - oBuffPosT += nb0T; - sizeCheck = fwrite(oBuffPosT, sizeT, seg0SizeT, foutput); - if (sizeCheck != seg0SizeT) EXM_THROW(68, "Write error : cannot write decoded block"); - } - oBuffPosT += seg0SizeT; - } - if (decodedBytes & maskT) /* size not multiple of sizeT (necessarily end of block) */ - { - const char* const restStart = (char*)oBuffEndT; - const char* restPtr = restStart; - size_t restSize = decodedBytes & maskT; - const char* const restEnd = restStart + restSize; - for (; (restPtr < restEnd) && (*restPtr == 0); restPtr++) ; - storedSkips += (unsigned) (restPtr - restStart); - if (restPtr != restEnd) - { - int seekResult = fseek(foutput, storedSkips, SEEK_CUR); - if (seekResult) EXM_THROW(68, "Skip error (end of block)"); - storedSkips = 0; - sizeCheck = fwrite(restPtr, 1, restEnd - restPtr, foutput); - if (sizeCheck != (size_t)(restEnd - restPtr)) EXM_THROW(68, "Write error : cannot write decoded end of block"); - } - } - } - else - { - sizeCheck = fwrite(outBuff, 1, decodedBytes, foutput); - if (sizeCheck != decodedBytes) EXM_THROW(68, "Write error : cannot write decoded block"); - } + storedSkips = LZ4IO_fwriteSparse(ress.dFile, ress.dstBuffer, decodedBytes, storedSkips); } } - } - if ((g_sparseFileSupport) && (storedSkips>0)) - { - int seekResult; - storedSkips --; - seekResult = fseek(foutput, storedSkips, SEEK_CUR); - if (seekResult != 0) EXM_THROW(69, "Final skip error (sparse file)\n"); - memset(outBuff, 0, 1); - sizeCheck = fwrite(outBuff, 1, 1, foutput); - if (sizeCheck != 1) EXM_THROW(69, "Write error : cannot write last zero\n"); - } + LZ4IO_fwriteSparseEnd(ress.dFile, storedSkips); + + return filesize; +} + + +static const size_t LZ4IO_dBufferSize = 64 KB; + +static unsigned long long LZ4IO_decompressFile(FILE* finput, FILE* foutput) +{ + unsigned long long filesize = 0; + void* inBuff; + void* outBuff; + const size_t inBuffSize = LZ4IO_dBufferSize; + const size_t outBuffSize = LZ4IO_dBufferSize; + LZ4F_decompressionContext_t dCtx; + LZ4F_errorCode_t errorCode; + decompressionJob_t ress; + + /* init */ + errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) EXM_THROW(60, "Can't create context : %s", LZ4F_getErrorName(errorCode)); + + /* Allocate Memory */ + inBuff = malloc(inBuffSize); + outBuff = malloc(outBuffSize); + if (!inBuff || !outBuff) EXM_THROW(61, "Allocation error : not enough memory"); + + /* Decompression Job */ + ress.dCtx = dCtx; + ress.dFile = foutput; + ress.dstBuffer = outBuff; + ress.dstBufferSize = outBuffSize; + ress.sFile = finput; + ress.srcBuffer = inBuff; + ress.srcBufferSize = inBuffSize; + + filesize = LZ4IO_decompressJob(ress); /* Free */ free(inBuff); free(outBuff); - errorCode = LZ4F_freeDecompressionContext(ctx); + errorCode = LZ4F_freeDecompressionContext(dCtx); if (LZ4F_isError(errorCode)) EXM_THROW(69, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); return filesize; @@ -812,6 +871,7 @@ static unsigned long long LZ4IO_passThrough(FILE* finput, FILE* foutput, unsigne void* buffer = malloc(64 KB); size_t read = 1, sizeCheck; unsigned long long total = MAGICNUMBER_SIZE; + unsigned storedSkips = 0; sizeCheck = fwrite(U32store, 1, MAGICNUMBER_SIZE, foutput); if (sizeCheck != MAGICNUMBER_SIZE) EXM_THROW(50, "Pass-through error at start"); @@ -820,10 +880,10 @@ static unsigned long long LZ4IO_passThrough(FILE* finput, FILE* foutput, unsigne { read = fread(buffer, 1, 64 KB, finput); total += read; - sizeCheck = fwrite(buffer, 1, read, foutput); - if (sizeCheck != read) EXM_THROW(50, "Pass-through error"); + storedSkips = LZ4IO_fwriteSparse(foutput, buffer, read, storedSkips); } + LZ4IO_fwriteSparseEnd(foutput, storedSkips); free(buffer); return total; } @@ -862,7 +922,7 @@ static unsigned long long selectDecoder( FILE* finput, FILE* foutput) return DEFAULT_DECOMPRESSOR(finput, foutput); case LEGACY_MAGICNUMBER: DISPLAYLEVEL(4, "Detected : Legacy format \n"); - return decodeLegacyStream(finput, foutput); + return LZ4IO_decodeLegacyStream(finput, foutput); case LZ4IO_SKIPPABLE0: DISPLAYLEVEL(4, "Skipping detected skippable area \n"); nbReadBytes = fread(U32store, 1, 4, finput); From eabc6d8d0054853274cd7839c252a0ae4bb8d929 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 16:34:37 +0100 Subject: [PATCH 066/121] New valgrind test with multiple files --- programs/Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/programs/Makefile b/programs/Makefile index 2b0b8a9..1caaab4 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -247,6 +247,9 @@ test-mem: lz4 datagen fuzzer frametest fullbench valgrind --leak-check=yes ./datagen -g50M > $(VOID) ./datagen -g16KB > tmp valgrind --leak-check=yes ./lz4 -9 -BD -f tmp $(VOID) + ./datagen -g16KB -s2 > tmp2 + ./datagen -g16KB -s3 > tmp3 + valgrind --leak-check=yes ./lz4 --force --multiple tmp tmp2 tmp3 ./datagen -g16MB > tmp valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp tmp2 valgrind --leak-check=yes ./lz4 -t tmp2 From 348f5099e42c23393ec9c8e6b403f4bca99bbb41 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 18:10:55 +0100 Subject: [PATCH 067/121] lz4io refactoring --- programs/lz4io.c | 145 +++++++++++++++++++++-------------------------- 1 file changed, 66 insertions(+), 79 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 01375be..9287979 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -402,25 +402,48 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output *********************************************/ typedef struct { - const char* srcFileName; - FILE* sFile; - const char* dstFileName; - FILE* dFile; - void* srcBuffer; + void* srcBuffer; size_t srcBufferSize; - void* dstBuffer; + void* dstBuffer; size_t dstBufferSize; LZ4F_compressionContext_t ctx; - unsigned compressionLevel; -} compressionJob_t; +} cRess_t; + +static cRess_t LZ4IO_createCResources(void) +{ + const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); + cRess_t ress; + LZ4F_errorCode_t errorCode; + + errorCode = LZ4F_createCompressionContext(&(ress.ctx), LZ4F_VERSION); + if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); + + /* Allocate Memory */ + ress.srcBuffer = malloc(blockSize); + ress.srcBufferSize = blockSize; + ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ + ress.dstBuffer = malloc(ress.dstBufferSize); + if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); + + return ress; +} + +static void LZ4IO_freeCResources(cRess_t ress) +{ + LZ4F_errorCode_t errorCode; + free(ress.srcBuffer); + free(ress.dstBuffer); + errorCode = LZ4F_freeCompressionContext(ress.ctx); + if (LZ4F_isError(errorCode)) EXM_THROW(38, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); +} -static int LZ4IO_compressJob(compressionJob_t ress) +static int LZ4IO_compressJob(cRess_t ress, const char* srcFileName, const char* dstFileName, int compressionLevel) { unsigned long long filesize = 0; unsigned long long compressedfilesize = 0; - FILE* finput = ress.sFile; - FILE* foutput = ress.dFile; + FILE* srcFile; + FILE* dstFile; void* const srcBuffer = ress.srcBuffer; void* const dstBuffer = ress.dstBuffer; const size_t dstBufferSize = ress.dstBufferSize; @@ -433,15 +456,21 @@ static int LZ4IO_compressJob(compressionJob_t ress) /* Init */ memset(&prefs, 0, sizeof(prefs)); + /* File check */ + { + int srcFileAbsent = LZ4IO_getFiles(srcFileName, dstFileName, &srcFile, &dstFile); + if (srcFileAbsent) return srcFileAbsent; + } + /* Set compression parameters */ prefs.autoFlush = 1; - prefs.compressionLevel = ress.compressionLevel; + prefs.compressionLevel = compressionLevel; prefs.frameInfo.blockMode = (blockMode_t)g_blockIndependence; prefs.frameInfo.blockSizeID = (blockSizeID_t)g_blockSizeId; prefs.frameInfo.contentChecksumFlag = (contentChecksum_t)g_streamChecksum; if (g_contentSizeFlag) { - unsigned long long fileSize = LZ4IO_GetFileSize(ress.srcFileName); + unsigned long long fileSize = LZ4IO_GetFileSize(srcFileName); prefs.frameInfo.contentSize = fileSize; /* == 0 if input == stdin */ if (fileSize==0) DISPLAYLEVEL(3, "Warning : cannot determine uncompressed frame content size \n"); @@ -450,12 +479,12 @@ static int LZ4IO_compressJob(compressionJob_t ress) /* Write Archive Header */ headerSize = LZ4F_compressBegin(ctx, dstBuffer, dstBufferSize, &prefs); if (LZ4F_isError(headerSize)) EXM_THROW(32, "File header generation failed : %s", LZ4F_getErrorName(headerSize)); - sizeCheck = fwrite(dstBuffer, 1, headerSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); if (sizeCheck!=headerSize) EXM_THROW(33, "Write error : cannot write header"); compressedfilesize += headerSize; /* read first block */ - readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, finput); + readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, srcFile); filesize += readSize; /* Main Loop */ @@ -470,11 +499,11 @@ static int LZ4IO_compressJob(compressionJob_t ress) DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", (unsigned)(filesize>>20), (double)compressedfilesize/filesize*100); /* Write Block */ - sizeCheck = fwrite(dstBuffer, 1, outSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, outSize, dstFile); if (sizeCheck!=outSize) EXM_THROW(35, "Write error : cannot write compressed block"); /* Read next block */ - readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, finput); + readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, srcFile); filesize += readSize; } @@ -482,10 +511,14 @@ static int LZ4IO_compressJob(compressionJob_t ress) headerSize = LZ4F_compressEnd(ctx, dstBuffer, dstBufferSize, NULL); if (LZ4F_isError(headerSize)) EXM_THROW(36, "End of file generation failed : %s", LZ4F_getErrorName(headerSize)); - sizeCheck = fwrite(dstBuffer, 1, headerSize, foutput); + sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); if (sizeCheck!=headerSize) EXM_THROW(37, "Write error : cannot write end of stream"); compressedfilesize += headerSize; + /* Release files */ + fclose (srcFile); + fclose (dstFile); + /* Final Status */ DISPLAYLEVEL(2, "\r%79s\r", ""); if (filesize == 0) @@ -502,40 +535,21 @@ static int LZ4IO_compressJob(compressionJob_t ress) } -int LZ4IO_compressFilename(const char* input_filename, const char* output_filename, int compressionLevel) +int LZ4IO_compressFilename(const char* srcFileName, const char* dstFileName, int compressionLevel) { clock_t start, end; - const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); - compressionJob_t ress; - LZ4F_errorCode_t errorCode; - + cRess_t ress; + int issueWithSrcFile = 0; /* Init */ start = clock(); - errorCode = LZ4F_createCompressionContext(&(ress.ctx), LZ4F_VERSION); - if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); - if (LZ4IO_getFiles(input_filename, output_filename, &(ress.sFile), &(ress.dFile))) - EXM_THROW(31, "File error"); - ress.compressionLevel = (unsigned)compressionLevel; + ress = LZ4IO_createCResources(); - /* Allocate Memory */ - ress.srcFileName = input_filename; - ress.srcBuffer = malloc(blockSize); - ress.srcBufferSize = blockSize; - ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ - ress.dstBuffer = malloc(ress.dstBufferSize); - if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); + /* Compress File */ + issueWithSrcFile += LZ4IO_compressJob(ress, srcFileName, dstFileName, compressionLevel); - /* Compress */ - LZ4IO_compressJob(ress); - - /* Close & Free */ - free(ress.srcBuffer); - free(ress.dstBuffer); - fclose(ress.sFile); - fclose(ress.dFile); - errorCode = LZ4F_freeCompressionContext(ress.ctx); - if (LZ4F_isError(errorCode)) EXM_THROW(38, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); + /* Free resources */ + LZ4IO_freeCResources(ress); /* Final Status */ end = clock(); @@ -544,7 +558,7 @@ int LZ4IO_compressFilename(const char* input_filename, const char* output_filena DISPLAYLEVEL(4, "Completed in %.2f sec \n", seconds); } - return 0; + return issueWithSrcFile; } @@ -556,54 +570,27 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); - const size_t blockSize = (size_t)LZ4IO_GetBlockSize_FromBlockId (g_blockSizeId); - compressionJob_t ress; - LZ4F_errorCode_t errorCode; + cRess_t ress; /* init */ - errorCode = LZ4F_createCompressionContext(&(ress.ctx), LZ4F_VERSION); - if (LZ4F_isError(errorCode)) EXM_THROW(30, "Allocation error : can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); - ress.compressionLevel = (unsigned)compressionLevel; - - /* Allocate Memory */ - ress.srcBuffer = malloc(blockSize); - ress.srcBufferSize = blockSize; - ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ - ress.dstBuffer = malloc(ress.dstBufferSize); - if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); + ress = LZ4IO_createCResources(); /* loop on each file */ for (i=0; i 0) return 1; - return 0; + return missing_files; } From 05a46fc59ab0934389055f97b7be5b5a6da2824d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 18:51:36 +0100 Subject: [PATCH 068/121] Changes LZ4F_compressBound() definition using NULL prefsPtr to cover worst case instead of default. --- lib/lz4frame.c | 3 ++- lib/lz4frame.h | 6 +++--- programs/lz4io.c | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 31cf9a5..8d4c029 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -472,12 +472,13 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds /* LZ4F_compressBound() : gives the size of Dst buffer given a srcSize to handle worst case situations. * The LZ4F_frameInfo_t structure is optional : -* you can provide NULL as argument, all preferences will then be set to default. +* you can provide NULL as argument, preferences will then be set to cover worst case situations. * */ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* preferencesPtr) { LZ4F_preferences_t prefsNull; memset(&prefsNull, 0, sizeof(prefsNull)); + prefsNull.frameInfo.contentChecksumFlag = contentChecksumEnabled; /* worst case */ { const LZ4F_preferences_t* prefsPtr = (preferencesPtr==NULL) ? &prefsNull : preferencesPtr; blockSizeID_t bid = prefsPtr->frameInfo.blockSizeID; diff --git a/lib/lz4frame.h b/lib/lz4frame.h index e5435cd..f22ef00 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -141,9 +141,9 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t cctx, void* dstBuffer, size_ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* prefsPtr); /* LZ4F_compressBound() : * Provides the minimum size of Dst buffer given srcSize to handle worst case situations. - * prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to default. - * Note that different preferences will produce different results. - * This function doesn't include frame termination cost (4 bytes, or 8 is frame checksum is enabled) + * Different preferences can produce different results. + * prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to cover worst case. + * This function includes frame termination cost (4 bytes, or 8 is frame checksum is enabled) */ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* cOptPtr); diff --git a/programs/lz4io.c b/programs/lz4io.c index 9287979..db34747 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -421,7 +421,7 @@ static cRess_t LZ4IO_createCResources(void) /* Allocate Memory */ ress.srcBuffer = malloc(blockSize); ress.srcBufferSize = blockSize; - ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* risk : real prefs may cost more */ + ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* cover worst case */ ress.dstBuffer = malloc(ress.dstBufferSize); if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); From 5a6652708439800e30d21e0cd24ed15cfb745a8b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 20:21:50 +0200 Subject: [PATCH 069/121] Revert "Add more descriptive frame errors" --- lib/lz4frame.c | 14 +++++++------- lib/lz4frame_static.h | 5 ----- 2 files changed, 7 insertions(+), 12 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index df41f63..31cf9a5 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -861,16 +861,16 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo blockSizeID = (BD>>4) & _3BITS; /* validate */ - if (version != 1) return (size_t)-ERROR_version_wrong; /* Version Number, only supported value */ - if (blockChecksumFlag != 0) return (size_t)-ERROR_unsupported_checksum; /* Only supported value for the time being */ - if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ - if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bit */ - if (blockSizeID < 4) return (size_t)-ERROR_unsupported_block_size; /* 4-7 only supported values for the time being */ - if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ + if (version != 1) return (size_t)-ERROR_GENERIC; /* Version Number, only supported value */ + if (blockChecksumFlag != 0) return (size_t)-ERROR_GENERIC; /* Only supported value for the time being */ + if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ + if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bit */ + if (blockSizeID < 4) return (size_t)-ERROR_GENERIC; /* 4-7 only supported values for the time being */ + if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ /* check */ HC = LZ4F_headerChecksum(srcPtr+4, frameHeaderSize-5); - if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_header_checksum_invalid; /* Bad header checksum error */ + if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_GENERIC; /* Bad header checksum error */ /* save */ dctxPtr->frameInfo.blockMode = (blockMode_t)blockMode; diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index b608d23..2e56400 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -57,11 +57,6 @@ extern "C" { ITEM(ERROR_wrongSrcPtr) \ ITEM(ERROR_decompressionFailed) \ ITEM(ERROR_checksum_invalid) \ - ITEM(ERROR_version_wrong) \ - ITEM(ERROR_unsupported_checksum) \ - ITEM(ERROR_reserved_flag_set) \ - ITEM(ERROR_unsupported_block_size) \ - ITEM(ERROR_header_checksum_invalid) \ ITEM(ERROR_maxCode) #define LZ4F_GENERATE_ENUM(ENUM) ENUM, From 3f4f623bc9d8635459ab45f978bd1fcfd52ddc75 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 14 Apr 2015 21:02:12 +0100 Subject: [PATCH 070/121] Valgrind tests generate errors --- programs/Makefile | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index 1caaab4..6008c7b 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -244,22 +244,22 @@ test-frametest32: frametest32 ./frametest32 $(FUZZER_TIME) test-mem: lz4 datagen fuzzer frametest fullbench - valgrind --leak-check=yes ./datagen -g50M > $(VOID) + valgrind --leak-check=yes --error-exitcode=1 ./datagen -g50M > $(VOID) ./datagen -g16KB > tmp - valgrind --leak-check=yes ./lz4 -9 -BD -f tmp $(VOID) + valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -BD -f tmp $(VOID) ./datagen -g16KB -s2 > tmp2 ./datagen -g16KB -s3 > tmp3 - valgrind --leak-check=yes ./lz4 --force --multiple tmp tmp2 tmp3 + valgrind --leak-check=yes --error-exitcode=1 ./lz4 --force --multiple tmp tmp2 tmp3 ./datagen -g16MB > tmp - valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp tmp2 - valgrind --leak-check=yes ./lz4 -t tmp2 - valgrind --leak-check=yes ./lz4 -bi1 tmp - valgrind --leak-check=yes ./fullbench -i1 tmp tmp2 + valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -B5D -f tmp tmp2 + valgrind --leak-check=yes --error-exitcode=1 ./lz4 -t tmp2 + valgrind --leak-check=yes --error-exitcode=1 ./lz4 -bi1 tmp + valgrind --leak-check=yes --error-exitcode=1 ./fullbench -i1 tmp tmp2 ./datagen -g256MB > tmp - valgrind --leak-check=yes ./lz4 -B4D -f -vq tmp $(VOID) + valgrind --leak-check=yes --error-exitcode=1 ./lz4 -B4D -f -vq tmp $(VOID) rm tmp* - valgrind --leak-check=yes ./fuzzer -i64 -t1 - valgrind --leak-check=yes ./frametest -i256 + valgrind --leak-check=yes --error-exitcode=1 ./fuzzer -i64 -t1 + valgrind --leak-check=yes --error-exitcode=1 ./frametest -i256 test-mem32: lz4c32 datagen # unfortunately, valgrind doesn't seem to work with non-native binary. If someone knows how to do a valgrind-test on a 32-bits exe with a 64-bits system... From b664a72145f5e48fbbb82b8efb52c72518b41aaf Mon Sep 17 00:00:00 2001 From: Charles Allen Date: Tue, 14 Apr 2015 13:33:43 -0700 Subject: [PATCH 071/121] Revert "Revert "Add more descriptive frame errors"" This reverts commit 5a6652708439800e30d21e0cd24ed15cfb745a8b. --- lib/lz4frame.c | 14 +++++++------- lib/lz4frame_static.h | 5 +++++ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 8d4c029..41f8b1e 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -862,16 +862,16 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo blockSizeID = (BD>>4) & _3BITS; /* validate */ - if (version != 1) return (size_t)-ERROR_GENERIC; /* Version Number, only supported value */ - if (blockChecksumFlag != 0) return (size_t)-ERROR_GENERIC; /* Only supported value for the time being */ - if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ - if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bit */ - if (blockSizeID < 4) return (size_t)-ERROR_GENERIC; /* 4-7 only supported values for the time being */ - if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_GENERIC; /* Reserved bits */ + if (version != 1) return (size_t)-ERROR_version_wrong; /* Version Number, only supported value */ + if (blockChecksumFlag != 0) return (size_t)-ERROR_unsupported_checksum; /* Only supported value for the time being */ + if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ + if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bit */ + if (blockSizeID < 4) return (size_t)-ERROR_unsupported_block_size; /* 4-7 only supported values for the time being */ + if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ /* check */ HC = LZ4F_headerChecksum(srcPtr+4, frameHeaderSize-5); - if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_GENERIC; /* Bad header checksum error */ + if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_header_checksum_invalid; /* Bad header checksum error */ /* save */ dctxPtr->frameInfo.blockMode = (blockMode_t)blockMode; diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index 2e56400..b608d23 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -57,6 +57,11 @@ extern "C" { ITEM(ERROR_wrongSrcPtr) \ ITEM(ERROR_decompressionFailed) \ ITEM(ERROR_checksum_invalid) \ + ITEM(ERROR_version_wrong) \ + ITEM(ERROR_unsupported_checksum) \ + ITEM(ERROR_reserved_flag_set) \ + ITEM(ERROR_unsupported_block_size) \ + ITEM(ERROR_header_checksum_invalid) \ ITEM(ERROR_maxCode) #define LZ4F_GENERATE_ENUM(ENUM) ENUM, From d7298d2059a4f2da7aa19122d3af2aacb931972b Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 00:17:06 +0900 Subject: [PATCH 072/121] Replace GCC_VERSION with LZ4_GCC_VERSION --- lib/lz4.c | 12 ++++++------ lib/lz4.h | 6 +++--- lib/lz4hc.h | 6 +++--- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index d8e4977..53cd4a9 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -94,8 +94,8 @@ # endif /* __STDC_VERSION__ */ #endif /* _MSC_VER */ -/* GCC_VERSION is defined into lz4.h */ -#if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) +/* LZ4_GCC_VERSION is defined into lz4.h */ +#if (LZ4_GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) #else # define expect(expr,value) (expr) @@ -261,7 +261,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanForward64( &r, (U64)val ); return (int)(r>>3); -# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctzll((U64)val) >> 3); # else static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 }; @@ -274,7 +274,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r; _BitScanForward( &r, (U32)val ); return (int)(r>>3); -# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctz((U32)val) >> 3); # else static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 }; @@ -290,7 +290,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanReverse64( &r, val ); return (unsigned)(r>>3); -# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_clzll((U64)val) >> 3); # else unsigned r; @@ -306,7 +306,7 @@ static unsigned LZ4_NbCommonBytes (register size_t val) unsigned long r = 0; _BitScanReverse( &r, (unsigned long)val ); return (unsigned)(r>>3); -# elif (defined(__clang__) || (GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) +# elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_clz((U32)val) >> 3); # else unsigned r; diff --git a/lib/lz4.h b/lib/lz4.h index 20e3c01..e831055 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -288,10 +288,10 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ #ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK # define LZ4_DEPRECATE_WARNING_DEFBLOCK -# define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) -# if (GCC_VERSION >= 405) || defined(__clang__) +# define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) +# if (LZ4_GCC_VERSION >= 405) || defined(__clang__) # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) -# elif (GCC_VERSION >= 301) +# elif (LZ4_GCC_VERSION >= 301) # define LZ4_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) diff --git a/lib/lz4hc.h b/lib/lz4hc.h index 95bb6fe..d7db3bd 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -145,10 +145,10 @@ using LZ4_saveDictHC(). You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ #ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK # define LZ4_DEPRECATE_WARNING_DEFBLOCK -# define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) -# if (GCC_VERSION >= 405) || defined(__clang__) +# define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) +# if (LZ4_GCC_VERSION >= 405) || defined(__clang__) # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) -# elif (GCC_VERSION >= 301) +# elif (LZ4_GCC_VERSION >= 301) # define LZ4_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) From e1283c7991b9b55348480dd114af1aa47410811b Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 00:19:13 +0900 Subject: [PATCH 073/121] Fix LZ4_DEPRECATED() for older/non-gcc/clang/MSVC compilers --- lib/lz4.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4.h b/lib/lz4.h index e831055..98296e9 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -297,7 +297,7 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") -# define LZ4_DEPRECATED +# define LZ4_DEPRECATED(message) # endif #endif // LZ4_DEPRECATE_WARNING_DEFBLOCK From 081bcca33bba4f6ac3e913b3cf2bc2ba58b6b697 Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 02:20:47 +0900 Subject: [PATCH 074/121] Issue#90 : Add LZ4F_ prefix - Add LZ4F_ prefix to LZ4 frame related enums. - Also add conditional macro LZ4F_DISABLE_OLD_ENUMS for testing and users who prefer strict prefixes. --- lib/lz4frame.c | 116 +++++++++++++++++++++--------------------- lib/lz4frame.h | 19 ++++--- lib/lz4frame_static.h | 6 ++- programs/lz4io.c | 6 +-- 4 files changed, 79 insertions(+), 68 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 41f8b1e..619481d 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -101,7 +101,7 @@ typedef unsigned long long U64; #define LZ4F_MAGICNUMBER 0x184D2204U #define LZ4F_BLOCKUNCOMPRESSED_FLAG 0x80000000U #define LZ4F_MAXHEADERFRAME_SIZE 15 -#define LZ4F_BLOCKSIZEID_DEFAULT max64KB +#define LZ4F_BLOCKSIZEID_DEFAULT LZ4F_max64KB static const size_t minFHSize = 5; static const U32 minHClevel = 3; @@ -157,7 +157,7 @@ static const char* LZ4F_errorStrings[] = { LZ4F_LIST_ERRORS(LZ4F_GENERATE_STRING unsigned LZ4F_isError(LZ4F_errorCode_t code) { - return (code > (LZ4F_errorCode_t)(-ERROR_maxCode)); + return (code > (LZ4F_errorCode_t)(-LZ4F_ERROR_maxCode)); } const char* LZ4F_getErrorName(LZ4F_errorCode_t code) @@ -177,7 +177,7 @@ static size_t LZ4F_getBlockSize(unsigned blockSizeID) if (blockSizeID == 0) blockSizeID = LZ4F_BLOCKSIZEID_DEFAULT; blockSizeID -= 4; - if (blockSizeID > 3) return (size_t)-ERROR_maxBlockSize_invalid; + if (blockSizeID > 3) return (size_t)-LZ4F_ERROR_maxBlockSize_invalid; return blockSizes[blockSizeID]; } @@ -236,15 +236,15 @@ static BYTE LZ4F_headerChecksum (const void* header, size_t length) /************************************** * Simple compression functions **************************************/ -static blockSizeID_t LZ4F_optimalBSID(const blockSizeID_t requestedBSID, const size_t srcSize) +static LZ4F_blockSizeID_t LZ4F_optimalBSID(const LZ4F_blockSizeID_t requestedBSID, const size_t srcSize) { - blockSizeID_t proposedBSID = max64KB; + LZ4F_blockSizeID_t proposedBSID = LZ4F_max64KB; size_t maxBlockSize = 64 KB; while (requestedBSID > proposedBSID) { if (srcSize <= maxBlockSize) return proposedBSID; - proposedBSID = (blockSizeID_t)((int)proposedBSID + 1); + proposedBSID = (LZ4F_blockSizeID_t)((int)proposedBSID + 1); maxBlockSize <<= 2; } return requestedBSID; @@ -314,12 +314,12 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf prefs.frameInfo.blockSizeID = LZ4F_optimalBSID(prefs.frameInfo.blockSizeID, srcSize); prefs.autoFlush = 1; if (srcSize <= LZ4F_getBlockSize(prefs.frameInfo.blockSizeID)) - prefs.frameInfo.blockMode = blockIndependent; /* no need for linked blocks */ + prefs.frameInfo.blockMode = LZ4F_blockIndependent; /* no need for linked blocks */ options.stableSrc = 1; if (dstMaxSize < LZ4F_compressFrameBound(srcSize, &prefs)) - return (size_t)-ERROR_dstMaxSize_tooSmall; + return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; errorCode = LZ4F_compressBegin(&cctxI, dstBuffer, dstMaxSize, &prefs); /* write header */ if (LZ4F_isError(errorCode)) return errorCode; @@ -357,14 +357,14 @@ LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* LZ4F_c LZ4F_cctx_internal_t* cctxPtr; cctxPtr = (LZ4F_cctx_internal_t*)ALLOCATOR(sizeof(LZ4F_cctx_internal_t)); - if (cctxPtr==NULL) return (LZ4F_errorCode_t)(-ERROR_allocation_failed); + if (cctxPtr==NULL) return (LZ4F_errorCode_t)(-LZ4F_ERROR_allocation_failed); cctxPtr->version = version; cctxPtr->cStage = 0; /* Next stage : write header */ *LZ4F_compressionContextPtr = (LZ4F_compressionContext_t)cctxPtr; - return OK_NoError; + return LZ4F_OK_NoError; } @@ -379,7 +379,7 @@ LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_comp FREEMEM(LZ4F_compressionContext); } - return OK_NoError; + return LZ4F_OK_NoError; } @@ -398,8 +398,8 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds BYTE* headerStart; size_t requiredBuffSize; - if (dstMaxSize < LZ4F_MAXHEADERFRAME_SIZE) return (size_t)-ERROR_dstMaxSize_tooSmall; - if (cctxPtr->cStage != 0) return (size_t)-ERROR_GENERIC; + if (dstMaxSize < LZ4F_MAXHEADERFRAME_SIZE) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; + if (cctxPtr->cStage != 0) return (size_t)-LZ4F_ERROR_GENERIC; memset(&prefNull, 0, sizeof(prefNull)); if (preferencesPtr == NULL) preferencesPtr = &prefNull; cctxPtr->prefs = *preferencesPtr; @@ -422,16 +422,16 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds if (cctxPtr->prefs.frameInfo.blockSizeID == 0) cctxPtr->prefs.frameInfo.blockSizeID = LZ4F_BLOCKSIZEID_DEFAULT; cctxPtr->maxBlockSize = LZ4F_getBlockSize(cctxPtr->prefs.frameInfo.blockSizeID); - requiredBuffSize = cctxPtr->maxBlockSize + ((cctxPtr->prefs.frameInfo.blockMode == blockLinked) * 128 KB); + requiredBuffSize = cctxPtr->maxBlockSize + ((cctxPtr->prefs.frameInfo.blockMode == LZ4F_blockLinked) * 128 KB); if (preferencesPtr->autoFlush) - requiredBuffSize = (cctxPtr->prefs.frameInfo.blockMode == blockLinked) * 64 KB; /* just needs dict */ + requiredBuffSize = (cctxPtr->prefs.frameInfo.blockMode == LZ4F_blockLinked) * 64 KB; /* just needs dict */ if (cctxPtr->maxBufferSize < requiredBuffSize) { cctxPtr->maxBufferSize = requiredBuffSize; FREEMEM(cctxPtr->tmpBuff); cctxPtr->tmpBuff = (BYTE*)ALLOCATOR(requiredBuffSize); - if (cctxPtr->tmpBuff == NULL) return (size_t)-ERROR_allocation_failed; + if (cctxPtr->tmpBuff == NULL) return (size_t)-LZ4F_ERROR_allocation_failed; } cctxPtr->tmpIn = cctxPtr->tmpBuff; cctxPtr->tmpInSize = 0; @@ -481,7 +481,7 @@ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* preferencesP prefsNull.frameInfo.contentChecksumFlag = contentChecksumEnabled; /* worst case */ { const LZ4F_preferences_t* prefsPtr = (preferencesPtr==NULL) ? &prefsNull : preferencesPtr; - blockSizeID_t bid = prefsPtr->frameInfo.blockSizeID; + LZ4F_blockSizeID_t bid = prefsPtr->frameInfo.blockSizeID; size_t blockSize = LZ4F_getBlockSize(bid); unsigned nbBlocks = (unsigned)(srcSize / blockSize) + 1; size_t lastBlockSize = prefsPtr->autoFlush ? srcSize % blockSize : blockSize; @@ -530,14 +530,14 @@ static int LZ4F_localLZ4_compressHC_limitedOutput_continue(void* ctx, const char return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); } -static compressFunc_t LZ4F_selectCompression(blockMode_t blockMode, U32 level) +static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, U32 level) { if (level < minHClevel) { - if (blockMode == blockIndependent) return LZ4F_localLZ4_compress_limitedOutput_withState; + if (blockMode == LZ4F_blockIndependent) return LZ4F_localLZ4_compress_limitedOutput_withState; return LZ4F_localLZ4_compress_limitedOutput_continue; } - if (blockMode == blockIndependent) return LZ4_compressHC2_limitedOutput_withStateHC; + if (blockMode == LZ4F_blockIndependent) return LZ4_compressHC2_limitedOutput_withStateHC; return LZ4F_localLZ4_compressHC_limitedOutput_continue; } @@ -572,8 +572,8 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d compressFunc_t compress; - if (cctxPtr->cStage != 1) return (size_t)-ERROR_GENERIC; - if (dstMaxSize < LZ4F_compressBound(srcSize, &(cctxPtr->prefs))) return (size_t)-ERROR_dstMaxSize_tooSmall; + if (cctxPtr->cStage != 1) return (size_t)-LZ4F_ERROR_GENERIC; + if (dstMaxSize < LZ4F_compressBound(srcSize, &(cctxPtr->prefs))) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; memset(&cOptionsNull, 0, sizeof(cOptionsNull)); if (compressOptionsPtr == NULL) compressOptionsPtr = &cOptionsNull; @@ -601,7 +601,7 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d dstPtr += LZ4F_compressBlock(dstPtr, cctxPtr->tmpIn, blockSize, compress, cctxPtr->lz4CtxPtr, cctxPtr->prefs.compressionLevel); - if (cctxPtr->prefs.frameInfo.blockMode==blockLinked) cctxPtr->tmpIn += blockSize; + if (cctxPtr->prefs.frameInfo.blockMode==LZ4F_blockLinked) cctxPtr->tmpIn += blockSize; cctxPtr->tmpInSize = 0; } } @@ -623,7 +623,7 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d } /* preserve dictionary if necessary */ - if ((cctxPtr->prefs.frameInfo.blockMode==blockLinked) && (lastBlockCompressed==fromSrcBuffer)) + if ((cctxPtr->prefs.frameInfo.blockMode==LZ4F_blockLinked) && (lastBlockCompressed==fromSrcBuffer)) { if (compressOptionsPtr->stableSrc) { @@ -632,13 +632,13 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d else { int realDictSize = LZ4F_localSaveDict(cctxPtr); - if (realDictSize==0) return (size_t)-ERROR_GENERIC; + if (realDictSize==0) return (size_t)-LZ4F_ERROR_GENERIC; cctxPtr->tmpIn = cctxPtr->tmpBuff + realDictSize; } } /* keep tmpIn within limits */ - if ((cctxPtr->tmpIn + blockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize) /* necessarily blockLinked && lastBlockCompressed==fromTmpBuffer */ + if ((cctxPtr->tmpIn + blockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize) /* necessarily LZ4F_blockLinked && lastBlockCompressed==fromTmpBuffer */ && !(cctxPtr->prefs.autoFlush)) { int realDictSize = LZ4F_localSaveDict(cctxPtr); @@ -654,7 +654,7 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d cctxPtr->tmpInSize = sizeToCopy; } - if (cctxPtr->prefs.frameInfo.contentChecksumFlag == contentChecksumEnabled) + if (cctxPtr->prefs.frameInfo.contentChecksumFlag == LZ4F_contentChecksumEnabled) XXH32_update(&(cctxPtr->xxh), srcBuffer, srcSize); cctxPtr->totalInSize += srcSize; @@ -679,8 +679,8 @@ size_t LZ4F_flush(LZ4F_compressionContext_t compressionContext, void* dstBuffer, if (cctxPtr->tmpInSize == 0) return 0; /* nothing to flush */ - if (cctxPtr->cStage != 1) return (size_t)-ERROR_GENERIC; - if (dstMaxSize < (cctxPtr->tmpInSize + 8)) return (size_t)-ERROR_dstMaxSize_tooSmall; /* +8 : block header(4) + block checksum(4) */ + if (cctxPtr->cStage != 1) return (size_t)-LZ4F_ERROR_GENERIC; + if (dstMaxSize < (cctxPtr->tmpInSize + 8)) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; /* +8 : block header(4) + block checksum(4) */ (void)compressOptionsPtr; /* not yet useful */ /* select compression function */ @@ -688,11 +688,11 @@ size_t LZ4F_flush(LZ4F_compressionContext_t compressionContext, void* dstBuffer, /* compress tmp buffer */ dstPtr += LZ4F_compressBlock(dstPtr, cctxPtr->tmpIn, cctxPtr->tmpInSize, compress, cctxPtr->lz4CtxPtr, cctxPtr->prefs.compressionLevel); - if (cctxPtr->prefs.frameInfo.blockMode==blockLinked) cctxPtr->tmpIn += cctxPtr->tmpInSize; + if (cctxPtr->prefs.frameInfo.blockMode==LZ4F_blockLinked) cctxPtr->tmpIn += cctxPtr->tmpInSize; cctxPtr->tmpInSize = 0; /* keep tmpIn within limits */ - if ((cctxPtr->tmpIn + cctxPtr->maxBlockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize)) /* necessarily blockLinked */ + if ((cctxPtr->tmpIn + cctxPtr->maxBlockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize)) /* necessarily LZ4F_blockLinked */ { int realDictSize = LZ4F_localSaveDict(cctxPtr); cctxPtr->tmpIn = cctxPtr->tmpBuff + realDictSize; @@ -725,7 +725,7 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t compressionContext, void* dstB LZ4F_writeLE32(dstPtr, 0); dstPtr+=4; /* endMark */ - if (cctxPtr->prefs.frameInfo.contentChecksumFlag == contentChecksumEnabled) + if (cctxPtr->prefs.frameInfo.contentChecksumFlag == LZ4F_contentChecksumEnabled) { U32 xxh = XXH32_digest(&(cctxPtr->xxh)); LZ4F_writeLE32(dstPtr, xxh); @@ -737,7 +737,7 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t compressionContext, void* dstB if (cctxPtr->prefs.frameInfo.contentSize) { if (cctxPtr->prefs.frameInfo.contentSize != cctxPtr->totalInSize) - return (size_t)-ERROR_frameSize_wrong; + return (size_t)-LZ4F_ERROR_frameSize_wrong; } return dstPtr - dstStart; @@ -762,11 +762,11 @@ LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* LZ LZ4F_dctx_internal_t* dctxPtr; dctxPtr = (LZ4F_dctx_internal_t*)ALLOCATOR(sizeof(LZ4F_dctx_internal_t)); - if (dctxPtr==NULL) return (LZ4F_errorCode_t)-ERROR_GENERIC; + if (dctxPtr==NULL) return (LZ4F_errorCode_t)-LZ4F_ERROR_GENERIC; dctxPtr->version = versionNumber; *LZ4F_decompressionContextPtr = (LZ4F_compressionContext_t)dctxPtr; - return OK_NoError; + return LZ4F_OK_NoError; } LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_decompressionContext) @@ -778,7 +778,7 @@ LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_ FREEMEM(dctxPtr->tmpOutBuffer); FREEMEM(dctxPtr); } - return OK_NoError; + return LZ4F_OK_NoError; } @@ -812,13 +812,13 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo const BYTE* srcPtr = (const BYTE*)srcVoidPtr; /* need to decode header to get frameInfo */ - if (srcSize < minFHSize) return (size_t)-ERROR_GENERIC; /* minimal header size */ + if (srcSize < minFHSize) return (size_t)-LZ4F_ERROR_GENERIC; /* minimal header size */ memset(&(dctxPtr->frameInfo), 0, sizeof(dctxPtr->frameInfo)); /* skippable frames */ if ((LZ4F_readLE32(srcPtr) & 0xFFFFFFF0U) == LZ4F_MAGIC_SKIPPABLE_START) { - dctxPtr->frameInfo.frameType = skippableFrame; + dctxPtr->frameInfo.frameType = LZ4F_skippableFrame; if (srcVoidPtr == (void*)(dctxPtr->header)) { dctxPtr->tmpInSize = srcSize; @@ -834,7 +834,7 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo } /* control magic number */ - if (LZ4F_readLE32(srcPtr) != LZ4F_MAGICNUMBER) return (size_t)-ERROR_frameType_unknown; + if (LZ4F_readLE32(srcPtr) != LZ4F_MAGICNUMBER) return (size_t)-LZ4F_ERROR_frameType_unknown; dctxPtr->frameInfo.frameType = LZ4F_frame; /* Flags */ @@ -874,9 +874,9 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_header_checksum_invalid; /* Bad header checksum error */ /* save */ - dctxPtr->frameInfo.blockMode = (blockMode_t)blockMode; - dctxPtr->frameInfo.contentChecksumFlag = (contentChecksum_t)contentChecksumFlag; - dctxPtr->frameInfo.blockSizeID = (blockSizeID_t)blockSizeID; + dctxPtr->frameInfo.blockMode = (LZ4F_blockMode_t)blockMode; + dctxPtr->frameInfo.contentChecksumFlag = (LZ4F_contentChecksum_t)contentChecksumFlag; + dctxPtr->frameInfo.blockSizeID = (LZ4F_blockSizeID_t)blockSizeID; dctxPtr->maxBlockSize = LZ4F_getBlockSize(blockSizeID); if (contentSizeFlag) dctxPtr->frameRemainingSize = dctxPtr->frameInfo.contentSize = LZ4F_readLE64(srcPtr+6); @@ -885,16 +885,16 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo if (contentChecksumFlag) XXH32_reset(&(dctxPtr->xxh), 0); /* alloc */ - bufferNeeded = dctxPtr->maxBlockSize + ((dctxPtr->frameInfo.blockMode==blockLinked) * 128 KB); + bufferNeeded = dctxPtr->maxBlockSize + ((dctxPtr->frameInfo.blockMode==LZ4F_blockLinked) * 128 KB); if (bufferNeeded > dctxPtr->maxBufferSize) /* tmp buffers too small */ { FREEMEM(dctxPtr->tmpIn); FREEMEM(dctxPtr->tmpOutBuffer); dctxPtr->maxBufferSize = bufferNeeded; dctxPtr->tmpIn = (BYTE*)ALLOCATOR(dctxPtr->maxBlockSize); - if (dctxPtr->tmpIn == NULL) return (size_t)-ERROR_GENERIC; + if (dctxPtr->tmpIn == NULL) return (size_t)-LZ4F_ERROR_GENERIC; dctxPtr->tmpOutBuffer= (BYTE*)ALLOCATOR(dctxPtr->maxBufferSize); - if (dctxPtr->tmpOutBuffer== NULL) return (size_t)-ERROR_GENERIC; + if (dctxPtr->tmpOutBuffer== NULL) return (size_t)-LZ4F_ERROR_GENERIC; } dctxPtr->tmpInSize = 0; dctxPtr->tmpInTarget = 0; @@ -1058,7 +1058,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, /* expect to continue decoding src buffer where it left previously */ if (dctxPtr->srcExpect != NULL) { - if (srcStart != dctxPtr->srcExpect) return (size_t)-ERROR_wrongSrcPtr; + if (srcStart != dctxPtr->srcExpect) return (size_t)-LZ4F_ERROR_wrongSrcPtr; } /* programmed as a state machine */ @@ -1143,7 +1143,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, dctxPtr->dStage = dstage_getSuffix; break; } - if (nextCBlockSize > dctxPtr->maxBlockSize) return (size_t)-ERROR_GENERIC; /* invalid cBlockSize */ + if (nextCBlockSize > dctxPtr->maxBlockSize) return (size_t)-LZ4F_ERROR_GENERIC; /* invalid cBlockSize */ dctxPtr->tmpInTarget = nextCBlockSize; if (LZ4F_readLE32(selectedIn) & LZ4F_BLOCKUNCOMPRESSED_FLAG) { @@ -1169,7 +1169,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= sizeToCopy; /* dictionary management */ - if (dctxPtr->frameInfo.blockMode==blockLinked) + if (dctxPtr->frameInfo.blockMode==LZ4F_blockLinked) LZ4F_updateDict(dctxPtr, dstPtr, sizeToCopy, dstStart, 0); srcPtr += sizeToCopy; @@ -1231,18 +1231,18 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, int (*decoder)(const char*, char*, int, int, const char*, int); int decodedSize; - if (dctxPtr->frameInfo.blockMode == blockLinked) + if (dctxPtr->frameInfo.blockMode == LZ4F_blockLinked) decoder = LZ4_decompress_safe_usingDict; else decoder = LZ4F_decompress_safe; decodedSize = decoder((const char*)selectedIn, (char*)dstPtr, (int)dctxPtr->tmpInTarget, (int)dctxPtr->maxBlockSize, (const char*)dctxPtr->dict, (int)dctxPtr->dictSize); - if (decodedSize < 0) return (size_t)-ERROR_GENERIC; /* decompression failed */ + if (decodedSize < 0) return (size_t)-LZ4F_ERROR_GENERIC; /* decompression failed */ if (dctxPtr->frameInfo.contentChecksumFlag) XXH32_update(&(dctxPtr->xxh), dstPtr, decodedSize); if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= decodedSize; /* dictionary management */ - if (dctxPtr->frameInfo.blockMode==blockLinked) + if (dctxPtr->frameInfo.blockMode==LZ4F_blockLinked) LZ4F_updateDict(dctxPtr, dstPtr, decodedSize, dstStart, 0); dstPtr += decodedSize; @@ -1256,13 +1256,13 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, int (*decoder)(const char*, char*, int, int, const char*, int); int decodedSize; - if (dctxPtr->frameInfo.blockMode == blockLinked) + if (dctxPtr->frameInfo.blockMode == LZ4F_blockLinked) decoder = LZ4_decompress_safe_usingDict; else decoder = LZ4F_decompress_safe; /* ensure enough place for tmpOut */ - if (dctxPtr->frameInfo.blockMode == blockLinked) + if (dctxPtr->frameInfo.blockMode == LZ4F_blockLinked) { if (dctxPtr->dict == dctxPtr->tmpOutBuffer) { @@ -1283,7 +1283,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, /* Decode */ decodedSize = decoder((const char*)selectedIn, (char*)dctxPtr->tmpOut, (int)dctxPtr->tmpInTarget, (int)dctxPtr->maxBlockSize, (const char*)dctxPtr->dict, (int)dctxPtr->dictSize); - if (decodedSize < 0) return (size_t)-ERROR_decompressionFailed; /* decompression failed */ + if (decodedSize < 0) return (size_t)-LZ4F_ERROR_decompressionFailed; /* decompression failed */ if (dctxPtr->frameInfo.contentChecksumFlag) XXH32_update(&(dctxPtr->xxh), dctxPtr->tmpOut, decodedSize); if (dctxPtr->frameInfo.contentSize) dctxPtr->frameRemainingSize -= decodedSize; dctxPtr->tmpOutSize = decodedSize; @@ -1299,7 +1299,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, memcpy(dstPtr, dctxPtr->tmpOut + dctxPtr->tmpOutStart, sizeToCopy); /* dictionary management */ - if (dctxPtr->frameInfo.blockMode==blockLinked) + if (dctxPtr->frameInfo.blockMode==LZ4F_blockLinked) LZ4F_updateDict(dctxPtr, dstPtr, sizeToCopy, dstStart, 1); dctxPtr->tmpOutStart += sizeToCopy; @@ -1319,7 +1319,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, case dstage_getSuffix: { size_t suffixSize = dctxPtr->frameInfo.contentChecksumFlag * 4; - if (dctxPtr->frameRemainingSize) return (size_t)-ERROR_frameSize_wrong; /* incorrect frame size decoded */ + if (dctxPtr->frameRemainingSize) return (size_t)-LZ4F_ERROR_frameSize_wrong; /* incorrect frame size decoded */ if (suffixSize == 0) /* frame completed */ { nextSrcSizeHint = 0; @@ -1360,7 +1360,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, { U32 readCRC = LZ4F_readLE32(selectedIn); U32 resultCRC = XXH32_digest(&(dctxPtr->xxh)); - if (readCRC != resultCRC) return (size_t)-ERROR_checksum_invalid; + if (readCRC != resultCRC) return (size_t)-LZ4F_ERROR_checksum_invalid; nextSrcSizeHint = 0; dctxPtr->dStage = dstage_getHeader; doAnotherStage = 0; @@ -1425,7 +1425,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, } /* preserve dictionary within tmp if necessary */ - if ( (dctxPtr->frameInfo.blockMode==blockLinked) + if ( (dctxPtr->frameInfo.blockMode==LZ4F_blockLinked) &&(dctxPtr->dict != dctxPtr->tmpOutBuffer) &&(!decompressOptionsPtr->stableDst) &&((unsigned)(dctxPtr->dStage-1) < (unsigned)(dstage_getSuffix-1)) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index f22ef00..1f13a07 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -62,16 +62,23 @@ const char* LZ4F_getErrorName(LZ4F_errorCode_t code); /* return error code str /************************************** * Frame compression types * ************************************/ -typedef enum { LZ4F_default=0, max64KB=4, max256KB=5, max1MB=6, max4MB=7 } blockSizeID_t; +typedef enum { LZ4F_default=0, LZ4F_max64KB=4, LZ4F_max256KB=5, LZ4F_max1MB=6, LZ4F_max4MB=7 } LZ4F_blockSizeID_t; +typedef enum { LZ4F_blockLinked=0, LZ4F_blockIndependent} LZ4F_blockMode_t; +typedef enum { LZ4F_noContentChecksum=0, LZ4F_contentChecksumEnabled } LZ4F_contentChecksum_t; +typedef enum { LZ4F_frame=0, LZ4F_skippableFrame } LZ4F_frameType_t; + +#ifndef LZ4F_DISABLE_OLD_ENUMS +typedef enum { /*LZ4F_default=0,*/ max64KB=4, max256KB=5, max1MB=6, max4MB=7 } blockSizeID_t; typedef enum { blockLinked=0, blockIndependent} blockMode_t; typedef enum { noContentChecksum=0, contentChecksumEnabled } contentChecksum_t; -typedef enum { LZ4F_frame=0, skippableFrame } frameType_t; +typedef enum { /*LZ4F_frame=0,*/ skippableFrame } frameType_t; +#endif typedef struct { - blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */ - blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */ - contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */ - frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */ + LZ4F_blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */ + LZ4F_blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */ + LZ4F_contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */ + LZ4F_frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */ unsigned long long contentSize; /* Size of uncompressed (original) content ; 0 == unknown */ unsigned reserved[2]; /* must be zero for forward compatibility */ } LZ4F_frameInfo_t; diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index b608d23..afcf7da 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -64,7 +64,11 @@ extern "C" { ITEM(ERROR_header_checksum_invalid) \ ITEM(ERROR_maxCode) -#define LZ4F_GENERATE_ENUM(ENUM) ENUM, +#ifndef LZ4F_DISABLE_OLD_ENUMS +#define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, ENUM = LZ4F_##ENUM, +#else +#define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, +#endif typedef enum { LZ4F_LIST_ERRORS(LZ4F_GENERATE_ENUM) } LZ4F_errorCodes; /* enum is exposed, to handle specific errors; compare function result to -enum value */ diff --git a/programs/lz4io.c b/programs/lz4io.c index db34747..6ca7ee7 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -465,9 +465,9 @@ static int LZ4IO_compressJob(cRess_t ress, const char* srcFileName, const char* /* Set compression parameters */ prefs.autoFlush = 1; prefs.compressionLevel = compressionLevel; - prefs.frameInfo.blockMode = (blockMode_t)g_blockIndependence; - prefs.frameInfo.blockSizeID = (blockSizeID_t)g_blockSizeId; - prefs.frameInfo.contentChecksumFlag = (contentChecksum_t)g_streamChecksum; + prefs.frameInfo.blockMode = (LZ4F_blockMode_t)g_blockIndependence; + prefs.frameInfo.blockSizeID = (LZ4F_blockSizeID_t)g_blockSizeId; + prefs.frameInfo.contentChecksumFlag = (LZ4F_contentChecksum_t)g_streamChecksum; if (g_contentSizeFlag) { unsigned long long fileSize = LZ4IO_GetFileSize(srcFileName); From 585bab81217ad2480803518140ed4956aad0459c Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 02:59:51 +0900 Subject: [PATCH 075/121] Issue#90 : Change old enum to macro to maximize compatibility --- lib/lz4frame.h | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 1f13a07..d1501da 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -68,10 +68,19 @@ typedef enum { LZ4F_noContentChecksum=0, LZ4F_contentChecksumEnabled } LZ4F_cont typedef enum { LZ4F_frame=0, LZ4F_skippableFrame } LZ4F_frameType_t; #ifndef LZ4F_DISABLE_OLD_ENUMS -typedef enum { /*LZ4F_default=0,*/ max64KB=4, max256KB=5, max1MB=6, max4MB=7 } blockSizeID_t; -typedef enum { blockLinked=0, blockIndependent} blockMode_t; -typedef enum { noContentChecksum=0, contentChecksumEnabled } contentChecksum_t; -typedef enum { /*LZ4F_frame=0,*/ skippableFrame } frameType_t; +# define max64KB LZ4F_max64KB +# define max256KB LZ4F_max256KB +# define max1MB LZ4F_max1MB +# define max4MB LZ4F_max4MB +# define blockSizeID_t LZ4F_blockSizeID_t +# define blockLinked LZ4F_blockLinked +# define blockIndependent LZ4F_blockIndependent +# define blockMode_t LZ4F_blockMode_t +# define noContentChecksum LZ4F_noContentChecksum +# define contentChecksumEnabled LZ4F_contentChecksumEnabled +# define contentChecksum_t LZ4F_contentChecksum_t +# define skippableFrame LZ4F_skippableFrame +# define frameType_t LZ4F_frameType_t #endif typedef struct { From 175890fe9bcd5029ffb33e2328037e0251e89ecb Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 03:00:36 +0900 Subject: [PATCH 076/121] Issue#90 : Change old enum names to new one --- programs/frametest.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/programs/frametest.c b/programs/frametest.c index 60cd765..8a51be8 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -297,21 +297,21 @@ int basicTests(U32 seed, double compressibility) } DISPLAYLEVEL(3, "Using 64 KB block : \n"); - prefs.frameInfo.blockSizeID = max64KB; - prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled; + prefs.frameInfo.blockSizeID = LZ4F_max64KB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = noContentChecksum; + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "Using 256 KB block : \n"); - prefs.frameInfo.blockSizeID = max256KB; - prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled; + prefs.frameInfo.blockSizeID = LZ4F_max256KB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); @@ -351,33 +351,33 @@ int basicTests(U32 seed, double compressibility) } DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = noContentChecksum; + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "Using 1 MB block : \n"); - prefs.frameInfo.blockSizeID = max1MB; - prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled; + prefs.frameInfo.blockSizeID = LZ4F_max1MB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = noContentChecksum; + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "Using 4 MB block : \n"); - prefs.frameInfo.blockSizeID = max4MB; - prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled; + prefs.frameInfo.blockSizeID = LZ4F_max4MB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = noContentChecksum; + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); if (LZ4F_isError(cSize)) goto _output_error; DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); @@ -595,9 +595,9 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi memset(&prefs, 0, sizeof(prefs)); memset(&cOptions, 0, sizeof(cOptions)); memset(&dOptions, 0, sizeof(dOptions)); - prefs.frameInfo.blockMode = (blockMode_t)BMId; - prefs.frameInfo.blockSizeID = (blockSizeID_t)BSId; - prefs.frameInfo.contentChecksumFlag = (contentChecksum_t)CCflag; + prefs.frameInfo.blockMode = (LZ4F_blockMode_t)BMId; + prefs.frameInfo.blockSizeID = (LZ4F_blockSizeID_t)BSId; + prefs.frameInfo.contentChecksumFlag = (LZ4F_contentChecksum_t)CCflag; prefs.frameInfo.contentSize = frameContentSize; prefs.autoFlush = autoflush; prefs.compressionLevel = FUZ_rand(&randState) % 5; From 3d46d4b144223a69ab7e6ed327d426a7b5bdf10f Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Wed, 15 Apr 2015 14:19:17 +0900 Subject: [PATCH 077/121] Fix LZ4_DEPRECATED() in lz4hc.h --- lib/lz4hc.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4hc.h b/lib/lz4hc.h index d7db3bd..2667044 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -154,7 +154,7 @@ using LZ4_saveDictHC(). # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") -# define LZ4_DEPRECATED +# define LZ4_DEPRECATED(message) # endif #endif // LZ4_DEPRECATE_WARNING_DEFBLOCK From 4e574e7395323d7fe0edb4313d79d5ab3b2607ae Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 15 Apr 2015 10:34:08 +0100 Subject: [PATCH 078/121] Updated lz4frame error names --- lib/lz4frame.c | 20 ++++++++++---------- lib/lz4frame_static.h | 14 +++++--------- programs/Makefile | 4 ++-- programs/frametest.c | 3 ++- programs/lz4.1 | 6 +++--- 5 files changed, 22 insertions(+), 25 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 619481d..6f53897 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -478,7 +478,7 @@ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* preferencesP { LZ4F_preferences_t prefsNull; memset(&prefsNull, 0, sizeof(prefsNull)); - prefsNull.frameInfo.contentChecksumFlag = contentChecksumEnabled; /* worst case */ + prefsNull.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; /* worst case */ { const LZ4F_preferences_t* prefsPtr = (preferencesPtr==NULL) ? &prefsNull : preferencesPtr; LZ4F_blockSizeID_t bid = prefsPtr->frameInfo.blockSizeID; @@ -862,16 +862,16 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo blockSizeID = (BD>>4) & _3BITS; /* validate */ - if (version != 1) return (size_t)-ERROR_version_wrong; /* Version Number, only supported value */ - if (blockChecksumFlag != 0) return (size_t)-ERROR_unsupported_checksum; /* Only supported value for the time being */ - if (((FLG>>0)&_2BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ - if (((BD>>7)&_1BIT) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bit */ - if (blockSizeID < 4) return (size_t)-ERROR_unsupported_block_size; /* 4-7 only supported values for the time being */ - if (((BD>>0)&_4BITS) != 0) return (size_t)-ERROR_reserved_flag_set; /* Reserved bits */ + if (version != 1) return (size_t)-LZ4F_ERROR_headerVersion_wrong; /* Version Number, only supported value */ + if (blockChecksumFlag != 0) return (size_t)-LZ4F_ERROR_blockChecksum_unsupported; /* Not supported for the time being */ + if (((FLG>>0)&_2BITS) != 0) return (size_t)-LZ4F_ERROR_reservedFlag_set; /* Reserved bits */ + if (((BD>>7)&_1BIT) != 0) return (size_t)-LZ4F_ERROR_reservedFlag_set; /* Reserved bit */ + if (blockSizeID < 4) return (size_t)-LZ4F_ERROR_maxBlockSize_invalid; /* 4-7 only supported values for the time being */ + if (((BD>>0)&_4BITS) != 0) return (size_t)-LZ4F_ERROR_reservedFlag_set; /* Reserved bits */ /* check */ HC = LZ4F_headerChecksum(srcPtr+4, frameHeaderSize-5); - if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-ERROR_header_checksum_invalid; /* Bad header checksum error */ + if (HC != srcPtr[frameHeaderSize-1]) return (size_t)-LZ4F_ERROR_headerChecksum_invalid; /* Bad header checksum error */ /* save */ dctxPtr->frameInfo.blockMode = (LZ4F_blockMode_t)blockMode; @@ -1058,7 +1058,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, /* expect to continue decoding src buffer where it left previously */ if (dctxPtr->srcExpect != NULL) { - if (srcStart != dctxPtr->srcExpect) return (size_t)-LZ4F_ERROR_wrongSrcPtr; + if (srcStart != dctxPtr->srcExpect) return (size_t)-LZ4F_ERROR_srcPtr_wrong; } /* programmed as a state machine */ @@ -1360,7 +1360,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, { U32 readCRC = LZ4F_readLE32(selectedIn); U32 resultCRC = XXH32_digest(&(dctxPtr->xxh)); - if (readCRC != resultCRC) return (size_t)-LZ4F_ERROR_checksum_invalid; + if (readCRC != resultCRC) return (size_t)-LZ4F_ERROR_contentChecksum_invalid; nextSrcSizeHint = 0; dctxPtr->dStage = dstage_getHeader; doAnotherStage = 0; diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index afcf7da..25afed4 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -50,20 +50,16 @@ extern "C" { ITEM(OK_NoError) ITEM(ERROR_GENERIC) \ ITEM(ERROR_maxBlockSize_invalid) ITEM(ERROR_blockMode_invalid) ITEM(ERROR_contentChecksumFlag_invalid) \ ITEM(ERROR_compressionLevel_invalid) \ + ITEM(ERROR_headerVersion_wrong) ITEM(ERROR_blockChecksum_unsupported) ITEM(ERROR_reservedFlag_set) \ ITEM(ERROR_allocation_failed) \ ITEM(ERROR_srcSize_tooLarge) ITEM(ERROR_dstMaxSize_tooSmall) \ - ITEM(ERROR_frameSize_wrong) \ - ITEM(ERROR_frameType_unknown) \ - ITEM(ERROR_wrongSrcPtr) \ + ITEM(ERROR_frameType_unknown) ITEM(ERROR_frameSize_wrong) \ + ITEM(ERROR_srcPtr_wrong) \ ITEM(ERROR_decompressionFailed) \ - ITEM(ERROR_checksum_invalid) \ - ITEM(ERROR_version_wrong) \ - ITEM(ERROR_unsupported_checksum) \ - ITEM(ERROR_reserved_flag_set) \ - ITEM(ERROR_unsupported_block_size) \ - ITEM(ERROR_header_checksum_invalid) \ + ITEM(ERROR_headerChecksum_invalid) ITEM(ERROR_contentChecksum_invalid) \ ITEM(ERROR_maxCode) +//#define LZ4F_DISABLE_OLD_ENUMS #ifndef LZ4F_DISABLE_OLD_ENUMS #define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, ENUM = LZ4F_##ENUM, #else diff --git a/programs/Makefile b/programs/Makefile index 6008c7b..7bbe060 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -226,10 +226,10 @@ test-lz4c32: lz4 lz4c32 datagen ./datagen -g6GB | ./lz4c32 -vq9BD | ./lz4 -qt test-fullbench: fullbench - ./fullbench --no-prompt $(TEST_FILES) + ./fullbench --no-prompt $(NB_LOOPS) $(TEST_FILES) test-fullbench32: fullbench32 - ./fullbench32 --no-prompt $(TEST_FILES) + ./fullbench32 --no-prompt $(NB_LOOPS) $(TEST_FILES) test-fuzzer: fuzzer ./fuzzer $(FUZZER_TIME) diff --git a/programs/frametest.c b/programs/frametest.c index 8a51be8..6adf408 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -678,7 +678,8 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi dOptions.stableDst = FUZ_rand(&randState) & 1; if (nonContiguousDst==2) dOptions.stableDst = 0; result = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, &dOptions); - if (result == (size_t)-ERROR_checksum_invalid) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); + if (result == (size_t)-LZ4F_ERROR_contentChecksum_invalid) + locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); CHECK(LZ4F_isError(result), "Decompression failed (error %i:%s)", (int)result, LZ4F_getErrorName((LZ4F_errorCode_t)result)); XXH64_update(&xxh64, op, (U32)oSize); totalOut += oSize; diff --git a/programs/lz4.1 b/programs/lz4.1 index eea248f..a9b5195 100644 --- a/programs/lz4.1 +++ b/programs/lz4.1 @@ -177,12 +177,12 @@ independently, and the resulting name of the compressed file will be block dependency (improve compression ratio) .TP .B \--[no-]frame-crc - disable stream checksum (default:enabled) + select frame checksum (default:enabled) .TP .B \--[no-]content-size - compressed file includes original size (default:not present) + header includes original size (default:not present) Note : this option can only be activated when the original size can be determined, -hence for a file. It won't work with unknown source size, such as stdin pipe. +hence for a file. It won't work with unknown source size, such as stdin or pipe. .TP .B \--[no-]sparse enable sparse file (default:disabled)(experimental) From a430b859cf552aa43215fe082e15a95c8bda5d6f Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 15 Apr 2015 16:24:43 +0100 Subject: [PATCH 079/121] Multiple files decompression refactoring --- programs/Makefile | 21 +++--- programs/lz4io.c | 179 ++++++++++++++++++++++------------------------ 2 files changed, 99 insertions(+), 101 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index 7bbe060..fa6b04d 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -191,7 +191,19 @@ test-lz4-frame-concatenation: lz4 datagen @rm *.test @echo frame concatenation test completed -test-lz4: lz4 datagen test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation +test-lz4-multiple: lz4 datagen + @echo ---- test multiple files ---- + @./datagen -s1 > tmp1 + @./datagen -s2 -g100K > tmp2 + @./datagen -s3 -g1M > tmp3 + ./lz4 -f -m tmp* + ls -ls tmp* + rm tmp1 tmp2 tmp3 + ./lz4 -df -m *.lz4 + ls -ls tmp* + @rm tmp* + +test-lz4: lz4 datagen test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation @echo ---- test lz4 basic compression/decompression ---- ./datagen -g0 | ./lz4 -v | ./lz4 -t ./datagen -g16KB | ./lz4 -9 | ./lz4 -t @@ -202,13 +214,6 @@ test-lz4: lz4 datagen test-lz4-sparse test-lz4-contentSize test-lz4-frame-concat ./datagen -g256MB | ./lz4 -vqB4D | ./lz4 -t ./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -qt ./datagen -g6GB | ./lz4 -vq9BD | ./lz4 -qt - @echo ---- test multiple input files ---- - @./datagen -s1 > file1 - @./datagen -s2 > file2 - @./datagen -s3 > file3 - ./lz4 -f -m file1 file2 file3 - ls -l file* - @rm file1 file2 file3 file1.lz4 file2.lz4 file3.lz4 @echo ---- test pass-through ---- ./datagen | ./lz4 -tf diff --git a/programs/lz4io.c b/programs/lz4io.c index 6ca7ee7..130a1de 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -159,7 +159,7 @@ static const int maxBlockSizeID = 7; #define EXTENDED_ARGUMENTS #define EXTENDED_HELP #define EXTENDED_FORMAT -#define DEFAULT_DECOMPRESSOR LZ4IO_decompressFile +#define DEFAULT_DECOMPRESSOR LZ4IO_decompressLZ4F /* ************************************************** */ @@ -751,16 +751,44 @@ static unsigned long long LZ4IO_decodeLegacyStream(FILE* finput, FILE* foutput) typedef struct { - FILE* sFile; - FILE* dFile; void* srcBuffer; size_t srcBufferSize; void* dstBuffer; size_t dstBufferSize; LZ4F_decompressionContext_t dCtx; -} decompressionJob_t; +} dRess_t; -static unsigned long long LZ4IO_decompressJob(decompressionJob_t ress) +static const size_t LZ4IO_dBufferSize = 64 KB; + +static dRess_t LZ4IO_createDResources(void) +{ + dRess_t ress; + LZ4F_errorCode_t errorCode; + + /* init */ + errorCode = LZ4F_createDecompressionContext(&ress.dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) EXM_THROW(60, "Can't create LZ4F context : %s", LZ4F_getErrorName(errorCode)); + + /* Allocate Memory */ + ress.srcBufferSize = LZ4IO_dBufferSize; + ress.srcBuffer = malloc(ress.srcBufferSize); + ress.dstBufferSize = LZ4IO_dBufferSize; + ress.dstBuffer = malloc(ress.dstBufferSize); + if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(61, "Allocation error : not enough memory"); + + return ress; +} + +static void LZ4IO_freeDResources(dRess_t ress) +{ + LZ4F_errorCode_t errorCode = LZ4F_freeDecompressionContext(ress.dCtx); + if (LZ4F_isError(errorCode)) EXM_THROW(69, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); + free(ress.srcBuffer); + free(ress.dstBuffer); +} + + +static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE* dstFile) { unsigned long long filesize = 0; LZ4F_errorCode_t errorCode; @@ -782,7 +810,7 @@ static unsigned long long LZ4IO_decompressJob(decompressionJob_t ress) size_t pos = 0; /* Read input */ - readSize = fread(ress.srcBuffer, 1, ress.srcBufferSize, ress.sFile); + readSize = fread(ress.srcBuffer, 1, ress.srcBufferSize, srcFile); if (!readSize) break; /* empty file or stream */ while (pos < readSize) @@ -799,69 +827,26 @@ static unsigned long long LZ4IO_decompressJob(decompressionJob_t ress) /* Write Block */ filesize += decodedBytes; DISPLAYUPDATE(2, "\rDecompressed : %u MB ", (unsigned)(filesize>>20)); - storedSkips = LZ4IO_fwriteSparse(ress.dFile, ress.dstBuffer, decodedBytes, storedSkips); + storedSkips = LZ4IO_fwriteSparse(dstFile, ress.dstBuffer, decodedBytes, storedSkips); } } } - LZ4IO_fwriteSparseEnd(ress.dFile, storedSkips); + LZ4IO_fwriteSparseEnd(dstFile, storedSkips); return filesize; } -static const size_t LZ4IO_dBufferSize = 64 KB; - -static unsigned long long LZ4IO_decompressFile(FILE* finput, FILE* foutput) -{ - unsigned long long filesize = 0; - void* inBuff; - void* outBuff; - const size_t inBuffSize = LZ4IO_dBufferSize; - const size_t outBuffSize = LZ4IO_dBufferSize; - LZ4F_decompressionContext_t dCtx; - LZ4F_errorCode_t errorCode; - decompressionJob_t ress; - - /* init */ - errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) EXM_THROW(60, "Can't create context : %s", LZ4F_getErrorName(errorCode)); - - /* Allocate Memory */ - inBuff = malloc(inBuffSize); - outBuff = malloc(outBuffSize); - if (!inBuff || !outBuff) EXM_THROW(61, "Allocation error : not enough memory"); - - /* Decompression Job */ - ress.dCtx = dCtx; - ress.dFile = foutput; - ress.dstBuffer = outBuff; - ress.dstBufferSize = outBuffSize; - ress.sFile = finput; - ress.srcBuffer = inBuff; - ress.srcBufferSize = inBuffSize; - - filesize = LZ4IO_decompressJob(ress); - - /* Free */ - free(inBuff); - free(outBuff); - errorCode = LZ4F_freeDecompressionContext(dCtx); - if (LZ4F_isError(errorCode)) EXM_THROW(69, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); - - return filesize; -} - - -static unsigned long long LZ4IO_passThrough(FILE* finput, FILE* foutput, unsigned char U32store[MAGICNUMBER_SIZE]) +static unsigned long long LZ4IO_passThrough(FILE* finput, FILE* foutput, unsigned char MNstore[MAGICNUMBER_SIZE]) { void* buffer = malloc(64 KB); size_t read = 1, sizeCheck; unsigned long long total = MAGICNUMBER_SIZE; unsigned storedSkips = 0; - sizeCheck = fwrite(U32store, 1, MAGICNUMBER_SIZE, foutput); - if (sizeCheck != MAGICNUMBER_SIZE) EXM_THROW(50, "Pass-through error at start"); + sizeCheck = fwrite(MNstore, 1, MAGICNUMBER_SIZE, foutput); + if (sizeCheck != MAGICNUMBER_SIZE) EXM_THROW(50, "Pass-through write error"); while (read) { @@ -877,9 +862,9 @@ static unsigned long long LZ4IO_passThrough(FILE* finput, FILE* foutput, unsigne #define ENDOFSTREAM ((unsigned long long)-1) -static unsigned long long selectDecoder( FILE* finput, FILE* foutput) +static unsigned long long selectDecoder(dRess_t ress, FILE* finput, FILE* foutput) { - unsigned char U32store[MAGICNUMBER_SIZE]; + unsigned char MNstore[MAGICNUMBER_SIZE]; unsigned magicNumber, size; int errorNb; size_t nbReadBytes; @@ -896,34 +881,34 @@ static unsigned long long selectDecoder( FILE* finput, FILE* foutput) } else { - nbReadBytes = fread(U32store, 1, MAGICNUMBER_SIZE, finput); + nbReadBytes = fread(MNstore, 1, MAGICNUMBER_SIZE, finput); if (nbReadBytes==0) return ENDOFSTREAM; /* EOF */ if (nbReadBytes != MAGICNUMBER_SIZE) EXM_THROW(40, "Unrecognized header : Magic Number unreadable"); - magicNumber = LZ4IO_readLE32(U32store); /* Little Endian format */ + magicNumber = LZ4IO_readLE32(MNstore); /* Little Endian format */ } if (LZ4IO_isSkippableMagicNumber(magicNumber)) magicNumber = LZ4IO_SKIPPABLE0; /* fold skippable magic numbers */ switch(magicNumber) { case LZ4IO_MAGICNUMBER: - return DEFAULT_DECOMPRESSOR(finput, foutput); + return LZ4IO_decompressLZ4F(ress, finput, foutput); case LEGACY_MAGICNUMBER: DISPLAYLEVEL(4, "Detected : Legacy format \n"); return LZ4IO_decodeLegacyStream(finput, foutput); case LZ4IO_SKIPPABLE0: DISPLAYLEVEL(4, "Skipping detected skippable area \n"); - nbReadBytes = fread(U32store, 1, 4, finput); + nbReadBytes = fread(MNstore, 1, 4, finput); if (nbReadBytes != 4) EXM_THROW(42, "Stream error : skippable size unreadable"); - size = LZ4IO_readLE32(U32store); /* Little Endian format */ + size = LZ4IO_readLE32(MNstore); /* Little Endian format */ errorNb = fseek(finput, size, SEEK_CUR); if (errorNb != 0) EXM_THROW(43, "Stream error : cannot skip skippable area"); - return selectDecoder(finput, foutput); + return selectDecoder(ress, finput, foutput); EXTENDED_FORMAT; default: if (nbCalls == 1) /* just started */ { if (g_overwrite) - return LZ4IO_passThrough(finput, foutput, U32store); + return LZ4IO_passThrough(finput, foutput, MNstore); EXM_THROW(44,"Unrecognized header : file cannot be decoded"); /* Wrong magic number at the beginning of 1st stream */ } DISPLAYLEVEL(2, "Stream followed by unrecognized data\n"); @@ -932,18 +917,16 @@ static unsigned long long selectDecoder( FILE* finput, FILE* foutput) } -int LZ4IO_decompressFilename(const char* input_filename, const char* output_filename) +static int LZ4IO_decompressFile_extRess(dRess_t ress, const char* input_filename, const char* output_filename) { unsigned long long filesize = 0, decodedSize=0; FILE* finput; FILE* foutput; - clock_t start, end; /* Init */ - start = clock(); if (LZ4IO_getFiles(input_filename, output_filename, &finput, &foutput)) - EXM_THROW(50, "File error"); + return 1; /* sparse file */ if (g_sparseFileSupport) { SET_SPARSE_FILE_MODE(foutput); } @@ -951,67 +934,77 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file /* Loop over multiple streams */ do { - decodedSize = selectDecoder(finput, foutput); + decodedSize = selectDecoder(ress, finput, foutput); if (decodedSize != ENDOFSTREAM) filesize += decodedSize; } while (decodedSize != ENDOFSTREAM); /* Final Status */ - end = clock(); DISPLAYLEVEL(2, "\r%79s\r", ""); DISPLAYLEVEL(2, "Successfully decoded %llu bytes \n", filesize); - if (end==start) end=start+1; - { - double seconds = (double)(end - start)/CLOCKS_PER_SEC; - DISPLAYLEVEL(4, "Done in %.2f s ==> %.2f MB/s\n", seconds, (double)filesize / seconds / 1024 / 1024); - } /* Close */ fclose(finput); fclose(foutput); - /* Error status = OK */ return 0; } +int LZ4IO_decompressFilename(const char* input_filename, const char* output_filename) +{ + dRess_t ress; + clock_t start, end; + int missingFiles = 0; + + start = clock(); + + ress = LZ4IO_createDResources(); + missingFiles += LZ4IO_decompressFile_extRess(ress, input_filename, output_filename); + LZ4IO_freeDResources(ress); + + end = clock(); + if (end==start) end=start+1; + { + double seconds = (double)(end - start)/CLOCKS_PER_SEC; + DISPLAYLEVEL(4, "Done in %.2f sec \n", seconds); + } + + return missingFiles; +} + + int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix) { int i; - int skipped_files = 0; - int missing_files = 0; + int skippedFiles = 0; + int missingFiles = 0; char* outFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); char* ifnSuffix = (char*)malloc(suffixSize + 1); + dRess_t ress; + + ress = LZ4IO_createDResources(); for (i=0; i 0) return 1; - if (missing_files > 0) return 1; - return 0; + return missingFiles + skippedFiles; } From d153aaa1c236ba464beec4171d76063b5109b18c Mon Sep 17 00:00:00 2001 From: Takayuki MATSUOKA Date: Thu, 16 Apr 2015 12:42:26 +0900 Subject: [PATCH 080/121] Add LZ4F_OBSOLETE_ENUM() to describe obsolete enums - Add LZ4F_OBSOLETE_ENUM() to describe obsolete enums - Add compatible typedefs for obsolete enum types - Remove constant macros --- lib/lz4frame.h | 62 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 44 insertions(+), 18 deletions(-) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index d1501da..428ad03 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -49,6 +49,14 @@ extern "C" { **************************************/ #include /* size_t */ +#ifndef LZ4F_OBSOLETE_ENUM +# ifndef LZ4F_DISABLE_OBSOLETE_ENUMS +# define LZ4F_OBSOLETE_ENUM(x) ,x +# else +# define LZ4F_OBSOLETE_ENUM(x) +# endif +#endif + /************************************** * Error management @@ -62,25 +70,43 @@ const char* LZ4F_getErrorName(LZ4F_errorCode_t code); /* return error code str /************************************** * Frame compression types * ************************************/ -typedef enum { LZ4F_default=0, LZ4F_max64KB=4, LZ4F_max256KB=5, LZ4F_max1MB=6, LZ4F_max4MB=7 } LZ4F_blockSizeID_t; -typedef enum { LZ4F_blockLinked=0, LZ4F_blockIndependent} LZ4F_blockMode_t; -typedef enum { LZ4F_noContentChecksum=0, LZ4F_contentChecksumEnabled } LZ4F_contentChecksum_t; -typedef enum { LZ4F_frame=0, LZ4F_skippableFrame } LZ4F_frameType_t; +typedef enum { + LZ4F_default=0, + LZ4F_max64KB=4, + LZ4F_max256KB=5, + LZ4F_max1MB=6, + LZ4F_max4MB=7 + LZ4F_OBSOLETE_ENUM(max64KB = LZ4F_max64KB) + LZ4F_OBSOLETE_ENUM(max256KB = LZ4F_max256KB) + LZ4F_OBSOLETE_ENUM(max1MB = LZ4F_max1MB) + LZ4F_OBSOLETE_ENUM(max4MB = LZ4F_max4MB) +} LZ4F_blockSizeID_t; -#ifndef LZ4F_DISABLE_OLD_ENUMS -# define max64KB LZ4F_max64KB -# define max256KB LZ4F_max256KB -# define max1MB LZ4F_max1MB -# define max4MB LZ4F_max4MB -# define blockSizeID_t LZ4F_blockSizeID_t -# define blockLinked LZ4F_blockLinked -# define blockIndependent LZ4F_blockIndependent -# define blockMode_t LZ4F_blockMode_t -# define noContentChecksum LZ4F_noContentChecksum -# define contentChecksumEnabled LZ4F_contentChecksumEnabled -# define contentChecksum_t LZ4F_contentChecksum_t -# define skippableFrame LZ4F_skippableFrame -# define frameType_t LZ4F_frameType_t +typedef enum { + LZ4F_blockLinked=0, + LZ4F_blockIndependent + LZ4F_OBSOLETE_ENUM(blockLinked = LZ4F_blockLinked) + LZ4F_OBSOLETE_ENUM(blockIndependent = LZ4F_blockIndependent) +} LZ4F_blockMode_t; + +typedef enum { + LZ4F_noContentChecksum=0, + LZ4F_contentChecksumEnabled + LZ4F_OBSOLETE_ENUM(noContentChecksum = LZ4F_noContentChecksum) + LZ4F_OBSOLETE_ENUM(contentChecksumEnabled = LZ4F_contentChecksumEnabled) +} LZ4F_contentChecksum_t; + +typedef enum { + LZ4F_frame=0, + LZ4F_skippableFrame + LZ4F_OBSOLETE_ENUM(skippableFrame = LZ4F_skippableFrame) +} LZ4F_frameType_t; + +#ifndef LZ4F_DISABLE_OBSOLETE_ENUMS +typedef LZ4F_blockSizeID_t blockSizeID_t; +typedef LZ4F_blockMode_t blockMode_t; +typedef LZ4F_frameType_t frameType_t; +typedef LZ4F_contentChecksum_t contentChecksum_t; #endif typedef struct { From 13c6e16333a63319ef0def9da6595a3bc7d2915d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 07:16:48 +0100 Subject: [PATCH 081/121] Removed status notification in multiple-files mode --- programs/Makefile | 6 +++--- programs/lz4cli.c | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index fa6b04d..6eec96e 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -193,9 +193,9 @@ test-lz4-frame-concatenation: lz4 datagen test-lz4-multiple: lz4 datagen @echo ---- test multiple files ---- - @./datagen -s1 > tmp1 - @./datagen -s2 -g100K > tmp2 - @./datagen -s3 -g1M > tmp3 + @./datagen -s1 > tmp1 2> $(VOID) + @./datagen -s2 -g100K > tmp2 2> $(VOID) + @./datagen -s3 -g1M > tmp3 2> $(VOID) ./lz4 -f -m tmp* ls -ls tmp* rm tmp1 tmp2 tmp3 diff --git a/programs/lz4cli.c b/programs/lz4cli.c index 8d7d5ca..d77ef5b 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -102,7 +102,7 @@ ***************************************/ #define DISPLAY(...) fprintf(stderr, __VA_ARGS__) #define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); } -static unsigned displayLevel = 2; /* 0 : no display ; 1: errors ; 2 : + result + interaction + warnings ; 3 : + progression; 4 : + information */ +static unsigned displayLevel = 2; /* 0 : no display ; 1: errors only ; 2 : downgradable normal ; 3 : non-downgradable normal; 4 : + information */ /************************************** @@ -521,8 +521,9 @@ int main(int argc, char** argv) /* Check if output is defined as console; trigger an error in this case */ if (!strcmp(output_filename,stdoutmark) && IS_CONSOLE(stdout) && !forceStdout) badusage(); - /* No warning message in pure pipe mode (stdin + stdout) */ + /* Downgrade notification level in pure pipe mode (stdin + stdout) and multiple file mode */ if (!strcmp(input_filename, stdinmark) && !strcmp(output_filename,stdoutmark) && (displayLevel==2)) displayLevel=1; + if ((multiple_inputs) && (displayLevel==2)) displayLevel=1; /* IO Stream/File */ From e328d41ef4124870c31cba1fe1f4eed282b8e53e Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 08:51:31 +0100 Subject: [PATCH 082/121] minor optimization for small files --- programs/lz4io.c | 91 ++++++++++++++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 38 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 130a1de..ba2a6e8 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -421,7 +421,7 @@ static cRess_t LZ4IO_createCResources(void) /* Allocate Memory */ ress.srcBuffer = malloc(blockSize); ress.srcBufferSize = blockSize; - ress.dstBufferSize = LZ4F_compressBound(blockSize, NULL); /* cover worst case */ + ress.dstBufferSize = LZ4F_compressFrameBound(blockSize, NULL); /* cover worst case */ ress.dstBuffer = malloc(ress.dstBufferSize); if (!ress.srcBuffer || !ress.dstBuffer) EXM_THROW(31, "Allocation error : not enough memory"); @@ -438,7 +438,7 @@ static void LZ4IO_freeCResources(cRess_t ress) } -static int LZ4IO_compressJob(cRess_t ress, const char* srcFileName, const char* dstFileName, int compressionLevel) +static int LZ4IO_compressFilename_extRess(cRess_t ress, const char* srcFileName, const char* dstFileName, int compressionLevel) { unsigned long long filesize = 0; unsigned long long compressedfilesize = 0; @@ -476,44 +476,63 @@ static int LZ4IO_compressJob(cRess_t ress, const char* srcFileName, const char* DISPLAYLEVEL(3, "Warning : cannot determine uncompressed frame content size \n"); } - /* Write Archive Header */ - headerSize = LZ4F_compressBegin(ctx, dstBuffer, dstBufferSize, &prefs); - if (LZ4F_isError(headerSize)) EXM_THROW(32, "File header generation failed : %s", LZ4F_getErrorName(headerSize)); - sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); - if (sizeCheck!=headerSize) EXM_THROW(33, "Write error : cannot write header"); - compressedfilesize += headerSize; - /* read first block */ - readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, srcFile); + readSize = fread(srcBuffer, (size_t)1, blockSize, srcFile); filesize += readSize; - /* Main Loop */ - while (readSize>0) + /* single-block file */ + if (readSize < blockSize) { - size_t outSize; - - /* Compress Block */ - outSize = LZ4F_compressUpdate(ctx, dstBuffer, dstBufferSize, srcBuffer, readSize, NULL); - if (LZ4F_isError(outSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(outSize)); - compressedfilesize += outSize; + /* Compress in single pass */ + size_t cSize = LZ4F_compressFrame(dstBuffer, dstBufferSize, srcBuffer, readSize, &prefs); + if (LZ4F_isError(cSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(cSize)); + compressedfilesize += cSize; DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", (unsigned)(filesize>>20), (double)compressedfilesize/filesize*100); /* Write Block */ - sizeCheck = fwrite(dstBuffer, 1, outSize, dstFile); - if (sizeCheck!=outSize) EXM_THROW(35, "Write error : cannot write compressed block"); - - /* Read next block */ - readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, srcFile); - filesize += readSize; + sizeCheck = fwrite(dstBuffer, 1, cSize, dstFile); + if (sizeCheck!=cSize) EXM_THROW(35, "Write error : cannot write compressed block"); } - /* End of Stream mark */ - headerSize = LZ4F_compressEnd(ctx, dstBuffer, dstBufferSize, NULL); - if (LZ4F_isError(headerSize)) EXM_THROW(36, "End of file generation failed : %s", LZ4F_getErrorName(headerSize)); + else - sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); - if (sizeCheck!=headerSize) EXM_THROW(37, "Write error : cannot write end of stream"); - compressedfilesize += headerSize; + /* multiple-blocks file */ + { + /* Write Archive Header */ + headerSize = LZ4F_compressBegin(ctx, dstBuffer, dstBufferSize, &prefs); + if (LZ4F_isError(headerSize)) EXM_THROW(32, "File header generation failed : %s", LZ4F_getErrorName(headerSize)); + sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); + if (sizeCheck!=headerSize) EXM_THROW(33, "Write error : cannot write header"); + compressedfilesize += headerSize; + + /* Main Loop */ + while (readSize>0) + { + size_t outSize; + + /* Compress Block */ + outSize = LZ4F_compressUpdate(ctx, dstBuffer, dstBufferSize, srcBuffer, readSize, NULL); + if (LZ4F_isError(outSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(outSize)); + compressedfilesize += outSize; + DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", (unsigned)(filesize>>20), (double)compressedfilesize/filesize*100); + + /* Write Block */ + sizeCheck = fwrite(dstBuffer, 1, outSize, dstFile); + if (sizeCheck!=outSize) EXM_THROW(35, "Write error : cannot write compressed block"); + + /* Read next block */ + readSize = fread(srcBuffer, (size_t)1, (size_t)blockSize, srcFile); + filesize += readSize; + } + + /* End of Stream mark */ + headerSize = LZ4F_compressEnd(ctx, dstBuffer, dstBufferSize, NULL); + if (LZ4F_isError(headerSize)) EXM_THROW(36, "End of file generation failed : %s", LZ4F_getErrorName(headerSize)); + + sizeCheck = fwrite(dstBuffer, 1, headerSize, dstFile); + if (sizeCheck!=headerSize) EXM_THROW(37, "Write error : cannot write end of stream"); + compressedfilesize += headerSize; + } /* Release files */ fclose (srcFile); @@ -521,14 +540,10 @@ static int LZ4IO_compressJob(cRess_t ress, const char* srcFileName, const char* /* Final Status */ DISPLAYLEVEL(2, "\r%79s\r", ""); - if (filesize == 0) - { - DISPLAYLEVEL(2, "Empty input\n"); - } - else { DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", - (unsigned long long) filesize, (unsigned long long) compressedfilesize, (double)compressedfilesize/filesize*100); + (unsigned long long) filesize, (unsigned long long) compressedfilesize, + (double)compressedfilesize/(filesize + !filesize)*100); /* avoid division by zero */ } return 0; @@ -546,7 +561,7 @@ int LZ4IO_compressFilename(const char* srcFileName, const char* dstFileName, int ress = LZ4IO_createCResources(); /* Compress File */ - issueWithSrcFile += LZ4IO_compressJob(ress, srcFileName, dstFileName, compressionLevel); + issueWithSrcFile += LZ4IO_compressFilename_extRess(ress, srcFileName, dstFileName, compressionLevel); /* Free resources */ LZ4IO_freeCResources(ress); @@ -583,7 +598,7 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, strcpy(outFileName, inFileNamesTable[i]); strcat(outFileName, suffix); - missing_files += LZ4IO_compressJob(ress, inFileNamesTable[i], outFileName, compressionLevel); + missing_files += LZ4IO_compressFilename_extRess(ress, inFileNamesTable[i], outFileName, compressionLevel); } /* Close & Free */ From 2cf8a19e9d0b462d20b004e25e6dab79a997372a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 09:57:46 +0100 Subject: [PATCH 083/121] minor header refactoring --- lib/lz4frame.h | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 428ad03..9c5ff37 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -49,14 +49,6 @@ extern "C" { **************************************/ #include /* size_t */ -#ifndef LZ4F_OBSOLETE_ENUM -# ifndef LZ4F_DISABLE_OBSOLETE_ENUMS -# define LZ4F_OBSOLETE_ENUM(x) ,x -# else -# define LZ4F_OBSOLETE_ENUM(x) -# endif -#endif - /************************************** * Error management @@ -70,6 +62,13 @@ const char* LZ4F_getErrorName(LZ4F_errorCode_t code); /* return error code str /************************************** * Frame compression types * ************************************/ +//#define LZ4F_DISABLE_OBSOLETE_ENUMS +#ifndef LZ4F_DISABLE_OBSOLETE_ENUMS +# define LZ4F_OBSOLETE_ENUM(x) ,x +#else +# define LZ4F_OBSOLETE_ENUM(x) +#endif + typedef enum { LZ4F_default=0, LZ4F_max64KB=4, @@ -110,12 +109,12 @@ typedef LZ4F_contentChecksum_t contentChecksum_t; #endif typedef struct { - LZ4F_blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */ - LZ4F_blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */ - LZ4F_contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */ - LZ4F_frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */ - unsigned long long contentSize; /* Size of uncompressed (original) content ; 0 == unknown */ - unsigned reserved[2]; /* must be zero for forward compatibility */ + LZ4F_blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */ + LZ4F_blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */ + LZ4F_contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */ + LZ4F_frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */ + unsigned long long contentSize; /* Size of uncompressed (original) content ; 0 == unknown */ + unsigned reserved[2]; /* must be zero for forward compatibility */ } LZ4F_frameInfo_t; typedef struct { @@ -185,7 +184,7 @@ size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* prefsPtr); * Provides the minimum size of Dst buffer given srcSize to handle worst case situations. * Different preferences can produce different results. * prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to cover worst case. - * This function includes frame termination cost (4 bytes, or 8 is frame checksum is enabled) + * This function includes frame termination cost (4 bytes, or 8 if frame checksum is enabled) */ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* cOptPtr); @@ -219,7 +218,7 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t * The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark), or 8 if optional frame checksum is enabled) * The function outputs an error code if it fails (can be tested using LZ4F_isError()) * The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. - * A successful call to LZ4F_compressEnd() makes cctx available again for future compression task. + * A successful call to LZ4F_compressEnd() makes cctx available again for next compression task. */ From 0ed2e7111e944abc1b7dddffd2c2a859e4b07e58 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 12:48:50 +0100 Subject: [PATCH 084/121] Static analyzer generates error codes on bug suspicion --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index b83d920..a9990df 100644 --- a/Makefile +++ b/Makefile @@ -104,10 +104,10 @@ clangtest: clean $(MAKE) all CC=clang CPPFLAGS="-Werror -Wconversion -Wno-sign-conversion" sanitize: clean - $(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T1mn" + $(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T1mn" NB_LOOPS=-i1 staticAnalyze: clean - scan-build -v $(MAKE) all CFLAGS=-g + scan-build --status-bugs -v $(MAKE) all CFLAGS=-g armtest: clean cd lib; $(MAKE) -e all CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" From 633c1ca810cb18f9e61f9cd4d90ad104305a0505 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 13:09:34 +0100 Subject: [PATCH 085/121] fixed minor leak --- programs/lz4io.c | 1 + 1 file changed, 1 insertion(+) diff --git a/programs/lz4io.c b/programs/lz4io.c index ba2a6e8..258af5b 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -1021,5 +1021,6 @@ int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSiz LZ4IO_freeDResources(ress); free(outFileName); + free(ifnSuffix); return missingFiles + skippedFiles; } From 2ed9dcc900a8a662a52d11968baf27f61a9bedd3 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 13:18:15 +0100 Subject: [PATCH 086/121] fix minor "divide by zero" risk --- programs/lz4io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index 258af5b..e2f2801 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -487,7 +487,8 @@ static int LZ4IO_compressFilename_extRess(cRess_t ress, const char* srcFileName, size_t cSize = LZ4F_compressFrame(dstBuffer, dstBufferSize, srcBuffer, readSize, &prefs); if (LZ4F_isError(cSize)) EXM_THROW(34, "Compression failed : %s", LZ4F_getErrorName(cSize)); compressedfilesize += cSize; - DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", (unsigned)(filesize>>20), (double)compressedfilesize/filesize*100); + DISPLAYUPDATE(2, "\rRead : %u MB ==> %.2f%% ", + (unsigned)(filesize>>20), (double)compressedfilesize/(filesize+!filesize)*100); /* avoid division by zero */ /* Write Block */ sizeCheck = fwrite(dstBuffer, 1, cSize, dstFile); From 9e92bee0443b45efa3de6b71f51863e8d66ead20 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 13:34:03 +0100 Subject: [PATCH 087/121] stronger arm tests --- Makefile | 4 ++-- programs/lz4io.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index a9990df..88c4016 100644 --- a/Makefile +++ b/Makefile @@ -110,8 +110,8 @@ staticAnalyze: clean scan-build --status-bugs -v $(MAKE) all CFLAGS=-g armtest: clean - cd lib; $(MAKE) -e all CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" - cd programs; $(MAKE) -e bins CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" + cd lib; $(MAKE) -e all CC=arm-linux-gnueabi-gcc CPPFLAGS="-Werror" + cd programs; $(MAKE) -e bins CC=arm-linux-gnueabi-gcc CPPFLAGS="-Werror" streaming-examples: cd examples; $(MAKE) -e test diff --git a/programs/lz4io.c b/programs/lz4io.c index e2f2801..e262304 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -289,12 +289,12 @@ static int LZ4IO_getFiles(const char* input_filename, const char* output_filenam fclose(*pfoutput); if (!g_overwrite) { - char ch = 'Y'; + int ch = 'Y'; DISPLAYLEVEL(2, "Warning : %s already exists\n", output_filename); if ((g_displayLevel <= 1) || (*pfinput == stdin)) EXM_THROW(11, "Operation aborted : %s already exists", output_filename); /* No interaction possible */ DISPLAYLEVEL(2, "Overwrite ? (Y/n) : "); - while((ch = (char)getchar()) != '\n' && ch != EOF) /* flush integrated */ + while((ch = getchar()) != '\n' && ch != EOF) /* flush integrated */ if ((ch!='Y') && (ch!='y')) EXM_THROW(12, "No. Operation aborted : %s already exists", output_filename); } } From bce2eeb9dfbca883003f21681e21df657fac90c4 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 16:27:16 +0100 Subject: [PATCH 088/121] Reclassified some notification messages as errors --- programs/lz4io.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index e262304..7d4fc55 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -269,7 +269,7 @@ static int LZ4IO_getFiles(const char* input_filename, const char* output_filenam if ( *pfinput==0 ) { - DISPLAYLEVEL(2, "Unable to access file for processing: %s\n", input_filename); + DISPLAYLEVEL(1, "Unable to access file for processing: %s\n", input_filename); return 1; } @@ -1010,7 +1010,7 @@ int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSiz if (ofnSize <= ifnSize-suffixSize+1) { free(outFileName); ofnSize = ifnSize + 20; outFileName = (char*)malloc(ofnSize); } if (ifnSize <= suffixSize || strcmp(ifnSuffix, suffix) != 0) { - DISPLAYLEVEL(2, "File extension doesn't match expected LZ4_EXTENSION (%4s); will not process file: %s\n", suffix, inFileNamesTable[i]); + DISPLAYLEVEL(1, "File extension doesn't match expected LZ4_EXTENSION (%4s); will not process file: %s\n", suffix, inFileNamesTable[i]); skippedFiles++; continue; } From 7644bee6435f05e0954b3da5f457550614a199b7 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 16 Apr 2015 16:45:58 +0100 Subject: [PATCH 089/121] test error message in multiple files mode --- programs/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/programs/Makefile b/programs/Makefile index 6eec96e..f61d708 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -201,6 +201,7 @@ test-lz4-multiple: lz4 datagen rm tmp1 tmp2 tmp3 ./lz4 -df -m *.lz4 ls -ls tmp* + ./lz4 -f -m tmp1 notHere tmp2; echo $$? @rm tmp* test-lz4: lz4 datagen test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation From 9fd4f1f9f780f1be38ad6695d62a9ba2ea426d6d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 17 Apr 2015 19:42:07 +0100 Subject: [PATCH 090/121] Sparse file support is now enabled by default --- programs/lz4.1 | 2 +- programs/lz4cli.c | 2 +- programs/lz4io.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/programs/lz4.1 b/programs/lz4.1 index a9b5195..a79fed3 100644 --- a/programs/lz4.1 +++ b/programs/lz4.1 @@ -185,7 +185,7 @@ independently, and the resulting name of the compressed file will be hence for a file. It won't work with unknown source size, such as stdin or pipe. .TP .B \--[no-]sparse - enable sparse file (default:disabled)(experimental) + sparse file support (default:enabled) .TP .B \-l use Legacy format (useful for Linux Kernel compression) diff --git a/programs/lz4cli.c b/programs/lz4cli.c index d77ef5b..da18169 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -175,7 +175,7 @@ static int usage_advanced(void) /* DISPLAY( " -BX : enable block checksum (default:disabled)\n"); *//* Option currently inactive */ DISPLAY( "--no-frame-crc : disable stream checksum (default:enabled)\n"); DISPLAY( "--content-size : compressed frame includes original size (default:not present)\n"); - DISPLAY( "--sparse : enable sparse file (default:disabled)(experimental)\n"); + DISPLAY( "--[no-]sparse : sparse file support (default:enabled)\n"); DISPLAY( "Benchmark arguments :\n"); DISPLAY( " -b : benchmark file(s)\n"); DISPLAY( " -i# : iteration loops [1-9](default : 3), benchmark mode only\n"); diff --git a/programs/lz4io.c b/programs/lz4io.c index 7d4fc55..e0c69d8 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -131,7 +131,7 @@ static int g_blockSizeId = LZ4IO_BLOCKSIZEID_DEFAULT; static int g_blockChecksum = 0; static int g_streamChecksum = 1; static int g_blockIndependence = 1; -static int g_sparseFileSupport = 0; +static int g_sparseFileSupport = 1; static int g_contentSizeFlag = 0; static const int minBlockSizeID = 4; From 47c304039954d73cea018fbd6b4eebc05877d3fc Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 18 Apr 2015 12:44:38 +0100 Subject: [PATCH 091/121] added --no-sparse test --- programs/Makefile | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index f61d708..910516f 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -151,15 +151,18 @@ test-travis: $(TRAVIS_TARGET) test-lz4-sparse: lz4 datagen @echo ---- test sparse file support ---- - ./datagen -g50M -P100 | ./lz4 -B4D | ./lz4 -dv --sparse > tmpB4 - ./datagen -g50M -P100 | ./lz4 -B5D | ./lz4 -dv --sparse > tmpB5 - ./datagen -g50M -P100 | ./lz4 -B6D | ./lz4 -dv --sparse > tmpB6 - ./datagen -g50M -P100 | ./lz4 -B7D | ./lz4 -dv --sparse > tmpB7 + ./datagen -g5M -P100 > tmpSrc + ./lz4 -B4D tmpSrc | ./lz4 -dv --sparse > tmpB4 + diff -s tmpSrc tmpB4 + ./lz4 -B5D tmpSrc | ./lz4 -dv --sparse > tmpB5 + diff -s tmpSrc tmpB5 + ./lz4 -B6D tmpSrc | ./lz4 -dv --sparse > tmpB6 + diff -s tmpSrc tmpB6 + ./lz4 -B7D tmpSrc | ./lz4 -dv --sparse > tmpB7 + diff -s tmpSrc tmpB7 + ./lz4 tmpSrc | ./lz4 -dv --no-sparse > tmpNoSparse + diff -s tmpSrc tmpNoSparse ls -ls tmp* - ./datagen -g50M -P100 | diff -s - tmpB4 - ./datagen -g50M -P100 | diff -s - tmpB5 - ./datagen -g50M -P100 | diff -s - tmpB6 - ./datagen -g50M -P100 | diff -s - tmpB7 ./datagen -s1 -g1200007 -P100 | ./lz4 | ./lz4 -dv --sparse > tmpOdd # Odd size file (to not finish on an exact nb of blocks) ./datagen -s1 -g1200007 -P100 | diff -s - tmpOdd ls -ls tmpOdd From 409f816267b00e2307fabc59cc6ddffcc605a1ec Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 19 Apr 2015 15:23:53 +0100 Subject: [PATCH 092/121] Updated LZ4F_getFrameInfo() behavior, related to uncomplete frame header decoding attempts --- lib/lz4frame.c | 82 ++++++++++++++++++++++++------------------- lib/lz4frame.h | 20 +++++------ lib/lz4frame_static.h | 16 +++++---- programs/frametest.c | 31 ++++++++++++++++ 4 files changed, 94 insertions(+), 55 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 6f53897..7fc1314 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -100,10 +100,11 @@ typedef unsigned long long U64; #define LZ4F_MAGIC_SKIPPABLE_START 0x184D2A50U #define LZ4F_MAGICNUMBER 0x184D2204U #define LZ4F_BLOCKUNCOMPRESSED_FLAG 0x80000000U -#define LZ4F_MAXHEADERFRAME_SIZE 15 #define LZ4F_BLOCKSIZEID_DEFAULT LZ4F_max64KB -static const size_t minFHSize = 5; +static const size_t minFHSize = 7; +static const size_t maxFHSize = 15; +static const size_t BHSize = 4; static const U32 minHClevel = 3; /************************************** @@ -263,7 +264,7 @@ size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* prefere prefs.frameInfo.blockSizeID = LZ4F_optimalBSID(prefs.frameInfo.blockSizeID, srcSize); prefs.autoFlush = 1; - headerSize = 15; /* header size, including magic number and frame content size*/ + headerSize = maxFHSize; /* header size, including magic number and frame content size*/ streamSize = LZ4F_compressBound(srcSize, &prefs); return headerSize + streamSize; @@ -398,7 +399,7 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds BYTE* headerStart; size_t requiredBuffSize; - if (dstMaxSize < LZ4F_MAXHEADERFRAME_SIZE) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; + if (dstMaxSize < maxFHSize) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; if (cctxPtr->cStage != 0) return (size_t)-LZ4F_ERROR_GENERIC; memset(&prefNull, 0, sizeof(prefNull)); if (preferencesPtr == NULL) preferencesPtr = &prefNull; @@ -789,8 +790,9 @@ LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_ typedef enum { dstage_getHeader=0, dstage_storeHeader, dstage_getCBlockSize, dstage_storeCBlockSize, dstage_copyDirect, - dstage_getCBlock, dstage_storeCBlock, dstage_decodeCBlock, - dstage_decodeCBlock_intoDst, dstage_decodeCBlock_intoTmp, dstage_flushOut, + dstage_getCBlock, dstage_storeCBlock, + dstage_decodeCBlock, dstage_decodeCBlock_intoDst, + dstage_decodeCBlock_intoTmp, dstage_flushOut, dstage_getSuffix, dstage_storeSuffix, dstage_getSFrameSize, dstage_storeSFrameSize, dstage_skipSkippable @@ -802,6 +804,7 @@ typedef enum { dstage_getHeader=0, dstage_storeHeader, or an error code (testable with LZ4F_isError()) output : set internal values of dctx, such as dctxPtr->frameInfo and dctxPtr->dStage. + input : srcVoidPtr points at the **beginning of the frame** */ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVoidPtr, size_t srcSize) { @@ -812,10 +815,10 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo const BYTE* srcPtr = (const BYTE*)srcVoidPtr; /* need to decode header to get frameInfo */ - if (srcSize < minFHSize) return (size_t)-LZ4F_ERROR_GENERIC; /* minimal header size */ + if (srcSize < minFHSize) return (size_t)-LZ4F_ERROR_frameHeader_incomplete; /* minimal frame header size */ memset(&(dctxPtr->frameInfo), 0, sizeof(dctxPtr->frameInfo)); - /* skippable frames */ + /* special case : skippable frames */ if ((LZ4F_readLE32(srcPtr) & 0xFFFFFFF0U) == LZ4F_MAGIC_SKIPPABLE_START) { dctxPtr->frameInfo.frameType = LZ4F_skippableFrame; @@ -846,10 +849,11 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo contentChecksumFlag = (FLG>>2) & _1BIT; /* Frame Header Size */ - frameHeaderSize = contentSizeFlag ? 15 : 7; + frameHeaderSize = contentSizeFlag ? maxFHSize : minFHSize; if (srcSize < frameHeaderSize) { + /* not enough input to fully decode frame header */ if (srcPtr != dctxPtr->header) memcpy(dctxPtr->header, srcPtr, srcSize); dctxPtr->tmpInSize = srcSize; @@ -920,28 +924,32 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo * The function result is an hint of the better srcSize to use for next call to LZ4F_decompress, * or an error code which can be tested using LZ4F_isError(). */ -LZ4F_errorCode_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t decompressionContext, LZ4F_frameInfo_t* frameInfoPtr, const void* srcBuffer, size_t* srcSizePtr) +LZ4F_errorCode_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t dCtx, LZ4F_frameInfo_t* frameInfoPtr, + const void* srcBuffer, size_t* srcSizePtr) { - LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)decompressionContext; + LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)dCtx; - if (dctxPtr->dStage == dstage_getHeader) + if (dctxPtr->dStage > dstage_storeHeader) /* note : requires dstage_* header related to be at beginning of enum */ { - size_t frameHeaderSize = LZ4F_decodeHeader(dctxPtr, srcBuffer, *srcSizePtr); - if (LZ4F_isError(frameHeaderSize)) return frameHeaderSize; - *srcSizePtr = frameHeaderSize; /* nb Bytes consumed */ - *frameInfoPtr = dctxPtr->frameInfo; /* copy into */ - dctxPtr->srcExpect = NULL; - return 4; /* nextSrcSizeHint : 4 == block header size */ + size_t o=0, i=0; + /* frameInfo already decoded */ + *srcSizePtr = 0; + *frameInfoPtr = dctxPtr->frameInfo; + return LZ4F_decompress(dCtx, NULL, &o, NULL, &i, NULL); + } + else + { + size_t o=0; + size_t nextSrcSize = LZ4F_decompress(dCtx, NULL, &o, srcBuffer, srcSizePtr, NULL); + if (dctxPtr->dStage <= dstage_storeHeader) /* note : requires dstage_* header related to be at beginning of enum */ + return (size_t)-LZ4F_ERROR_frameHeader_incomplete; + *frameInfoPtr = dctxPtr->frameInfo; + return nextSrcSize; } - - /* frameInfo already decoded */ - *srcSizePtr = 0; - *frameInfoPtr = dctxPtr->frameInfo; - return 0; } -/* redirector, with common prototype */ +/* trivial redirector, for common prototype */ static int LZ4F_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize, const char* dictStart, int dictSize) { (void)dictStart; (void)dictSize; @@ -1071,7 +1079,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, case dstage_getHeader: { - if (srcEnd-srcPtr >= 7) + if ((size_t)(srcEnd-srcPtr) >= maxFHSize) /* enough to decode - shortcut */ { LZ4F_errorCode_t errorCode = LZ4F_decodeHeader(dctxPtr, srcPtr, srcEnd-srcPtr); if (LZ4F_isError(errorCode)) return errorCode; @@ -1079,7 +1087,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, break; } dctxPtr->tmpInSize = 0; - dctxPtr->tmpInTarget = 7; + dctxPtr->tmpInTarget = minFHSize; /* minimum to attempt decode */ dctxPtr->dStage = dstage_storeHeader; } @@ -1092,7 +1100,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, srcPtr += sizeToCopy; if (dctxPtr->tmpInSize < dctxPtr->tmpInTarget) { - nextSrcSizeHint = (dctxPtr->tmpInTarget - dctxPtr->tmpInSize) + 4; + nextSrcSizeHint = (dctxPtr->tmpInTarget - dctxPtr->tmpInSize) + BHSize; /* rest of header + nextBlockHeader */ doAnotherStage = 0; /* not enough src data, ask for some more */ break; } @@ -1105,10 +1113,10 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, case dstage_getCBlockSize: { - if ((srcEnd - srcPtr) >= 4) + if ((size_t)(srcEnd - srcPtr) >= BHSize) { selectedIn = srcPtr; - srcPtr += 4; + srcPtr += BHSize; } else { @@ -1121,15 +1129,15 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, if (dctxPtr->dStage == dstage_storeCBlockSize) case dstage_storeCBlockSize: { - size_t sizeToCopy = 4 - dctxPtr->tmpInSize; + size_t sizeToCopy = BHSize - dctxPtr->tmpInSize; if (sizeToCopy > (size_t)(srcEnd - srcPtr)) sizeToCopy = srcEnd - srcPtr; memcpy(dctxPtr->tmpIn + dctxPtr->tmpInSize, srcPtr, sizeToCopy); srcPtr += sizeToCopy; dctxPtr->tmpInSize += sizeToCopy; - if (dctxPtr->tmpInSize < 4) /* not enough input to get full cBlockSize; wait for more */ + if (dctxPtr->tmpInSize < BHSize) /* not enough input to get full cBlockSize; wait for more */ { - nextSrcSizeHint = 4 - dctxPtr->tmpInSize; - doAnotherStage=0; + nextSrcSizeHint = BHSize - dctxPtr->tmpInSize; + doAnotherStage = 0; break; } selectedIn = dctxPtr->tmpIn; @@ -1153,7 +1161,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, dctxPtr->dStage = dstage_getCBlock; if (dstPtr==dstEnd) { - nextSrcSizeHint = nextCBlockSize + 4; + nextSrcSizeHint = nextCBlockSize + BHSize; doAnotherStage = 0; } break; @@ -1180,7 +1188,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, break; } dctxPtr->tmpInTarget -= sizeToCopy; /* still need to copy more */ - nextSrcSizeHint = dctxPtr->tmpInTarget + 4; + nextSrcSizeHint = dctxPtr->tmpInTarget + BHSize; doAnotherStage = 0; break; } @@ -1208,7 +1216,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, srcPtr += sizeToCopy; if (dctxPtr->tmpInSize < dctxPtr->tmpInTarget) /* need more input */ { - nextSrcSizeHint = (dctxPtr->tmpInTarget - dctxPtr->tmpInSize) + 4; + nextSrcSizeHint = (dctxPtr->tmpInTarget - dctxPtr->tmpInSize) + BHSize; doAnotherStage=0; break; } @@ -1311,7 +1319,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, dctxPtr->dStage = dstage_getCBlockSize; break; } - nextSrcSizeHint = 4; + nextSrcSizeHint = BHSize; doAnotherStage = 0; /* still some data to flush */ break; } diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 9c5ff37..e871046 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -33,7 +33,7 @@ */ /* LZ4F is a stand-alone API to create LZ4-compressed frames - * fully conformant to specification v1.4.1. + * fully conformant to specification v1.5.1. * All related operations, including memory management, are handled by the library. * You don't need lz4.h when using lz4frame.h. * */ @@ -254,18 +254,16 @@ size_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t dctx, LZ4F_frameInfo_t* frameInfoPtr, const void* srcBuffer, size_t* srcSizePtr); /* LZ4F_getFrameInfo() - * This function decodes frame header information, such as blockSize. - * It is optional : you could start by calling directly LZ4F_decompress() instead. - * The objective is to extract header information without starting decompression, typically for allocation purposes. - * The function will work only if srcBuffer points at the beginning of the frame, - * and *srcSizePtr is large enough to decode the whole header (typically, between 7 & 15 bytes). - * The result is copied into an LZ4F_frameInfo_t structure, which is pointed by frameInfoPtr, and must be already allocated. - * LZ4F_getFrameInfo() can also be used *after* starting decompression, on a valid LZ4F_decompressionContext_t. + * This function decodes frame header information (such as max blockSize, frame checksum, etc.). + * Its usage is optional : you can start by calling directly LZ4F_decompress() instead. + * The objective is to extract frame header information, typically for allocation purposes. + * LZ4F_getFrameInfo() can also be used anytime *after* starting decompression, on any valid LZ4F_decompressionContext_t. + * The result is *copied* into an existing LZ4F_frameInfo_t structure which must be already allocated. * The number of bytes read from srcBuffer will be provided within *srcSizePtr (necessarily <= original value). - * It is basically the frame header size. - * You are expected to resume decompression from where it stopped (srcBuffer + *srcSizePtr) * The function result is an hint of how many srcSize bytes LZ4F_decompress() expects for next call, - * or an error code which can be tested using LZ4F_isError(). + * or an error code which can be tested using LZ4F_isError() + * (typically, when there is not enough src bytes to fully decode the frame header) + * You are expected to resume decompression from where it stopped (srcBuffer + *srcSizePtr) */ size_t LZ4F_decompress(LZ4F_decompressionContext_t dctx, diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index 25afed4..0d90975 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -40,9 +40,17 @@ extern "C" { #endif /* lz4frame_static.h should be used solely in the context of static linking. + * It contains definitions which may still change overtime. + * Never use it in the context of DLL linking. * */ +/************************************** +* Includes +**************************************/ +#include "lz4frame.h" + + /************************************** * Error management * ************************************/ @@ -53,7 +61,7 @@ extern "C" { ITEM(ERROR_headerVersion_wrong) ITEM(ERROR_blockChecksum_unsupported) ITEM(ERROR_reservedFlag_set) \ ITEM(ERROR_allocation_failed) \ ITEM(ERROR_srcSize_tooLarge) ITEM(ERROR_dstMaxSize_tooSmall) \ - ITEM(ERROR_frameType_unknown) ITEM(ERROR_frameSize_wrong) \ + ITEM(ERROR_frameHeader_incomplete) ITEM(ERROR_frameType_unknown) ITEM(ERROR_frameSize_wrong) \ ITEM(ERROR_srcPtr_wrong) \ ITEM(ERROR_decompressionFailed) \ ITEM(ERROR_headerChecksum_invalid) ITEM(ERROR_contentChecksum_invalid) \ @@ -68,12 +76,6 @@ extern "C" { typedef enum { LZ4F_LIST_ERRORS(LZ4F_GENERATE_ENUM) } LZ4F_errorCodes; /* enum is exposed, to handle specific errors; compare function result to -enum value */ -/************************************** - Includes -**************************************/ -#include "lz4frame.h" - - #if defined (__cplusplus) } #endif diff --git a/programs/frametest.c b/programs/frametest.c index 6adf408..ed131d2 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -277,6 +277,37 @@ int basicTests(U32 seed, double compressibility) if (crcDest != crcOrig) goto _output_error; DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); + DISPLAYLEVEL(4, "Reusing decompression context \n"); + { + size_t oSize = 0; + size_t iSize = 0; + LZ4F_frameInfo_t fi; + + DISPLAYLEVEL(3, "Start by feeding 0 bytes, to get next input size : "); + errorCode = LZ4F_decompress(dCtx, NULL, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, " %u \n", (unsigned)errorCode); + + DISPLAYLEVEL(3, "get FrameInfo on null input : "); + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; + DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); + + DISPLAYLEVEL(3, "get FrameInfo on not enough input : "); + iSize = 6; + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; + DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); + ip += iSize; + + DISPLAYLEVEL(3, "get FrameInfo on enough input : "); + iSize = 15 - iSize; + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, " correctly decoded \n"); + ip += iSize; + } + DISPLAYLEVEL(3, "Byte after byte : \n"); while (ip < iend) { From 197982ec6cf449734f78d849ed4a845e075b2cf4 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 09:24:25 +0100 Subject: [PATCH 093/121] Fixed unfinished frame (issue #75) --- NEWS | 8 ++++++-- programs/Makefile | 8 +++++++- programs/frametest.c | 14 ++++++++++++++ programs/lz4io.c | 21 +++++++++++++-------- 4 files changed, 40 insertions(+), 11 deletions(-) diff --git a/NEWS b/NEWS index 462a23b..e797ff7 100644 --- a/NEWS +++ b/NEWS @@ -1,9 +1,13 @@ r129: +New : LZ4 CLI improved performance when compressing/decompressing multiple files (#86, kind contribution from Kyle J. Harper & Takayuki Matsuoka) Added : LZ4_compress_fast() -Changed: New compression functions LZ4_compress_safe() - Suggested by Evan Nemerson & Takayuki Matsuoka +Changed: Sparse file support enabled by default +Changed: New function names LZ4_compress_safe() - Suggested by Evan Nemerson & Takayuki Matsuoka +Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Greg Slazinski & Evan Nemerson +Changed: Enums converted to LZ4F_ namespace convention - by Takayuki Matsuoka Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka -Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Greg Slazinski, Evan Nemerson & Takayuki Matsuoka Modified:Obsolete functions generate warnings - Suggested by Evan Nemerson & Takayuki Matsuoka +Fixed : Bug #75 (unfinished stream), reported by Yongwoon Cho Updated: Documentation converted to MarkDown r128: diff --git a/programs/Makefile b/programs/Makefile index 910516f..b0e30d8 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -207,7 +207,13 @@ test-lz4-multiple: lz4 datagen ./lz4 -f -m tmp1 notHere tmp2; echo $$? @rm tmp* -test-lz4: lz4 datagen test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation +# test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation +test-lz4: lz4 datagen + @echo ---- test erroneous data ---- + ./datagen -s1 | ./lz4 -vf - tmp + tmpSize= $(stat -c%s "tmp") + echo "Size of tmp = $tmpSize bytes." + rm tmp* @echo ---- test lz4 basic compression/decompression ---- ./datagen -g0 | ./lz4 -v | ./lz4 -t ./datagen -g16KB | ./lz4 -9 | ./lz4 -t diff --git a/programs/frametest.c b/programs/frametest.c index ed131d2..237fd4a 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -278,6 +278,20 @@ int basicTests(U32 seed, double compressibility) DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); DISPLAYLEVEL(4, "Reusing decompression context \n"); + { + size_t iSize = compressedBufferSize - 4; + DISPLAYLEVEL(3, "Missing last 4 bytes : "); + errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + if (!errorCode) goto _output_error; + DISPLAYLEVEL(3, "indeed, request %u bytes \n", (unsigned)errorCode); + iSize = errorCode; + errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &iSize, NULL); + if (errorCode != 0) goto _output_error; + crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + if (crcDest != crcOrig) goto _output_error; + } + { size_t oSize = 0; size_t iSize = 0; diff --git a/programs/lz4io.c b/programs/lz4io.c index e0c69d8..cbf366b 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -807,7 +807,7 @@ static void LZ4IO_freeDResources(dRess_t ress) static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE* dstFile) { unsigned long long filesize = 0; - LZ4F_errorCode_t errorCode; + LZ4F_errorCode_t nextToLoad; unsigned storedSkips = 0; /* Init feed with magic number (already consumed from FILE* sFile) */ @@ -815,8 +815,8 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE size_t inSize = MAGICNUMBER_SIZE; size_t outSize= 0; LZ4IO_writeLE32(ress.srcBuffer, LZ4IO_MAGICNUMBER); - errorCode = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &outSize, ress.srcBuffer, &inSize, NULL); - if (LZ4F_isError(errorCode)) EXM_THROW(62, "Header error : %s", LZ4F_getErrorName(errorCode)); + nextToLoad = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &outSize, ress.srcBuffer, &inSize, NULL); + if (LZ4F_isError(nextToLoad)) EXM_THROW(62, "Header error : %s", LZ4F_getErrorName(nextToLoad)); } /* Main Loop */ @@ -824,18 +824,20 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE { size_t readSize; size_t pos = 0; + size_t decodedBytes = ress.dstBufferSize; /* Read input */ readSize = fread(ress.srcBuffer, 1, ress.srcBufferSize, srcFile); - if (!readSize) break; /* empty file or stream */ + if (!readSize) + break; /* empty file or stream */ - while (pos < readSize) + while (nextToLoad && ((pos < readSize) || (decodedBytes == ress.dstBufferSize))) /* still to read, or still to flush */ { /* Decode Input (at least partially) */ size_t remaining = readSize - pos; - size_t decodedBytes = ress.dstBufferSize; - errorCode = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &decodedBytes, (char*)(ress.srcBuffer)+pos, &remaining, NULL); - if (LZ4F_isError(errorCode)) EXM_THROW(66, "Decompression error : %s", LZ4F_getErrorName(errorCode)); + decodedBytes = ress.dstBufferSize; + nextToLoad = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &decodedBytes, (char*)(ress.srcBuffer)+pos, &remaining, NULL); + if (LZ4F_isError(nextToLoad)) EXM_THROW(66, "Decompression error : %s", LZ4F_getErrorName(nextToLoad)); pos += remaining; if (decodedBytes) @@ -850,6 +852,9 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE LZ4IO_fwriteSparseEnd(dstFile, storedSkips); + if (nextToLoad!=0) + EXM_THROW(67, "Unfinished stream"); + return filesize; } From c035b7a99a6057b249718e2f0d163f0f1dfe05dc Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 09:26:16 +0100 Subject: [PATCH 094/121] Restored make-lz4 --- programs/Makefile | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/programs/Makefile b/programs/Makefile index b0e30d8..910516f 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -207,13 +207,7 @@ test-lz4-multiple: lz4 datagen ./lz4 -f -m tmp1 notHere tmp2; echo $$? @rm tmp* -# test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation -test-lz4: lz4 datagen - @echo ---- test erroneous data ---- - ./datagen -s1 | ./lz4 -vf - tmp - tmpSize= $(stat -c%s "tmp") - echo "Size of tmp = $tmpSize bytes." - rm tmp* +test-lz4: lz4 datagen test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation @echo ---- test lz4 basic compression/decompression ---- ./datagen -g0 | ./lz4 -v | ./lz4 -t ./datagen -g16KB | ./lz4 -9 | ./lz4 -t From e18aa907985da9ccbeb1684517d0f8e77e189984 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 09:51:00 +0100 Subject: [PATCH 095/121] Fixed frametest --- programs/frametest.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/programs/frametest.c b/programs/frametest.c index 237fd4a..fedb78d 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -280,13 +280,15 @@ int basicTests(U32 seed, double compressibility) DISPLAYLEVEL(4, "Reusing decompression context \n"); { size_t iSize = compressedBufferSize - 4; + const BYTE* cBuff = (const BYTE*) compressedBuffer; DISPLAYLEVEL(3, "Missing last 4 bytes : "); - errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &iSize, NULL); + errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); if (LZ4F_isError(errorCode)) goto _output_error; if (!errorCode) goto _output_error; DISPLAYLEVEL(3, "indeed, request %u bytes \n", (unsigned)errorCode); + cBuff += iSize; iSize = errorCode; - errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &iSize, NULL); + errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); if (errorCode != 0) goto _output_error; crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); if (crcDest != crcOrig) goto _output_error; @@ -327,7 +329,6 @@ int basicTests(U32 seed, double compressibility) { size_t oSize = oend-op; size_t iSize = 1; - //DISPLAY("%7i \n", (int)(ip-(BYTE*)compressedBuffer)); errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); if (LZ4F_isError(errorCode)) goto _output_error; op += oSize; @@ -335,7 +336,7 @@ int basicTests(U32 seed, double compressibility) } crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); if (crcDest != crcOrig) goto _output_error; - DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); + DISPLAYLEVEL(3, "Regenerated %u/%u bytes \n", (unsigned)(op-(BYTE*)decodedBuffer), COMPRESSIBLE_NOISE_LENGTH); errorCode = LZ4F_freeDecompressionContext(dCtx); if (LZ4F_isError(errorCode)) goto _output_error; From cbcdd88ccb97632015cf3732b46f8800e62e337b Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 10:05:54 +0100 Subject: [PATCH 096/121] Fixed frame concatenation --- programs/lz4io.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/programs/lz4io.c b/programs/lz4io.c index cbf366b..40bdbb7 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -820,18 +820,19 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE } /* Main Loop */ - for (;;) + for (;nextToLoad;) { size_t readSize; size_t pos = 0; size_t decodedBytes = ress.dstBufferSize; /* Read input */ - readSize = fread(ress.srcBuffer, 1, ress.srcBufferSize, srcFile); + if (nextToLoad > ress.srcBufferSize) nextToLoad = ress.srcBufferSize; + readSize = fread(ress.srcBuffer, 1, nextToLoad, srcFile); if (!readSize) break; /* empty file or stream */ - while (nextToLoad && ((pos < readSize) || (decodedBytes == ress.dstBufferSize))) /* still to read, or still to flush */ + while ((pos < readSize) || (decodedBytes == ress.dstBufferSize)) /* still to read, or still to flush */ { /* Decode Input (at least partially) */ size_t remaining = readSize - pos; @@ -839,6 +840,7 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE nextToLoad = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &decodedBytes, (char*)(ress.srcBuffer)+pos, &remaining, NULL); if (LZ4F_isError(nextToLoad)) EXM_THROW(66, "Decompression error : %s", LZ4F_getErrorName(nextToLoad)); pos += remaining; + if (!nextToLoad) break; if (decodedBytes) { From a01e10dbdc58ae3bcef3348c6b196fc733b942b7 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 12:12:02 +0100 Subject: [PATCH 097/121] Changed LZ4F compressionLevel from unsigned to signed, in anticipation for LZ4_compress_fast() integration. --- lib/lz4frame.c | 12 ++++++------ lib/lz4frame.h | 6 +++--- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 7fc1314..eb38390 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -105,7 +105,7 @@ typedef unsigned long long U64; static const size_t minFHSize = 7; static const size_t maxFHSize = 15; static const size_t BHSize = 4; -static const U32 minHClevel = 3; +static const int minHClevel = 3; /************************************** * Structures and local types @@ -306,7 +306,7 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf if (prefs.frameInfo.contentSize != 0) prefs.frameInfo.contentSize = (U64)srcSize; /* auto-correct content size if selected (!=0) */ - if (prefs.compressionLevel < minHClevel) + if (prefs.compressionLevel < (int)minHClevel) { cctxI.lz4CtxPtr = &lz4ctx; cctxI.lz4CtxLevel = 1; @@ -334,7 +334,7 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; - if (prefs.compressionLevel >= minHClevel) /* no allocation necessary with lz4 fast */ + if (prefs.compressionLevel >= (int)minHClevel) /* no allocation necessary with lz4 fast */ FREEMEM(cctxI.lz4CtxPtr); return (dstPtr - dstStart); @@ -407,11 +407,11 @@ size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* ds /* ctx Management */ { - U32 tableID = cctxPtr->prefs.compressionLevelprefs.compressionLevel < minHClevel) ? 1 : 2; /* 0:nothing ; 1:LZ4 table ; 2:HC tables */ if (cctxPtr->lz4CtxLevel < tableID) { FREEMEM(cctxPtr->lz4CtxPtr); - if (cctxPtr->prefs.compressionLevelprefs.compressionLevel < minHClevel) cctxPtr->lz4CtxPtr = (void*)LZ4_createStream(); else cctxPtr->lz4CtxPtr = (void*)LZ4_createStreamHC(); @@ -531,7 +531,7 @@ static int LZ4F_localLZ4_compressHC_limitedOutput_continue(void* ctx, const char return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); } -static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, U32 level) +static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int level) { if (level < minHClevel) { diff --git a/lib/lz4frame.h b/lib/lz4frame.h index e871046..85ebce3 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -119,9 +119,9 @@ typedef struct { typedef struct { LZ4F_frameInfo_t frameInfo; - unsigned compressionLevel; /* 0 == default (fast mode); values above 16 count as 16 */ - unsigned autoFlush; /* 1 == always flush (reduce need for tmp buffer) */ - unsigned reserved[4]; /* must be zero for forward compatibility */ + int compressionLevel; /* 0 == default (fast mode); values above 16 count as 16; values below 0 count as 0 */ + unsigned autoFlush; /* 1 == always flush (reduce need for tmp buffer) */ + unsigned reserved[4]; /* must be zero for forward compatibility */ } LZ4F_preferences_t; From 72e679438f9ead31e671e0738b7730f846ebde70 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 20 Apr 2015 12:23:33 +0100 Subject: [PATCH 098/121] Updated LZ4F_freeDecompressionContext(), to provide stage hint as result --- lib/lz4frame.c | 4 +++- lib/lz4frame.h | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index eb38390..a70dbaf 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -772,14 +772,16 @@ LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* LZ LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_decompressionContext) { + LZ4F_errorCode_t result = LZ4F_OK_NoError; LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)LZ4F_decompressionContext; if (dctxPtr != NULL) /* can accept NULL input, like free() */ { + result = (LZ4F_errorCode_t)dctxPtr->dStage; FREEMEM(dctxPtr->tmpIn); FREEMEM(dctxPtr->tmpOutBuffer); FREEMEM(dctxPtr); } - return LZ4F_OK_NoError; + return result; } diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 85ebce3..b5ba2fb 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -245,6 +245,8 @@ LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t dctx) * The function will provide a pointer to a fully allocated and initialized LZ4F_decompressionContext_t object. * The result is an errorCode, which can be tested using LZ4F_isError(). * dctx memory can be released using LZ4F_freeDecompressionContext(); + * The result of LZ4F_freeDecompressionContext() is indicative of the current state of decompressionContext when being released. + * That is, it should be == 0 if decompression has been completed fully and correctly. */ From f11afafe234ea45a6d80f8b21df1d603ace4e025 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 21 Apr 2015 18:31:35 +0100 Subject: [PATCH 099/121] Removed LZ4_compress() (obsolete) from lz4 --- lib/lz4.c | 2 +- lib/lz4.h | 2 +- programs/bench.c | 10 +++++----- programs/lz4io.c | 12 ++++++------ 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 53cd4a9..490f9ce 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -685,7 +685,7 @@ int LZ4_compress_safe(const char* source, char* dest, int inputSize, int maxOutp #if (HEAPMODE) void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* malloc-calloc aligned on 8-bytes boundaries */ #else - U64 ctx[LZ4_STREAMSIZE_U64] = {0}; /* Ensure data is aligned on 8-bytes boundaries */ + U64 ctx[LZ4_STREAMSIZE_U64]; /* Ensure data is aligned on 8-bytes boundaries */ #endif int result = LZ4_compress_safe_extState(ctx, source, dest, inputSize, maxOutputSize); diff --git a/lib/lz4.h b/lib/lz4.h index 98296e9..cf06a8e 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -302,7 +302,7 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS #endif // LZ4_DEPRECATE_WARNING_DEFBLOCK /* Obsolete compression functions */ -/* These functions are planned to generate warnings by r131 approximately */ +/* These functions are planned to start generate warnings by r131 approximately */ int LZ4_compress (const char* source, char* dest, int sourceSize); int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); diff --git a/programs/bench.c b/programs/bench.c index d3b060c..384a425 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -58,9 +58,9 @@ #include "lz4.h" #define COMPRESSOR0 LZ4_compress_local -static int LZ4_compress_local(const char* src, char* dst, int size, int clevel) { (void)clevel; return LZ4_compress(src, dst, size); } +static int LZ4_compress_local(const char* src, char* dst, int srcSize, int dstSize, int clevel) { (void)clevel; return LZ4_compress_safe(src, dst, srcSize, dstSize); } #include "lz4hc.h" -#define COMPRESSOR1 LZ4_compressHC2 +#define COMPRESSOR1 LZ4_compressHC_safe #define DEFAULTCOMPRESSOR COMPRESSOR0 #include "xxhash.h" @@ -121,8 +121,8 @@ struct chunkParameters struct compressionParameters { - int (*compressionFunction)(const char*, char*, int, int); - int (*decompressionFunction)(const char*, char*, int); + int (*compressionFunction)(const char* src, char* dst, int srcSize, int dstSize, int cLevel); + int (*decompressionFunction)(const char* src, char* dst, int dstSize); }; @@ -371,7 +371,7 @@ int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel) while(BMK_GetMilliSpan(milliTime) < TIMELOOP) { for (chunkNb=0; chunkNb %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); + DISPLAYUPDATE(2, "\rRead : %i MB ==> %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); /* Write Block */ LZ4IO_writeLE32(out_buff, outSize); From b805d581b97be95fcc000134a54aa7c591b3ef09 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 21 Apr 2015 19:07:31 +0100 Subject: [PATCH 100/121] Removed obsolete functions from lz4 cli --- lib/lz4frame.c | 8 ++++---- lib/lz4hc.c | 3 ++- programs/lz4io.c | 12 +++++++++--- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index a70dbaf..a2b6a22 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -516,19 +516,19 @@ static size_t LZ4F_compressBlock(void* dst, const void* src, size_t srcSize, com static int LZ4F_localLZ4_compress_limitedOutput_withState(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compress_limitedOutput_withState(ctx, src, dst, srcSize, dstSize); + return LZ4_compress_safe_extState(ctx, src, dst, srcSize, dstSize); } static int LZ4F_localLZ4_compress_limitedOutput_continue(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compress_limitedOutput_continue((LZ4_stream_t*)ctx, src, dst, srcSize, dstSize); + return LZ4_compress_safe_continue((LZ4_stream_t*)ctx, src, dst, srcSize, dstSize); } static int LZ4F_localLZ4_compressHC_limitedOutput_continue(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compressHC_limitedOutput_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); + return LZ4_compressHC_safe_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); } static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int level) @@ -538,7 +538,7 @@ static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int lev if (blockMode == LZ4F_blockIndependent) return LZ4F_localLZ4_compress_limitedOutput_withState; return LZ4F_localLZ4_compress_limitedOutput_continue; } - if (blockMode == LZ4F_blockIndependent) return LZ4_compressHC2_limitedOutput_withStateHC; + if (blockMode == LZ4F_blockIndependent) return LZ4_compressHC_safe_extStateHC; return LZ4F_localLZ4_compressHC_limitedOutput_continue; } diff --git a/lib/lz4hc.c b/lib/lz4hc.c index 4f1272c..909481e 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -673,7 +673,8 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS * Deprecated Functions ***********************************/ /* Deprecated compression functions */ -int LZ4_compressHC(const char* source, char* dest, int inputSize) { return LZ4_compressHC2(source, dest, inputSize, 0); } +/* These functions are planned to start generate warnings by r131 approximately */ +int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, 0); } int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, cLevel); } diff --git a/programs/lz4io.c b/programs/lz4io.c index efe1fcd..88f94dd 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -322,12 +322,18 @@ static void LZ4IO_writeLE32 (void* p, unsigned value32) dstPtr[3] = (unsigned char)(value32 >> 24); } +static int LZ4IO_LZ4_compress(const char* src, char* dst, int srcSize, int dstSize, int cLevel) +{ + (void)cLevel; + return LZ4_compress_safe(src, dst, srcSize, dstSize); +} + /* LZ4IO_compressFilename_Legacy : * This function is intentionally "hidden" (not published in .h) * It generates compressed streams using the old 'legacy' format */ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output_filename, int compressionlevel) { - int (*compressionFunction)(const char* src, char* dst, int srcSize, int dstSize); + int (*compressionFunction)(const char* src, char* dst, int srcSize, int dstSize, int cLevel); unsigned long long filesize = 0; unsigned long long compressedfilesize = MAGICNUMBER_SIZE; char* in_buff; @@ -341,7 +347,7 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output /* Init */ start = clock(); - if (compressionlevel < 3) compressionFunction = LZ4_compress_safe; else compressionFunction = LZ4_compressHC_limitedOutput; + if (compressionlevel < 3) compressionFunction = LZ4IO_LZ4_compress; else compressionFunction = LZ4_compressHC_safe; if (LZ4IO_getFiles(input_filename, output_filename, &finput, &foutput)) EXM_THROW(20, "File error"); @@ -366,7 +372,7 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output filesize += inSize; /* Compress Block */ - outSize = compressionFunction(in_buff, out_buff+4, inSize, outBuffSize); + outSize = compressionFunction(in_buff, out_buff+4, inSize, outBuffSize, compressionlevel); compressedfilesize += outSize+4; DISPLAYUPDATE(2, "\rRead : %i MB ==> %.2f%% ", (int)(filesize>>20), (double)compressedfilesize/filesize*100); From b05d3d71a6ae2a0f711dfa0a046b8b0095978045 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 22 Apr 2015 00:57:39 +0100 Subject: [PATCH 101/121] Frame content size disabled by default when using LZ4F_compressFrame(), to be in better coherence with the advanced API LZ4F_compress_update() --- lib/lz4frame.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index a2b6a22..8f82f6c 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -301,7 +301,6 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf else { memset(&prefs, 0, sizeof(prefs)); - prefs.frameInfo.contentSize = (U64)srcSize; } if (prefs.frameInfo.contentSize != 0) prefs.frameInfo.contentSize = (U64)srcSize; /* auto-correct content size if selected (!=0) */ From 9c6fb8b160e72454cb50ca78ceca428cb246fd8a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 23 Apr 2015 07:46:35 +0100 Subject: [PATCH 102/121] Added LZ4_compress_fast_extState() --- lib/lz4.c | 44 +++++++++++++++++++++++--------------------- lib/lz4.h | 22 ++++++++++++---------- 2 files changed, 35 insertions(+), 31 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 490f9ce..e9fe467 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -697,29 +697,36 @@ int LZ4_compress_safe(const char* source, char* dest, int inputSize, int maxOutp } +int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) +{ + MEM_INIT(state, 0, LZ4_STREAMSIZE); + + if (acceleration == 0) + { + if (inputSize < LZ4_64Klimit) + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); + else + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, ACCELERATION_DEFAULT); + } + else + { + if (inputSize < LZ4_64Klimit) + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); + else + return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); + } +} + + int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) { #if (HEAPMODE) void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ #else - U64 ctx[LZ4_STREAMSIZE_U64] = {0}; /* Ensure data is aligned on 8-bytes boundaries */ + U64 ctx[LZ4_STREAMSIZE_U64]; /* Ensure data is aligned on 8-bytes boundaries */ #endif - int result; - if (acceleration == 0) - { - if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); - else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, ACCELERATION_DEFAULT); - } - else - { - if (inputSize < LZ4_64Klimit) - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); - else - result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); - } + int result = LZ4_compress_fast_extState(ctx, source, dest, inputSize, maxOutputSize, acceleration); #if (HEAPMODE) FREEMEM(ctx); @@ -740,11 +747,6 @@ LZ4_stream_t* LZ4_createStream(void) return lz4s; } -/* - * LZ4_initStream - * Use this function once, to init a newly allocated LZ4_stream_t structure - * Return : 1 if OK, 0 if error - */ void LZ4_resetStream (LZ4_stream_t* LZ4_stream) { MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t)); diff --git a/lib/lz4.h b/lib/lz4.h index cf06a8e..a2091fa 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -118,16 +118,6 @@ LZ4_compressBound() : */ int LZ4_compressBound(int inputSize); -/* -LZ4_compress_fast() : - Same as LZ4_compress_limitedOutput, but allows to select an "acceleration" factor. - The larger the value, the faster the algorithm, but also the lesser the compression. - So it's a trade-off, which can be fine tuned, selecting whichever value you want. - An acceleration value of "0" means "use Default value", which is typically about 15 (see lz4.c source code). - Note : this function is "safe", even if its name does not say it. It's just faster and compress less. -*/ -int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); - /* LZ4_compress_safe_extState() : Same compression function, just using an externally allocated memory space to store compression state. @@ -138,6 +128,18 @@ LZ4_compress_safe_extState() : int LZ4_sizeofState(void); int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); +/* +LZ4_compress_fast() : + Same as LZ4_compress_safe(), but allows to select an "acceleration" factor. + The larger the acceleration value, the faster the algorithm, but also the lesser the compression. + It's a trade-off, which can be fine tuned, selecting whichever value you want. + An acceleration value of "0" means "use Default value", which is typically 17 (see lz4.c source code). + An acceleration value of "1" is the same as regular LZ4_compress_safe() + Note : this function is "safe", even if its name does not explicitly contain the word. It's just faster and compress less. +*/ +int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); +int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration); + /* LZ4_decompress_fast() : From a9a24e8b7f0fc17e8f667ed4e2cc8f8f9f9ee12f Mon Sep 17 00:00:00 2001 From: Eric Berge Date: Thu, 23 Apr 2015 15:59:09 -0500 Subject: [PATCH 103/121] cmake support for AIX, HPUX, Solaris and Windows The following changes allow for builds on AIX and HPUX with the native (non-gcc) compilers, as well as Visual Studio 2008 and Visual Studio 2012. Also work around a build error with gcc on Solaris which fails due to the system detecting an attempt to use C99 mode with an XPG mode less than XPG6. --- cmake_unofficial/CMakeLists.txt | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/cmake_unofficial/CMakeLists.txt b/cmake_unofficial/CMakeLists.txt index 1782ea0..77d9ca5 100644 --- a/cmake_unofficial/CMakeLists.txt +++ b/cmake_unofficial/CMakeLists.txt @@ -17,9 +17,11 @@ ENDIF() option(BUILD_TOOLS "Build the command line tools" ON) option(BUILD_LIBS "Build the libraries in addition to the tools" ON) +if(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX) if(UNIX AND BUILD_LIBS) add_definitions(-fPIC) endif() +endif() set(LZ4_DIR ../lib/) set(PRG_DIR ../programs/) @@ -62,7 +64,12 @@ endif() #warnings +if(MSVC) +ADD_DEFINITIONS("-W4") +endif() +if(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX) ADD_DEFINITIONS("-Wall") +endif() if(CMAKE_COMPILER_IS_GNUCXX) ADD_DEFINITIONS("-Wextra") ADD_DEFINITIONS("-Wundef") @@ -70,7 +77,10 @@ ADD_DEFINITIONS("-Wshadow") ADD_DEFINITIONS("-Wcast-align") ADD_DEFINITIONS("-Wstrict-prototypes") endif(CMAKE_COMPILER_IS_GNUCXX) +if((CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX) AND + (NOT CMAKE_SYSTEM_NAME MATCHES "SunOS")) ADD_DEFINITIONS("-std=c99") +endif() ADD_DEFINITIONS("-DLZ4_VERSION=\"${CPACK_PACKAGE_VERSION_PATCH}\"") INCLUDE_DIRECTORIES (${LZ4_DIR}) From d6dc0a410d0be93551f247eeb871603d2db11c17 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 24 Apr 2015 10:15:12 +0100 Subject: [PATCH 104/121] streaming API : Improved ring buffer management --- lib/lz4.c | 3 +- lib/lz4.h | 39 ++++++++------ lib/lz4hc.c | 3 +- programs/fuzzer.c | 132 ++++++++++++++++++++++++++++------------------ programs/lz4io.c | 34 ++++++------ 5 files changed, 124 insertions(+), 87 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index e9fe467..538aad4 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -1023,8 +1023,7 @@ FORCE_INLINE int LZ4_decompress_generic( { /* match can be copied as a single segment from external dictionary */ match = dictEnd - (lowPrefix-match); - memcpy(op, match, length); - op += length; + memmove(op, match, length); op += length; } else { diff --git a/lib/lz4.h b/lib/lz4.h index a2091fa..68f43a7 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -74,15 +74,14 @@ int LZ4_compress_safe (const char* source, char* dest, int sourceSize, int max int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize); /* -LZ4_compress_limitedOutput() : +LZ4_compress_safe() : Compresses 'sourceSize' bytes from buffer 'source' - into already allocated 'dest' of size 'maxOutputSize'. - Compression runs faster when 'maxOutputSize' >= LZ4_compressBound(sourceSize). - That's because in such case, it is guaranteed to compress within 'dest' budget, even in worst case scenario. - Compressing into a more limited space budget requires additional checks. - If the function cannot compress 'source' into a limited 'dest' budget, - compression stops *immediately*, and result of the function is zero. - It greatly accelerates behavior on non-compressible input, but as a consequence, 'dest' content is not valid either. + into already allocated 'dest' buffer of size 'maxDestSize'. + Compression is guaranteed to succeed if 'maxDestSize' >= LZ4_compressBound(sourceSize). + It also runs faster, so it's a recommended setting. + If the function cannot compress 'source' into a more limited 'dest' budget, + compression stops *immediately*, and the function result is zero. + As a consequence, 'dest' content is not valid. This function never writes outside 'dest' buffer, nor read outside 'source' buffer. sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE maxDestSize : full or partial size of buffer 'dest' (which must be already allocated) @@ -132,13 +131,13 @@ int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int LZ4_compress_fast() : Same as LZ4_compress_safe(), but allows to select an "acceleration" factor. The larger the acceleration value, the faster the algorithm, but also the lesser the compression. - It's a trade-off, which can be fine tuned, selecting whichever value you want. + It's a trade-off. It can be fine tuned, with each successive value providing an additional +2/3% to speed. An acceleration value of "0" means "use Default value", which is typically 17 (see lz4.c source code). An acceleration value of "1" is the same as regular LZ4_compress_safe() Note : this function is "safe", even if its name does not explicitly contain the word. It's just faster and compress less. */ -int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxOutputSize, unsigned acceleration); -int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration); +int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxDestSize, unsigned acceleration); +int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, unsigned acceleration); /* @@ -260,8 +259,18 @@ int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dicti *_continue() : These decoding functions allow decompression of multiple blocks in "streaming" mode. Previously decoded blocks *must* remain available at the memory position where they were decoded (up to 64 KB) - If this condition is not possible, save the relevant part of decoded data into a safe buffer, - and indicate where is its new address using LZ4_setStreamDecode() + In the case of a ring buffers, decoding buffer must be either : + - Exactly same size as encoding buffer, with same update rule (block boundaries at same positions) + In which case, the decoding & encoding ring buffer can have any size, including very small ones ( < 64 KB). + - Larger than encoding buffer, by a minimum of maxBlockSize more bytes. + maxBlockSize is implementation dependent. It's the maximum size you intend to compress into a single block. + In which case, encoding and decoding buffers do not need to be synchronized, + and encoding ring buffer can have any size, including small ones ( < 64 KB). + - _At least_ 64 KB + 8 bytes + maxBlockSize. + In which case, encoding and decoding buffers do not need to be synchronized, + and encoding ring buffer can have any size, including larger than decoding buffer. + Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer, + and indicate where it is saved using LZ4_setStreamDecode() */ int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxDecompressedSize); int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize); @@ -271,8 +280,8 @@ int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const ch Advanced decoding functions : *_usingDict() : These decoding functions work the same as - a combination of LZ4_setDictDecode() followed by LZ4_decompress_x_continue() - They are stand-alone and don't use nor update an LZ4_streamDecode_t structure. + a combination of LZ4_setStreamDecode() followed by LZ4_decompress_x_continue() + They are stand-alone. They don't need nor update an LZ4_streamDecode_t structure. */ int LZ4_decompress_safe_usingDict (const char* source, char* dest, int compressedSize, int maxDecompressedSize, const char* dictStart, int dictSize); int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalSize, const char* dictStart, int dictSize); diff --git a/lib/lz4hc.c b/lib/lz4hc.c index 909481e..e154ca9 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -620,7 +620,8 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr, } /* Check if blocks follow each other */ - if ((const BYTE*)source != ctxPtr->end) LZ4HC_setExternalDict(ctxPtr, (const BYTE*)source); + if ((const BYTE*)source != ctxPtr->end) + LZ4HC_setExternalDict(ctxPtr, (const BYTE*)source); /* Check overlapping input/dictionary space */ { diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 1704b6e..685553b 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -153,10 +153,10 @@ static U32 FUZ_rand(U32* src) { U32 rand32 = *src; rand32 *= PRIME1; - rand32 += PRIME2; + rand32 ^= PRIME2; rand32 = FUZ_rotl32(rand32, 13); *src = rand32; - return rand32 >> 3; + return rand32; } @@ -168,27 +168,28 @@ static void FUZ_fillCompressibleNoiseBuffer(void* buffer, size_t bufferSize, dou size_t pos = 0; U32 P32 = (U32)(32768 * proba); - // First Byte - BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); + /* First Bytes */ + while (pos < 20) + BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); while (pos < bufferSize) { - // Select : Literal (noise) or copy (within 64K) + /* Select : Literal (noise) or copy (within 64K) */ if (FUZ_RAND15BITS < P32) { - // Copy (within 64K) + /* Copy (within 64K) */ size_t match, d; size_t length = FUZ_RANDLENGTH + 4; size_t offset = FUZ_RAND15BITS + 1; - if (offset > pos) offset = pos; + while (offset > pos) offset >>= 1; d = pos + length; - if (d > bufferSize) d = bufferSize; + while (d > bufferSize) d = bufferSize; match = pos - offset; while (pos < d) BBuffer[pos++] = BBuffer[match++]; } else { - // Literal (noise) + /* Literal (noise) */ size_t d; size_t length = FUZ_RANDLENGTH; d = pos + length; @@ -299,6 +300,16 @@ static void FUZ_displayUpdate(unsigned testNb) } +static void FUZ_findDiff(const void* buff1, const void* buff2) +{ + const BYTE* b1 = (const BYTE*)buff1; + const BYTE* b2 = (const BYTE*)buff2; + size_t i=0; + while (b1[i]==b2[i]) i++; + DISPLAY("Wrong Byte at position %u\n", (unsigned)i); +} + + static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double compressibility, U32 duration) { unsigned long long bytes = 0; @@ -391,17 +402,17 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_CHECKTEST(ret==0, "LZ4_compressHC() failed"); HCcompressedSize = ret; - // Test compression HC using external state + /* Test compression HC using external state */ FUZ_DISPLAYTEST; ret = LZ4_compressHC_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize); FUZ_CHECKTEST(ret==0, "LZ4_compressHC_withStateHC() failed"); - // Test compression using external state + /* Test compression using external state */ FUZ_DISPLAYTEST; ret = LZ4_compress_withState(stateLZ4, block, compressedBuffer, blockSize); FUZ_CHECKTEST(ret==0, "LZ4_compress_withState() failed"); - // Test compression + /* Test compression */ FUZ_DISPLAYTEST; ret = LZ4_compress(block, compressedBuffer, blockSize); FUZ_CHECKTEST(ret==0, "LZ4_compress() failed"); @@ -411,7 +422,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c crcOrig = XXH32(block, blockSize, 0); - // Test decoding with output size being exactly what's necessary => must work + /* Test decoding with output size being exactly what's necessary => must work */ FUZ_DISPLAYTEST; ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize); FUZ_CHECKTEST(ret<0, "LZ4_decompress_fast failed despite correct space"); @@ -419,19 +430,19 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c crcCheck = XXH32(decodedBuffer, blockSize, 0); FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast corrupted decoded data"); - // Test decoding with one byte missing => must fail + /* Test decoding with one byte missing => must fail */ FUZ_DISPLAYTEST; decodedBuffer[blockSize-1] = 0; ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize-1); FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small"); FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast overrun specified output buffer"); - // Test decoding with one byte too much => must fail + /* Test decoding with one byte too much => must fail */ FUZ_DISPLAYTEST; ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize+1); FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too large"); - // Test decoding with output size exactly what's necessary => must work + /* Test decoding with output size exactly what's necessary => must work */ FUZ_DISPLAYTEST; decodedBuffer[blockSize] = 0; ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize); @@ -570,9 +581,9 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_DISPLAYTEST; ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize, decodedBuffer, dictSize); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_withPrefix64k did not regenerate original data"); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_withPrefix64k corrupted decoded data"); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); /* Compress using External dictionary */ FUZ_DISPLAYTEST; @@ -593,7 +604,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize); FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer"); - // Decompress with dictionary as external + /* Decompress with dictionary as external */ FUZ_DISPLAYTEST; decodedBuffer[blockSize] = 0; ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize, dict, dictSize); @@ -601,11 +612,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_fast_usingDict overrun specified output buffer size") crcCheck = XXH32(decodedBuffer, blockSize, 0); if (crcCheck!=crcOrig) - { - int i=0; - while (block[i]==decodedBuffer[i]) i++; - printf("Wrong Byte at position %i/%i\n", i, blockSize); - } + FUZ_findDiff(block, decodedBuffer); FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_usingDict corrupted decoded data (dict %i)", dictSize); FUZ_DISPLAYTEST; @@ -619,7 +626,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_DISPLAYTEST; decodedBuffer[blockSize-1] = 0; ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_withDict should have failed : wrong original size (-1 byte)"); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)"); FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size"); FUZ_DISPLAYTEST; @@ -640,9 +647,9 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c } } - // Compress HC using External dictionary + /* Compress HC using External dictionary */ FUZ_DISPLAYTEST; - dict -= (FUZ_rand(&randState) & 7); // even bigger separation + dict -= (FUZ_rand(&randState) & 7); /* even bigger separation */ if (dict < (char*)CNBuffer) dict = (char*)CNBuffer; LZ4_resetStreamHC (&LZ4dictHC, FUZ_rand(&randState) & 0x7); LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); @@ -667,14 +674,9 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size") crcCheck = XXH32(decodedBuffer, blockSize, 0); if (crcCheck!=crcOrig) - { - int i=0; - while (block[i]==decodedBuffer[i]) i++; - printf("Wrong Byte at position %i/%i\n", i, blockSize); - } + FUZ_findDiff(block, decodedBuffer); FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); - /* ***** End of tests *** */ /* Fill stats */ bytes += blockSize; @@ -723,13 +725,13 @@ static void FUZ_unitTests(void) char ringBuffer[ringBufferSize]; U32 randState = 1; - // Init + /* Init */ FUZ_fillCompressibleNoiseBuffer(testInput, testInputSize, 0.50, &randState); - // 32-bits address space overflow test + /* 32-bits address space overflow test */ FUZ_AddressOverflow(); - // LZ4 streaming tests + /* LZ4 streaming tests */ { LZ4_stream_t* statePtr; LZ4_stream_t streamingState; @@ -737,12 +739,12 @@ static void FUZ_unitTests(void) U64 crcNew; int result; - // Allocation test + /* Allocation test */ statePtr = LZ4_createStream(); FUZ_CHECKTEST(statePtr==NULL, "LZ4_createStream() allocation failed"); LZ4_freeStream(statePtr); - // simple compression test + /* simple compression test */ crcOrig = XXH64(testInput, testCompressedSize, 0); LZ4_resetStream(&streamingState); result = LZ4_compress_limitedOutput_continue(&streamingState, testInput, testCompressed, testCompressedSize, testCompressedSize-1); @@ -753,7 +755,7 @@ static void FUZ_unitTests(void) crcNew = XXH64(testVerify, testCompressedSize, 0); FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); - // ring buffer test + /* ring buffer test */ { XXH64_state_t xxhOrig; XXH64_state_t xxhNew; @@ -787,7 +789,7 @@ static void FUZ_unitTests(void) crcNew = XXH64_digest(&xxhNew); FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); - // prepare next message + /* prepare next message */ iNext += messageSize; rNext += messageSize; dNext += messageSize; @@ -798,21 +800,20 @@ static void FUZ_unitTests(void) } } - // LZ4 HC streaming tests + /* LZ4 HC streaming tests */ { LZ4_streamHC_t* sp; LZ4_streamHC_t sHC; - //XXH64_state_t xxh; U64 crcOrig; U64 crcNew; int result; - // Allocation test + /* Allocation test */ sp = LZ4_createStreamHC(); FUZ_CHECKTEST(sp==NULL, "LZ4_createStreamHC() allocation failed"); LZ4_freeStreamHC(sp); - // simple compression test + /* simple HC compression test */ crcOrig = XXH64(testInput, testCompressedSize, 0); LZ4_resetStreamHC(&sHC, 0); result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput, testCompressed, testCompressedSize, testCompressedSize-1); @@ -823,7 +824,7 @@ static void FUZ_unitTests(void) crcNew = XXH64(testVerify, testCompressedSize, 0); FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); - // simple dictionary compression test + /* simple dictionary HC compression test */ crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); LZ4_resetStreamHC(&sHC, 0); LZ4_loadDictHC(&sHC, testInput, 64 KB); @@ -855,7 +856,7 @@ static void FUZ_unitTests(void) FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() dictionary decompression corruption"); } - // remote dictionary HC compression test + /* remote dictionary HC compression test */ crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); LZ4_resetStreamHC(&sHC, 0); LZ4_loadDictHC(&sHC, testInput, 32 KB); @@ -965,19 +966,46 @@ static void FUZ_unitTests(void) XXH64_state_t xxhOrig; XXH64_state_t xxhNew; LZ4_streamDecode_t decodeState; - const U32 maxMessageSizeLog = 10; + const U32 maxMessageSizeLog = 12; const U32 maxMessageSizeMask = (1< dBufferSize) dNext = 0; + while (totalMessageSize < 9 MB) { XXH64_update(&xxhOrig, testInput + iNext, messageSize); @@ -987,18 +1015,20 @@ static void FUZ_unitTests(void) FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed"); result = LZ4_decompress_safe_continue(&decodeState, testCompressed, testVerify + dNext, result, messageSize); - FUZ_CHECKTEST(result!=(int)messageSize, "ringBuffer : LZ4_decompress_safe() test failed"); + FUZ_CHECKTEST(result!=(int)messageSize, "64K D.ringBuffer : LZ4_decompress_safe() test failed"); XXH64_update(&xxhNew, testVerify + dNext, messageSize); crcNew = XXH64_digest(&xxhNew); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); + if (crcOrig != crcNew) + FUZ_findDiff(testInput + iNext, testVerify + dNext); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption during small decoder-side ring buffer test"); /* prepare next message */ dNext += messageSize; totalMessageSize += messageSize; messageSize = (FUZ_rand(&randState) & maxMessageSizeMask) + 1; iNext = (FUZ_rand(&randState) & 65535); - if (dNext + messageSize > dBufferSize) dNext = 0; + if (dNext > dBufferSize) dNext = 0; } } } diff --git a/programs/lz4io.c b/programs/lz4io.c index 88f94dd..fb1f428 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -443,7 +443,11 @@ static void LZ4IO_freeCResources(cRess_t ress) if (LZ4F_isError(errorCode)) EXM_THROW(38, "Error : can't free LZ4F context resource : %s", LZ4F_getErrorName(errorCode)); } - +/* + * LZ4IO_compressFilename_extRess() + * result : 0 : compression completed correctly + * 1 : missing or pb opening srcFileName + */ static int LZ4IO_compressFilename_extRess(cRess_t ress, const char* srcFileName, const char* dstFileName, int compressionLevel) { unsigned long long filesize = 0; @@ -463,10 +467,7 @@ static int LZ4IO_compressFilename_extRess(cRess_t ress, const char* srcFileName, memset(&prefs, 0, sizeof(prefs)); /* File check */ - { - int srcFileAbsent = LZ4IO_getFiles(srcFileName, dstFileName, &srcFile, &dstFile); - if (srcFileAbsent) return srcFileAbsent; - } + if (LZ4IO_getFiles(srcFileName, dstFileName, &srcFile, &dstFile)) return 1; /* Set compression parameters */ prefs.autoFlush = 1; @@ -547,11 +548,8 @@ static int LZ4IO_compressFilename_extRess(cRess_t ress, const char* srcFileName, /* Final Status */ DISPLAYLEVEL(2, "\r%79s\r", ""); - { - DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", - (unsigned long long) filesize, (unsigned long long) compressedfilesize, - (double)compressedfilesize/(filesize + !filesize)*100); /* avoid division by zero */ - } + DISPLAYLEVEL(2, "Compressed %llu bytes into %llu bytes ==> %.2f%%\n", + filesize, compressedfilesize, (double)compressedfilesize/(filesize + !filesize)*100); /* avoid division by zero */ return 0; } @@ -588,8 +586,8 @@ int LZ4IO_compressFilename(const char* srcFileName, const char* dstFileName, int int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix, int compressionLevel) { int i; - int missing_files = 0; - char* outFileName = (char*)malloc(FNSPACE); + int missed_files = 0; + char* dstFileName = (char*)malloc(FNSPACE); size_t ofnSize = FNSPACE; const size_t suffixSize = strlen(suffix); cRess_t ress; @@ -601,18 +599,18 @@ int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, for (i=0; i Date: Fri, 24 Apr 2015 11:04:21 +0100 Subject: [PATCH 105/121] Fixed minor static analyzer warning --- programs/fuzzer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 685553b..2bf3cfe 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -968,7 +968,7 @@ static void FUZ_unitTests(void) LZ4_streamDecode_t decodeState; const U32 maxMessageSizeLog = 12; const U32 maxMessageSizeMask = (1< Date: Fri, 24 Apr 2015 13:26:53 +0100 Subject: [PATCH 106/121] Modified lz4frame context typedef, to enforce stricter alignment condition --- lib/lz4frame.c | 7 ++++--- lib/lz4frame.h | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 8f82f6c..c6b9634 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -283,6 +283,7 @@ size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* prefere size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_preferences_t* preferencesPtr) { LZ4F_cctx_internal_t cctxI; + LZ4F_compressionContext_t cctx = (LZ4F_compressionContext_t)(&cctxI); LZ4_stream_t lz4ctx; LZ4F_preferences_t prefs; LZ4F_compressOptions_t options; @@ -321,15 +322,15 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf if (dstMaxSize < LZ4F_compressFrameBound(srcSize, &prefs)) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; - errorCode = LZ4F_compressBegin(&cctxI, dstBuffer, dstMaxSize, &prefs); /* write header */ + errorCode = LZ4F_compressBegin(cctx, dstBuffer, dstMaxSize, &prefs); /* write header */ if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; /* header size */ - errorCode = LZ4F_compressUpdate(&cctxI, dstPtr, dstEnd-dstPtr, srcBuffer, srcSize, &options); + errorCode = LZ4F_compressUpdate(cctx, dstPtr, dstEnd-dstPtr, srcBuffer, srcSize, &options); if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; - errorCode = LZ4F_compressEnd(&cctxI, dstPtr, dstEnd-dstPtr, &options); /* flush last block, and generate suffix */ + errorCode = LZ4F_compressEnd(cctx, dstPtr, dstEnd-dstPtr, &options); /* flush last block, and generate suffix */ if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; diff --git a/lib/lz4frame.h b/lib/lz4frame.h index b5ba2fb..4698c4b 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -146,7 +146,7 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf /********************************** * Advanced compression functions * ********************************/ -typedef void* LZ4F_compressionContext_t; +typedef size_t* LZ4F_compressionContext_t; typedef struct { unsigned stableSrc; /* 1 == src content will remain available on future calls to LZ4F_compress(); avoid saving src content within tmp buffer as future dictionary */ @@ -226,7 +226,7 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t * Decompression functions * *********************************/ -typedef void* LZ4F_decompressionContext_t; +typedef size_t* LZ4F_decompressionContext_t; typedef struct { unsigned stableDst; /* guarantee that decompressed data will still be there on next function calls (avoid storage into tmp buffers) */ From c9cbb8fc069f93708cbc6a053cf06f8c965f28e5 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 24 Apr 2015 14:28:06 +0100 Subject: [PATCH 107/121] Increased aligment requirement for lz4frame context pointer --- lib/lz4frame.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 4698c4b..6c4dbed 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -132,7 +132,7 @@ size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* prefere size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_preferences_t* preferencesPtr); /* LZ4F_compressFrame() - * Compress an entire srcBuffer into a valid LZ4 frame, as defined by specification v1.5 + * Compress an entire srcBuffer into a valid LZ4 frame, as defined by specification v1.5.1 * The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case. * You can get the minimum value of dstMaxSize by using LZ4F_compressFrameBound() * If this condition is not respected, LZ4F_compressFrame() will fail (result is an errorCode) @@ -146,7 +146,7 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf /********************************** * Advanced compression functions * ********************************/ -typedef size_t* LZ4F_compressionContext_t; +typedef long long* LZ4F_compressionContext_t; /* must be aligned on 8-bytes */ typedef struct { unsigned stableSrc; /* 1 == src content will remain available on future calls to LZ4F_compress(); avoid saving src content within tmp buffer as future dictionary */ @@ -226,7 +226,7 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t * Decompression functions * *********************************/ -typedef size_t* LZ4F_decompressionContext_t; +typedef long long* LZ4F_decompressionContext_t; /* must be aligned on 8-bytes */ typedef struct { unsigned stableDst; /* guarantee that decompressed data will still be there on next function calls (avoid storage into tmp buffers) */ From ad2dd6df3f62340c6f4ad6d70e805ade7037afbd Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 27 Apr 2015 02:31:56 +0100 Subject: [PATCH 108/121] moved lz4frame context types to incomplete typedef --- lib/lz4frame.c | 61 +++++++++++++++++++++++++------------------------- lib/lz4frame.h | 12 +++++----- 2 files changed, 36 insertions(+), 37 deletions(-) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index c6b9634..5fad2e8 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -39,7 +39,7 @@ You can contact the author at : /************************************** -Compiler Options +* Compiler Options **************************************/ #ifdef _MSC_VER /* Visual Studio */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ @@ -107,10 +107,11 @@ static const size_t maxFHSize = 15; static const size_t BHSize = 4; static const int minHClevel = 3; + /************************************** * Structures and local types **************************************/ -typedef struct +typedef struct LZ4F_cctx_s { LZ4F_preferences_t prefs; U32 version; @@ -124,9 +125,9 @@ typedef struct XXH32_state_t xxh; void* lz4CtxPtr; U32 lz4CtxLevel; /* 0: unallocated; 1: LZ4_stream_t; 3: LZ4_streamHC_t */ -} LZ4F_cctx_internal_t; +} LZ4F_cctx_t; -typedef struct +typedef struct LZ4F_dctx_s { LZ4F_frameInfo_t frameInfo; U32 version; @@ -146,7 +147,7 @@ typedef struct size_t tmpOutStart; XXH32_state_t xxh; BYTE header[16]; -} LZ4F_dctx_internal_t; +} LZ4F_dctx_t; /************************************** @@ -282,8 +283,7 @@ size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* prefere */ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_preferences_t* preferencesPtr) { - LZ4F_cctx_internal_t cctxI; - LZ4F_compressionContext_t cctx = (LZ4F_compressionContext_t)(&cctxI); + LZ4F_cctx_t cctxI; LZ4_stream_t lz4ctx; LZ4F_preferences_t prefs; LZ4F_compressOptions_t options; @@ -298,11 +298,10 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf cctxI.version = LZ4F_VERSION; cctxI.maxBufferSize = 5 MB; /* mess with real buffer size to prevent allocation; works because autoflush==1 & stableSrc==1 */ - if (preferencesPtr!=NULL) prefs = *preferencesPtr; + if (preferencesPtr!=NULL) + prefs = *preferencesPtr; else - { memset(&prefs, 0, sizeof(prefs)); - } if (prefs.frameInfo.contentSize != 0) prefs.frameInfo.contentSize = (U64)srcSize; /* auto-correct content size if selected (!=0) */ @@ -322,15 +321,15 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf if (dstMaxSize < LZ4F_compressFrameBound(srcSize, &prefs)) return (size_t)-LZ4F_ERROR_dstMaxSize_tooSmall; - errorCode = LZ4F_compressBegin(cctx, dstBuffer, dstMaxSize, &prefs); /* write header */ + errorCode = LZ4F_compressBegin(&cctxI, dstBuffer, dstMaxSize, &prefs); /* write header */ if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; /* header size */ - errorCode = LZ4F_compressUpdate(cctx, dstPtr, dstEnd-dstPtr, srcBuffer, srcSize, &options); + errorCode = LZ4F_compressUpdate(&cctxI, dstPtr, dstEnd-dstPtr, srcBuffer, srcSize, &options); if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; - errorCode = LZ4F_compressEnd(cctx, dstPtr, dstEnd-dstPtr, &options); /* flush last block, and generate suffix */ + errorCode = LZ4F_compressEnd(&cctxI, dstPtr, dstEnd-dstPtr, &options); /* flush last block, and generate suffix */ if (LZ4F_isError(errorCode)) return errorCode; dstPtr += errorCode; @@ -342,8 +341,8 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf /*********************************** -* Advanced compression functions -* *********************************/ +* Advanced compression functions +***********************************/ /* LZ4F_createCompressionContext() : * The first thing to do is to create a compressionContext object, which will be used in all compression operations. @@ -355,9 +354,9 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf */ LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* LZ4F_compressionContextPtr, unsigned version) { - LZ4F_cctx_internal_t* cctxPtr; + LZ4F_cctx_t* cctxPtr; - cctxPtr = (LZ4F_cctx_internal_t*)ALLOCATOR(sizeof(LZ4F_cctx_internal_t)); + cctxPtr = (LZ4F_cctx_t*)ALLOCATOR(sizeof(LZ4F_cctx_t)); if (cctxPtr==NULL) return (LZ4F_errorCode_t)(-LZ4F_ERROR_allocation_failed); cctxPtr->version = version; @@ -371,7 +370,7 @@ LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* LZ4F_c LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_compressionContext) { - LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)LZ4F_compressionContext; + LZ4F_cctx_t* cctxPtr = (LZ4F_cctx_t*)LZ4F_compressionContext; if (cctxPtr != NULL) /* null pointers can be safely provided to this function, like free() */ { @@ -393,7 +392,7 @@ LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_comp size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_preferences_t* preferencesPtr) { LZ4F_preferences_t prefNull; - LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)compressionContext; + LZ4F_cctx_t* cctxPtr = (LZ4F_cctx_t*)compressionContext; BYTE* const dstStart = (BYTE*)dstBuffer; BYTE* dstPtr = dstStart; BYTE* headerStart; @@ -542,7 +541,7 @@ static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int lev return LZ4F_localLZ4_compressHC_limitedOutput_continue; } -static int LZ4F_localSaveDict(LZ4F_cctx_internal_t* cctxPtr) +static int LZ4F_localSaveDict(LZ4F_cctx_t* cctxPtr) { if (cctxPtr->prefs.compressionLevel < minHClevel) return LZ4_saveDict ((LZ4_stream_t*)(cctxPtr->lz4CtxPtr), (char*)(cctxPtr->tmpBuff), 64 KB); @@ -563,7 +562,7 @@ typedef enum { notDone, fromTmpBuffer, fromSrcBuffer } LZ4F_lastBlockStatus; size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* compressOptionsPtr) { LZ4F_compressOptions_t cOptionsNull; - LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)compressionContext; + LZ4F_cctx_t* cctxPtr = (LZ4F_cctx_t*)compressionContext; size_t blockSize = cctxPtr->maxBlockSize; const BYTE* srcPtr = (const BYTE*)srcBuffer; const BYTE* const srcEnd = srcPtr + srcSize; @@ -673,7 +672,7 @@ size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* d */ size_t LZ4F_flush(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* compressOptionsPtr) { - LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)compressionContext; + LZ4F_cctx_t* cctxPtr = (LZ4F_cctx_t*)compressionContext; BYTE* const dstStart = (BYTE*)dstBuffer; BYTE* dstPtr = dstStart; compressFunc_t compress; @@ -714,7 +713,7 @@ size_t LZ4F_flush(LZ4F_compressionContext_t compressionContext, void* dstBuffer, */ size_t LZ4F_compressEnd(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* compressOptionsPtr) { - LZ4F_cctx_internal_t* cctxPtr = (LZ4F_cctx_internal_t*)compressionContext; + LZ4F_cctx_t* cctxPtr = (LZ4F_cctx_t*)compressionContext; BYTE* const dstStart = (BYTE*)dstBuffer; BYTE* dstPtr = dstStart; size_t errorCode; @@ -760,20 +759,20 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t compressionContext, void* dstB */ LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* LZ4F_decompressionContextPtr, unsigned versionNumber) { - LZ4F_dctx_internal_t* dctxPtr; + LZ4F_dctx_t* dctxPtr; - dctxPtr = (LZ4F_dctx_internal_t*)ALLOCATOR(sizeof(LZ4F_dctx_internal_t)); + dctxPtr = (LZ4F_dctx_t*)ALLOCATOR(sizeof(LZ4F_dctx_t)); if (dctxPtr==NULL) return (LZ4F_errorCode_t)-LZ4F_ERROR_GENERIC; dctxPtr->version = versionNumber; - *LZ4F_decompressionContextPtr = (LZ4F_compressionContext_t)dctxPtr; + *LZ4F_decompressionContextPtr = (LZ4F_decompressionContext_t)dctxPtr; return LZ4F_OK_NoError; } LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t LZ4F_decompressionContext) { LZ4F_errorCode_t result = LZ4F_OK_NoError; - LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)LZ4F_decompressionContext; + LZ4F_dctx_t* dctxPtr = (LZ4F_dctx_t*)LZ4F_decompressionContext; if (dctxPtr != NULL) /* can accept NULL input, like free() */ { result = (LZ4F_errorCode_t)dctxPtr->dStage; @@ -808,7 +807,7 @@ typedef enum { dstage_getHeader=0, dstage_storeHeader, dctxPtr->frameInfo and dctxPtr->dStage. input : srcVoidPtr points at the **beginning of the frame** */ -static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVoidPtr, size_t srcSize) +static size_t LZ4F_decodeHeader(LZ4F_dctx_t* dctxPtr, const void* srcVoidPtr, size_t srcSize) { BYTE FLG, BD, HC; unsigned version, blockMode, blockChecksumFlag, contentSizeFlag, contentChecksumFlag, blockSizeID; @@ -929,7 +928,7 @@ static size_t LZ4F_decodeHeader(LZ4F_dctx_internal_t* dctxPtr, const void* srcVo LZ4F_errorCode_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t dCtx, LZ4F_frameInfo_t* frameInfoPtr, const void* srcBuffer, size_t* srcSizePtr) { - LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)dCtx; + LZ4F_dctx_t* dctxPtr = (LZ4F_dctx_t*)dCtx; if (dctxPtr->dStage > dstage_storeHeader) /* note : requires dstage_* header related to be at beginning of enum */ { @@ -959,7 +958,7 @@ static int LZ4F_decompress_safe (const char* source, char* dest, int compressedS } -static void LZ4F_updateDict(LZ4F_dctx_internal_t* dctxPtr, const BYTE* dstPtr, size_t dstSize, const BYTE* dstPtr0, unsigned withinTmp) +static void LZ4F_updateDict(LZ4F_dctx_t* dctxPtr, const BYTE* dstPtr, size_t dstSize, const BYTE* dstPtr0, unsigned withinTmp) { if (dctxPtr->dictSize==0) dctxPtr->dict = (BYTE*)dstPtr; /* priority to dictionary continuity */ @@ -1047,7 +1046,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, const void* srcBuffer, size_t* srcSizePtr, const LZ4F_decompressOptions_t* decompressOptionsPtr) { - LZ4F_dctx_internal_t* dctxPtr = (LZ4F_dctx_internal_t*)decompressionContext; + LZ4F_dctx_t* dctxPtr = (LZ4F_dctx_t*)decompressionContext; LZ4F_decompressOptions_t optionsNull; const BYTE* const srcStart = (const BYTE*)srcBuffer; const BYTE* const srcEnd = srcStart + *srcSizePtr; diff --git a/lib/lz4frame.h b/lib/lz4frame.h index 6c4dbed..05fbc5f 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -144,9 +144,9 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf /********************************** - * Advanced compression functions - * ********************************/ -typedef long long* LZ4F_compressionContext_t; /* must be aligned on 8-bytes */ +* Advanced compression functions +**********************************/ +typedef struct LZ4F_cctx_s* LZ4F_compressionContext_t; /* must be aligned on 8-bytes */ typedef struct { unsigned stableSrc; /* 1 == src content will remain available on future calls to LZ4F_compress(); avoid saving src content within tmp buffer as future dictionary */ @@ -223,10 +223,10 @@ size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t /*********************************** - * Decompression functions - * *********************************/ +* Decompression functions +***********************************/ -typedef long long* LZ4F_decompressionContext_t; /* must be aligned on 8-bytes */ +typedef struct LZ4F_dctx_s* LZ4F_decompressionContext_t; /* must be aligned on 8-bytes */ typedef struct { unsigned stableDst; /* guarantee that decompressed data will still be there on next function calls (avoid storage into tmp buffers) */ From 1e751a74f4caf2265b52a03577df3631f1a777a6 Mon Sep 17 00:00:00 2001 From: Eric Berge Date: Tue, 28 Apr 2015 09:00:23 -0500 Subject: [PATCH 109/121] Install the lz4frame.h header in the cmake build --- cmake_unofficial/CMakeLists.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmake_unofficial/CMakeLists.txt b/cmake_unofficial/CMakeLists.txt index 77d9ca5..4c3eb65 100644 --- a/cmake_unofficial/CMakeLists.txt +++ b/cmake_unofficial/CMakeLists.txt @@ -25,7 +25,7 @@ endif() set(LZ4_DIR ../lib/) set(PRG_DIR ../programs/) -set(LZ4_SRCS_LIB ${LZ4_DIR}lz4.c ${LZ4_DIR}lz4hc.c ${LZ4_DIR}lz4.h ${LZ4_DIR}lz4hc.h ${LZ4_DIR}lz4frame.c ${LZ4_DIR}xxhash.c) +set(LZ4_SRCS_LIB ${LZ4_DIR}lz4.c ${LZ4_DIR}lz4hc.c ${LZ4_DIR}lz4.h ${LZ4_DIR}lz4hc.h ${LZ4_DIR}lz4frame.c ${LZ4_DIR}lz4frame.h ${LZ4_DIR}xxhash.c) set(LZ4_SRCS ${LZ4_DIR}lz4frame.c ${LZ4_DIR}xxhash.c ${PRG_DIR}bench.c ${PRG_DIR}lz4cli.c ${PRG_DIR}lz4io.c) if(BUILD_TOOLS AND NOT BUILD_LIBS) @@ -53,6 +53,7 @@ if(BUILD_LIBS) install(FILES ${LZ4_DIR}/lz4.h ${LZ4_DIR}/lz4hc.h + ${LZ4_DIR}/lz4frame.h DESTINATION include ) From 2a974d73c35476eb66275dd0d9448a6efa51bf56 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 2 May 2015 15:44:43 +0100 Subject: [PATCH 110/121] refactored lz4hc --- lib/lz4hc.c | 92 ++++++++++++++++++++++++++--------------------------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/lib/lz4hc.c b/lib/lz4hc.c index e154ca9..1db3d98 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -1,53 +1,53 @@ /* -LZ4 HC - High Compression Mode of LZ4 -Copyright (C) 2011-2015, Yann Collet. + LZ4 HC - High Compression Mode of LZ4 + Copyright (C) 2011-2015, Yann Collet. -BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) + BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 - - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c + You can contact the author at : + - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ /************************************** - Tuning Parameter +* Tuning Parameter **************************************/ static const int LZ4HC_compressionLevel_default = 9; /************************************** - Includes +* Includes **************************************/ #include "lz4hc.h" /************************************** - Local Compiler Options +* Local Compiler Options **************************************/ #if defined(__GNUC__) # pragma GCC diagnostic ignored "-Wunused-function" @@ -59,18 +59,18 @@ static const int LZ4HC_compressionLevel_default = 9; /************************************** - Common LZ4 definition +* Common LZ4 definition **************************************/ #define LZ4_COMMONDEFS_ONLY #include "lz4.c" /************************************** - Local Constants +* Local Constants **************************************/ #define DICTIONARY_LOGSIZE 16 #define MAXD (1<> ((MINMATCH*8)-HASH_LOG)) -#define DELTANEXT(p) chainTable[(size_t)(p) & MAXD_MASK] -#define GETNEXT(p) ((p) - (size_t)DELTANEXT(p)) +//#define DELTANEXTU16(p) chainTable[(p) & MAXD_MASK] /* flexible, MAXD dependent */ +#define DELTANEXTU16(p) chainTable[(U16)(p)] /* faster */ static U32 LZ4HC_hashPtr(const void* ptr) { return HASH_FUNCTION(LZ4_read32(ptr)); } /************************************** - HC Compression +* HC Compression **************************************/ static void LZ4HC_init (LZ4HC_Data_Structure* hc4, const BYTE* start) { @@ -141,7 +141,7 @@ FORCE_INLINE void LZ4HC_Insert (LZ4HC_Data_Structure* hc4, const BYTE* ip) U32 h = LZ4HC_hashPtr(base+idx); size_t delta = idx - HashTable[h]; if (delta>MAX_DISTANCE) delta = MAX_DISTANCE; - chainTable[idx & 0xFFFF] = (U16)delta; + DELTANEXTU16(idx) = (U16)delta; HashTable[h] = idx; idx++; } @@ -197,7 +197,7 @@ FORCE_INLINE int LZ4HC_InsertAndFindBestMatch (LZ4HC_Data_Structure* hc4, /* I if (mlt > ml) { ml = mlt; *matchpos = base + matchIndex; } /* virtual matchpos */ } } - matchIndex -= chainTable[matchIndex & 0xFFFF]; + matchIndex -= DELTANEXTU16(matchIndex); } return (int)ml; @@ -274,7 +274,7 @@ FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch ( if ((int)mlt > longest) { longest = (int)mlt; *matchpos = base + matchIndex + back; *startpos = ip+back; } } } - matchIndex -= chainTable[matchIndex & 0xFFFF]; + matchIndex -= DELTANEXTU16(matchIndex); } return longest; @@ -557,8 +557,8 @@ int LZ4_compressHC_safe(const char* source, char* dest, int inputSize, int maxOu /************************************** - * Streaming Functions - * ************************************/ +* Streaming Functions +**************************************/ /* allocation */ LZ4_streamHC_t* LZ4_createStreamHC(void) { return (LZ4_streamHC_t*)malloc(sizeof(LZ4_streamHC_t)); } int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { free(LZ4_streamHCPtr); return 0; } @@ -671,8 +671,8 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS /*********************************** - * Deprecated Functions - ***********************************/ +* Deprecated Functions +***********************************/ /* Deprecated compression functions */ /* These functions are planned to start generate warnings by r131 approximately */ int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } From 791512cfddac1dd789ebfdbf1880553a17b57ccb Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 2 May 2015 19:10:28 +0100 Subject: [PATCH 111/121] Fixed bug 9318 --- lib/lz4.c | 10 ++++++---- programs/fuzzer.c | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 538aad4..6fed55a 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -767,7 +767,8 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) const BYTE* const dictEnd = p + dictSize; const BYTE* base; - if (dict->initCheck) LZ4_resetStream(LZ4_dict); /* Uninitialized structure detected */ + if (dict->initCheck) /* Uninitialized structure detected */ + LZ4_resetStream(LZ4_dict); if (dictSize < (int)HASH_UNIT) { @@ -776,7 +777,8 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) return 0; } - if (p <= dictEnd - 64 KB) p = dictEnd - 64 KB; + if ((dictEnd - p) > 64 KB) p = dictEnd - 64 KB; + dict->currentOffset += 64 KB; base = p - dict->currentOffset; dict->dictionary = p; dict->dictSize = (U32)(dictEnd - p); @@ -784,7 +786,7 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) while (p <= dictEnd-HASH_UNIT) { - LZ4_putPosition(p, dict, byU32, base); + LZ4_putPosition(p, dict->hashTable, byU32, base); p+=3; } @@ -1260,7 +1262,7 @@ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, in int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_safe(source, dest, inputSize, LZ4_compressBound(inputSize)); } int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, dstSize); } int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize)); } -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_safe_continue(LZ4_stream, source, dest, inputSize, maxOutputSize); } +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_safe_continue(LZ4_stream, src, dst, srcSize, maxDstSize); } int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_safe_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize)); } /* diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 2bf3cfe..9fd4648 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -596,7 +596,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c FUZ_DISPLAYTEST; LZ4_loadDict(&LZ4dict, dict, dictSize); ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize-1); - FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer"); + FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize); FUZ_DISPLAYTEST; LZ4_loadDict(&LZ4dict, dict, dictSize); From 1b17bf2ab8cf66dd2b740eca376e2d46f7ad7041 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 3 May 2015 17:57:46 +0100 Subject: [PATCH 112/121] New lz4 API, using LZ4_compress_fast() --- lib/lz4.c | 115 ++++++++++-------------- lib/lz4.h | 77 ++++++++-------- lib/lz4frame.c | 4 +- programs/bench.c | 2 +- programs/fullbench.c | 207 ++++++++++++++++++++++++++----------------- programs/lz4io.c | 2 +- 6 files changed, 214 insertions(+), 193 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 6fed55a..3c182aa 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -47,7 +47,7 @@ * ACCELERATION_DEFAULT : * Select the value of "acceleration" for LZ4_compress_fast() when parameter == 0 */ -#define ACCELERATION_DEFAULT 17 +#define ACCELERATION_DEFAULT 1 /************************************** @@ -71,12 +71,6 @@ /************************************** * Compiler Options **************************************/ -#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ -/* "restrict" is a known keyword */ -#else -# define restrict /* Disable restrict */ -#endif - #ifdef _MSC_VER /* Visual Studio */ # define FORCE_INLINE static __forceinline # include @@ -220,7 +214,7 @@ static void LZ4_wildCopy(void* dstPtr, const void* srcPtr, void* dstEnd) /************************************** - Common Constants +* Common Constants **************************************/ #define MINMATCH 4 @@ -660,53 +654,17 @@ _last_literals: } -int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, volatile int acceleration) { - MEM_INIT(state, 0, LZ4_STREAMSIZE); + LZ4_resetStream((LZ4_stream_t*)state); + if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; if (maxOutputSize >= LZ4_compressBound(inputSize)) { if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, 1); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, acceleration); else - return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); - } - else - { - if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, 1); - else - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, 1); - } -} - -int LZ4_compress_safe(const char* source, char* dest, int inputSize, int maxOutputSize) -{ -#if (HEAPMODE) - void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* malloc-calloc aligned on 8-bytes boundaries */ -#else - U64 ctx[LZ4_STREAMSIZE_U64]; /* Ensure data is aligned on 8-bytes boundaries */ -#endif - - int result = LZ4_compress_safe_extState(ctx, source, dest, inputSize, maxOutputSize); - -#if (HEAPMODE) - FREEMEM(ctx); -#endif - return result; -} - - -int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) -{ - MEM_INIT(state, 0, LZ4_STREAMSIZE); - - if (acceleration == 0) - { - if (inputSize < LZ4_64Klimit) - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, ACCELERATION_DEFAULT); - else - return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, ACCELERATION_DEFAULT); + return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); } else { @@ -718,23 +676,45 @@ int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int } -int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, unsigned acceleration) +int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { #if (HEAPMODE) - void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U64, 8); /* Aligned on 8-bytes boundaries */ + void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t_internal)); /* malloc-calloc always properly aligned */ #else - U64 ctx[LZ4_STREAMSIZE_U64]; /* Ensure data is aligned on 8-bytes boundaries */ + LZ4_stream_t ctx; + void* ctxPtr = &ctx; #endif - int result = LZ4_compress_fast_extState(ctx, source, dest, inputSize, maxOutputSize, acceleration); + int result = LZ4_compress_fast_extState(ctxPtr, source, dest, inputSize, maxOutputSize, acceleration); #if (HEAPMODE) - FREEMEM(ctx); + FREEMEM(ctxPtr); #endif return result; } +int LZ4_compress_default(const char* source, char* dest, int inputSize, int maxOutputSize) +{ + return LZ4_compress_fast(source, dest, inputSize, maxOutputSize, 1); +} + + +/* hidden debug function */ +/* strangely enough, gcc generates faster code when this function is uncommented, even if unused */ +int LZ4_compress_fast_force(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) +{ + LZ4_stream_t ctx; + + LZ4_resetStream(&ctx); + + if (inputSize < LZ4_64Klimit) + return LZ4_compress_generic(&ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); + else + return LZ4_compress_generic(&ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); +} + + /***************************************** * Experimental : Streaming functions *****************************************/ @@ -767,7 +747,7 @@ int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) const BYTE* const dictEnd = p + dictSize; const BYTE* base; - if (dict->initCheck) /* Uninitialized structure detected */ + if ((dict->initCheck) || (dict->currentOffset > 1 GB)) /* Uninitialized structure, or reuse overflow */ LZ4_resetStream(LZ4_dict); if (dictSize < (int)HASH_UNIT) @@ -815,7 +795,7 @@ static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src) } -int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_fast_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_stream; const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; @@ -824,6 +804,7 @@ int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_stream, const char* source, ch if (streamPtr->initCheck) return 0; /* Uninitialized structure detected */ if ((streamPtr->dictSize>0) && (smallest>dictEnd)) smallest = dictEnd; LZ4_renormDictT(streamPtr, smallest); + if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; /* Check overlapping input/dictionary space */ { @@ -842,9 +823,9 @@ int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_stream, const char* source, ch { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, dictSmall, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, dictSmall, acceleration); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, noDictIssue, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, noDictIssue, acceleration); streamPtr->dictSize += (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; @@ -854,9 +835,9 @@ int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_stream, const char* source, ch { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, dictSmall, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, dictSmall, acceleration); else - result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, noDictIssue, 1); + result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, noDictIssue, acceleration); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; @@ -929,7 +910,7 @@ FORCE_INLINE int LZ4_decompress_generic( ) { /* Local Variables */ - const BYTE* restrict ip = (const BYTE*) source; + const BYTE* ip = (const BYTE*) source; const BYTE* const iend = ip + inputSize; BYTE* op = (BYTE*) dest; @@ -1258,12 +1239,12 @@ int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compres * Obsolete Functions ***************************************************/ /* obsolete compression functions */ -int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_safe(source, dest, inputSize, maxOutputSize); } -int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_safe(source, dest, inputSize, LZ4_compressBound(inputSize)); } -int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, dstSize); } -int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_safe_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize)); } -int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_safe_continue(LZ4_stream, src, dst, srcSize, maxDstSize); } -int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_safe_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize)); } +int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_default(source, dest, inputSize, maxOutputSize); } +int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_default(source, dest, inputSize, LZ4_compressBound(inputSize)); } +int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, dstSize, 1); } +int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize), 1); } +int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_fast_continue(LZ4_stream, src, dst, srcSize, maxDstSize, 1); } +int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_fast_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize), 1); } /* These function names are deprecated and should no longer be used. diff --git a/lib/lz4.h b/lib/lz4.h index 68f43a7..b597064 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -70,11 +70,11 @@ int LZ4_versionNumber (void); * Simple Functions **************************************/ -int LZ4_compress_safe (const char* source, char* dest, int sourceSize, int maxDestSize); +int LZ4_compress_default(const char* source, char* dest, int sourceSize, int maxDestSize); int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize); /* -LZ4_compress_safe() : +LZ4_compress_default() : Compresses 'sourceSize' bytes from buffer 'source' into already allocated 'dest' buffer of size 'maxDestSize'. Compression is guaranteed to succeed if 'maxDestSize' >= LZ4_compressBound(sourceSize). @@ -86,7 +86,7 @@ LZ4_compress_safe() : sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE maxDestSize : full or partial size of buffer 'dest' (which must be already allocated) return : the number of bytes written into buffer 'dest' (necessarily <= maxOutputSize) - or 0 if compression fails + or 0 if compression fails LZ4_decompress_safe() : compressedSize : is obviously the source size @@ -95,7 +95,7 @@ LZ4_decompress_safe() : If the destination buffer is not large enough, decoding will stop and output an error code (<0). If the source stream is detected malformed, the function will stop decoding and return a negative result. This function is protected against buffer overflow exploits, including malicious data packets. - It never writes outside of output buffer, nor reads outside of input buffer. + It never writes outside output buffer, nor reads outside input buffer. */ @@ -108,36 +108,35 @@ LZ4_decompress_safe() : /* LZ4_compressBound() : Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible) - This function is primarily useful for memory allocation purposes (output buffer size). + This function is primarily useful for memory allocation purposes (destination buffer size). Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). - - inputSize : max supported value is LZ4_MAX_INPUT_SIZE - return : maximum output size in a "worst case" scenario - or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE) + Note that LZ4_compress_default() compress faster when dest buffer size is >= LZ4_compressBound(srcSize) + inputSize : max supported value is LZ4_MAX_INPUT_SIZE + return : maximum output size in a "worst case" scenario + or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE) */ int LZ4_compressBound(int inputSize); /* -LZ4_compress_safe_extState() : - Same compression function, just using an externally allocated memory space to store compression state. - Use LZ4_sizeofState() to know how much memory must be allocated, - and then, provide it as 'void* state' to compression functions. - Note that 'state' ptr must be aligned on 4-bytes boundaries. +LZ4_compress_fast() : + Same as LZ4_compress_default(), but allows to select an "acceleration" factor. + The larger the acceleration value, the faster the algorithm, but also the lesser the compression. + It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed. + An acceleration value of "0" means "use Default value" (see lz4.c) + An acceleration value of "1" is the same as regular LZ4_compress_default() */ -int LZ4_sizeofState(void); -int LZ4_compress_safe_extState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxDestSize, int acceleration); + /* -LZ4_compress_fast() : - Same as LZ4_compress_safe(), but allows to select an "acceleration" factor. - The larger the acceleration value, the faster the algorithm, but also the lesser the compression. - It's a trade-off. It can be fine tuned, with each successive value providing an additional +2/3% to speed. - An acceleration value of "0" means "use Default value", which is typically 17 (see lz4.c source code). - An acceleration value of "1" is the same as regular LZ4_compress_safe() - Note : this function is "safe", even if its name does not explicitly contain the word. It's just faster and compress less. +LZ4_compress_fast_extState() : + Same compression function, just using an externally allocated memory space to store compression state. + Use LZ4_sizeofState() to know how much memory must be allocated, + and allocate it on 8-bytes boundaries (using malloc() typically). + Then, provide it as 'void* state' to compression function. */ -int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxDestSize, unsigned acceleration); -int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, unsigned acceleration); +int LZ4_sizeofState(void); +int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, int acceleration); /* @@ -186,7 +185,7 @@ typedef struct { long long table[LZ4_STREAMSIZE_U64]; } LZ4_stream_t; * LZ4_resetStream * Use this function to init an allocated LZ4_stream_t structure */ -void LZ4_resetStream (LZ4_stream_t* LZ4_streamPtr); +void LZ4_resetStream (LZ4_stream_t* streamPtr); /* * LZ4_createStream will allocate and initialize an LZ4_stream_t structure @@ -195,7 +194,7 @@ void LZ4_resetStream (LZ4_stream_t* LZ4_streamPtr); * They are more future proof, in case of a change of LZ4_stream_t size. */ LZ4_stream_t* LZ4_createStream(void); -int LZ4_freeStream (LZ4_stream_t* LZ4_streamPtr); +int LZ4_freeStream (LZ4_stream_t* streamPtr); /* * LZ4_loadDict @@ -204,27 +203,27 @@ int LZ4_freeStream (LZ4_stream_t* LZ4_streamPtr); * Loading a size of 0 is allowed. * Return : dictionary size, in bytes (necessarily <= 64 KB) */ -int LZ4_loadDict (LZ4_stream_t* LZ4_streamPtr, const char* dictionary, int dictSize); +int LZ4_loadDict (LZ4_stream_t* streamPtr, const char* dictionary, int dictSize); /* - * LZ4_compress_safe_continue - * Compress data block 'source', using data from previous blocks to improve compression ratio. + * LZ4_compress_fast_continue + * Compress buffer content 'src', using data from previously compressed blocks as dictionary to improve compression ratio. * Important : Previous data blocks are assumed to still be present and unmodified ! - * dest buffer must be already allocated. - * if maxOutpuSize >= (inputSize), compression is guaranteed to succeed. - * if not, and if target size objective cannot be met, compression stops, and function returns a zero. + * 'dst' buffer must be already allocated. + * If maxDstSize >= LZ4_compressBound(srcSize), compression is guaranteed to succeed, and runs faster. + * If not, and if compressed data cannot fit into 'dst' buffer size, compression stops, and function returns a zero. */ -int LZ4_compress_safe_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_fast_continue (LZ4_stream_t* streamPtr, const char* src, char* dst, int srcSize, int maxDstSize, int acceleration); /* * LZ4_saveDict * If previously compressed data block is not guaranteed to remain available at its memory location * save it into a safer place (char* safeBuffer) * Note : you don't need to call LZ4_loadDict() afterwards, - * dictionary is immediately usable, you can therefore call again LZ4_compress_continue() + * dictionary is immediately usable, you can therefore call LZ4_compress_fast_continue() * Return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error */ -int LZ4_saveDict (LZ4_stream_t* LZ4_streamPtr, char* safeBuffer, int dictSize); +int LZ4_saveDict (LZ4_stream_t* streamPtr, char* safeBuffer, int dictSize); /************************************************ @@ -310,7 +309,7 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") # define LZ4_DEPRECATED(message) # endif -#endif // LZ4_DEPRECATE_WARNING_DEFBLOCK +#endif /* LZ4_DEPRECATE_WARNING_DEFBLOCK */ /* Obsolete compression functions */ /* These functions are planned to start generate warnings by r131 approximately */ @@ -338,8 +337,8 @@ LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); /* Obsolete streaming decoding functions */ -LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize); -LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize); +LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* src, char* dst, int compressedSize, int maxDstSize); +LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* src, char* dst, int originalSize); #if defined (__cplusplus) diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 5fad2e8..408e871 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -515,13 +515,13 @@ static size_t LZ4F_compressBlock(void* dst, const void* src, size_t srcSize, com static int LZ4F_localLZ4_compress_limitedOutput_withState(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compress_safe_extState(ctx, src, dst, srcSize, dstSize); + return LZ4_compress_limitedOutput_withState(ctx, src, dst, srcSize, dstSize); } static int LZ4F_localLZ4_compress_limitedOutput_continue(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compress_safe_continue((LZ4_stream_t*)ctx, src, dst, srcSize, dstSize); + return LZ4_compress_limitedOutput_continue((LZ4_stream_t*)ctx, src, dst, srcSize, dstSize); } static int LZ4F_localLZ4_compressHC_limitedOutput_continue(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) diff --git a/programs/bench.c b/programs/bench.c index 384a425..3a93cf9 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -58,7 +58,7 @@ #include "lz4.h" #define COMPRESSOR0 LZ4_compress_local -static int LZ4_compress_local(const char* src, char* dst, int srcSize, int dstSize, int clevel) { (void)clevel; return LZ4_compress_safe(src, dst, srcSize, dstSize); } +static int LZ4_compress_local(const char* src, char* dst, int srcSize, int dstSize, int clevel) { (void)clevel; return LZ4_compress_default(src, dst, srcSize, dstSize); } #include "lz4hc.h" #define COMPRESSOR1 LZ4_compressHC_safe #define DEFAULTCOMPRESSOR COMPRESSOR0 diff --git a/programs/fullbench.c b/programs/fullbench.c index ed53819..0d08a40 100644 --- a/programs/fullbench.c +++ b/programs/fullbench.c @@ -30,7 +30,7 @@ #define _CRT_SECURE_NO_WARNINGS #define _CRT_SECURE_NO_DEPRECATE /* VS2005 */ -// Unix Large Files support (>4GB) +/* Unix Large Files support (>4GB) */ #if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits requires specific definitions # define _LARGEFILE_SOURCE # define _FILE_OFFSET_BITS 64 @@ -47,17 +47,17 @@ /************************************** * Includes **************************************/ -#include // malloc -#include // fprintf, fopen, ftello64 -#include // stat64 -#include // stat64 -#include // strcmp +#include /* malloc, free */ +#include /* fprintf, fopen, ftello64 */ +#include /* stat64 */ +#include /* stat64 */ +#include /* strcmp */ -// Use ftime() if gettimeofday() is not available on your target +/* Use ftime() if gettimeofday() is not available on your target */ #if defined(BMK_LEGACY_TIMER) -# include // timeb, ftime +# include /* timeb, ftime */ #else -# include // gettimeofday +# include /* gettimeofday */ #endif #include "lz4.h" @@ -75,16 +75,11 @@ # define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) #endif -// GCC does not support _rotl outside of Windows -#if !defined(_WIN32) -# define _rotl(x,r) ((x << r) | (x >> (32 - r))) -#endif - /************************************** * Basic Types **************************************/ -#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L // C99 +#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # include typedef uint8_t BYTE; typedef uint16_t U16; @@ -118,7 +113,7 @@ #define GB *(1U<<30) #define KNUTH 2654435761U -#define MAX_MEM (1984 MB) +#define MAX_MEM (1920 MB) #define DEFAULT_CHUNKSIZE (4 MB) #define ALL_COMPRESSORS 0 @@ -139,42 +134,42 @@ struct chunkParameters /************************************** -* MACRO +* Macros **************************************/ #define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define PROGRESS(...) no_prompt ? 0 : DISPLAY(__VA_ARGS__) - +#define PROGRESS(...) g_noPrompt ? 0 : DISPLAY(__VA_ARGS__) /************************************** * Benchmark Parameters **************************************/ -static int chunkSize = DEFAULT_CHUNKSIZE; -static int nbIterations = NBLOOPS; -static int BMK_pause = 0; -static int compressionTest = 1; -static int decompressionTest = 1; -static int compressionAlgo = ALL_COMPRESSORS; -static int decompressionAlgo = ALL_DECOMPRESSORS; -static int no_prompt = 0; +static int g_chunkSize = DEFAULT_CHUNKSIZE; +static int g_nbIterations = NBLOOPS; +static int g_pause = 0; +static int g_compressionTest = 1; +static int g_compressionAlgo = ALL_COMPRESSORS; +static int g_decompressionTest = 1; +static int g_decompressionAlgo = ALL_DECOMPRESSORS; +static int g_noPrompt = 0; -void BMK_SetBlocksize(int bsize) +static void BMK_setBlocksize(int bsize) { - chunkSize = bsize; - DISPLAY("-Using Block Size of %i KB-\n", chunkSize>>10); + g_chunkSize = bsize; + DISPLAY("-Using Block Size of %i KB-\n", g_chunkSize>>10); } -void BMK_SetNbIterations(int nbLoops) +static void BMK_setNbIterations(int nbLoops) { - nbIterations = nbLoops; - DISPLAY("- %i iterations -\n", nbIterations); + g_nbIterations = nbLoops; + DISPLAY("- %i iterations -\n", g_nbIterations); } -void BMK_SetPause(void) +static void BMK_setPause(void) { - BMK_pause = 1; + g_pause = 1; } + /********************************************************* * Private functions *********************************************************/ @@ -253,7 +248,7 @@ static U64 BMK_GetFileSize(char* infilename) struct stat statbuf; r = stat(infilename, &statbuf); #endif - if (r || !S_ISREG(statbuf.st_mode)) return 0; // No good... + if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ return (U64)statbuf.st_size; } @@ -403,11 +398,46 @@ static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSiz return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); } -static int local_LZ4_compress_fast(const char* in, char* out, int inSize) +static int local_LZ4_compress_default_large(const char* in, char* out, int inSize) +{ + return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)); +} + +static int local_LZ4_compress_default_small(const char* in, char* out, int inSize) +{ + return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compress_fast0(const char* in, char* out, int inSize) { return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); } +static int local_LZ4_compress_fast1(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 1); +} + +static int local_LZ4_compress_fast2(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 2); +} + +static int local_LZ4_compress_fast17(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 17); +} + +static int local_LZ4_compress_fast_extState0(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast_extState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); +} + +static int local_LZ4_compress_fast_continue0(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); +} + static int local_LZ4_compress_withState(const char* in, char* out, int inSize) { return LZ4_compress_withState(&LZ4_stream, in, out, inSize); @@ -575,10 +605,10 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); /* Allocation */ - chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)chunkSize)+1) * sizeof(struct chunkParameters)); + chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)g_chunkSize)+1) * sizeof(struct chunkParameters)); orig_buff = (char*) malloc(benchedSize); - nbChunks = (int) ((benchedSize + (chunkSize-1)) / chunkSize); - maxCompressedChunkSize = LZ4_compressBound(chunkSize); + nbChunks = (int) ((benchedSize + (g_chunkSize-1)) / g_chunkSize); + maxCompressedChunkSize = LZ4_compressBound(g_chunkSize); compressedBuffSize = nbChunks * maxCompressedChunkSize; compressed_buff = (char*)malloc((size_t)compressedBuffSize); if(!chunkP || !orig_buff || !compressed_buff) @@ -619,7 +649,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) DISPLAY(" %s : \n", inFileName); /* Bench Compression Algorithms */ - for (cAlgNb=1; (cAlgNb <= NB_COMPRESSION_ALGORITHMS) && (compressionTest); cAlgNb++) + for (cAlgNb=0; (cAlgNb <= NB_COMPRESSION_ALGORITHMS) && (g_compressionTest); cAlgNb++) { const char* compressorName; int (*compressionFunction)(const char*, char*, int); @@ -627,7 +657,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) double bestTime = 100000000.; /* filter compressionAlgo only */ - if ((compressionAlgo != ALL_COMPRESSORS) && (compressionAlgo != cAlgNb)) continue; + if ((g_compressionAlgo != ALL_COMPRESSORS) && (g_compressionAlgo != cAlgNb)) continue; /* Init data chunks */ { @@ -635,12 +665,12 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) size_t remaining = benchedSize; char* in = orig_buff; char* out = compressed_buff; - nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); + nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); for (i=0; i chunkSize) { chunkP[i].origSize = chunkSize; remaining -= chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } + chunkP[i].origBuffer = in; in += g_chunkSize; + if ((int)remaining > g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; chunkP[i].compressedSize = 0; } @@ -648,34 +678,44 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) switch(cAlgNb) { - case 1 : compressionFunction = LZ4_compress; compressorName = "LZ4_compress"; break; - case 2 : compressionFunction = local_LZ4_compress_limitedOutput; compressorName = "LZ4_compress_limitedOutput"; break; - case 3 : compressionFunction = local_LZ4_compress_withState; compressorName = "LZ4_compress_withState"; break; - case 4 : compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; - case 5 : compressionFunction = local_LZ4_compress_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_continue"; break; - case 6 : compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_limitedOutput_continue"; break; - case 7 : compressionFunction = local_LZ4_compress_fast; compressorName = "LZ4_compress_fast"; break; - case 8 : compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; - case 9 : compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; - case 10 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; - case 11: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; - case 12: compressionFunction = local_LZ4_compressHC_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_continue"; break; - case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; - case 14: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; - case 15: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; + case 0 : DISPLAY("Compression functions : \n"); continue; + case 1 : compressionFunction = local_LZ4_compress_default_large; compressorName = "LZ4_compress_default"; break; + case 2 : compressionFunction = local_LZ4_compress_default_small; compressorName = "LZ4_compress_default(small dst)"; break; + case 3 : compressionFunction = local_LZ4_compress_fast0; compressorName = "LZ4_compress_fast(0)"; break; + case 4 : compressionFunction = local_LZ4_compress_fast1; compressorName = "LZ4_compress_fast(1)"; break; + case 5 : compressionFunction = local_LZ4_compress_fast2; compressorName = "LZ4_compress_fast(2)"; break; + case 6 : compressionFunction = local_LZ4_compress_fast17; compressorName = "LZ4_compress_fast(17)"; break; + case 7 : compressionFunction = local_LZ4_compress_fast_extState0; compressorName = "LZ4_compress_fast_extState(0)"; break; + case 8 : compressionFunction = local_LZ4_compress_fast_continue0; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_fast_continue(0)"; break; + + case 10: compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; + case 11: compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; + case 12 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; + case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; + case 14: compressionFunction = local_LZ4_compressHC_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_continue"; break; + case 15: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; + case 20: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; + case 30: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; chunkP[0].origSize = (int)benchedSize; nbChunks=1; break; - case 16: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; + case 40: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; LZ4_loadDict(&LZ4_stream, chunkP[0].origBuffer, chunkP[0].origSize); break; - case 17: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; + case 41: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; LZ4_loadDictHC(&LZ4_streamHC, chunkP[0].origBuffer, chunkP[0].origSize); break; + case 60: DISPLAY("Obsolete compression functions : \n"); continue; + case 61: compressionFunction = LZ4_compress; compressorName = "LZ4_compress"; break; + case 62: compressionFunction = local_LZ4_compress_limitedOutput; compressorName = "LZ4_compress_limitedOutput"; break; + case 63: compressionFunction = local_LZ4_compress_withState; compressorName = "LZ4_compress_withState"; break; + case 64: compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; + case 65: compressionFunction = local_LZ4_compress_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_continue"; break; + case 66: compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_limitedOutput_continue"; break; default : continue; /* unknown ID : just skip */ } - for (loopNb = 1; loopNb <= nbIterations; loopNb++) + for (loopNb = 1; loopNb <= g_nbIterations; loopNb++) { double averageTime; int milliTime; @@ -721,12 +761,12 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) char* in = orig_buff; char* out = compressed_buff; - nbChunks = (int) (((int)benchedSize + (chunkSize-1))/ chunkSize); + nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); for (i=0; i chunkSize) { chunkP[i].origSize = chunkSize; remaining -= chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } + chunkP[i].origBuffer = in; in += g_chunkSize; + if ((int)remaining > g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; chunkP[i].compressedSize = 0; } @@ -738,16 +778,17 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) } /* Decompression Algorithms */ - for (dAlgNb=1; (dAlgNb <= NB_DECOMPRESSION_ALGORITHMS) && (decompressionTest); dAlgNb++) + for (dAlgNb=0; (dAlgNb <= NB_DECOMPRESSION_ALGORITHMS) && (g_decompressionTest); dAlgNb++) { const char* dName; int (*decompressionFunction)(const char*, char*, int, int); double bestTime = 100000000.; - if ((decompressionAlgo != ALL_DECOMPRESSORS) && (decompressionAlgo != dAlgNb)) continue; + if ((g_decompressionAlgo != ALL_DECOMPRESSORS) && (g_decompressionAlgo != dAlgNb)) continue; switch(dAlgNb) { + case 0: DISPLAY("Decompression functions : \n"); continue; case 1: decompressionFunction = local_LZ4_decompress_fast; dName = "LZ4_decompress_fast"; break; case 3: decompressionFunction = local_LZ4_decompress_fast_usingDict; dName = "LZ4_decompress_fast_usingDict"; break; case 4: decompressionFunction = LZ4_decompress_safe; dName = "LZ4_decompress_safe"; break; @@ -774,7 +815,7 @@ int fullSpeedBench(char** fileNamesTable, int nbFiles) { size_t i; for (i=0; i= '0') && (argument[1]<= '9')) { - compressionAlgo *= 10; - compressionAlgo += argument[1] - '0'; + g_compressionAlgo *= 10; + g_compressionAlgo += argument[1] - '0'; argument++; } break; // Select decompression algorithm only case 'd': - compressionTest = 0; + g_compressionTest = 0; while ((argument[1]>= '0') && (argument[1]<= '9')) { - decompressionAlgo *= 10; - decompressionAlgo += argument[1] - '0'; + g_decompressionAlgo *= 10; + g_decompressionAlgo += argument[1] - '0'; argument++; } break; @@ -921,7 +962,7 @@ int main(int argc, char** argv) { int B = argument[1] - '0'; int S = 1 << (8 + 2*B); - BMK_SetBlocksize(S); + BMK_setBlocksize(S); argument++; break; } @@ -936,13 +977,13 @@ _exit_blockProperties: if ((argument[1] >='0') && (argument[1] <='9')) { int iters = argument[1] - '0'; - BMK_SetNbIterations(iters); + BMK_setNbIterations(iters); argument++; } break; // Pause at the end (hidden option) - case 'p': BMK_SetPause(); break; + case 'p': BMK_setPause(); break; // Unknown command default : badusage(exename); return 1; diff --git a/programs/lz4io.c b/programs/lz4io.c index fb1f428..16fc879 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -325,7 +325,7 @@ static void LZ4IO_writeLE32 (void* p, unsigned value32) static int LZ4IO_LZ4_compress(const char* src, char* dst, int srcSize, int dstSize, int cLevel) { (void)cLevel; - return LZ4_compress_safe(src, dst, srcSize, dstSize); + return LZ4_compress_fast(src, dst, srcSize, dstSize, 1); } /* LZ4IO_compressFilename_Legacy : From 1171303a4ff6566f7f9d4b1f91d4d73f0bf00584 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 3 May 2015 18:02:48 +0100 Subject: [PATCH 113/121] Updated streaming examples --- examples/blockStreaming_doubleBuffer.c | 4 ++-- examples/blockStreaming_lineByLine.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/blockStreaming_doubleBuffer.c b/examples/blockStreaming_doubleBuffer.c index 58f2bdb..59355da 100644 --- a/examples/blockStreaming_doubleBuffer.c +++ b/examples/blockStreaming_doubleBuffer.c @@ -53,8 +53,8 @@ void test_compress(FILE* outFp, FILE* inpFp) { char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)]; - const int cmpBytes = LZ4_compress_safe_continue( - lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf)); + const int cmpBytes = LZ4_compress_fast_continue( + lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf), 1); if(cmpBytes <= 0) { break; } diff --git a/examples/blockStreaming_lineByLine.c b/examples/blockStreaming_lineByLine.c index 8ae268f..c4fd2e3 100644 --- a/examples/blockStreaming_lineByLine.c +++ b/examples/blockStreaming_lineByLine.c @@ -64,8 +64,8 @@ static void test_compress( #endif { - const int cmpBytes = LZ4_compress_safe_continue( - lz4Stream, inpPtr, cmpBuf, inpBytes, cmpBufBytes); + const int cmpBytes = LZ4_compress_fast_continue( + lz4Stream, inpPtr, cmpBuf, inpBytes, cmpBufBytes, 1); if (cmpBytes <= 0) break; write_uint16(outFp, (uint16_t) cmpBytes); write_bin(outFp, cmpBuf, cmpBytes); From b4348a47189f7bce93537a85906d26b09be7e802 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 3 May 2015 18:06:01 +0100 Subject: [PATCH 114/121] Fixed minor Visual warning --- lib/lz4.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/lz4.c b/lib/lz4.c index 3c182aa..bb40f76 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -654,7 +654,7 @@ _last_literals: } -int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, volatile int acceleration) +int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { LZ4_resetStream((LZ4_stream_t*)state); if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; From e05088d0eb500d8d673e081929620e538df3d718 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 3 May 2015 20:57:21 +0100 Subject: [PATCH 115/121] Updated lz4hc API --- examples/Makefile | 25 ++---- examples/blockStreaming_doubleBuffer.c | 8 +- examples/blockStreaming_lineByLine.c | 12 +-- lib/lz4frame.c | 4 +- lib/lz4hc.c | 37 ++++----- lib/lz4hc.h | 101 +++++++++++++------------ programs/bench.c | 2 +- programs/lz4io.c | 2 +- 8 files changed, 94 insertions(+), 97 deletions(-) diff --git a/examples/Makefile b/examples/Makefile index 0c4cf13..808b511 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -1,6 +1,7 @@ # ########################################################################## # LZ4 examples - Makefile # Copyright (C) Yann Collet 2011-2014 +# # GPL v2 License # # This program is free software; you can redistribute it and/or modify @@ -21,29 +22,17 @@ # - LZ4 source repository : http://code.google.com/p/lz4/ # - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c # ########################################################################## -# lz4 : Command Line Utility, supporting gzip-like arguments -# lz4c : CLU, supporting also legacy lz4demo arguments -# lz4c32: Same as lz4c, but forced to compile in 32-bits mode -# fuzzer : Test tool, to check lz4 integrity on target platform -# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode -# fullbench : Precisely measure speed for each LZ4 function variant -# fullbench32: Same as fullbench, but forced to compile in 32-bits mode +# This makefile compile and test +# example programs, using (mostly) LZ4 streaming library, +# kindly provided by Takayuki Matsuoka # ########################################################################## -CC := $(CC) CFLAGS ?= -O3 -CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -Wno-missing-braces # Wno-missing-braces required due to GCC <4.8.3 bug -FLAGS = -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) +CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes +FLAGS := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) TESTFILE= Makefile -LZ4DIR = ../lib - - -# Minimize test target for Travis CI's Build Matrix -ifeq ($(LZ4_TRAVIS_CI_ENV),-m32) -CFLAGS += -m32 -else ifeq ($(LZ4_TRAVIS_CI_ENV),-m64) -endif +LZ4DIR := ../lib # Define *.exe as extension for Windows systems diff --git a/examples/blockStreaming_doubleBuffer.c b/examples/blockStreaming_doubleBuffer.c index 59355da..efe6fc6 100644 --- a/examples/blockStreaming_doubleBuffer.c +++ b/examples/blockStreaming_doubleBuffer.c @@ -38,11 +38,13 @@ size_t read_bin(FILE* fp, void* array, size_t arrayBytes) { void test_compress(FILE* outFp, FILE* inpFp) { - LZ4_stream_t lz4Stream_body = { 0 }; + LZ4_stream_t lz4Stream_body; LZ4_stream_t* lz4Stream = &lz4Stream_body; char inpBuf[2][BLOCK_BYTES]; int inpBufIndex = 0; + + LZ4_resetStream(lz4Stream); for(;;) { char* const inpPtr = inpBuf[inpBufIndex]; @@ -71,12 +73,14 @@ void test_compress(FILE* outFp, FILE* inpFp) void test_decompress(FILE* outFp, FILE* inpFp) { - LZ4_streamDecode_t lz4StreamDecode_body = { 0 }; + LZ4_streamDecode_t lz4StreamDecode_body; LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body; char decBuf[2][BLOCK_BYTES]; int decBufIndex = 0; + LZ4_setStreamDecode(lz4StreamDecode, NULL, 0); + for(;;) { char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)]; int cmpBytes = 0; diff --git a/examples/blockStreaming_lineByLine.c b/examples/blockStreaming_lineByLine.c index c4fd2e3..f449aa3 100644 --- a/examples/blockStreaming_lineByLine.c +++ b/examples/blockStreaming_lineByLine.c @@ -42,8 +42,8 @@ static void test_compress( { LZ4_stream_t* const lz4Stream = LZ4_createStream(); const size_t cmpBufBytes = LZ4_COMPRESSBOUND(messageMaxBytes); - char* const cmpBuf = malloc(cmpBufBytes); - char* const inpBuf = malloc(ringBufferBytes); + char* const cmpBuf = (char*) malloc(cmpBufBytes); + char* const inpBuf = (char*) malloc(ringBufferBytes); int inpOffset = 0; for ( ; ; ) @@ -90,8 +90,8 @@ static void test_decompress( size_t ringBufferBytes) { LZ4_streamDecode_t* const lz4StreamDecode = LZ4_createStreamDecode(); - char* const cmpBuf = malloc(LZ4_COMPRESSBOUND(messageMaxBytes)); - char* const decBuf = malloc(ringBufferBytes); + char* const cmpBuf = (char*) malloc(LZ4_COMPRESSBOUND(messageMaxBytes)); + char* const decBuf = (char*) malloc(ringBufferBytes); int decOffset = 0; for ( ; ; ) @@ -125,8 +125,8 @@ static int compare(FILE* f0, FILE* f1) { int result = 0; const size_t tempBufferBytes = 65536; - char* const b0 = malloc(tempBufferBytes); - char* const b1 = malloc(tempBufferBytes); + char* const b0 = (char*) malloc(tempBufferBytes); + char* const b1 = (char*) malloc(tempBufferBytes); while(0 == result) { diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 408e871..d73436c 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -527,7 +527,7 @@ static int LZ4F_localLZ4_compress_limitedOutput_continue(void* ctx, const char* static int LZ4F_localLZ4_compressHC_limitedOutput_continue(void* ctx, const char* src, char* dst, int srcSize, int dstSize, int level) { (void) level; - return LZ4_compressHC_safe_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); + return LZ4_compress_HC_continue((LZ4_streamHC_t*)ctx, src, dst, srcSize, dstSize); } static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int level) @@ -537,7 +537,7 @@ static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int lev if (blockMode == LZ4F_blockIndependent) return LZ4F_localLZ4_compress_limitedOutput_withState; return LZ4F_localLZ4_compress_limitedOutput_continue; } - if (blockMode == LZ4F_blockIndependent) return LZ4_compressHC_safe_extStateHC; + if (blockMode == LZ4F_blockIndependent) return LZ4_compress_HC_extStateHC; return LZ4F_localLZ4_compressHC_limitedOutput_continue; } diff --git a/lib/lz4hc.c b/lib/lz4hc.c index 1db3d98..fa05ee3 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -538,20 +538,20 @@ _Search3: int LZ4_sizeofStateHC(void) { return sizeof(LZ4HC_Data_Structure); } -int LZ4_compressHC_safe_extStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) +int LZ4_compress_HC_extStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel) { if (((size_t)(state)&(sizeof(void*)-1)) != 0) return 0; /* Error : state is not aligned for pointers (32 or 64 bits) */ - LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)source); - if (maxOutputSize < LZ4_compressBound(inputSize)) - return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput); + LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)src); + if (maxDstSize < LZ4_compressBound(srcSize)) + return LZ4HC_compress_generic (state, src, dst, srcSize, maxDstSize, compressionLevel, limitedOutput); else - return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, noLimit); + return LZ4HC_compress_generic (state, src, dst, srcSize, maxDstSize, compressionLevel, noLimit); } -int LZ4_compressHC_safe(const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) +int LZ4_compress_HC(const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel) { LZ4HC_Data_Structure state; - return LZ4_compressHC_safe_extStateHC(&state, source, dest, inputSize, maxOutputSize, compressionLevel); + return LZ4_compress_HC_extStateHC(&state, src, dst, srcSize, maxDstSize, compressionLevel); } @@ -639,7 +639,7 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr, return LZ4HC_compress_generic (ctxPtr, source, dest, inputSize, maxOutputSize, ctxPtr->compressionLevel, limit); } -int LZ4_compressHC_safe_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize) +int LZ4_compress_HC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize) { if (maxOutputSize < LZ4_compressBound(inputSize)) return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput); @@ -675,19 +675,20 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS ***********************************/ /* Deprecated compression functions */ /* These functions are planned to start generate warnings by r131 approximately */ -int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } -int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, 0); } -int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compressHC_safe (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } -int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compressHC_safe(src, dst, srcSize, maxDstSize, cLevel); } -int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); } -int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe_extStateHC (state, src, dst, srcSize, maxDstSize, 0); } -int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compressHC_safe_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } -int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compressHC_safe_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); } -int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compressHC_safe_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); } -int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compressHC_safe_continue (ctx, src, dst, srcSize, maxDstSize); } +int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } +int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, 0); } +int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } +int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, cLevel); } +int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); } +int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, maxDstSize, 0); } +int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } +int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); } +int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); } +int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, maxDstSize); } /* Deprecated streaming functions */ +/* These functions currently generate deprecation warnings */ int LZ4_sizeofStreamStateHC(void) { return LZ4_STREAMHCSIZE; } int LZ4_resetStreamStateHC(void* state, const char* inputBuffer) diff --git a/lib/lz4hc.h b/lib/lz4hc.h index 2667044..f8461b4 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -47,18 +47,18 @@ extern "C" { /************************************** * Block Compression **************************************/ -int LZ4_compressHC_safe (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +int LZ4_compress_HC (const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel); /* -LZ4_compressHC_safe : - return : the number of bytes in compressed buffer dest - or 0 if compression fails. - note : destination buffer must be already allocated. - To guarantee compression completion, size it to handle worst cases situations (data not compressible) - Worst case size evaluation is provided by function LZ4_compressBound() (see "lz4.h") - inputSize : Max supported value is LZ4_MAX_INPUT_SIZE (see "lz4.h") - compressionLevel : Recommended values are between 4 and 9, although any value between 0 and 16 will work. - 0 means use default 'compressionLevel' value. - Values >16 behave the same as 16. +LZ4_compress_HC : + Destination buffer 'dst' must be already allocated. + Compression completion is guaranteed if 'dst' buffer is sized to handle worst circumstances (data not compressible) + Worst size evaluation is provided by function LZ4_compressBound() (see "lz4.h") + srcSize : Max supported value is LZ4_MAX_INPUT_SIZE (see "lz4.h") + compressionLevel : Recommended values are between 4 and 9, although any value between 0 and 16 will work. + 0 means "use default value" (see lz4hc.c). + Values >16 behave the same as 16. + return : the number of bytes written into buffer 'dst' + or 0 if compression fails. */ @@ -68,18 +68,18 @@ LZ4_compressHC_safe : int LZ4_sizeofStateHC(void); -int LZ4_compressHC_safe_extStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); - +int LZ4_compress_HC_extStateHC(void* state, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel); /* -This function is provided should you prefer to allocate memory for compression tables with your own allocation methods. -To know how much memory must be allocated for the compression tables, use : -int LZ4_sizeofStateHC(); +LZ4_compress_HC_extStateHC() : + Use this function if you prefer to manually allocate memory for compression tables. + To know how much memory must be allocated for the compression tables, use : + int LZ4_sizeofStateHC(); -Note that tables must be aligned for pointer (32 or 64 bits), otherwise compression will fail (return code 0). + Allocated memory must be aligned on 8-bytes boundaries (which a normal malloc() will do properly). -The allocated memory can be provided to the compression functions using 'void* state' parameter. -LZ4_compressHC_safe_extStateHC() is equivalent to previously described function. -It just uses externally allocated memory for stateHC instead of allocating their own (on stack, or on heap). + The allocated memory can then be provided to the compression functions using 'void* state' parameter. + LZ4_compress_HC_extStateHC() is equivalent to previously described function. + It just uses externally allocated memory for stateHC. */ @@ -90,53 +90,56 @@ It just uses externally allocated memory for stateHC instead of allocating their #define LZ4_STREAMHCSIZE_SIZET (LZ4_STREAMHCSIZE / sizeof(size_t)) typedef struct { size_t table[LZ4_STREAMHCSIZE_SIZET]; } LZ4_streamHC_t; /* -LZ4_streamHC_t -This structure allows static allocation of LZ4 HC streaming state. -State must then be initialized using LZ4_resetStreamHC() before first use. + LZ4_streamHC_t + This structure allows static allocation of LZ4 HC streaming state. + State must then be initialized using LZ4_resetStreamHC() before first use. -Static allocation should only be used with statically linked library. -If you want to use LZ4 as a DLL, please use construction functions below, which are more future-proof. + Static allocation should only be used in combination with static linking. + If you want to use LZ4 as a DLL, please use construction functions below, which are future-proof. */ LZ4_streamHC_t* LZ4_createStreamHC(void); -int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr); +int LZ4_freeStreamHC (LZ4_streamHC_t* streamHCPtr); /* -These functions create and release memory for LZ4 HC streaming state. -Newly created states are already initialized. -Existing state space can be re-used anytime using LZ4_resetStreamHC(). -If you use LZ4 as a DLL, please use these functions instead of direct struct allocation, -to avoid size mismatch between different versions. + These functions create and release memory for LZ4 HC streaming state. + Newly created states are already initialized. + Existing state space can be re-used anytime using LZ4_resetStreamHC(). + If you use LZ4 as a DLL, use these functions instead of static structure allocation, + to avoid size mismatch between different versions. */ -void LZ4_resetStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel); -int LZ4_loadDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, const char* dictionary, int dictSize); +void LZ4_resetStreamHC (LZ4_streamHC_t* streamHCPtr, int compressionLevel); +int LZ4_loadDictHC (LZ4_streamHC_t* streamHCPtr, const char* dictionary, int dictSize); -int LZ4_compressHC_safe_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); +int LZ4_compress_HC_continue (LZ4_streamHC_t* streamHCPtr, const char* src, char* dst, int srcSize, int maxDstSize); -int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int maxDictSize); +int LZ4_saveDictHC (LZ4_streamHC_t* streamHCPtr, char* safeBuffer, int maxDictSize); /* -These functions compress data in successive blocks of any size, using previous blocks as dictionary. -One key assumption is that each previous block will remain read-accessible while compressing next block. + These functions compress data in successive blocks of any size, using previous blocks as dictionary. + One key assumption is that previous blocks (up to 64 KB) remain read-accessible while compressing next blocks. + There is an exception for ring buffers, which can be smaller 64 KB. + Such case is automatically detected and correctly handled by LZ4_compress_HC_continue(). -Before starting compression, state must be properly initialized, using LZ4_resetStreamHC(). -A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional). + Before starting compression, state must be properly initialized, using LZ4_resetStreamHC(). + A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional). -Then, use LZ4_compressHC_safe_continue() to compress each successive block. -It works like LZ4_compressHC_safe(), but use previous memory blocks to improve compression. -Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression. + Then, use LZ4_compress_HC_continue() to compress each successive block. + It works like LZ4_compress_HC(), but use previous memory blocks as dictionary to improve compression. + Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression. + As a reminder, size 'dst' buffer to handle worst cases, using LZ4_compressBound(), to ensure success of compression operation. -If, for any reason, previous data block can't be preserved in memory during next compression block, -you must save it to a safer memory space, -using LZ4_saveDictHC(). + If, for any reason, previous data blocks can't be preserved unmodified in memory during next compression block, + you must save it to a safer memory space, using LZ4_saveDictHC(). + Return value of LZ4_saveDictHC() is the size of dictionary effectively saved into 'safeBuffer'. */ /************************************** - * Deprecated Functions - * ************************************/ +* Deprecated Functions +**************************************/ /* Deprecate Warnings */ /* Should these warnings messages be a problem, it is generally possible to disable them, @@ -175,8 +178,8 @@ int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, cons LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (const char* inputBuffer); LZ4_DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); LZ4_DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); -LZ4_DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); -LZ4_DEPRECATED("use LZ4_compressHC_safe_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); +LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); +LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); LZ4_DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); diff --git a/programs/bench.c b/programs/bench.c index 3a93cf9..9f949c4 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -60,7 +60,7 @@ #define COMPRESSOR0 LZ4_compress_local static int LZ4_compress_local(const char* src, char* dst, int srcSize, int dstSize, int clevel) { (void)clevel; return LZ4_compress_default(src, dst, srcSize, dstSize); } #include "lz4hc.h" -#define COMPRESSOR1 LZ4_compressHC_safe +#define COMPRESSOR1 LZ4_compress_HC #define DEFAULTCOMPRESSOR COMPRESSOR0 #include "xxhash.h" diff --git a/programs/lz4io.c b/programs/lz4io.c index 16fc879..15c93f4 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -347,7 +347,7 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output /* Init */ start = clock(); - if (compressionlevel < 3) compressionFunction = LZ4IO_LZ4_compress; else compressionFunction = LZ4_compressHC_safe; + if (compressionlevel < 3) compressionFunction = LZ4IO_LZ4_compress; else compressionFunction = LZ4_compress_HC; if (LZ4IO_getFiles(input_filename, output_filename, &finput, &foutput)) EXM_THROW(20, "File error"); From 05b0aa62a785bbf17b2452a0e7cda04bc630cc27 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 4 May 2015 14:43:37 +0100 Subject: [PATCH 116/121] Updated readme --- NEWS | 8 ++++---- README.md | 48 +++++++++++++++++++++++++++++++++--------------- 2 files changed, 37 insertions(+), 19 deletions(-) diff --git a/NEWS b/NEWS index e797ff7..08622f2 100644 --- a/NEWS +++ b/NEWS @@ -1,14 +1,14 @@ r129: -New : LZ4 CLI improved performance when compressing/decompressing multiple files (#86, kind contribution from Kyle J. Harper & Takayuki Matsuoka) Added : LZ4_compress_fast() +Changed: New lz4 and lz4hc compression API. Previous function prototypes still supported. Changed: Sparse file support enabled by default -Changed: New function names LZ4_compress_safe() - Suggested by Evan Nemerson & Takayuki Matsuoka +New : LZ4 CLI improved performance compressing/decompressing multiple files (#86, kind contribution from Kyle J. Harper & Takayuki Matsuoka) Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Greg Slazinski & Evan Nemerson Changed: Enums converted to LZ4F_ namespace convention - by Takayuki Matsuoka Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka -Modified:Obsolete functions generate warnings - Suggested by Evan Nemerson & Takayuki Matsuoka +Modified:Obsolete functions generate warnings - Suggested by Evan Nemerson, contributed by Takayuki Matsuoka Fixed : Bug #75 (unfinished stream), reported by Yongwoon Cho -Updated: Documentation converted to MarkDown +Updated: Documentation converted to MarkDown format r128: New : lz4cli sparse file support (Requested by Neil Wilson, and contributed by Takayuki Matsuoka) diff --git a/README.md b/README.md index 5d506d4..d050029 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,12 @@ It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems. -A high compression derivative, called LZ4_HC, is also provided. -It trades CPU time for compression ratio. +Speed can be tuned dynamically, selecting an "acceleration" factor +which trades compression ratio for more speed up. +On the other end, a high compression derivative, LZ4_HC, is also provided, +trading CPU time for improved compression ratio. +All versions feature the same excellent decompression speed. + |Branch |Status | |------------|---------| @@ -37,26 +41,40 @@ The reference system uses a Core i5-4300U @1.9GHz. Benchmark evaluates the compression of reference [Silesia Corpus] in single-thread mode. -| Compressor | Ratio | Compression | Decompression | -| ---------- | ----- | ----------- | ------------- | -| memcpy | 1.000 | 4200 MB/s | 4200 MB/s | -|**LZ4 fast (r129)**| 1.607 |**680 MB/s** | **2220 MB/s** | -|**LZ4 (r129)** |**2.101**|**385 MB/s** | **1850 MB/s** | -| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | -| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | -| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | -| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | -|**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | -| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | +| Compressor | Ratio | Compression | Decompression | +| ---------- | ----- | ----------- | ------------- | +| memcpy | 1.000 | 4200 MB/s | 4200 MB/s | +|**LZ4 fast 17 (r129)**| 1.607 |**690 MB/s** | **2220 MB/s** | +|**LZ4 default (r129)**|**2.101**|**385 MB/s** | **1850 MB/s** | +| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s | +| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s | +| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s | +| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s | +|**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** | +| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s | + + +Documentation +------------------------- The raw LZ4 block compression format is detailed within [lz4_Block_format]. -Compressing an arbitrarily long file or data stream requires multiple blocks. +To compress an arbitrarily long file or data stream, multiple blocks are required. Organizing these blocks and providing a common header format to handle their content is the purpose of the Frame format, defined into [lz4_Frame_format]. -Interoperable versions of LZ4 must respect this frame format too. +Interoperable versions of LZ4 must respect this frame format. + + +Other source versions +------------------------- + +Beyond the C reference source, +many contributors have created versions of lz4 in multiple languages. +A list of these sources is maintained on the [LZ4 Homepage]. + [Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029 [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [lz4_Block_format]: lz4_Block_format.md [lz4_Frame_format]: lz4_Frame_format.md +[LZ4 Homepage]: http://www.lz4.info From 1c3e633c48766c58df949887297dc5838170a33f Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 6 May 2015 01:58:24 +0100 Subject: [PATCH 117/121] Added compilation flag -Wcast-qual --- lib/lz4.c | 26 ++++---- lib/lz4.h | 4 +- lib/lz4frame.c | 20 +++--- lib/lz4hc.c | 11 ++-- lib/lz4hc.h | 4 +- lib/xxhash.c | 144 +++++++++++++++++++++---------------------- lib/xxhash.h | 6 +- programs/Makefile | 2 +- programs/frametest.c | 4 +- programs/lz4cli.c | 2 +- programs/lz4io.c | 8 +-- 11 files changed, 114 insertions(+), 117 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index bb40f76..c511456 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -351,7 +351,7 @@ typedef struct { U32 currentOffset; U32 initCheck; const BYTE* dictionary; - const BYTE* bufferStart; + BYTE* bufferStart; /* obsolete, used for slideInputBuffer */ U32 dictSize; } LZ4_stream_t_internal; @@ -1064,11 +1064,11 @@ FORCE_INLINE int LZ4_decompress_generic( if (endOnInput) return (int) (((char*)op)-dest); /* Nb of output bytes decoded */ else - return (int) (((char*)ip)-source); /* Nb of input bytes read */ + return (int) (((const char*)ip)-source); /* Nb of input bytes read */ /* Overflow error detected */ _output_error: - return (int) (-(((char*)ip)-source))-1; + return (int) (-(((const char*)ip)-source))-1; } @@ -1092,9 +1092,9 @@ int LZ4_decompress_fast(const char* source, char* dest, int originalSize) typedef struct { - BYTE* externalDict; + const BYTE* externalDict; size_t extDictSize; - BYTE* prefixEnd; + const BYTE* prefixEnd; size_t prefixSize; } LZ4_streamDecode_t_internal; @@ -1126,7 +1126,7 @@ int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dicti { LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode; lz4sd->prefixSize = (size_t) dictSize; - lz4sd->prefixEnd = (BYTE*) dictionary + dictSize; + lz4sd->prefixEnd = (const BYTE*) dictionary + dictSize; lz4sd->externalDict = NULL; lz4sd->extDictSize = 0; return 1; @@ -1215,7 +1215,7 @@ FORCE_INLINE int LZ4_decompress_usingDict_generic(const char* source, char* dest return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, withPrefix64k, (BYTE*)dest-64 KB, NULL, 0); return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, noDict, (BYTE*)dest-dictSize, NULL, 0); } - return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, usingExtDict, (BYTE*)dest, (BYTE*)dictStart, dictSize); + return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } int LZ4_decompress_safe_usingDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) @@ -1231,7 +1231,7 @@ int LZ4_decompress_fast_usingDict(const char* source, char* dest, int originalSi /* debug function */ int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) { - return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, (BYTE*)dest, (BYTE*)dictStart, dictSize); + return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } @@ -1260,23 +1260,23 @@ int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; } -static void LZ4_init(LZ4_stream_t_internal* lz4ds, const BYTE* base) +static void LZ4_init(LZ4_stream_t_internal* lz4ds, BYTE* base) { MEM_INIT(lz4ds, 0, LZ4_STREAMSIZE); lz4ds->bufferStart = base; } -int LZ4_resetStreamState(void* state, const char* inputBuffer) +int LZ4_resetStreamState(void* state, char* inputBuffer) { if ((((size_t)state) & 3) != 0) return 1; /* Error : pointer is not aligned on 4-bytes boundary */ - LZ4_init((LZ4_stream_t_internal*)state, (const BYTE*)inputBuffer); + LZ4_init((LZ4_stream_t_internal*)state, (BYTE*)inputBuffer); return 0; } -void* LZ4_create (const char* inputBuffer) +void* LZ4_create (char* inputBuffer) { void* lz4ds = ALLOCATOR(8, LZ4_STREAMSIZE_U64); - LZ4_init ((LZ4_stream_t_internal*)lz4ds, (const BYTE*)inputBuffer); + LZ4_init ((LZ4_stream_t_internal*)lz4ds, (BYTE*)inputBuffer); return lz4ds; } diff --git a/lib/lz4.h b/lib/lz4.h index b597064..00304d8 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -331,9 +331,9 @@ int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const cha /* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */ /* Obsolete streaming functions; use new streaming interface whenever possible */ -LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (char* inputBuffer); LZ4_DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); -LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, char* inputBuffer); LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); /* Obsolete streaming decoding functions */ diff --git a/lib/lz4frame.c b/lib/lz4frame.c index d73436c..d1733d0 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -140,7 +140,7 @@ typedef struct LZ4F_dctx_s size_t tmpInSize; size_t tmpInTarget; BYTE* tmpOutBuffer; - BYTE* dict; + const BYTE* dict; size_t dictSize; BYTE* tmpOut; size_t tmpOutSize; @@ -961,7 +961,7 @@ static int LZ4F_decompress_safe (const char* source, char* dest, int compressedS static void LZ4F_updateDict(LZ4F_dctx_t* dctxPtr, const BYTE* dstPtr, size_t dstSize, const BYTE* dstPtr0, unsigned withinTmp) { if (dctxPtr->dictSize==0) - dctxPtr->dict = (BYTE*)dstPtr; /* priority to dictionary continuity */ + dctxPtr->dict = (const BYTE*)dstPtr; /* priority to dictionary continuity */ if (dctxPtr->dict + dctxPtr->dictSize == dstPtr) /* dictionary continuity */ { @@ -971,7 +971,7 @@ static void LZ4F_updateDict(LZ4F_dctx_t* dctxPtr, const BYTE* dstPtr, size_t dst if (dstPtr - dstPtr0 + dstSize >= 64 KB) /* dstBuffer large enough to become dictionary */ { - dctxPtr->dict = (BYTE*)dstPtr0; + dctxPtr->dict = (const BYTE*)dstPtr0; dctxPtr->dictSize = dstPtr - dstPtr0 + dstSize; return; } @@ -987,7 +987,7 @@ static void LZ4F_updateDict(LZ4F_dctx_t* dctxPtr, const BYTE* dstPtr, size_t dst { size_t preserveSize = dctxPtr->tmpOut - dctxPtr->tmpOutBuffer; size_t copySize = 64 KB - dctxPtr->tmpOutSize; - BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize - dctxPtr->tmpOutStart; + const BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize - dctxPtr->tmpOutStart; if (dctxPtr->tmpOutSize > 64 KB) copySize = 0; if (copySize > preserveSize) copySize = preserveSize; @@ -1003,10 +1003,10 @@ static void LZ4F_updateDict(LZ4F_dctx_t* dctxPtr, const BYTE* dstPtr, size_t dst if (dctxPtr->dictSize + dstSize > dctxPtr->maxBufferSize) /* tmp buffer not large enough */ { size_t preserveSize = 64 KB - dstSize; /* note : dstSize < 64 KB */ - memcpy(dctxPtr->dict, dctxPtr->dict + dctxPtr->dictSize - preserveSize, preserveSize); + memcpy(dctxPtr->tmpOutBuffer, dctxPtr->dict + dctxPtr->dictSize - preserveSize, preserveSize); dctxPtr->dictSize = preserveSize; } - memcpy(dctxPtr->dict + dctxPtr->dictSize, dstPtr, dstSize); + memcpy(dctxPtr->tmpOutBuffer + dctxPtr->dictSize, dstPtr, dstSize); dctxPtr->dictSize += dstSize; return; } @@ -1277,10 +1277,10 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, { if (dctxPtr->dictSize > 128 KB) { - memcpy(dctxPtr->dict, dctxPtr->dict + dctxPtr->dictSize - 64 KB, 64 KB); + memcpy(dctxPtr->tmpOutBuffer, dctxPtr->dict + dctxPtr->dictSize - 64 KB, 64 KB); dctxPtr->dictSize = 64 KB; } - dctxPtr->tmpOut = dctxPtr->dict + dctxPtr->dictSize; + dctxPtr->tmpOut = dctxPtr->tmpOutBuffer + dctxPtr->dictSize; } else /* dict not within tmp */ { @@ -1444,7 +1444,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, { size_t preserveSize = dctxPtr->tmpOut - dctxPtr->tmpOutBuffer; size_t copySize = 64 KB - dctxPtr->tmpOutSize; - BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize - dctxPtr->tmpOutStart; + const BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize - dctxPtr->tmpOutStart; if (dctxPtr->tmpOutSize > 64 KB) copySize = 0; if (copySize > preserveSize) copySize = preserveSize; @@ -1456,7 +1456,7 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t decompressionContext, else { size_t newDictSize = dctxPtr->dictSize; - BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize; + const BYTE* oldDictEnd = dctxPtr->dict + dctxPtr->dictSize; if ((newDictSize) > 64 KB) newDictSize = 64 KB; memcpy(dctxPtr->tmpOutBuffer, oldDictEnd - newDictSize, newDictSize); diff --git a/lib/lz4hc.c b/lib/lz4hc.c index fa05ee3..a22ab2b 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -91,7 +91,7 @@ typedef struct const BYTE* end; /* next block here to continue on current prefix */ const BYTE* base; /* All index relative to this position */ const BYTE* dictBase; /* alternate base for extDict */ - const BYTE* inputBuffer;/* deprecated */ + BYTE* inputBuffer; /* deprecated */ U32 dictLimit; /* below that point, need extDict */ U32 lowLimit; /* below that point, no more dict */ U32 nextToUpdate; /* index from which to continue dictionary update */ @@ -119,7 +119,6 @@ static void LZ4HC_init (LZ4HC_Data_Structure* hc4, const BYTE* start) MEM_INIT(hc4->chainTable, 0xFF, sizeof(hc4->chainTable)); hc4->nextToUpdate = 64 KB; hc4->base = start - 64 KB; - hc4->inputBuffer = start; hc4->end = start; hc4->dictBase = start - 64 KB; hc4->dictLimit = 64 KB; @@ -628,7 +627,7 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr, const BYTE* sourceEnd = (const BYTE*) source + inputSize; const BYTE* dictBegin = ctxPtr->dictBase + ctxPtr->lowLimit; const BYTE* dictEnd = ctxPtr->dictBase + ctxPtr->dictLimit; - if ((sourceEnd > dictBegin) && ((BYTE*)source < dictEnd)) + if ((sourceEnd > dictBegin) && ((const BYTE*)source < dictEnd)) { if (sourceEnd > dictEnd) sourceEnd = dictEnd; ctxPtr->lowLimit = (U32)(sourceEnd - ctxPtr->dictBase); @@ -691,17 +690,19 @@ int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, /* These functions currently generate deprecation warnings */ int LZ4_sizeofStreamStateHC(void) { return LZ4_STREAMHCSIZE; } -int LZ4_resetStreamStateHC(void* state, const char* inputBuffer) +int LZ4_resetStreamStateHC(void* state, char* inputBuffer) { if ((((size_t)state) & (sizeof(void*)-1)) != 0) return 1; /* Error : pointer is not aligned for pointer (32 or 64 bits) */ LZ4HC_init((LZ4HC_Data_Structure*)state, (const BYTE*)inputBuffer); + ((LZ4HC_Data_Structure*)state)->inputBuffer = (BYTE*)inputBuffer; return 0; } -void* LZ4_createHC (const char* inputBuffer) +void* LZ4_createHC (char* inputBuffer) { void* hc4 = ALLOCATOR(1, sizeof(LZ4HC_Data_Structure)); LZ4HC_init ((LZ4HC_Data_Structure*)hc4, (const BYTE*)inputBuffer); + ((LZ4HC_Data_Structure*)hc4)->inputBuffer = (BYTE*)inputBuffer; return hc4; } diff --git a/lib/lz4hc.h b/lib/lz4hc.h index f8461b4..431f7c8 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -175,13 +175,13 @@ int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, cons int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /* Streaming functions following the older model; should no longer be used */ -LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (char* inputBuffer); LZ4_DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); LZ4_DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); LZ4_DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); -LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, const char* inputBuffer); +LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, char* inputBuffer); #if defined (__cplusplus) diff --git a/lib/xxhash.c b/lib/xxhash.c index a4a3fbe..e6fb8f1 100644 --- a/lib/xxhash.c +++ b/lib/xxhash.c @@ -29,13 +29,12 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - xxHash source repository : https://github.com/Cyan4973/xxHash -- public discussion board : https://groups.google.com/forum/#!forum/lz4c */ /************************************** * Tuning parameters -***************************************/ +**************************************/ /* Unaligned memory access is automatically enabled for "common" CPU, such as x86. * For others CPU, the compiler will be more cautious, and insert extra code to ensure aligned access is respected. * If you know your target CPU supports unaligned memory access, you want to force this option manually to improve performance. @@ -93,10 +92,7 @@ static void* XXH_malloc(size_t s) { return malloc(s); } static void XXH_free (void* p) { free(p); } /* for memcpy() */ #include -static void* XXH_memcpy(void* dest, const void* src, size_t size) -{ - return memcpy(dest,src,size); -} +static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } /************************************** @@ -104,17 +100,17 @@ static void* XXH_memcpy(void* dest, const void* src, size_t size) ***************************************/ #if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # include -typedef uint8_t BYTE; -typedef uint16_t U16; -typedef uint32_t U32; -typedef int32_t S32; -typedef uint64_t U64; + typedef uint8_t BYTE; + typedef uint16_t U16; + typedef uint32_t U32; + typedef int32_t S32; + typedef uint64_t U64; #else -typedef unsigned char BYTE; -typedef unsigned short U16; -typedef unsigned int U32; -typedef signed int S32; -typedef unsigned long long U64; + typedef unsigned char BYTE; + typedef unsigned short U16; + typedef unsigned int U32; + typedef signed int S32; + typedef unsigned long long U64; #endif static U32 XXH_read32(const void* memPtr) @@ -133,7 +129,7 @@ static U64 XXH_read64(const void* memPtr) -/***************************************** +/****************************************** * Compiler-specific Functions and Macros ******************************************/ #define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) @@ -175,7 +171,55 @@ static U64 XXH_swap64 (U64 x) #endif -/************************************** +/*************************************** +* Architecture Macros +***************************************/ +typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; +#ifndef XXH_CPU_LITTLE_ENDIAN /* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example using a compiler switch */ +static const int one = 1; +# define XXH_CPU_LITTLE_ENDIAN (*(const char*)(&one)) +#endif + + +/***************************** +* Memory reads +*****************************/ +typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; + +FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align) +{ + if (align==XXH_unaligned) + return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); + else + return endian==XXH_littleEndian ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr); +} + +FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian) +{ + return XXH_readLE32_align(ptr, endian, XXH_unaligned); +} + +FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align) +{ + if (align==XXH_unaligned) + return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); + else + return endian==XXH_littleEndian ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr); +} + +FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian) +{ + return XXH_readLE64_align(ptr, endian, XXH_unaligned); +} + + +/*************************************** +* Macros +***************************************/ +#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */ + + +/*************************************** * Constants ***************************************/ #define PRIME32_1 2654435761U @@ -191,55 +235,7 @@ static U64 XXH_swap64 (U64 x) #define PRIME64_5 2870177450012600261ULL -/*************************************** -* Architecture Macros -****************************************/ -typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; -#ifndef XXH_CPU_LITTLE_ENDIAN /* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example using a compiler switch */ -static const int one = 1; -# define XXH_CPU_LITTLE_ENDIAN (*(char*)(&one)) -#endif - - -/************************************** -* Macros -***************************************/ -#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */ - - -/**************************** -* Memory reads -*****************************/ -typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; - -FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align) -{ - if (align==XXH_unaligned) - return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); - else - return endian==XXH_littleEndian ? *(U32*)ptr : XXH_swap32(*(U32*)ptr); -} - -FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian) -{ - return XXH_readLE32_align(ptr, endian, XXH_unaligned); -} - -FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align) -{ - if (align==XXH_unaligned) - return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); - else - return endian==XXH_littleEndian ? *(U64*)ptr : XXH_swap64(*(U64*)ptr); -} - -FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian) -{ - return XXH_readLE64_align(ptr, endian, XXH_unaligned); -} - - -/**************************** +/***************************** * Simple Hash Functions *****************************/ FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align) @@ -319,7 +315,7 @@ FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH } -unsigned int XXH32 (const void* input, size_t len, unsigned seed) +unsigned XXH32 (const void* input, size_t len, unsigned seed) { #if 0 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ @@ -331,7 +327,7 @@ unsigned int XXH32 (const void* input, size_t len, unsigned seed) XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; # if !defined(XXH_USE_UNALIGNED_ACCESS) - if ((((size_t)input) & 3) == 0) /* Input is aligned, let's leverage the speed advantage */ + if ((((size_t)input) & 3) == 0) /* Input is 4-bytes aligned, leverage the speed benefit */ { if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned); @@ -488,7 +484,7 @@ unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed } /**************************************************** - * Advanced Hash Functions +* Advanced Hash Functions ****************************************************/ /*** Allocation ***/ @@ -672,9 +668,9 @@ XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void* input, size_t l FORCE_INLINE U32 XXH32_digest_endian (const XXH32_state_t* state_in, XXH_endianess endian) { - XXH_istate32_t* state = (XXH_istate32_t*) state_in; + const XXH_istate32_t* state = (const XXH_istate32_t*) state_in; const BYTE * p = (const BYTE*)state->mem32; - BYTE* bEnd = (BYTE*)(state->mem32) + state->memsize; + const BYTE* bEnd = (const BYTE*)(state->mem32) + state->memsize; U32 h32; if (state->total_len >= 16) @@ -826,9 +822,9 @@ XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void* input, size_t l FORCE_INLINE U64 XXH64_digest_endian (const XXH64_state_t* state_in, XXH_endianess endian) { - XXH_istate64_t * state = (XXH_istate64_t *) state_in; + const XXH_istate64_t * state = (const XXH_istate64_t *) state_in; const BYTE * p = (const BYTE*)state->mem64; - BYTE* bEnd = (BYTE*)state->mem64 + state->memsize; + const BYTE* bEnd = (const BYTE*)state->mem64 + state->memsize; U64 h64; if (state->total_len >= 32) diff --git a/lib/xxhash.h b/lib/xxhash.h index 34eea73..0df5ac1 100644 --- a/lib/xxhash.h +++ b/lib/xxhash.h @@ -29,7 +29,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - - xxHash source repository : http://code.google.com/p/xxhash/ + - xxHash source repository : https://github.com/Cyan4973/xxHash */ /* Notice extracted from xxHash homepage : @@ -57,8 +57,8 @@ Q.Score is a measure of quality of the hash function. It depends on successfully passing SMHasher test set. 10 is a perfect score. -A new 64-bits version, named XXH64, is available since r35. -It offers better speed for 64-bits applications. +A 64-bits version, named XXH64, is available since r35. +It offers much better speed, but for 64-bits applications only. Name Speed on 64 bits Speed on 32 bits XXH64 13.8 GB/s 1.9 GB/s XXH32 6.8 GB/s 6.0 GB/s diff --git a/programs/Makefile b/programs/Makefile index 910516f..39335db 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -39,7 +39,7 @@ RELEASE?= r129 DESTDIR?= PREFIX ?= /usr/local CFLAGS ?= -O3 -CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\" +CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-qual -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\" FLAGS := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) BINDIR := $(PREFIX)/bin diff --git a/programs/frametest.c b/programs/frametest.c index fedb78d..46ec030 100644 --- a/programs/frametest.c +++ b/programs/frametest.c @@ -568,8 +568,8 @@ _output_error: static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, unsigned nonContiguous) { int p=0; - BYTE* b1=(BYTE*)buff1; - BYTE* b2=(BYTE*)buff2; + const BYTE* b1=(const BYTE*)buff1; + const BYTE* b2=(const BYTE*)buff2; if (nonContiguous) { DISPLAY("Non-contiguous output test (%i bytes)\n", (int)size); diff --git a/programs/lz4cli.c b/programs/lz4cli.c index da18169..970856d 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -470,7 +470,7 @@ int main(int argc, char** argv) if (!decode) DISPLAYLEVEL(4, "Blocks size : %i KB\n", blockSize>>10); /* No input filename ==> use stdin */ - if (multiple_inputs) input_filename = inFileNames[0], output_filename = (char*)(inFileNames[0]); + if (multiple_inputs) input_filename = inFileNames[0], output_filename = (const char*)(inFileNames[0]); if(!input_filename) { input_filename=stdinmark; } /* Check if input or output are defined as console; trigger an error in this case */ diff --git a/programs/lz4io.c b/programs/lz4io.c index 15c93f4..209f5ed 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -630,10 +630,10 @@ static unsigned LZ4IO_readLE32 (const void* s) static unsigned LZ4IO_fwriteSparse(FILE* file, const void* buffer, size_t bufferSize, unsigned storedSkips) { - size_t* const bufferT = (size_t*)buffer; /* Buffer is supposed malloc'ed, hence aligned on size_t */ - size_t* ptrT = bufferT; + const size_t* const bufferT = (const size_t*)buffer; /* Buffer is supposed malloc'ed, hence aligned on size_t */ + const size_t* ptrT = bufferT; size_t bufferSizeT = bufferSize / sizeT; - size_t* const bufferTEnd = bufferT + bufferSizeT; + const size_t* const bufferTEnd = bufferT + bufferSizeT; static const size_t segmentSizeT = (32 KB) / sizeT; if (!g_sparseFileSupport) /* normal write */ @@ -679,7 +679,7 @@ static unsigned LZ4IO_fwriteSparse(FILE* file, const void* buffer, size_t buffer if (bufferSize & maskT) /* size not multiple of sizeT : implies end of block */ { - const char* const restStart = (char*)bufferTEnd; + const char* const restStart = (const char*)bufferTEnd; const char* restPtr = restStart; size_t restSize = bufferSize & maskT; const char* const restEnd = restStart + restSize; From efbebd2a9955c6a6eabc558f00643abb4690edac Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 6 May 2015 02:29:04 +0100 Subject: [PATCH 118/121] Added : LZ4_compress_destSize() --- lib/Makefile | 4 +- lib/lz4.c | 221 +++++++++++++++++++++++++++++++++++++++++++++- lib/lz4.h | 15 ++++ programs/fuzzer.c | 35 ++++++++ 4 files changed, 269 insertions(+), 6 deletions(-) diff --git a/lib/Makefile b/lib/Makefile index 8aa15de..ef671d3 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -31,7 +31,7 @@ # ################################################################ # Version numbers -VERSION ?= 128 +VERSION ?= 129 LIBVER_MAJOR=`sed -n '/define LZ4_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h` LIBVER_MINOR=`sed -n '/define LZ4_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h` LIBVER_PATCH=`sed -n '/define LZ4_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h` @@ -40,7 +40,7 @@ LIBVER=$(LIBVER_MAJOR).$(LIBVER_MINOR).$(LIBVER_PATCH) DESTDIR?= PREFIX ?= /usr/local CFLAGS ?= -O3 -CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic +CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wcast-qual Wstrict-prototypes -pedantic LIBDIR?= $(PREFIX)/lib INCLUDEDIR=$(PREFIX)/include diff --git a/lib/lz4.c b/lib/lz4.c index c511456..55f2359 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -679,7 +679,7 @@ int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { #if (HEAPMODE) - void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t_internal)); /* malloc-calloc always properly aligned */ + void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ #else LZ4_stream_t ctx; void* ctxPtr = &ctx; @@ -715,9 +715,222 @@ int LZ4_compress_fast_force(const char* source, char* dest, int inputSize, int m } -/***************************************** -* Experimental : Streaming functions -*****************************************/ +/******************************** +* destSize variant +********************************/ + +static int LZ4_compress_destSize_generic( + void* const ctx, + const char* const src, + char* const dst, + int* const srcSizePtr, + const int targetDstSize, + const tableType_t tableType) +{ + const BYTE* ip = (const BYTE*) src; + const BYTE* base = (const BYTE*) src; + const BYTE* lowLimit = (const BYTE*) src; + const BYTE* anchor = ip; + const BYTE* const iend = ip + *srcSizePtr; + const BYTE* const mflimit = iend - MFLIMIT; + const BYTE* const matchlimit = iend - LASTLITERALS; + + BYTE* op = (BYTE*) dst; + BYTE* const oend = op + targetDstSize; + BYTE* const oMaxLit = op + targetDstSize - 2 /* offset */ - 8 /* because 8+MINMATCH==MFLIMIT */ - 1 /* token */; + BYTE* const oMaxMatch = op + targetDstSize - (LASTLITERALS + 1 /* token */); + BYTE* const oMaxSeq = oMaxLit - 1 /* token */; + + U32 forwardH; + + + /* Init conditions */ + if (targetDstSize < 1) return 0; /* Impossible to store anything */ + if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ + if ((tableType == byU16) && (*srcSizePtr>=LZ4_64Klimit)) return 0; /* Size too large (not within 64K limit) */ + if (*srcSizePtr> LZ4_skipTrigger); + + if (unlikely(forwardIp > mflimit)) + goto _last_literals; + + match = LZ4_getPositionOnHash(h, ctx, tableType, base); + forwardH = LZ4_hashPosition(forwardIp, tableType); + LZ4_putPositionOnHash(ip, h, ctx, tableType, base); + + } while ( ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip)) + || (LZ4_read32(match) != LZ4_read32(ip)) ); + } + + /* Catch up */ + while ((ip>anchor) && (match > lowLimit) && (unlikely(ip[-1]==match[-1]))) { ip--; match--; } + + { + /* Encode Literal length */ + unsigned litLength = (unsigned)(ip - anchor); + token = op++; + if (op + ((litLength+240)/255) + litLength > oMaxLit) + { + /* Not enough space for a last match */ + op--; + goto _last_literals; + } + if (litLength>=RUN_MASK) + { + unsigned len = litLength - RUN_MASK; + *token=(RUN_MASK<= 255 ; len-=255) *op++ = 255; + *op++ = (BYTE)len; + } + else *token = (BYTE)(litLength< oMaxMatch) + { + /* Match description too long : reduce it */ + matchLength = (15-1) + (oMaxMatch-op) * 255; + } + //printf("offset %5i, matchLength%5i \n", (int)(ip-match), matchLength + MINMATCH); + ip += MINMATCH + matchLength; + + if (matchLength>=ML_MASK) + { + *token += ML_MASK; + matchLength -= ML_MASK; + while (matchLength >= 255) { matchLength-=255; *op++ = 255; } + *op++ = (BYTE)matchLength; + } + else *token += (BYTE)(matchLength); + } + + anchor = ip; + + /* Test end of block */ + if (ip > mflimit) break; + if (op > oMaxSeq) break; + + /* Fill table */ + LZ4_putPosition(ip-2, ctx, tableType, base); + + /* Test next position */ + match = LZ4_getPosition(ip, ctx, tableType, base); + LZ4_putPosition(ip, ctx, tableType, base); + if ( (match+MAX_DISTANCE>=ip) + && (LZ4_read32(match)==LZ4_read32(ip)) ) + { token=op++; *token=0; goto _next_match; } + + /* Prepare next loop */ + forwardH = LZ4_hashPosition(++ip, tableType); + } + +_last_literals: + /* Encode Last Literals */ + { + size_t lastRunSize = (size_t)(iend - anchor); + if (op + 1 /* token */ + ((lastRunSize+240)/255) /* litLength */ + lastRunSize /* literals */ > oend) + { + /* adapt lastRunSize to fill 'dst' */ + lastRunSize = (oend-op) - 1; + lastRunSize -= (lastRunSize+240)/255; + } + ip = anchor + lastRunSize; + + if (lastRunSize >= RUN_MASK) + { + size_t accumulator = lastRunSize - RUN_MASK; + *op++ = RUN_MASK << ML_BITS; + for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; + *op++ = (BYTE) accumulator; + } + else + { + *op++ = (BYTE)(lastRunSize<= LZ4_compressBound(*srcSizePtr)) /* compression success is guaranteed */ + { + return LZ4_compress_fast_extState(state, src, dst, *srcSizePtr, targetDstSize, 1); + } + else + { + if (*srcSizePtr < LZ4_64Klimit) + return LZ4_compress_destSize_generic(state, src, dst, srcSizePtr, targetDstSize, byU16); + else + return LZ4_compress_destSize_generic(state, src, dst, srcSizePtr, targetDstSize, LZ4_64bits() ? byU32 : byPtr); + } +} + + +int LZ4_compress_destSize(const char* src, char* dst, int* srcSizePtr, int targetDstSize) +{ +#if (HEAPMODE) + void* ctx = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ +#else + LZ4_stream_t ctxBody; + void* ctx = &ctxBody; +#endif + + int result = LZ4_compress_destSize_extState(ctx, src, dst, srcSizePtr, targetDstSize); + +#if (HEAPMODE) + FREEMEM(ctx); +#endif + return result; +} + + + +/******************************** +* Streaming functions +********************************/ LZ4_stream_t* LZ4_createStream(void) { diff --git a/lib/lz4.h b/lib/lz4.h index 00304d8..93388c2 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -139,6 +139,21 @@ int LZ4_sizeofState(void); int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, int acceleration); +/* +LZ4_compress_destSize() : + Reverse the logic, by compressing as much data as possible from 'source' buffer + into already allocated buffer 'dest' of size 'targetDestSize'. + This function either compresses the entire 'source' content into 'dest' if it's large enough, + or fill 'dest' buffer completely with as much data as possible from 'source'. + Original idea by WiredTiger team. + *sourceSizePtr : will be modified to indicate how many bytes where read from 'source' to fill 'dest'. + New value is necessarily <= old value. + return : Nb bytes written into 'dest' (necessarily <= targetDestSize) + or 0 if compression fails +*/ +int LZ4_compress_destSize (const char* source, char* dest, int* sourceSizePtr, int targetDestSize); + + /* LZ4_decompress_fast() : originalSize : is the original and therefore uncompressed size diff --git a/programs/fuzzer.c b/programs/fuzzer.c index 9fd4648..6588290 100644 --- a/programs/fuzzer.c +++ b/programs/fuzzer.c @@ -396,6 +396,41 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c /* Compression tests */ + /* Test compression destSize */ + FUZ_DISPLAYTEST; + { + int srcSize = blockSize; + int targetSize = srcSize * ((FUZ_rand(&randState) & 127)+1) >> 7; + char endCheck = FUZ_rand(&randState) & 255; + compressedBuffer[targetSize] = endCheck; + ret = LZ4_compress_destSize(block, compressedBuffer, &srcSize, targetSize); + FUZ_CHECKTEST(ret > targetSize, "LZ4_compress_destSize() result larger than dst buffer !"); + FUZ_CHECKTEST(compressedBuffer[targetSize] != endCheck, "LZ4_compress_destSize() overwrite dst buffer !"); + FUZ_CHECKTEST(srcSize > blockSize, "LZ4_compress_destSize() fed more than src buffer !"); + DISPLAY("destSize : %7i/%7i; content%7i/%7i ", ret, targetSize, srcSize, blockSize); + if (targetSize>0) + { + FUZ_CHECKTEST((ret==0), "LZ4_compress_destSize() compression failed"); + /* check correctness */ + FUZ_DISPLAYTEST; + + crcOrig = XXH32(block, srcSize, 0); + compressedSize = ret; + endCheck = FUZ_rand(&randState) & 255; + decodedBuffer[srcSize] = endCheck; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, srcSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe() failed on data compressed by LZ4_compress_destSize"); + FUZ_CHECKTEST(ret!=srcSize, "LZ4_decompress_safe() failed : did not fully decompressed data"); + FUZ_CHECKTEST(decodedBuffer[srcSize] != endCheck, "LZ4_decompress_safe() overwrite dst buffer !"); + crcCheck = XXH32(decodedBuffer, srcSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe() corrupted decoded data"); + + DISPLAY(" OK \n"); + } + else + DISPLAY(" \n"); + } + /* Test compression HC */ FUZ_DISPLAYTEST; ret = LZ4_compressHC(block, compressedBuffer, blockSize); From b4ef93a83707614b66e3f44706d9992048119768 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 6 May 2015 10:26:59 +0100 Subject: [PATCH 119/121] Fixed typo --- lib/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Makefile b/lib/Makefile index ef671d3..4be1499 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -40,7 +40,7 @@ LIBVER=$(LIBVER_MAJOR).$(LIBVER_MINOR).$(LIBVER_PATCH) DESTDIR?= PREFIX ?= /usr/local CFLAGS ?= -O3 -CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wcast-qual Wstrict-prototypes -pedantic +CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wcast-qual -Wstrict-prototypes -pedantic LIBDIR?= $(PREFIX)/lib INCLUDEDIR=$(PREFIX)/include From 672bfdebd0975289dbbdecabeeda9fabb499475a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 6 May 2015 20:23:06 +0100 Subject: [PATCH 120/121] Updated comments --- lib/lz4.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/lz4.h b/lib/lz4.h index 93388c2..20e3d48 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -89,10 +89,10 @@ LZ4_compress_default() : or 0 if compression fails LZ4_decompress_safe() : - compressedSize : is obviously the source size - maxDecompressedSize : is the size of the destination buffer, which must be already allocated. - return : the number of bytes decompressed into the destination buffer (necessarily <= maxDecompressedSize) - If the destination buffer is not large enough, decoding will stop and output an error code (<0). + compressedSize : is the precise full size of the compressed block. + maxDecompressedSize : is the size of destination buffer, which must be already allocated. + return : the number of bytes decompressed into destination buffer (necessarily <= maxDecompressedSize) + If destination buffer is not large enough, decoding will stop and output an error code (<0). If the source stream is detected malformed, the function will stop decoding and return a negative result. This function is protected against buffer overflow exploits, including malicious data packets. It never writes outside output buffer, nor reads outside input buffer. From fdd0029c3778768d0e8a80ce238d981f946c8b78 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 6 May 2015 20:57:35 +0100 Subject: [PATCH 121/121] minor parsing update --- lz4_Block_format.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/lz4_Block_format.md b/lz4_Block_format.md index b933a6a..a120a0b 100644 --- a/lz4_Block_format.md +++ b/lz4_Block_format.md @@ -45,17 +45,20 @@ There can be any number of bytes following the token. There is no "size limit". (Side note : this is why a not-compressible input block is expanded by 0.4%). Example 1 : A length of 48 will be represented as : -- 15 : value for the 4-bits High field -- 33 : (=48-15) remaining length to reach 48 + + - 15 : value for the 4-bits High field + - 33 : (=48-15) remaining length to reach 48 Example 2 : A length of 280 will be represented as : -- 15 : value for the 4-bits High field -- 255 : following byte is maxed, since 280-15 >= 255 -- 10 : (=280 - 15 - 255) ) remaining length to reach 280 + + - 15 : value for the 4-bits High field + - 255 : following byte is maxed, since 280-15 >= 255 + - 10 : (=280 - 15 - 255) ) remaining length to reach 280 Example 3 : A length of 15 will be represented as : -- 15 : value for the 4-bits High field -- 0 : (=15-15) yes, the zero must be output + + - 15 : value for the 4-bits High field + - 0 : (=15-15) yes, the zero must be output Following the token and optional length bytes, are the literals themselves. They are exactly as numerous as previously decoded (length of literals). @@ -84,7 +87,7 @@ we output additional bytes, one at a time, with values ranging from 0 to 255. They are added to total to provide the final match length. A 255 value means there is another byte to read and add. There is no limit to the number of optional bytes that can be output this way. -(This points towards a maximum achievable compression ratio of ~250). +(This points towards a maximum achievable compression ratio of about 250). With the offset and the matchlength, the decoder can now proceed to copy the data from the already decoded buffer. @@ -98,8 +101,7 @@ There are specific parsing rules to respect in order to remain compatible with assumptions made by the decoder : 1. The last 5 bytes are always literals -2. The last match must start at least 12 bytes before end of block. - +2. The last match must start at least 12 bytes before end of block. Consequently, a block with less than 13 bytes cannot be compressed. These rules are in place to ensure that the decoder