AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
2,469 Commits 13 Branches 31 Tags 6.8 MiB
2964b8a6f6
Commit Graph

337 Commits

Author SHA1 Message Date
W. Felix Handte
efc419a6d4 Replace calloc() Calls With malloc() Where Possible 2018-03-12 14:58:43 -04:00
W. Felix Handte
80790c587b Copy the Dict Table Into the Context for Large Compressions 2018-03-12 14:58:43 -04:00
W. Felix Handte
14ce912b70 Switch Current Offset to 1 Only When in External Dictionary Context Mode 2018-03-12 14:58:43 -04:00
W. Felix Handte
cea09d67a9 Hoist Table Reset One Level Up 2018-03-12 14:58:43 -04:00
W. Felix Handte
68c6bd17b8 Set Dictionary Context Pointer Rather than Copying the Context In 2018-03-12 14:58:43 -04:00
W. Felix Handte
73cc39327e Lookup Matches in Separate Dictionary Context 2018-03-12 14:58:43 -04:00
W. Felix Handte
62cb52b341 Initialize Current Offset to 1 2018-03-12 14:58:43 -04:00
W. Felix Handte
b3628cb0c5 Avoid Resetting the Context When Possible 2018-03-12 14:58:43 -04:00
W. Felix Handte
aa36e118f1 Const-ify Table Arg to LZ4_getPosition(OnHash) 2018-03-12 14:58:43 -04:00
W. Felix Handte
d6a3024dbb Add LZ4_compress_fast_safeExtState Function 2018-03-12 14:58:43 -04:00
W. Felix Handte
5709891de6 Add a Table Type Field to LZ4_stream_t 2018-03-12 14:58:43 -04:00
W. Felix Handte
6933f5ad9c Remove Obsolete Stream Functions to Free Space in LZ4_stream_t 2018-03-12 14:58:43 -04:00
W. Felix Handte
6d156fea56 Allow Empty Dictionaries 2018-03-12 14:58:43 -04:00
Yann Collet
7173a631db edge case : compress up to end-mflimit (12 bytes) 
The LZ4 block format specification
states that the last match must start
at a minimum distance of 12 bytes from the end of the block.

However, out of an abundance of caution,
the reference implementation would actually stop searching matches
at 13 bytes from the end of the block.

This patch fixes this small detail.
The new version is now able to properly compress a limit case
such as `aaaaaaaabaaa\n`
as reported by Gao Xiang (@hsiangkao).

Obviously, it doesn't change a lot of things.
This is just one additional match candidate per block, with a maximum match length of 7 (since last 5 bytes must remain literals).

With default policy, blocks are 4 MB long, so it doesn't happen too often
Compressing silesia.tar at default level 1 saves 5 bytes (100930101 -> 100930096).
At max level 12, it saves a grand 16 bytes (77389871 -> 77389855).

The impact is a bit more visible when blocks are smaller, hence more numerous.
For example, compressing silesia with blocks of 64 KB (using -12 -B4D) saves 543 bytes (77304583 -> 77304040).
So the smaller the packet size, the more visible the impact.

And it happens we have a ton of scenarios with little blocks using LZ4 compression ...

And a useless "hooray" sidenote :
the patch improves the LZ4 compression record of silesia (using -12 -B7D --no-frame-crc) by 16 bytes (77270672 -> 77270656)
and the record on enwik9 by 44 bytes (371680396 -> 371680352) (previously claimed by [smallz4](http://create.stephan-brumme.com/smallz4/) ).
 2018-02-24 11:47:53 -08:00
Yann Collet
219abab74b removed LZ4_copy8 
better use memcpy() directly
 2018-02-11 22:20:09 -08:00
Yann Collet
3ad3b0f850 slightly improved decompression speed (~+1-2%) 
by making shortcut slightly more common
 2018-02-11 01:43:20 -08:00
Ben Boeckel
c4671be550 intel: do not use __attribute__((packed)) on Windows 
On Windows, the Intel compiler is closer to MSVC rather than GCC and
does not support the GCC attribute syntax.

Fixes #468
 2018-02-08 09:15:27 -05:00
Yann Collet
6bbe45e1b8 remove register keyword 
deprecated in newer C++ versions,
and dubious utility
 2017-12-04 17:10:23 -08:00
Yann Collet
dac26084a9
Merge pull request #416 from lz4/newopt 
Improve Optimal parser
 2017-11-09 14:13:30 -08:00
Yann Collet
9221419c6f added LZ4_FORCEINLINE to counter gcc regression 
as recommended by @terrelln
 2017-11-06 17:29:27 -08:00
Yann Collet
d51f046628 2-stages LZ4_count 
separate first branch from the rest of the compare loop
to get dedicated prediction.

measured a 3-4% compression speed improvement.
 2017-11-06 15:42:50 -08:00
Sylvestre Ledru
4fed595dac Only ignore with C++17 2017-11-06 16:16:02 +01:00
Sylvestre Ledru
cca7618f09 When building with a C++ compiler, remove the 'register' keyword to silent a warning 
For example, with clang:

lz4.c:XXX:36: error: 'register' storage class specifier is deprecated and incompatible with C++17 [-Werror,-Wdeprecated-register]
static unsigned LZ4_NbCommonBytes (register reg_t val)
                                   ^~~~~~~~~
 2017-11-05 11:48:03 +01:00
Yann Collet
8e16eb0cd1 fixed last lost bytes in maximal mode 
even gained 2 bytes on calgary.tar...
added conditional traces `g_debuglog_enable`
 2017-11-02 14:53:06 -07:00
Yann Collet
15b0d229c1 Merge branch 'dev' into btopt 2017-10-31 17:55:01 -07:00
Yann Collet
a5731d6b26 minor change, to help store forwarding 
in a marginal case (offset==4)
 2017-10-31 15:51:56 -07:00
Yann Collet
9378f76e41 extended shortcut match length to 18 2017-10-31 14:20:25 -07:00
Yann Collet
ace334a4c9 minor : coding style : use ML_MASK constant 2017-10-31 12:22:15 -07:00
Yann Collet
3f173052ae added comments, as suggested by @terrelln 2017-10-31 11:49:57 -07:00
Yann Collet
e0914ff70c more complete shortcut - passes tests 2017-10-30 16:07:15 -07:00
Yann Collet
a31b7058cb small modification of lz4 decoder to shortcut common case (short branch). 2017-10-25 10:10:53 +02:00
Yann Collet
6cec68de39 added assert 2017-10-19 16:47:25 -07:00
Rei Odaira
73bcf90e51 Use the optimization level of O2 for the decompression functions on ppc64le with gcc, to avoid harmful unrolling and SIMDization with O3 2017-10-13 14:53:37 -05:00
Yann Collet
f6b31bf0d0 fix #404 
static analyzer `cppcheck` complains about a shift-by-32 on a 32 bits value,
which is an undefined behavior.
However, the flagged code path is never triggered in 32-bits mode,
(actually, it's not even generated if DCE kicks in),
the shift-by-32 is necessarily performed on a 64-bits value.

While it doesn't change anything regarding lz4 code generation, for both 32 and 64 bits mode,
(can be checked by md5sum on the generated binary),
the shift has been rewritten in a way which should please this static analyzer,
since it now pretends to shift by 16 on 32-bits cpu (note : it doesn't matter since the code will not even be generated in this case).

Note : this is a blind fix, the new code has not been tested with cppcheck, because cppcheck only works on Windows.
Other static analyzer, such as scan-build, do not trigger this false positive.
 2017-09-30 10:35:55 -07:00
Yann Collet
a30cba08f4 fixed a bunch of -Wcomma warnings 
reported by @rvandermeulen (#398)
 2017-09-10 14:32:38 -07:00
Yann Collet
b500c5282e fix #397 : decompression failed when using a combination of extDict + low memory address 
Reported and fixed by @jscheid

Note : we are missing a test case to include it in the CI
 2017-09-07 12:48:24 -07:00
Yann Collet
00418f7611 bench : made decompression speed evaluation same time as compression 
minor : slightly modified an example do avoid disabling a gcc warning through #pragma
 2017-09-07 12:12:36 -07:00
tcpan
f423d016cf FIX: added prefix to FORCE_INLINE to prevent redefinition error during compilation when used with other libraries that define FORCE_INLINE 2017-08-24 10:14:20 -04:00
Yann Collet
8593ba8831 Merge pull request #380 from lz4/dictID 
Frame Dictionary API
 2017-08-15 14:12:04 -07:00
Yann Collet
31f2cdf4d2 implemented dictionary compression in lz4frame 
note : only compression API is implemented and tested
still to do : decompression API
 2017-08-09 16:51:19 -07:00
Alex Deymo
76ef6d0ab0 Allow to predefine FORCE_INLINE macro. 
FORCE_INLINE macro is defined based on the compiler used. When using
gcc, it will include "__attribute__((always_inline))" forcing gcc to
always inline all the functions marked as FORCE_INLINE. However, this
can cause a performance degradation of about 15%.

This patch allows to set the FORCE_INLINE macro from the compiler
command line to either "static" or "static inline" giving allowing it
to inline functions as needed when performing optimizations.
 2017-08-07 10:34:33 +02:00
Yann Collet
6ad3a983db fix #369 
The bug would make the bt search read one byte in an invalid memory region,
and make a branch decision based on its value.
Impact was small (missed compression opportunity).
It only happens in -BD mode, with extDict-prefix overlapping matches.

The bt match search is supposed to work also in extDict mode.
In which case, the match ptr can point into Dict.
When the match was overlapping Dict<->Prefix,
match[matchLength] would end up outside of Dict, in an invalid memory area.
The correction ensures that in such a case,
match[matchLength] ends up at intended location, inside prefix.
 2017-06-26 11:29:05 -07:00
Yann Collet
a8dd86d93e changed macro HEAPMODE into LZ4_HEAPMODE 
This macro is susceptible to be triggered from user side
typically through compiler flag (-DLZ4_HEAPMODE=1).
In which case, it makes sense to prefix the macro
since we want to reduce potential side-effect on namespace.
 2017-05-01 22:32:21 -07:00
Yann Collet
e2827775ee make __packed memory access default for gcc 
It's always as good or better then memcpy()
but depends on gcc-specific extension.

solves https://github.com/facebook/zstd/issues/620
 2017-03-30 12:22:17 -07:00
Yann Collet
cd35f0d98c LZ4_compress_HC_destSize() uses LZ4HC_compress_generic() code path 
Limits compression level to 10,
to remain compatible with Hash Chain.
 2017-03-16 15:10:38 -07:00
Yann Collet
5eb547a608 fix #283 : implement LZ4_versionString(). 2016-12-04 20:05:36 +01:00
Yann Collet
a91983daa0 highly improved speed on -mx32 mode 
Now -mx32 is fastest mode on x64 CPU
 2016-11-18 17:05:06 -08:00
Yann Collet
8e1ea59ad6 attempt to fix sanitize32 package dependency 2016-11-17 18:30:46 -08:00
Yann Collet
1abecbc33c fix 32-bits mode. 
Large File support for Mac OS-X in 32-bits mode
Fixed potential undefined behavior
Changed makefile for 32-bits mode
 2016-11-17 13:02:06 -08:00
Yann Collet
4c5c711b7a fixed minor conversion warning 2016-11-14 14:53:15 -08:00
Yann Collet
f094f53144 silence a minor msan warning 2016-11-14 14:38:46 -08:00
Yann Collet
0fe5c187a5 fixed minor msan warning 2016-11-14 11:34:10 -08:00
Yann Collet
bb97e8a6fd fixed __GNUC__ macro 2016-11-12 13:30:02 -08:00
Yann Collet
3580d96980 enabled deprecation warnings on remaining obsolete functions 2016-11-12 08:48:42 -08:00
Nick Terrell
85aeb0e4bb Expose internal types to remove strict aliasing 2016-11-11 13:00:02 -08:00
Yann Collet
1f246a9899 Fixed #178 fullbench on small input 2016-11-10 15:31:59 -08:00
Yann Collet
e9089bcbe4 small compression ratio and speed improvement on small files 2016-11-07 17:31:38 -08:00
Yann Collet
f6a7c6b578 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev 2016-11-07 10:31:43 -08:00
Nick Terrell
920bf21714 Fix LZ4_decompress_fast_continue() bug 
It specified the external dictionary location incorrectly.
Add tests that expose this bug with both normal compilation and ASAN.
 2016-11-04 20:01:23 -07:00
Yann Collet
207f478bab minor refactor 2016-11-04 19:04:32 -07:00
Nick Terrell
f30c56c083 Quiet gcc-4.6.3 narrowing warning 2016-11-04 12:34:28 -07:00
Yann Collet
f878c08b76 better correctness on big-endian 64-bits platforms 2016-11-04 07:24:21 -07:00
Yann Collet
26dec498cf small compression speed improvement on 64-bits systems 2016-11-04 07:18:03 -07:00
Yann Collet
258a5e7fa4 updated comments 2016-11-03 17:14:25 -07:00
Przemyslaw Skibinski
84cedb4632 updated links to LZ4 repository 2016-11-03 15:12:57 +01:00
Yann Collet
33993542af removed test artefacts 2016-09-06 16:46:52 +02:00
Yann Collet
5bd3eaa481 minor comments 2016-08-21 00:08:38 +02:00
Yann Collet
9578ff539d minor refactor 2016-08-20 23:59:45 +02:00
Yann Collet
a7672dc4f9 minor decompression speed gains 2016-08-20 23:52:20 +02:00
Ben Wiederhake
d3e2d80266 Use https wherever possible 2016-07-08 00:46:02 +02:00
Yann Collet
6b1600f41e changed : default benchmark LZ4_decompress_safe() 2016-06-29 17:48:28 +02:00
Yann Collet
6923f3e230 Minor refactoring (code style) 2016-06-29 17:12:24 +02:00
Yann Collet
410ec81543 Fixed : alignment warning 2016-06-29 17:02:09 +02:00
Yann Collet
bc342ab94c minor refactoring 2016-06-29 16:01:24 +02:00
Yann Collet
c8a573d32c very minor decompression speed gain 2016-06-29 15:04:19 +02:00
Yann Collet
e27e7316dd minor code refactoring 2016-06-29 14:54:07 +02:00
Yann Collet
8938e10742 minor compression speed improvement 2016-06-29 14:07:54 +02:00
Yann Collet
c77c0c1086 minor refactoring 2016-06-29 13:21:23 +02:00
Yann Collet
3c03326004 minor refactoring (coding style) 2016-06-29 12:54:23 +02:00
Irwan Djajadi
bc70a4b72e fixed non-C99 compilers 2016-04-19 15:06:15 -05:00
Irwan Djajadi
10e3eecc56 fixed non-C99 compiler 2016-04-19 15:01:35 -05:00
Irwan Djajadi
973baa2c19 Merge branch 'heapmode_alloc_failure_fix' into dev 2016-04-19 08:22:56 -05:00
Irwan Djajadi
3e5bb67537 alloc failure fix 2016-04-18 15:51:32 -05:00
John Zhuge
cf6652f9b9 lz4cli: print library version 2016-04-01 07:07:25 -07:00
Julius Werner
dc868cd5b1 Allow for safe in-place decoding 
Add a check to allow safe "in-place" decoding (meaning that the
beginning of the source buffer partially overlaps the end of the
destination buffer). This is usually possible as long as the output
stops at least 15 bytes before the end of the input buffer (7 to account
for the extra spill from LZ4_wildCopy, 4 for a possible block checksum,
and 4 for the terminating block header), but in some pathological edge
cases it could be possible for the output stream to overwrite a byte in
the input stream before it gets decoded. With this patch the decoder
will reliably detect those cases and return a decoding error.

Signed-off-by: Julius Werner <jwerner@chromium.org>
 2016-02-12 22:52:51 -08:00
Yann Collet
2c80138121 heapmode macro option for lz4hc 2015-10-21 15:00:48 +01:00
Yann Collet
fb6fece770 Updated Makefile and .travis 2015-09-01 15:59:24 +01:00
Yann Collet
9aa5504f0c New unalign access method 2015-08-19 18:16:17 +01:00
Yann Collet
d15ac1f1a7 Slightly improved compression speed on ARMv6 2015-08-16 02:33:57 +01:00
Yann Collet
0f2bf0c54e Improved performance on ARMv6 2015-08-16 01:54:55 +01:00
Yann Collet
9517bd9234 minor simplification lz4_readarch() 2015-08-06 03:49:29 +01:00
Yann Collet
00eedb37d1 minor refactor 2015-07-16 09:38:10 +01:00
Yann Collet
3e579afa6d Slightly improved decompression speed 
Slightly decreased decompression stack usage in 64-bits mode
 2015-07-16 09:03:55 +01:00
Yann Collet
795a2cdb1b sligthly improved decompression speed 2015-07-16 08:55:48 +01:00
Yann Collet
91c1b9a682 Performance fix : big compression speed boost for clang (+30%) 2015-05-21 01:17:56 +01:00
Yann Collet
05c3f66bb8 Updated a few comments 2015-05-12 10:10:20 +01:00
Yann Collet
efbebd2a99 Added : LZ4_compress_destSize() 2015-05-06 02:29:04 +01:00
Yann Collet
1c3e633c48 Added compilation flag -Wcast-qual 2015-05-06 01:58:24 +01:00
Yann Collet
b4348a4718 Fixed minor Visual warning 2015-05-03 18:06:01 +01:00
Yann Collet
1b17bf2ab8 New lz4 API, using LZ4_compress_fast() 2015-05-03 17:57:46 +01:00
Yann Collet
791512cfdd Fixed bug 9318 2015-05-02 19:10:28 +01:00
Yann Collet
d6dc0a410d streaming API : Improved ring buffer management 2015-04-24 10:15:12 +01:00
Yann Collet
9c6fb8b160 Added LZ4_compress_fast_extState() 2015-04-23 07:46:35 +01:00
Yann Collet
f11afafe23 Removed LZ4_compress() (obsolete) from lz4 2015-04-21 18:31:35 +01:00
Takayuki MATSUOKA
d7298d2059 Replace GCC_VERSION with LZ4_GCC_VERSION 2015-04-15 06:39:04 +09:00
Yann Collet
2852b9e439 Fixed issue #84 2015-04-12 15:17:10 +01:00
Yann Collet
138673df5e fixed minor g++ warning 2015-04-12 09:37:34 +01:00
Yann Collet
81fdd9df23 Fixed a few Valgrind warnings 2015-04-12 09:29:52 +01:00
Yann Collet
973e385fa3 Implemented obsolete warning message 2015-04-11 18:59:22 +01:00
Yann Collet
f344fbd3ca Fixed a few warnings from -fsanitize=undefined 2015-04-09 22:59:07 +01:00
Yann Collet
2f8a4c32f9 New LZ4_compress_safe() API 2015-04-09 13:34:38 +01:00
Yann Collet
78d2dfd427 fullbench : tests of _limitedOutput variants intentionnally provides less memory space than safe (LZ4_compressBound()) 2015-04-01 18:21:03 +01:00
Yann Collet
28e237e954 simplified LZ4_compress_limitedOutput() 2015-04-01 15:53:08 +01:00
Yann Collet
76a03c1035 simplified LZ4_compress_withState() 2015-04-01 14:59:42 +01:00
Yann Collet
6625068902 simplified LZ4_compress() 2015-04-01 14:48:24 +01:00
Yann Collet
a761546b1b Fix : minor warning under Visual 2015-03-31 18:52:52 +01:00
Yann Collet
5b9fb69715 minor tweak 2015-03-30 22:39:08 +01:00
Yann Collet
4c227a487e Added LZ4_compress_fast() 2015-03-30 21:32:25 +01:00
Yann Collet
6c69dc176c faster compression in 64 bits mode 2015-03-30 18:34:15 +01:00
Yann Collet
a357f434f0 Fixed cast-align warnings on 32-bits 2015-03-25 18:06:40 +01:00
Yann Collet
7cf4e5c941 Updates tests & Man pages 2015-03-22 13:42:00 +01:00
Yann Collet
7d87d43e61 Updated lz4io sparse file support (alignment properties) 2015-03-17 18:02:01 +01:00
Yann Collet
bbcfe2144d Added : clang test 2015-03-15 14:19:47 +01:00
Yann Collet
45b0642bf5 scan-build tests 2015-03-15 01:42:27 +01:00
Yann Collet
45a357fd17 Improved sparse file support 2015-03-13 02:24:08 +01:00
Yann Collet
e3f33d2579 Fixed minor warnings 2015-03-11 18:38:44 +01:00
Yann Collet
ceec6fa849 g++ compatibility 2015-03-10 16:57:42 +01:00
Yann Collet
32a85fc76d NetBSD compatibility (#48) 2015-03-07 19:30:45 +01:00
Yann Collet
8a9fb8cf32 Fixed : older compiler don't like nameless unions, reported by Cheyi Lin 2014-12-16 22:03:16 +01:00
Yann Collet
95cc6cef64 Fixed : bug within LZ4 HC streaming mode, reported by James Boyle 2014-12-16 02:13:19 +01:00
Yann Collet
7a8cd69960 Fixed : GCC 4.9 bug on highest performance settings, reported by Greg Slazinski 2014-12-14 14:29:15 +01:00
Yann Collet
e68d7dcf22 Fixed : LZ4_compress_limitedOutput() bug, as reported by Christopher Speller 2014-12-10 17:58:15 +01:00
Yann Collet
3477cbac73 stronger tests for LZ4_compress_limitedOutput() 2014-12-10 09:05:44 +01:00
Yann Collet
b827ecf728 Clarified a few comments 2014-12-03 23:19:11 +01:00
Yann Collet
65ee6b09c4 Fixed : deprecated function LZ4_slideInputBufferHC() 2014-12-03 19:17:10 +01:00
Yann Collet
118296aeb0 variable ref renamed 2014-12-02 23:57:15 +01:00
Yann Collet
d008c87151 New directory structure : library source files into /lib directory 2014-11-30 23:32:12 +01:00