AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
1,853 Commits 13 Branches 31 Tags 6.8 MiB
d28389b2dc
Commit Graph

686 Commits

Author SHA1 Message Date
W. Felix Handte
85cac61dd8 Don't Segfault on Malloc Failure 2018-04-20 19:35:51 -04:00
W. Felix Handte
756ed402da Sign-Extend -1 to Pointer Width 2018-04-20 17:56:26 -04:00
W. Felix Handte
86b381e40b Fix Constant Value 2018-04-20 17:13:40 -04:00
W. Felix Handte
1d2500d44e Handle Index Underflows Safely 2018-04-20 17:13:03 -04:00
W. Felix Handte
7874cf06b3 Consts and Asserts and Other Minor Nits 2018-04-20 15:30:08 -04:00
W. Felix Handte
3f087cf1cb Add Comments on New Public APIs 2018-04-20 15:00:53 -04:00
W. Felix Handte
d7347f9eea Add API for Attaching Dictionaries 2018-04-20 14:59:34 -04:00
W. Felix Handte
ca833f928f Also Reset the Chain Table 2018-04-20 14:16:27 -04:00
W. Felix Handte
8f118cf6e9 Remove inputBuffer from Context, Work Around its Absence 2018-04-20 14:08:06 -04:00
W. Felix Handte
0064e8ebc7 Remove Commented Out Support for Match Continuation over Segment Boundary 2018-04-20 13:14:37 -04:00
W. Felix Handte
14c577d4c9 Fix Signedness of Comparison 2018-04-19 20:54:35 -04:00
W. Felix Handte
f4b13e17ea Don't Clear the Dictionary Context Until No Longer Useful 2018-04-19 20:54:35 -04:00
W. Felix Handte
0abc23f72e Copy DictCtx into Working Context on Inputs Larger than 4 KB 2018-04-19 20:54:35 -04:00
W. Felix Handte
b67de2a327 Force Inline on HashChain 2018-04-19 20:54:35 -04:00
W. Felix Handte
22e16d5b50 Split DictCtx-using Code Into Separate Inlining Chain 2018-04-19 20:54:35 -04:00
W. Felix Handte
0a2abacd90 Use Fast Reset in LZ4F Again 2018-04-19 20:54:35 -04:00
W. Felix Handte
61c7ceffed Use Fast Reset API in LZ4F 2018-04-19 20:54:35 -04:00
W. Felix Handte
3591fe8ab8 Add Fast Reset Paths 2018-04-19 20:54:35 -04:00
W. Felix Handte
8db291bc1d Remove Match Upper Bounds Check 2018-04-19 20:54:35 -04:00
W. Felix Handte
8f9a2db0e1 Fix Some Cast/Conversion Warnings 2018-04-19 20:54:35 -04:00
W. Felix Handte
221211d7d0 Fix Offset Math 2018-04-19 20:54:35 -04:00
W. Felix Handte
a1beba13f7 Reset Stream in LZ4_compress_HC 2018-04-19 20:54:35 -04:00
W. Felix Handte
bdd7af6f71 Don't Bother Clearing Chain Table for Working Contexts 2018-04-19 20:54:35 -04:00
W. Felix Handte
895e76cc20 Push Previous Compression Offsets into the Past 2018-04-19 20:54:35 -04:00
W. Felix Handte
22db704a73 Shift Dict Limit Checks out of the Loop 2018-04-19 20:54:35 -04:00
W. Felix Handte
4f7b7a8ffa Clear Tables on Dict Load 2018-04-19 20:54:35 -04:00
W. Felix Handte
b88a0b4e88 Only Perform Dict Lookup if Attempts Remain 2018-04-19 20:54:35 -04:00
W. Felix Handte
b6c35ed642 Avoid Resetting Chain Table 2018-04-19 20:54:35 -04:00
W. Felix Handte
595ea58289 Avoid Resetting Hash Table 2018-04-19 20:54:35 -04:00
W. Felix Handte
66d217e240 Perform Lookups into the Dictionary Context 2018-04-19 20:54:35 -04:00
W. Felix Handte
6289ff4fb1 Call LZ4F_applyCDict Even on NULL CDict 2018-04-19 20:54:35 -04:00
W. Felix Handte
fdeead0b09 Set dictCtx Rather than memcpy'ing Ctx 2018-04-19 20:54:35 -04:00
W. Felix Handte
a992d11fc2 Fully Bounds Check Hash Table Reads 2018-04-19 20:54:35 -04:00
W. Felix Handte
f895b9a6c6 Add a Dictionary Context Pointer to the HC Context 2018-04-19 20:54:35 -04:00
W. Felix Handte
e75153f508 Add Debug Log Statements to HC 2018-04-19 20:54:35 -04:00
Yann Collet
62d7cdcc74
Merge pull request #503 from lz4/l120 
minor length reduction of several large lines
 2018-04-19 11:50:20 -07:00
Yann Collet
dede47f13b
Merge pull request #502 from lhacc1/dev 
Wrap likely/unlikely macroses with #ifndef
 2018-04-19 10:52:48 -07:00
Yann Collet
46058d71aa modified indentation for consistency 2018-04-19 10:50:40 -07:00
Baruch Siach
95bde2a4ae lib: allow to disable shared libraries 
Just like BUILD_STATIC=no disables static libraries, BUILD_SHARED=no
disabled shared libraries. This is useful to support toolchains that do
not support shared libraries.
 2018-04-19 12:28:11 +03:00
Yann Collet
4785bd6a35 minor length reduction of several large lines 2018-04-18 16:49:27 -07:00
Dmitrii Rodionov
ea6ed46fc2 Wrap likely/unlikely macroses with #ifndef 
It prevent redefine error when project using lz4 has its own likely/unlikely
macroses.
 2018-04-18 12:20:56 +03:00
Yann Collet
5ad4599c5a fixed LZ4_compress_fast_extState_fastReset() in 32-bit mode 2018-04-17 16:47:56 -07:00
Yann Collet
88cca1723e fix dictDelta setting error 
wrong test
 2018-04-17 16:18:37 -07:00
Yann Collet
1520642183 fix matchIndex overflow 
can happen with dictCtx
 2018-04-17 15:29:17 -07:00
Yann Collet
ce78d10c1f Merge branch 'dev' into lowAddr 2018-04-17 12:07:22 -07:00
W. Felix Handte
aedc447804 Always Bump Offset by 64 KB in LZ4_loadDict() 
This actually ensures the guarantee referred to in the comment in
LZ4_compress_fast_continue().
 2018-04-17 14:09:00 -04:00
Yann Collet
da3b5ba6f0 fixed dictCtx compression 2018-04-16 23:59:42 -07:00
Yann Collet
444211d259 edited a few traces for debugging 2018-04-16 17:16:08 -07:00
Yann Collet
a3aeb34184 fixed minor format warnings 2018-04-16 16:54:03 -07:00
Yann Collet
e928064797 fixed gcc performance regression 2018-04-16 15:11:28 -07:00
Yann Collet
d2bcfa31f5 fixed minor unused variable warning 2018-04-13 02:45:32 -07:00
Yann Collet
c40bac31d3 added comment on variables required after _next_match 2018-04-13 02:26:14 -07:00
Yann Collet
54ec83ce1f fixed potential ptrdiff_t overflow (32-bits mode) 
Also removed pointer comparison, which should solve #485
 2018-04-13 02:10:53 -07:00
Cyan4973
57afa36795 compatibility with gcc-4.4 string.h version 
Someone found it would be a great idea to define there a global variable under the very generic name "index".
Cause problem with shadow warnings, so no variable can be named "index" now ...

Also : automatically update API manual
 2018-04-13 01:01:54 -07:00
test4973
db9aa785c5 fixed : counting matches which overlap extDict and prefix 2018-04-12 16:12:21 -07:00
test4973
8af32ce6f7 modified a few traces for debug 2018-04-12 13:35:19 -07:00
test4973
1838803948 fixed LZ4_compress_fast_extState_fastReset() 2018-04-11 16:49:40 -07:00
test4973
b183066793 Merge branch 'dev' into lowAddr 2018-04-11 16:45:19 -07:00
W. Felix Handte
056ea63215 Fix Silly Warning (const-ness in declaration has no effect on value types!) 2018-04-11 18:42:09 -04:00
W. Felix Handte
51a56c47c0 Minor Fixes 2018-04-11 18:06:48 -04:00
W. Felix Handte
3a0c571272 Add a LZ4_STATIC_LINKING_ONLY Macro to Guard Experimental APIs 2018-04-11 18:06:10 -04:00
W. Felix Handte
afa52c9b95 Expose dictCtx Functionality in LZ4 2018-04-11 16:28:56 -04:00
W. Felix Handte
21f0c9700b Rename _extState_noReset -> _extState_fastReset and Edit Comments 2018-04-11 15:13:01 -04:00
W. Felix Handte
c18bff933b Remove Extraneous Assignment (clearedTable == 0) 2018-04-11 15:12:34 -04:00
W. Felix Handte
59c7d95121 Expose a Faster Stream Reset Function 2018-04-10 13:26:17 -04:00
test4973
cf2f06a6c5 fixed minor conversion warning 
ptr diff -> U32
 2018-04-09 17:08:17 -07:00
test4973
b28abb9f18 Merge branch 'dev' into lowAddr 2018-04-09 16:23:39 -07:00
W. Felix Handte
f88dc90055 Avoid Calling LZ4_prepareTable() in LZ4_compress_fast_continue() 2018-04-09 16:17:33 -04:00
W. Felix Handte
5622c276e1 Return to Allowing Early Returns in LZ4_compress_generic() 
Or: `goto` Considered Harmful

Or: https://xkcd.com/292/
 2018-04-06 19:28:08 -04:00
test4973
f9992fa37f noticed a bug when re-using hash table 
./fuzzer -vv -s4217 -t7518
 2018-04-05 19:09:24 -07:00
test4973
f4e06e28e6 fixed byPtr mode 
switch to byU32 when src address is < 64K

note : byPtr is still useful in 32-bits, as it's about ~10% faster
 2018-04-05 18:29:42 -07:00
test4973
b4be1e0a74 fixed byPtr match search 2018-04-05 17:52:54 -07:00
test4973
f2a4d6ef37 fixed immediate match search 2018-04-05 17:16:33 -07:00
test4973
64a3e41aca changed LZ4_compress_generic() logic 
to use indexes (U32) instead of Ptr.

byPtr is still present.
 2018-04-05 16:43:46 -07:00
test4973
6d931b6a93 fixed lz4 compression starting at small address 
when using byU32 and byU16 modes
 2018-04-05 12:40:33 -07:00
test4973
43132af808 Merge branch 'dev' into lowAddr 2018-04-04 11:38:55 -07:00
Yann Collet
8c763aa900
Merge pull request #487 from felixhandte/better-obsoletion-comment 
Better Describe Functionality of Obsolete Streaming Functions
 2018-03-21 14:52:53 -07:00
W. Felix Handte
126f18d3e0 Also Fix a Comment 2018-03-21 11:48:35 -04:00
W. Felix Handte
a3a9b80dff Better Describe Functionality of Obsolete Streaming Functions 2018-03-21 11:39:41 -04:00
Yann Collet
863e24892d fix comment style 2018-03-21 07:07:24 -07:00
W. Felix Handte
b0a18896fe Move LZ4_compress_fast_extState_noReset Declaration to Unstable Section 2018-03-14 15:59:33 -04:00
W. Felix Handte
66b6fbfe6f Restore the Other Old Streaming Functions in a Degraded Fashion 2018-03-14 15:53:10 -04:00
W. Felix Handte
c852f20c39 Switch ALLOC() to ALLOC_AND_ZERO() to Paper Over Existing Uninitialized Read 2018-03-13 17:47:34 -04:00
W. Felix Handte
995756f218 Split lz4CtxLevel into Two Fields 2018-03-13 17:45:09 -04:00
W. Felix Handte
640db34e43 Another Allocation Fail Check 2018-03-13 17:35:44 -04:00
W. Felix Handte
146e676531 Restore LZ4_sizeofStreamState, We Didn't Actually Need to Delete It 2018-03-13 15:42:03 -04:00
W. Felix Handte
2be38a7429 Rename Enums and Add Comment 2018-03-13 15:16:52 -04:00
W. Felix Handte
b8e9c77855 Whitespace Fixes 2018-03-13 13:20:37 -04:00
W. Felix Handte
5149767a1b Add NULL Checks 2018-03-12 18:35:09 -04:00
W. Felix Handte
299f34909a Simpler Ternary Statements 2018-03-12 18:35:09 -04:00
W. Felix Handte
f9fef255a1 Renames and Comment Fixes 2018-03-12 18:34:41 -04:00
W. Felix Handte
1df5d911aa Hoist LZ4F Dictionary Setup into Helper LZ4F_applyCDict() 2018-03-12 16:11:55 -04:00
W. Felix Handte
3ecc1d7a5b Minor Style Fixes 2018-03-12 16:11:44 -04:00
W. Felix Handte
e34716cc3f Preserve currentOffset==0 When Possible 2018-03-12 14:59:28 -04:00
W. Felix Handte
5f8967b983 Specialize _extState() for Clean Ctx Rather Than Calling _safeExtState() 2018-03-12 14:58:43 -04:00
W. Felix Handte
6716325ae8 Remove Switch In Favor of Ternary Statement 2018-03-12 14:58:43 -04:00
W. Felix Handte
b4335a6585 Further Avoid a dictionary==NULL Check 2018-03-12 14:58:43 -04:00
W. Felix Handte
64bcbf400e Optimize Dict Check Condition 2018-03-12 14:58:43 -04:00
W. Felix Handte
b78cf67c96 Move to 4KB Cut-Off 2018-03-12 14:58:43 -04:00
W. Felix Handte
00eadadbfc Reset Table on Inputs Larger than 2KB 2018-03-12 14:58:43 -04:00
W. Felix Handte
d571d0cdba Avoid DictSmall Checks By Strategically Bumping CurrentOffset 2018-03-12 14:58:43 -04:00
W. Felix Handte
1c4601d643 Restore DictIssue Check 2018-03-12 14:58:43 -04:00
W. Felix Handte
d6ed9a7799 Avoid dictionary == NULL Check 2018-03-12 14:58:43 -04:00
W. Felix Handte
efc419a6d4 Replace calloc() Calls With malloc() Where Possible 2018-03-12 14:58:43 -04:00
W. Felix Handte
80790c587b Copy the Dict Table Into the Context for Large Compressions 2018-03-12 14:58:43 -04:00
W. Felix Handte
9dcd9abc14 Make LZ4F_compressFrame_usingCDict Take a Compression Context 2018-03-12 14:58:43 -04:00
W. Felix Handte
14ce912b70 Switch Current Offset to 1 Only When in External Dictionary Context Mode 2018-03-12 14:58:43 -04:00
W. Felix Handte
cea09d67a9 Hoist Table Reset One Level Up 2018-03-12 14:58:43 -04:00
W. Felix Handte
68c6bd17b8 Set Dictionary Context Pointer Rather than Copying the Context In 2018-03-12 14:58:43 -04:00
W. Felix Handte
73cc39327e Lookup Matches in Separate Dictionary Context 2018-03-12 14:58:43 -04:00
W. Felix Handte
62cb52b341 Initialize Current Offset to 1 2018-03-12 14:58:43 -04:00
W. Felix Handte
7060bcabf0 Only Re-Alloc / Reset When Needed When Switching Between Regular and High Compression Modes 2018-03-12 14:58:43 -04:00
W. Felix Handte
b3628cb0c5 Avoid Resetting the Context When Possible 2018-03-12 14:58:43 -04:00
W. Felix Handte
aa36e118f1 Const-ify Table Arg to LZ4_getPosition(OnHash) 2018-03-12 14:58:43 -04:00
W. Felix Handte
d6a3024dbb Add LZ4_compress_fast_safeExtState Function 2018-03-12 14:58:43 -04:00
W. Felix Handte
5709891de6 Add a Table Type Field to LZ4_stream_t 2018-03-12 14:58:43 -04:00
W. Felix Handte
6933f5ad9c Remove Obsolete Stream Functions to Free Space in LZ4_stream_t 2018-03-12 14:58:43 -04:00
W. Felix Handte
6d156fea56 Allow Empty Dictionaries 2018-03-12 14:58:43 -04:00
Yann Collet
b5233d3726 updated LZ4F_compressBound() documentation 
to clarify it includes potentially buffered data.
 2018-02-27 23:23:27 -08:00
Yann Collet
860ff77909
Merge pull request #478 from lz4/mergeOpt 
merge lz4opt.h into lz4hc.c
 2018-02-26 14:06:31 -08:00
Yann Collet
39fda9a447 bumped version number to v1.8.2 
updated NEWS was current progresses
 2018-02-26 13:50:04 -08:00
Yann Collet
ba115386fa update code comment on LZ4 streaming interface 
notably regarding LZ4_saveDict() speed advantage,
answering #477.
 2018-02-26 13:31:18 -08:00
Yann Collet
550b40849f merge lz4opt.h into lz4hc.c 
Having a dedicated file for optimal parser
made sense during its creation,
it allowed Przemyslaw to work more freely on lz4opt, with less dependency on lz4hc,
moreover, the optimal parser was more complex, with its own search functions.

Since the optimal was rewritten last year, it's now a lot lighter.
It makes more sense now to integrate it directly inside lz4hc.c,
making it easier to edit (editors are a bit "lost" inside a `*.h` dependent on its #include position),
it also reduces the number of files in the project,
which fits pretty well with lz4 objectives.
(adding lz4hc requires "just" lz4hc.h and lz4hc.c).
 2018-02-25 00:32:09 -08:00
Yann Collet
7173a631db edge case : compress up to end-mflimit (12 bytes) 
The LZ4 block format specification
states that the last match must start
at a minimum distance of 12 bytes from the end of the block.

However, out of an abundance of caution,
the reference implementation would actually stop searching matches
at 13 bytes from the end of the block.

This patch fixes this small detail.
The new version is now able to properly compress a limit case
such as `aaaaaaaabaaa\n`
as reported by Gao Xiang (@hsiangkao).

Obviously, it doesn't change a lot of things.
This is just one additional match candidate per block, with a maximum match length of 7 (since last 5 bytes must remain literals).

With default policy, blocks are 4 MB long, so it doesn't happen too often
Compressing silesia.tar at default level 1 saves 5 bytes (100930101 -> 100930096).
At max level 12, it saves a grand 16 bytes (77389871 -> 77389855).

The impact is a bit more visible when blocks are smaller, hence more numerous.
For example, compressing silesia with blocks of 64 KB (using -12 -B4D) saves 543 bytes (77304583 -> 77304040).
So the smaller the packet size, the more visible the impact.

And it happens we have a ton of scenarios with little blocks using LZ4 compression ...

And a useless "hooray" sidenote :
the patch improves the LZ4 compression record of silesia (using -12 -B7D --no-frame-crc) by 16 bytes (77270672 -> 77270656)
and the record on enwik9 by 44 bytes (371680396 -> 371680352) (previously claimed by [smallz4](http://create.stephan-brumme.com/smallz4/) ).
 2018-02-24 11:47:53 -08:00
Yann Collet
71e16fa11a
Merge pull request #471 from lz4/fasterHC 
Faster HC
 2018-02-20 21:04:07 -08:00
Yann Collet
25b16e8a2e added one assert() 
suggested by @terrelln
 2018-02-20 15:25:45 -08:00
Yann Collet
d74f079748 update API doc regarding double-buffer strategy 
answering question #473
 2018-02-18 11:00:33 -08:00
Yann Collet
d3a13397d9 slight hc speed benefit (~+1%) 
by optimizing countback
 2018-02-12 00:01:58 -08:00
Yann Collet
219abab74b removed LZ4_copy8 
better use memcpy() directly
 2018-02-11 22:20:09 -08:00
Yann Collet
2b674bf02f slightly improved hc compression speed (+~1-2%) 
by removing bad candidates faster.
 2018-02-11 02:45:36 -08:00
Yann Collet
3ad3b0f850 slightly improved decompression speed (~+1-2%) 
by making shortcut slightly more common
 2018-02-11 01:43:20 -08:00
Ben Boeckel
c4671be550 intel: do not use __attribute__((packed)) on Windows 
On Windows, the Intel compiler is closer to MSVC rather than GCC and
does not support the GCC attribute syntax.

Fixes #468
 2018-02-08 09:15:27 -05:00
Yann Collet
ea25250c99 fixed code comment as detected in #466 
Also clarified a few API code comments
and updated associated html documentation
 2018-02-07 02:21:25 -08:00
Yann Collet
20e969e579 fuzzer: added low address compression test 
is expected to work on linux+gcc only.
 2018-02-05 15:19:00 -08:00
Nick Terrell
e832a3d87a Clarify the requirements of the LZ4 streaming API 2018-02-01 16:08:59 -08:00
Yann Collet
5fd3ac7904 Merge branch 'dev' into frameCompress 2018-01-31 17:18:57 -08:00
Yann Collet
87fb7a1d03 refactored frameCompress example 
to better reflect LZ4F API usage.
 2018-01-31 14:33:16 -08:00
Asger Hautop Drewsen
c129f480e7 Always prefer c++14 attributes if available 2018-01-31 20:24:44 +01:00
Asger Hautop Drewsen
865bd83e13 Ensure LZ4_DEPRECATED("...") is before LZ4LIB_API 
When using clang++ with std c++14 or c++17 you would get the error "an attribute list cannot appear here" when including "lz4.h" as the visibility attribute is before the c++ attribute.
This ensures that the [[deprecated]] c++ attribute is before everything
else in the function declarations.
 2018-01-31 13:33:07 +01:00
Nick Terrell
30e92f320c [lz4hc] level == 0 means default, not level 1 2018-01-22 12:50:06 -08:00
Po-Chuan Hsieh
75b81bbbf0 Change file format back to ASCII (from UTF-8) 
- Replace U+00A0 by space
- Fix build failure of archivers/py-borgbackup in FreeBSD

Reference: https://bugs.FreeBSD.org/225235
 2018-01-18 03:13:05 +08:00
Yann Collet
5e7780d2d8 lz4frame : removed some intermediate stage from LZ4F_decompress() 
ensure some strange jump cases are not possible
(they were already not possible, but static analyzer couldn't understand it).
 2018-01-14 00:15:07 -08:00
Yann Collet
18b4c66d25 ensure a ptr is non-null 
with an assert()
to help static analyzer understanding this condition.
 2018-01-13 22:47:46 -08:00
Yann Collet
4d61ebc9c8 modified formulation for LZ4F_compressBound() 
previous version used an intentional overflow,
which is defined since it uses unsigned type,
but static analyzer complain about it.
 2018-01-13 22:39:39 -08:00
Po-Chuan Hsieh
47bf1a9f01 Fix lz4 version 2018-01-14 06:38:03 +08:00
Yann Collet
c423dc21bd updated LZ4F_decompress() documentation 2018-01-13 13:16:31 -08:00
Yann Collet
58199f1311
Merge pull request #443 from terrelln/440 
[lz4f] Skip memcpy() on empty dictionary
 2018-01-10 19:06:21 +01:00
W. Felix Handte
ebef34fe79 Add Option to Make lz4frame_static.h Functions Visible in Shared Objects 
In some contexts, *cough*like at facebook*cough*, dynamic linking is used in
contexts which aren't truly dynamic. That is, the guarantee is maintained that
a program will only ever execute against the library version it was compiled
to interact with.

For those situations, introduce a compile-time flag that overrides hiding
these unstable APIs in shared objects.
 2018-01-08 14:46:22 -05:00
Yann Collet
0b203b04f6
Merge pull request #434 from lz4/pattern 
conditional pattern analysis
 2018-01-06 06:58:41 +01:00
Nick Terrell
c2dd686e96 [lz4f] Skip memcpy() on empty dictionary 2018-01-05 14:30:49 -08:00