AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
1,935 Commits 13 Branches 31 Tags 6.8 MiB
49073aca86
Commit Graph

205 Commits

Author SHA1 Message Date
Alexey Tourbin
5603d30f81 lz4.c: fixed the LZ4_decompress_safe_continue case 
The previous change broke decoding with a ring buffer.  That's because
I didn't realize that the "double dictionary mode" was possible, i.e.
that the decoding routine can look both at the first part of the
dictionary passed as prefix and the second part passed via dictStart+dictSize.

So this change introduces the LZ4_decompress_safe_doubleDict helper,
which handles this "double dictionary" situation.  (This is a bit of
a misnomer, there is only one dictionary, but I can't think of a better
name, and perhaps the designation is not all too bad.)  The helper is
used only once, in LZ4_decompress_safe_continue, it should be inlined
with LZ4_FORCE_O2_GCC_PPC64LE attached to LZ4_decompress_safe_continue.

(Also, in the helper functions, I change the dictStart parameter type
to "const void*", to avoid a cast when calling helpers.  In the helpers,
the upcast to "BYTE*" is still required, for compatibility with C++.)

So this fixes the case of LZ4_decompress_safe_continue, and I'm
surprised by the fact that the fuzzer is now happy and does not detect
a similar problem with LZ4_decompress_fast_continue.  So before fixing
LZ4_decompress_fast_continue, the next logical step is to enhance
the fuzzer.
 2018-04-26 08:23:54 +03:00
W. Felix Handte
27c6eec18d Multiply-Include Header to Check Guard Macro Correctness 2018-04-24 18:50:03 -04:00
W. Felix Handte
2dfc7cbe82 Change Over Includes in the Project 2018-04-24 16:22:28 -04:00
Yann Collet
b2637ab7b2
Merge pull request #512 from lz4/HC_dict 
In-place unmutable dictionaries for LZ4HC
 2018-04-24 13:18:40 -07:00
W. Felix Handte
db9deb7b74 Remove the Framebench Tool 2018-04-24 11:58:51 -04:00
Cyan4973
bd06fde104 fullbench compiled without assert() 
to better reflect release speed
 2018-04-23 15:42:27 -07:00
Nick Terrell
672799e814 Fix compilation error and assert. 2018-04-23 14:21:02 -07:00
Nick Terrell
bb83cad98f Fix input size validation edge cases 
The bug is a read up to 2 bytes past the end of the buffer.
There are three cases for this bug, one for each test case added.

* An empty input causes `token = *ip++` to read one byte too far.
* A one byte input with `(token >> ML_BITS) == RUN_MASK` causes
  one extra byte to be read without validation. This could be
  combined with the first bug to cause 2 extra bytes to be read.
* The case pointed out in issue #508, where `ip == iend` at the
  beginning of the loop after taking the shortcut.

Benchmarks show no regressions on clang or gcc-7 on both my mac
and devserver.

Fixes #508.
 2018-04-23 13:34:18 -07:00
W. Felix Handte
209c9c29d1 Add Some Simple Fuzzer Tests 2018-04-20 15:16:41 -04:00
W. Felix Handte
e0d8add791 Fix Framebench Output Buffer Sizing 2018-04-19 20:45:48 -04:00
W. Felix Handte
9dae661b1f Fix Cast 2018-04-19 20:45:48 -04:00
W. Felix Handte
f646c512e9 Print Failure Message in Framebench 2018-04-19 20:45:48 -04:00
W. Felix Handte
9d971fd5c3 Switch to Unaligned Samples to Compress Different Blobs Each Time 2018-04-19 20:45:47 -04:00
W. Felix Handte
66f0c29aa4 Fix Framebench Statistics 2018-04-19 20:45:47 -04:00
W. Felix Handte
09df7a05f9 Add Run Name to Frame Bench Output 2018-04-19 20:45:47 -04:00
W. Felix Handte
a9a62321ff Auto-Calculate Appropriate Repetition Count 2018-04-19 20:45:47 -04:00
W. Felix Handte
d4ee7554fc Print More Detailed Results Inside bench(), Add Compression Levels 2018-04-19 20:45:47 -04:00
W. Felix Handte
ffb2d8b0ed Check Compressed Buffer is Correct in Frame Bench 2018-04-19 20:45:47 -04:00
W. Felix Handte
0bc13ab69c Add HC Calls to Framebench 2018-04-19 20:45:47 -04:00
W. Felix Handte
b9836b2a03 Restore Framebench Tool 
This reverts commit 70f14823a4.
 2018-04-19 20:45:47 -04:00
Yann Collet
444211d259 edited a few traces for debugging 2018-04-16 17:16:08 -07:00
Yann Collet
4aff9b10b5 fixed fuzzer tests 
which were modified in parallel within branc `dev`
 2018-04-16 16:14:28 -07:00
Yann Collet
b5528ac86c Merge branch 'dev' into lowAddr 2018-04-16 16:12:38 -07:00
W. Felix Handte
9f0f6b89bb Further Test that ExtDictCtx Mode Produces the Exact Same Output 2018-04-13 11:50:52 -04:00
W. Felix Handte
6dd64e0776 Add Tests for LZ4_attach_dictionary and Friends 2018-04-13 11:50:52 -04:00
test4973
8af32ce6f7 modified a few traces for debug 2018-04-12 13:35:19 -07:00
Yann Collet
ca388790e6 allow system-defined CPPFLAGS in /tests 2018-04-11 16:41:25 -07:00
Yann Collet
7b3cd10579 reduced test time on circle-ci 2018-04-11 16:31:43 -07:00
test4973
ad7e040384 fix minor conversion warning 
cast from void not implicit for C++
 2018-04-09 20:38:00 -07:00
test4973
f9992fa37f noticed a bug when re-using hash table 
./fuzzer -vv -s4217 -t7518
 2018-04-05 19:09:24 -07:00
test4973
6d931b6a93 fixed lz4 compression starting at small address 
when using byU32 and byU16 modes
 2018-04-05 12:40:33 -07:00
test4973
43132af808 Merge branch 'dev' into lowAddr 2018-04-04 11:38:55 -07:00
W. Felix Handte
ebdcbc359b Add Dependency to Fix Parallel make test Runs 
When run with `-jN`, the `rm tmp*` can run in the middle of the `test-lz4-dict`
job, which will then fail, finding its files to have been axed. This adds a
dependency between the two.
 2018-03-21 11:31:22 -04:00
test4973
c3f0ed28ff added low address fuzzer tests 2018-03-21 07:14:13 -07:00
W. Felix Handte
70f14823a4 Remove Framebench Tool 2018-03-14 15:59:36 -04:00
W. Felix Handte
c4aef7cd62 Restore checkTag Cleaning 2018-03-13 15:18:08 -04:00
W. Felix Handte
f9fef255a1 Renames and Comment Fixes 2018-03-12 18:34:41 -04:00
W. Felix Handte
9dcd9abc14 Make LZ4F_compressFrame_usingCDict Take a Compression Context 2018-03-12 14:58:43 -04:00
W. Felix Handte
b3628cb0c5 Avoid Resetting the Context When Possible 2018-03-12 14:58:43 -04:00
W. Felix Handte
f34fb3c42d Add Bounds Check to locateBuffDiff 2018-03-12 14:58:43 -04:00
W. Felix Handte
8c006b19bb Add a Benchmarking Tool For Compression with Context Re-Use 2018-03-12 14:58:43 -04:00
Yann Collet
0ddd1ceb1d added target make check 
according to GNU Makefile conventions,
the Makefile should feature a make check target
to self-test the generated program:
https://www.gnu.org/prep/standards/html_node/Standard-Targets.html .

this is much less thorough and less taxing than `make test`,
and can be run on any target in a reasonable timeframe (several seconds).
 2018-02-26 14:09:46 -08:00
Yann Collet
20e969e579 fuzzer: added low address compression test 
is expected to work on linux+gcc only.
 2018-02-05 15:19:00 -08:00
Yann Collet
b077a99347 added checkTag 
checkTag verifies that provided tag and library version match.
It's started automatically in circleCI when a new tag is created.
 2018-01-14 16:57:25 -08:00
Yann Collet
e2900241e8 object files in /tests 
object files created from /tests are compiled with specific flags (`-g -DLZ4_DEBUG=1`)
which are not welcomed in release binary.
`lib/*.o` files created from /tests are now stored in /tests, to avoid unintentional mix.
 2017-12-27 13:02:40 +01:00
Yann Collet
7d2f30c7d1 lz4opt supports _destSize 
no longer limited to level 9
 2017-12-22 12:47:59 +01:00
Yann Collet
fdde4311fb /tests programs compiled with LZ4_DEBUG=1 
to enable assert() within /lib
 2017-12-22 11:54:43 +01:00
Yann Collet
1025546347 unified HC levels 
LZ4_setCompressionLevel() can be users accross the whole range of HC levels
No more transition issue between Optimal and HC modes
 2017-11-03 11:28:28 -07:00
W. Felix Handte
93f8284c17 Add some tests verifying command line dictionary functionality 2017-10-09 20:18:03 -04:00
Yann Collet
2ed3612477 removed test-fasttest 2017-08-24 18:30:00 -07:00
Yann Collet
c04655bebf fixed make recurrence from /tests 2017-08-24 17:58:13 -07:00
Yann Collet
517dd95d92 removed fasttest 2017-08-24 17:03:03 -07:00
Yann Collet
82ccdec2fb minor : added header license 2017-08-24 16:59:47 -07:00
Yann Collet
5637d8856f make test is compatible with parallel execution (-j#) 
each test section runs with its own set of files
 2017-08-21 15:26:03 -07:00
Yann Collet
930a692110 cli : restored command -BX to enable block checksum (#322) 2017-08-11 17:58:46 -07:00
Yann Collet
77f99d2922 restored block checksum capability at lz4frame API level 2017-08-11 17:46:52 -07:00
Yann Collet
757497ae3d implemented lz4frame decompression API 2017-08-10 16:53:57 -07:00
Yann Collet
4531637ecd support dictionary compression with independent blocks 2017-08-10 12:12:53 -07:00
Yann Collet
d8aafe2c52 dictionary compression correctly uses compression level 
Not obvious : copying the state was copying cdict's compression level
 2017-08-10 00:48:19 -07:00
Yann Collet
31f2cdf4d2 implemented dictionary compression in lz4frame 
note : only compression API is implemented and tested
still to do : decompression API
 2017-08-09 16:51:19 -07:00
Yann Collet
a82dadfbae added dictID inside LZ4F_frameInfo_t 
Compressor can set dictID on LZ4F_compressBegin()
Decompressor can retrieve it using LZ4F_getFrameInfo()
 2017-08-08 17:43:11 -07:00
Jussi Kukkonen
e14b4c5a3e tests/Makefile: don't use LIBDIR as variable 
LIBDIR may be overriden with a environment variable: In this case make
clean breaks in tests/. Use another variable name.
 2017-06-26 13:32:40 +03:00
Yann Collet
d18084c655 report where decompression ends (#313) 
suggested by @ehem
note : only works for files < 2 GB
 2017-06-12 15:47:43 -07:00
Yann Collet
2600a154be fix (minor) g++ compatibility for frametest 2017-05-10 17:06:31 -07:00
Yann Collet
e60cbb5cac added test for LZ4F_resetDecompressionContext() 2017-05-10 16:28:36 -07:00
Yann Collet
2012e4de9e fixed c_standards tests 
and added entry "make list"
 2017-05-10 14:51:09 -07:00
Yann Collet
e2c9b19122 lz4frame : Added negative compression levels 2017-04-09 01:41:36 -07:00
Yann Collet
fc31257ab2 added LZ4F_resetDecompressionContext() 2017-03-28 17:36:12 -07:00
Yann Collet
f0a7651fce Safer LZ4_getFrameInfo() 
LZ4_getFrameInfo() is now guaranteed to keep dctx state clean,
even in case of failure.
 2017-03-28 17:10:01 -07:00
Dmitry V. Levin
b1daffc4e5 cli: add GNU separator -- specifying that all following arguments are files 
This option is supported by other compressors with compatible cli,
so add it to lz4 as well for better compatibility.
 2017-03-23 03:48:51 +00:00
Yann Collet
0d073d4d28 added extern C for lz4.h static section 
should make the file more compatible with C++ compiler, such as Visual or g++
 2017-03-17 15:11:09 -07:00
Yann Collet
aae447fffd LZ4_compress_HC_continue_destSize() works as intended up to level 10 
It's incompatible with btopt though,
so cLevel >= 11 feature much reduced performance (degraded mode)
 2017-03-16 15:41:30 -07:00
Yann Collet
cd35f0d98c LZ4_compress_HC_destSize() uses LZ4HC_compress_generic() code path 
Limits compression level to 10,
to remain compatible with Hash Chain.
 2017-03-16 15:10:38 -07:00
Yann Collet
430b7d32b3 created LZ4_HC_STATIC_LINKING_ONLY section 
where are exposed new prototypes *_destSize()
 2017-03-16 02:16:24 -07:00
remittor
66b26a389f tests: fuzzer: Add test for LZ4_compressHC_destSize 2017-03-08 11:13:28 +03:00
Dmitry V. Levin
0dfb0b9dad Fix test-lz4-basic 
When no output filename is specified and stdout is not a terminal,
lz4 doesn't attempt to guess an output filename and uses stdout for
output.

This change fixes test-lz4-basic when run without a terminal
by specifying output filenames.
 2017-03-05 23:28:25 +00:00
Przemyslaw Skibinski
d3eaf78838 remove MOREFLAGS from test-lz4c32 2017-02-16 16:30:16 +01:00
Przemyslaw Skibinski
ace22424d8 circle.yml: fixed lz4c32 target (2) 2017-02-15 23:19:30 +01:00
Przemyslaw Skibinski
0d77c85d9d circle.yml: fixed lz4c32 target 2017-02-15 22:10:11 +01:00
Przemyslaw Skibinski
4ee9bd5c3e improved Circle CI tests 2017-02-15 18:24:42 +01:00
Eric Siegerman
b89cac7b2e Don't use "foo && false || true" 
Replace it with either:
    test ! -f $FILE_THAT_SHOULD_NOT_EXIST
or:
    ! $COMMAND_THAT_SHOULD_FAIL

as appropriate.
 2017-02-14 14:17:06 -05:00
Przemyslaw Skibinski
c139eb40b0 added OpenBSD NetBSD SunOS to list of supported make install OSes 2017-02-10 15:00:27 +01:00
Przemyslaw Skibinski
489d1618c9 added "This Makefile is validated for" 2017-01-19 16:28:08 +01:00
Przemyslaw Skibinski
d2b51c22d3 fuzzer: tests more compression levels 2016-12-28 17:47:10 +01:00
Przemyslaw Skibinski
1c80b9af4e LZ4HC_getSearchNum 2016-12-28 15:18:19 +01:00
Przemyslaw Skibinski
c8b31263cc added FUZ_CLEVEL_DEFAULT 2016-12-28 13:08:38 +01:00
Przemyslaw Skibinski
a3d61cf3df improved test-lz4-opt-parser 2016-12-27 15:38:07 +01:00
Przemyslaw Skibinski
f4575f4f14 added test-lz4-opt-parser 2016-12-27 13:14:04 +01:00
Przemyslaw Skibinski
efaa9b7bc7 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev 
# Conflicts:
#	lib/Makefile
 2016-12-27 10:43:07 +01:00
Przemyslaw Skibinski
ea51ad0cee Solaris: working "make install" 2016-12-23 00:02:01 +01:00
Przemyslaw Skibinski
9b4b081c15 "make test" is now compatible with Solaris 2016-12-22 17:18:39 +01:00
Yann Collet
b41f9bb132 Merge pull request #292 from inikep/dev 
improved POSIX
 2016-12-22 11:58:29 +01:00
Przemyslaw Skibinski
f084b65383 test Large File support for Mac OS-X in 32-bits mode 2016-12-21 17:13:38 +01:00
Przemyslaw Skibinski
973bc79740 util.h and platform.h based on zstd 2016-12-21 16:10:09 +01:00
Yann Collet
7cf0bb97b2 LZ4F_compressBound(0) provides upper bound for LZ4F_flush() and LZ4F_compressEnd() [#290, suggested by @vtermanis] 2016-12-21 14:18:01 +01:00
Przemyslaw Skibinski
3c6ce9c665 fixed gcc 4.4 support 2016-12-21 10:39:27 +01:00
Przemyslaw Skibinski
64cbc4e1dc improved MinGW support 2016-12-21 10:22:40 +01:00
Przemyslaw Skibinski
9546ba62d0 executables use basic types from util.h 2016-12-21 09:44:59 +01:00
Przemyslaw Skibinski
2cfef13952 executables use platform.h 2016-12-20 18:05:43 +01:00
Yann Collet
de93e9e5d8 fix #285 : lz4cat doesn't work with stdin (reported by @beiDei8z) 2016-12-08 14:25:04 -08:00