AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
920 Commits 13 Branches 31 Tags 6.8 MiB
1b16805941
Commit Graph

357 Commits

Author SHA1 Message Date
Yann Collet
e277511ab6 Merge pull request #54 from t-mat/gc-issue/155 
Gc issue/155
 2015-03-08 19:06:17 +01:00
Yann Collet
32a85fc76d NetBSD compatibility (#48) 2015-03-07 19:30:45 +01:00
Yann Collet
488029ec3b Updated : compress multiple files 2015-03-07 13:23:00 +01:00
Yann Collet
046bd3abee Merge pull request #52 from KyleJHarper/r128/multiple_inputs 
Adding support for multiple file processing.  Fixes Google code issue 151.
 2015-03-03 01:04:40 +01:00
Takayuki MATSUOKA
dcdd628d5e Fix sentinel bit pattern 
Set all bits of U64.
 2015-03-02 13:38:30 +09:00
Takayuki MATSUOKA
de5c930c90 Fix sentinel size miscalculation 
Since is_nul() style wild-comparing stride is U64, we must add sizeof(U64) to the sentinel.
 2015-03-02 13:38:29 +09:00
Takayuki MATSUOKA
eed7952101 Add GNU coreutil's is_nul() method to isSparse() 
See original source
http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/system.h
 2015-03-02 13:38:28 +09:00
Takayuki MATSUOKA
b372f45093 Add Neil's method to isSparse() 2015-03-02 13:38:27 +09:00
Takayuki MATSUOKA
01a24afdcf Improve isSparse() 2015-03-02 13:38:25 +09:00
Takayuki MATSUOKA
4a5d92b1d9 Adjust coding style 2015-03-02 13:38:24 +09:00
Takayuki MATSUOKA
97679fa5a1 Google Code Issue 155: lz4 cli should support sparse file 
https://code.google.com/p/lz4/issues/detail?id=155

This is experimental implementation. Just a proof of concept.
It works Linux and Windows.

# Build

To build experimental version, define 'LZ4IO_ENABLE_SPARSE_FILE' like the following command :

    make lz4programs 'CFLAGS=-O3 -DLZ4IO_ENABLE_SPARSE_FILE=1'
    ./programs/lz4 -h

You will see "EXPERIMENTAL_SPARSE_FILE" as lz4 revision :

    "*** LZ4 command line interface 64-bits EXPERIMENTAL_SPARSE_FILE, by Yann Collet (...) ***"

# Experiment

This experimental version adds option "-x" for sparse file for decompression.
You can use this option like this :

    ./programs/lz4 -9 -f my-file
    ./programs/lz4 -d -f -x my-file.lz4 my-file.lz4.out
    cmp my-file my-file.lz4.out
 2015-03-02 13:38:23 +09:00
kyle
fa27d236b4 Added support for multiple input files to act more like other compressors. For example: gzip file1 file2 file3. You can now do: lz4 [args] -m file1 file2 file3. Fixes 151. 2015-03-01 19:00:40 -06:00
Yann Collet
67f3b4141a Merge pull request #49 from t-mat/msvc-fseeki64 
Avoid MSVC fseek()'s 2GiB barrier
 2015-03-02 00:41:38 +01:00
Yann Collet
e68d1c98c1 restored lz4 hc compression ratio 2015-03-02 00:11:34 +01:00
Takayuki MATSUOKA
41b6ed3c5b Replace fseek with _fseeki64 to avoid MSVC's 2GiB barrier 
Since MSVC fseek()'s SEEK_CUR mode has 2GiB barrier, our fseek() calling in program/lz4io.c/selectDecoder() will fail for large (>2GiB) .lz4 file.
This commit just replace fseek with _fseeki64 by macro. Second argument is automatically casted to __int64 (signed 64bit integer). Other things are completely same.
 2015-02-14 12:48:11 +09:00
Yann Collet
f68eead36c Added : -pedantic compilation option 2014-12-17 12:32:49 +01:00
Yann Collet
0569a68edd C90 compatibility 2014-12-17 12:20:17 +01:00
Yann Collet
8a9fb8cf32 Fixed : older compiler don't like nameless unions, reported by Cheyi Lin 2014-12-16 22:03:16 +01:00
Yann Collet
95cc6cef64 Fixed : bug within LZ4 HC streaming mode, reported by James Boyle 2014-12-16 02:13:19 +01:00
Yann Collet
daa320f3f7 New : lz4frame integrated into liblz4 (v1.5.0) 2014-12-13 15:05:46 +01:00
Yann Collet
7dd50ed2f6 minor memtest correction 2014-12-10 23:00:50 +01:00
Yann Collet
e68d7dcf22 Fixed : LZ4_compress_limitedOutput() bug, as reported by Christopher Speller 2014-12-10 17:58:15 +01:00
Yann Collet
3477cbac73 stronger tests for LZ4_compress_limitedOutput() 2014-12-10 09:05:44 +01:00
Yann Collet
c5decf7562 minor refactoring 2014-12-06 17:10:54 +01:00
Yann Collet
9f6826fdc9 lz4cli : legacy arguments are now disabled by default 2014-12-01 01:25:18 +01:00
Yann Collet
d008c87151 New directory structure : library source files into /lib directory 2014-11-30 23:32:12 +01:00
Yann Collet
6de52c2a8c LZ4IO : modified to use lz4frame instead 2014-11-30 17:59:31 +01:00
Yann Collet
d0a6cccf75 Fixed : some minor Visual warnings 2014-11-30 12:58:00 +01:00
Yann Collet
326cada9fa Updated lz4hc : re-use most shared elements from lz4 (endianess / align / bus detection routines) 2014-11-29 20:19:39 +01:00
Yann Collet
dc43a1f6b5 Fixed : decompression issue on 32-bits CPU without unaligned memory access 2014-11-29 16:41:28 +01:00
Yann Collet
2d98faf238 Improved decoding speed 2014-11-27 22:44:36 +01:00
Yann Collet
86fa21b301 Added : $(EXT) within install for cross-compilation support 2014-11-23 18:36:04 +01:00
Yann Collet
2ad37dbe58 Corrected lz4io 2014-11-23 01:14:04 +01:00
Yann Collet
73554386ed Makefile : refactored Travis CI tests 2014-11-04 20:56:38 +01:00
Yann Collet
89d8b98d0d Fixed issue 44 : stopped concatenated stream on null size frame (thanks to Brendan O'Connor) 2014-11-04 12:11:14 +01:00
Yann Collet
d01bb4a193 Makefile : added -Wcast-align compilation warning 2014-11-04 10:32:50 +01:00
Yann Collet
7050d4b767 Updated unsynchronized buffers tests 2014-11-04 10:04:37 +01:00
Yann Collet
b005e1b42e fullbench : display Algorithm Nb 2014-11-03 09:18:15 +01:00
Yann Collet
61289dea1d Optimized LZ4_saveDictHC() 2014-11-02 22:32:12 +01:00
Yann Collet
55e8a47474 Added : valgrind test for fuzzer 2014-10-29 01:47:43 +01:00
Yann Collet
5a5a15969e minor fuzzer test correction 2014-10-29 01:39:47 +01:00
Yann Collet
d4f2d8061e Fixed : frametest for Travis CI build matrix 2014-10-28 15:58:25 +01:00
Yann Collet
9bf331afc6 new Travis CI build matrix 2014-10-28 15:35:43 +01:00
Yann Collet
6436eca4fa Added : frametest32 2014-10-28 11:32:42 +01:00
Yann Collet
7a677a77a6 LZ4 HC streaming integrated within lz4frame 2014-10-28 00:39:06 +01:00
Yann Collet
508855c488 HC streaming : support for very long streaming (> 4 GB) scenarios 2014-10-26 11:22:15 +01:00
Yann Collet
2b421e97d4 HC streaming : support small ringbuffer scenarios 2014-10-25 20:52:10 +01:00
Yann Collet
e468a4eb96 fixed LZ4F_compressFrameBound (dynamic block resize) 2014-10-22 00:36:32 +01:00
Yann Collet
e450018588 LZ4 HC : fixed small dictionary streaming compression 2014-10-21 00:12:55 +01:00
Yann Collet
3dab5f476a LZ4 HC : External Dictionary compression : First implementation 2014-10-20 01:08:21 +01:00
Yann Collet
d618673057 fuzzer : easier random replay 2014-10-19 16:06:33 +01:00
Yann Collet
f91d5aae79 fuzzer : fixed test 21 2014-10-19 15:25:35 +01:00
Yann Collet
6b89dca1a9 fuzzer : faster jump to testNb 2014-10-19 13:47:22 +01:00
Yann Collet
68ea218a72 fuzzer : fixed minor command lines issues 2014-10-18 13:27:32 +01:00
Yann Collet
d239a23337 updated LZ4HC API 2014-10-18 11:18:14 +01:00
Yann Collet
4da47a2c67 fuzzer : pause at the end is disabled by default 2014-10-10 20:58:42 +01:00
Yann Collet
92574da1a0 modified : fuzzer automatically selects seed 2014-10-09 22:15:57 +01:00
Yann Collet
2ab6f9a387 Updated : xxHash to r37 2014-10-06 11:13:56 +01:00
Yann Collet
c14a342cc2 Fixed : LZ4F_compressBound() using null preferencesPtr 
Updated : frametest, to check LZ4F_compressBound() using null preferencesPtr
 2014-10-06 00:48:45 +01:00
Yann Collet
8c1ae99837 fixed : clang warnings 2014-09-22 18:42:00 +01:00
Yann Collet
abb6f7806a Added : suooirt for s390x, thanks to Nobuhiro Iwamatsu 
Multiple warning fixes for Visual Studio 2012
 2014-09-22 17:38:17 +01:00
Yann Collet
6e1049872a LZ4F_compressFrame : fix potential crash on selecting custom preferences 
frame fuzzer tests : new random tests using LZ4F_compressFrame
 2014-09-22 02:59:42 +01:00
Yann Collet
eeb9011467 lz4frame : support stableDst option 
Multiple bugfixes within lz4frame decompression
Small decompression speed improvements
Improved fuzzer test, with more thorough and complex tests
 2014-09-21 09:56:21 +01:00
Yann Collet
d71b9e25b7 small improvements to lz4frame compression 2014-09-15 00:59:30 +01:00
Yann Collet
562b34f660 changed : lz4 test mode (-t) to no longer ask for confirmation, as suggested by Nguyen Thary 2014-09-13 23:49:45 +01:00
Yann Collet
7716c03dbf changed xxhash directory 
changed makefile
 2014-09-13 23:44:07 +01:00
Yann Collet
3f80a22819 Merge pull request #26 from Cyan4973/frame 
Frame
 2014-09-14 00:30:01 +02:00
Yann Collet
457dc35e6a Restored variable output size fuzzer test 
Quickfix frame decompression
Small speed optimization frame decompression
 2014-09-13 21:21:41 +01:00
Yann Collet
56c2b79ed0 Frame decompression speed optimization 2014-09-13 19:49:01 +01:00
Yann Collet
38912f55e3 fullbench : added benchmark for LZ4F_decompress() 2014-09-13 15:24:16 +01:00
Yann Collet
658ab6cfca LZ4F_decompressFrame : implemented srcSizeHint as function result 2014-09-13 12:15:54 +01:00
Yann Collet
bd704cf70a lz4frame : implemented option stableSrc 
Improved LZ4_compressFrame() speed
 2014-09-13 10:08:55 +01:00
Yann Collet
a586208597 lz4frame : autoflush mode : saves memory, and improves compression speed 
LZ4F_getFrameInfo() fix, thanks to Christopher Jackson
 2014-09-12 19:28:44 +01:00
Yann Collet
b1d022fa72 slightly improved frame compression speed 2014-09-11 22:27:14 +01:00
Yann Collet
c71de79688 Improved lz4frame compression speed 
Added : fullbench test -c14 (LZ4F_compressFrame)
 2014-09-10 22:17:03 +01:00
Yann Collet
ed4a6bf2cb More tests : random flushes 2014-09-10 13:53:42 +01:00
Yann Collet
0400451ac2 Fix : streaming mode bug (re-using context & buffers) 2014-09-10 13:00:39 +01:00
Yann Collet
eac83cd850 Added : streaming mode 2014-09-09 23:54:22 +01:00
Yann Collet
6b407c65a4 Added : valgrind verification for frametest 2014-09-07 14:15:33 +01:00
Yann Collet
d1d1f8835d API change : compressionContext more easily reusable 2014-09-07 12:57:09 +01:00
Yann Collet
cf28c27809 Added : fuzzer : ability to jump to specified testNb 2014-09-07 12:12:46 +01:00
Yann Collet
cdececa38d Added : variable dstBuffer size decompression test 2014-09-07 11:04:29 +01:00
Yann Collet
535120bbe3 More complex compression tests 2014-09-07 09:13:02 +01:00
Yann Collet
04f5b17b72 Improved frame decompression test 
Fixed : frame decompression bug
 2014-09-06 22:48:03 +01:00
Yann Collet
03bc760af3 minor frametest display improvement 2014-09-06 13:06:28 +01:00
Yann Collet
e6d9a17188 minor test code refactoring 2014-09-06 12:19:26 +01:00
Yann Collet
5c32b50b94 More tests (variation of blocksize & checksum) 
fixed : checksum error on dealing with uncompressed blocks
 2014-09-06 09:47:28 +01:00
Yann Collet
fd8665320f Added some more tests 2014-09-05 16:32:04 +01:00
Yann Collet
b03f8f0e6f fixed : LZ4F_decompress() 
more decompress tests
 2014-09-04 22:56:51 +01:00
Yann Collet
e619cfe852 Completed first version of lz4frame decompress 
Added a first decompression test
 2014-09-03 19:49:59 +01:00
Yann Collet
f7f67e778c Added : preliminary frame decompression function 2014-09-01 22:44:02 +01:00
Yann Collet
2f33c77d53 upgraded lz4frame compression tests 2014-08-31 23:14:20 +01:00
Yann Collet
f66721d303 lz4frame compression API v1 completed 
Added : test program frametest
 2014-08-30 18:14:44 +01:00
Yann Collet
3b4e3f2b01 minor fuzzer code cleanup 2014-08-30 18:13:45 +01:00
Yann Collet
14a28c0633 minor fuzzer cleaning 2014-08-30 12:32:09 +01:00
Yann Collet
557ff22334 Fixed crlf issue 2014-08-26 15:53:39 +02:00
Yann Collet
da07465b64 Simplified travis test flow 2014-08-26 14:39:09 +01:00
Yann Collet
ff94eec374 Added macro, for large file support on 32-bits AIX 2014-08-20 16:37:34 +01:00
Yann Collet
a74f7ee811 Update xxHash to r36 2014-08-15 10:24:24 +01:00
Yann Collet
4dc1eca5a9 updated release number 2014-08-13 16:44:44 +01:00
Yann Collet
205735a9f2 Fixed crlf issue 2014-08-06 13:50:38 +02:00
Yann Collet
4ea832b8ff Updated Makefile : make dist 
Minor formatting changes for xxhash
 2014-08-06 08:55:08 +01:00
Yann Collet
a0755949b6 fix : minor compilation warning under mingw 2014-08-05 06:42:40 +01:00
Yann Collet
ea74cc4676 Added : HC streaming test 2014-08-01 19:21:26 +01:00
Yann Collet
e4aaa681ca fix : minor compilation warning under mingw 2014-07-29 21:57:09 +01:00
Yann Collet
a50c102f5a Added : Makefile : install for kFreeBSD and Hurd (Nobuhiro Iwamatsu) 2014-07-26 15:15:00 +01:00
Yann Collet
2c15ec421a Makefile : some corrections suggested by Michał Górny (issue 136) 2014-07-21 21:01:59 +01:00
Yann Collet
2f93f3afd5 Updated xxHash to r35 2014-07-19 10:03:53 +01:00
Yann Collet
a154628e70 Makefile : added -Wstrict-prototype, as suggested by Larry Doolittle 2014-07-14 23:29:40 +01:00
Yann Collet
fbe14d128e Modified : lz4 streaming API, strong types 2014-07-14 23:04:10 +01:00
Yann Collet
d3c43d3251 Modified : streaming API (fast compression) 2014-07-05 16:48:49 +01:00
Yann Collet
e002818cf6 Modified : Makefile : added -Wshadow option, suggest by Boris Faure 2014-07-05 13:33:57 +01:00
Yann Collet
fd51d0567b updated fuzzer, faster and cleaner overflow tests 2014-07-05 12:50:05 +01:00
Yann Collet
bdb5bcefff update to r119 2014-07-02 22:03:58 +01:00
Yann Collet
3a9427237d Stronger fuzzer tests 
Stronger fix 134
 2014-07-02 18:02:29 +01:00
Yann Collet
a3089e5b29 stronger fuzzer tests 2014-07-02 14:38:16 +01:00
Yann Collet
402c5874af New fuzzer test (issue 134) 2014-07-02 09:38:34 +01:00
Yann Collet
790db934f1 updated lz4cat man page 2014-06-24 18:59:51 +01:00
Yann Collet
da5373197e Fixed : issue 52 (reported by Ludwig Strigeus) 2014-06-22 11:25:04 +01:00
Yann Collet
8d66dd7cd5 Fixed : using loadDict() with small dictionaries 2014-06-21 17:01:15 +01:00
Yann Collet
6e1179a1e3 bugfix : streaming tiny messages from within very small ringbuffer (Takayuki's streaming example 2) 2014-06-19 22:54:16 +01:00
Yann Collet
d517d609d9 Fixed : streaming compression using small (<64KB) dictionary buffers 2014-06-17 21:41:59 +01:00
Yann Collet
2f0a717a35 LZ4 Streaming : check overlapping input/dictionary 2014-06-14 16:56:24 +01:00
Yann Collet
1c5a6304a2 CLI : can select compression level > 9 2014-06-11 22:02:46 +01:00
Yann Collet
0c62103105 restored LZ4 HC streaming mode 2014-06-11 21:40:16 +01:00
Yann Collet
661e4ddb78 lz4io : reduced memory usage in streaming mode 2014-06-10 06:10:08 +01:00
Yann Collet
135f11b54e Obsolete "external allocation" functions 
(convergence towards LZ4_compress_continue() )
 2014-06-09 16:46:03 +01:00
Yann Collet
598bde9a69 converge towards LZ4_compress_continue() 2014-06-09 02:42:39 +01:00
Yann Collet
a79180f51d New : valgrind memtest 2014-06-09 01:01:04 +01:00
Yann Collet
1e053a290a new test tool : datagen 2014-06-03 23:44:49 +01:00
Yann Collet
b636779b0e unified structure model 2014-06-02 07:07:19 +01:00
Yann Collet
4b43c2bd98 Continuous streaming mode (automatic) 2014-05-21 19:46:36 +01:00
Yann Collet
914f8a9a2c Added : fuzzer code for LZ4_compress_limitedOutput_usingDict() 2014-05-20 23:36:27 +01:00
Yann Collet
8c38ddd7e6 Introduce : LZ4_compress_limitedOutput_usingDict() 2014-05-20 23:26:03 +01:00
Yann Collet
9ac680972a restored Travis Build Matrix 2014-05-20 19:14:53 +01:00
Yann Collet
7f92b09174 Fixed : compilation errors using Visual 2012 2014-05-20 01:53:48 +01:00
Yann Collet
4db6b03fce First version of Block Streaming API : LZ4_compress_usingDict() 2014-05-20 00:40:29 +01:00
Yann Collet
7bcb3b2e9f changed naming convention to *_usingDict() 2014-05-04 13:26:05 +01:00
Yann Collet
200d87c684 Added : *_withDict to fullbench 2014-05-03 19:56:08 +01:00
Takayuki MATSUOKA
dca4bc3595 Add Travis CI's Build Matrix 
See also : http://docs.travis-ci.com/user/build-configuration/#The-Build-Matrix
 2014-05-01 00:29:13 +09:00
Yann Collet
42cb61367a fuzzer : added compressibility parameter (-p#) 2014-04-29 23:44:49 +01:00
Yann Collet
c75f73a2c0 fixed : allocation within fuzzer 
new command line options for fuzzer (see -h)
 2014-04-29 21:39:08 +01:00
Yann Collet
ff0db1dfb9 modified : rand generator 2014-04-28 23:49:31 +01:00
Yann Collet
b4e85853c6 fixed : read error into fuzzer 2014-04-28 23:17:49 +01:00
Yann Collet
03b9c5fcae Introduce "External Dictionary" de/compression API 2014-04-28 21:45:35 +01:00
Yann Collet
c4a99c817a fuzzer title shows 32/64 bits mode 2014-04-24 10:12:58 +01:00
Yann Collet
f6beaa779d re-order travis tests 
fix : remove man pages on uninstall
 2014-04-23 00:54:32 +01:00
Takayuki MATSUOKA
a41790a759 Add build target 'test' 2014-04-16 23:26:15 +09:00
Takayuki MATSUOKA
0390c7dd9d Add command-line switch to support Travis-CI 
Add command-line switch '--no-prompt' to fullbench and fuzzer.
'--no-prompt' enables :

 - Throtting stdout to satisfy Travis-CI's stdout limit (10MB).
 - Prevent Travis-CI's build timeouts and ignore pause by getchar().
   http://docs.travis-ci.com/user/build-configuration/#Build-Timeouts
 2014-04-16 23:25:56 +09:00
Yann Collet
5b91330ce2 Added lz4c & lz4cat man pages 2014-04-15 16:47:48 +02:00
Yann Collet
374d6ac35c Fixed issue 127 & 128 2014-04-15 15:03:17 +02:00
yann.collet.73@gmail.com
37be46701a quickfix (issue 124 & 125) 
git-svn-id: https://lz4.googlecode.com/svn/trunk@116 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-03-24 20:59:20 +00:00
yann.collet.73@gmail.com
848fdf3c15 Added : lz4cat utility for posx systems 
OS-X compatible dynamic library, thanks to Jan Stepien
corrected : relative symbolic link for versioned library

git-svn-id: https://lz4.googlecode.com/svn/trunk@115 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-03-23 11:15:37 +00:00
yann.collet.73@gmail.com
a9d8640c1f Makefile : library correctly compiled with -O3 switch (issue 114) 
Makefile : library compilation compatible with clang
Makefile : library is versioned and linked (issue 119)
lz4.h : no more static inline prototypes (issue 116)
man : improved header/footer (issue 111)
Makefile : Use system default $(CC) & $(MAKE) variables (issue 112)
xxhash : updated to r34

git-svn-id: https://lz4.googlecode.com/svn/trunk@114 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-03-12 14:51:59 +00:00
yann.collet.73@gmail.com
69dc85b8ab Large decompression speed improvement for GCC 32-bits. Thanks to Valery Croizier ! 
LZ4HC : Compression Level is now a programmable parameter (CLI from 4 to 9)
Separated IO routines from command line (lz4io.c)
Version number into lz4.h (suggested by Francesc Alted)


git-svn-id: https://lz4.googlecode.com/svn/trunk@113 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-02-04 14:11:10 +00:00
yann.collet.73@gmail.com
96a60a52eb quickfix (issue 107 & issue 103) 
git-svn-id: https://lz4.googlecode.com/svn/trunk@112 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-01-08 18:50:11 +00:00
yann.collet.73@gmail.com
648678788c Makefile : added capability to install libraries 
Modified Directory tree, to better separate libraries from programs.


git-svn-id: https://lz4.googlecode.com/svn/trunk@111 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
 2014-01-07 18:47:50 +00:00