AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
1,792 Commits 13 Branches 31 Tags 6.8 MiB
bf6fd938e5
Commit Graph

660 Commits

Author SHA1 Message Date
Yann Collet
bf6fd938e5
Merge pull request #537 from lz4/xpHCmf2 
Speed optimization for optimal parser
 2018-05-07 13:06:43 -07:00
Yann Collet
ba1c7148a5 renamed variable for clarity 2018-05-07 12:14:26 -07:00
Yann Collet
abb1f70e17
Merge pull request #538 from lz4/frameTestError 
Fix frametest error
 2018-05-07 11:33:53 -07:00
Yann Collet
200b2960d5 fixed minor conversion warning 2018-05-06 18:26:14 -07:00
Yann Collet
24b9c485db small PA optimization 
which measurably improves speed
on levels 9+
 2018-05-06 16:53:33 -07:00
Yann Collet
d7b6c726ed small extDict : fixed side-effect 
don't fix dictionaries of size 0.
setting dictEnd == source triggers prefix mode,
thus removing possibility to use CDict.
 2018-05-05 19:59:00 -07:00
Yann Collet
af12733467 fixed frametest error 
The error can be reproduced using following command :
./frametest -v -i100000000 -s1659 -t31096808

It's actually a bug in the stream LZ4 API,
when starting a new stream
and providing a first chunk to complete with size < MINMATCH.
In which case, the chunk becomes a dictionary.
No hash was generated and stored,
but the chunk is accessible as default position 0 points to dictStart,
and position 0 is still within MAX_DISTANCE.
Then, next attempt to read 32-bits from position 0 fails.

The issue would have been mitigated by starting from index 64 KB,
effectively eliminating position 0 as too far away.

The proper fix is to eliminate such "dictionary" as too small.
Which is what this patch does.
 2018-05-05 18:24:11 -07:00
Yann Collet
cdb0275b7f lz4hc: fixed PA / SC parameter order 
also :
reserved PA for levels 9+ (instead of 8+).
In most cases, speed is lower, and compression benefit is not worth.
 2018-05-05 14:32:57 -07:00
Yann Collet
a4e918d7a6 lz4hc: SC only enabled for opt parser 
the trade off is not good for regular HC parser :
compression is a little bit better, but speed cost is too large in comparison.
 2018-05-05 14:25:37 -07:00
Yann Collet
d097bf93f8 fixed SC.opt integration with regular HC parser 
Only enabled when searching forward.

note : it slighly improves compression ratio,
but measurably decreases speed.
Trade-off to analyse.
 2018-05-05 13:46:45 -07:00
Yann Collet
fa89a9e18b lz4hc: fixed performance issue 
when combining both PA and CS optimizations
 2018-05-05 13:31:03 -07:00
Yann Collet
9699ba5ddf integrated chain swapper into HC match finder 
slower than expected
Pattern analyzer and Chain Swapper
work slower when both activated.
Reasons unclear.
 2018-05-04 19:13:33 -07:00
Nick Terrell
a7cc0b590a Fix make install 
* Uninstall didn't remove the pkg-config correctly.
* Fix `mandir`
* Allow overriding either upper- or lower-case location variables, but
  always use the lower case variables.
* Add test case that ensures overriding both upper- and lower-case
  variables is the same, and that the directory is empty after uninstall.
 2018-05-04 13:50:23 -07:00
Yann Collet
434ace7244 implemented search accelerator 
greatly improves speed compared to non-accelerated,
especially for slower files.

On my laptop, -b12 :
```
calgary.tar :  4.3 MB/s =>  9.0 MB/s
enwik7      : 10.2 MB/s => 13.3 MB/s
silesia.tar :  4.0 MB/s =>  8.7 MB/s
```

Note : this is the simplified version,
without handling dictionaries, external buffer, nor pattern analyzer.
Current `dev` branch on these samples gives :
```
calgary.tar :  4.2 MB/s
enwik7      :  9.7 MB/s
silesia.tar :  3.5 MB/s
```

interestingly, it's slower,
presumably due to handling of dictionaries.
 2018-05-03 16:31:41 -07:00
Yann Collet
dc42707107 created LZ4HC_FindLongestMatch() 
simplified match finder
only searching forward and within current buffer,
for easier testing of optimizations.
 2018-05-03 15:38:32 -07:00
Yann Collet
f3e84ffd41
Merge pull request #529 from felixhandte/lz4f-fast-reset-for-streaming-only 
LZ4F: Only Reset the LZ4_stream_t when Init'ing a Streaming Block
 2018-05-03 15:37:51 -07:00
Yann Collet
ffbff1f360 Merge branch 'dev' into lz4fRingBuffer 2018-05-03 11:54:57 -07:00
Yann Collet
95607a749b
Merge pull request #528 from lz4/complexShortcut 
Faster decoding speed
 2018-05-03 11:35:50 -07:00
Cyan4973
2e2c9f6ff3 fix comments / indentation 
as requested by @terrelln
 2018-05-03 07:56:33 -07:00
W. Felix Handte
5406c2e479 Only Reset the LZ4 Stream when Init'ing a Streaming Block 2018-05-03 00:03:20 -04:00
Yann Collet
c25eb16666 random lz4f clarifications 
the initial intention was to update lz4f ring buffer strategy,
but lz4f doesn't use ring buffer.
Instead, it uses the destination buffer as much as possible,
and merely copies just what's required to preserve history
into its own buffer, at the end.
Pretty efficient.

This patch just clarifies a few comments and add some assert().
It's built on top of #528.

It also updates doc.
 2018-05-02 16:05:42 -07:00
Yann Collet
858d12e3e1 Merge branch 'dev' into lz4fRingBuffer 2018-05-02 14:24:24 -07:00
Yann Collet
85be6b8f6d increased nbAttempts for lz4 -12 
shaves one more kilobyte from silesia.tar
 2018-05-02 14:22:35 -07:00
Yann Collet
93cf628a08 introduce LZ4_decoderRingBufferSize() 
fuzzer : fix and robustify ring buffer tests
 2018-05-02 13:01:04 -07:00
Yann Collet
1a191b3f8d simplify shortcut 2018-05-02 10:33:12 -07:00
Yann Collet
0114b63b40 Merge branch 'dev' into complexShortcut 2018-05-02 10:08:30 -07:00
Yann Collet
bd470ccd38
Merge pull request #521 from lz4/BD_deterministic 
fix lz4hc -BD non-determinism
 2018-04-30 20:40:34 -07:00
Cyan4973
6a7d501fed renamed variable for clarity 
lowLimit -> lowestMatchIndex
 2018-04-30 18:56:16 -07:00
Yann Collet
8c574990a9 lz4hc changed variable 
to reduce confusion
dictLowLimit => dictStart
 2018-04-30 16:08:16 -07:00
Yann Collet
4c696613a0 clarified streaming decompression function 
restrictions for ring buffer
 2018-04-30 15:55:33 -07:00
Yann Collet
90374271c2
Merge pull request #527 from svpv/fastDec 
lz4.c: two-stage shortcut for LZ4_decompress_generic
 2018-04-30 15:32:37 -07:00
Yann Collet
c32e0319a5
Merge pull request #523 from svpv/makeV1 
lib/Makefile: show commands with V=1
 2018-04-29 08:56:44 -07:00
Yann Collet
41ad238bf9
Merge pull request #515 from svpv/refactorDec 
lz4.c: refactor the decoding routines
 2018-04-29 07:41:35 -07:00
Alexey Tourbin
45f8603aae lz4.c: two-stage shortcut for LZ4_decompress_generic 2018-04-28 11:16:57 +03:00
Alexey Tourbin
69242a8a08 lib/Makefile: show commands with V=1 
`make V=1` will now show the commands executed to build the library.
A similar technique is used in e.g. linux/Makefile.

The bulk of this change is produced with the following vim command:

    :g!/^\t@echo\>/s/^\t@/\t\$(Q)/
 2018-04-28 07:22:26 +03:00
Yann Collet
1e6ca25af3
Merge pull request #520 from felixhandte/frame-dict-nits 
Minor Fixes to Dictionary Preparation in LZ4 Frame
 2018-04-27 13:52:30 -07:00
Yann Collet
de7b274d99 Merge branch 'dev' into BD_deterministic 2018-04-27 12:59:20 -07:00
Yann Collet
19b1267d44 fix lz4hc -BD non-determinism 
related to chain table update
 2018-04-27 12:46:49 -07:00
Yann Collet
72e99c8939 lz4hc : minor editions for clarity 2018-04-27 12:28:58 -07:00
Yann Collet
47d70e755e
Merge pull request #519 from lz4/fdParser 
Faster decoding speed
 2018-04-27 11:46:29 -07:00
W. Felix Handte
fefc40fc0a Avoid Possibly Redundant Table Clears When Loading HC Dict 2018-04-27 14:10:27 -04:00
W. Felix Handte
5076aa3e35 Remove Redundant LZ4_resetStream() Call 2018-04-27 13:59:02 -04:00
W. Felix Handte
7d11e34413 Rename LZ4F_applyCDict() -> LZ4F_initStream() 2018-04-27 13:57:10 -04:00
Yann Collet
d294dd7fc6 ensure favorDecSpeed is properly initialized 
also :
- fix a potential malloc error
- proper use of ALLOC macro inside lz4hc
- update html API doc
 2018-04-27 09:04:09 -07:00
Alexey Tourbin
d81a434c3d lz4.c: fixed the LZ4_decompress_fast_continue case 
The change is very similar to that of the LZ4_decompress_safe_continue
case.  The only reason a make this a separate change is to ensure that
the fuzzer, after it's been enhanced, can detect the flaw in
LZ4_decompress_fast_continue, and that the change indeed fixes the flaw.
 2018-04-27 15:10:12 +03:00
Yann Collet
0fb3a3b199 fixed a number of minor cast warnings 2018-04-26 18:08:28 -07:00
Yann Collet
00909b27b1
Merge pull request #518 from felixhandte/fix-517-dict-size-truncation 
Limit Dictionary Size During LZ4F Decompression
 2018-04-26 16:47:50 -07:00
Yann Collet
5c7d3812d9 fasterDecSpeed can be triggered from cli with --favor-decSpeed 2018-04-26 15:49:32 -07:00
Yann Collet
3792d00168 favorDecSpeed feature can be triggered from lz4frame 
and lz4hc.
 2018-04-26 15:18:44 -07:00
W. Felix Handte
0858362f28 Merge _destSize Compress Variant into LZ4_compress_generic() 2018-04-26 18:01:08 -04:00