Commit Graph

68 Commits

Author SHA1 Message Date
Yann Collet
0b203b04f6
Merge pull request #434 from lz4/pattern
conditional pattern analysis
2018-01-06 06:58:41 +01:00
Yann Collet
7d2f30c7d1 lz4opt supports _destSize
no longer limited to level 9
2017-12-22 12:47:59 +01:00
Yann Collet
9753ac4c91 conditional pattern analysis
Pattern analysis (currently limited to long ranges of identical bytes)
is actually detrimental to performance
when `nbSearches` is low.

Reason is : `nbSearches` provides a built-in protection for these cases.
The problem with patterns is that they dramatically increase the number of candidates to visit.
But with a low nbSearches, the match finder just aborts early.

In such cases, pattern analysis adds some complexity without reducing total nb of candidates.
It actually increases compression ratio a little bit, by filtering only "good" candidates,
but at a measurable speed cost, so it's not a good trade-off.

This patch makes pattern analysis optional.
It's enabled for levels 8+ only.
2017-12-22 08:07:25 +01:00
Yann Collet
dc3ed5b6a7 added code comments 2017-11-08 17:56:20 -08:00
Yann Collet
63f6039fb3 added constant TRAILING_LITERALS
which is more explicit than its value `3`.
reported by @terrelln
2017-11-08 17:47:24 -08:00
Yann Collet
f93b595718 lz4opt: simplified match finder invocation to LZ4HC_FindLongerMatch() 2017-11-08 17:11:51 -08:00
Yann Collet
897f5e9834 removed the ip++ at the beginning of block
The first byte used to be skipped
to avoid a infinite self-comparison.
This is no longer necessary, since init() ensures that index starts at 64K.

The first byte is also useless to search when each block is independent,
but it's no longer the case when blocks are linked.

Removing the first-byte-skip saves
about 10 bytes / MB on files compressed with -BD4 (linked blocks 64Kb),
which feels correct as each MB has 16 blocks of 64KB.
2017-11-07 17:37:31 -08:00
Yann Collet
89821ac7a1 minor comment edit 2017-11-03 11:49:56 -07:00
Yann Collet
a1f4a0d983 moved ctx->end handling from parsers
responsibility better handled one layer above (LZ4HC_compress_generic())
2017-11-03 10:48:55 -07:00
Yann Collet
c9bbad53ff removed ctx->searchNum
nbSearches now transmitted directly as function parameter
easier to track and debug
2017-11-03 10:30:52 -07:00
Yann Collet
e2eca62046 LZ4_compress_HC_continue_destSize() now compatible with optimal parser
levels 11+
2017-11-03 02:03:19 -07:00
Yann Collet
3b222d2d96 removes matches[] table
saves stack space
clearer match finder interface (no more table to fill)
2017-11-03 01:37:43 -07:00
Yann Collet
320e1d51ac removed useless parameter from hash chain matchfinder
used to be present for compatibility with binary tree matchfinder
2017-11-03 01:20:30 -07:00
Yann Collet
81667a1e96 removed code and reference to binary tree match finder
reduced size of LZ4HC state
2017-11-03 01:18:12 -07:00
Yann Collet
82c1aed419 improved level 11 speed 2017-11-03 00:59:05 -07:00
Yann Collet
890c0553d0 optimized skip strategy for level 12 2017-11-03 00:15:52 -07:00
Yann Collet
a1c5343d89 more generic skip formula
improving speed
2017-11-02 18:54:18 -07:00
Yann Collet
e06cb03c11 small adaptations for intermediate level 11 2017-11-02 16:25:10 -07:00
Yann Collet
4b81885800 partial search, while preserving compression ratio
tag interesting places
2017-11-02 15:37:18 -07:00
Yann Collet
bd992f12e4 searching match leading strictly farther does not work
sometimes, it's better to re-use same match but start it later,
in order to get shorter matchlength code
2017-11-02 15:05:45 -07:00
Yann Collet
8e16eb0cd1 fixed last lost bytes in maximal mode
even gained 2 bytes on calgary.tar...
added conditional traces `g_debuglog_enable`
2017-11-02 14:53:06 -07:00
Yann Collet
0ff4df1b94 changed strategy : opt[] path is complete after each match
previous strategy would leave a few "bad choices"
on the ground they would be fixed later,
but that requires passing through each position to make the fix
and cannot give the end position of the last useful match.
2017-11-02 13:44:57 -07:00
Yann Collet
931c5c20d0 fixed minor overflow mistake in optimal parser
saving 20 bytes on calgary.tar
2017-10-30 17:47:54 -07:00
Yann Collet
ab4bd93f59 fixed minor initialization warning 2017-10-30 16:10:25 -07:00
Yann Collet
16a4337473 added hash chain with conditional length
not a success yet
2017-10-25 07:07:08 +02:00
Yann Collet
a12cdf00c3 lz4opt: added hash chain search 2017-10-20 17:04:29 -07:00
Yann Collet
fd6bd5107b switched many types to int 2017-10-20 15:33:52 -07:00
Yann Collet
d813134619 removed SET_PRICE macro 2017-10-20 13:44:49 -07:00
Yann Collet
fa064c8a8c removed one macro usage 2017-10-20 13:32:45 -07:00
Yann Collet
ee62faee08 minor refactor
reduce variable scope
remove one macro usage
2017-10-20 12:05:00 -07:00
Yann Collet
fc879fe170 lz4opt: refactor sequence reverse traversal 2017-10-20 11:32:15 -07:00
Yann Collet
c058753393 refactor variable matchnum
separate initial and iterative search
renamed nb_matches
2017-10-20 11:24:56 -07:00
Yann Collet
7bb0a617ee simplified initial cost conditions
llen integrated in opt[]
2017-10-20 11:00:10 -07:00
Yann Collet
6cec68de39 added assert 2017-10-19 16:47:25 -07:00
Yann Collet
ac2ad52257 renamed last_pos into last_match_pos 2017-10-19 16:43:36 -07:00
Yann Collet
708e2cbb60 simplified early exit when single solution 2017-10-19 16:39:40 -07:00
tcpan
f423d016cf FIX: added prefix to FORCE_INLINE to prevent redefinition error during compilation when used with other libraries that define FORCE_INLINE 2017-08-24 10:14:20 -04:00
Yann Collet
6ad3a983db fix #369
The bug would make the bt search read one byte in an invalid memory region,
and make a branch decision based on its value.
Impact was small (missed compression opportunity).
It only happens in -BD mode, with extDict-prefix overlapping matches.

The bt match search is supposed to work also in extDict mode.
In which case, the match ptr can point into Dict.
When the match was overlapping Dict<->Prefix,
match[matchLength] would end up outside of Dict, in an invalid memory area.
The correction ensures that in such a case,
match[matchLength] ends up at intended location, inside prefix.
2017-06-26 11:29:05 -07:00
Yann Collet
a8dd86d93e changed macro HEAPMODE into LZ4_HEAPMODE
This macro is susceptible to be triggered from user side
typically through compiler flag (-DLZ4_HEAPMODE=1).
In which case, it makes sense to prefix the macro
since we want to reduce potential side-effect on namespace.
2017-05-01 22:32:21 -07:00
Yann Collet
68044acfec Merge branch 'optlz4opt' of github.com:Cyan4973/lz4 into optlz4opt 2017-03-20 09:58:00 -07:00
Yann Collet
b8bc70022b minor refactor 2017-03-20 05:04:12 -07:00
Yann Collet
cea8f60df1 slight btopt speed improvement
removing a useless test
2017-03-20 02:33:02 -07:00
Yann Collet
f513020a6d slight btopt speed improvement
removing a useless test
2017-03-17 18:07:53 -07:00
Yann Collet
8bd32a17b1 made SET_PRICE macro more usable
previous version would use argument to also change target member.
Now, only values are transferred
2017-03-17 17:42:47 -07:00
Yann Collet
118b47f3d1 improved lz4opt speed (~4%) 2017-03-17 16:53:35 -07:00
Yann Collet
eff6166eb8 minor price function optimization 2017-03-17 15:51:21 -07:00
Yann Collet
cd35f0d98c LZ4_compress_HC_destSize() uses LZ4HC_compress_generic() code path
Limits compression level to 10,
to remain compatible with Hash Chain.
2017-03-16 15:10:38 -07:00
Przemyslaw Skibinski
312d88249f removed nextToUpdateBT 2016-12-28 15:38:59 +01:00
Przemyslaw Skibinski
3d5bb38977 clean logging 2016-12-28 15:37:12 +01:00
Przemyslaw Skibinski
1c80b9af4e LZ4HC_getSearchNum 2016-12-28 15:18:19 +01:00