AuroraMiddleware/lz4
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

lz4_Block_format.md: clarify on short inputs and restrictions

Browse Source 
It occurred to me that the formula "The last 5 bytes are always
literals", on the list of "assumptions made by the decoder", is
remarkably ambiguous.  Suppose the decoder is presented with 5 bytes.
Are they literals?  It may seem that the decoder degenerates
to memcpy on short inputs.  But of course the answer is no,
so the formula needs some clarification.

Parsing restrictions should be explained as well, otherwise they look
like arbitrary numbers.  The 5-byte restriction has been mentioned
recently in connection with the shortcut in LZ4_decompress_generic,
so I add that.  The second restriction is left to be explained
by the author.

I also took the liberty to explain that empty inputs "are either
unrepresentable or can be represented with a null byte".  This wording
may actually have some merit: it leaves for the implementation,
as opposed to the spec, to decide whether the encoder can compress
empty inputs, and whether the decoder can produce an empty output
(which the implementation should further clarify).
This commit is contained in:
Alexey Tourbin 2018-04-25 01:40:12 +03:00
parent 62d7cdcc74
commit ff9b4cf826
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
doc/lz4_Block_format.md
View File

@ -109,15 +109,24 @@ Parsing restrictions
There are specific parsing rules to respect in order to remain compatible
with assumptions made by the decoder :
1. The last 5 bytes are always literals
1. The last 5 bytes are always literals. In other words, the last five bytes
from the uncompressed input (or all bytes, if the input has less than five
bytes) must be encoded as literals on behalf of the last sequence.
The last sequence is incomplete, and stops right after the literals.
2. The last match must start at least 12 bytes before end of block.
Consequently, a block with less than 13 bytes cannot be compressed.
These rules are in place to ensure that the decoder
will never read beyond the input buffer, nor write beyond the output buffer.
Note that the last sequence is also incomplete,
and stops right after literals.
1. To copy literals from a non-last sequence, an 8-byte copy instruction
can always be safely issued (without reading past the input), because
the literals are followed by a 2-byte offset, and the last sequence
is at least 1+5 bytes long.
2. TODO: explain the benefits of the second restriction.
Empty inputs are either unrepresentable or can be represented with a null byte,
which can be interpreted as a token without literals and without a match.
Additional notes