zstd/lib/compress
Yann Collet 281f06e01f saves 3-bytes on small input with streaming API
zstd streaming API was adding a null-block at end of frame for small input.

Reason is : on small input, a single block is enough.
ZSTD_CStream would size its input buffer to expect a single block of this size,
automatically triggering a flush on reaching this size.

Unfortunately, that last byte was generally received before the "end" directive (at least in `fileio`).
The later "end" directive would force the creation of a 3-bytes last block to indicate end of frame.

The solution is to not flush automatically, which is btw the expected behavior.
It happens in this case because blocksize is defined with exactly the same size as input.
Just adding one-byte is enough to stop triggering the automatic flush.

I initially looked at another solution, solving the problem directly in the compression context.
But it felt awkward.
Now, the underlying compression API `ZSTD_compressContinue()` would take the decision the close a frame
on reaching its expected end (`pledgedSrcSize`).
This feels awkward, a responsability over-reach, beyond the definition of this API.
ZSTD_compressContinue() is clearly documented as a guaranteed flush,
with ZSTD_compressEnd() generating a guaranteed end.

I faced similar issue when trying to port a similar mechanism at the higher streaming layer.
Having ZSTD_CStream end a frame automatically on reaching `pledgedSrcSize` can surprise the caller,
since it did not explicitly requested an end of frame.
The only sensible action remaining after that is to end the frame with no additional input.
This adds additional logic in the ZSTD_CStream state to check this condition.
Plus some potential confusion on the meaning of ZSTD_endStream() with no additional input (ending confirmation ? new 0-size frame ?)

In the end, just enlarging input buffer by 1 byte feels the least intrusive change.
It's also a contract remaining inside the streaming layer, so the logic is contained in this part of the code.

The patch also introduces a new test checking that size of small frame is as expected, without additional 3-bytes null block.
2017-12-14 11:47:02 -08:00
..
fse_compress.c [zstd] Backport kernel patch from @ColinIanKing 2017-09-25 16:18:23 -07:00
huf_compress.c Ensure dictionary Huff table can encode any symbol 2017-10-03 13:22:13 -07:00
zstd_compress_internal.h fix #944 : ZSTDMT with large files and dictionary now works correctly 2017-12-12 18:04:58 -08:00
zstd_compress.c saves 3-bytes on small input with streaming API 2017-12-14 11:47:02 -08:00
zstd_double_fast.c renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_double_fast.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_fast.c renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_fast.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_lazy.c strategy ZSTD_btopt pinned to (0) variant (faster one) 2017-11-20 11:53:20 -08:00
zstd_lazy.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_ldm.c Increase maximum window size 2017-09-26 14:00:01 -07:00
zstd_ldm.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_opt.c zstd_opt: added comments, as requested by @terrelln 2017-11-29 15:19:00 -08:00
zstd_opt.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstdmt_compress.c fix #944 : ZSTDMT with large files and dictionary now works correctly 2017-12-12 18:04:58 -08:00
zstdmt_compress.h fix #944 : ZSTDMT with large files and dictionary now works correctly 2017-12-12 18:04:58 -08:00