zstd/lib/compress
Nick Terrell 674534a700 [zstd] Fix data corruption in niche use case
* Extract the overflow correction into a helper function.
* Load the dictionary `ZSTD_CHUNKSIZE_MAX = 512 MB` bytes at a time
  and overflow correct between each chunk.

Data corruption could happen when all these conditions are true:

* You are using multithreading mode
* Your overlap size is >= 512 MB (implies window size >= 512 MB)
* You are using a strategy >= ZSTD_btlazy
* You are compressing more than 4 GB

The problem is that when loading a large dictionary we don't do
overflow correction. We can only load 512 MB at a time, and may
need to do overflow correction before each chunk.
2019-06-21 15:47:31 -07:00
..
fse_compress.c Spelling (#1582) 2019-04-12 11:18:11 -07:00
hist.c refactor HUF_compress_internal for clarity 2018-10-26 13:21:37 -07:00
hist.h refactor HUF_compress_internal for clarity 2018-10-26 13:21:37 -07:00
huf_compress.c fix confusion between unsigned <-> U32 2018-12-21 18:09:41 -08:00
zstd_compress_internal.h minor code cleaning for new index invalidation strategy 2019-05-31 16:52:37 -07:00
zstd_compress.c [zstd] Fix data corruption in niche use case 2019-06-21 15:47:31 -07:00
zstd_double_fast.c removed UINT32_MAX 2019-05-31 17:27:07 -07:00
zstd_double_fast.h Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
zstd_fast.c Made fast strategy compatible with new offset validation strategy 2019-05-31 16:34:55 -07:00
zstd_fast.h Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
zstd_lazy.c extended exact window size to greedy/lazy modes 2019-05-31 16:08:48 -07:00
zstd_lazy.h Spelling (#1582) 2019-04-12 11:18:11 -07:00
zstd_ldm.c Spelling (#1582) 2019-04-12 11:18:11 -07:00
zstd_ldm.h changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
zstd_opt.c [opt] Add asserts for corruption in ZSTD_updateTree() 2019-06-21 15:22:29 -07:00
zstd_opt.h changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 2018-12-06 15:00:52 -08:00
zstdmt_compress.c [zstdmt] Update assert to use ZSTD_WINDOWLOG_MAX 2019-06-21 15:39:33 -07:00
zstdmt_compress.h [libzstd] Require ZSTD_MULTITHREAD to create a ZSTDMT_CCtx 2019-04-15 23:04:46 -07:00