zstd/lib/compress
Yann Collet 5c2f2ebfdb zstdmt via compress_generic: reduce opportunity to free/create mtctx
`zstreamtest --newapi` (and `--opaqueapi`) create and destroy way too many threads
resulting in failure of tsan tests,
and potentially connected to the qemu flaky tests.

This is because, at each test, the nb of threads can be changed (random).

The `--no-big-tests` directive reduce this choice to 1/2 threads,
in order to limit memory usage, especially for qemu and 32-bits builds.
Unfortunately, swapping between 1 and 2 threads is enough to constantly create/destroy new mtctx.

This patch takes advantage of the following property :
via compress_generic, no internal mtctx is needed for nbThreads < 2.
As a consequence, when nbThreads == 2, the currently active mtctx is necessarily good.

This dramatically reduces the nb of thread creations when invoking `zstreamtest --newapi --no-big-tests`
(only when parent cctx itself is created, which is randomized to 1/256 tests).

Expected outcome :
- at a minimum : tsan tests shall now work continuously without exploding the thread counter
- at best : flaky qemu tests on `zstreamtest --newapi --no-big-tests` may stop being flaky, due to less stress from constant thread creation/destruction

Real world impact :
minimal, I don't expect users to constantly change `nbThreads` between each invocation.
If `nbThreads` remains stable, existing implementation re-uses existing mtctx.

Also : `zstreamtest --newapi` but without `--no-big-tests` doesn't benefit as much,
since this test can select a random `nbThreads` value between 1 and 4.
The current patch only reduces opportunity to free/create mtctx (for example : 2->1->2 doesn't need a new mtctx)
but doesn't completely eliminate it, since `nbThreads` can still change between 2/3/4.
A more complete solution could be to only use 2 out of 4 allocated threads, thus keeping the pool at a constant size.
This would require a larger change to `POOL_*` api though.
2017-12-16 12:48:13 -08:00
..
fse_compress.c [zstd] Backport kernel patch from @ColinIanKing 2017-09-25 16:18:23 -07:00
huf_compress.c Ensure dictionary Huff table can encode any symbol 2017-10-03 13:22:13 -07:00
zstd_compress_internal.h fix #944 : ZSTDMT with large files and dictionary now works correctly 2017-12-12 18:04:58 -08:00
zstd_compress.c zstdmt via compress_generic: reduce opportunity to free/create mtctx 2017-12-16 12:48:13 -08:00
zstd_double_fast.c renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_double_fast.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_fast.c renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_fast.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_lazy.c strategy ZSTD_btopt pinned to (0) variant (faster one) 2017-11-20 11:53:20 -08:00
zstd_lazy.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_ldm.c Increase maximum window size 2017-09-26 14:00:01 -07:00
zstd_ldm.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstd_opt.c zstd_opt: added comments, as requested by @terrelln 2017-11-29 15:19:00 -08:00
zstd_opt.h renamed zstd_compress.h into zstd_compress_internal.h 2017-11-07 16:15:23 -08:00
zstdmt_compress.c zstdmt via compress_generic: reduce opportunity to free/create mtctx 2017-12-16 12:48:13 -08:00
zstdmt_compress.h zstdmt via compress_generic: reduce opportunity to free/create mtctx 2017-12-16 12:48:13 -08:00