AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,480 Commits 1 Branch 0 Tags 27 MiB
5808027abf
Commit Graph

2447 Commits

Author SHA1 Message Date
Yann Collet
5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Yann Collet
5892dd5da4
Merge pull request #1255 from terrelln/norm-fix 
[FSE] Fix division by zero
 2018-08-02 11:48:56 -07:00
Nick Terrell
dc5a67cb7b Disallow tableLog == srcLog 2018-08-02 11:12:17 -07:00
Jennifer Liu
f5228f2c44 Refactoring 2018-07-31 13:58:54 -07:00
Jennifer Liu
4e29bc2469 Use CDict instead of CCtx in analyzeEntropy 2018-07-31 10:36:45 -07:00
cyan4973
3f535007e4 fix %zu support under minGW 
and relevant test on Appveyor
 2018-07-30 16:56:18 +02:00
cyan4973
aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
Nick Terrell
9889bca530 [FSE] Fix division by zero 
When the primary normalization method fails, and
`(1 << tableLog) == (maxSymbolValue + 1)`, and every symbol gets assigned
normalized weight 1 or -1 in the first loop, then the next division can
raise `SIGFPE`.
 2018-07-27 17:30:03 -07:00
Yann Collet
6e490a2f09
Merge pull request #1237 from terrelln/init-cstream-adv 
Set requestedParams in ZSTD_initCStream*()
 2018-07-18 16:33:30 +02:00
cyan4973
9597b438e9 fix #1241 
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
 2018-07-17 18:52:57 +02:00
cyan4973
53e1f0504e zstdmt debug traces compatibles with mingw 
since mingw does not have `sys/times.h`,
remove this path when detecting mingw compilation.
 2018-07-17 14:39:44 +02:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev 
Split samples when building dictionary for COVER
 2018-07-13 13:26:15 -07:00
Nick Terrell
6d222c437c Set requestedParams in ZSTD_initCStream*() 
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
 2018-07-12 18:35:55 -07:00
Jennifer Liu
612b346ed5 Add explanation for split=100 2018-07-11 15:50:28 -07:00
Jennifer Liu
5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu
456f290e31 Change back to splitPoint<=0 2018-07-09 13:53:25 -07:00
Jennifer Liu
7efabb2cf6 Only make 0.0 default splitPoint 2018-07-09 12:26:53 -07:00
Yann Collet
bbd78df59b add build macro NO_PREFETCH 
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
 2018-07-06 17:06:04 -07:00
Jennifer Liu
015a00af0f Change cover_sum back to 2 parameters and fix splitPoint issues 2018-07-06 14:24:18 -07:00
Jennifer Liu
0bbff01211 Fix testing parameter 2018-07-05 22:40:32 -07:00
Jennifer Liu
a085d1aae1 Allow splitPoint==1.0 (using all samples for both training and testing) 2018-07-05 10:38:45 -07:00
Jennifer Liu
0881184c89 Some edits based on pull request comments 2018-07-03 17:53:27 -07:00
Jennifer Liu
16e75e8804 Update minimal training sample size 2018-07-03 12:07:06 -07:00
Jennifer Liu
348e5f77a9 Add split=# to cli 2018-06-29 17:54:41 -07:00
Jennifer Liu
52fbbbcb6b Explicitly cast double to unsigned 2018-06-29 16:17:20 -07:00
Jennifer Liu
f9d19b83fb Fix variable declaration problem 2018-06-29 15:46:56 -07:00
Jennifer Liu
e061d84016 Another fix to comparator 2018-06-29 15:38:08 -07:00
Jennifer Liu
59797d3328 Fix splitPoint floating point comparison problem 2018-06-29 12:47:03 -07:00
Jennifer Liu
0ef06f2e8a Split samples into train and test sets 2018-06-29 12:33:34 -07:00
Yann Collet
121aa2c388
Merge pull request #1211 from facebook/staticAssert 
updated DEBUG_STATIC_ASSERT()
 2018-06-27 12:19:17 -07:00
Yann Collet
4489daec09 slightly adjusted default-distribution threshold 
depending on strategy.
fast favors faster compression and decompression speeds.
 2018-06-26 20:10:45 -07:00
Yann Collet
ff773bfcde zeroise freq table with memset() 
improves decoding speed by ~5% in github_users sample set
 2018-06-26 17:24:41 -07:00
Yann Collet
7b9bbf77c9 switched to a sizeof() version 
avoid -Werror=unused-variable issue
 2018-06-26 14:08:35 -07:00
Yann Collet
f98ec46979 updated DEBUG_STATIC_ASSERT() 
following suggestion from #1209
 2018-06-26 12:04:59 -07:00
Nick Terrell
b426bcc097
[zstdmt] Fix jobsize bugs (#1205) 
[zstdmt] Fix jobsize bugs

* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
  Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
 2018-06-25 15:21:08 -07:00
Yann Collet
3b53bfe4f3
Merge pull request #1200 from felixhandte/zstd-attach-dict-pref 
Add CCtx Param Controlling Dict Attachment Behavior
 2018-06-25 12:42:31 -07:00
Yann Collet
31769ce702 error on no forward progress 
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.

In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.

This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.

The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
 2018-06-22 17:58:21 -07:00
Yann Collet
3934e010a2
Merge pull request #1197 from facebook/poolResize 
Thread Pool resize
 2018-06-22 14:20:07 -07:00
Yann Collet
fbd5dfc1b1 changed POOL_resize() return type to int 
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
 2018-06-22 12:14:59 -07:00
Yann Collet
1d5648ca10
Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict 
ZSTD_btopt: Support Searching the Dictionary Context In-Place
 2018-06-22 11:53:23 -07:00
Yann Collet
f6242d30b7
Merge pull request #1202 from facebook/barelyCompressible 
Increase threshold detection of poorly compressible data
 2018-06-22 11:52:52 -07:00
Yann Collet
698fd00afb huf: increase threshold detection of poorly compressible data 2018-06-21 18:32:38 -07:00
Yann Collet
243cd9d8bb add a cond_broadcast after resize 
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
 2018-06-21 18:04:58 -07:00
Yann Collet
818e72b4d5 added extended POOL test 
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
 2018-06-21 14:58:59 -07:00
W. Felix Handte
01bb1c1016 Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00
W. Felix Handte
3e91dc4d6a Add Repcode Bounds Check 2018-06-21 15:54:41 -04:00
W. Felix Handte
5bd3d4b7d2 Add Debug Log Statement 2018-06-21 15:54:07 -04:00
W. Felix Handte
3caba150c6 Fix dmsBtLow Test 2018-06-21 15:53:40 -04:00
W. Felix Handte
5da9bbc38e Conceivably Dedup ZSTD_noDict and ZSTD_dictMatchState _insertBt1 Impls 
By reverting to the bool extDict flag, we call ZSTD_insertBt1 with the same
const args in both non-extDict dictModes.
 2018-06-21 11:20:01 -04:00
Yann Collet
6de249c1c6 fixed: bug when counting nb of active threads 
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
 2018-06-20 18:28:49 -07:00