AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,955 Commits 1 Branch 0 Tags 27 MiB
b9693d3a49
Commit Graph

445 Commits

Author SHA1 Message Date
Yann Collet
626040ab53 changed PREFETCH() macro into PREFETCH_L2() 
which is more accurate
 2018-11-12 17:05:32 -08:00
Yann Collet
1b4a9c518b
Merge pull request #1410 from facebook/prefetch_dec 
improve long-range decoder speed
 2018-11-08 18:41:58 -08:00
Yann Collet
9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Nick Terrell
a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Yann Collet
acd75a1448 fixed a second memset() on NULL 
not sure why it only triggers now,
this code has been around for a while.

Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
 2018-10-29 15:03:57 -07:00
Yann Collet
2b4914082e created zstd_decompress_block module 
isolate all logic associated with block decompression
into its own module.

zstd_decompress is still in charge
of context creation/destruction,
frames, headers, streaming, special blocks, etc.

Compressed blocks themselves are now handled within zstd_decompress_block .
 2018-10-25 16:28:41 -07:00
Yann Collet
ccd2d426fc separate DDict logic into its own module 
created zstd_ddict.c within lib/decompress
 2018-10-23 17:25:49 -07:00
Ori Livneh
f31715f5e0 Enable use of bswap intrinsics in clang 
Necessary because clang disguises itself as an older (__GNUC_MINOR__ = 2) GCC.
 2018-10-11 15:01:09 -04:00
Yann Collet
6ed3b526e4 restored bitMask for shift values 
since corrupted bitstreams can generate too large values.

This slightly reduces the benefits from clang on my laptop.
gcc results and code generation are not affected.
 2018-10-10 18:29:50 -07:00
Yann Collet
c012e9540a removed one assert() 
that can be triggered by a corrupted bitstream.
 2018-10-10 17:33:04 -07:00
Yann Collet
7791f192ee removed one assert() 
which can be triggered when input is corrupted.
 2018-10-10 16:39:15 -07:00
Yann Collet
d3ec23313d improved decompression speed 
while reviewing #1364,
I found a decompression speed improvement.

On my laptop, the new code decompresses +5-6% faster on clang
and +2-3% faster on gcc.

not bad for an accidental optimization...
 2018-10-10 15:48:43 -07:00
Nick Terrell
109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Yann Collet
292d8e4a83 added some tests based on limits.h 
in order to ensure proper type mapping
when not using stdint.h
 2018-09-23 23:57:30 -07:00
Yann Collet
0e5b447aaa
Merge pull request #1316 from facebook/coldDict 
Cold dictionary mitigation
 2018-09-14 10:37:46 -07:00
Yann Collet
5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet
2618253da2 fixed PREFETCH() macro 
for corner cases and platforms without this instruction
 2018-09-12 16:15:37 -07:00
Nick Terrell
f6daddf2db Also allow x86 2018-09-12 12:05:32 -07:00
Nick Terrell
1e0bac6a9c [libzstd] Fix cpu for MSFT ARM 
The `__cpuid()` and `__cpuidex()` intrinsics are only available
on x86 and x86_64.
 2018-09-12 10:35:16 -07:00
Yann Collet
4de344d505 added conditional prefetch 
depending on amount of work to do.
 2018-09-12 10:29:47 -07:00
Yann Collet
63a519dbf6 implemented first prefetch 
based on dictID.
dictContent is prefetched up to 32 KB
(no contentSize adaptation)
 2018-09-11 17:23:44 -07:00
Nick Terrell
5e580de6da [zstd] Fix seqStore growth 
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
 2018-08-28 13:24:44 -07:00
Nick Terrell
924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
Yann Collet
6e66bbf5dd fixed several minor issues detected by scan-build 
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
 2018-08-14 16:55:35 -07:00
Yann Collet
bbd78df59b add build macro NO_PREFETCH 
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
 2018-07-06 17:06:04 -07:00
Yann Collet
121aa2c388
Merge pull request #1211 from facebook/staticAssert 
updated DEBUG_STATIC_ASSERT()
 2018-06-27 12:19:17 -07:00
Yann Collet
ff773bfcde zeroise freq table with memset() 
improves decoding speed by ~5% in github_users sample set
 2018-06-26 17:24:41 -07:00
Yann Collet
7b9bbf77c9 switched to a sizeof() version 
avoid -Werror=unused-variable issue
 2018-06-26 14:08:35 -07:00
Yann Collet
f98ec46979 updated DEBUG_STATIC_ASSERT() 
following suggestion from #1209
 2018-06-26 12:04:59 -07:00
Yann Collet
fbd5dfc1b1 changed POOL_resize() return type to int 
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
 2018-06-22 12:14:59 -07:00
Yann Collet
243cd9d8bb add a cond_broadcast after resize 
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
 2018-06-21 18:04:58 -07:00
Yann Collet
818e72b4d5 added extended POOL test 
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
 2018-06-21 14:58:59 -07:00
Yann Collet
6de249c1c6 fixed: bug when counting nb of active threads 
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
 2018-06-20 18:28:49 -07:00
Yann Collet
6b48eb12c0 change control of threadLimit 
now limits maximum nb of active threads
even when queueSize > 1.
 2018-06-20 14:35:39 -07:00
Yann Collet
62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet
166901dc72 reduced POOL_resize() restriction 
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
 2018-06-19 18:07:18 -07:00
Yann Collet
4567c57199 finalized POOL_resize() 
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
 2018-06-19 16:03:12 -07:00
Yann Collet
1c714fda3f introduced POOL_resize() 
not complete yet :
finalize behavior in case of unfinished expansion
 2018-06-18 20:46:39 -07:00
Yann Collet
d8462ecba2 Merge branch 'dev' into huf_rename 2018-06-14 20:42:10 -04:00
Yann Collet
9698d2fb72
Merge pull request #1189 from facebook/hist 
histogram module
 2018-06-14 20:39:52 -04:00
Yann Collet
1adf84ccb7 renamed all HUF_decompress*X4*() functions into *X2 
to underline they generate up to 2 symbols per decoding,
in preparation for a future *X3 variant.
 2018-06-14 15:17:03 -04:00
Yann Collet
a09af5eb6b renamed all HUF_decompress*X2*() functions into *X1 
to underline they generate one symbol per decoding operation.

The new naming scheme will make it easier to introduce an *X3 variant.
 2018-06-14 15:08:43 -04:00
Yann Collet
fc682263d0 fixed g_debuglevel variable name 
in debug.h
 2018-06-13 20:02:33 -04:00
Yann Collet
2d76defbfe grouped all histogram functions into hist.c 
renamed functions with HIST_* prefix
 2018-06-13 19:49:31 -04:00
Yann Collet
fa41bcc2c2 grouped debug functions into debug.h 
There were 2 competing set of debug functions
within zstd_internal.h and bitstream.h.
They were mostly duplicate, and required care to avoid messing with each other.

There is now a single implementation, shared by both.

Significant change :
The macro variable ZSTD_DEBUG does no longer exist,
it has been replaced by DEBUGLEVEL,
which required modifying several source files.
 2018-06-13 15:43:09 -04:00
Yann Collet
463a0fe38b simplified optimal parser 
removed "cached" structure.
prices are now saved in the optimal table.

Primarily done for simplification.
Might improve speed by a little.
But actually, and surprisingly, also improves ratio in some circumstances.
 2018-05-29 14:07:25 -07:00
Yann Collet
b5ef32fea7 Merge branch 'dev' into fracFse 2018-05-24 14:09:49 -07:00
Yann Collet
776128d16f fix corner case when requiring cost of an FSE symbol 
ensure that, when frequency[symbol]==0,
result is (tableLog + 1) bits
with both upper-bit and fractional-bit estimates.

Also : enable BIT_DEBUG in /tests
 2018-05-24 13:59:11 -07:00
Nick Terrell
f2d0924b87 Variable declarations 2018-05-23 14:58:58 -07:00
Nick Terrell
c92dd11940 Error if reported size is too large in edge case 2018-05-23 14:47:20 -07:00