Commit Graph

457 Commits

Author SHA1 Message Date
Yann Collet
ededcfca57 fix confusion between unsigned <-> U32
as suggested in #1441.

generally U32 and unsigned are the same thing,
except when they are not ...

case : 32-bit compilation for MIPS (uint32_t == unsigned long)

A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).

Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
  but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
  but the caller user a pointer to U32 instead.

These fixes are useful.

However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.

As a consequence, the amount of code changed is larger than I would expect for such a patch.

Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.

So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.

I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.

Thoughts ?
2018-12-21 18:09:41 -08:00
Yann Collet
e4ae24c229
Merge pull request #1420 from felixhandte/zstd-decompress-minimal
Various Macros to Allow Building Extremely Minimal Decoder Library
2018-12-20 15:17:37 -08:00
W. Felix Handte
8e61ac8161 Use Unused Variable in ERR_getErrorString() 2018-12-19 12:36:10 -08:00
W. Felix Handte
c560e34c86 Add HUF_FORCE_DECOMPRESS_X2 2018-12-18 13:36:39 -08:00
W. Felix Handte
432314b58a Rename HUF_DECOMPRESS_MINIMAL -> HUF_FORCE_DECOMPRESS_X1 2018-12-18 13:36:39 -08:00
W. Felix Handte
605dd576ee Remove Error Strings with ZSTD_STRIP_ERROR_STRINGS 2018-12-18 13:36:39 -08:00
W. Felix Handte
9d5f3963ff Add Option to Not Request Inlining with ZSTD_NO_INLINE 2018-12-18 13:36:39 -08:00
W. Felix Handte
f45c9df42e Totally Hide/Disable X2 Variants when HUF_DECOMPRESS_MINIMAL is Defined 2018-12-18 13:36:39 -08:00
Yann Collet
373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Yann Collet
9c3265a53f
Merge pull request #1417 from facebook/advancedAPI
Advanced API
2018-12-10 18:48:15 -08:00
Ryan Schmidt
ef4df0df4a Fix i386 build failure "Junk character 13" 2018-11-16 02:16:21 -06:00
Yann Collet
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
2018-11-13 18:12:34 -08:00
Yann Collet
626040ab53 changed PREFETCH() macro into PREFETCH_L2()
which is more accurate
2018-11-12 17:05:32 -08:00
Yann Collet
1b4a9c518b
Merge pull request #1410 from facebook/prefetch_dec
improve long-range decoder speed
2018-11-08 18:41:58 -08:00
Yann Collet
9126da5b5c improve long-range decoder speed
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
2018-11-08 12:47:46 -08:00
Nick Terrell
a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Yann Collet
acd75a1448 fixed a second memset() on NULL
not sure why it only triggers now,
this code has been around for a while.

Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
2018-10-29 15:03:57 -07:00
Yann Collet
2b4914082e created zstd_decompress_block module
isolate all logic associated with block decompression
into its own module.

zstd_decompress is still in charge
of context creation/destruction,
frames, headers, streaming, special blocks, etc.

Compressed blocks themselves are now handled within zstd_decompress_block .
2018-10-25 16:28:41 -07:00
Yann Collet
ccd2d426fc separate DDict logic into its own module
created zstd_ddict.c within lib/decompress
2018-10-23 17:25:49 -07:00
Ori Livneh
f31715f5e0 Enable use of bswap intrinsics in clang
Necessary because clang disguises itself as an older (__GNUC_MINOR__ = 2) GCC.
2018-10-11 15:01:09 -04:00
Yann Collet
6ed3b526e4 restored bitMask for shift values
since corrupted bitstreams can generate too large values.

This slightly reduces the benefits from clang on my laptop.
gcc results and code generation are not affected.
2018-10-10 18:29:50 -07:00
Yann Collet
c012e9540a removed one assert()
that can be triggered by a corrupted bitstream.
2018-10-10 17:33:04 -07:00
Yann Collet
7791f192ee removed one assert()
which can be triggered when input is corrupted.
2018-10-10 16:39:15 -07:00
Yann Collet
d3ec23313d improved decompression speed
while reviewing #1364,
I found a decompression speed improvement.

On my laptop, the new code decompresses +5-6% faster on clang
and +2-3% faster on gcc.

not bad for an accidental optimization...
2018-10-10 15:48:43 -07:00
Nick Terrell
109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Yann Collet
292d8e4a83 added some tests based on limits.h
in order to ensure proper type mapping
when not using stdint.h
2018-09-23 23:57:30 -07:00
Yann Collet
0e5b447aaa
Merge pull request #1316 from facebook/coldDict
Cold dictionary mitigation
2018-09-14 10:37:46 -07:00
Yann Collet
5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet
2618253da2 fixed PREFETCH() macro
for corner cases and platforms without this instruction
2018-09-12 16:15:37 -07:00
Nick Terrell
f6daddf2db Also allow x86 2018-09-12 12:05:32 -07:00
Nick Terrell
1e0bac6a9c [libzstd] Fix cpu for MSFT ARM
The `__cpuid()` and `__cpuidex()` intrinsics are only available
on x86 and x86_64.
2018-09-12 10:35:16 -07:00
Yann Collet
4de344d505 added conditional prefetch
depending on amount of work to do.
2018-09-12 10:29:47 -07:00
Yann Collet
63a519dbf6 implemented first prefetch
based on dictID.
dictContent is prefetched up to 32 KB
(no contentSize adaptation)
2018-09-11 17:23:44 -07:00
Nick Terrell
5e580de6da [zstd] Fix seqStore growth
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
Nick Terrell
924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
Yann Collet
6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
Yann Collet
bbd78df59b add build macro NO_PREFETCH
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
2018-07-06 17:06:04 -07:00
Yann Collet
121aa2c388
Merge pull request #1211 from facebook/staticAssert
updated DEBUG_STATIC_ASSERT()
2018-06-27 12:19:17 -07:00
Yann Collet
ff773bfcde zeroise freq table with memset()
improves decoding speed by ~5% in github_users sample set
2018-06-26 17:24:41 -07:00
Yann Collet
7b9bbf77c9 switched to a sizeof() version
avoid -Werror=unused-variable issue
2018-06-26 14:08:35 -07:00
Yann Collet
f98ec46979 updated DEBUG_STATIC_ASSERT()
following suggestion from #1209
2018-06-26 12:04:59 -07:00
Yann Collet
fbd5dfc1b1 changed POOL_resize() return type to int
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
Yann Collet
243cd9d8bb add a cond_broadcast after resize
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
Yann Collet
818e72b4d5 added extended POOL test
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
Yann Collet
6de249c1c6 fixed: bug when counting nb of active threads
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
2018-06-20 18:28:49 -07:00
Yann Collet
6b48eb12c0 change control of threadLimit
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
Yann Collet
62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet
166901dc72 reduced POOL_resize() restriction
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
Yann Collet
4567c57199 finalized POOL_resize()
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
Yann Collet
1c714fda3f introduced POOL_resize()
not complete yet :
finalize behavior in case of unfinished expansion
2018-06-18 20:46:39 -07:00