Commit Graph

57 Commits

Author SHA1 Message Date
Carl Woffenden
901ea61f83 Tweaks to create a single-file decoder
The CHECK_F macros differ slightly (but eventually do the same thing). Older GCC needs to fallback on the old-style pragma optimisation flags.
2019-08-21 17:49:17 +02:00
W. Felix Handte
0d606ee3db Fix Incorrect assert() 2018-12-18 13:36:39 -08:00
W. Felix Handte
c2d51637d9 Add Mutual-Exclusion Error 2018-12-18 13:36:39 -08:00
W. Felix Handte
c560e34c86 Add HUF_FORCE_DECOMPRESS_X2 2018-12-18 13:36:39 -08:00
W. Felix Handte
abd1567d3c Move HUF_DGEN Up Out of X1 Definitions 2018-12-18 13:36:39 -08:00
W. Felix Handte
432314b58a Rename HUF_DECOMPRESS_MINIMAL -> HUF_FORCE_DECOMPRESS_X1 2018-12-18 13:36:39 -08:00
W. Felix Handte
f45c9df42e Totally Hide/Disable X2 Variants when HUF_DECOMPRESS_MINIMAL is Defined 2018-12-18 13:36:39 -08:00
W. Felix Handte
f9cb348776 Add HUF_DECOMPRESS_MINIMAL Macro, Which Avoids Using X2 Variants 2018-12-18 13:36:39 -08:00
Yann Collet
87406548f0 reduced DDict size, by -2KB
corresponding to the removal of workspace
which is needed while building huffman table
and is now either present in DCtx,
or temporarily borrowed from available FSE table space.
2018-09-06 17:07:53 -07:00
Yann Collet
b7e5ebef2a grouped X2 function together 2018-06-14 20:41:50 -04:00
Yann Collet
1adf84ccb7 renamed all HUF_decompress*X4*() functions into *X2
to underline they generate up to 2 symbols per decoding,
in preparation for a future *X3 variant.
2018-06-14 15:17:03 -04:00
Yann Collet
a09af5eb6b renamed all HUF_decompress*X2*() functions into *X1
to underline they generate one symbol per decoding operation.

The new naming scheme will make it easier to introduce an *X3 variant.
2018-06-14 15:08:43 -04:00
Yann Collet
fa41bcc2c2 grouped debug functions into debug.h
There were 2 competing set of debug functions
within zstd_internal.h and bitstream.h.
They were mostly duplicate, and required care to avoid messing with each other.

There is now a single implementation, shared by both.

Significant change :
The macro variable ZSTD_DEBUG does no longer exist,
it has been replaced by DEBUGLEVEL,
which required modifying several source files.
2018-06-13 15:43:09 -04:00
Yann Collet
ba2ad9b6b9 implemented fractional bit cost evaluation
for FSE symbols.

While it seems to work, the gains are negligible compared to rough maxNbBits evaluation.
There are even a few losses sometimes, that still need to be explained.
Furthermode, there are still cases where btlazy2 does a better job than btopt,
which seems rather strange too.
2018-05-08 17:43:13 -07:00
Yann Collet
a909c293c6 Merge branch 'dev' into hufCompress 2018-03-14 16:11:25 -07:00
Yann Collet
a95a88af57 removed huf_compress_impl.h
re-imported all functions inside huf_compress.c
for easier source editing.

Also updated a bunch of code comments
for clarification.
2018-03-13 14:14:05 -07:00
Yann Collet
17626ba56e restored ZSTD_decodeSequence() into zstd_decompress.c 2018-03-09 18:03:25 -08:00
Yann Collet
b01552a07a force inlining of HUF_decodeSymbol*() functions
which was not done properly by gcc 4.8
resulting in major performance difference.

ex :
zstd -b1 silesia.tar
before : dec 680 MB/s
after  : dec 710 MB/s  (without bmi2)
after  : dec 770 MB/s  (with DYNAMIC_BMI2)
2018-03-01 11:31:45 -08:00
Yann Collet
6cdf690441 minor cleaning of huff0
Update code documentation, and properly names a few "magic constants".
Also, HUF_compress_internal() gets a cleaner way
to determine size of tables inside workspace.
2018-02-26 14:52:23 -08:00
Yann Collet
653383f74a minor nit from Mac XCode 2018-02-22 15:44:26 -08:00
Yann Collet
0fd4df6ed3 Implemented BMI2 functions directly within huf_decompress.c
This makes it easier to edit for maintenance and evolutions
(I plan to experiment modifications in huffman decompression functions).

The methology followed seems broadly applicable to other BMI2 modules.

Performance was tracked rigorously at each step,
there is no noticeable loss (nor win) of performance compared to `#include` version.

Note however that 4X decoder variants tend to be extremely sensitive to code alignment.
This source code resulted in pretty good performance for gcc 7.2 and 7.3,
but future changes (even in other parts of the code) might trigger the issue again.
2018-02-22 10:51:47 -08:00
Nick Terrell
4319132312 [decompress] Support BMI2 2018-02-13 17:00:15 -08:00
Yann Collet
f9e6590715 Merge pull request #796 from terrelln/is-error
[FSE][HUF] Inline error checks
2017-08-15 12:37:28 -07:00
Nick Terrell
07c6ff588e [FSE][HUF] Inline error checks
Caught by Clang's optimization remarks.
2017-08-15 11:23:28 -07:00
Nick Terrell
565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Nick Terrell
abe12b3399 [libzstd] Fix bug in Huffman decompresser
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
2017-08-07 12:37:48 -07:00
Stella Lau
4c71f59c77 Clarify typedef of rankVal_t and rankValCol_t 2017-06-30 09:52:20 -07:00
Stella Lau
28f711ef95 Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK 2017-06-30 09:38:11 -07:00
Stella Lau
70ad6829e7 Delegate HUF_decompress4X_hufOnly to workspace version 2017-06-29 16:22:32 -07:00
Stella Lau
104c4d57c1 Fix bitshift error 2017-06-29 15:40:49 -07:00
Stella Lau
fedc94de8c Fix pointer casting warning 2017-06-29 13:04:15 -07:00
Stella Lau
c6a5275a28 Fix alignment warnings with pointer casting 2017-06-29 12:39:34 -07:00
Stella Lau
99e315999c Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2 2017-06-29 11:49:59 -07:00
Nick Terrell
55fc1f91fd [zstd] Fix up formatting edge cases for clang-format 2017-05-24 13:50:10 -07:00
Yann Collet
a41a4ed39a Merge pull request #594 from terrelln/bugs
Small fixes
2017-03-08 14:56:07 -08:00
Nick Terrell
81512e9ebe Avoid '#define inline /* ... */'
Take definition of `FORCE_INLINE` from `zstd_internal.h`.
2017-03-08 14:00:21 -08:00
Yann Collet
1f2c95c5f3 minor code refactor in HUF module 2017-03-05 21:07:20 -08:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage
saved 128 KB
2017-02-27 00:27:30 -08:00
Anders Oleson
517577bf53 spelling fixes in comments
i.e. occurred labeled Huffman
2017-02-20 12:08:59 -08:00
Yann Collet
2238312c2f fix dict loading 2016-12-02 11:36:11 -08:00
Yann Collet
b89af20353 reduced table sizes for HUF_readDTableX4 2016-12-01 18:24:59 -08:00
Yann Collet
1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
Yann Collet
af1960396b sync huff0 2016-08-19 19:38:19 +02:00
Yann Collet
9b998e4d08 Fixed decompression of literals in dictionary mode 2016-06-15 23:11:20 +02:00
Yann Collet
cd98f93cff Fixed decompression issue with invalid data 2016-06-11 23:26:22 +02:00
Yann Collet
237ad4beb3 Added single-stream decompression variant using external DTable 2016-06-11 01:46:03 +02:00
Yann Collet
289bbd52e5 Updated huff0 2016-06-11 01:31:54 +02:00
Yann Collet
662a541431 updated huff0 - now generates a common HUF_DTable type for all decoding tables 2016-06-08 11:11:02 +02:00
Yann Collet
130fe11394 merged huf_static.h into huf.h . Requires HUF_STATIC_LINKING_ONLY macro. 2016-06-05 00:42:28 +02:00
inikep
36fac00149 removed calloc calls from lib/ 2016-06-03 13:23:04 +02:00