AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,826 Commits 1 Branch 0 Tags 27 MiB
9a91afe6ef
Commit Graph

318 Commits

Author SHA1 Message Date
Yann Collet
8977224b9b Merge pull request #859 from terrelln/31 
Prepare for ZSTD_WINDOWLOG_MAX == 31
 2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Nick Terrell
5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Yann Collet
ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() 
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
 2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min() 
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
 2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Yann Collet
8a5c0c98ae restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+) 2017-09-01 11:56:57 -07:00
Yann Collet
36aa8b5999 improved decoding speed 2017-09-01 11:40:59 -07:00
Yann Collet
3704507774 fixed decompression bug reported by @Etsukata (#828) 2017-09-01 00:05:37 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
2dbcfc6994 Merge pull request #794 from terrelln/force-inline 
[libzstd] Fix FORCE_INLINE macro
 2017-08-15 12:03:44 -07:00
Nick Terrell
565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Roman Gershman
b9d4f4fb74 Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged 2017-08-13 13:29:42 +03:00
Nick Terrell
9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Yann Collet
2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Nick Terrell
de0414b736 [libzstd] Pull CTables into sub-structure 2017-07-12 19:49:19 -07:00
Yann Collet
0f4fc6c20a fixed several conversion warnings 2017-07-07 17:13:12 -07:00
Yann Collet
9bde061a0b fixed minor Visual compilation limitation 2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf fixed minor cast warning 2017-07-07 16:09:47 -07:00
Yann Collet
ead4dd48f6 new field frameHeader.headerSize 2017-07-07 15:51:24 -07:00
Yann Collet
46396523c0 ZSTD_getFrameHeader : control of windowSize limits is delegated to caller 
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
 2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b extended frameHeader.windowSize to unsigned long long 2017-07-07 14:19:01 -07:00
Yann Collet
f04deff4fc fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk 2017-07-06 01:42:46 -07:00
Yann Collet
d75c0e71c4 minor code refactoring 2017-07-05 18:10:07 -07:00
Yann Collet
5051dd39ca Merge pull request #743 from facebook/fullbench 
compress_generic() automatic optimization opportunities
 2017-07-03 21:26:38 -07:00
Nick Terrell
c80fc50a8d [libzstd] Fix memcpy() on potential NULL source 
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
  to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
  the source should not be `NULL`.
 2017-07-03 12:31:55 -07:00
Yann Collet
2485f88bf8 fixed legacy version init bug 2017-07-01 09:09:34 -07:00
Yann Collet
7f40bb1c39 Merge pull request #742 from stellamplau/stack-space 
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
 2017-06-30 14:50:23 -07:00
Stella Lau
99e315999c Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2 2017-06-29 11:49:59 -07:00
Yann Collet
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet
7d3816183f exposed ZSTD_MAGIC_DICTIONARY in zstd.h 
makes it easier to explain ZSTD_dictMode
 2017-06-27 13:50:34 -07:00
Yann Collet
c7fb884eea fixed minor conversion warning 2017-06-26 18:02:23 -07:00
Yann Collet
dde10b23fe refactored ZSTD_estimateDStreamSize() 
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
 2017-06-26 17:44:26 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet
1e4129b27b fixed dangling pointer risk, detected by @terrelln 2017-06-21 13:26:10 -07:00
Yann Collet
e51d51bdf7 fixed memcpy() overlap 2017-06-20 17:44:55 -07:00
Yann Collet
c3bce24ef4 fixed potential dangling pointer, detected by @terrelln 2017-06-20 16:09:11 -07:00
Yann Collet
695a0a3449 fixed IA64 compilation error, by @mcmilk 2017-06-19 15:27:30 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Yann Collet
58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet
8ddf4c22d5 fixed missing initialization 2017-06-02 17:16:49 -07:00
Yann Collet
33a7e679e5 significant zlib wrapper code refactoring 
code indentation
variable scope and names
constify

Only coding style changes.
The logic should remain the same.
 2017-06-02 17:10:49 -07:00
Yann Collet
4effccbf56 zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing 
more generic and safer than using own routing for magic number comparison
 2017-06-02 14:27:11 -07:00
Yann Collet
ae728a43b8 removed defaultCustomMem 
now ZSTD_customCMem is promoted as new default.

Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.

ZSTD_customCMem is public constant
(defaultCustomMem was private only).

Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.

Fixed zlibwrapper which depended on defaultCustomMem.
 2017-05-30 17:11:39 -07:00
Yann Collet
b8136f019a static dctx is incompatible with legacy support 
documented, and runtime tested
 2017-05-27 00:03:08 -07:00
Yann Collet
cdf7e82222 Added ZSTD_initStaticCDict() 2017-05-25 18:05:49 -07:00
Yann Collet
57827f906f added ZSTD_initStaticDDict() 2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference 
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
 2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet
ba183005d3 merged DStream's inBuff and outBuff into a single buffer 
Saves one malloc().
Also : makes it easier to implement static allocation
 2017-05-24 15:42:24 -07:00
Nick Terrell
55fc1f91fd [zstd] Fix up formatting edge cases for clang-format 2017-05-24 13:50:10 -07:00
Yann Collet
2e4db3e531 fixed performance regression with ZSTD_decompress() on small files 
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.

Fixed by only initialization relevant pointers and size fields.
 2017-05-24 13:15:19 -07:00
Yann Collet
11ea2f7fda Merged ZSTD_DCtx and ZSTD_DStream objects 
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
 2017-05-23 16:19:43 -07:00
Yann Collet
fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX 
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
 2017-05-19 10:51:30 -07:00
Yann Collet
7101434ec9 pedantic : added one error check 
on a function which (today) never fails.
But who knows, maybe tomorrow ...
 2017-05-16 16:28:24 -07:00
Yann Collet
4eff8136aa added prototype ZSTD_decompressBegin_usingDDict (#700) 2017-05-16 16:05:27 -07:00
Yann Collet
cef02d9317 changed variable name, for clarity 
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
 2017-05-10 11:14:08 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader 
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
 2017-05-09 15:46:07 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() 
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
 2017-05-08 17:51:49 -07:00
Yann Collet
b184589c4c minor code refactoring for clarity 2017-05-01 11:35:47 -07:00
Yann Collet
89f50deec7 minor code refactoring 
clearer tables
 2017-04-28 16:52:36 -07:00
Yann Collet
003a244324 DStream : ensure correct size of internal buffers in case of error 2017-04-05 15:28:56 -07:00
Sean Purcell
4708394bdd Remove extra 'F' from skippable magic mask 2017-03-29 11:46:57 -07:00
Nick Terrell
eaf69b07f0 Zero pointers after freeing 2017-03-21 13:20:59 -07:00
Sean Purcell
334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Sean Purcell
784082f49c Change gotoDict type to uPtrDiff 2017-03-10 10:34:45 -08:00
Sean Purcell
8fe5c6862c Fix undefined behaviour in decompressor 2017-03-10 10:17:42 -08:00
Yann Collet
fe5d27062e disable prefetch-decode for 32-bits target 
This decoder variant is detrimental to x86 architecture
likely due to register pressure.

Note that the variant is disabled for all 32-bits targets.
It's unclear if it would help for different architectures,
such as ARM, MIPS or PowerPC.
 2017-03-02 17:09:21 -08:00
Sean Purcell
3d95925a59 Merge remote-tracking branch 'origin/dev' into m32 2017-03-02 15:17:56 -08:00
Yann Collet
fdb0fd34b3 Merge pull request #583 from terrelln/set-dictid 
Set dictID to 0 for content only dictionaries
 2017-03-02 13:15:31 -08:00
Nick Terrell
3475b9b431 Set dictID to 0 for content only dictionaries 2017-03-02 12:33:02 -08:00
Sean Purcell
d44703d145 Offsets >= 32MB in 32-bits mode 2017-03-01 16:27:56 -08:00
Yann Collet
4bcc69b761 solves warnings when compiling with global XXH_STATIC_LINKING_ONLY 
XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include.
The main idea is to keep this setting local :
user module shall explicitly understand and accept the static linking restriction
which becomes transparent when triggering the macro at project level.
Global definition also triggers redefinition warnings for user modules which do locally define the macro.

This new version compiles lib and cli without warning when the macro is set globally.
That's not a scenario to be recommended, since it trades a local effect for a global one,
but it was easy enough to provide from zstd side.
 2017-03-01 11:33:25 -08:00
Yann Collet
31432cc57d Merge pull request #579 from iburinoc/multiframe 
Check to ensure ddict isn't null before dereference
 2017-03-01 11:02:04 -08:00
Sean Purcell
a81d4fee58 Check to ensure ddict isn't null before dereference 2017-02-28 15:28:29 -08:00
Yann Collet
22d79762ef fixed multi frames 2017-02-28 02:12:42 -08:00
Yann Collet
a33ae64204 fixed decoding skippable frames 2017-02-28 01:15:28 -08:00
Yann Collet
d1760113ec Improved speed of ZSTD_decompressStream() 
When ZSTD_decompressStream() detects
that there is enough space in dst
to complete decompression in a single pass,
delegates to ZSTD_decompress(),
for an extra ~5% speed boost
 2017-02-28 00:14:28 -08:00
Yann Collet
0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet
bd7fa21deb added ZSTD_refDDict() 
Now DDict does no longer depends on DCtx duplication
 2017-02-26 14:43:07 -08:00
Yann Collet
d73eebc00f loadEntropy works on new ZSTD_entropy_t type 2017-02-26 10:16:42 -08:00
Yann Collet
8629f0e41f created entropy structure type 2017-02-25 18:33:31 -08:00
Yann Collet
8dff956dbf Added DDict unit test in fuzzer 
also : slightly modified loadEntropy :
know src must points at start of dictionary
 2017-02-25 10:11:15 -08:00
Sean Purcell
9757cc811b Update comment 2017-02-22 12:28:21 -08:00
Sean Purcell
9050e1925e Change name to to findFrameCompressedSize and add skippable support 2017-02-22 12:12:34 -08:00
Anders Oleson
517577bf53 spelling fixes in comments 
i.e. occurred labeled Huffman
 2017-02-20 12:08:59 -08:00
Sean Purcell
6b010dec80 execSequence copies up to 2*WILDCOPY_OVERLENGTH extra 2017-02-16 12:05:40 -08:00
Sean Purcell
887eaa9e21 Fix wildcopy overwriting data still in window 2017-02-15 16:43:45 -08:00
Sean Purcell
d7bfcac18a Expose frameSrcSize to experimental API 2017-02-10 11:55:44 -08:00
Sean Purcell
ba2ad9f25c ZSTD_decompress now handles multiple frames 2017-02-08 14:50:10 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00