Yann Collet
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Yann Collet
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
Yann Collet
f04deff4fc
fixed #718 , reported by @GregSlazinski, solution suggested by @mcmilk
2017-07-06 01:42:46 -07:00
Yann Collet
d75c0e71c4
minor code refactoring
2017-07-05 18:10:07 -07:00
Yann Collet
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
Nick Terrell
c80fc50a8d
[libzstd] Fix memcpy() on potential NULL source
...
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
the source should not be `NULL`.
2017-07-03 12:31:55 -07:00
Yann Collet
2485f88bf8
fixed legacy version init bug
2017-07-01 09:09:34 -07:00
Yann Collet
7f40bb1c39
Merge pull request #742 from stellamplau/stack-space
...
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-30 14:50:23 -07:00
Stella Lau
4c71f59c77
Clarify typedef of rankVal_t and rankValCol_t
2017-06-30 09:52:20 -07:00
Stella Lau
28f711ef95
Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK
2017-06-30 09:38:11 -07:00
Stella Lau
70ad6829e7
Delegate HUF_decompress4X_hufOnly to workspace version
2017-06-29 16:22:32 -07:00
Stella Lau
104c4d57c1
Fix bitshift error
2017-06-29 15:40:49 -07:00
Stella Lau
fedc94de8c
Fix pointer casting warning
2017-06-29 13:04:15 -07:00
Stella Lau
c6a5275a28
Fix alignment warnings with pointer casting
2017-06-29 12:39:34 -07:00
Stella Lau
99e315999c
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-29 11:49:59 -07:00
Yann Collet
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
Yann Collet
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
Yann Collet
c7fb884eea
fixed minor conversion warning
2017-06-26 18:02:23 -07:00
Yann Collet
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
Yann Collet
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
Yann Collet
1e4129b27b
fixed dangling pointer risk, detected by @terrelln
2017-06-21 13:26:10 -07:00
Yann Collet
e51d51bdf7
fixed memcpy() overlap
2017-06-20 17:44:55 -07:00
Yann Collet
c3bce24ef4
fixed potential dangling pointer, detected by @terrelln
2017-06-20 16:09:11 -07:00
Yann Collet
695a0a3449
fixed IA64 compilation error, by @mcmilk
2017-06-19 15:27:30 -07:00
Paul Cruz
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
Yann Collet
58e8d793e1
made debug definitions common within zstd_internal.h
2017-06-02 18:20:48 -07:00
Yann Collet
8ddf4c22d5
fixed missing initialization
2017-06-02 17:16:49 -07:00
Yann Collet
33a7e679e5
significant zlib wrapper code refactoring
...
code indentation
variable scope and names
constify
Only coding style changes.
The logic should remain the same.
2017-06-02 17:10:49 -07:00
Yann Collet
4effccbf56
zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing
...
more generic and safer than using own routing for magic number comparison
2017-06-02 14:27:11 -07:00
Yann Collet
ae728a43b8
removed defaultCustomMem
...
now ZSTD_customCMem is promoted as new default.
Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.
ZSTD_customCMem is public constant
(defaultCustomMem was private only).
Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.
Fixed zlibwrapper which depended on defaultCustomMem.
2017-05-30 17:11:39 -07:00
Yann Collet
beb62b15a8
Merge branch 'dev' into advancedAPI2
...
Fixed conflic in zstd_decompress.c
2017-05-30 16:18:57 -07:00
Yann Collet
b8136f019a
static dctx is incompatible with legacy support
...
documented, and runtime tested
2017-05-27 00:03:08 -07:00
Yann Collet
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
Yann Collet
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
Yann Collet
ba183005d3
merged DStream's inBuff and outBuff into a single buffer
...
Saves one malloc().
Also : makes it easier to implement static allocation
2017-05-24 15:42:24 -07:00
Nick Terrell
55fc1f91fd
[zstd] Fix up formatting edge cases for clang-format
2017-05-24 13:50:10 -07:00
Yann Collet
2e4db3e531
fixed performance regression with ZSTD_decompress() on small files
...
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.
Fixed by only initialization relevant pointers and size fields.
2017-05-24 13:15:19 -07:00
Yann Collet
11ea2f7fda
Merged ZSTD_DCtx and ZSTD_DStream objects
...
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
2017-05-23 16:19:43 -07:00
Yann Collet
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
Yann Collet
7101434ec9
pedantic : added one error check
...
on a function which (today) never fails.
But who knows, maybe tomorrow ...
2017-05-16 16:28:24 -07:00
Yann Collet
4eff8136aa
added prototype ZSTD_decompressBegin_usingDDict ( #700 )
2017-05-16 16:05:27 -07:00
Yann Collet
cef02d9317
changed variable name, for clarity
...
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
Yann Collet
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet
b184589c4c
minor code refactoring for clarity
2017-05-01 11:35:47 -07:00
Yann Collet
89f50deec7
minor code refactoring
...
clearer tables
2017-04-28 16:52:36 -07:00
Yann Collet
003a244324
DStream : ensure correct size of internal buffers in case of error
2017-04-05 15:28:56 -07:00
Sean Purcell
4708394bdd
Remove extra 'F' from skippable magic mask
2017-03-29 11:46:57 -07:00
Nick Terrell
eaf69b07f0
Zero pointers after freeing
2017-03-21 13:20:59 -07:00
Sean Purcell
334cb34edb
ZSTD_LEGACY_SUPPORT defines lowest supported version
2017-03-13 14:32:30 -07:00
Sean Purcell
784082f49c
Change gotoDict type to uPtrDiff
2017-03-10 10:34:45 -08:00
Sean Purcell
8fe5c6862c
Fix undefined behaviour in decompressor
2017-03-10 10:17:42 -08:00
Yann Collet
a41a4ed39a
Merge pull request #594 from terrelln/bugs
...
Small fixes
2017-03-08 14:56:07 -08:00
Nick Terrell
81512e9ebe
Avoid '#define inline /* ... */'
...
Take definition of `FORCE_INLINE` from `zstd_internal.h`.
2017-03-08 14:00:21 -08:00
Yann Collet
1f2c95c5f3
minor code refactor in HUF module
2017-03-05 21:07:20 -08:00
Yann Collet
fe5d27062e
disable prefetch-decode for 32-bits target
...
This decoder variant is detrimental to x86 architecture
likely due to register pressure.
Note that the variant is disabled for all 32-bits targets.
It's unclear if it would help for different architectures,
such as ARM, MIPS or PowerPC.
2017-03-02 17:09:21 -08:00
Sean Purcell
3d95925a59
Merge remote-tracking branch 'origin/dev' into m32
2017-03-02 15:17:56 -08:00
Yann Collet
fdb0fd34b3
Merge pull request #583 from terrelln/set-dictid
...
Set dictID to 0 for content only dictionaries
2017-03-02 13:15:31 -08:00
Nick Terrell
3475b9b431
Set dictID to 0 for content only dictionaries
2017-03-02 12:33:02 -08:00
Sean Purcell
d44703d145
Offsets >= 32MB in 32-bits mode
2017-03-01 16:27:56 -08:00
Yann Collet
4bcc69b761
solves warnings when compiling with global XXH_STATIC_LINKING_ONLY
...
XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include.
The main idea is to keep this setting local :
user module shall explicitly understand and accept the static linking restriction
which becomes transparent when triggering the macro at project level.
Global definition also triggers redefinition warnings for user modules which do locally define the macro.
This new version compiles lib and cli without warning when the macro is set globally.
That's not a scenario to be recommended, since it trades a local effect for a global one,
but it was easy enough to provide from zstd side.
2017-03-01 11:33:25 -08:00
Yann Collet
31432cc57d
Merge pull request #579 from iburinoc/multiframe
...
Check to ensure ddict isn't null before dereference
2017-03-01 11:02:04 -08:00
Sean Purcell
a81d4fee58
Check to ensure ddict isn't null before dereference
2017-02-28 15:28:29 -08:00
Yann Collet
22d79762ef
fixed multi frames
2017-02-28 02:12:42 -08:00
Yann Collet
a33ae64204
fixed decoding skippable frames
2017-02-28 01:15:28 -08:00
Yann Collet
d1760113ec
Improved speed of ZSTD_decompressStream()
...
When ZSTD_decompressStream() detects
that there is enough space in dst
to complete decompression in a single pass,
delegates to ZSTD_decompress(),
for an extra ~5% speed boost
2017-02-28 00:14:28 -08:00
Yann Collet
0b9b894b2d
reduced ZSTD_DDict memory usage
...
saved 128 KB
2017-02-27 00:27:30 -08:00
Yann Collet
bd7fa21deb
added ZSTD_refDDict()
...
Now DDict does no longer depends on DCtx duplication
2017-02-26 14:43:07 -08:00
Yann Collet
d73eebc00f
loadEntropy works on new ZSTD_entropy_t type
2017-02-26 10:16:42 -08:00
Yann Collet
8629f0e41f
created entropy structure type
2017-02-25 18:33:31 -08:00
Yann Collet
8dff956dbf
Added DDict unit test in fuzzer
...
also : slightly modified loadEntropy :
know src must points at start of dictionary
2017-02-25 10:11:15 -08:00
Sean Purcell
9757cc811b
Update comment
2017-02-22 12:28:21 -08:00
Sean Purcell
9050e1925e
Change name to to findFrameCompressedSize and add skippable support
2017-02-22 12:12:34 -08:00
Anders Oleson
517577bf53
spelling fixes in comments
...
i.e. occurred labeled Huffman
2017-02-20 12:08:59 -08:00
Sean Purcell
6b010dec80
execSequence copies up to 2*WILDCOPY_OVERLENGTH extra
2017-02-16 12:05:40 -08:00
Sean Purcell
887eaa9e21
Fix wildcopy overwriting data still in window
2017-02-15 16:43:45 -08:00
Sean Purcell
d7bfcac18a
Expose frameSrcSize to experimental API
2017-02-10 11:55:44 -08:00
Sean Purcell
ba2ad9f25c
ZSTD_decompress now handles multiple frames
2017-02-08 14:50:10 -08:00
Sean Purcell
4e709712e1
Decompressed size functions now handle multiframes and distinguish cases
...
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
Yann Collet
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Sean Purcell
57d423c5df
Don't create dict in streaming apis if dictSize == 0
2017-01-17 14:31:35 -08:00
Sean Purcell
834ab50fa3
Fixed decompress_usingDict not propagating corrupted dictionary error
2017-01-11 17:31:34 -08:00
Yann Collet
aca113f4f5
fixed ZSTD_sizeof_?Dict()
2016-12-23 22:25:03 +01:00
Yann Collet
0819abe3c1
added ZSTD_createDDict_byReference() body
2016-12-21 19:25:15 +01:00
Yann Collet
4e5eea61a8
added ZSTD_createDDict_byReference()
2016-12-21 16:44:35 +01:00
Nick Terrell
8157a4c3cc
Fix dictionary loading bug causing an MSAN failure
...
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
2016-12-20 10:47:52 -08:00
Yann Collet
35168679bd
Merge pull request #478 from terrelln/wildcopy-ub
...
Fix execSequence wildcopy undefined behavior
2016-12-13 11:33:00 +01:00
Nick Terrell
064a143520
Fix execSequence wildcopy undefined behavior
...
execSequence relied on pointer overflow to handle cases where
`sequence.matchLength < 8`. Instead of passing an `size_t` to
wildcopy, pass a `ptrdiff_t`.
2016-12-12 19:01:23 -08:00
Nick Terrell
e474aa55b4
Fix decompression buffer overrun
...
Allows an adversary to write up to 3 bytes beyond the end of the buffer.
Occurs if the match overlaps the `extDict` and `currentPrefix`, and the
match length in the `currentPrefix` is less than `MINMATCH`, and
`op-(16-MINMATCH) >= oMatchEnd > op-16`.
2016-12-12 18:05:30 -08:00
Yann Collet
825dffbc43
moved zbuff source files into lib/deprecated
2016-12-05 19:28:19 -08:00
Yann Collet
8f8e2b0b4a
fixed initialization warning
2016-12-05 18:00:50 -08:00
Yann Collet
e7a41a5955
added : dictID retrieval functions.
...
added : unit tests for dictID retrieval functions
2016-12-05 16:21:06 -08:00
Yann Collet
9ffbeea875
API : changed : streaming decompression : implicit reset on starting new frames
2016-12-02 18:37:38 -08:00
Yann Collet
2238312c2f
fix dict loading
2016-12-02 11:36:11 -08:00
Yann Collet
b89af20353
reduced table sizes for HUF_readDTableX4
2016-12-01 18:24:59 -08:00
Yann Collet
ff504de391
minor decompression speed improvement
2016-11-29 17:42:46 -08:00
Yann Collet
a56ac2815c
restored normal decoder speed
2016-11-29 15:30:23 -08:00
Yann Collet
37870d7a66
fixed minor visual warning
2016-11-29 14:31:57 -08:00
Yann Collet
4f5350f610
long matches support overflow
2016-11-29 13:12:24 -08:00
Yann Collet
52e136ed3d
long decoder compatible with round and separate buffers
2016-11-28 19:59:11 -08:00
Yann Collet
ce3527ca0c
combined normal and long decoder
2016-11-28 18:38:52 -08:00
Yann Collet
8993bee997
restored normal mode
2016-11-28 16:11:30 -08:00
Yann Collet
764e70a4f3
added decodeSequencesLong
2016-11-28 15:50:16 -08:00
Yann Collet
73f88a66f1
added prefetch
2016-11-23 15:43:30 -08:00
Yann Collet
50524bf0da
delayed decompression
2016-11-23 15:11:07 -08:00
Nick Terrell
4359d21ad7
Merge two memset() calls into one
2016-11-14 17:52:51 -08:00
Nick Terrell
24701de877
Fix uninitialized memory read
2016-11-14 13:57:05 -08:00
Yann Collet
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
Yann Collet
31e660e7aa
more accurate default maximum window size
2016-10-29 03:56:45 -07:00
Yann Collet
2115724c22
Merge pull request #430 from terrelln/exec-sequences
...
ZSTD_execSequence() accepts match in last 7 bytes
2016-10-28 10:45:05 -07:00
Nick Terrell
10bfd0c0d5
Fix ZSTD_execSequence() performance regression
...
Commit ae1cb3b3d0
caused the regression.
It is an instruction alignment issue, because if it is `U64 i` instead
of `U32 i`, the regression returns. This patch fixes the regression
in gcc, but only gets some of the clang performance back.
Benchmarks:
Run on `silesia.tar`. I only show levels 1-5 because the performance
regression was uniform across all levels. I did one run on levels
1-19 and it looked good.
| Build | Level | Before | While | After |
|-------|-------|-------:|------:|------:|
| gcc | 1 | 931.4 | 904.4 | 932.8 |
| gcc | 2 | 849.1 | 822.6 | 851.2 |
| gcc | 3 | 815.6 | 790.6 | 818.9 |
| gcc | 4 | 794.1 | 770.7 | 798.0 |
| gcc | 5 | 785.7 | 760.7 | 788.8 |
| clang | 1 | 705.5 | 683.2 | 693.8 |
| clang | 2 | 670.0 | 649.2 | 660.7 |
| clang | 3 | 659.6 | 639.8 | 651.4 |
| clang | 4 | 652.5 | 634.7 | 645.9 |
| clang | 5 | 646.9 | 625.5 | 637.7 |
2016-10-27 16:19:57 -07:00
Nick Terrell
eb7873a048
ZSTD_execSequence() accepts match in last 7 bytes
...
The zstd reference compressor will not emit a match in the last 7
bytes of a block. The decompressor will also not accept a match
in the last 7 bytes. This patch makes the decompressor accept a
match in the last 7 bytes.
2016-10-25 21:24:15 -07:00
Yann Collet
335ad5d4d4
added ZSTD_initDStream_usingDDict() .
...
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
2016-10-25 17:47:02 -07:00
Nick Terrell
f698ad6deb
Merge remote-tracking branch 'upstream/dev' into fixes
...
* upstream/dev:
added doc\zstd_manual.html
added contrib\gen_html
zstd_compression_format.md moved to doc/
Fix small bug in ZSTD_execSequence()
improved ZSTD_compressBlock_opt_extDict_generic
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
zstd_opt.h: small improvement in compression ratio
improved dicitonary segment merge
use implicit rules to compile zstd_decompress.c
detect early impossible decompression scenario in legacy decoder v0.5
no repeat mode in legacy v0.5
fixed invalid invocation of dictionary in legacy decoder v0.5
fix edge case
fix command line interpretation
fixed minor corner case
zstd.h: added the Introduction section
fixed clang 3.5 warnings
zstd.h: updated comments
2016-10-24 13:10:13 -07:00
Yann Collet
4239a207dd
Merge pull request #425 from inikep/dev11
...
Doc
2016-10-24 11:11:40 -07:00
Przemyslaw Skibinski
3ee94a7600
zstd_compression_format.md moved to doc/
2016-10-24 15:58:07 +02:00
Yann Collet
97611611a3
Merge pull request #423 from terrelln/exec-seq-patch
...
Fix small bug in ZSTD_execSequence()
2016-10-21 17:02:06 -07:00
Nick Terrell
ae1cb3b3d0
Fix small bug in ZSTD_execSequence()
...
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.
Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.
These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.
Introduced by commit 7158584399
2016-10-21 12:13:44 -07:00
Yann Collet
da3bd8b6de
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
2016-10-20 20:11:00 -07:00
Nick Terrell
bb68062c59
Unitialized memory read in ZSTD_decodeSeqHeaders()
...
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
2016-10-18 16:41:33 -07:00
Yann Collet
06573e17be
fixed minor corner case
2016-10-17 17:28:28 -07:00
Nick Terrell
4db751668f
Fix buffer overrun in ZSTD_loadEntropy()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`. This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.
The benchmarks look good, there is no obvious performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 268.2 MB/s , 701.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.5 MB/s , 666.9 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 154.9 MB/s , 655.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 128.9 MB/s , 648.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.4 MB/s , 633.4 MB/s
> ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 266.1 MB/s , 703.7 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.0 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.2 MB/s , 656.2 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 133.2 MB/s , 647.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.3 MB/s , 633.3 MB/s
2016-10-17 15:51:15 -07:00
Yann Collet
7933434fdf
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-10-14 13:32:35 -07:00
Yann Collet
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
Nick Terrell
3b9cdf9220
Fix ubsan failures (pass NULL to memcpy)
2016-10-12 20:54:42 -07:00
Yann Collet
5d919e7ac3
added ZSTD_error_frameParameter_windowTooLarge ( #403 )
2016-10-12 17:29:24 -07:00
Nick Terrell
7158584399
Fix ZSTD_execSequence() edge case
2016-10-12 10:05:26 -07:00
Yann Collet
2f2639438a
zstreamtest can fuzztest pledgedSrcSize
2016-09-26 14:06:08 +02:00
Yann Collet
51f4d566c2
small decompression speed boost for very small data
2016-09-22 15:57:28 +02:00
Yann Collet
d7c6589df8
support ZSTD_sizeof_*() on NULL
...
added ZSTD_sizeof_CDict()
2016-09-15 02:57:27 +02:00
Yann Collet
e91c4b4cef
introduced ZSTD_resetDStream() .
...
added : ZSTD_sizeof_DDict()
2016-09-14 16:55:44 +02:00
Yann Collet
d092d77cfc
minor variable renaming
2016-09-14 16:14:57 +02:00
Yann Collet
26ec254066
new strategy for faster DDict decompression
2016-09-13 16:52:16 +02:00
Yann Collet
b3060f7a9e
changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed.
2016-09-09 16:44:16 +02:00
Yann Collet
95d07d7447
introduced CHECK_E
2016-09-06 16:38:51 +02:00
Yann Collet
3e21ec5b01
introduced CHECK_F
2016-09-06 15:36:19 +02:00
Yann Collet
5c956d593c
FORCE_INLINE common definition
2016-09-06 15:05:19 +02:00
Yann Collet
7c83dfd5c2
ZSTD_frameHeaderSize_prefix ( #340 ), as result of ZSTD_initStream
2016-09-05 19:47:43 +02:00
Yann Collet
1563bfeabc
fixing FORCE_INLINE for older compilers ( #330 )
2016-09-02 11:44:21 -07:00
David Lam
e10f7f3dcb
merge
2016-08-30 12:03:36 -07:00
Yann Collet
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
David Lam
da9d3b7057
Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c
2016-08-29 17:31:51 -07:00
Yann Collet
23b6e05d8e
ZSTD_malloc() and ZSTD_free(), to simplify customMem
2016-08-28 21:05:43 -07:00
Yann Collet
5f53b0335e
fixed continuation context
2016-08-28 10:00:49 -07:00
Yann Collet
767d8f66fa
legacy contexts can be re-used
2016-08-28 08:19:47 -07:00
Yann Collet
4bf317dd00
first version supporting legacy streams (transparent decoding)
2016-08-28 07:43:34 -07:00
inikep
a3a47ec4d0
Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other
2016-08-24 21:25:49 +02:00
inikep
e416e30019
remove unnecessary comments
2016-08-24 17:32:09 +02:00
Yann Collet
17e482efdd
added ZSTD_setDStreamParameter()
2016-08-23 16:58:10 +02:00
Yann Collet
3071c3e303
STREAM_WINDOW_MAX : protect streaming from unreasonable memory requirements
2016-08-23 01:34:34 +02:00
Yann Collet
70e3b31306
fixed playtests on os-x
2016-08-23 01:18:06 +02:00
Yann Collet
cb3276329a
added sizeof CStream and DStream
2016-08-23 00:31:59 +02:00
Yann Collet
8baf78a291
minor coding style
2016-08-20 13:04:20 +02:00
Yann Collet
1bee2d5e08
slight decompression speed improvement
2016-08-20 02:59:04 +02:00
Yann Collet
af1960396b
sync huff0
2016-08-19 19:38:19 +02:00
Yann Collet
18442c1482
minor refactoring
2016-08-18 01:40:32 +02:00
Yann Collet
53e17fbd5e
updated streaming API
2016-08-17 01:39:22 +02:00
Yann Collet
104e5b072d
added : streaming decompression API
2016-08-16 15:11:28 +02:00
inikep
038d1497c9
fixed compilation with Visual Studio 2005
2016-08-10 14:30:10 +02:00
Yann Collet
917fe188f1
Implemented repOffset "minus 1" on ll==0
2016-07-31 04:01:57 +02:00
Yann Collet
66f69e58d2
restore decompression speed on fizzle
2016-07-30 15:32:47 +02:00
Yann Collet
f714f59c16
fixed visual warning
2016-07-30 12:05:28 +02:00
Yann Collet
761f8dbbd2
back to normal table cell copy
2016-07-30 11:43:53 +02:00
Yann Collet
3c6b808870
minor decompression speed gains
2016-07-30 03:20:47 +02:00
Yann Collet
c00d30fbe4
Merge pull request #264 from inikep/dev08
...
Dev08
2016-07-29 17:42:30 +02:00
Yann Collet
4c5bbf64f9
fixed : frame concatenation without checksum
2016-07-28 20:30:25 +02:00
Yann Collet
60ba31c570
zbuff uses ZSTD_compressEnd()
2016-07-28 19:55:09 +02:00
Yann Collet
c991cc1828
new frame end, 32-bits checksums
2016-07-28 00:55:43 +02:00
inikep
003c7a8568
optimal parser: removed ZSTD_REP_INIT
2016-07-27 11:07:13 +02:00
Eric Biggers
0a55e7a0bb
ZSTD_decompressFrame(): use remainingSize instead of iend - ip
...
Same behavior, but no need to have redundant variables.
2016-07-26 13:22:27 -07:00
Eric Biggers
aa6c70bf60
ZSTD_decompressFrame(): pass up error code from ZSTD_decodeFrameHeader()
2016-07-26 13:22:27 -07:00
Eric Biggers
e4d0265ea9
Replace remaining references to "direct mode" with "single segment mode"
2016-07-26 13:22:27 -07:00
Yann Collet
cbc5e9dc19
fixes oob read
2016-07-24 18:02:04 +02:00
Yann Collet
7ed5e33b89
minor comment changes
2016-07-24 14:26:11 +02:00
Yann Collet
10b9c13d07
fixed doc on cLevel default, reported by Oliver Lange
2016-07-24 01:21:53 +02:00
Yann Collet
f8e7b5363f
unified encoding types
2016-07-23 16:31:49 +02:00
Yann Collet
c2e1a68d81
changed streamNb order to 1-4-4-4
2016-07-22 17:30:52 +02:00
Yann Collet
772d912c2f
more complete support for literals repeat mode
2016-07-22 15:04:25 +02:00
Yann Collet
9f2d82d4a4
fixed : big-endian decoding
2016-07-22 14:37:10 +02:00
Yann Collet
32faf6c8e7
fixed conversion warnings
2016-07-22 14:37:09 +02:00
Yann Collet
5e45a5fbb3
force loop-align to 32 for zstd_decompress
2016-07-22 14:37:09 +02:00
Yann Collet
5288ac0cb7
changed filed order
2016-07-22 14:37:09 +02:00
Yann Collet
198e6aac44
Literals header fields use little endian convention
2016-07-22 14:37:09 +02:00
Yann Collet
6fa05a2371
cBlockSize uses little-endian convention
2016-07-22 14:37:09 +02:00
Yann Collet
cf05b9d477
ZSTD_getBlockSizeMax()
2016-07-18 16:52:10 +02:00
Yann Collet
972e5806ee
fixed : premature frame end on zero-sized raw block - reported by @ebiggers
2016-07-17 15:39:24 +02:00
Yann Collet
d158c35e9f
added ZSTD_estimateDCtxSize()
2016-07-11 13:46:25 +02:00
Yann Collet
8e0ee681b8
added ZSTD_sizeofDCtx()
2016-07-11 13:09:52 +02:00
Yann Collet
3ae543ce75
added ZSTD_estimateCCtxSize()
2016-07-11 03:12:17 +02:00
Yann Collet
722e14bb65
fixed compilation error in decompression module
2016-07-08 19:22:16 +02:00
Yann Collet
bd10607063
updated spec
2016-07-08 19:16:57 +02:00
Yann Collet
c5fb5b7fcd
support offset > 128 MB
2016-07-08 13:13:37 +02:00
Yann Collet
19c27d27f1
simplified legacy functions, no longer need magic number
2016-07-07 14:40:13 +02:00
Yann Collet
f323bf7d32
added : ZSTD_getDecompressedSize()
2016-07-07 13:14:21 +02:00
Yann Collet
f246cf5423
ZSTD_decompress_usingDDict() compatible with Legacy mode
2016-07-06 20:32:27 +02:00
Yann Collet
517e1ba623
fixed dictBuilder issue with HC levels. Reported by Bartosz Taudul.
2016-07-06 12:35:09 +02:00
Yann Collet
fe07eaa972
simplified ZSTD_decodeSequence()
2016-07-06 02:25:44 +02:00
Yann Collet
9ca73364e6
updated spec
2016-07-05 10:53:38 +02:00
Yann Collet
f9cac7a734
Added GNU separator --
, to specifies that all following arguments are necessary file names (and not commands). Suggested by @chipturner ( #230 )
2016-07-04 18:18:24 +02:00
Yann Collet
23f05ccc6b
updated specifications
2016-07-04 16:13:11 +02:00
Yann Collet
2fa9904844
update specification and comments
2016-07-01 20:55:28 +02:00
Yann Collet
c093208ab8
fix : potential leak ( #229 )
2016-06-30 14:07:30 +02:00
Yann Collet
d4f4e58ee1
fixed ZSTD_decompressBlock() using multiple blocks
2016-06-27 01:31:35 +02:00
Yann Collet
e4811ba761
Modified : ZSTD_createDDict() accepts dictionary < 8 bytes in pure content mode (reported by @chipturner)
2016-06-19 23:06:54 +02:00
Yann Collet
06d9a73b48
minor refactor, using WILDCOPY_OVERLENGTH
macro instead of hard-coded 8
2016-06-19 14:27:21 +02:00
Yann Collet
4948f270b3
make room for reserved "information bit" in frame header
2016-06-16 15:38:51 +02:00
Yann Collet
80d033fb43
fixed ptr arithmetic warning
2016-06-16 01:41:50 +02:00
Yann Collet
736d419289
strengthened dict loading on decompresson side
2016-06-16 01:05:04 +02:00
Yann Collet
8e36a9c169
decoder restores repOffsets from dictionary
2016-06-16 01:05:04 +02:00
Yann Collet
9b998e4d08
Fixed decompression of literals in dictionary mode
2016-06-15 23:11:20 +02:00
Yann Collet
d059092897
fixed conversion warnings
2016-06-14 15:34:24 +02:00
Yann Collet
4266c0a2fd
adding inter-blocks rep-offsets
2016-06-14 01:49:25 +02:00
Yann Collet
cd98f93cff
Fixed decompression issue with invalid data
2016-06-11 23:26:22 +02:00
Yann Collet
37fece22e8
enable repeat-entropic-stats mode
2016-06-11 02:52:42 +02:00
Yann Collet
d60a5bf900
Literal decompression builds Huffman tables within shared space (for later re-use)
2016-06-11 02:35:31 +02:00
Yann Collet
237ad4beb3
Added single-stream decompression variant using external DTable
2016-06-11 01:46:03 +02:00
Yann Collet
289bbd52e5
Updated huff0
2016-06-11 01:31:54 +02:00
Yann Collet
9dd12742f3
litBlockType_t
is an enum
2016-06-10 00:12:26 +02:00
Yann Collet
662a541431
updated huff0 - now generates a common HUF_DTable type for all decoding tables
2016-06-08 11:11:02 +02:00
Yann Collet
302fb53a76
Removed ZSTD_*_usingPrepared?Ctx()
declaration from public space
2016-06-07 12:16:49 +02:00
Yann Collet
81e13ef7cf
first implementation of the new dictionary API (untested)
2016-06-07 00:51:51 +02:00
Yann Collet
9d504ae85b
Added decoding of RLE blocks
2016-06-06 19:52:35 +02:00
Yann Collet
673f0d7cdc
new frame format, allowing custom window size
2016-06-06 00:26:38 +02:00
Yann Collet
d0e2cd15cb
Merged fse_static
into fse.h
. Now requires FSE_STATIC_LINKING_ONLY
macro.
2016-06-05 00:58:01 +02:00
Yann Collet
130fe11394
merged huf_static.h
into huf.h
. Requires HUF_STATIC_LINKING_ONLY
macro.
2016-06-05 00:42:28 +02:00
Yann Collet
d3b7f8d21f
Merged zstd_static.h
into zstd.h
. Now requires ZSTD_STATIC_LINKING_ONLY
macro
2016-06-04 19:47:02 +02:00
Yann Collet
ac110a1f21
Removed ZBUFF internal util function from public area
2016-06-04 19:16:49 +02:00
Yann Collet
5347aee8f7
merged zbuff_static.h
into zbuff.h
. Now requires ZBUFF_STATIC_LINKING_ONLY
macro
2016-06-04 19:12:48 +02:00
Yann Collet
198d127b35
minor comment change (unfinished description of new header format)
2016-06-04 18:40:55 +02:00
Yann Collet
f4f5affdf7
restore ZBUFF full-block-size, for better performance on small input
2016-06-03 23:09:28 +02:00
Yann Collet
ab7b6f1ece
Merge pull request #198 from inikep/dev070
...
Dev070
2016-06-03 21:37:49 +02:00
inikep
3640396b1a
fixed: deallocation of structures in case of error in ZBUFF_createCCtx and ZBUFF_createDCtx
2016-06-03 16:36:50 +02:00
Yann Collet
fe48775868
minor decoder code refactoring
2016-06-03 15:41:51 +02:00
inikep
3763c77f6b
defaultCustomNULL replaced with defaultCustomMem
2016-06-03 13:28:20 +02:00
inikep
36fac00149
removed calloc calls from lib/
2016-06-03 13:23:04 +02:00
inikep
db2f540414
added defaultCustomNULL
2016-06-03 12:56:56 +02:00
inikep
2866951558
opaque parameter for custom memory allocation functions
2016-06-02 13:04:18 +02:00
inikep
9242816b56
fparamsPtr->windowLog==0 means that a frame is skippable
2016-06-01 18:47:04 +02:00
inikep
13f42d9085
VS2010 project: reverted zstdlib.rc
2016-06-01 14:44:31 +02:00
inikep
5c2771710d
Merge remote-tracking branch 'refs/remotes/Cyan4973/dev070' into dev070
...
# Conflicts:
# .gitignore
# lib/decompress/zstd_decompress.c
# programs/zbufftest.c
2016-06-01 09:16:11 +02:00
Yann Collet
202844ebd0
fixed zbufftest :
2016-06-01 00:44:36 +02:00
Yann Collet
8e3a36a6db
decompression validates frame content checksum
2016-06-01 00:18:28 +02:00
inikep
a6b942018d
Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070
...
# Conflicts:
# .travis.yml
# Makefile
# lib/common/zstd_static.h
# programs/Makefile
# projects/VS2008/zstd/zstd.vcproj
# projects/VS2008/zstdlib/zstdlib.vcproj
# projects/cmake/lib/CMakeLists.txt
# projects/cmake/programs/CMakeLists.txt
2016-06-01 00:07:09 +02:00