Commit Graph

1083 Commits

Author SHA1 Message Date
Yann Collet
49d105cfcf better warning and error messages in case of dictionary training failure (#292) 2016-08-18 15:02:11 +02:00
Alexander Borzunov
0f6f17a14f Rename ZSTDLIB_API to ZDICTLIB_API in zdict.h 2016-08-18 16:47:06 +05:00
Alexander Borzunov
1f48382b1a Export functions related to dictionary compression from DLL 2016-08-18 16:12:49 +05:00
Yann Collet
e80d15304a Merge pull request #296 from inikep/Other
Other
2016-08-18 11:48:48 +02:00
inikep
a7bb322a93 removed never referenced functions 2016-08-18 10:30:21 +02:00
Yann Collet
18442c1482 minor refactoring 2016-08-18 01:40:32 +02:00
Yann Collet
c411902230 fixed g++ conversion warning 2016-08-17 01:50:54 +02:00
Yann Collet
53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet
655393cc72 updated doc for streaming API 2016-08-16 15:11:28 +02:00
Yann Collet
104e5b072d added : streaming decompression API 2016-08-16 15:11:28 +02:00
Yann Collet
5a0c8e2439 new streaming API (compression) 2016-08-16 15:11:27 +02:00
Yann Collet
ba92046031 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-08-11 22:10:04 +02:00
Yann Collet
e9b414d825 fixed msan warning (#281) 2016-08-11 22:09:09 +02:00
inikep
5f49eba512 added usage of rep[0]-1 for the optimal parser 2016-08-10 15:01:53 +02:00
inikep
98e08cbe34 fixed: tree not updated after finding very long rep matches 2016-08-10 15:00:30 +02:00
inikep
038d1497c9 fixed compilation with Visual Studio 2005 2016-08-10 14:30:10 +02:00
inikep
48849f86f0 fixed compilation with Intel Compiler with Windows 2016-08-10 14:26:35 +02:00
Yann Collet
1ea5622a32 updated xxhash 2016-08-10 09:40:08 +02:00
Yann Collet
666398e7ed added : xxhash namespace enforced from xxhash.h.
added : xxhash namespace test.
removed : -DXXH_NAMESPACE
2016-08-10 08:16:51 +02:00
Yann Collet
8ded0b84aa update xxhash to v0.6.2 2016-08-10 07:40:40 +02:00
Yann Collet
280f9a8754 minor comment 2016-08-08 00:44:00 +02:00
Yann Collet
e0b4a2d40f fixed dictionary generation, reported by Bartosz Taudul 2016-08-03 03:36:03 +02:00
Yann Collet
ae40b18d55 bumped library number 2016-08-03 01:59:21 +02:00
Yann Collet
0763905f44 ZSTD_compress_usingCDict() correctly provides original size by default in frame header
Fixed dictionary examples
2016-08-03 01:57:57 +02:00
Yann Collet
bf2bc112bb bench : controlled display update when loading lot of files 2016-08-02 23:48:13 +02:00
Yann Collet
346efccc35 fixed doc typo 2016-08-02 14:26:00 +02:00
Yann Collet
f116e87f59 fixed analyzer warning 2016-08-01 19:15:18 +02:00
Yann Collet
9ba929f1d4 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-08-01 02:26:59 +02:00
Yann Collet
3ca750372d updated doc (#269) 2016-08-01 02:26:20 +02:00
Yann Collet
c55eb18c11 Merge pull request #267 from inikep/dev08
fixed ZSTD_compressBlock_opt_extDict_generic
2016-07-31 22:00:16 +02:00
inikep
056df510aa fixed ZSTD_compressBlock_opt_extDict_generic 2016-07-31 20:08:53 +02:00
Yann Collet
917fe188f1 Implemented repOffset "minus 1" on ll==0 2016-07-31 04:01:57 +02:00
Yann Collet
2a2ba3691c Merge pull request #266 from jrmarino/master
Enable build on FreeBSD ports (includes DragonFly BSD) [dev branch]
2016-07-31 02:21:26 +02:00
jrmarino
0d07ec0c0c Enable build on FreeBSD ports (includes DragonFly BSD)
Zstd has been introduced to FreeBSD ports
(http://www.freshports.org/archivers/zstd/) which DragonFly BSD also
uses.  FreeBSD and DragonFly use the install targets (albeit modified in
some cases) so they must be added to the associated Makefile filters.
2016-07-30 19:11:15 -05:00
Yann Collet
8cebfd1d26 fix attempt on test-zstd-speed 2016-07-31 01:59:23 +02:00
Yann Collet
66f69e58d2 restore decompression speed on fizzle 2016-07-30 15:32:47 +02:00
Yann Collet
3b2bd1d11c zstd_opt uses same tables as zstd_compress 2016-07-30 13:21:41 +02:00
Yann Collet
f714f59c16 fixed visual warning 2016-07-30 12:05:28 +02:00
Yann Collet
761f8dbbd2 back to normal table cell copy 2016-07-30 11:43:53 +02:00
Yann Collet
3c6b808870 minor decompression speed gains 2016-07-30 03:20:47 +02:00
Yann Collet
70a9ff4af3 fixed too large selectivity level, reported by Ilona Papava 2016-07-30 01:09:14 +02:00
Yann Collet
c0ce4f1211 slightly improved compression speed 2016-07-30 00:55:13 +02:00
Yann Collet
ed57d8530a new seqStore 2016-07-29 21:22:17 +02:00
Yann Collet
6b615d32cd Updated API comments, following suggestions by Bryan O'Sullivan 2016-07-29 19:40:37 +02:00
Yann Collet
c00d30fbe4 Merge pull request #264 from inikep/dev08
Dev08
2016-07-29 17:42:30 +02:00
inikep
6b68ba2079 zstd_opt.h: fixed checking of rep codes (2) 2016-07-29 16:45:39 +02:00
inikep
59b86fc141 zstd_opt.h: fixed checking of rep codes 2016-07-29 11:00:33 +02:00
Yann Collet
6a82f0f8bf minor comments 2016-07-29 00:55:45 +02:00
Yann Collet
ffa7d0ac1e clarified comment 2016-07-28 21:01:17 +02:00
Yann Collet
4c5bbf64f9 fixed : frame concatenation without checksum 2016-07-28 20:30:25 +02:00
Yann Collet
60ba31c570 zbuff uses ZSTD_compressEnd() 2016-07-28 19:55:09 +02:00
Yann Collet
16e73033ad introduced stage zbf_end 2016-07-28 16:32:34 +02:00
Yann Collet
62470b4bab Changed ZSTD_compressEnd() 2016-07-28 15:29:08 +02:00
Yann Collet
e7bf9156d1 Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎ 2016-07-28 05:00:57 +02:00
Yann Collet
d469a98c01 Clarified API comments, from suggestions by ‎Bryan O'Sullivan‎ 2016-07-28 04:55:03 +02:00
Yann Collet
19c1002e46 applied ZSTD_compressContinueThenEnd() 2016-07-28 01:25:46 +02:00
Yann Collet
5b56739b63 created ZSTD_compressContinueThenEnd() 2016-07-28 01:17:22 +02:00
Yann Collet
c991cc1828 new frame end, 32-bits checksums 2016-07-28 00:55:43 +02:00
Yann Collet
d4180cad9c minor code refactoring 2016-07-27 21:21:36 +02:00
Yann Collet
731ef16fc1 minor code style refactoring 2016-07-27 21:05:12 +02:00
Yann Collet
4b9ca0a6b5 minor example variation 2016-07-27 19:53:19 +02:00
Yann Collet
4110534886 ZSTD_maxCLevel() is promoted to "stable" API (#254, by @FrancescAlted) 2016-07-27 15:09:11 +02:00
Yann Collet
55a8bea0b5 fixed dictionary generation 2016-07-27 14:48:47 +02:00
Yann Collet
c154d9d6a2 better support for large dictionaries (> 128 KB) 2016-07-27 14:37:00 +02:00
Yann Collet
07626dfa51 improved dictbuilder notifications on selectivity 2016-07-27 13:28:46 +02:00
Yann Collet
f796f7ab45 removed fastscan mode 2016-07-27 12:53:54 +02:00
Yann Collet
dd25a27702 added tutorial warning messages for dictBuilder 2016-07-27 12:43:09 +02:00
inikep
003c7a8568 optimal parser: removed ZSTD_REP_INIT 2016-07-27 11:07:13 +02:00
Yann Collet
04cdd8660d Merge pull request #262 from ebiggers/misc_updates
Miscellaneous updates
2016-07-27 01:25:45 +02:00
Eric Biggers
0a55e7a0bb ZSTD_decompressFrame(): use remainingSize instead of iend - ip
Same behavior, but no need to have redundant variables.
2016-07-26 13:22:27 -07:00
Eric Biggers
aa6c70bf60 ZSTD_decompressFrame(): pass up error code from ZSTD_decodeFrameHeader() 2016-07-26 13:22:27 -07:00
Eric Biggers
e4d0265ea9 Replace remaining references to "direct mode" with "single segment mode" 2016-07-26 13:22:27 -07:00
Yann Collet
d50f9db3ea Improved speed on clang and gcc -O2, thanks to @ebiggers ! (#263) 2016-07-26 21:30:35 +02:00
Yann Collet
7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
inikep
4178f5c289 fixed gcc warning: always_inline function might not be inlinable 2016-07-25 21:17:45 +02:00
inikep
fca90f8f60 legacy decoder for v0.7 format 2016-07-25 17:49:08 +02:00
Yann Collet
cbc5e9dc19 fixes oob read 2016-07-24 18:02:04 +02:00
Yann Collet
38b75ddeb2 removed special case all-1 huffman distribution 2016-07-24 15:35:59 +02:00
Yann Collet
7ed5e33b89 minor comment changes 2016-07-24 14:26:11 +02:00
Yann Collet
10b9c13d07 fixed doc on cLevel default, reported by Oliver Lange 2016-07-24 01:21:53 +02:00
Yann Collet
f8e7b5363f unified encoding types 2016-07-23 16:31:49 +02:00
Yann Collet
571a59034a changed enccoding type order : raw, rle, compressed, repeat-stats 2016-07-23 15:52:05 +02:00
Yann Collet
c2e1a68d81 changed streamNb order to 1-4-4-4 2016-07-22 17:30:52 +02:00
Yann Collet
772d912c2f more complete support for literals repeat mode 2016-07-22 15:04:25 +02:00
Yann Collet
9f2d82d4a4 fixed : big-endian decoding 2016-07-22 14:37:10 +02:00
Yann Collet
32faf6c8e7 fixed conversion warnings 2016-07-22 14:37:09 +02:00
Yann Collet
5e45a5fbb3 force loop-align to 32 for zstd_decompress 2016-07-22 14:37:09 +02:00
Yann Collet
5288ac0cb7 changed filed order 2016-07-22 14:37:09 +02:00
Yann Collet
198e6aac44 Literals header fields use little endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
6fa05a2371 cBlockSize uses little-endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
7bf72bbf5e update header to v0.8 2016-07-22 14:37:09 +02:00
Yann Collet
5894ea8d01 updated cLevels 2016-07-22 14:36:46 +02:00
Yann Collet
d5c5a77990 minor comments clarifications 2016-07-20 13:35:14 +02:00
Yann Collet
572b817be3 Merge pull request #253 from gymdis/heapmode_off_legacy_fix
Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0
2016-07-19 13:52:03 +02:00
Christopher Bergqvist
780a9fa857 Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0 2016-07-19 13:25:38 +02:00
Yann Collet
cf05b9d477 ZSTD_getBlockSizeMax() 2016-07-18 16:52:10 +02:00
Yann Collet
16aa38b0e0 minor doc clarifications 2016-07-18 03:52:47 +02:00
Yann Collet
85f3919960 moved zstd.h to /lib 2016-07-17 20:42:21 +02:00
Yann Collet
9375590462 update version to v0.7.5 2016-07-17 16:44:18 +02:00
Yann Collet
e557fd5e92 minor compression level corrections 2016-07-17 16:21:37 +02:00
Yann Collet
d54b2d23b4 minor static assert for 32/64 bits system. Suggested by @ebiggers 2016-07-17 15:53:18 +02:00
Yann Collet
972e5806ee fixed : premature frame end on zero-sized raw block - reported by @ebiggers 2016-07-17 15:39:24 +02:00
luben karavelov
10f999f856 Add legacy support for the low-level streaming API 2016-07-17 01:03:26 +02:00
Yann Collet
6cacd34d44 minor formatting changes 2016-07-15 17:58:13 +02:00
Yann Collet
f6ff53cd4e implemented dictID reserved ranges 2016-07-15 17:03:38 +02:00
Yann Collet
98c8884999 added target zstd in root Makefile 2016-07-15 16:12:38 +02:00
Yann Collet
961b6a0e34 ZSTD_compressBlock() limits block size depending on windowLog parameter 2016-07-15 11:58:49 +02:00
Yann Collet
227cc39e15 improved efficiency for large messages with small dictionaries 2016-07-15 11:27:09 +02:00
Yann Collet
ea2ecdc315 fixed issue with small dictionary 2016-07-14 23:27:31 +02:00
Yann Collet
e9ed5cdc94 fixed minor coverity warning 2016-07-14 21:02:57 +02:00
Yann Collet
b23e1ce319 removed debugging traces 2016-07-14 17:46:38 +02:00
Yann Collet
17508f1a16 fixed a few minor coverity warnings 2016-07-14 17:18:20 +02:00
Yann Collet
8847238cac simplified ZSTD_estimateCCtxSize() 2016-07-14 17:05:38 +02:00
Yann Collet
69c2cdb45c fixed conversion warning 2016-07-14 16:52:45 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
3c174f4da9 fixed minor coverity warning 2016-07-13 17:25:53 +02:00
Yann Collet
2b1a3638e6 changed macro name to ZSTDCLI_CLEVEL_DEFAULT 2016-07-13 15:16:00 +02:00
Yann Collet
3c242e79d3 updated compression levels table 2016-07-13 14:56:24 +02:00
Yann Collet
fbc69f8649 changed for #245 2016-07-13 13:52:58 +02:00
Yann Collet
eed2081e55 fixed conversion warning 2016-07-12 15:11:40 +02:00
Yann Collet
a43a854cdb updated paramgrill 2016-07-12 13:42:10 +02:00
Yann Collet
73d74a05b9 fixed dfast strategy 2016-07-12 13:03:48 +02:00
Yann Collet
45dc35628c first version of doubleFast 2016-07-12 09:47:31 +02:00
Yann Collet
d158c35e9f added ZSTD_estimateDCtxSize() 2016-07-11 13:46:25 +02:00
Yann Collet
8e0ee681b8 added ZSTD_sizeofDCtx() 2016-07-11 13:09:52 +02:00
Yann Collet
3ae543ce75 added ZSTD_estimateCCtxSize() 2016-07-11 03:12:17 +02:00
Yann Collet
25c506601c promote ZSTD_getDecompressedSize() to stable API 2016-07-10 01:46:18 +02:00
Yann Collet
3b6ae77e15 comment clarification 2016-07-08 23:42:22 +02:00
Yann Collet
722e14bb65 fixed compilation error in decompression module 2016-07-08 19:22:16 +02:00
Yann Collet
bd10607063 updated spec 2016-07-08 19:16:57 +02:00
Yann Collet
c5fb5b7fcd support offset > 128 MB 2016-07-08 13:13:37 +02:00
Yann Collet
ed3845d3fa introduced ZSTD_WINDOWLOG_MAX_32 (#239), suggested by @GregSlazinski 2016-07-08 12:57:10 +02:00
Yann Collet
26f681451f updated doc 2016-07-08 11:45:08 +02:00
Yann Collet
19c27d27f1 simplified legacy functions, no longer need magic number 2016-07-07 14:40:13 +02:00
Yann Collet
e72efeb0a1 removed "error_public.h" dependency from "zstd.h" 2016-07-07 14:17:40 +02:00
Yann Collet
974f52fc5d Added "dictionary decompression" example 2016-07-07 14:08:00 +02:00
Yann Collet
e09d38e921 removed mem.h dependency from zbuff.h (experimental section) 2016-07-07 13:17:37 +02:00
Yann Collet
f323bf7d32 added : ZSTD_getDecompressedSize() 2016-07-07 13:14:21 +02:00
Yann Collet
52c04fe58f removed mem.h dependency from zstd.h (experimental section) 2016-07-07 11:53:18 +02:00
Yann Collet
f246cf5423 ZSTD_decompress_usingDDict() compatible with Legacy mode 2016-07-06 20:32:27 +02:00
Yann Collet
29652e2618 sample set limitation closer to 2 GB 2016-07-06 16:25:46 +02:00
Yann Collet
99b045b70a dictBuilder protection vs huge sample sets (>2 GB) 2016-07-06 16:12:38 +02:00
Yann Collet
445d49d898 fixed conversion warning 2016-07-06 13:27:22 +02:00
Yann Collet
a295b3170f fixed conversion warning 2016-07-06 13:13:12 +02:00
Yann Collet
517e1ba623 fixed dictBuilder issue with HC levels. Reported by Bartosz Taudul. 2016-07-06 12:35:09 +02:00
Yann Collet
fe07eaa972 simplified ZSTD_decodeSequence() 2016-07-06 02:25:44 +02:00
Yann Collet
9ca73364e6 updated spec 2016-07-05 10:53:38 +02:00
Yann Collet
f9cac7a734 Added GNU separator --, to specifies that all following arguments are necessary file names (and not commands). Suggested by @chipturner (#230) 2016-07-04 18:18:24 +02:00
Yann Collet
23f05ccc6b updated specifications 2016-07-04 16:13:11 +02:00
Yann Collet
d916c908e0 updated doc 2016-07-04 00:42:58 +02:00
Yann Collet
698cb63305 Updated specifications 2016-07-03 18:49:35 +02:00
Yann Collet
d57dffbe76 ZSTD_storeSeq takes an U32 as offset type 2016-07-03 01:48:26 +02:00
Yann Collet
302ff036f6 simplified repcodes for lazy_extDict 2016-07-03 01:28:16 +02:00
Yann Collet
9634f67107 fix lazy parser 2016-07-03 01:23:58 +02:00
Yann Collet
92d75667e4 fix for fast mode 2016-07-03 01:10:53 +02:00
Yann Collet
5e734ad09b revert fix 2016-07-02 23:55:34 +02:00
Yann Collet
0d5bf8f06f fixed risk of segfault on very large files (multiple GB) 2016-07-02 21:39:47 +02:00
Yann Collet
2fa9904844 update specification and comments 2016-07-01 20:55:28 +02:00
Yann Collet
c093208ab8 fix : potential leak (#229) 2016-06-30 14:07:30 +02:00
Yann Collet
6c6e1751f6 use ZSTD_getParams() to simplify code 2016-06-27 15:28:45 +02:00
Yann Collet
3d2cd7f816 Introduced ZSTD_getParams()
bench now uses ZSTD_createCDict_advanced()
2016-06-27 15:12:26 +02:00
Yann Collet
529d9c7dee updated version to v0.7.2 2016-06-27 10:03:10 +02:00
Yann Collet
d4f4e58ee1 fixed ZSTD_decompressBlock() using multiple blocks 2016-06-27 01:31:35 +02:00
Yann Collet
63b5e7a2ea Improved comments 2016-06-26 17:42:15 +02:00
Yann Collet
3755eb8fea fixed strict-aliasing warning on gcc6 2016-06-22 13:15:53 +02:00
Yann Collet
23042929da Fixed : dictBuilder fails if first sample is too small 2016-06-22 11:05:34 +02:00
Yann Collet
391a128794 fix : segfault in command line during automatic overwrite protection mode 2016-06-21 17:06:25 +02:00
Yann Collet
bda68c253b refactored ZBUFF_compressEnd() for better maintainability 2016-06-21 15:18:11 +02:00
Yann Collet
aa29226b7c fix : ZBUFF_compressEnd() gives right amount remaining to flush, including future epilogue 2016-06-21 14:04:57 +02:00
Yann Collet
f15c1cb00c Fixed : ZBUFF_compressEnd() called multiple times with too small dst buffer (#206) 2016-06-21 13:11:48 +02:00
Yann Collet
a49e066b26 clarified comments on ZSTD_compressContinue() 2016-06-21 11:54:03 +02:00
Yann Collet
d4f38d0dcd updated library to v0.7.1 2016-06-21 10:15:43 +02:00
Yann Collet
22d76322ce minor refactor 2016-06-21 08:01:51 +02:00
Yann Collet
a436a529bc minor : fast_extDict does no longer skip first byte 2016-06-20 23:34:04 +02:00
Yann Collet
4623d11571 new correction, less extreme replacement value 2016-06-20 19:15:37 +02:00
Yann Collet
5477cc25f7 fixed corruption error related to inter-blocks rep-offset 2016-06-20 18:31:25 +02:00
Yann Collet
e4811ba761 Modified : ZSTD_createDDict() accepts dictionary < 8 bytes in pure content mode (reported by @chipturner) 2016-06-19 23:06:54 +02:00
Yann Collet
06d9a73b48 minor refactor, using WILDCOPY_OVERLENGTH macro instead of hard-coded 8 2016-06-19 14:27:21 +02:00
Yann Collet
19cab46f2f Joined seqStore initialization at dispatch point 2016-06-17 12:54:52 +02:00
Yann Collet
510cff3570 minor comment change 2016-06-16 16:39:55 +02:00
Yann Collet
4948f270b3 make room for reserved "information bit" in frame header 2016-06-16 15:38:51 +02:00
Yann Collet
23ba41533a Fixed zstd_opt encoding error with repeat-offsets 2016-06-16 13:20:46 +02:00
Yann Collet
80d033fb43 fixed ptr arithmetic warning 2016-06-16 01:41:50 +02:00
Yann Collet
ad39b7a718 zdict stores standard rep-offset.
It can use custom ones, but the proper formula and impact on statistics is not done yet.
2016-06-16 01:14:41 +02:00
Yann Collet
736d419289 strengthened dict loading on decompresson side 2016-06-16 01:05:04 +02:00
Yann Collet
8e36a9c169 decoder restores repOffsets from dictionary 2016-06-16 01:05:04 +02:00
Yann Collet
52a0622beb RepsCodes are saved into Dict
(uncomplete : need decompression to regenerate them)
2016-06-16 01:05:04 +02:00
Yann Collet
efd0b4993a fixed fuzzer error (inter-block repeated offsets) 2016-06-16 00:53:56 +02:00
Yann Collet
9b998e4d08 Fixed decompression of literals in dictionary mode 2016-06-15 23:11:20 +02:00
Yann Collet
d059092897 fixed conversion warnings 2016-06-14 15:34:24 +02:00
Yann Collet
45c03c564f fixed corruption with inter-blocks repeated offsets 2016-06-14 13:46:11 +02:00
Yann Collet
4266c0a2fd adding inter-blocks rep-offsets 2016-06-14 01:49:25 +02:00
Yann Collet
43dfe01919 Check repIndex for validity 2016-06-13 21:43:06 +02:00
Yann Collet
18c8f79f3e fixed gcc warning on uninitialized structure variable 2016-06-12 22:51:52 +02:00
Yann Collet
cd98f93cff Fixed decompression issue with invalid data 2016-06-11 23:26:22 +02:00
Yann Collet
37fece22e8 enable repeat-entropic-stats mode 2016-06-11 02:52:42 +02:00
Yann Collet
d60a5bf900 Literal decompression builds Huffman tables within shared space (for later re-use) 2016-06-11 02:35:31 +02:00
Yann Collet
237ad4beb3 Added single-stream decompression variant using external DTable 2016-06-11 01:46:03 +02:00
Yann Collet
289bbd52e5 Updated huff0 2016-06-11 01:31:54 +02:00
Yann Collet
1869f7966e Merge pull request #205 from inikep/dev
legacy decoder for v0.6
2016-06-10 17:13:07 +02:00