AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,105 Commits 1 Branch 0 Tags 27 MiB
9822f97721
Commit Graph

223 Commits

Author SHA1 Message Date
Yann Collet
6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet
9f95e445ab minor comment clarifications 2017-05-16 17:26:43 -07:00
Yann Collet
0bdb575c31 Merge branch 'dev' into advancedAPI2 2017-05-16 16:32:29 -07:00
Yann Collet
bfff8999c5 added prototype ZSTD_versionString() 2017-05-16 16:12:23 -07:00
Yann Collet
4eff8136aa added prototype ZSTD_decompressBegin_usingDDict (#700) 2017-05-16 16:05:27 -07:00
Yann Collet
a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1) 
this makes it possible to use `0` to mean:
"do not change strategy"
 2017-05-12 16:29:19 -07:00
Yann Collet
add66f816d changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams() 
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
 2017-05-12 16:01:15 -07:00
Yann Collet
b0edb7fb0e added ZSTD_CCtx_setParameter() 2017-05-12 15:31:53 -07:00
Yann Collet
db8e21d5a0 made ZSTD_compress_generic() definition accessible 
note that the implementation is not done yet.
 2017-05-12 13:46:49 -07:00
Yann Collet
33eb7ac6b6 updated Advanced API proposal 
only declarations in zstd.h
 2017-05-12 12:36:11 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt 
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
 2017-05-10 16:49:58 -07:00
Yann Collet
cef02d9317 changed variable name, for clarity 
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
 2017-05-10 11:14:08 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader 
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
 2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7 regroup memory usage function declarations 
in a single paragraph in zstd.h, for clarity
 2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() 
for clarity
 2017-05-08 18:24:16 -07:00
Yann Collet
51652522a2 bumped version number 2017-05-08 17:52:46 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() 
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
 2017-05-08 17:51:49 -07:00
Yann Collet
fc5145955a updated ZSTD_estimateCCtxSize() 
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
 2017-05-08 17:07:59 -07:00
Yann Collet
791d744279 Updated ZSTD_sizeof_CCtx() 
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
 2017-05-08 16:17:30 -07:00
Yann Collet
0be6fd3429 merged CCtx and CStream as a single same object 
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
 2017-05-08 16:08:01 -07:00
Yann Collet
a51cab6e68 Merge pull request #678 from facebook/apiChange 
Breaking API Change around CDict
 2017-04-28 10:02:45 -07:00
Yann Collet
a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet
8b669535f8 bumped version number to v1.2.0 2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50 added ZSTD_initCStream_usingCDict_advanced() 2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Yann Collet
768df129d2 changed ZSTD_compressBegin_usingCDict() 
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
 2017-04-26 15:42:10 -07:00
Yann Collet
71ddeb67b1 made room in workspace for FSE tables 
still need to be transferred from CCtx into workspace
 2017-04-20 22:54:54 -07:00
Yann Collet
715b9aa113 created ZSTD_compressBegin_usingCDict_advanced() 2017-04-18 13:55:53 -07:00
Yann Collet
af4f45b682 Improved code comments for block functions 2017-04-18 03:17:44 -07:00
Yann Collet
4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict() 
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
 2017-04-17 18:29:06 -07:00
Yann Collet
36c2a03757 updated comments for ZSTD_resetCStream() 2017-04-05 22:06:21 -07:00
Yann Collet
7cf78f1be7 Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference 
Will issue an error (GENERIC) is cdict==NULL
 2017-04-04 12:38:14 -07:00
Yann Collet
f3dfcdccd1 bump version number 2017-03-21 12:18:28 -07:00
Sean Purcell
dec2b96536 Add functions missing from manual, and fix parameter alignment 2017-03-14 11:24:09 -07:00
Nick Terrell
eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Nick Terrell
e65aab8e0f Remove 'mem.h' dependency from ZSTD_WINDOWLOG_MAX 2017-03-08 15:40:13 -08:00
Sean Purcell
d44703d145 Offsets >= 32MB in 32-bits mode 2017-03-01 16:27:56 -08:00
Yann Collet
14312d833e zstdmt : fix : loading prefix from previous segments 
There used to be a (very small) chance that
loading prefix from previous segment
would be confused with a real zstd dictionary.
For that to happen, the prefix needs to start
with the same value as dictionary magic.
That's 1 chance in 4 billions if all values have equal probability.
But in fact, since some values are more common (0x00000000 for example)
others are less common, and dictionary magic was selected to be one of them,
so probabilities are likely even lower.

Anyway, this risk is no down to zero
by adding a new CCtx parameter : ZSTD_p_forceRawDict

Current parameter policy : the parameter "stick" to its CCtx,
so any dictionary loading after ZSTD_p_forceRawDict is set
will be loaded in "raw" ("content only") mode,
even if CCtx is re-used multiple times with multiple different dictionary.
It's up to the user to reset this value differently if it needs so.
 2017-02-23 23:42:12 -08:00
Yann Collet
831b4890ce minor tests/Makefile refactoring 
and update of zstd_manual,html
 2017-02-23 23:09:10 -08:00
Sean Purcell
64417cd2ff Describe ambiguity around skippable frames 2017-02-22 13:29:01 -08:00
Sean Purcell
9757cc811b Update comment 2017-02-22 12:28:21 -08:00
Sean Purcell
9050e1925e Change name to to findFrameCompressedSize and add skippable support 2017-02-22 12:12:34 -08:00
Anders Oleson
517577bf53 spelling fixes in comments 
i.e. occurred labeled Huffman
 2017-02-20 12:08:59 -08:00
Yann Collet
58af614ef2 push version and NEWS to v1.1.4 2017-02-13 18:32:44 -08:00
Sean Purcell
d7bfcac18a Expose frameSrcSize to experimental API 2017-02-10 11:55:44 -08:00
Sean Purcell
269b2cd3d8 Documentation updates 2017-02-09 13:25:30 -08:00
Sean Purcell
2db7249265 Make pledgedSrcSize meaning clear for other functions 
- Added tests
- Moved new size functions to static link only
 2017-02-09 11:49:58 -08:00
Sean Purcell
0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Yann Collet
ef33d00532 fixed : ZSTD_setCCtxParameter() properly exposed in DLL 2017-01-26 12:24:21 -08:00
Yann Collet
4a62f79ec9 fixed clang documentation warning 2017-01-26 09:16:56 -08:00
Yann Collet
06e7697f96 added test of new parameter ZSTD_p_forceWindow 2017-01-25 16:39:03 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet
c593348722 ZSTDMT_initCStream_usingDict() can outlive dict 
Like ZSTD_initCStream_usingDict(),
ZSTDMT_initCStream_usingDict() now keep a copy of dict internally.
This way, dict can be released :
it does not longer have to outlive all future compression sessions.
 2017-01-22 16:44:15 -08:00
Yann Collet
d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
Yann Collet
dab5ea93f2 Merge pull request #515 from iburinoc/emptydict 
Don't create dict in streaming apis if dictSize == 0
 2017-01-19 09:02:42 -08:00
Sean Purcell
0b5370ae38 Prefix notes with /**< 2017-01-18 13:45:02 -08:00
Sean Purcell
57d423c5df Don't create dict in streaming apis if dictSize == 0 2017-01-17 14:31:35 -08:00
Przemyslaw Skibinski
d72f4b6b7a added "Makefile is validated" 2017-01-17 12:40:06 +01:00
Yann Collet
5eb749e734 ZSTDMT_compress() creates a single frame 
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
 2017-01-11 18:21:25 +01:00
Yann Collet
0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet
ba75e9d8c3 fix : zlib wrapper compile in gnu90 mode 2016-12-21 19:57:18 +01:00
Yann Collet
4e5eea61a8 added ZSTD_createDDict_byReference() 2016-12-21 16:44:35 +01:00
Yann Collet
1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Yann Collet
d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Nick Terrell
8de46ab51a Export all API functions 2016-12-16 13:27:30 -08:00
Yann Collet
2b36b238d3 changed variable name to estimatedSrcSize, to emphasize it does not need to be exact 2016-12-13 17:59:55 +01:00
Yann Collet
e795c8a5f6 Added ZSTD_initCStream_srcSize(). 
Added relevant test cases in zstreamtest
 2016-12-13 17:00:14 +01:00
Przemyslaw Skibinski
7687913178 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-12-08 10:42:42 +01:00
Yann Collet
426a9d4b71 changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal. 2016-12-07 16:39:34 -08:00
Przemyslaw Skibinski
4da53219a0 zstd Manual updated to 1.1.2 2016-12-07 11:18:40 +01:00
Yann Collet
379908be3d fixed zstd.h for manual 2016-12-06 10:36:15 -08:00
Yann Collet
825dffbc43 moved zbuff source files into lib/deprecated 2016-12-05 19:28:19 -08:00
Yann Collet
e7a41a5955 added : dictID retrieval functions. 
added : unit tests for dictID retrieval functions
 2016-12-05 16:21:06 -08:00
Yann Collet
9ffbeea875 API : changed : streaming decompression : implicit reset on starting new frames 2016-12-02 18:37:38 -08:00
Nick Terrell
05c00f2ff7 Fix ZSTD_STATIC_LINKING_ONLY with double include 2016-11-29 11:54:39 -08:00
Przemyslaw Skibinski
179555c1d1 working fullbench-dll 2016-11-15 18:05:46 +01:00
Yann Collet
fd3be6bc97 bump version number to 1.1.2 2016-11-07 14:35:41 -08:00
Yann Collet
11812260d1 Merge pull request #439 from terrelln/dev 
ZSTD_compress_usingDict() specify dict == NULL
 2016-11-03 14:15:36 -07:00
Nick Terrell
d82efd8a70 ZSTD_compress_usingDict() when dict gets loaded 
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
 2016-11-02 18:07:16 -07:00
Yann Collet
179b19776f fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h 
Added : ZSTD_isFrame() in experimental section
 2016-11-02 17:30:49 -07:00
Yann Collet
ee5b725823 ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used 2016-10-27 14:20:55 -07:00
Yann Collet
9516234e67 first sketch for ZSTD_initCStream_usingCDict() 2016-10-25 16:19:52 -07:00
Yann Collet
62d9a7ddfd Merge pull request #429 from inikep/btopt2 
Btopt2
 2016-10-25 14:48:43 -07:00
Przemyslaw Skibinski
5c5f01f3da added ZSTD_btopt2 strategy 2016-10-25 12:25:07 +02:00
Yann Collet
37d130031d updated comments on context re-use 2016-10-24 17:22:12 -07:00
Przemyslaw Skibinski
984b66cd72 added contrib\gen_html 2016-10-24 15:59:51 +02:00
Przemyslaw Skibinski
575ab00db7 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-10-20 11:01:52 +02:00
Yann Collet
f52cd03e73 bumped version number 2016-10-11 17:29:27 -07:00
inikep
2d2613399a zstd.h: added the Introduction section 2016-10-06 16:28:21 +02:00
inikep
ba1db376ac fixed clang 3.5 warnings 2016-10-06 14:22:48 +02:00
inikep
82057aa7ec zstd.h: updated comments 2016-10-06 13:23:52 +02:00
Yann Collet
cf409a7e2a fixed : init*_advanced() followed by reset() with different pledgedSrcSiz 2016-09-26 16:41:05 +02:00
Yann Collet
97b378a6f8 Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize. 2016-09-21 17:20:19 +02:00
Yann Collet
1eb2fdc74f bumped version number 2016-09-18 12:21:47 +02:00
Yann Collet
4cb212938c introduced ZSTD_resetCStream() 2016-09-15 14:54:07 +02:00
Yann Collet
d7c6589df8 support ZSTD_sizeof_*() on NULL 
added ZSTD_sizeof_CDict()
 2016-09-15 02:57:27 +02:00