AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,034 Commits 1 Branch 0 Tags 27 MiB
8a9741b3ee
Commit Graph

260 Commits

Author SHA1 Message Date
Yann Collet
fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet
2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet
114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet
19e5f2a35b removed some constants and _simpleArgs() from staging 
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
 2018-11-19 17:38:15 -08:00
Yann Collet
5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet
4542b9f1cf updated manual 2018-11-15 10:52:24 -08:00
Yann Collet
21a42bf5f9 added advanced decompression api 2018-11-14 16:54:54 -08:00
Yann Collet
cf9f4b63b8 fixed fuzz test src code 2018-11-14 14:46:49 -08:00
Yann Collet
7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet
2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK 
and updated several API comments
 2018-11-13 17:36:35 -08:00
Yann Collet
b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet
768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet
092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet
fc20b3c441 added flag -Wc++-compat 
for library and cli
 2018-10-26 16:38:23 -07:00
Yann Collet
9bf00707c7 minor clarifications of history update rules 2018-10-26 15:51:51 -07:00
Ulrich Kunitz
f0fe9b0f02 Reverted removal of a trailing space. 
My editor removes trailing spaces while saving. Not confusing things I
reverted that change.
 2018-10-23 08:43:19 +02:00
Ulrich Kunitz
4f702e4445 Fixed a typo 
I fixed a typo in the last commit. Many thanks to @terrelin for pointing
that out.
 2018-10-23 08:36:50 +02:00
Ulrich Kunitz
c7942caff0 Clarify special case of offset history update 
If the current sequence has literal length of zero then an offset value
of three is handled in a special manner. While I implemented a golang
decoder I had to consult the educational decoder for clarification on
the update of the offset history in that case. This commit provides the
clarification that the offset value Repeated_Offset1-1 is handled as a
new offset is added to the offset history accordingly.
 2018-10-22 23:46:43 +02:00
Yann Collet
1e6208e75e bumped version number to v1.3.7 
updated documentation
 2018-10-11 14:40:12 -07:00
Yann Collet
68bec4c5bb added graph for ZSTD_compress_usingCDict() in v1.3.6 2018-10-04 14:39:11 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt 
Adaptive compression
 2018-09-26 14:39:08 -07:00
Yann Collet
72a3adf826 updated format documentation 
to match last edits of RFC8478.
 2018-09-25 16:34:26 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
a54c86cfc6 defined a minimum negative level 
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
 2018-09-20 16:52:03 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Yann Collet
005f000aed updated documentation of *refPrefix() 
indicating the equivalence with `diff` operation.
 2018-09-18 13:07:08 -07:00
Yann Collet
7269fe6cd3 minor code comment update 2018-09-14 16:06:35 -07:00
Yann Collet
0403148315
Merge pull request #1295 from felixhandte/hdr-intro-comment-negative-lvls 
Proposed Update to Zstd.h Introduction Comment
 2018-09-14 15:29:19 -07:00
Yann Collet
674dd21bd0 final parameter tuning 2018-09-12 17:25:34 -07:00
Yann Collet
55a8f84a2c spec clarification 
following #1305 comments from @ulikunitz
 2018-09-05 12:31:33 -07:00
W. Felix Handte
8e944ccade Regenerate Documentation 2018-08-22 17:07:34 -07:00
Yann Collet
33f7709c71 fileio: changed parameter type from ptr to plain structure 
safer : this parameter is read-only,
we don't want original structure to be modified
 2018-08-13 13:02:03 -07:00
Nick Terrell
c1a7defee1 Small fixes to zstd specification 
Update to keep in sync with the RFC.
 2018-07-10 15:07:36 -07:00
Yann Collet
a55ffbb31b removed unused imaged 2018-06-27 16:03:46 -07:00
Yann Collet
c0007893b4 added graph for cdict speed v1.3.5 improvements 2018-06-27 15:58:34 -07:00
Yann Collet
4489daec09 slightly adjusted default-distribution threshold 
depending on strategy.
fast favors faster compression and decompression speeds.
 2018-06-26 20:10:45 -07:00
Yann Collet
d70c4a5074
Merge pull request #1201 from facebook/rfcUpdate 
updated Zstandard frame format
 2018-06-22 11:53:50 -07:00
Yann Collet
c1e6347717 fixed minor typos, detected by @terrelln 2018-06-21 18:08:11 -07:00
Yann Collet
7639db939f updated Zstandard frame format 
adding clarifications from IETF RFC DISCUSS.
 2018-06-21 17:55:55 -07:00
Yann Collet
2d76defbfe grouped all histogram functions into hist.c 
renamed functions with HIST_* prefix
 2018-06-13 19:49:31 -04:00
Yann Collet
a57b4df85f removed literalCompression directive 
in this version, literal compression is always disabled for ZSTD_fast strategy.

Performance parity between ZSTD_compress_advanced() and ZSTD_compress_generic()
 2018-06-07 15:24:12 -07:00
Yann Collet
24319975b6 bumped version number to v1.3.5 2018-06-06 15:51:55 -07:00
Yann Collet
830fd4468f better make -j all behavior 
avoid concurrent compilation of libzstd
 2018-06-06 12:47:16 -07:00
Yann Collet
a4c9c4defe update Zstandard format specification 
answering a few questions from IETF RFC Discuss stage.
 2018-05-31 10:47:44 -07:00
Nick Terrell
73f4c890cd Clarify what happens when Number_of_Sequences == 0 2018-05-22 16:12:33 -07:00
Yann Collet
82ad249645 Clarifications of Zstandard format specification 
from IETF RFC review
 2018-04-30 12:36:55 -07:00
Yann Collet
12f60b8c98 clarified documentation related to refPrefix() 2018-04-25 10:17:06 -07:00
Yann Collet
ace856a835 updated documentation of streaming compression api 2018-04-24 14:44:27 -07:00
Nick Terrell
6bb6e6178a Add zstdmt long range mode graphs 2018-03-23 14:06:05 -07:00
Yann Collet
d6c6b62958 updated graph for dictionary compression 2018-03-22 18:44:56 -07:00
Yann Collet
184e1c8b8c updated benchmark figures 2018-03-22 18:22:10 -07:00
Yann Collet
b7b80e1569 fixed paramgrill 2018-03-21 16:09:15 -07:00
Yann Collet
a99c4a3621 Merge branch 'dev' into advancedDecompress 2018-03-21 06:08:28 -07:00
Yann Collet
6873fec658 changed dictMore for dictContentType 
which seems clearer to describe what the variable/argument is about.
 2018-03-20 15:13:14 -07:00
Yann Collet
31b54b6eea updated ZSTD_initStaticDDict() prototype 
can also specify dictContentType.
 2018-03-20 14:52:02 -07:00
Yann Collet
353117c5d7 implemented ZSTD_DCtx_loadDictionary*() 
this required updating ZSTD_createDDict_advanced()
to accept a dictContentType parameter (raw, full, auto).
 2018-03-20 13:40:29 -07:00
Yann Collet
c8b3d389fd updated CCtxParams API 
to respect naming convention :
ZSTD_CCtxParams_*()
 2018-03-19 15:07:26 -07:00
Yann Collet
97816400ca added /contrib projects to make all 2018-03-15 16:40:14 -07:00
Yann Collet
2291b85a1e changed ZSTD_p_literalCompression into ZSTD_p_compressLiterals 
prefer verb+object construction
 2018-03-12 11:44:10 -07:00
Yann Collet
a57d43d4d4 updated documentation of targetLength 2018-03-12 11:35:01 -07:00
Yann Collet
564cb1b640 update doc/README.md on provided tools to test 3rd party implementations 2018-02-27 17:37:05 -08:00
Yann Collet
9945e60ac4 Merge branch 'dev' into flexibleLevel 2018-02-10 11:54:49 -08:00
Yann Collet
209df52ba2 Changed nbThreads for nbWorkers 
This makes it easier to explain that nbWorkers=0 --> single-threaded mode,
while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread).
No need for an additional asynchronous mode flag.
nbWorkers>=2 works the same as nbThreads>=2 previously.
 2018-02-01 19:29:30 -08:00
Yann Collet
823a28a1f4
Merge pull request #1000 from facebook/progressiveFlush 
Progressive flush
 2018-01-30 22:49:47 -08:00
Yann Collet
b173b8d8da updated zstd api manual 2018-01-30 15:05:12 -08:00
Yann Collet
2cb0740b6b zstdmt: changed naming convention 
to avoid confusion with blocks.

also:
- jobs are cut into chunks of 512KB now, to reduce nb of mutex calls.
- fix function declaration ZSTD_getBlockSizeMax()
- fix outdated comment
 2018-01-30 14:43:36 -08:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4 
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
 2018-01-27 22:23:26 -08:00
Yann Collet
8e128eaf05 zstdmt : refactor job members 
grouped by sharing properties
 2018-01-26 10:20:38 -08:00
Yann Collet
997e4d0ccd added POOL_tryAdd() 2018-01-18 14:39:51 -08:00
Yann Collet
394eec697b Introduce ZSTD_getFrameProgression() 
Produces 3 statistics for ongoing frame compression :
- ingested
- consumed (effectively compressed)
- produced

Ingested can be larger than consumed due to buffering effect.

For the time being, this patch mostly fixes the % ratio issue,
since it computes consumed / produced,
instead of ingested / produced.

That being said, update is not "smooth",
because on a slow enough setting,
fileio spends most of its time waiting for a worker to complete its job.

This could be improved thanks to more granular flushing
i.e. start flushing before ongoing job is fully completed.
 2018-01-17 16:39:02 -08:00
Shawn Landden
914d983879
fix unbounded range 
I think you meant 8 MiB or smaller, instead of an unbounded (and illogical) range
 2017-12-21 16:15:12 -08:00
Yann Collet
78de28239f minor readme formatting update 2017-12-15 13:26:39 -08:00
Yann Collet
2cff66b62f version bump to v1.3.3 2017-12-14 16:11:20 -08:00
Yann Collet
bd88f633ac zstreamtest : in -T#s, s considered a suffix meaning "seconds" 
avoid unintentionnally triggering `seedset`,
so that seed gets automatically determined when not set.
 2017-11-27 12:15:23 -08:00
Yann Collet
4154aec679 fixed comment, as suggested by @terrelln 2017-11-21 10:26:17 -08:00
Yann Collet
fccb46fbe0 minor spelling fixes 2017-11-18 11:28:00 -08:00
Yann Collet
ca1a9ebac5 fixed zlib wrapper 
it was invoking ZSTD_initCStream_advanced() with pledgedSrcSize==0 and contentSizeFlag=1
which means "empty"
while the intention was to mean "unknown".

The contentSizeFlag==1 is new, it is a consequence of setting this value to 1 by default.

The solution selected here is to pass ZSTD_CONTENTSIZE_UNKNOWN to mean "unknown".
So contentSizeFlag remains set (it wasn't in previous versions).
 2017-10-18 11:22:23 -07:00
Yann Collet
5eed8e7a55 changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN 
to mean "pledgedSrcSize is not known at init time" instead of `0`.
Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown,
to avoid breaking 3rd party applications which depend on this behavior.
But this value is no longer recommended to mean "unknown".

In some future version, it might be possible to switch "0" to mean "empty",
as is already the case for several prototypes.
The advantage is that pledgedSrcSize field would have same behavior accross entire API,
making it easier to reason about.

Note that all concerned prototypes belong to the "experimental" API section.

srcSize is controlled at end of compression,
so if someone uses "0" to mean "unknown" while it effectively means "empty",
this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong
 2017-10-14 00:32:06 -07:00
Nick Terrell
6d8778d8e7 [doc] Add images for release notes 2017-10-09 16:44:31 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus 
a compressed block must be strictly smaller than its decompressed size.
 2017-10-07 15:19:52 -07:00
Yann Collet
ecf1778e23 updated ZSTD_format_e value validation 
also updated manual
 2017-09-27 11:19:21 -07:00
Yann Collet
52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet
f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet
6ee05a02b8 added ZSTD_decompress_generic() 
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
 2017-09-25 15:41:48 -07:00
Yann Collet
96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce added prototypes for advanced parameters for decompression API 
required to decode custom formats
 2017-09-24 15:57:29 -07:00
Yann Collet
086b9597d9 added ability to split input files for dictionary training 
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
 2017-09-14 16:45:10 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet
ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min() 
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
 2017-09-09 01:03:29 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau
8c33cfe0bc Add ldm documentation in README 2017-09-06 15:21:01 -07:00
Yann Collet
0c314cde4b updated zstd API manual 
for new CCtxParams object
 2017-08-31 18:28:19 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00