Commit Graph

1475 Commits

Author SHA1 Message Date
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
2017-05-10 16:49:58 -07:00
Yann Collet
ba41b26405 Merge pull request #689 from facebook/cctxMerge
Cctx merge
2017-05-10 14:53:54 -07:00
Yann Collet
cef02d9317 changed variable name, for clarity
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
Yann Collet
669346fe8b fixed ZSTD_estimateCStreamSize()
https://github.com/facebook/zstd/pull/689#discussion_r115637721
2017-05-10 11:08:00 -07:00
Yann Collet
6fb2f24132 shortened ZSTD_createCStream_Advanced()
https://github.com/facebook/zstd/pull/689#discussion_r115637613
2017-05-10 11:06:06 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7 regroup memory usage function declarations
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
for clarity
2017-05-08 18:24:16 -07:00
Yann Collet
51652522a2 bumped version number 2017-05-08 17:52:46 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet
7855366598 Updated ZSTD_freeCCtx()
which can also contain streaming buffers now.
Redirected ZSTD_freeCStream() towards it.
2017-05-08 17:15:00 -07:00
Yann Collet
fc5145955a updated ZSTD_estimateCCtxSize()
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
Yann Collet
791d744279 Updated ZSTD_sizeof_CCtx()
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
2017-05-08 16:17:30 -07:00
Yann Collet
0be6fd3429 merged CCtx and CStream as a single same object
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet
d47709b6ea Merge pull request #654 from iburinoc/splittable
[RFC] Splittable Format and API
2017-05-08 13:41:56 -07:00
Yann Collet
a00e9599f1 removed -g from DEBUGFLAGS
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
2017-05-04 17:24:29 -07:00
Yann Collet
606c04c228 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-02 12:13:52 -07:00
Yann Collet
072484a3bf Merge pull request #683 from terrelln/odev
[CLI] Make cover the default dictionary builder
2017-05-02 12:13:23 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell
865918dd04 Fix typo in zdict.h 2017-05-02 11:02:37 -07:00
Yann Collet
b184589c4c minor code refactoring for clarity 2017-05-01 11:35:47 -07:00
Yann Collet
33c38b0925 fixed const in prototype, that Visual doesn't accept 2017-05-01 11:12:30 -07:00
Yann Collet
f39a6731ec sync bitstream.h from fse library 2017-05-01 09:56:03 -07:00
Yann Collet
202082f285 sync bitstream from FSE project
add assert into unsafe *_fast() variants
2017-04-28 17:00:31 -07:00
Yann Collet
89f50deec7 minor code refactoring
clearer tables
2017-04-28 16:52:36 -07:00
Yann Collet
68a7d3d49a added HUF_PUBLIC_API macro to huf.h
to make it possible to control symbol visibility.
Also : better separation and comments between "public" and "static" sections
2017-04-28 12:46:48 -07:00
Yann Collet
a51cab6e68 Merge pull request #678 from facebook/apiChange
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
Yann Collet
29297c6751 Changed default level 18 (large input)
Previous -18 : 4.7 MB/s, R:3.833
New -18 : 5.1 MB/s. R:3.825

It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s)
The new level 18 also uses significantly less memory.
And, it makes a good transition between level 17 (mml5)
and level 19 (mml3).
Up to now, there was no level with mml4.

(note : minmatch setting can have a large impact on some (specific) datasets)
2017-04-27 17:44:01 -07:00
Yann Collet
a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet
d3694e6c70 removed C4204 2017-04-27 14:29:35 -07:00
Yann Collet
1c3ab0c77f fixed init error on Visual 2008 2017-04-27 12:57:11 -07:00
Yann Collet
8b669535f8 bumped version number to v1.2.0 2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50 added ZSTD_initCStream_usingCDict_advanced() 2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet
69a54d138a fixed compilation warning : declaration-after-statement 2017-04-27 01:11:26 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced()
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Yann Collet
768df129d2 changed ZSTD_compressBegin_usingCDict()
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
2017-04-26 15:42:10 -07:00
Yann Collet
e42afbc6fa Comply with suggested comments by @terrelln
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
2017-04-26 11:39:35 -07:00
Sean Purcell
7d37ca1d5b Merge remote-tracking branch 'origin/dev' into splittable 2017-04-21 14:18:39 -07:00
Yann Collet
7271203bdb transferred entropy scratch space from CCtx into workSpace
Saved 6 KB
2017-04-20 23:21:19 -07:00
Yann Collet
a408645f50 made some room for entropy scratch space 2017-04-20 23:09:39 -07:00
Yann Collet
71aaa32c3c transferred FSE tables from CCtx into workspace
Saved 5 KB from CCtx
2017-04-20 23:03:38 -07:00
Yann Collet
71ddeb67b1 made room in workspace for FSE tables
still need to be transferred from CCtx into workspace
2017-04-20 22:54:54 -07:00
Yann Collet
a34a39c183 changed size evaluation of entropy tables
so that memcpy() does no longer depends on fse pointer being a static table
2017-04-20 18:26:25 -07:00
Yann Collet
7bb60b17d8 init entropy table pointers only once
per workSpace resize
2017-04-20 17:38:56 -07:00
Yann Collet
e6fa70a0a1 reorganized ZSTD_resetCCtx_internal()
clearer separation between variables and buffers
clearer buffers category
kept static buffers at the beginning, favoring cache locality
(it will be easier to add FSE tables there later)

This break a few assumptions that hashTable was always at the beginning.
This is fixed.
And remaining assumptions (namely that tables stand next to each other in memory)
are now tested with assert.
2017-04-20 17:28:31 -07:00
Yann Collet
c17e020c9a disable assert when compiling paramgrill
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
2017-04-20 12:50:02 -07:00