Commit Graph

1677 Commits

Author SHA1 Message Date
Yann Collet
e51d51bdf7 fixed memcpy() overlap 2017-06-20 17:44:55 -07:00
Yann Collet
466f92eaa6 removed one useless streaming compression stage, detected by @terrelln 2017-06-20 16:25:29 -07:00
Yann Collet
c3bce24ef4 fixed potential dangling pointer, detected by @terrelln 2017-06-20 16:09:11 -07:00
Yann Collet
78b8234554 fixed comments, following suggestion by @terrelln 2017-06-20 14:26:48 -07:00
Yann Collet
b44ab82f7a ensure new ZSTD_strategy starts at value 1 2017-06-20 14:11:49 -07:00
Yann Collet
c08e649e95 first implementation of bench.c with new API ZSTD_compress_generic()
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.

Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
Yann Collet
695a0a3449 fixed IA64 compilation error, by @mcmilk 2017-06-19 15:27:30 -07:00
Yann Collet
fe234bf48b fix attempts : fullbench for VS2008 2017-06-19 15:23:19 -07:00
Nick Terrell
55f9cd4942 [libzstd] Fix UBSAN failure 2017-06-19 15:12:28 -07:00
Yann Collet
bf99150be3 update new api presentation in zstd.h and manual 2017-06-19 12:56:25 -07:00
Yann Collet
d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Yuri
92bafda406 INSTALL_DATA instead of INSTALL_LIB for libzstd.a
INSTALL_LIB can be passed -s flag to strip symbols. Static libraries should not be stripped, only dynamic ones should be stripped.
2017-06-17 00:23:41 -07:00
Yann Collet
381e66cfbd added ZSTD_clampCParams()
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
Yann Collet
aee916e37c fixed +/-1 error for pledgedSrcSizePlusOne 2017-06-16 17:02:35 -07:00
Yann Collet
d3de3d51a3 fix attempt 2 : Visual sign conversion warning 2017-06-16 16:51:33 -07:00
Yann Collet
944be54774 fixed attempt : minor Visual sign conversion warning 2017-06-16 14:05:01 -07:00
Yann Collet
b26728c9c8 added ZSTD_startNewCompression() 2017-06-16 14:00:46 -07:00
Yann Collet
a0ba849fe6 changed frameContentSize field to pledgedSrcSizePlusOne
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.

PlusOne is meant to have 0 (default) == unknown
2017-06-16 13:29:17 -07:00
Yann Collet
2cf7755da7 fix : pledgedSrcSize correctly reset to unknown in "continue" mode 2017-06-16 12:34:41 -07:00
Yann Collet
9e73f2f320 fix : correctly reset pledgedSrcSize to unknown status
when starting a new compression with an existing context
2017-06-16 12:24:01 -07:00
Yann Collet
33873f0e74 fixed : new advanced AIP : setting nbThreads to the same value > 1 2017-06-16 12:04:21 -07:00
Yann Collet
559ee82e90 fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps 2017-06-16 11:58:21 -07:00
Yann Collet
bd18c885a3 added ZSTD_CCtx_reset 2017-06-16 10:17:50 -07:00
Yann Collet
cc9f9b7f4c protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag 2017-06-15 18:17:34 -07:00
Yann Collet
05ae4b2190 added protection : MT incompatible with Static allocation 2017-06-15 18:03:34 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Yann Collet
f129fd3970 disabled MT code path when ZSTD_MULTITHREAD is not defined 2017-06-11 18:46:09 -07:00
Yann Collet
23aace9778 added control stage to MT mode 2017-06-11 18:32:36 -07:00
Yann Collet
f35e2de61c linked newAPI to ZSTDMT 2017-06-05 18:32:48 -07:00
cyan4973
c59162e053 minor fix for -Wdocumentation 2017-06-05 00:12:13 -07:00
cyan4973
8bcbf42617 fixed g++ prototype mismatch 2017-06-04 23:52:00 -07:00
Yann Collet
8c910d2097 updated ZSTDMT streaming API
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet
8ddf4c22d5 fixed missing initialization 2017-06-02 17:16:49 -07:00
Yann Collet
33a7e679e5 significant zlib wrapper code refactoring
code indentation
variable scope and names
constify

Only coding style changes.
The logic should remain the same.
2017-06-02 17:10:49 -07:00
Yann Collet
4effccbf56 zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing
more generic and safer than using own routing for magic number comparison
2017-06-02 14:27:11 -07:00
Yann Collet
dcb7535352 ensure zlibwrapper uses ZSTD_malloc() and ZSTD_free()
which is compatible with { NULL, NULL, NULL }
2017-06-02 14:01:21 -07:00
Yann Collet
b877e834b1 minor indent 2017-06-02 13:47:11 -07:00
Yann Collet
6056e4c3eb added POOL_sizeof() for single-thread 2017-06-02 11:36:47 -07:00
Yann Collet
c35e535002 added support for multithreading parameters 2017-06-01 18:44:06 -07:00
Yann Collet
c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof()
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
2017-06-01 17:56:14 -07:00
Yann Collet
cd2892fd1e protected impossible switch(){default:} with assert(0)
can be converted into assume(0) in some future
2017-06-01 09:44:54 -07:00
Yann Collet
06589fe516 Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-05-31 10:03:20 -07:00
Yann Collet
18ab5affa5 fixed visual warning 2017-05-31 09:59:22 -07:00
Yann Collet
9a691e0f55 fixed visual warnings 2017-05-31 01:17:44 -07:00
Yann Collet
01b1549f83 finally converted ZSTD_compressStream_generic() to use {in,ou}Buffer
replacing the older read/write variables from ZBUFF_* era.
Mostly to help code readability.

Fixed relevant callers.
2017-05-30 18:10:26 -07:00
Yann Collet
c4f46b94ce ZSTD_createCCtx_advanced() now uses ZSTD_calloc()
initially uses calloc() instead of memset().

Performance improvement is unlikely measurable,
since ZSTD_CCtx is now very small,
with all tables transferred into workSpace.
2017-05-30 17:45:37 -07:00
Yann Collet
deee6e523f expose ZSTD_compress_generic_simpleArgs()
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
2017-05-30 17:42:00 -07:00
Yann Collet
ae728a43b8 removed defaultCustomMem
now ZSTD_customCMem is promoted as new default.

Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.

ZSTD_customCMem is public constant
(defaultCustomMem was private only).

Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.

Fixed zlibwrapper which depended on defaultCustomMem.
2017-05-30 17:11:39 -07:00
Yann Collet
5bcef1ada2 removed mtctx->cstream
use the first cctx in pool when ZSTDMT is used in single-thread mode
now that cctx and cstream are the same object.
2017-05-30 16:37:19 -07:00
Yann Collet
beb62b15a8 Merge branch 'dev' into advancedAPI2
Fixed conflic in zstd_decompress.c
2017-05-30 16:18:57 -07:00
Yann Collet
44e45e8423 added ZSTDMT_createCCtx_advanced()
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
Yann Collet
f45ca527a1 Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-05-30 10:02:03 -07:00
Yann Collet
b6dec4c3ae fixed minor cast warning 2017-05-27 17:09:06 -07:00
Yann Collet
e071159101 mtctx->jobs allocate its own memory space
to make ZSTDMT_CCtx_s size predictable
so that it can be included in CCtx
2017-05-27 00:21:33 -07:00
Yann Collet
b8136f019a static dctx is incompatible with legacy support
documented, and runtime tested
2017-05-27 00:03:08 -07:00
Yann Collet
7028cbd7fd fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader 2017-05-25 18:29:08 -07:00
Yann Collet
cdf7e82222 Added ZSTD_initStaticCDict() 2017-05-25 18:05:49 -07:00
Dmitry V. Levin
1ea655c765 Fix typo in libzstd.a-mt make rules
The macro name is ZSTD_MULTITHREAD, not ZSTD_MULTHREAD.

Fixes: ca6fae7808 ("Add MT enabled targets for libzstd")
2017-05-25 23:43:05 +00:00
Yann Collet
57827f906f added ZSTD_initStaticDDict() 2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet
ba183005d3 merged DStream's inBuff and outBuff into a single buffer
Saves one malloc().
Also : makes it easier to implement static allocation
2017-05-24 15:42:24 -07:00
Nick Terrell
55fc1f91fd [zstd] Fix up formatting edge cases for clang-format 2017-05-24 13:50:10 -07:00
Yann Collet
2e4db3e531 fixed performance regression with ZSTD_decompress() on small files
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.

Fixed by only initialization relevant pointers and size fields.
2017-05-24 13:15:19 -07:00
Yann Collet
11ea2f7fda Merged ZSTD_DCtx and ZSTD_DStream objects
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
2017-05-23 16:19:43 -07:00
Yann Collet
b81f19ffce move MEM_readMINMATCH() into zstd_opt.h
which is its only user.
Use case too narrow to belong to mem.h.
renamed to ZSTD_readMINMATCH()
2017-05-23 15:41:55 -07:00
Yann Collet
c7fe262dc9 added ZSTD_initStaticCCtx()
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
Yann Collet
5ac72b417c Buffered are now allocated inside workSpace 2017-05-23 11:18:24 -07:00
Yann Collet
1880337c30 Simplifier compression call graph
Everything converge towards ZSTD_compressBegin_internal
which delegated to ZSTD_copyCCtx_internal if cdict!=NULL.

This simplifies routing which was previously depending on cdict.
2017-05-22 18:21:51 -07:00
Yann Collet
b0739bcf8f simplified reset by removing full-reset policy
this was meant to be applied prior to dictionary loading.
But effectively, it seems redundant with later loading stage,
so it can be skipped safely.
2017-05-22 17:45:15 -07:00
Yann Collet
1ad7c82eb5 Implemented separation between requested and applied parameters
first version to pass cli tests with -DZSTD_NEWAPI
2017-05-22 17:06:04 -07:00
Yann Collet
24de7b0346 Implemented ZSTD_CCtx_refCDict() 2017-05-22 13:05:45 -07:00
Yann Collet
ee970398b2 Merge branch 'dev' into advancedAPI2 2017-05-22 12:33:56 -07:00
Yann Collet
8b21ec42a9 ZSTD_compress_generic() can handle dictionary compression 2017-05-19 19:46:15 -07:00
Nick Terrell
a1280406b0 [libzstd] Allow users to define custom visibility 2017-05-19 18:01:59 -07:00
Yann Collet
334a288d0d ZSTD_CCtx_setParameter() only works during initialization stage
and generate a stage_wrong error otherwise.
2017-05-19 11:04:41 -07:00
Yann Collet
48855fa0d2 fixed declaration-after-statement warning 2017-05-19 10:56:11 -07:00
Yann Collet
fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
Yann Collet
009d604e00 ZSTD_compress_generic() supports multiple successive frames
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
Yann Collet
6d4fef36de Added ZSTD_compress_generic()
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
Yann Collet
23c256e44b removed useless variable from CCtx
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
Yann Collet
9f95e445ab minor comment clarifications 2017-05-16 17:26:43 -07:00
Yann Collet
0bdb575c31 Merge branch 'dev' into advancedAPI2 2017-05-16 16:32:29 -07:00
Yann Collet
7101434ec9 pedantic : added one error check
on a function which (today) never fails.
But who knows, maybe tomorrow ...
2017-05-16 16:28:24 -07:00
Yann Collet
bfff8999c5 added prototype ZSTD_versionString() 2017-05-16 16:12:23 -07:00
Yann Collet
4eff8136aa added prototype ZSTD_decompressBegin_usingDDict (#700) 2017-05-16 16:05:27 -07:00
Yann Collet
2d4d31c18a removed gcc compilation flag -Wbad-function-cast
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
2017-05-16 11:34:38 -07:00
Yann Collet
133f0aee54 fixed redundant declarations in legacy v0.5 and v0.7 decoders
triggered by new flag -Wredundant-decls
2017-05-15 17:44:04 -07:00
Yann Collet
83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Yann Collet
a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1)
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
Yann Collet
add66f816d changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams()
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
2017-05-12 16:01:15 -07:00
Yann Collet
b0edb7fb0e added ZSTD_CCtx_setParameter() 2017-05-12 15:31:53 -07:00
Yann Collet
ef738c1b23 better error code when compressing using NULL CDict
which is not allowed (but detected, and generates an error).
2017-05-12 13:55:25 -07:00
Yann Collet
db8e21d5a0 made ZSTD_compress_generic() definition accessible
note that the implementation is not done yet.
2017-05-12 13:46:49 -07:00
Yann Collet
33eb7ac6b6 updated Advanced API proposal
only declarations in zstd.h
2017-05-12 12:36:11 -07:00
Yann Collet
bd1964a988 Merge pull request #696 from joscollin/wip-lib-legacy-fallthrough-warn
lib/legacy: warning: this statement may fall through
2017-05-11 10:45:01 -07:00
Yann Collet
4c1cfc0bb6 Merge pull request #695 from joscollin/wip-lib-compress-fallthrough-warn
lib/compress: warning: this statement may fall through
2017-05-11 10:44:27 -07:00
Jos Collin
280510f2d5 lib/legacy: warning: this statement may fall through
The following warning appears during build at sevaral places.

../lib/legacy/zstd_v04.c:819:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
             case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16);

../lib/legacy/zstd_v05.c:821:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
             case 7: bitD->bitContainer += (size_t)(((const BYTE*)(bitD->start))[6]) << (sizeof(size_t)*8 - 16);

../lib/legacy/zstd_v06.c:913:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
             case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);

../lib/legacy/zstd_v07.c:583:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
             case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) <<
             (sizeof(bitD->bitContainer)*8 - 16);

Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 14:27:40 +05:30
Jos Collin
7cd7a7564b lib/compress: warning: this statement may fall through
The following warning appears during build.

../lib/compress/huf_compress.c: In function ‘HUF_compress1X_usingCTable’:
../lib/compress/huf_compress.c:444:8: warning: this statement may fall through [-Wimplicit-fallthrough=]
     if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream)
        ^
../lib/compress/huf_compress.c:465:18: note: in expansion of macro ‘HUF_FLUSHBITS_2’
                  HUF_FLUSHBITS_2(&bitC);
                  ^~~~~~~~~~~~~~~
../lib/compress/huf_compress.c:466:9: note: here
         case 2 : HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);

../lib/compress/zstd_compress.c: In function ‘ZSTD_compressStream_generic’:
../lib/compress/zstd_compress.c:3366:34: warning: this statement may fall through [-Wimplicit-fallthrough=]
                 zcs->streamStage = zcss_flush;   /* pass-through to flush stage */
                 ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
../lib/compress/zstd_compress.c:3369:9: note: here
         case zcss_flush:

Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 13:17:26 +05:30
Jos Collin
05286fdd5a lib/common: warning: this statement may fall through
The following warning appears during the build. Fixed the review comments too.

zstd/lib/common/bitstream.h: In function ‘BIT_initDStream’:
zstd/lib/common/bitstream.h:277:33: warning: this statement may fall through [-Wimplicit-fallthrough=]
      case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) <<
      (sizeof(bitD->bitContainer)*8 - 16);

Signed-off-by: Jos Collin <jcollin@redhat.com>
2017-05-11 09:10:02 +05:30
Nick Terrell
374f868354 Update whitespace 2017-05-10 17:48:42 -07:00
Nick Terrell
5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
2017-05-10 16:49:58 -07:00
Yann Collet
ba41b26405 Merge pull request #689 from facebook/cctxMerge
Cctx merge
2017-05-10 14:53:54 -07:00
Yann Collet
cef02d9317 changed variable name, for clarity
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
Yann Collet
669346fe8b fixed ZSTD_estimateCStreamSize()
https://github.com/facebook/zstd/pull/689#discussion_r115637721
2017-05-10 11:08:00 -07:00
Yann Collet
6fb2f24132 shortened ZSTD_createCStream_Advanced()
https://github.com/facebook/zstd/pull/689#discussion_r115637613
2017-05-10 11:06:06 -07:00
Yann Collet
f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8 changed name frameParams into frameHeader
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7 regroup memory usage function declarations
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
for clarity
2017-05-08 18:24:16 -07:00
Yann Collet
51652522a2 bumped version number 2017-05-08 17:52:46 -07:00
Yann Collet
a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet
7855366598 Updated ZSTD_freeCCtx()
which can also contain streaming buffers now.
Redirected ZSTD_freeCStream() towards it.
2017-05-08 17:15:00 -07:00
Yann Collet
fc5145955a updated ZSTD_estimateCCtxSize()
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
Yann Collet
791d744279 Updated ZSTD_sizeof_CCtx()
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
2017-05-08 16:17:30 -07:00
Yann Collet
0be6fd3429 merged CCtx and CStream as a single same object
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet
d47709b6ea Merge pull request #654 from iburinoc/splittable
[RFC] Splittable Format and API
2017-05-08 13:41:56 -07:00
Yann Collet
a00e9599f1 removed -g from DEBUGFLAGS
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
2017-05-04 17:24:29 -07:00
Yann Collet
606c04c228 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-02 12:13:52 -07:00
Yann Collet
072484a3bf Merge pull request #683 from terrelln/odev
[CLI] Make cover the default dictionary builder
2017-05-02 12:13:23 -07:00
Nick Terrell
f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell
020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell
f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell
865918dd04 Fix typo in zdict.h 2017-05-02 11:02:37 -07:00
Yann Collet
b184589c4c minor code refactoring for clarity 2017-05-01 11:35:47 -07:00
Yann Collet
33c38b0925 fixed const in prototype, that Visual doesn't accept 2017-05-01 11:12:30 -07:00
Yann Collet
f39a6731ec sync bitstream.h from fse library 2017-05-01 09:56:03 -07:00
Yann Collet
202082f285 sync bitstream from FSE project
add assert into unsafe *_fast() variants
2017-04-28 17:00:31 -07:00
Yann Collet
89f50deec7 minor code refactoring
clearer tables
2017-04-28 16:52:36 -07:00
Yann Collet
68a7d3d49a added HUF_PUBLIC_API macro to huf.h
to make it possible to control symbol visibility.
Also : better separation and comments between "public" and "static" sections
2017-04-28 12:46:48 -07:00
Yann Collet
a51cab6e68 Merge pull request #678 from facebook/apiChange
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
Yann Collet
29297c6751 Changed default level 18 (large input)
Previous -18 : 4.7 MB/s, R:3.833
New -18 : 5.1 MB/s. R:3.825

It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s)
The new level 18 also uses significantly less memory.
And, it makes a good transition between level 17 (mml5)
and level 19 (mml3).
Up to now, there was no level with mml4.

(note : minmatch setting can have a large impact on some (specific) datasets)
2017-04-27 17:44:01 -07:00
Yann Collet
a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet
d3694e6c70 removed C4204 2017-04-27 14:29:35 -07:00
Yann Collet
1c3ab0c77f fixed init error on Visual 2008 2017-04-27 12:57:11 -07:00
Yann Collet
8b669535f8 bumped version number to v1.2.0 2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50 added ZSTD_initCStream_usingCDict_advanced() 2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet
69a54d138a fixed compilation warning : declaration-after-statement 2017-04-27 01:11:26 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced()
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Yann Collet
768df129d2 changed ZSTD_compressBegin_usingCDict()
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
2017-04-26 15:42:10 -07:00
Yann Collet
e42afbc6fa Comply with suggested comments by @terrelln
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
2017-04-26 11:39:35 -07:00
Sean Purcell
7d37ca1d5b Merge remote-tracking branch 'origin/dev' into splittable 2017-04-21 14:18:39 -07:00
Yann Collet
7271203bdb transferred entropy scratch space from CCtx into workSpace
Saved 6 KB
2017-04-20 23:21:19 -07:00
Yann Collet
a408645f50 made some room for entropy scratch space 2017-04-20 23:09:39 -07:00
Yann Collet
71aaa32c3c transferred FSE tables from CCtx into workspace
Saved 5 KB from CCtx
2017-04-20 23:03:38 -07:00
Yann Collet
71ddeb67b1 made room in workspace for FSE tables
still need to be transferred from CCtx into workspace
2017-04-20 22:54:54 -07:00
Yann Collet
a34a39c183 changed size evaluation of entropy tables
so that memcpy() does no longer depends on fse pointer being a static table
2017-04-20 18:26:25 -07:00
Yann Collet
7bb60b17d8 init entropy table pointers only once
per workSpace resize
2017-04-20 17:38:56 -07:00