Commit Graph

551 Commits

Author SHA1 Message Date
Yann Collet
9c3265a53f
Merge pull request #1417 from facebook/advancedAPI
Advanced API
2018-12-10 18:48:15 -08:00
Yann Collet
3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_*
for naming consistency
2018-12-05 17:26:02 -08:00
Lzu Tao
beb13bd87e Move contrib/meson to build/meson 2018-12-01 23:18:59 +07:00
Lzu Tao
c0e71cae55 Add enable_lz4 build option and fix lzma dependency 2018-12-01 23:18:59 +07:00
Lzu Tao
5c4965c351 Add pedantic flag 2018-12-01 23:18:59 +07:00
Lzu Tao
6f3f1a8d3a No install zstd_manual.html 2018-12-01 23:18:59 +07:00
Lzu Tao
f660825d9f Install missed zstdgrep and zstdless 2018-12-01 23:18:59 +07:00
Lzu Tao
3f27e2a072 Install zstdmt.1 manpage [skip ci] 2018-12-01 23:18:59 +07:00
Lzu Tao
d3134a3ed3 Rename meson variables 2018-12-01 23:18:59 +07:00
Lzu Tao
1985e427c7 Add manpage install warning [skip ci]
We link new manpages with gz compressed format of the target manpage.
I have not tested it on Windows. So just place a warning here.
2018-12-01 23:18:59 +07:00
Lzu Tao
9c862c6a53 Fix manpage symlinks [skip ci] 2018-12-01 23:18:59 +07:00
Lzu Tao
d79df2a370 Apply new InstallSymlink script 2018-12-01 23:18:59 +07:00
Lzu Tao
ef2e761937 Helper script to install symlink in meson 2018-12-01 23:18:59 +07:00
Lzu Tao
3175188407 No need these helpers 2018-12-01 23:18:59 +07:00
Lzu Tao
337f914dc8 Fix lib soversion and no install cover.h header 2018-12-01 23:18:59 +07:00
Lzu Tao
c9f0144302 Fix meson tests build 2018-12-01 23:18:59 +07:00
Lzu Tao
5a36a57cf5 Bump to 1.3.8 and fix run_command function
The run_command is run from an unspecified directory. Therefore we cannot assume
which directory it is running our command.
2018-12-01 23:18:59 +07:00
Lzu Tao
8a160680d1 Update legacy support to 5 2018-12-01 23:18:59 +07:00
Lzu Tao
f727808731 Minor fix for meson build
Use files function instead of constructing path with meson.current_source_dir()
2018-12-01 23:18:59 +07:00
Lzu Tao
9a721e5216 Update meson build system
NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
build may not work as expected.

* Use meson >= 0.47.0 cause we use install_man function
* Add three helper Python script:
  * CopyFile.py: To copy file
  * CreateSymlink.py: To make symlink (both Windows and Unix)
  * GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
  These help emulating equivalent functions in CMake and Makefile.
* Use subdir from meson to split meson.build
  * Add contrib build
  * Fix other build
* Add new build options
  * build_programs: Enable programs build
  * build_contrib: Enable contrib build
  * build_tests: Enable tests build
  * use_static_runtime: Link to static run-time libraries on MSVC
  * zlib_support: Enable zlib support
  * lzma_support: Enable lzma support
2018-11-28 01:08:34 +07:00
Lzu Tao
9bd8f6a00c Rename and update build instruction in README file to README.md 2018-11-28 01:08:34 +07:00
Lzu Tao
2abd5139a5 Add meson build guide 2018-11-28 01:08:34 +07:00
Yann Collet
5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet
b83d1e7714 removed some static const variables
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
2018-11-13 16:56:32 -08:00
Yann Collet
b830ccca5c changed benchfn api
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
2018-11-13 13:12:50 -08:00
Yann Collet
d38063f8ae separated bench module into benchfn and benchzstd
it shall be possible to use benchfn
without any dependency on zstd.
2018-11-13 11:01:59 -08:00
Yann Collet
483759a3de Improves decompression speed when using cold dictionary
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
2018-11-08 17:00:23 -08:00
Rohit Jain
705e0b18ab Making changes to make it compile on my laptop 2018-10-11 15:51:57 -07:00
Yann Collet
123fac6b6d fix pzstd compatibility with mingw
some details changed with introduction of gcc7
2018-09-21 17:36:00 -07:00
Yann Collet
00ce26725b
Merge pull request #1324 from ko-zu/fixclangcode
Fix largeNbDicts bench for clangbuild
2018-09-17 14:10:17 -07:00
Nick Terrell
8f27e8cf3d
Merge pull request #1322 from azat-archive/seekable-fixes-pull
Fixes read write past end of input buffer.
2018-09-17 11:04:51 -07:00
ko-zu
b053bec2f4 Fix largeNbDicts bench for clangbuild
Remove unsigned to size_t promotion to fix implicit down conversion errors in clangbuild target.
2018-09-17 13:09:08 +09:00
Azat Khuzhin
d707692e05
seekable_decompression: support offset greater then UNIT_MAX 2018-09-16 18:05:32 +03:00
Azat Khuzhin
b52867a97f
zstdseek_decompress: fix decompression with data left in input buffer 2018-09-16 18:05:32 +03:00
Yann Collet
c49ccbc8e7 largeNbDicts : can select a nb of blocks
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
2018-09-12 11:31:28 -07:00
Yann Collet
50b216146f
Merge pull request #1304 from facebook/largeNbDicts
contrib/largeNbDicts
2018-09-06 09:50:56 -07:00
Yann Collet
c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet
1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet
11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet
0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet
f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet
39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet
39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet
4086b2871b largeNbDicts compatible with multiple source files
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
2018-08-30 14:38:49 -07:00
Yann Collet
a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00
Yann Collet
d89fa814c1 added a README
for documentation
2018-08-28 18:19:19 -07:00
Yann Collet
6444c50035 increases randomness of ddict ptrs 2018-08-28 18:13:46 -07:00
Yann Collet
6c398df241 level, block size and nb dicts can be set on command line 2018-08-28 18:05:31 -07:00
Yann Collet
0c66a44d1b first working test program
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
2018-08-28 15:47:07 -07:00
Yann Collet
274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00