Yann Collet
6782725155
first sketch for largeNbDicts test program
2018-08-26 19:29:12 -07:00
Jennifer Liu
9d6ed9def3
Merge fastCover into DictBuilder ( #1274 )
...
* Minor fix
* Run non-optimize FASTCOVER 5 times in benchmark
* Merge fastCover into dictBuilder
* Fix mixed declaration issue
* Add fastcover to symbol.c
* Add fastCover.c and cover.h to build
* Change fastCover.c to fastcover.c
* Update benchmark to run FASTCOVER in dictBuilder
* Undo spliting fastcover_param into cover_param and f
* Remove convert param functions
* Assign f to parameter
* Add zdict.h to Makefile in lib
* Add cover.h to BUCK
* Cast 1 to U64 before shifting
* Remove trimming of zero freq head and tail in selectSegment and rebenchmark
* Remove f as a separate parameter of tryParam
* Read 8 bytes when d is 6
* Add trimming off zero frequency head and tail
* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)
* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary
* Change nbDmer to always read 8 bytes even when d=6
* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER
* Update comments and benchmarking result
* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover
* Add dictType enum and fix bug about passing zParam when converting to coverParam
* Combine finalize and skip into a single parameter
* Update acceleration parameters and benchmark on 3 sample sets
* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets
* Initialize variables outside of for loop in benchmark.c
* Update benchmark result for hg-manifest
* Remove cover.h from install-includes
* Add explanation of f
* Set default compression level for trainFromBuffer to 3
* Add assertion of fastCoverParams in DiB_trainFromFiles
* Add checkTotalCompressedSize function + some minor fixes
* Add test for multithreading fastCovr
* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish
* Free segmentFreqs
* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment
* Add FASTCOVER_MEMMULT
* Minor fix
* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet
36d6165a2d
Makefile: added variable SCANBUILD
...
so that a different version of scan-build can be selected
2018-08-16 16:44:13 -07:00
Yann Collet
42a02ab745
fixed minor warnings issued by scan-build
2018-08-15 14:36:02 -07:00
Jennifer Liu
0acb0abd1e
Add non-optimize FASTCOVER ( #1260 )
...
* Add non-optimize FASTCOVER
* Minor fix
* Pass param as value instead of pointer
2018-08-01 11:06:16 -07:00
Jennifer Liu
4e29bc2469
Use CDict instead of CCtx in analyzeEntropy
2018-07-31 10:36:45 -07:00
Jennifer Liu
31229e527b
Increment frequency for every dmer occurence within same sample instead of at most once per sample
2018-07-30 12:54:22 -07:00
Jennifer Liu
51b109c1b5
Delete old benchmarking result
2018-07-27 17:31:33 -07:00
Jennifer Liu
53ef22a4bc
Undo deleting clean in make
2018-07-27 16:56:50 -07:00
Jennifer Liu
96d84ee235
Revert test.sh
2018-07-27 16:54:05 -07:00
Jennifer Liu
61262f6c0d
Save segmentFreqs in ctx instead of malloc and memset in SelectSegment
2018-07-27 16:51:38 -07:00
Jennifer Liu
49b398e93f
Use same param after optimizing cover and fastCover and record k and d for benchmarking
2018-07-27 13:39:19 -07:00
Jennifer Liu
759c543312
Rerun cover and fastCover with optimized values
2018-07-26 19:03:01 -07:00
Jennifer Liu
3d7941ce41
Benchmark different f values
2018-07-26 16:24:13 -07:00
Jennifer Liu
3b163e0b5b
Add array to keep track of frequency within active segment, fix malloc bug, update benchmarking result
2018-07-26 13:53:13 -07:00
Jennifer Liu
2333ecb173
Allow d=6
2018-07-25 18:10:09 -07:00
Jennifer Liu
1e85f314d8
Benchmark fast cover optimize vs k=200
2018-07-25 17:53:38 -07:00
Jennifer Liu
d1fc507ef9
Initial benchmarking result for fastCover
2018-07-25 17:05:54 -07:00
Jennifer Liu
f5407e398a
Make hash value const
2018-07-25 16:54:08 -07:00
Jennifer Liu
7f3f70f766
Add Fast Cover Dictionary Builder
2018-07-25 16:34:07 -07:00
Nick Terrell
77068a8447
Merge pull request #1246 from jennifermliu/benchmark
...
Benchmark dictionary builders
2018-07-20 18:09:31 -07:00
Jennifer Liu
b6c5d4982c
Minor fix
2018-07-20 17:41:22 -07:00
Jennifer Liu
71e767ac09
Refactoring and benchmark without dictionary
2018-07-20 17:03:47 -07:00
Jennifer Liu
470c8d42f4
Benchmark dictionary builders
2018-07-20 11:32:39 -07:00
Nick Terrell
4d1ad5cdb2
Merge pull request #1238 from jennifermliu/random
...
Add random dictionary builder
2018-07-19 13:52:15 -07:00
Jennifer Liu
0c5eaef248
Update Makefile
2018-07-19 13:44:27 -07:00
Jennifer Liu
5bb46a898e
Rename cleanup
2018-07-18 12:15:49 -07:00
Jennifer Liu
52e7cf0e40
Add cleanup to trainfromFiles and move RANDOM_segment_t declaration
2018-07-18 10:40:13 -07:00
Jennifer Liu
ce09fb723d
Update freeSampleInfo
2018-07-17 16:13:40 -07:00
Jennifer Liu
896ff0644a
Fix deallocation problem and add documentation
2018-07-17 16:01:44 -07:00
Jennifer Liu
e6fe405838
Make test PHONY target
2018-07-17 12:42:53 -07:00
Jennifer Liu
49acfaeaec
Move file loading functions to new file for access by benchmarking tool
2018-07-17 12:35:09 -07:00
Jennifer Liu
4d32339b75
Remove CLevel cli option which was accidentally added back in the last commit
2018-07-16 18:59:18 -07:00
Jennifer Liu
1f7fa5cdd6
Fix spacing and Edit Makefile (now run with make instead of make run)
2018-07-16 16:31:59 -07:00
Jennifer Liu
b5806d33db
Refactor RANDOM
2018-07-16 16:03:04 -07:00
Codecat
044cd81ce6
Fix wrong conditions
2018-07-14 12:34:03 +02:00
Codecat
1a61bdb9c0
Update zstd.lua
2018-07-14 12:27:42 +02:00
Jennifer Liu
0e5fbc10fa
Update README
2018-07-13 17:41:09 -07:00
Jennifer Liu
31731df4da
Remove clevel and update documentation
2018-07-13 17:38:53 -07:00
Jennifer Liu
a23a3b95f9
Add random dictionary builder
2018-07-13 16:05:14 -07:00
Codecat
23d77c531e
Added premake4/GENie script to contrib folder
2018-07-11 18:02:18 +02:00
Jon Turney
87579d51ee
meson: fix build
2018-07-02 15:06:46 +01:00
Ryan Schmidt
bf30b9caf4
Add CXXFLAGS to ALL_LDFLAGS ( #1178 )
...
pzstd requires C++11, which older C++ standard libraries like libstdc++
as used on OS X 10.8 and earlier don't support. The user might address
this by setting "CXXFLAGS=-stdlib=libc++". This flag must be used both
at compile time and at link time. Asking the user to also put the flag
in LDFLAGS is undesirable because then the flag would also be used when
linking C code, which would be inappropriate.
2018-06-14 13:05:33 -07:00
Yann Collet
750ee87a92
Merge pull request #1175 from ryandesign/macos
...
Fix name of macOS
2018-06-13 11:32:06 -04:00
Ryan Schmidt
8ebd0f3a17
Remove duplicate CXXFLAGS
2018-06-10 18:04:21 -05:00
Ryan Schmidt
b567ce9d68
Fix name of macOS
2018-06-09 14:31:17 -05:00
Yann Collet
97c60cdf36
fixed seekable_format type mismatch
...
and some minor "unused variable" warnings.
Also : zstd_seekable.h is actually depending on zstd.h for ZSTDLIB_API
2018-06-06 13:10:29 -07:00
Yann Collet
30ee23e905
ensure seekable_format/examples generated libzstd.a
...
when it's not already present in the expected directory
2018-06-06 12:09:58 -07:00
Chris Lamb
2dbe408a49
Make the build reproducible
...
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.
This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.
This was originally filed in Debian as #897904 [1].
[0] https://reproducible-builds.org/
[1] https://bugs.debian.org/897904
Signed-off-by: Chris Lamb <lamby@debian.org>
2018-05-04 08:39:51 -07:00
Adam Borowski
00d52282d0
Use -pthread rather than -lpthread.
...
It can have other effects, such as pulling in extra libraries. Without it,
riscv build fails with undefined reference to `__atomic_compare_exchange_1'.
2018-04-09 01:50:49 +02:00
Yann Collet
9e6ba88363
moved snap to /contrib
2018-03-19 16:15:06 -07:00
Yann Collet
a7b532a347
added docker readme
2018-03-19 16:13:12 -07:00
Yann Collet
b06db3b3c5
Merge pull request #1052 from Varunram/dockerfile
...
Add Dockerfile
2018-03-19 16:07:41 -07:00
Varunram
7616200eaf
Add Dockerfile
...
Dockerfile initially proposed by @gyscos at #880
2018-03-18 14:53:48 +05:30
Yann Collet
5373e44ba7
fixed contrib/adaptive-compression
2018-03-15 17:10:15 -07:00
Yann Collet
355cb645bf
fixed seekable format example
2018-03-15 16:29:28 -07:00
Yann Collet
38cbcb5f1a
removed LRM exploratory experiment
2018-03-15 16:26:08 -07:00
Yann Collet
e916b9090e
gen_html: changed CFLAGS for CXXFLAGS
...
since it's associated with $(CXX)
2018-03-09 11:52:14 -08:00
Shawn Landden
ea41b580eb
meson: allow -Dlegacy_support=true, fix -Dlegacy_support=0
2017-12-29 10:18:33 -08:00
Shawn Landden
6ff43c0051
get soversion right
2017-12-24 10:05:43 -08:00
Shawn Landden
daffe435c0
meson: support differn't legacy levels.
...
Default to v0.4.0+
2017-12-21 15:47:38 -08:00
Shawn Landden
3ddfa42fe8
meson: fix build
...
used absolute paths which are deprecated in meson, also missing some sources
that got split
also move source files each to their own line so future diffs are clearer.
2017-12-19 22:02:03 -08:00
Yann Collet
c173dbd6e7
no longer supported starting C++17
2017-12-04 18:00:53 -08:00
Yann Collet
0d58aaf6f0
/contrib: fixed license header
...
removed last reference to PATENTS file
2017-10-02 02:07:17 -07:00
Yann Collet
f1a3ce34b7
Merge pull request #826 from terrelln/license
...
[linux-kernel] Update license
2017-08-31 12:52:23 -07:00
Nick Terrell
f9252d8347
[linux-kernel] Update license
2017-08-31 12:48:36 -07:00
Yann Collet
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
Yann Collet
394bdd7db9
changed license for examples
...
intentionnally this time
2017-08-29 09:24:11 -07:00
Yann Collet
dd63533b29
Merge pull request #797 from paulcruz74/documentation
...
Updating README.md
2017-08-16 00:37:55 -07:00
Paul Cruz
733ca51360
Updating README.md
2017-08-15 17:48:23 -07:00
Yann Collet
2daeab12f4
Merge pull request #795 from terrelln/squashfs-tools2
...
[kernel] Update squashfs-tools patch
2017-08-15 11:59:41 -07:00
Nick Terrell
57e2df6651
[kernel] Update squashfs-tools patch
2017-08-14 22:43:36 -07:00
Paul Cruz
fc90469587
updated program name print statement
2017-08-10 16:11:59 -07:00
Paul Cruz
75c3ad947c
Merge remote-tracking branch 'upstream/dev' into dev
2017-08-10 14:46:31 -07:00
Yann Collet
8915b9b518
Merge pull request #786 from terrelln/squashfs-tools
...
[linux-kernel] Update patches for v4 and v5
2017-08-09 16:00:11 -07:00
Nick Terrell
8b6702a00d
[linux-kernel] Update patches for v5
2017-08-09 13:03:40 -07:00
Yann Collet
a1fe656165
Merge pull request #779 from paulcruz74/adapt-approach-4
...
Patching warnings, adding ability to bound compression variation
2017-08-07 14:50:38 -07:00
Paul Cruz
7069bb9458
Merge branch 'adapt-approach-4' into bug-fixes
2017-08-07 13:18:53 -07:00
Paul Cruz
e100a311eb
removed direct assignment of 22, used ZSTD_maxCLevel() instead
2017-08-07 13:11:07 -07:00
Nick Terrell
7393b49fbd
[linux-kernel] Update patches for v4
2017-08-04 16:57:03 -07:00
Paul Cruz
01237e3b35
changed multi to zstd-adaptive in the help menu
2017-08-03 15:13:49 -07:00
Paul Cruz
8be7bba08c
added mutex for compression level to avoid data race
2017-08-02 10:27:33 -07:00
Paul Cruz
69ef22c0ac
added detach statements to prevent resource leak
2017-08-01 17:36:13 -07:00
Paul Cruz
f8c7b191e7
Merge branch 'adapt-approach-4' into dev
2017-07-31 15:42:57 -07:00
Paul Cruz
0295737ad7
change signal to broadcast for jobCompressed condition varaible since multiple threads waiting
2017-07-31 13:43:03 -07:00
Paul Cruz
cf92221671
Merge remote-tracking branch 'upstream/dev' into dev
2017-07-31 11:43:40 -07:00
Paul Cruz
c2f1bda8a1
Merge branch 'adapt-approach-4' into dev
2017-07-31 11:43:01 -07:00
Paul Cruz
9ea7df03de
add install target in makefile
2017-07-31 11:04:17 -07:00
Paul Cruz
f60cd3f99b
print defaults and range, remove EXT
2017-07-31 09:47:09 -07:00
Paul Cruz
e22b60cb76
removed ternary operation, added assert statement, check to make sure initial compression level is within bounds
2017-07-28 17:46:51 -07:00
Paul Cruz
cb9af53e77
delete empty line
2017-07-28 17:28:25 -07:00
Paul Cruz
51788225db
remove exe extension from makefile, reinclude pthread flag
2017-07-28 17:27:36 -07:00
Yann Collet
ee27f6937c
Merge pull request #758 from stellamplau/ldm
...
Add working prototype of a long distance matcher
2017-07-28 17:07:21 -07:00
Paul Cruz
4d904ac800
add flags for multithreading
2017-07-28 16:12:58 -07:00
Paul Cruz
0f4cb67b00
add tests for compression bounds, fix another warning
2017-07-28 15:55:02 -07:00
Paul Cruz
ff54fced64
patched style errors, add ability to bound compression level variation
2017-07-28 15:30:46 -07:00
Stella Lau
8fae41c412
Return error code in verify() and minor code cleanup
2017-07-27 17:14:05 -07:00
Stella Lau
1294a4a897
Fix typo
2017-07-27 15:49:46 -07:00
