AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,454 Commits 1 Branch 0 Tags 27 MiB
master
Commit Graph

653 Commits

Author SHA1 Message Date
Yann Collet
0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet
274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00
Yann Collet
6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274) 
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
 2018-08-23 12:06:20 -07:00
Yann Collet
36d6165a2d Makefile: added variable SCANBUILD 
so that a different version of scan-build can be selected
 2018-08-16 16:44:13 -07:00
Yann Collet
42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Jennifer Liu
0acb0abd1e Add non-optimize FASTCOVER (#1260) 
* Add non-optimize FASTCOVER

* Minor fix

* Pass param as value instead of pointer
 2018-08-01 11:06:16 -07:00
Jennifer Liu
4e29bc2469 Use CDict instead of CCtx in analyzeEntropy 2018-07-31 10:36:45 -07:00
Jennifer Liu
31229e527b Increment frequency for every dmer occurence within same sample instead of at most once per sample 2018-07-30 12:54:22 -07:00
Jennifer Liu
51b109c1b5 Delete old benchmarking result 2018-07-27 17:31:33 -07:00
Jennifer Liu
53ef22a4bc Undo deleting clean in make 2018-07-27 16:56:50 -07:00
Jennifer Liu
96d84ee235 Revert test.sh 2018-07-27 16:54:05 -07:00
Jennifer Liu
61262f6c0d Save segmentFreqs in ctx instead of malloc and memset in SelectSegment 2018-07-27 16:51:38 -07:00
Jennifer Liu
49b398e93f Use same param after optimizing cover and fastCover and record k and d for benchmarking 2018-07-27 13:39:19 -07:00
Jennifer Liu
759c543312 Rerun cover and fastCover with optimized values 2018-07-26 19:03:01 -07:00
Jennifer Liu
3d7941ce41 Benchmark different f values 2018-07-26 16:24:13 -07:00
Jennifer Liu
3b163e0b5b Add array to keep track of frequency within active segment, fix malloc bug, update benchmarking result 2018-07-26 13:53:13 -07:00
Jennifer Liu
2333ecb173 Allow d=6 2018-07-25 18:10:09 -07:00
Jennifer Liu
1e85f314d8 Benchmark fast cover optimize vs k=200 2018-07-25 17:53:38 -07:00
Jennifer Liu
d1fc507ef9 Initial benchmarking result for fastCover 2018-07-25 17:05:54 -07:00
Jennifer Liu
f5407e398a Make hash value const 2018-07-25 16:54:08 -07:00
Jennifer Liu
7f3f70f766 Add Fast Cover Dictionary Builder 2018-07-25 16:34:07 -07:00
Nick Terrell
77068a8447
Merge pull request #1246 from jennifermliu/benchmark 
Benchmark dictionary builders
 2018-07-20 18:09:31 -07:00
Jennifer Liu
b6c5d4982c Minor fix 2018-07-20 17:41:22 -07:00
Jennifer Liu
71e767ac09 Refactoring and benchmark without dictionary 2018-07-20 17:03:47 -07:00
Jennifer Liu
470c8d42f4 Benchmark dictionary builders 2018-07-20 11:32:39 -07:00
Nick Terrell
4d1ad5cdb2
Merge pull request #1238 from jennifermliu/random 
Add random dictionary builder
 2018-07-19 13:52:15 -07:00
Jennifer Liu
0c5eaef248 Update Makefile 2018-07-19 13:44:27 -07:00
Jennifer Liu
5bb46a898e Rename cleanup 2018-07-18 12:15:49 -07:00
Jennifer Liu
52e7cf0e40 Add cleanup to trainfromFiles and move RANDOM_segment_t declaration 2018-07-18 10:40:13 -07:00
Jennifer Liu
ce09fb723d Update freeSampleInfo 2018-07-17 16:13:40 -07:00
Jennifer Liu
896ff0644a Fix deallocation problem and add documentation 2018-07-17 16:01:44 -07:00
Jennifer Liu
e6fe405838 Make test PHONY target 2018-07-17 12:42:53 -07:00
Jennifer Liu
49acfaeaec Move file loading functions to new file for access by benchmarking tool 2018-07-17 12:35:09 -07:00
Jennifer Liu
4d32339b75 Remove CLevel cli option which was accidentally added back in the last commit 2018-07-16 18:59:18 -07:00
Jennifer Liu
1f7fa5cdd6 Fix spacing and Edit Makefile (now run with make instead of make run) 2018-07-16 16:31:59 -07:00
Jennifer Liu
b5806d33db Refactor RANDOM 2018-07-16 16:03:04 -07:00
Codecat
044cd81ce6
Fix wrong conditions 2018-07-14 12:34:03 +02:00
Codecat
1a61bdb9c0
Update zstd.lua 2018-07-14 12:27:42 +02:00
Jennifer Liu
0e5fbc10fa Update README 2018-07-13 17:41:09 -07:00
Jennifer Liu
31731df4da Remove clevel and update documentation 2018-07-13 17:38:53 -07:00
Jennifer Liu
a23a3b95f9 Add random dictionary builder 2018-07-13 16:05:14 -07:00
Codecat
23d77c531e
Added premake4/GENie script to contrib folder 2018-07-11 18:02:18 +02:00
Jon Turney
87579d51ee
meson: fix build 2018-07-02 15:06:46 +01:00
Ryan Schmidt
bf30b9caf4 Add CXXFLAGS to ALL_LDFLAGS (#1178) 
pzstd requires C++11, which older C++ standard libraries like libstdc++
as used on OS X 10.8 and earlier don't support. The user might address
this by setting "CXXFLAGS=-stdlib=libc++". This flag must be used both
at compile time and at link time. Asking the user to also put the flag
in LDFLAGS is undesirable because then the flag would also be used when
linking C code, which would be inappropriate.
 2018-06-14 13:05:33 -07:00
Yann Collet
750ee87a92
Merge pull request #1175 from ryandesign/macos 
Fix name of macOS
 2018-06-13 11:32:06 -04:00
Ryan Schmidt
8ebd0f3a17 Remove duplicate CXXFLAGS 2018-06-10 18:04:21 -05:00
Ryan Schmidt
b567ce9d68 Fix name of macOS 2018-06-09 14:31:17 -05:00
Yann Collet
97c60cdf36 fixed seekable_format type mismatch 
and some minor "unused variable" warnings.
Also : zstd_seekable.h is actually depending on zstd.h for ZSTDLIB_API
 2018-06-06 13:10:29 -07:00
Yann Collet
30ee23e905 ensure seekable_format/examples generated libzstd.a 
when it's not already present in the expected directory
 2018-06-06 12:09:58 -07:00
Chris Lamb
2dbe408a49 Make the build reproducible 
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.

This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.

This was originally filed in Debian as #897904 [1].

 [0] https://reproducible-builds.org/
 [1] https://bugs.debian.org/897904

Signed-off-by: Chris Lamb <lamby@debian.org>
 2018-05-04 08:39:51 -07:00
Adam Borowski
00d52282d0 Use -pthread rather than -lpthread. 
It can have other effects, such as pulling in extra libraries.  Without it,
riscv build fails with undefined reference to `__atomic_compare_exchange_1'.
 2018-04-09 01:50:49 +02:00
Yann Collet
9e6ba88363 moved snap to /contrib 2018-03-19 16:15:06 -07:00
Yann Collet
a7b532a347 added docker readme 2018-03-19 16:13:12 -07:00
Yann Collet
b06db3b3c5
Merge pull request #1052 from Varunram/dockerfile 
Add Dockerfile
 2018-03-19 16:07:41 -07:00
Varunram
7616200eaf Add Dockerfile 
Dockerfile initially proposed by @gyscos at #880
 2018-03-18 14:53:48 +05:30
Yann Collet
5373e44ba7 fixed contrib/adaptive-compression 2018-03-15 17:10:15 -07:00
Yann Collet
355cb645bf fixed seekable format example 2018-03-15 16:29:28 -07:00
Yann Collet
38cbcb5f1a removed LRM exploratory experiment 2018-03-15 16:26:08 -07:00
Yann Collet
e916b9090e gen_html: changed CFLAGS for CXXFLAGS 
since it's associated with $(CXX)
 2018-03-09 11:52:14 -08:00
Shawn Landden
ea41b580eb meson: allow -Dlegacy_support=true, fix -Dlegacy_support=0 2017-12-29 10:18:33 -08:00
Shawn Landden
6ff43c0051 get soversion right 2017-12-24 10:05:43 -08:00
Shawn Landden
daffe435c0 meson: support differn't legacy levels. 
Default to v0.4.0+
 2017-12-21 15:47:38 -08:00
Shawn Landden
3ddfa42fe8 meson: fix build 
used absolute paths which are deprecated in meson, also missing some sources
that got split

also move source files each to their own line so future diffs are clearer.
 2017-12-19 22:02:03 -08:00
Yann Collet
c173dbd6e7 no longer supported starting C++17 2017-12-04 18:00:53 -08:00
Yann Collet
0d58aaf6f0 /contrib: fixed license header 
removed last reference to PATENTS file
 2017-10-02 02:07:17 -07:00
Yann Collet
f1a3ce34b7 Merge pull request #826 from terrelln/license 
[linux-kernel] Update license
 2017-08-31 12:52:23 -07:00
Nick Terrell
f9252d8347 [linux-kernel] Update license 2017-08-31 12:48:36 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Yann Collet
394bdd7db9 changed license for examples 
intentionnally this time
 2017-08-29 09:24:11 -07:00
Yann Collet
dd63533b29 Merge pull request #797 from paulcruz74/documentation 
Updating README.md
 2017-08-16 00:37:55 -07:00
Paul Cruz
733ca51360 Updating README.md 2017-08-15 17:48:23 -07:00
Yann Collet
2daeab12f4 Merge pull request #795 from terrelln/squashfs-tools2 
[kernel] Update squashfs-tools patch
 2017-08-15 11:59:41 -07:00
Nick Terrell
57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Paul Cruz
fc90469587 updated program name print statement 2017-08-10 16:11:59 -07:00
Paul Cruz
75c3ad947c Merge remote-tracking branch 'upstream/dev' into dev 2017-08-10 14:46:31 -07:00
Yann Collet
8915b9b518 Merge pull request #786 from terrelln/squashfs-tools 
[linux-kernel] Update patches for v4 and v5
 2017-08-09 16:00:11 -07:00
Nick Terrell
8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Yann Collet
a1fe656165 Merge pull request #779 from paulcruz74/adapt-approach-4 
Patching warnings, adding ability to bound compression variation
 2017-08-07 14:50:38 -07:00
Paul Cruz
7069bb9458 Merge branch 'adapt-approach-4' into bug-fixes 2017-08-07 13:18:53 -07:00
Paul Cruz
e100a311eb removed direct assignment of 22, used ZSTD_maxCLevel() instead 2017-08-07 13:11:07 -07:00
Nick Terrell
7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Paul Cruz
01237e3b35 changed multi to zstd-adaptive in the help menu 2017-08-03 15:13:49 -07:00
Paul Cruz
8be7bba08c added mutex for compression level to avoid data race 2017-08-02 10:27:33 -07:00
Paul Cruz
69ef22c0ac added detach statements to prevent resource leak 2017-08-01 17:36:13 -07:00
Paul Cruz
f8c7b191e7 Merge branch 'adapt-approach-4' into dev 2017-07-31 15:42:57 -07:00
Paul Cruz
0295737ad7 change signal to broadcast for jobCompressed condition varaible since multiple threads waiting 2017-07-31 13:43:03 -07:00
Paul Cruz
cf92221671 Merge remote-tracking branch 'upstream/dev' into dev 2017-07-31 11:43:40 -07:00
Paul Cruz
c2f1bda8a1 Merge branch 'adapt-approach-4' into dev 2017-07-31 11:43:01 -07:00
Paul Cruz
9ea7df03de add install target in makefile 2017-07-31 11:04:17 -07:00
Paul Cruz
f60cd3f99b print defaults and range, remove EXT 2017-07-31 09:47:09 -07:00
Paul Cruz
e22b60cb76 removed ternary operation, added assert statement, check to make sure initial compression level is within bounds 2017-07-28 17:46:51 -07:00
Paul Cruz
cb9af53e77 delete empty line 2017-07-28 17:28:25 -07:00
Paul Cruz
51788225db remove exe extension from makefile, reinclude pthread flag 2017-07-28 17:27:36 -07:00
Yann Collet
ee27f6937c Merge pull request #758 from stellamplau/ldm 
Add working prototype of a long distance matcher
 2017-07-28 17:07:21 -07:00
Paul Cruz
4d904ac800 add flags for multithreading 2017-07-28 16:12:58 -07:00
Paul Cruz
0f4cb67b00 add tests for compression bounds, fix another warning 2017-07-28 15:55:02 -07:00
Paul Cruz
ff54fced64 patched style errors, add ability to bound compression level variation 2017-07-28 15:30:46 -07:00
Stella Lau
8fae41c412 Return error code in verify() and minor code cleanup 2017-07-27 17:14:05 -07:00
Stella Lau
1294a4a897 Fix typo 2017-07-27 15:49:46 -07:00
Stella Lau
627621839c Add checks in initialization code 2017-07-27 15:37:37 -07:00
Stella Lau
c105f605e6 Update README 2017-07-27 11:11:35 -07:00
Yann Collet
e1222544be Merge pull request #753 from paulcruz74/adapt-approach-3 
adaptive compression v1
 2017-07-27 10:00:10 -07:00
Paul Cruz
2320e7378a remove unused variable, add documentation for context fields 2017-07-26 17:02:47 -07:00
Stella Lau
9eaf3d22d0 Allow HASH_ONLY_EVERY_LOG to be configured in ldm.h 2017-07-26 16:43:25 -07:00
Paul Cruz
ab5a78547e fix leaky abstraction regarding measuring completion 2017-07-26 16:40:05 -07:00
Paul Cruz
715f36ca81 added definitions for conversion constants, moved forced compression check to top of adaptCompressionLevel, used ZSTD_BLOCKSIZE_MAX 2017-07-26 15:52:15 -07:00
Paul Cruz
6c1c1242fc set the window log value before performing compression 2017-07-26 14:29:59 -07:00
Stella Lau
40759bade9 Add README and clean up code 2017-07-26 13:24:03 -07:00
Paul Cruz
a959cc881a moved reset of completion to right after wait 2017-07-26 10:34:48 -07:00
Paul Cruz
305d5ee70f change to >= convergence counter 2017-07-26 10:20:29 -07:00
Paul Cruz
be92a38d6a decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait 2017-07-26 10:05:10 -07:00
Stella Lau
e9161637b2 Allow parameters to be modified from a separate file 2017-07-25 18:13:27 -07:00
Paul Cruz
0b18d21e03 building on readme, added another help tip in the menu 2017-07-25 17:47:02 -07:00
Paul Cruz
7cc74e0b27 adding more to readme 2017-07-25 16:55:16 -07:00
Paul Cruz
8dbb07d822 updated progress bar with better representation of time, added const 2017-07-25 16:03:43 -07:00
Paul Cruz
9a132707af changing time units to seconds 2017-07-25 15:26:26 -07:00
Stella Lau
629c300118 Rename and remove unneeded files 2017-07-25 15:17:36 -07:00
Paul Cruz
31a9ed9883 updated const values, added more comments 2017-07-25 14:53:40 -07:00
Paul Cruz
5cfbf609a4 removed old debug statements no longer being used 2017-07-25 14:31:48 -07:00
Paul Cruz
0882cd1981 progress bar -- don't print num jobs, time elapsed shown in seconds 2017-07-25 14:26:55 -07:00
Paul Cruz
310c12d07e moved debug statements to a compiler flag 2017-07-25 14:08:39 -07:00
Paul Cruz
e02c79f833 started using decrease cooldown so that compression level would not decrease several times in a row 2017-07-25 11:16:27 -07:00
Paul Cruz
85d7c919f6 created independent function for controlling how completion relates to compression level change 2017-07-25 10:32:14 -07:00
Paul Cruz
6f1e260edd added mechanism for getting rid of spikes 2017-07-25 10:01:10 -07:00
Paul Cruz
700758d676 added help statement for -p, switched it to hide progress bar now that progress bar is default 2017-07-24 16:26:20 -07:00
Paul Cruz
df3754b6ed add quiet option, make progress bar default 2017-07-24 16:19:07 -07:00
Stella Lau
0295a27133 Experiment with not using a checksum 2017-07-24 15:26:44 -07:00
Paul Cruz
4dc83ca64c compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance 2017-07-24 15:14:58 -07:00
Paul Cruz
0ee3f8c2f8 adding more debug 2017-07-24 15:06:11 -07:00
Paul Cruz
8328f8192a updating debug statements again 2017-07-24 14:40:23 -07:00
Paul Cruz
d3d759301f changing position of endline for debug 2017-07-24 13:47:39 -07:00
Stella Lau
08a6e9a141 Minor code cleanup 2017-07-24 13:22:00 -07:00
Stella Lau
6eefa32911 Deduplicate code 2017-07-24 12:40:59 -07:00
Stella Lau
8ed9220102 Experiment with eviction policies and minor code cleanup 2017-07-24 12:05:43 -07:00
Paul Cruz
e508f632d6 updated comments and debug statements 2017-07-24 11:01:36 -07:00
Stella Lau
eb16da647d Minor clean up 2017-07-24 10:20:53 -07:00
Stella Lau
1a188fe864 Fix overflow bug when calculating hash 2017-07-24 10:20:53 -07:00
Stella Lau
0b8fb1703b Experiment with 64-bit hash insertion policy 2017-07-24 10:20:30 -07:00
Stella Lau
273c17b350 Experiment with 64-bit hash and checksum 2017-07-24 10:19:50 -07:00
Paul Cruz
483d936b87 reduced competition for completion mutex by separating mutex use based on which values is updated 2017-07-23 14:09:16 -07:00
Paul Cruz
880f08d104 change how completion is measured in compression thread 2017-07-23 10:18:54 -07:00
Paul Cruz
08d9e42ec6 removed useless measurements 2017-07-21 18:02:55 -07:00
Paul Cruz
95bef759b3 switched over to model where reading only waits on compression thread 2017-07-21 17:49:39 -07:00
Paul Cruz
6455ec482c taking the maximum of the completion level reads in order to determine which one was waiting more 2017-07-21 16:05:01 -07:00
Paul Cruz
05fe8dd47c updating debug statements 2017-07-21 14:06:24 -07:00
Paul Cruz
db109f8fef measure multiple completion levels during each wait 2017-07-21 13:38:24 -07:00
Paul Cruz
721c6a8b97 added bounding to compression level change 2017-07-21 09:30:24 -07:00
Paul Cruz
e929d3b787 added priority decision making for adapt compression level 2017-07-21 09:26:35 -07:00
Paul Cruz
9259c7afa4 semi working version that stabilizes 2017-07-20 18:45:33 -07:00
Paul Cruz
82e488770c fixed bug where writeSize could be zero 2017-07-20 16:38:02 -07:00
Paul Cruz
a19916425d reworked adaptCompressionLevel to only account for completion information 2017-07-20 16:19:16 -07:00
Nick Terrell
7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Paul Cruz
7ab758a640 changed how completion is actually sampled 2017-07-20 10:53:51 -07:00
Stella Lau
13a01ffb27 Fix off-by-one in size calculations 2017-07-19 17:24:09 -07:00
Stella Lau
2427a154cb Minor refactoring 2017-07-19 16:56:28 -07:00
Paul Cruz
dcf609f835 make adaptCompressionLevel oscillate less 2017-07-19 16:36:33 -07:00
Paul Cruz
2a22c7915e call ZSTD_compressBegin() once 2017-07-19 16:00:54 -07:00
Paul Cruz
6767abe652 fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0) 2017-07-19 14:54:15 -07:00
Stella Lau
030264ca51 Experiment with integrating ZSTD_count with findBestMatch 2017-07-19 14:14:26 -07:00
Paul Cruz
42382c1216 added some debug statements, adjusted end condition 2017-07-19 13:30:07 -07:00
Paul Cruz
5a85c57e30 set up new calculations compression completion progress 2017-07-19 11:47:17 -07:00
Paul Cruz
f1ac518b59 split compression into smaller blocks 2017-07-19 11:23:40 -07:00
Paul Cruz
338951cd48 moved compression adapt to avoid warning 2017-07-19 10:23:46 -07:00
Paul Cruz
4497ecf297 change compression level only right before actually performing compression. When waiting, only update waiting statistics. 2017-07-19 10:14:00 -07:00
Paul Cruz
e11bf55d0b added mechanism for measuring how much of a job has been created 2017-07-19 10:10:47 -07:00
Paul Cruz
559ea4ff25 split up read process into smaller chunks 2017-07-19 09:59:17 -07:00
Paul Cruz
6119cd2164 added additional print for help menu 2017-07-19 09:43:17 -07:00
Stella Lau
4352e09cb0 Avoid recounting match lengths with ZSTD_count 2017-07-18 18:35:25 -07:00
Stella Lau
1fa223859f Switch to using ZSTD_count instead of function pointer 2017-07-18 18:05:10 -07:00
Paul Cruz
3d7f1afadd changed createCCtx() to split into initialization and creation 2017-07-18 17:32:36 -07:00
Paul Cruz
2c4e4ddc50 added mutex for stats struct 2017-07-18 15:55:58 -07:00
Paul Cruz
ad66faf16a added progress check for filewriting, put important shared data behind mutex when being read from/written to 2017-07-18 15:23:11 -07:00
Stella Lau
19258f51c1 Make the meaning of LDM_MEMORY_USAGE consistent across tables 2017-07-18 14:25:39 -07:00
Paul Cruz
a34bc30237 setting up basic readme 2017-07-18 13:31:02 -07:00
Paul Cruz
29c36cf051 rename completion variable, split up fwrite operations in order to track progress 2017-07-18 13:30:29 -07:00
Paul Cruz
ae47eab2fd changed test cases to use -s setting on the diffs 2017-07-18 12:58:50 -07:00
Stella Lau
fc41a87964 Experiment with using a lag when hashing 2017-07-17 18:13:09 -07:00
Paul Cruz
5af04c57b0 change parameters for compression level adapt 2017-07-17 17:59:50 -07:00
Paul Cruz
b3c9e02bb6 added signal to other threads whenever error occurs 2017-07-17 15:34:58 -07:00
Stella Lau
a00e406231 Remove version archive 2017-07-17 15:17:32 -07:00
Stella Lau
15a041adbf Add function to get valid entries only from table 2017-07-17 15:16:58 -07:00
Paul Cruz
6be22f1f84 swap buffers instead of copying memory over 2017-07-17 14:39:10 -07:00
Paul Cruz
708238e07e open file outside of adaptCCtx, pass to the output thread 2017-07-17 14:01:13 -07:00
Stella Lau
4bb42b02c1 Add basic chaining table 2017-07-17 11:53:54 -07:00
Paul Cruz
044e40db5a removed freeCCtx() calls from createCCtx() so that it is not called twice during errors 2017-07-17 11:19:23 -07:00
Paul Cruz
50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Stella Lau
ca300ce6e0 Decouple hash table from compression function 2017-07-14 17:17:00 -07:00
Paul Cruz
1ab3f06f00 updated tests to use different seeds when executing different tests 2017-07-14 16:29:29 -07:00
Stella Lau
6e443b4960 Move hash table access for own functions 2017-07-14 14:27:55 -07:00
Stella Lau
2d8e6c6608 Add more statistics 2017-07-14 12:31:01 -07:00
Stella Lau
55f960e8db Add percentages to offset histogram 2017-07-14 11:00:20 -07:00
Stella Lau
4db7f12ef3 Add offset histogram 2017-07-14 10:52:03 -07:00
Paul Cruz
0c8b9436b7 removed goto statements for the most part 2017-07-13 16:38:20 -07:00
Stella Lau
175a6c6029 [ldm] Minor refactoring 2017-07-13 16:16:31 -07:00
Stella Lau
361c06df75 Add min/max offset to stats 2017-07-13 15:29:41 -07:00
Paul Cruz
65a4ce2635 added tests for forced compression level 2017-07-13 14:57:24 -07:00
Paul Cruz
0d9665cef5 added additional tests for performance, allowed force compression level for testing purposes 2017-07-13 14:46:54 -07:00