Commit Graph

5574 Commits

Author SHA1 Message Date
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet
1af27a7ed7
Merge pull request #1287 from tgurr/cmake-docdir
Use GNUInstallDirs DOCDIR for installing the manual
2018-08-21 14:48:50 -07:00
Yann Collet
801e3bcd97
Merge pull request #1290 from edenzik/ezik/1119-safe-strcpy-in-fileio
Fixed unsafe string copy and concat in `fileio.c`.
2018-08-21 13:18:44 -07:00
Eden Zik
78af534f82 Fixed unsafe string copy and concat in fileio.c.
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c
2018-08-20 22:15:24 -04:00
Yann Collet
ea0b5fc193
Merge pull request #1285 from facebook/scanbuild
static analyzer tests
2018-08-17 16:38:41 -07:00
Timo Gurr
e400a86f17 Use GNUInstallDirs DOCDIR for installing the manual 2018-08-17 13:50:00 +02:00
Yann Collet
77fa1d6e42
Merge pull request #1284 from GeorgeLu97/negativeLvlDisplay
Fix negative lvl display value
2018-08-16 18:20:37 -07:00
George Lu
3959ba15e6 Clarify README 2018-08-16 17:22:29 -07:00
George Lu
8175b28f03 Fix negative lvl display value
Also fix synthetic benchmark parameter setting
2018-08-16 16:46:37 -07:00
Yann Collet
36d6165a2d Makefile: added variable SCANBUILD
so that a different version of scan-build can be selected
2018-08-16 16:44:13 -07:00
Yann Collet
1515f0bb0d fixed more issues detected by recent version of scan-build
test run on Linux
2018-08-16 15:20:25 -07:00
Yann Collet
1dd3156f2b
Merge pull request #1281 from GeorgeLu97/cleanUp
Clean up paramgrill code
2018-08-15 17:56:45 -07:00
Yann Collet
31224cc126 added static analyzer tests to travis CI 2018-08-15 17:44:27 -07:00
Yann Collet
5291d9ac31 fix scope of scan-build tests
exclude zlib code
2018-08-15 17:41:44 -07:00
Yann Collet
da55865e47 ensure dependency for zlib wrapper 2018-08-15 16:43:13 -07:00
George Lu
239e114d62 prune comments 2018-08-15 16:04:34 -07:00
George Lu
8a296d3e1f Move Stuff around
Group similar functions together, remove outdated comments
2018-08-15 16:04:34 -07:00
George Lu
3f8b10baa1 consts 2018-08-15 16:04:34 -07:00
George Lu
46be2ef5d8 Remove unused stuff 2018-08-15 16:04:34 -07:00
Yann Collet
512919415c
Merge pull request #1280 from GeorgeLu97/betterDisplay
Better Paramgrill Display
2018-08-15 15:20:56 -07:00
Yann Collet
42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
George Lu
b234870c33 clarify display README 2018-08-15 14:29:49 -07:00
George Lu
ee77ddc28d Fix wraparound 2018-08-15 14:01:32 -07:00
George Lu
1e8d352930 silencing params 2018-08-15 14:01:32 -07:00
George Lu
2c5fdae0ae Clean up repetitive display
Add documentation
2018-08-15 14:01:32 -07:00
George Lu
4d9c6f51b8 -q -v options 2018-08-15 14:01:32 -07:00
George Lu
3dcfe5cc2c begin display changes 2018-08-15 14:01:32 -07:00
Yann Collet
3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
Yann Collet
a01b964207
Merge pull request #1277 from GeorgeLu97/mtoptions
memoTableOptions
2018-08-15 13:45:17 -07:00
George Lu
b1d9ca737a Add memoTable options
-hashing memotable
-no memotable
2018-08-15 10:19:38 -07:00
Yann Collet
6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
Yann Collet
90a3901d1e
Merge pull request #1279 from GeorgeLu97/benchSA
Fix scan-build warnings in bench.c
2018-08-14 16:44:28 -07:00
George Lu
8c918edd3a MAke it easier to add params
Make memoTable size limited
2018-08-14 16:15:46 -07:00
George Lu
96725989ef Temp fix perf regression 2018-08-14 16:14:37 -07:00
Yann Collet
0c9772b084
Merge pull request #1276 from GeorgeLu97/fadt
forceAttachDict
2018-08-14 16:06:18 -07:00
George Lu
e89f1fb45c Fix scan-build warnings in bench.c 2018-08-14 14:44:47 -07:00
George Lu
3f2d024dca forceAttachDict 2018-08-14 14:24:41 -07:00
George Lu
e3c679484a Add Time Checks
Fix double -> U64 display
2018-08-14 14:24:41 -07:00
George Lu
88dda92285 Reduce Duplication
Change Defaults
Asserts actually disabled in paramgrill + fullbench
2018-08-14 14:24:41 -07:00
George Lu
f581ccd267 Doc Updates
Add option to pass in existing parameters in use
2018-08-14 14:24:41 -07:00
Yann Collet
e19c79900c
Merge pull request #1275 from GeorgeLu97/multilvl
MultiLvl
2018-08-14 14:17:28 -07:00
George Lu
76acba025d scan-build 2018-08-14 12:13:05 -07:00
George Lu
614aaa3ae1 rebase clevel 2018-08-14 10:53:04 -07:00
Yann Collet
a25d8000ec
Merge pull request #1278 from edenzik/fb/ezik/better-cmake-support-1081
Cmake now builds with CMAKE_BUILD_TYPE=Release by default
2018-08-13 18:38:05 -07:00
Eden Zik
a6df961497 Cmake now builds with CMAKE_BUILD_TYPE=Release by default, both while
being invoked from the main Makefile (via cmakebuild) or directly from
the build/cmake directory. Suggested by @pdknsk (#1081).
2018-08-13 20:28:52 -04:00
George Lu
3b36fe5c68 strategy switching 2018-08-13 16:36:14 -07:00
George Lu
d4730a4f66 Update fulltable to use same interface
Add seperateFiles flag
2018-08-13 16:15:52 -07:00
George Lu
43b4971ca8 Renames, Documentation Updates 2018-08-13 16:15:52 -07:00
George Lu
a884b76bc2 Style Changes
Add single run dictionaries
Change MB to be consistent 1 << 20 rather than 1,000,000
2018-08-13 16:15:52 -07:00
George Lu
b3544217b7 Cleanup 2018-08-13 16:15:52 -07:00