Commit Graph

874 Commits

Author SHA1 Message Date
Nick Terrell
3b56bb1e4c Fix decodecorpus 2018-08-23 17:48:06 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet
801e3bcd97
Merge pull request #1290 from edenzik/ezik/1119-safe-strcpy-in-fileio
Fixed unsafe string copy and concat in `fileio.c`.
2018-08-21 13:18:44 -07:00
Eden Zik
78af534f82 Fixed unsafe string copy and concat in fileio.c.
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c
2018-08-20 22:15:24 -04:00
Yann Collet
ea0b5fc193
Merge pull request #1285 from facebook/scanbuild
static analyzer tests
2018-08-17 16:38:41 -07:00
George Lu
3959ba15e6 Clarify README 2018-08-16 17:22:29 -07:00
George Lu
8175b28f03 Fix negative lvl display value
Also fix synthetic benchmark parameter setting
2018-08-16 16:46:37 -07:00
George Lu
239e114d62 prune comments 2018-08-15 16:04:34 -07:00
George Lu
8a296d3e1f Move Stuff around
Group similar functions together, remove outdated comments
2018-08-15 16:04:34 -07:00
George Lu
3f8b10baa1 consts 2018-08-15 16:04:34 -07:00
George Lu
46be2ef5d8 Remove unused stuff 2018-08-15 16:04:34 -07:00
George Lu
b234870c33 clarify display README 2018-08-15 14:29:49 -07:00
George Lu
ee77ddc28d Fix wraparound 2018-08-15 14:01:32 -07:00
George Lu
1e8d352930 silencing params 2018-08-15 14:01:32 -07:00
George Lu
2c5fdae0ae Clean up repetitive display
Add documentation
2018-08-15 14:01:32 -07:00
George Lu
4d9c6f51b8 -q -v options 2018-08-15 14:01:32 -07:00
George Lu
3dcfe5cc2c begin display changes 2018-08-15 14:01:32 -07:00
Yann Collet
3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
George Lu
b1d9ca737a Add memoTable options
-hashing memotable
-no memotable
2018-08-15 10:19:38 -07:00
George Lu
8c918edd3a MAke it easier to add params
Make memoTable size limited
2018-08-14 16:15:46 -07:00
George Lu
96725989ef Temp fix perf regression 2018-08-14 16:14:37 -07:00
George Lu
3f2d024dca forceAttachDict 2018-08-14 14:24:41 -07:00
George Lu
e3c679484a Add Time Checks
Fix double -> U64 display
2018-08-14 14:24:41 -07:00
George Lu
88dda92285 Reduce Duplication
Change Defaults
Asserts actually disabled in paramgrill + fullbench
2018-08-14 14:24:41 -07:00
George Lu
f581ccd267 Doc Updates
Add option to pass in existing parameters in use
2018-08-14 14:24:41 -07:00
George Lu
76acba025d scan-build 2018-08-14 12:13:05 -07:00
George Lu
614aaa3ae1 rebase clevel 2018-08-14 10:53:04 -07:00
George Lu
3b36fe5c68 strategy switching 2018-08-13 16:36:14 -07:00
George Lu
d4730a4f66 Update fulltable to use same interface
Add seperateFiles flag
2018-08-13 16:15:52 -07:00
George Lu
43b4971ca8 Renames, Documentation Updates 2018-08-13 16:15:52 -07:00
George Lu
a884b76bc2 Style Changes
Add single run dictionaries
Change MB to be consistent 1 << 20 rather than 1,000,000
2018-08-13 16:15:52 -07:00
George Lu
b3544217b7 Cleanup 2018-08-13 16:15:52 -07:00
George Lu
8ff0de15e4 Generalize, macro magic numbers 2018-08-13 16:15:52 -07:00
George Lu
3a2e95eba4 Perf improvements
try decay
strategy selection skipping
2018-08-13 16:15:52 -07:00
George Lu
2bdfe6ca71 Better Display 2018-08-13 16:15:52 -07:00
George Lu
f67d040c39 Bugfixes, style changes
Complete euclidean distance climb
2018-08-13 16:15:52 -07:00
George Lu
5f4502fc07 New climb
feas part 2 uses euclidean metric
2018-08-13 16:15:52 -07:00
George Lu
13611249a5 Table
Compiling
+Euclidean Metric
2018-08-13 16:15:52 -07:00
George Lu
0cea754024 Revert "Reorder declaration"
This reverts commit 3ac2c22485.
2018-08-13 16:15:34 -07:00
George Lu
486e586eed Revert "Default lvl 1"
This reverts commit 0cc75d6ee0.
2018-08-13 16:13:46 -07:00
George Lu
0cc75d6ee0 Default lvl 1
MB to 2^20
2018-08-13 14:55:56 -07:00
George Lu
3ac2c22485 Reorder declaration 2018-08-09 16:38:32 -07:00
George Lu
bfe8392e23 Remove ctx from benchMem 2018-08-09 12:07:57 -07:00
George Lu
0ece2e5cdc Add consts
+ fix gcc-8 warnings
2018-08-09 11:38:09 -07:00
George Lu
6f480927af argument parsing cleanup
+ clarifying comment
2018-08-09 10:42:58 -07:00
George Lu
ad16a69408 Readability improvements, renaming 2018-08-09 10:42:58 -07:00
George Lu
8278a49cb6 const srcPtrs 2018-08-09 10:42:58 -07:00
George Lu
3d230db853 Change speed representation from floating point to integral 2018-08-09 10:42:58 -07:00
George Lu
8faeb41679 Update Documentation
Change comment // to  /* */
Add more description of what functions do
Remove outdated comments
2018-08-09 10:42:58 -07:00
George Lu
dd270b2f75 Renaming / Style fixes 2018-08-09 10:42:58 -07:00