9d6ed9def3
* Minor fix * Run non-optimize FASTCOVER 5 times in benchmark * Merge fastCover into dictBuilder * Fix mixed declaration issue * Add fastcover to symbol.c * Add fastCover.c and cover.h to build * Change fastCover.c to fastcover.c * Update benchmark to run FASTCOVER in dictBuilder * Undo spliting fastcover_param into cover_param and f * Remove convert param functions * Assign f to parameter * Add zdict.h to Makefile in lib * Add cover.h to BUCK * Cast 1 to U64 before shifting * Remove trimming of zero freq head and tail in selectSegment and rebenchmark * Remove f as a separate parameter of tryParam * Read 8 bytes when d is 6 * Add trimming off zero frequency head and tail * Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed) * Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary * Change nbDmer to always read 8 bytes even when d=6 * Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER * Update comments and benchmarking result * Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover * Add dictType enum and fix bug about passing zParam when converting to coverParam * Combine finalize and skip into a single parameter * Update acceleration parameters and benchmark on 3 sample sets * Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets * Initialize variables outside of for loop in benchmark.c * Update benchmark result for hg-manifest * Remove cover.h from install-includes * Add explanation of f * Set default compression level for trainFromBuffer to 3 * Add assertion of fastCoverParams in DiB_trainFromFiles * Add checkTotalCompressedSize function + some minor fixes * Add test for multithreading fastCovr * Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish * Free segmentFreqs * Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment * Add FASTCOVER_MEMMULT * Minor fix * Update benchmarking result
40 lines
1.5 KiB
C
40 lines
1.5 KiB
C
/*
|
|
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
|
* All rights reserved.
|
|
*
|
|
* This source code is licensed under both the BSD-style license (found in the
|
|
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
|
* in the COPYING file in the root directory of this source tree).
|
|
* You may select, at your option, one of the above-listed licenses.
|
|
*/
|
|
|
|
/* This library is designed for a single-threaded console application.
|
|
* It exit() and printf() into stderr when it encounters an error condition. */
|
|
|
|
#ifndef DIBIO_H_003
|
|
#define DIBIO_H_003
|
|
|
|
|
|
/*-*************************************
|
|
* Dependencies
|
|
***************************************/
|
|
#define ZDICT_STATIC_LINKING_ONLY
|
|
#include "zdict.h" /* ZDICT_params_t */
|
|
|
|
|
|
/*-*************************************
|
|
* Public functions
|
|
***************************************/
|
|
/*! DiB_trainFromFiles() :
|
|
Train a dictionary from a set of files provided by `fileNamesTable`.
|
|
Resulting dictionary is written into file `dictFileName`.
|
|
`parameters` is optional and can be provided with values set to 0, meaning "default".
|
|
@return : 0 == ok. Any other : error.
|
|
*/
|
|
int DiB_trainFromFiles(const char* dictFileName, unsigned maxDictSize,
|
|
const char** fileNamesTable, unsigned nbFiles, size_t chunkSize,
|
|
ZDICT_legacy_params_t* params, ZDICT_cover_params_t* coverParams,
|
|
ZDICT_fastCover_params_t* fastCoverParams, int optimize);
|
|
|
|
#endif
|