Commit Graph

1225 Commits

Author SHA1 Message Date
George Lu
50d612f4f0 Interleave compression/decompression
Fix Bugs
2018-06-25 15:01:03 -07:00
George Lu
d6121ad0e1 Opaque State
And minor fixups (comments/alignment/checks/fix memory leak)
2018-06-25 08:07:43 -07:00
George Lu
ab26f24c9c benchFunction Timed Wrappers
Add BMK_benchFunctionTimed
Add BMK_init_customResultCont..
Change benchMem to use benchFunctionTimed
Minor Fixes/Adjustments
2018-06-21 16:23:55 -07:00
George Lu
a8eea99ebe Incremental Display + Fn Separations
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
2018-06-21 16:23:18 -07:00
George Lu
a3c8b59990 Fix cli no print
Change looping behavior to match old
2018-06-18 15:38:14 -07:00
George Lu
e482e328cd Reorder Arguments
make initFn nullable
2018-06-18 13:21:42 -07:00
George Lu
0d1ee22990 Requested Changes
Add Comment
Simplify Interface (Remove resultSet)
Reorder Arguments
Remove customBench displayLevel
Reorder bench.h
Change benchFiles return type to match advanced
Rename stuff
2018-06-18 12:01:12 -07:00
George Lu
8522346322 Make Fullbench use new function
Rearrange Args
Add nothing function
Use new function, change locals to match
New Display
Comment cleanup
Change builds
2018-06-15 11:37:49 -04:00
George Lu
20f4f32379 Add to bench
-Remove global variables
-Remove gv setting functions
-Add advancedParams struct
-Add defaultAdvancedParams();
-Change return type of bench Files
-Change cli to use new interface
-Changed error returns to own struct value
-Change default compression benchmark to use decompress_generic
-Add CustomBench function
-Add Documentation for new functions
2018-06-14 14:23:24 -04:00
Yann Collet
c986dbf241
Merge pull request #1168 from GeorgeLu97/paramgrillfeatures
Have paramgrill share bench.c benchmarking function
2018-06-13 11:38:29 -04:00
George Lu
01d940b670 Requested changes
-Remove g_displaylevel/setNotificationLevel function
-Add extern "C"
-Remove averaging
-Reorder arguments

More fixes

-Added BMK_return_t (result + possible error)
-Correct comment'
-Nullcheck ctx, dctx when allocated
-Remove extra assert
2018-06-12 17:02:44 -04:00
Ryan Schmidt
b567ce9d68 Fix name of macOS 2018-06-09 14:31:17 -05:00
George Lu
0e808d608b Make paramgrill use bench.c benchmarking 2018-06-08 12:01:05 -07:00
Yann Collet
d3615c28db
Merge pull request #1159 from GeorgeLu97/suffixlist
Unknown Suffix Error
2018-06-01 14:00:10 -07:00
George Lu
8984cc93d6 update display 2018-05-31 18:04:05 -07:00
George Lu
547096d672 update man 2018-05-31 18:03:52 -07:00
George Lu
c9b1068298 removed strcats 2018-05-31 17:47:29 -07:00
George Lu
5ff30fe2e5 Unknown Suffix Error
Changed so only compiled formats are printed in list of supported extensions
2018-05-31 16:13:36 -07:00
George Lu
140f59d38e Added --format=zstd
title
2018-05-31 15:29:35 -07:00
Yann Collet
174bd3d4a7
Merge pull request #1131 from facebook/zstdcli
minor: control numeric argument overflow
2018-05-14 11:53:58 -07:00
Yann Collet
9cd5c63771 cli: control numeric argument overflow
exit on overflow
backported from paramgrill
added associated test case
2018-05-12 14:29:33 -07:00
Yann Collet
b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
cyan4973
62487b5e76 fixed decoding bogus lz4 frame
FIO would keep presenting data after an LZ4F decoding error
resulting in a NULL pointer dereference
when associated with older liblz4 version (< v1.8.1.2)
2018-04-23 18:50:16 -07:00
Yann Collet
1da629f2ad
Merge pull request #1104 from terrelln/fast-train
Allow negative compression levels in training
2018-04-09 14:16:20 -07:00
Nick Terrell
569e2abccd Allow negative compression levels in training
* Set `dictCLevel` in `zstdcli.c`.
* Only set to default level if the compression level `== 0`, not `<= 0`.
2018-04-09 12:12:03 -07:00
Björn Ketelaars
e5ea8d272a fix typo in programs/zstd.{1,1.md}
s/nodictID/no-dictID/g
2018-04-05 06:44:46 +02:00
Yann Collet
7188862d32
Merge pull request #1086 from hagemt/hagemt-patch-1
Correct small typo in manual (man file and markdown)
2018-03-30 20:45:10 -06:00
Tor E Hagemann
c7a5e60bc6
Update zstd.1.md 2018-03-30 15:25:32 -07:00
Tor E Hagemann
292d370ab4
Update zstd.1 2018-03-30 14:53:57 -07:00
Yann Collet
525f3fab33 restored ability to manually set overlapLog 2018-03-28 11:33:41 -06:00
Yann Collet
01082a39bd restored simple status line during zstd compression
the more advanced one, featuring amount of data buffered,
is triggered on `-v`.
2018-03-22 17:49:46 -07:00
Yann Collet
153bc1c004 removed limit ZSTD_TARGETLENGTH_MAX
this makes it possible to specify extremely large negative compression levels,
achieving the side effect as "no compression".

It will also be possible to define larger targetlength for ultra compression mode.

There is no adverse side effect due to removing this limit.
2018-03-21 15:50:05 -07:00
Yann Collet
353117c5d7 implemented ZSTD_DCtx_loadDictionary*()
this required updating ZSTD_createDDict_advanced()
to accept a dictContentType parameter (raw, full, auto).
2018-03-20 13:40:29 -07:00
Yann Collet
4c5cbac179
Merge pull request #1041 from facebook/fasterFast
Negative compression levels
2018-03-13 21:32:46 -07:00
Yann Collet
bd7bb94361
Merge pull request #1044 from baldurk/remove-utf8-characters
Remove non-ASCII characters in header file comments
2018-03-13 13:22:07 -07:00
Baldur Karlsson
430a2fec19 Remove non-ASCII characters in header file comments
* Replaced a non-breaking space and an en dash with a plain space and
  a hyphen.
* This means the files are simple ASCII and less likely to run into
  codepage issues.
2018-03-13 20:05:53 +00:00
Jesse Talavera-Greenberg
2f70fbf2a3
Made -H's printout specify the semantics of -T0 2018-03-12 20:43:32 -04:00
Yann Collet
a57d43d4d4 updated documentation of targetLength 2018-03-12 11:35:01 -07:00
Yann Collet
f24566b597 minor bench improvements
- do not test level 0, as it is converted into level 3,
  which feels strange when compressing multiple levels
- Use direct synchronous mode when a single worker is requested.
2018-03-12 04:02:57 -07:00
Yann Collet
6a9b41b731 create command --fast[=#]
access negative compression levels from command line
for both compression and benchmark modes.

also : ensure proper propagation of parameters
through ZSTD_compress_generic() interface.

added relevant cli tests.
2018-03-11 20:01:23 -07:00
Yann Collet
a70f7e10fa Merge branch 'benchDecode' into longOffsetMode 2018-03-05 14:09:00 -08:00
Yann Collet
03e7e14192 fix benchmark issue when measuring only decoding speed
zstd bench module can focus on decompression speed _only_.
This is useful when trying to measure performance
on large input data compressed using a high level
as compression time becomes problematic (too long).

This mode is triggered by command : zstd -b -d

Problem was : in such a mode,
measured decoding speed was > 10% slower
than in nominal mode (compression + decompression),
making decompression benchmark mode much less useful.

This patch fixes the issue.
It's not completely clear why, but
moving the `memcpy()` operation sooner in the pipeline fixed it.

I can still measure some difference, but it is in the < 2% range,
so it's much more tolerable.

also : it doesn't matter anymore in which order are selected
commands `-b` and `-d`.
The combination always triggers bench_decodeOnly mode.
2018-03-05 13:57:41 -08:00
Yann Collet
41bd10446e Merge branch 'dev' into longOffsetMode 2018-03-05 13:10:10 -08:00
Yann Collet
b91ddf0ae6 Merge branch 'dev' into longOffsetMode 2018-03-05 11:59:54 -08:00
Conrad Meyer
606374269c FIO_addFInfo: Fully initialize output 'total' struct
Silence a Coverity warning about 'windowSize' being uninitialized.
(Yes, nothing that calls this routine actually uses the windowSize
value.  Still, appeasing Coverity is pretty harmless in this case.)
2018-02-28 15:23:05 -08:00
Yann Collet
25d00d10fc fixed minor conversion warning 2018-02-20 16:52:28 -08:00
Yann Collet
3538a535bf use TIMELOOP_NANOSEC
as suggested by @terrelln
2018-02-20 15:33:56 -08:00
Yann Collet
d3364aa39e improve benchmark measurement for small inputs
by invoking time() once per batch, instead of once per compression / decompression.
Batch is dynamically resized so that each round lasts approximately 1 second.

Also : increases time accuracy to nanosecond
2018-02-20 14:58:40 -08:00
Yann Collet
5cb1144872 fixed --single-thread
was incorrectly set to -T0 (use as many cores as possible) previously
2018-02-13 14:56:35 -08:00
Yann Collet
04a3f85ce7 fixed gcc warning on a switch code path 2018-02-09 16:16:27 -08:00