Commit Graph

7796 Commits

Author SHA1 Message Date
Yann Collet
8b9cdd2597 fixed overlapping count & workspace special case 2020-07-26 22:40:21 -07:00
Yann Collet
051232223f optimized histogram
new version easier to vectorize
leads to smaller code and faster execution
notably at the last recombination stage
(basically, fixed cost per block).

Assembly inspected with godbolt

On my laptop, with `clang` and `-mavx2` :
2K block : 1280 MB/s -> 1550 MB/s
8K block : 1750 MB/s -> 1860 MB/s
2020-07-26 22:24:22 -07:00
Yann Collet
15c5e20023 updated documentation 2020-07-23 12:10:57 -07:00
Yann Collet
e33e5eece3
Merge pull request #2250 from facebook/twoFieldsArgs
CLI: simplified handling of separated-fields arguments
2020-07-22 09:49:34 -07:00
Yann Collet
cf6c4fcbd5
Merge pull request #2249 from facebook/fsefix
import some minor fixes from FSE project
2020-07-21 08:23:37 -07:00
Yann Collet
610c41f90f improved 2-args macro to support both syntaxes
both `--long-commmand=arg` and `--long-command arg`
are supported by macro `NEXT_FIELD()`
so that the long-command only needs to be listed once.

This extends the syntax to support new syntaxes like
`-o=FILE` or `-D=dict`,
though there is no need to advertise this capability for the time being.

Also : added `NEXT_UINT32()`,
which is wrapper around `NEXT_FIELD()`
to read integer parameters.
Use the wrapper to new field, such as `--memlimit`
which can now support both syntaxes too.
2020-07-20 17:41:32 -07:00
Yann Collet
23941eec04 added tests for newly enabled syntax
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
2020-07-17 13:31:15 -07:00
Yann Collet
7280c1dafa extended 2-fields macro usage to number arguments
leading to further simplications
2020-07-17 13:09:23 -07:00
Yann Collet
a7e5652672 CLI: simplified handling of separated-fields arguments
this patch reduces complexity associated with
commands requiring a separated arguments
such as :
-o filename
-D dictionary
--output-dir-flat dir
--output-dir-mirror dir

It used to be a multi-stage process with explicit context,
it's now simplified as a single macro.

Thanks to this simplification,
separated arguments logic has also been extended to
--patch-from XXX
--filelist XXX
which extends existing capability using =XXX
--patch-from=XXX
--filelist=XXX

Separated argument is useful for filenames and directories,
as it benefits from shell expansion
such as ~/dir/file
where the ~ is automatically translated by the shell.
In contrast --long-command=FILE does not interpret FILE,
so ~/ is transmitted as is to the main() function,
generally resulting in incorrect file name.
2020-07-17 12:46:36 -07:00
Yann Collet
40f20b6453
Merge pull request #2197 from neheb/hj
meson: remove build requirement for distutils
2020-07-17 09:16:11 -07:00
Yann Collet
62df5bca4c
Merge pull request #2243 from facebook/uclibc
fix uclibc's st_mtim
2020-07-17 09:14:20 -07:00
Yann Collet
c224367ede ensure workspace is large enough
even when MAX_TABLELOG is reduced
2020-07-16 20:33:50 -07:00
Yann Collet
21c273da84 import some minor fixes from FSE project 2020-07-16 20:25:15 -07:00
Felix Handte
a35f135617
Merge pull request #2246 from ffontaine/dev
programs/util.c: fix build without st_mtime
2020-07-16 12:40:43 -04:00
Yann Collet
0d0fc4544c
Merge pull request #2247 from facebook/revert-2244-dev
Revert "Fix -Wunused-variable under FUZZING_BUILD_MODE..."
2020-07-15 12:42:31 -07:00
Yann Collet
a44671b281
Revert "Fix -Wunused-variable under FUZZING_BUILD_MODE..." 2020-07-15 12:42:18 -07:00
Yann Collet
7d3d5f2a20
Merge pull request #2244 from hctim/dev
Fix -Wunused-variable under FUZZING_BUILD_MODE...
2020-07-15 12:40:17 -07:00
Fabrice Fontaine
26d01bdb26 programs/util.c: fix build without st_mtime
Since version 1.4.5 and commit
5af8cb7aea, if st_mtime is not defined,
programs/util.c uses utime without including utime.h which will raise
the following build failure on some of the buildroot autobuilders:

util.c: In function 'UTIL_setFileStat':
util.c:161:24: error: storage size of 'timebuf' isn't known
         struct utimbuf timebuf;
                        ^~~~~~~

Fixes:
 - http://autobuild.buildroot.org/results/be902c5d110f37bce622a2215191f155b7d3e7e0

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
2020-07-15 21:24:13 +02:00
Mitch Phillips
23b55d6b3e Fix -Wunused-variable under FUZZING_BUILD_MODE...
Fuzzing build modes (FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) doesn't
necessarily imply that assert() is enabled, according to the manual.

When the current do-nothing is expanded under -Wunused-variable (-Wall),
it results in unused variables in some of the FUZZING_BUILD_MODE...
blocks.

This patch extends the do-nothing to avoid the unused variable.
2020-07-14 09:03:02 -07:00
Yann Collet
1dcc478796 fix uclibc's st_mtim
reported and suggested by @ewildgoose (#1872)
2020-07-13 14:16:33 -07:00
Nick Terrell
bd21e4b264
Merge pull request #2242 from evverx/cifuzz-ubsan-msan
cifuzz: turn on UBsan and MSan
2020-07-13 14:26:13 -04:00
Evgeny Vereshchagin
8e20f133e7 cifuzz: turn on UBsan and MSan 2020-07-13 01:20:11 +00:00
Yann Collet
16b353b207 minor doc clarification regarding MT parameters 2020-07-11 02:16:52 -07:00
Nick Terrell
bdf716bc64
Merge pull request #2239 from evverx/evverx-cifuzz
cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets
2020-07-10 11:26:15 -04:00
Yann Collet
a3296dae38
Merge pull request #2219 from xxie24/output-dir-mirror
Add output-dir-mirror option
2020-07-10 01:21:47 -07:00
Yann Collet
865ecf67ae
Merge pull request #2183 from tonytheodore/patch-1
Install pkg-config file on MINGW also
2020-07-10 01:20:59 -07:00
Yann Collet
78ebe4b5de
Merge pull request #2237 from marxin/gcda-to-gitignore
Add GCC PGO data files to .gitignore.
2020-07-09 08:48:36 -07:00
Yann Collet
c0ee26ec15
Merge pull request #2236 from marxin/pgo-fix-comment
Update comment about PGO build.
2020-07-09 08:48:04 -07:00
Yann Collet
2fbc438a5d
Merge pull request #2231 from marxin/use-threads-in-example
Use threads in streaming_compression.c example.
2020-07-09 08:41:44 -07:00
Martin Liska
7276ab15bc
Add GCC PGO data files to .gitignore. 2020-07-09 09:32:54 +02:00
Martin Liska
5b9d6b992a
Update comment about PGO build. 2020-07-09 09:31:51 +02:00
Martin Liska
461348c09a
Use threads in streaming_compression.c example.
Fixes #1584.
2020-07-09 09:20:06 +02:00
Evgeny Vereshchagin
3b9df5896f cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets
It should address https://github.com/google/oss-fuzz/issues/3982
2020-07-09 06:02:48 +00:00
Yann Collet
978cea5d56
Merge pull request #2230 from cwoffenden/minor-tidy
Remove superfluous build-time macros (single-file libs)
2020-07-08 11:49:40 -07:00
Yann Collet
fedecb29d8
Merge pull request #2235 from bimbashrestha/decom-macros
[actions] removing flakey min-decomp
2020-07-08 11:49:04 -07:00
Bimba Shrestha
2598af451f removing min decomp 2020-07-08 11:11:45 -07:00
Carl Woffenden
9bfd371d9f Remove superfluous build-time options 2020-07-08 11:39:17 +02:00
Yann Collet
2cdd33ae16
Merge pull request #2227 from yoshihitoh/single-file-dict-emscripten
[contrib] Fix single-file compilation error on Emscripten build.
2020-07-07 08:51:20 -07:00
yoshihitoh
bc5fe33243 Change Emscripten docker image to the official one which is used for testing the single-file feature. 2020-07-07 19:10:12 +09:00
Yann Collet
f014c3cd40
Merge pull request #2228 from marxin/rename-ZSTDMT_NBTHREADS_MAX
Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX.
2020-07-06 16:06:49 -07:00
Martin Liska
4307e870a2
Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX.
Rename it in zstd man page.
2020-07-07 00:18:20 +02:00
yoshihitoh
a035654ab8 Enable docker based Emscripten build on single file library test if emcc is not available. 2020-07-05 22:25:20 +09:00
Nick Terrell
a2b6a8fbc6
Merge pull request #2220 from dkcasset/fix-lib-Makefile-older-sed-version
Replace -E option with equivalent -r for older versions of sed
2020-06-29 17:19:24 -07:00
Nick Terrell
3690fe30f3
Merge pull request #2221 from BioDataAnalysis/bda_install_dll
Install MSVC runtime shared libraries (DLL) with cmake
2020-06-29 17:14:50 -07:00
dkcasset
82e7e2b47e Add variable for sed extended RE option (defaults to -E) 2020-06-29 13:44:23 -07:00
Mario Emmenlauer
37f36b9670 build/cmake/lib/CMakeLists.txt: Install shared library (dll) for Windows 2020-06-29 14:15:12 +02:00
yoshihitoh
c6548eac8e Rename static vars to avoid redefinition error. 2020-06-29 10:51:50 +09:00
dkcasset
b0ed66ef92 Replace -E option with equivalent -r for older versions of sed 2020-06-26 10:43:28 -07:00
Xin Xie
9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Nick Terrell
7afd5d85d3
Merge pull request #2218 from terrelln/assert-seq
Fix unused variable warnings in fuzzing build mode without asserts
2020-06-22 17:41:18 -07:00