Commit Graph

1457 Commits

Author SHA1 Message Date
Conrad Meyer
fe82637069 Fix #1425 - Use physical core count API on FreeBSD
Similar to Apple, use the native physical core count sysctl, when available.

This is a little repetitive (it's basically the __APPLE__ method plus the
otherBSD method concatenated together) but seemed clearer than any way that
would totally eliminate repetition.

The __FreeBSD_version check only tests the version of the FreeBSD kernel
that zstd is compiled on; importantly, it may be run on a different version.
So the compile-time check is a little naive and needs to be able to fallback
to work on older versions of FreeBSD.  For a similar reason, it may make
sense to simply eliminate the __FreeBSD_version check entirely.  The
tradeoff is that a spurious sysctlbyname would be issued when -T0 is used on
older kernels.
2019-01-04 11:57:12 -08:00
Yann Collet
6b7a1d6127 fixed detection of input==output on Visual
due to bad support of inode identifiers.
On Visual, option is limited to same file name,
which is imperfect, but way better than disabling the feature entirely.

It's enough to pass associated tests.
2018-12-26 15:51:34 -08:00
Yann Collet
ae1d6bd48e fixed playTests.sh for minGW
On Windows, the equivalent of `/dev/null` is `NUL`.
When tests are run under msys2/minGW,
the environment identifies itself as Windows,
hence the script uses `NUL` instead of `/dev/null`
but the environment will consider `NUL` to be a regular file name.
Consequently, `NUL` will be overwritten during tests,
triggering an error.

This patch uses flag `-f` to force such overwrite
passing the test.
2018-12-26 15:19:09 -08:00
Yann Collet
1fdba696ca fixed VS2008 project
bench.c => benchfn.c + benchzstd.c
wrong util.c from prior patch

and warnings :
C4127
2018-12-25 16:14:14 -08:00
Yann Collet
ededcfca57 fix confusion between unsigned <-> U32
as suggested in #1441.

generally U32 and unsigned are the same thing,
except when they are not ...

case : 32-bit compilation for MIPS (uint32_t == unsigned long)

A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).

Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
  but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
  but the caller user a pointer to U32 instead.

These fixes are useful.

However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.

As a consequence, the amount of code changed is larger than I would expect for such a patch.

Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.

So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.

I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.

Thoughts ?
2018-12-21 18:09:41 -08:00
Yann Collet
8f35c7f94c
Merge pull request #1466 from facebook/noDictPresent
fixed : better error message
2018-12-20 19:01:27 -08:00
Yann Collet
18434d76b8 added strerror in comment
as suggested by @felixhandte
2018-12-20 17:27:08 -08:00
Yann Collet
41b45b84a1
Merge pull request #1465 from facebook/noFilePresent
fixed : detection of non-existing file
2018-12-20 17:21:04 -08:00
Yann Collet
ed2fb6bd57 fixed : better error message when dictionary missing
during benchmark.
Also : refactored ZSTD_fillHashTable(),
just for readability (it does the same thing)
2018-12-20 17:20:07 -08:00
Yann Collet
6e9512a70c
Merge pull request #1463 from yijinfb/getenv
Add support for environment variable ZSTD_CLEVEL in CLI
2018-12-20 15:17:00 -08:00
Yann Collet
0ed8ee4a37 fixed wrong assert condition 2018-12-20 14:46:23 -08:00
Yann Collet
ffba142406 fixed file identity detection in 32-bit mode
also :
some library decided to use `index` as a global variable declared in standard header
shadowing the ones used in fastcover.c  :(
2018-12-20 14:30:30 -08:00
Yann Collet
65a441a8f0 fixed stdlib implementation of time functions
generated redefinitions
2018-12-20 14:02:50 -08:00
Yann Collet
72dbf1bcd0 removed strncpy() from util.c
as Visual surprisingly complains about their usage.
Replaced by memcpy()
2018-12-20 12:27:12 -08:00
Yann Collet
105fa953cb use strerror() to generate error message
as suggested by @terrelln .

also:
- hopefully fixed Windows version
- changed the test, so that it passes on non-english OS stdlib errors.
2018-12-20 09:16:40 -08:00
Yi Jin
9b2d708850 change function name: _readU32FromChar() -> readU32FromCharChecked() 2018-12-19 23:41:18 -08:00
Yann Collet
173ef9dea2 fixed : detection of non-existing file
better error message
with test
2018-12-19 18:30:57 -08:00
Yi Jin
f7b1841b6f make init_cLevel(...) static 2018-12-19 17:56:45 -08:00
Yi Jin
30ffc24ad7 fix the code ahead of declaration issue 2018-12-19 17:49:04 -08:00
Yi Jin
0700335f57 change int init_cLevel() to int init_cLevel(void) 2018-12-19 17:38:28 -08:00
Yann Collet
0f2d443e10 added system error strings
to more failure cases in fileio.c
as requested by fbpkg
2018-12-19 17:25:58 -08:00
Yi Jin
cdc7bbf8b2 edit README.md 2018-12-19 17:09:54 -08:00
Yi Jin
29c7d82390 add a section on restricted support of environment variables to README.md 2018-12-19 17:06:56 -08:00
Yi Jin
26a9ae3f5f refactor readU32FromChar(...), improve init_cLevel(...), and add env var ZSTD_CLEVEL tests 2018-12-19 16:45:42 -08:00
Yi Jin
5324b1e386 add support for setting compression level through environment variable ZSTD_CLEVEL 2018-12-19 13:26:27 -08:00
Nick Terrell
7abd6acebf Also compare device thanks to @felixhandte 2018-12-18 15:40:27 -08:00
Nick Terrell
cd2c8defad [zstdcli] Refuse to overwrite input file
Compare the input and output files by their inode number and
refuse to open the output file if the input file is the same.

This doesn't work when (de)compressing multiple files to a single
file, but that is a very uncommon use case, mostly used for
benchmarking by me.

Fixes #1422.
2018-12-18 15:29:54 -08:00
Yann Collet
517d8c984c
Merge pull request #1449 from facebook/ovlog_def
overlapLog default values
2018-12-18 09:45:53 -08:00
Nick Terrell
bdfcaecc0a [zstdcli] Add --no-progress flag
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
2018-12-14 11:50:25 -08:00
Yann Collet
1993f5d412 fixed ovlog tests
and updated man page
2018-12-12 21:09:14 -08:00
Yann Collet
f2f86d369b Merge branch 'btultra2' into ovlog_def 2018-12-12 20:58:14 -08:00
Yann Collet
9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet
c313a85ee2 removed exception code for overlapLog level 22
CLI used to set overlapLog at value 9 when level == 22.
This is no longer necessary (handled internally within library)
2018-12-11 18:13:06 -08:00
Yann Collet
9b784dec7f changed parameter name to ZSTD_c_overlapLog
from overlapSizeLog.

Reasoning :
`overlapLog` is already used everwhere, in the code, command line and documentation.
`ZSTD_c_overlapSizeLog` feels unnecessarily different.
2018-12-11 16:55:33 -08:00
Yann Collet
9c3265a53f
Merge pull request #1417 from facebook/advancedAPI
Advanced API
2018-12-10 18:48:15 -08:00
Yann Collet
d613fd9afe linked btultra2 as strategy9
and ensure zstdbench detects out-of-bound parameters
2018-12-06 19:27:37 -08:00
Yann Collet
34aa401afd updated documentation
introducing ZSTD_btultra2
2018-12-06 17:22:19 -08:00
Yann Collet
be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy
also : fixed paramgrill, and limit conditions
2018-12-06 15:00:52 -08:00
Yann Collet
3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_*
for naming consistency
2018-12-05 17:26:02 -08:00
Yann Collet
34e146f548 advanced decompression function replaces by normal streaming one
advanced parameters compatible with ZSTD_decompressStream().
2018-12-04 10:28:36 -08:00
Yann Collet
7ef7dc561a check availability of --color=never command on grep and egrep
before applying them.
Fixes #1436
2018-12-03 15:46:55 -08:00
Yann Collet
d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2()
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
2018-11-30 11:25:56 -08:00
Yann Collet
f6591e4a04
Merge pull request #1430 from lzutao/fix-zstdgrep
Fix  #1428 - zstdgrep now returns 1 on unmatch
2018-11-28 14:07:40 -08:00
Lzu Tao
3d18b4764d Prevent globbing on non-quoting variable 2018-11-29 03:04:40 +07:00
Lzu Tao
c046e0b626 Fix #1428 - zstdgrep now returns 1 on unmatch 2018-11-29 02:45:01 +07:00
Denis Ahrens
71b8ee1bf1 fixed a typo 2018-11-28 04:51:16 +01:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
Yann Collet
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int
for compatibility with compression level
2018-11-20 11:53:01 -08:00
Ryan Schmidt
46d470c20c Fix feature detection with multiple -arch flags
When multiple -arch flags are used, the compiler invokes itself once for
each architecture. Apparently, input on stdin is consumed by the
compilation of the first arch and is no longer available to the
compilation of the second arch, which results in a build failure and the
potentially incorrect determination that a feature is not available. So
write the feature detection source to a file instead of using stdin.
2018-11-16 03:49:15 -06:00