AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,943 Commits 1 Branch 0 Tags 27 MiB
41012193ad
Commit Graph

1825 Commits

Author SHA1 Message Date
Yann Collet
14e9010bb5
Merge pull request #1844 from AhmedAbdellah19/adding_read_files_from_file_feature 
Adding --file=FILE feature
 2019-10-25 10:11:47 -07:00
W. Felix Handte
91c3f545cc Fix Build; Refactor 2019-10-24 20:18:57 -04:00
Felix Handte
506e1a1176
Merge branch 'tzst-ext-tmp' into tzst-ext 2019-10-24 17:49:34 -04:00
Ahmed Abdellah
5249085e11 fixing free const char** filenamesTable 2019-10-24 20:54:40 +01:00
Yann Collet
37f90930da
Merge pull request #1839 from bimbashrestha/named_pipes 
Allowing named pipes to go through in zstdcli
 2019-10-24 10:53:12 -07:00
Ahmed Abdellah
5f9e868ee8 fixing type conversion error 2019-10-24 16:20:58 +01:00
Ahmed Abdellah
0b3096596a fixing AppVeyor errors 2019-10-24 15:25:48 +01:00
Ahmed Abdellah
cddb05ef8c fixing some warning 2019-10-24 14:42:37 +01:00
Ahmed Abdellah
5e206fdd53 fixing some warning 2019-10-24 14:21:01 +01:00
Ahmed Abdellah
0e6a73b148 fixing newTable issues and some warnings 2019-10-24 13:51:12 +01:00
Ahmed Abdellah
639bb46954 removing extra logs 2019-10-24 11:35:26 +01:00
Ahmed Abdellah
849b8c6de8 fixing continuous integeration errors and removing a lot of logs 2019-10-24 11:10:13 +01:00
Ahmed Abdellah
47712c9b15 fixing c90 issue in util.c cont. 2019-10-24 10:30:05 +01:00
Ahmed Abdellah
c799f33899 fixing c90 issue in util.c cont. again 2019-10-24 10:23:36 +01:00
Ahmed Abdellah
8a9741b3ee fixing c90 issue in util.c cont. 2019-10-24 10:17:31 +01:00
Ahmed Abdellah
aefa18ee38 fixing c90 issue in util.c 2019-10-24 10:12:51 +01:00
Ahmed Abdellah
f43e45954f fixing memory leak issue and removing c90 issue 2019-10-24 09:39:16 +01:00
Ahmed Abdellah
8cbe42fcb0 solving the rest of C90 issues in defining local variables in middle of code and comparing uncompatible types 2019-10-23 20:22:07 +01:00
Ahmed Abdellah
9a454e9724 solving C90 issues in defining local variables in middle of code and comparing uncompatible types 2019-10-23 20:15:18 +01:00
AhmedAbdellah19
7a732e2bb2
Merge branch 'dev' into adding_read_files_from_file_feature 2019-10-23 20:29:01 +02:00
Bimba Shrestha
8a3974807e Allowing named pipes to go through zstdcli 2019-10-22 15:23:22 -07:00
Yann Collet
514f941a70
Merge pull request #1836 from facebook/dirFlat2 
Added --output-dir-flat=
 2019-10-22 15:19:04 -07:00
Nick Terrell
b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data 
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
 2019-10-21 21:16:17 -07:00
Yann Collet
989e0f0cee Added --output-dir-flat= 
which is a preferred construction when applicable.

Also, added --output-dir-flat into man page.
 2019-10-21 14:39:18 -07:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix 
Windows fixes
 2019-10-21 13:45:17 -07:00
Yann Collet
632e07747c minor refactor of FIO_compressGzFrame() 
mostly narrowing variables lifetime and mutability.
 2019-10-21 12:14:59 -07:00
Yann Collet
ca73c218be added mode documentation on ZSTD_CLEVEL 
providing range restriction [1-19] explicitly
partially answering #1829
 2019-10-18 17:22:45 -07:00
Yann Collet
0492c57013 fixed visual defined test 2019-10-18 17:08:52 -07:00
Yann Collet
f379637025 removed Visual warning 
for pointer casts
 2019-10-18 17:05:42 -07:00
Yann Collet
1bd6c15b6b fix function cast warning on Windows with gcc9 2019-10-18 15:45:57 -07:00
Yann Collet
58c59341d3 updated windows binary resources 2019-10-18 15:41:12 -07:00
Yann Collet
29e46ed0bd fix test on windows 
isDirectory() doesn't work on Windows
if directory name is followed by '/'
 2019-10-18 14:28:34 -07:00
Yann Collet
ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Yann Collet
caf40d0ae4 fix : no output file opened in test mode 
also : redistributed error code within fileio.c
for more precise diagnosis.
 2019-10-17 16:58:49 -07:00
Yann Collet
0a24d4ef18 removed regular file test on Windows 
since it does not work well on this platform
(tested with MinGW).

Note : could be an issue within UTIL_isRegularFile()
 2019-10-17 16:39:47 -07:00
Yann Collet
0ee360982d improved test mode -t 
The test mode do no longer open a file `/dev/null` nor write anything to output.

This is supposed to be more efficient than writing to `/dev/null`,
and more universal too : the previous method was failing on Windows.
 2019-10-17 16:09:53 -07:00
Yann Collet
1795133c45 refactored FIO_compressMultipleFilenames() prototype 
for consistency
 2019-10-17 15:32:03 -07:00
Yann Collet
7f86ae2867 fixed multiple implicit casts 2019-10-17 15:27:25 -07:00
Yann Collet
157479af0c fixed isSameFile() 2019-10-17 14:31:42 -07:00
Yann Collet
a71256a2ee fix several cast 2019-10-17 11:01:20 -07:00
Yann Collet
000404311f fix incorrect dictName/FileName comparison on Windows 
inode identification does not seem to work on Windows,
even with on a msys2 posix layer.
 2019-10-17 10:56:14 -07:00
Yann Collet
19741c7d99
Merge pull request #1815 from facebook/zlibwrap 
make zlibWrapper strict ISO-C90 compatible
 2019-10-16 16:45:15 -07:00
Yann Collet
6309be677c minor comments & refactoring 2019-10-15 16:09:18 -07:00
Ahmed Abdellah
779ea72953 Adding --file=FILE feature 2019-10-15 07:49:13 +01:00
Yann Collet
8b6d96827c
Merge pull request #1731 from chungy/stdin_nochmod 
[programs] don’t do chmod when coming from stdin or multiple files
 2019-10-14 09:34:56 -07:00
Yann Collet
b3e92ad346
Merge pull request #1812 from senhuang42/output-dir-flat 
Support for --output-dir-flat: output into a directory
 2019-10-14 09:25:05 -07:00
Eric van Gyzen
46ee10dfb5 Include errno in all fwrite error messages in fileio.c 2019-10-10 09:21:29 -05:00
Sen Huang
c5ebb37051 Only check for filename collisions when using --output-dir-flat 2019-10-09 09:39:52 -04:00
Sen Huang
6e406b55ee Casting to void* to avoid C4090 warnings 2019-10-08 09:54:59 -04:00
Sen Huang
332b5988d8 Suppress C4090 warning 2019-10-08 09:44:24 -04:00
Sen Huang
f9b1da6f62 Hopefully fix VS compiler warning 2019-10-08 00:15:28 -04:00
Yann Collet
e0d413d648 fixed init warning 2019-10-04 15:09:52 -07:00
Sen Huang
6b81bfb257 Changed function name, fix test 2019-10-03 15:23:49 -04:00
Sen Huang
64bc441d7d Now constructs final destination path without allocating new table 2019-10-03 13:53:04 -04:00
Sen Huang
4dc604cab8 Addressed comments on malloc 2019-10-02 18:34:42 -04:00
Sen Huang
c763457e0a Static analyze fix 2019-10-02 15:30:24 -04:00
Sen Huang
b93f1b2a30 CI Tests fix 2019-10-02 11:29:34 -04:00
Sen Huang
147b761454 Removed strdup() dependency 2019-10-02 11:18:14 -04:00
Sen Huang
1e4d602412 Modified message 2019-10-02 11:11:35 -04:00
Sen Huang
f80437c586 Add support for --output-dir-flat 
New flag to specify output directory destination for multiple files.
 2019-10-02 11:08:20 -04:00
Yann Collet
ad2a2785f7 bump version number to v1.4.4 
so that future reports on `dev` branch use this number instead
 2019-09-24 15:15:33 -07:00
Sergey Ponomarev
dafe796e39 #1790 short tar's extensions tgz, txz, tlz4m .tzst should be decompressed with .tar suffix 2019-09-18 09:23:10 +03:00
Sergey Ponomarev
bfb4d830b2 FIO_determineDstName: extract dstFileNameEndPos variable 2019-09-18 09:21:00 +03:00
Sergey Ponomarev
59f369a6da Add short tar's extensions .tgz (.tar.gz), .txz (.tar.xz), .tzst (.tar.zst) 2019-09-14 21:30:15 +03:00
Sergey Ponomarev
a101721f4e Use one strstr() call instead of chain of strcmp() 2019-09-14 21:26:27 +03:00
Sergey Ponomarev
7d9cd22e21 #754 Add a hint about -o option 2019-09-14 21:23:47 +03:00
Sergey Ponomarev
8cc815a941 #754 sufixlist->SUFFIX_LIST 2019-09-14 21:15:24 +03:00
Sergey Ponomarev
b804dd3e5b #754 move sufixlist upper and improve error message on missing suffix 2019-09-14 21:14:43 +03:00
W. Felix Handte
e1ec8004cc Formatting and Clean Up 2019-09-12 16:27:05 -04:00
W. Felix Handte
5a9baae9cf Set M-Time Nanoseconds 2019-09-12 11:50:33 -04:00
Felix Handte
6ae1ec96bc
Merge pull request #1708 from neheb/dev 
zstd: Don't use utime on Linux
 2019-09-12 11:44:31 -04:00
Yann Collet
41416f0927
Merge pull request #1773 from bimbashrestha/rle_first_block_decompression_fix 
Removing redundant condition in decompression, making first block rle…
 2019-09-10 11:17:29 -07:00
Sen Huang
62616c4d90 fixes for windows compilation 2019-09-06 13:20:50 -07:00
Sen Huang
30bff50e06 fixes for tests and segfault 2019-09-06 11:10:53 -07:00
Bimba Shrestha
d687d603e4 Removing redundant condition in decompression, making first block rles valid to deocmpress 2019-09-06 10:46:19 -07:00
Sen Huang
a9c807a948 kill memory leaks, cleanup, fix some dumb bugs 2019-09-06 10:17:04 -07:00
Sen Huang
6beb3c0159 added decompression support 2019-09-05 17:56:24 -07:00
Sen Huang
7f98b46876 adding support for -O flag: multiple files into one directory for compressions (decompression to come) 2019-09-05 16:03:35 -07:00
Mike Swanson
25878b2de9 [programs] don’t do chmod when coming from stdin or multiple files 
Fixes #1729
 2019-09-02 18:02:32 -07:00
Nick Magerko
493f95c7df Fix merge conflicts 2019-08-22 11:51:41 -07:00
Nick Terrell
a505463710
Merge pull request #1726 from nmagerko/stream-size 
Add --stream-size=# option
 2019-08-22 11:31:15 -07:00
Nick Magerko
fd486a846a Differentiate --stream-size from --size-hint 2019-08-22 09:37:47 -07:00
Nick Magerko
2cdda8b3c4 Minor documentation update 2019-08-22 09:13:28 -07:00
Nick Magerko
05d7479a50 Document --size-hint 2019-08-20 14:08:26 -07:00
Nick Magerko
2d39b43906 Use int for srcSizeHint when sensible 2019-08-19 16:49:25 -07:00
Nick Magerko
dffbac5f89 Add --size-hint=# option 2019-08-19 11:38:49 -07:00
Nick Magerko
30bfa228e8 Keep content size flag set in stream size mode 2019-08-19 11:20:28 -07:00
Nick Magerko
a24dc3a935 Remove extraneous variables 2019-08-19 11:14:56 -07:00
Nick Magerko
f781cf672b Remove extraneous parameter 2019-08-19 11:07:43 -07:00
Nick Magerko
bbd83c2ab3 Update man page 2019-08-19 09:11:22 -07:00
Nick Magerko
c403b12f9d Set pledged size just before compression 2019-08-19 09:01:31 -07:00
Nick Magerko
af0c9501d1 Add --stream-size=# command 2019-08-16 11:34:39 -07:00
Ed Maste
b81d7cc6a0 remove extraneous doubled ;s 2019-08-15 21:17:06 -04:00
Yann Collet
01b2331ad1 bumped version number 
to v1.4.3
 2019-08-05 17:17:16 +02:00
Yann Collet
26351071bc
Merge pull request #1706 from LeeYoung624/dev 
add NULL pointer check in util.c
 2019-08-05 15:56:50 +02:00
Yann Collet
387e20d4f0 fixed minor conversion warning in datagen 2019-08-02 18:02:54 +02:00
Yann Collet
37f47e51a8 fixed datagen 
to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.

also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
 2019-08-02 17:34:53 +02:00
Rosen Penev
41e90653fe
zstd: Don't use utime on Linux 
utime is deprecated by POSIX 2008 and optionally not available with
uClibc-ng.

Got rid of a few useless headers in timefn.h.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
 2019-07-31 12:29:13 -07:00
LeeYoung624
c5caaf52a4 bug fix : NULL pointer 2019-07-29 17:05:50 +08:00
LeeYoung624
793b94b354 memory leak fix 2019-07-25 21:07:57 +08:00
Yann Collet
1f16245e9e updated man page 2019-07-24 16:04:37 -07:00
Yann Collet
0d38ee3c30
Merge pull request #1690 from piguin/dev 
fix compiling errors with clang-8
 2019-07-24 15:37:05 -07:00
Tyler Tran
5a61e66f7b previous commit did not undo all changes 2019-07-24 13:53:50 -07:00
Tyler Tran
12d60a9bd9 removing changes to zstd.1 2019-07-24 13:52:34 -07:00
Tyler Tran
f8c1d7979c modifying minor nit 2019-07-22 16:36:44 -07:00
Tyler Tran
02da4497f0 Adding documentation for shrink flag PR #1656 2019-07-22 16:33:22 -07:00
Qin Li
04a9d6b828 fix compiling errors with clang-8 
Compiling with clang-8 fails with the following errors:

largeNbDicts.c:562:37: error: implicit conversion turns floating-point
number into integer: 'const double' to 'U64' (aka 'unsigned long')
[-Werror,-Wfloat-conversion]
        U64 const dTime_ns = result.nanoSecPerRun;
                  ~~~~~~~~   ~~~~~~~^~~~~~~~~~~~~

zstdcli.c:300:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
 * @return 1 means that cover parameters were correct
   ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

zstdcli.c:301:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
 * @return 0 in case of malformed parameters
   ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 2019-07-18 19:41:00 -07:00
Yann Collet
8eda16c9f6
Merge pull request #1677 from LeeYoung624/gitignore_fix 
fix gitignore errors
 2019-07-09 14:36:38 -07:00
Yann Collet
b8ec4b0fd6 updated version number (to v1.4.1) 
also : added doc on context re-use, as suggested by @scherepanov at #1676
 2019-07-09 11:43:59 -07:00
LeeYoung624
654cb9d439 fix gitignore errors 2019-07-09 21:08:13 +08:00
Yann Collet
096714d1b8
Merge pull request #1671 from ephiepark/dev 
Adding targetCBlockSize param
 2019-07-03 17:47:44 -07:00
Ephraim Park
9007701670 Adding targetCBlockSize param 2019-07-03 15:41:52 -07:00
Yann Collet
857e608b51
Merge pull request #1658 from facebook/memset 
memset() rather than reduceIndex()
 2019-07-01 15:01:43 -07:00
Tyler-Tran
c55d2e7ba3 Adding shrinking flag for cover and fastcover (#1656) 
* Changed ERROR(GENERIC) excluding inits

* editing git ignore

* Edited init functions to size_t returns

* moved declarations earlier

* resolved issues with changes to init functions

* fixed style and an error check

* attempting to add tests that might trigger changes

* added && die to cases expecting to fail

* resolved no die on expected failed command

* fixed accel to be incorrect value

* Adding an automated shrinking option

* Fixing build

* finalizing fixes

* fix?

* Removing added comment in cover.h

* Styling fixes

* Merging with fb dev

* removing megic number for default regression

* Requested revisions

* fixing support for fast cover

* fixing casting errors

* parenthesis fix

* fixing some build nits

* resolving travis ci syntax

* might resolve all compilation issues

* removed unused variable

* remodeling the selectDict function

* fixing bad memory access

* fixing error checks

* fixed erroring check in selectDict

* fixing mixed declarations

* modify mixed declaration

* fixing nits and adding test cases

* Adding requested changes + fixed bug for error checking

* switched double comparison from != to <

* fixed declaration typing

* refactoring COVER_best_finish() and changing shrinkDict

* removing the const's

* modifying ZDICT_optimizeTrainFromBuffer_cover functions

* fixing potential bad memcpy

* fixing the error function for dict size
 2019-06-27 16:26:57 -07:00
Yann Collet
944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Yann Collet
1166782eb7
Merge pull request #1640 from felixhandte/lstat-macro-guard 
Protect lstat() With Better Macro Guard
 2019-06-14 17:30:03 -07:00
Mike Swanson
af80f6dfac [programs] Don’t try to chmod a dst file if it can’t be opened 
Repairs an oversight in my last commit, thanks @Cyan4973
 2019-06-09 01:52:45 -07:00
Mike Swanson
3968160a91 [programs] set chmod 600 after opening destination file 
This resolves a race condition where zstd or unzstd may expose read
permissions beyond the original file allowed.  Mode 600 is used
temporarily during the compression and decompression write stage
and the new file inherits the original file’s mode at the end.

Fixes #1630
 2019-06-08 21:54:02 -07:00
W. Felix Handte
d2c48042e6 Switch Macro Guarding lstat() 2019-06-07 15:32:28 -04:00
Yosuke Tanigawa
f46a3603c9
Update README.md 
Fix minor typo in the documentation.
 2019-06-05 09:11:21 -07:00
Ephraim Park
e498bb64b6 zstdcli : align output message with previous message 2019-06-04 09:42:18 -07:00
Ephraim Park
a38601f05a zstdcli : remove extra semicolon 2019-06-04 09:25:16 -07:00
Ephraim Park
ef2c836955 zstdcli : trying to fix declaration after statement 2019-06-04 09:23:49 -07:00
Ephraim Park
5fe974261b zstdcli : moving cpu load calculation from FIO_compressFilename_dstFile to FIO_compressFilename_internal 2019-06-04 09:04:35 -07:00
Ephraim Park
2841c79e87 zstdcli : fixing mixed declarations and code error 2019-06-03 17:36:09 -07:00
Ephraim Park
43d463eccb zstdcli : moving start time and cpu initialization after potential prompt 2019-06-03 17:31:14 -07:00
Ephraim Park
5029ebce2f zstdcli : exposing cpu load indicator for each file on -vv mode 2019-06-03 14:38:40 -07:00
W. Felix Handte
61025d5b7d zstdgrep: Handle -f Flag 2019-05-24 16:55:59 -04:00
Rosen Penev
23b5ee1236
fileio: Disable BACKTRACE under uClibc 
uClibc does not support BACKTRACE. It also defines __GLIBC__ and minor as both 2.
This fixes compilation under such a setup.
 2019-05-06 15:02:47 -07:00
cev
c4a40dbf65 set followLinks option true to cat, zcat and gzcat programs 2019-04-27 19:06:29 -04:00
myzhang1029
f837326f11
Fix #1591 - Not building on MinGW-W64 
Add a static function LONG_TELL for the forth #if branch
 2019-04-24 20:06:05 +08:00
Josh Soref
a880ca239b Spelling (#1582) 
* spelling: accidentally

* spelling: across

* spelling: additionally

* spelling: addresses

* spelling: appropriate

* spelling: assumed

* spelling: available

* spelling: builder

* spelling: capacity

* spelling: compiler

* spelling: compressibility

* spelling: compressor

* spelling: compression

* spelling: contract

* spelling: convenience

* spelling: decompress

* spelling: description

* spelling: deflate

* spelling: deterministically

* spelling: dictionary

* spelling: display

* spelling: eliminate

* spelling: preemptively

* spelling: exclude

* spelling: failure

* spelling: independence

* spelling: independent

* spelling: intentionally

* spelling: matching

* spelling: maximum

* spelling: meaning

* spelling: mishandled

* spelling: memory

* spelling: occasionally

* spelling: occurrence

* spelling: official

* spelling: offsets

* spelling: original

* spelling: output

* spelling: overflow

* spelling: overridden

* spelling: parameter

* spelling: performance

* spelling: probability

* spelling: receives

* spelling: redundant

* spelling: recompression

* spelling: resources

* spelling: sanity

* spelling: segment

* spelling: series

* spelling: specified

* spelling: specify

* spelling: subtracted

* spelling: successful

* spelling: return

* spelling: translation

* spelling: update

* spelling: unrelated

* spelling: useless

* spelling: variables

* spelling: variety

* spelling: verbatim

* spelling: verification

* spelling: visited

* spelling: warming

* spelling: workers

* spelling: with
 2019-04-12 11:18:11 -07:00
Yann Collet
1e01560b83 fixed timespec_get() initialization bug on some targets 
not sure why, but msan fires an "unitialized variable" error
when time gets properly initialized by timespec_get().
Maybe in some cases, not all bytes of the structure are initialized ?
Or maybe msan fails to detect the initialization ?

Anyway, pre-initializing the variable before passing it to timespec_get() works.
 2019-04-11 13:46:30 -07:00
Yann Collet
30c26ab726 fixed minor warning 
unused variable when assert() turned off in fileio.c
 2019-04-10 17:48:56 -07:00
Yann Collet
fbdd30d68e fixed cmake build script for test programs 2019-04-10 17:47:01 -07:00
Yann Collet
526ec646b7 alternate static assert 
to circumvent Visual's C4804 warning
 2019-04-10 16:05:02 -07:00
Yann Collet
9703a59121 fixed minor conversion warning 2019-04-10 15:54:55 -07:00
Yann Collet
885476fb5b FreeBSD_11 specific fix 
C11 mandates the definition of timespec_get() and TIME_UTC.
However, FreeBSD11 announce C11 compliance, but does not provifr timespec_get(),
breaking link stage for benchfn.
Since it does not provide TIME_UTC either, which is also required by C11,
test this macro: this will automatically rule out FreeBSD 11 for this code path
(it will use the backup C90 path instead, based on clock_t).

The issue seeems fixed in FreeBSD 12.
 2019-04-10 15:22:18 -07:00
Yann Collet
2c6b14ed22 fixed Windows header 
cmake build script: added timefn
 2019-04-10 14:54:13 -07:00
Yann Collet
3d346579d8 no more need for CLOCK_MONOTONIC 2019-04-10 14:16:39 -07:00
Yann Collet
36d2dfd846 moved C11 code path to timespec_get 2019-04-10 14:15:11 -07:00
Yann Collet
4765929271 fixed perror include 2019-04-10 14:04:11 -07:00
Yann Collet
70802cde6d fixed error message 
using stdlib's perror()
 2019-04-10 14:01:18 -07:00
Yann Collet
4b8185c7fc tried a blindfix for unix + c11 2019-04-10 13:26:27 -07:00
Yann Collet
59a7116cc2 benchfn dependencies reduced to only timefn 
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.

Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.

Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
 2019-04-10 12:37:03 -07:00
Yann Collet
094c000904 Merge branch 'dev' into benchfn 2019-04-10 11:57:05 -07:00
Yann Collet
90c0462d63 minor presentation refactoring 
and removed some // comment style
 2019-04-10 10:03:06 -07:00
Ori Peleg
bdeb4786b5 Fixed -r typo 2019-04-10 13:37:41 +03:00
shakeelrao
dca73db30c fix srcSize typo and add new UTIL func to comment 2019-03-28 17:50:34 -07:00
shakeelrao
5333e41ab3 Add NULL check for dict 2019-03-24 00:23:50 -07:00
shakeelrao
8ea219d8c6 Modify error msg 2019-03-23 21:59:30 -07:00
shakeelrao
1290933d19 Implement file check 2019-03-23 21:53:13 -07:00
shakeelrao
e5811e5520 Extract file comparison into utility func 2019-03-23 19:04:56 -07:00
Nick Terrell
19ca3fbc03 [zstdcli] Respect --[no-]compress-literals in benchmark mode 2019-02-15 16:27:39 -08:00
Nick Terrell
0c53c5ad4a [zstdcli] Add a flag to control literals compression 2019-02-15 15:00:45 -08:00
Nick Terrell
e1a799e5de [util] Fix UTIL_isLink() for FreeBSD 2019-02-11 17:44:10 -08:00
Björn Ketelaars
482b84f07b Make detection of symbolic links more consistent 
While fixing the detection of symbolic links on OpenBSD I noticed
inconsistent behaviour:

$ echo hello > hello
$ ln -s hello world
$ zstd hello world
Warning : world is a symbolic link, ignoring
hello                :316.67%   (     6 =>     19 bytes, hello.zst
$ ls *.zst
hello.zst
$ zstd world
world                :316.67%   (     6 =>     19 bytes, world.zst)
$ ls *.zst
hello.zst world.zst
 2019-02-12 01:56:05 +01:00
Björn Ketelaars
1e4dc2e5f1 Detect symbolic links on OpenBSD 
In #1520 it is described that FreeBSD doesn't detect symbolic links. The
same is true for OpenBSD. This diff fixes this issue for OpenBSD. I'm
guessing that something similar works for FreeBSD as well. However, I'm
unable to test this.
 2019-02-12 01:56:05 +01:00
Peter (Stig) Edwards
1951c1b3a0
-Wformat-security not needed with -Wformat=2 2019-02-01 09:31:46 +00:00
Yann Collet
fb756156f5 fixed static size of benchFnState 
added a static assert to ensure condition is respected on target platform
 2019-01-30 17:59:07 -08:00
Yann Collet
b8701102e0 fixed benchzstd to use new version of benchfn 
returning a double type
 2019-01-25 15:11:50 -08:00
Yann Collet
8c3ba44ada Merge branch 'fileio_lz4' into benchfn 2019-01-25 14:53:56 -08:00
Yann Collet
07e047873e fixed fileio.c compilation with LZ4 enabled 
was broken by #1505.

I'm surprised it passed CI tests.
LZ4 tests are part of the "Extended" tests on Travis CI,
which are run on "master" and in "cron" jobs.

Since latest cron job did not failed,
especially this one : https://travis-ci.org/facebook/zstd/jobs/484365040
it suggests cron jobs are no longer using `dev` branch.

To be investigated
 2019-01-25 14:42:44 -08:00
Yann Collet
f75ad2edcf added ability to create timedFnState on stack 2019-01-25 14:22:25 -08:00
Yann Collet
6b2f26791e updated benchfn.h code comments 2019-01-25 12:18:33 -08:00
Karl Ostmo
5e220bf4b5 Remove global parameters, pass into public functions instead 2019-01-23 16:06:27 -08:00
Karl Ostmo
4fa585aee3 fix --list on truncated files 
fseek() doesn't indicate when it moves past the end of a file.
Consequently, if a file is truncated within its last block, the error would't be detected.

This PR adds a test scenario that induces this situation using a small compressed file of only one block in size.
This test is added to tests/playTests.sh

Check is implemented by ensuring that the filehandle position is equal to the filesize upon exit.
 2019-01-15 19:03:29 -08:00
Lzu Tao
7b6a8840c5 Fix #1428 - zstdgrep returned 1 on match and unmatch 
- Use ZCAT for testing zstdgrep in case of non-install yet
- tests: Add file test for zstdgrep
 2019-01-06 23:22:55 +07:00
Conrad Meyer
fe82637069 Fix #1425 - Use physical core count API on FreeBSD 
Similar to Apple, use the native physical core count sysctl, when available.

This is a little repetitive (it's basically the __APPLE__ method plus the
otherBSD method concatenated together) but seemed clearer than any way that
would totally eliminate repetition.

The __FreeBSD_version check only tests the version of the FreeBSD kernel
that zstd is compiled on; importantly, it may be run on a different version.
So the compile-time check is a little naive and needs to be able to fallback
to work on older versions of FreeBSD.  For a similar reason, it may make
sense to simply eliminate the __FreeBSD_version check entirely.  The
tradeoff is that a spurious sysctlbyname would be issued when -T0 is used on
older kernels.
 2019-01-04 11:57:12 -08:00
Yann Collet
6b7a1d6127 fixed detection of input==output on Visual 
due to bad support of inode identifiers.
On Visual, option is limited to same file name,
which is imperfect, but way better than disabling the feature entirely.

It's enough to pass associated tests.
 2018-12-26 15:51:34 -08:00
Yann Collet
ae1d6bd48e fixed playTests.sh for minGW 
On Windows, the equivalent of `/dev/null` is `NUL`.
When tests are run under msys2/minGW,
the environment identifies itself as Windows,
hence the script uses `NUL` instead of `/dev/null`
but the environment will consider `NUL` to be a regular file name.
Consequently, `NUL` will be overwritten during tests,
triggering an error.

This patch uses flag `-f` to force such overwrite
passing the test.
 2018-12-26 15:19:09 -08:00
Yann Collet
1fdba696ca fixed VS2008 project 
bench.c => benchfn.c + benchzstd.c
wrong util.c from prior patch

and warnings :
C4127
 2018-12-25 16:14:14 -08:00
Yann Collet
ededcfca57 fix confusion between unsigned <-> U32 
as suggested in #1441.

generally U32 and unsigned are the same thing,
except when they are not ...

case : 32-bit compilation for MIPS (uint32_t == unsigned long)

A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).

Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
  but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
  but the caller user a pointer to U32 instead.

These fixes are useful.

However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.

As a consequence, the amount of code changed is larger than I would expect for such a patch.

Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.

So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.

I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.

Thoughts ?
 2018-12-21 18:09:41 -08:00
Yann Collet
8f35c7f94c
Merge pull request #1466 from facebook/noDictPresent 
fixed : better error message
 2018-12-20 19:01:27 -08:00
Yann Collet
18434d76b8 added strerror in comment 
as suggested by @felixhandte
 2018-12-20 17:27:08 -08:00
Yann Collet
41b45b84a1
Merge pull request #1465 from facebook/noFilePresent 
fixed : detection of non-existing file
 2018-12-20 17:21:04 -08:00
Yann Collet
ed2fb6bd57 fixed : better error message when dictionary missing 
during benchmark.
Also : refactored ZSTD_fillHashTable(),
just for readability (it does the same thing)
 2018-12-20 17:20:07 -08:00
Yann Collet
6e9512a70c
Merge pull request #1463 from yijinfb/getenv 
Add support for environment variable ZSTD_CLEVEL in CLI
 2018-12-20 15:17:00 -08:00
Yann Collet
0ed8ee4a37 fixed wrong assert condition 2018-12-20 14:46:23 -08:00
Yann Collet
ffba142406 fixed file identity detection in 32-bit mode 
also :
some library decided to use `index` as a global variable declared in standard header
shadowing the ones used in fastcover.c  :(
 2018-12-20 14:30:30 -08:00
Yann Collet
65a441a8f0 fixed stdlib implementation of time functions 
generated redefinitions
 2018-12-20 14:02:50 -08:00
Yann Collet
72dbf1bcd0 removed strncpy() from util.c 
as Visual surprisingly complains about their usage.
Replaced by memcpy()
 2018-12-20 12:27:12 -08:00
Yann Collet
105fa953cb use strerror() to generate error message 
as suggested by @terrelln .

also:
- hopefully fixed Windows version
- changed the test, so that it passes on non-english OS stdlib errors.
 2018-12-20 09:16:40 -08:00
Yi Jin
9b2d708850 change function name: _readU32FromChar() -> readU32FromCharChecked() 2018-12-19 23:41:18 -08:00
Yann Collet
173ef9dea2 fixed : detection of non-existing file 
better error message
with test
 2018-12-19 18:30:57 -08:00
Yi Jin
f7b1841b6f make init_cLevel(...) static 2018-12-19 17:56:45 -08:00
Yi Jin
30ffc24ad7 fix the code ahead of declaration issue 2018-12-19 17:49:04 -08:00
Yi Jin
0700335f57 change int init_cLevel() to int init_cLevel(void) 2018-12-19 17:38:28 -08:00
Yann Collet
0f2d443e10 added system error strings 
to more failure cases in fileio.c
as requested by fbpkg
 2018-12-19 17:25:58 -08:00
Yi Jin
cdc7bbf8b2 edit README.md 2018-12-19 17:09:54 -08:00
Yi Jin
29c7d82390 add a section on restricted support of environment variables to README.md 2018-12-19 17:06:56 -08:00
Yi Jin
26a9ae3f5f refactor readU32FromChar(...), improve init_cLevel(...), and add env var ZSTD_CLEVEL tests 2018-12-19 16:45:42 -08:00
Yi Jin
5324b1e386 add support for setting compression level through environment variable ZSTD_CLEVEL 2018-12-19 13:26:27 -08:00
Nick Terrell
7abd6acebf Also compare device thanks to @felixhandte 2018-12-18 15:40:27 -08:00
Nick Terrell
cd2c8defad [zstdcli] Refuse to overwrite input file 
Compare the input and output files by their inode number and
refuse to open the output file if the input file is the same.

This doesn't work when (de)compressing multiple files to a single
file, but that is a very uncommon use case, mostly used for
benchmarking by me.

Fixes #1422.
 2018-12-18 15:29:54 -08:00
Yann Collet
517d8c984c
Merge pull request #1449 from facebook/ovlog_def 
overlapLog default values
 2018-12-18 09:45:53 -08:00
Nick Terrell
bdfcaecc0a [zstdcli] Add --no-progress flag 
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
 2018-12-14 11:50:25 -08:00
Yann Collet
1993f5d412 fixed ovlog tests 
and updated man page
 2018-12-12 21:09:14 -08:00
Yann Collet
f2f86d369b Merge branch 'btultra2' into ovlog_def 2018-12-12 20:58:14 -08:00
Yann Collet
9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet
c313a85ee2 removed exception code for overlapLog level 22 
CLI used to set overlapLog at value 9 when level == 22.
This is no longer necessary (handled internally within library)
 2018-12-11 18:13:06 -08:00
Yann Collet
9b784dec7f changed parameter name to ZSTD_c_overlapLog 
from overlapSizeLog.

Reasoning :
`overlapLog` is already used everwhere, in the code, command line and documentation.
`ZSTD_c_overlapSizeLog` feels unnecessarily different.
 2018-12-11 16:55:33 -08:00
Yann Collet
9c3265a53f
Merge pull request #1417 from facebook/advancedAPI 
Advanced API
 2018-12-10 18:48:15 -08:00
Yann Collet
d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet
34aa401afd updated documentation 
introducing ZSTD_btultra2
 2018-12-06 17:22:19 -08:00
Yann Collet
be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet
3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet
34e146f548 advanced decompression function replaces by normal streaming one 
advanced parameters compatible with ZSTD_decompressStream().
 2018-12-04 10:28:36 -08:00
Yann Collet
7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet
d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Yann Collet
f6591e4a04
Merge pull request #1430 from lzutao/fix-zstdgrep 
Fix  #1428 - zstdgrep now returns 1 on unmatch
 2018-11-28 14:07:40 -08:00
Lzu Tao
3d18b4764d Prevent globbing on non-quoting variable 2018-11-29 03:04:40 +07:00
Lzu Tao
c046e0b626 Fix #1428 - zstdgrep now returns 1 on unmatch 2018-11-29 02:45:01 +07:00
Denis Ahrens
71b8ee1bf1 fixed a typo 2018-11-28 04:51:16 +01:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Ryan Schmidt
46d470c20c Fix feature detection with multiple -arch flags 
When multiple -arch flags are used, the compiler invokes itself once for
each architecture. Apparently, input on stdin is consumed by the
compilation of the first arch and is no longer available to the
compilation of the second arch, which results in a build failure and the
potentially incorrect determination that a feature is not available. So
write the feature detection source to a file instead of using stdin.
 2018-11-16 03:49:15 -06:00
Yann Collet
5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet
06c8d5a4f4 Merge branch 'dev' into advancedAPI 
fixed rsyncable
 2018-11-15 10:51:24 -08:00
Nick Terrell
4316409427 [cli] Set the job size 2018-11-14 17:00:29 -08:00
Nick Terrell
f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
Yann Collet
7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet
5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet
c584e84e68
Merge pull request #1415 from facebook/benchfn 
Separating benchfn and benchzstd
 2018-11-14 13:00:08 -08:00
Yann Collet
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet
2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK 
and updated several API comments
 2018-11-13 17:36:35 -08:00
Yann Collet
b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet
092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet
3ba0d6dd27 fixed decode-only test condition 2018-11-13 14:15:12 -08:00
Yann Collet
a29e3d1d01 fix debug trace 2018-11-13 13:27:22 -08:00
Yann Collet
b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet
9867cdb847 benchfn can provided faulty return value 
with BMK_extract_returnValue()
 2018-11-13 12:01:17 -08:00
Yann Collet
d38063f8ae separated bench module into benchfn and benchzstd 
it shall be possible to use benchfn
without any dependency on zstd.
 2018-11-13 11:01:59 -08:00
Yann Collet
9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Yann Collet
8bed4012bd fixed decompression-only benchmark 2018-11-08 12:36:39 -08:00
Bernhard M. Wiedemann
572279e73e pgo: also optimize compression 
and clean *.o so that linking does not fail from code
that is partially augmented with coverage generation instructions
 2018-10-29 17:20:03 +01:00
Bernhard M. Wiedemann
a38ad53ace zstd-pgo: do not make clean and zstd in parallel 
otherwise, there might not be a ./zstd to run
 2018-10-29 15:11:28 +01:00
Yann Collet
fc20b3c441 added flag -Wc++-compat 
for library and cli
 2018-10-26 16:38:23 -07:00
Yann Collet
fab150ce9f fix : removed zstdmt symlink on uninstall 2018-10-22 12:51:56 -07:00
Yann Collet
f181799082 fix decodecorpus incorrect frame generation 
fix #1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.

The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
 2018-10-20 18:56:21 -07:00
Yann Collet
f2cff22804
Merge pull request #1368 from rkjain89/test-branch 
Moving Code To util.c
 2018-10-19 16:31:10 -07:00
Sam Russell
622699706c fix up zgrep/zless references 2018-10-16 08:42:02 +13:00
Sam Russell
490f652974 Add man pages for zstdgrep and zstdless 2018-10-15 22:25:03 +13:00
Rohit Jain
c7251e5151 Address most comments 2018-10-11 18:05:15 -07:00
Rohit Jain
91b2fed8f8 Moving more function to the new C file 2018-10-11 17:34:47 -07:00
Rohit Jain
a47f6e6f2d Removing static and making extern 2018-10-11 16:51:29 -07:00
Yann Collet
5d419071be
Merge pull request #1369 from facebook/v137 
bumped version number to v1.3.7
 2018-10-11 15:59:57 -07:00
Rohit Jain
705e0b18ab Making changes to make it compile on my laptop 2018-10-11 15:51:57 -07:00
Rohit Jain
b19140bc13 Fixed makefile and removed multiple definitions from util.h (UTIL_getTime) 2018-10-11 15:17:40 -07:00