Commit Graph

7556 Commits

Author SHA1 Message Date
W. Felix Handte
005ffd3bdb Fix Sed Regex Escaping for Extended Regex Rules...... 2020-05-05 14:54:04 -04:00
W. Felix Handte
bdfb50f7c4 Uhhh It's Still Failing, Let's Try Printing Debug Info? 2020-05-05 13:58:22 -04:00
W. Felix Handte
229cbd056e Fix Fallbacks to Work on FreeBSD 2020-05-05 13:18:45 -04:00
W. Felix Handte
6381b7ee3f Add Fallback When realpath Doesn't Support --relative-to 2020-05-04 18:59:47 -04:00
W. Felix Handte
c7cba8e87c Mark zstddeclib-in.c Not Executable 2020-05-04 18:59:26 -04:00
W. Felix Handte
b48f6c7d26 Try to Fix Single File Library Combiner Script to Handle Relative Includes 2020-05-04 15:20:26 -04:00
W. Felix Handte
7b75d772b1 Remove Useless Assignment in Makefile 2020-05-04 15:20:26 -04:00
W. Felix Handte
7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
W. Felix Handte
6028827fee Rewrite Include Paths to be Relative
Addresses #1998.
2020-05-04 15:20:26 -04:00
Bimba Shrestha
0301ef5d04
[bench] Extending largeNbDicts to compression (#2089)
* adding cdict_collection_t

* adding shuffleCDictionaries()

* adding compressInstructions

* adding compress()

* integrating compression into bench()

* copy paste error fix

* static analyzer uninit value complaint fix

* changing to control

* removing assert

* changing to control

* moving memcpy to seperate function

* fixing static analyzer complaint

* another hacky solution attempt

* Copying createbuffer logic
2020-05-04 10:42:22 -07:00
Felix Handte
7e9aabd652
Merge pull request #2099 from felixhandte/compile-under-pedantic
Compile Under `-pedantic -Werror` and `-std=c90`
2020-05-04 10:07:13 -07:00
Felix Handte
816ed80774
Merge pull request #1984 from MeghnaM/1636-Reduce-stack-usage-of-HUF_sort
Reduce stack usage of HUF_sort()
2020-05-04 08:15:31 -07:00
W. Felix Handte
2cf72d56a6 Try to Fix MSVC Error
It's complaining about the `memcpy`s, saying:

"warning C4090: 'function': different 'const' qualifiers"

Let's try explicitly casting to the argument types...
2020-05-04 10:59:15 -04:00
W. Felix Handte
6528fa088e Do a -pedantic Build in CircleCI 2020-05-04 10:59:15 -04:00
W. Felix Handte
dacbcd2cc1 Fix Up Some Pointer Handling in Tests 2020-05-04 10:59:15 -04:00
W. Felix Handte
3764859060 Switch Helper Declaration to Not Force Inline
It was causing build issues in ANSI mode.
2020-05-04 10:59:15 -04:00
W. Felix Handte
c7da66c9cf Purge C++-Style Comments (// ...), Make Compilation Succeed Under C90 2020-05-04 10:59:15 -04:00
W. Felix Handte
952427aebf Avoid inline Keyword in C90
Previously we would use it for all gcc-like compilations, even when a
restrictive mode that disallowed it had been selected.
2020-05-04 10:59:15 -04:00
W. Felix Handte
baa4e2e36c Don't Evaluate Arguments to Dummy Function 2020-05-04 10:59:15 -04:00
W. Felix Handte
450542d3a7 Allow Empty Format Strings in Error Macro Invocations
`-Wall` implies `-Wformat-zero-length`, which will cause compilation to fail
under `-Werror` when an empty string is passed as the format string to a
`printf`-family function. This commit moves us back to prefixing the provided
format string, which successfully avoids that warning.

However, this removes the failure mode where that `RAWLOG` invocation would
fail to compile when no format string was provided at all (which was desirable
to avoid having code that would successfully compile normally but fail under
`-pedantic`, which *does* require that a non-zero number of args are provided).

So this commit also introduces a function which does nothing at all, but will
fail to compile if not provided with at least one argument, which is a string.
This successfully links the compilability of pedantic and non-pedantic builds.
2020-05-04 10:59:15 -04:00
W. Felix Handte
6696933b32 Make All Invocations Start With Literal Format String 2020-05-04 10:59:15 -04:00
W. Felix Handte
2745f7a7d5 Make Error Macro Invocation Without Info String Fail to Compile
Even without `-pedantic`, these macros will now fail to compile unless you
provide an info string argument. This will prevent us from regressing.
2020-05-04 10:59:15 -04:00
W. Felix Handte
5e5f262612 Add (Possibly Empty) Info Strings to All Variadic Error Handling Macro Invocations 2020-05-04 10:58:55 -04:00
Nick Terrell
e103d7b4a6
Fix superblock mode (#2100)
Fixes:

Enable RLE blocks for superblock mode
Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table.
Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table.
Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes.
Respect disableLiteralsCompression in superblock mode
Fix superblock mode to handle a block consisting of only compressed literals
Fix a off by 1 error in superblock mode that disabled it whenever there were last literals
Fix superblock mode with long literals/matches (> 0xFFFF)
Allow superblock mode to repeat Huffman tables
Respect ZSTD_minGain().
Tests:

Simple check for the condition in #2096.
When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much.
Remaining limitations:

O(targetCBlockSize^2) because we recompute statistics every sequence
Unable to split literals of length > targetCBlockSize into multiple sequences
Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead.
...
Fixes #2096
2020-05-01 16:11:47 -07:00
Meghna Malhotra
0adfc8dfce Fix broken CI; make changes in response to the comments 2020-05-01 13:45:48 -07:00
Meghna Malhotra
53d76dc20f Remove magic constant and made other changes addressing the comments 2020-05-01 13:45:48 -07:00
Meghna Malhotra
cc7c29595d Fixed tests to use correct workspace size 2020-05-01 13:45:48 -07:00
Meghna Malhotra
fe8402b522 WIP: Still getting an error 2020-05-01 13:45:48 -07:00
Meghna Malhotra
a084d959bd WIP: Increased wksp size, but it's segfaulting 2020-05-01 13:45:48 -07:00
Meghna Malhotra
fdb2780c47 Move rank table into HUF_buildCTable_wksp() 2020-05-01 13:45:48 -07:00
Bimba Shrestha
0eaaa4d3a7
Merge pull request #2101 from bimbashrestha/meson
[build] fix meson build issue
2020-05-01 12:52:37 -05:00
Bimba Shrestha
834a8f838a meson msvc build fix 2020-05-01 09:04:09 -05:00
Yann Collet
da2748a855
Merge pull request #2097 from facebook/underlink
Fix underlinked libzstd
2020-04-30 10:16:24 -07:00
Yann Collet
6f62a9caaa fixed zstd-nolegacy target
when compiled as part of allVariants

(though I'm unsure why it was working before ...)
2020-04-29 11:56:21 -07:00
Yann Collet
7ea2ae6649 added test linking user program to multi-threaded libzstd 2020-04-28 21:18:29 -07:00
Yann Collet
f77fd5ced0 generalized pattern rules 2020-04-28 18:43:55 -07:00
Yann Collet
c6ae2e83bc fix libzstd-mt underlinking issue
fix #2045
When compiling `libzstd` in multithreading mode,
the `libzstd-mt` recipe would not include `-pthread`,
resulting in an underlinked dynamic library.

Added a test on Travis to check that the library is fully linked.

This makes it possible, in some future release,
to build a multi-threaded `libzstd` dynamic library by default
as it would no longer impact the build script of user programs.
2020-04-28 18:29:20 -07:00
Nick Terrell
0ed07f6dfe
Merge pull request #2094 from terrelln/stable-dst
[lib] Add ZSTD_d_stableOutBuffer + fix single-pass mode for empty frames
2020-04-28 17:53:24 -07:00
Yann Collet
f17ac423b2 new tests created new artifacts
they were not properly ignored
2020-04-28 15:58:22 -07:00
Nick Terrell
55a57d46be Add extra warnings about not modifying the ZSTD_outBuffer 2020-04-28 12:07:42 -07:00
Nick Terrell
108a5572a5
Merge pull request #2048 from nocnokneo/ctest-support
Add CTest support
2020-04-28 11:01:13 -07:00
Nick Terrell
1343b815f8 [fuzz] Fuzz test ZSTD_d_stableOutBuffer 2020-04-27 20:04:04 -07:00
Nick Terrell
77a2945c43 Add some comments 2020-04-27 20:04:04 -07:00
Nick Terrell
f33de06c3e [lib] Fix single-pass mode for empty frames 2020-04-27 20:04:01 -07:00
Nick Terrell
a4ff217baf [lib] Add ZSTD_d_stableOutBuffer 2020-04-27 18:09:44 -07:00
Nick Terrell
b104f8e3eb [zstd] Fix typo in ZSTD_dParameter 2020-04-27 12:12:28 -07:00
Bimba Shrestha
6b4a3e019f
Merge pull request #2088 from bimbashrestha/bug
[bug] Making compressStream2 fail when passing rawContent but claiming fullDict
2020-04-23 14:16:56 -05:00
Bimba Shrestha
861a73744d
Merge pull request #2086 from bimbashrestha/quick3
[bug] passing cParams.hashLog instead of comprParams->hashLog for ldm trigger
2020-04-23 13:02:57 -05:00
Bimba Shrestha
1875f616ce passing dictContentType instead of rawContent every time 2020-04-21 22:29:35 -07:00
Bimba Shrestha
f7a7409a49 adding fail test when passing wrong fullDict using refPrefix 2020-04-21 22:26:48 -07:00