AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,760 Commits 1 Branch 0 Tags 27 MiB
05c0a072b7
Commit Graph

924 Commits

Author SHA1 Message Date
George Lu
3959ba15e6 Clarify README 2018-08-16 17:22:29 -07:00
George Lu
8175b28f03 Fix negative lvl display value 
Also fix synthetic benchmark parameter setting
 2018-08-16 16:46:37 -07:00
George Lu
239e114d62 prune comments 2018-08-15 16:04:34 -07:00
George Lu
8a296d3e1f Move Stuff around 
Group similar functions together, remove outdated comments
 2018-08-15 16:04:34 -07:00
George Lu
3f8b10baa1 consts 2018-08-15 16:04:34 -07:00
George Lu
46be2ef5d8 Remove unused stuff 2018-08-15 16:04:34 -07:00
George Lu
b234870c33 clarify display README 2018-08-15 14:29:49 -07:00
George Lu
ee77ddc28d Fix wraparound 2018-08-15 14:01:32 -07:00
George Lu
1e8d352930 silencing params 2018-08-15 14:01:32 -07:00
George Lu
2c5fdae0ae Clean up repetitive display 
Add documentation
 2018-08-15 14:01:32 -07:00
George Lu
4d9c6f51b8 -q -v options 2018-08-15 14:01:32 -07:00
George Lu
3dcfe5cc2c begin display changes 2018-08-15 14:01:32 -07:00
Yann Collet
3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
George Lu
b1d9ca737a Add memoTable options 
-hashing memotable
-no memotable
 2018-08-15 10:19:38 -07:00
George Lu
8c918edd3a MAke it easier to add params 
Make memoTable size limited
 2018-08-14 16:15:46 -07:00
George Lu
96725989ef Temp fix perf regression 2018-08-14 16:14:37 -07:00
George Lu
3f2d024dca forceAttachDict 2018-08-14 14:24:41 -07:00
George Lu
e3c679484a Add Time Checks 
Fix double -> U64 display
 2018-08-14 14:24:41 -07:00
George Lu
88dda92285 Reduce Duplication 
Change Defaults
Asserts actually disabled in paramgrill + fullbench
 2018-08-14 14:24:41 -07:00
George Lu
f581ccd267 Doc Updates 
Add option to pass in existing parameters in use
 2018-08-14 14:24:41 -07:00
George Lu
76acba025d scan-build 2018-08-14 12:13:05 -07:00
George Lu
614aaa3ae1 rebase clevel 2018-08-14 10:53:04 -07:00
George Lu
3b36fe5c68 strategy switching 2018-08-13 16:36:14 -07:00
George Lu
d4730a4f66 Update fulltable to use same interface 
Add seperateFiles flag
 2018-08-13 16:15:52 -07:00
George Lu
43b4971ca8 Renames, Documentation Updates 2018-08-13 16:15:52 -07:00
George Lu
a884b76bc2 Style Changes 
Add single run dictionaries
Change MB to be consistent 1 << 20 rather than 1,000,000
 2018-08-13 16:15:52 -07:00
George Lu
b3544217b7 Cleanup 2018-08-13 16:15:52 -07:00
George Lu
8ff0de15e4 Generalize, macro magic numbers 2018-08-13 16:15:52 -07:00
George Lu
3a2e95eba4 Perf improvements 
try decay
strategy selection skipping
 2018-08-13 16:15:52 -07:00
George Lu
2bdfe6ca71 Better Display 2018-08-13 16:15:52 -07:00
George Lu
f67d040c39 Bugfixes, style changes 
Complete euclidean distance climb
 2018-08-13 16:15:52 -07:00
George Lu
5f4502fc07 New climb 
feas part 2 uses euclidean metric
 2018-08-13 16:15:52 -07:00
George Lu
13611249a5 Table 
Compiling
+Euclidean Metric
 2018-08-13 16:15:52 -07:00
George Lu
0cea754024 Revert "Reorder declaration" 
This reverts commit 3ac2c22485.
 2018-08-13 16:15:34 -07:00
George Lu
486e586eed Revert "Default lvl 1" 
This reverts commit 0cc75d6ee0.
 2018-08-13 16:13:46 -07:00
George Lu
0cc75d6ee0 Default lvl 1 
MB to 2^20
 2018-08-13 14:55:56 -07:00
Yann Collet
09c9cf3f51 simplified rateLimiter 
resists better to changing in/out conditions
limits risks of "catching up"
 2018-08-13 12:13:47 -07:00
Yann Collet
e11f91b039 remove error message for Ctrl+C 2018-08-13 11:48:25 -07:00
Yann Collet
f3aa510738 rateLimiter does not "catch up" when input speed is slow 2018-08-13 11:38:55 -07:00
Yann Collet
a996b1fd2d fixed rate limited for high speed 2018-08-10 17:39:00 -07:00
Yann Collet
681a382eea added rateLimiter.py, by @felixhandte 
this rate limiter avoid the problem of `pv`
which "catch up" after a blocked period
instead of preserving a constant speed cap.
 2018-08-10 12:25:52 -07:00
George Lu
3ac2c22485 Reorder declaration 2018-08-09 16:38:32 -07:00
George Lu
bfe8392e23 Remove ctx from benchMem 2018-08-09 12:07:57 -07:00
George Lu
0ece2e5cdc Add consts 
+ fix gcc-8 warnings
 2018-08-09 11:38:09 -07:00
George Lu
6f480927af argument parsing cleanup 
+ clarifying comment
 2018-08-09 10:42:58 -07:00
George Lu
ad16a69408 Readability improvements, renaming 2018-08-09 10:42:58 -07:00
George Lu
8278a49cb6 const srcPtrs 2018-08-09 10:42:58 -07:00
George Lu
3d230db853 Change speed representation from floating point to integral 2018-08-09 10:42:58 -07:00
George Lu
8faeb41679 Update Documentation 
Change comment // to  /* */
Add more description of what functions do
Remove outdated comments
 2018-08-09 10:42:58 -07:00
George Lu
dd270b2f75 Renaming / Style fixes 2018-08-09 10:42:58 -07:00
George Lu
e148db366e Separate capacity vs size 
Also:
Make suggested fixes
-varInds_t
-reorder some arguments
-remove code duplication
-update README / -h
-Fix memory leaks
 2018-08-09 10:42:58 -07:00
George Lu
df026e159f Fix windows implicit casting bugs 2018-08-09 10:42:58 -07:00
George Lu
0f91b039ff Add Levels 2018-08-09 10:42:58 -07:00
George Lu
7b5b3d7ae3 BenchMem with block compressed sizes passed back up 2018-08-09 10:42:58 -07:00
George Lu
3adc217ea4 Total Changes: 
Add different constraint types (decompression speed, compression memory, parameter constraints)
Separate search space by strategy + strategy selection
Memoize results
Real random restarts
Support multiple files
Support Dictionary inputs
Debug Macro for extra printing
 2018-08-09 10:42:58 -07:00
George Lu
fab4438801 Dictionary + Multiple file Loading 2018-08-09 10:42:58 -07:00
George Lu
eb21b7f482 Not crashing 2018-08-09 10:42:58 -07:00
George Lu
5f49034520 Working V1 2018-08-09 10:42:58 -07:00
George Lu
cffb6da339 Parses additional parameters 
Additional constraint checking

Minor fixes

more param parsing

Add Memory

Change paramVariation

work on feasibility

reformat bench

Changed Paramgrill to use bench.c benchmarking

customlevel macro

Printing Flag

Minor changes

Explicit casting

Makefile fix

casting, type fix

Printing Flag

Minor Changes

comments, helper fn's
 2018-08-09 10:42:58 -07:00
Yann Collet
5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Nick Terrell
b9faaa1dc3 [FSE] Add division by zero test 2018-07-30 13:24:09 -07:00
cyan4973
3f535007e4 fix %zu support under minGW 
and relevant test on Appveyor
 2018-07-30 16:56:18 +02:00
cyan4973
aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
cyan4973
3d4b09a5aa support %zu under mingw 2018-07-30 16:29:20 +02:00
Yann Collet
effa84c8d1
Merge pull request #1230 from terrelln/train-out 
zstdcli: Allow -o before --train
 2018-07-18 16:34:10 +02:00
Yann Collet
6e490a2f09
Merge pull request #1237 from terrelln/init-cstream-adv 
Set requestedParams in ZSTD_initCStream*()
 2018-07-18 16:33:30 +02:00
cyan4973
7d1bc9cc8c fix minor conversion warning 2018-07-18 16:10:23 +02:00
cyan4973
9597b438e9 fix #1241 
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
 2018-07-17 18:52:57 +02:00
Nick Terrell
58b8219475 zstdcli: Allow -o before --train 
Only set the default value if `outFileName` is unset.

Fixes #1227.
 2018-07-16 12:45:34 -07:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev 
Split samples when building dictionary for COVER
 2018-07-13 13:26:15 -07:00
Nick Terrell
6d222c437c Set requestedParams in ZSTD_initCStream*() 
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
 2018-07-12 18:35:55 -07:00
Dmitry V. Levin
7ca12a1455 tests: use /dev/zero instead of /dev/random on all systems except GNU/Hurd 
https://github.com/facebook/zstd/pull/1124 broke the test suite
on Linux.  Looks like GNU/Hurd is the only operating system where
/dev/random is available for writing by unprivileged processes.

https://github.com/facebook/zstd/pull/1222 reverted the change
introduced by https://github.com/facebook/zstd/pull/1124 for OpenBSD
only, other operating systems need that change to be reverted, too.

Fixes: 2dde9d5aba ("Write to /dev/random for test")
Complements: 276988f794 ("OpenBSD is unable to write to /dev/zero")
 2018-07-11 12:41:50 +00:00
Jennifer Liu
a085d1aae1 Allow splitPoint==1.0 (using all samples for both training and testing) 2018-07-05 10:38:45 -07:00
Yann Collet
1bf0784975
Merge pull request #1206 from GeorgeLu97/FullBenchParam 
Fullbench Param
 2018-07-02 13:33:53 -07:00
Yann Collet
689bfecd48
Merge pull request #1188 from GeorgeLu97/BenchModule 
Bench module
 2018-07-02 13:33:27 -07:00
Jennifer Liu
1a14f8639c Update COVER dictionary builder tests 2018-07-02 11:37:04 -07:00
Björn Ketelaars
527dbf89ec xz/lzma warning causes test to fail 
OpenBSD's port building infrastructure is able to build in a privilege
separated mode. It uses a privilege drop model. Regression tests fail in
this mode as xz/lzma is unable to set file group and errors out with:

xz: tmp.xz: Cannot set the file group: Operation not permitted
gmake[1]: *** [Makefile:307: zstd-playTests] Error 2

Actually it is not a xz/lzma error but a warning causing zstd's
regression test to fail. Proposed fix is to have xz/lzma not set the
exit status to 2 even if a condition worth a warning was detected (-Q
flag).
 2018-06-30 13:54:34 +02:00
Björn Ketelaars
276988f794 OpenBSD is unable to write to /dev/zero 
https://github.com/facebook/zstd/pull/1124 fixes an issue with GNU/Hurd
being unable to write to /dev/zero. Implemented fix is writing to
/dev/random instead.

On OpenBSD a regular user is unable to write to /dev/random because of
permissions set on this device. Result is failing a regression test.

Proposed solution should work for all platforms.
 2018-06-30 13:54:34 +02:00
Yann Collet
328ec7e34c fix test-zstd can be run with parallel compilation 
fix #1221
 2018-06-29 16:52:21 -07:00
W. Felix Handte
8e7bdc18d6 Fix Tests of --list Behavior with stdin 2018-06-29 16:31:22 -04:00
Yann Collet
a2c3a4cd0e
Merge pull request #1214 from jennifermliu/dev 
Make --fast=0 fail
 2018-06-27 18:53:39 -07:00
Jennifer Liu
1ab57a7ce1 Redirect failed test result to INTOVOID and update comment about parsing fast command 2018-06-27 16:27:45 -07:00
W. Felix Handte
c10561c752 poolTests.c: Fix Interval Var Type 2018-06-27 19:15:38 -04:00
Jennifer Liu
aef8486fee Make fast=0 fail 2018-06-27 14:27:27 -07:00
George Lu
ceb4b9e670 New fullbench args 
-l# gives some level
--zstd= style parameters also supported
 2018-06-25 18:43:55 -07:00
Nick Terrell
b426bcc097
[zstdmt] Fix jobsize bugs (#1205) 
[zstdmt] Fix jobsize bugs

* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
  Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
 2018-06-25 15:21:08 -07:00
Yann Collet
3b53bfe4f3
Merge pull request #1200 from felixhandte/zstd-attach-dict-pref 
Add CCtx Param Controlling Dict Attachment Behavior
 2018-06-25 12:42:31 -07:00
George Lu
d6121ad0e1 Opaque State 
And minor fixups (comments/alignment/checks/fix memory leak)
 2018-06-25 08:07:43 -07:00
Yann Collet
31769ce702 error on no forward progress 
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.

In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.

This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.

The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
 2018-06-22 17:58:21 -07:00
Yann Collet
3934e010a2
Merge pull request #1197 from facebook/poolResize 
Thread Pool resize
 2018-06-22 14:20:07 -07:00
Yann Collet
fbd5dfc1b1 changed POOL_resize() return type to int 
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
 2018-06-22 12:14:59 -07:00
George Lu
a8eea99ebe Incremental Display + Fn Separations 
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
 2018-06-21 16:23:18 -07:00
Yann Collet
818e72b4d5 added extended POOL test 
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
 2018-06-21 14:58:59 -07:00
W. Felix Handte
01bb1c1016 Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00
Yann Collet
7d80ada5ca added a test for POOL (multithreading) 
ensuring all jobs in queue are nonetheless completed
when POOL is instructed to end abruptly (POOL_free())
 2018-06-21 12:24:36 -07:00
Yann Collet
6de249c1c6 fixed: bug when counting nb of active threads 
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
 2018-06-20 18:28:49 -07:00
Yann Collet
62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet
c0b6ce95b1
Merge pull request #1179 from supertopher/dev 
Improves UX for --list command's lack of support for pipes
 2018-06-19 14:36:30 -07:00
Topher Lubaway
5bac1db28f Tests to verify piped input to --list exits 1 
I'm following the pattern that i saw in the rest of the test file
please tell me if i am using the wrong conventions
 2018-06-19 09:56:37 -07:00
Yann Collet
c9e8ee93a7 removed specific --opaqueapi test 
from zstreamtest.

This test is now integrated within --newapi,
which dynamically switches between the 2 modes randomly.

The main outcome is reduced testing time.
 2018-06-18 19:20:37 -07:00