AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,031 Commits 1 Branch 0 Tags 27 MiB
5b2f4ac1a8
Commit Graph

1370 Commits

Author SHA1 Message Date
Sen Huang
5b2f4ac1a8 merge 2019-10-21 19:02:52 -04:00
Nick Terrell
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset 
[zstdmt] Don't memset the jobDescription
 2019-10-21 15:53:57 -07:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix 
Windows fixes
 2019-10-21 13:45:17 -07:00
Sen Huang
676f89902a Added multiplier, renamed new enum to something more useful 2019-10-21 15:36:12 -04:00
Sen Huang
1f3a51fb52 Updated forceAttachDict param bounds 2019-10-21 15:36:12 -04:00
Sen Huang
8f69c47643 Add enum to decision process 2019-10-21 15:36:12 -04:00
Sen Huang
e4de8b098a Added support for forcing new CDict behavior and updated enum 2019-10-21 15:36:12 -04:00
Sen Huang
9294f4826b Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
f0fccc8847 Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
bb2df8c499 Trailing whitespace 2019-10-21 15:36:12 -04:00
Sen Huang
cf51501d2f Fix test 2019-10-21 15:36:12 -04:00
Sen Huang
ea3cb6988f Cast to BYTE to appease appveyor 2019-10-21 15:36:12 -04:00
Sen Huang
a727a85a7e merge conflicts round 2 2019-10-21 15:36:12 -04:00
Sen Huang
053a35fd64 formatting 2019-10-21 15:35:33 -04:00
Sen Huang
3fa4daaa55 Fix error 2019-10-21 15:35:33 -04:00
Sen Huang
3328348c63 Add compressionlevel to cdict 2019-10-21 15:32:39 -04:00
Felix Handte
cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan 
Add Poisoned Redzones to the Workspace When Compiling with ASAN
 2019-10-21 12:15:17 -04:00
Nick Terrell
0bc39bc3a0 [zstdmt] Don't memset the jobDescription 2019-10-18 15:05:51 -07:00
Yann Collet
1795133c45 refactored FIO_compressMultipleFilenames() prototype 
for consistency
 2019-10-17 15:32:03 -07:00
Yann Collet
6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api 
Adding api for extracting sequences from seqstore
 2019-10-10 13:11:18 -07:00
W. Felix Handte
ede31da2ea Fix CCtx Size Estimation 2019-10-10 15:02:08 -04:00
W. Felix Handte
bd6a20b8a0 Expand Default Redzone Size 2019-10-10 13:45:55 -04:00
W. Felix Handte
2c80a9f8ac Check if CCtx in Workspace after Null Check 2019-10-10 13:40:16 -04:00
W. Felix Handte
0ffae7e440 Stop Allocating Extra Space for Table Redzones 2019-10-10 13:40:16 -04:00
W. Felix Handte
a07037b784 Don't Try to Redzone the Tables 2019-10-10 13:40:16 -04:00
W. Felix Handte
0cc481ef66 Fix Workspace Size Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte
b6c0a02a17 Fix ZSTD_sizeof_matchState() Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte
8cffd6ed08 Avoid ASAN Failure in ZSTD_cwksp_free() 2019-10-10 13:40:16 -04:00
W. Felix Handte
ef0b5707c5 Refactor Freeing CCtxes / CDicts Inside Workspaces 2019-10-10 13:40:16 -04:00
W. Felix Handte
143b296cf6 Surround Workspace Allocs with Dead Zone 2019-10-10 13:40:16 -04:00
W. Felix Handte
19a0955ec9 Add ZSTD_cwksp_alloc_size() to Help Calculate Needed Workspace Size 2019-10-10 13:40:16 -04:00
W. Felix Handte
da88c35d41 Stop Assuming Tables are Adjacent 2019-10-10 13:40:16 -04:00
W. Felix Handte
35c30d6ca7 Poison Unused Workspace Memory 2019-10-10 13:40:16 -04:00
Bimba Shrestha
36528b96c4 Manually moving instead of memcpy on decoder and using genBuffer() 2019-10-03 09:26:51 -07:00
Bimba Shrestha
61ec4c2e7f Cleaning sequence parsing logic 2019-10-03 06:42:40 -07:00
Bimba Shrestha
c04245b257 Replacing assert with memory_allocation error code throw 2019-09-23 15:42:16 -07:00
Bimba Shrestha
be0bebd24e Adding test and null check for malloc 2019-09-23 15:08:18 -07:00
Nick Terrell
5cb7615f1f Add UNUSED_ATTR to ZSTD_storeSeq() 2019-09-20 21:37:13 -07:00
Nick Terrell
5dc0a1d659 HINT_INLINE ZSTD_storeSeq() 
Clang on Mac wasn't inlining `ZSTD_storeSeq()` in level 1, which was
causing a 5% performance regression. This fixes it.
 2019-09-20 16:39:27 -07:00
Bimba Shrestha
f3c4fd17e3 Passing in dummy dst buffer of compressbound(srcSize) 2019-09-20 15:50:58 -07:00
Nick Terrell
44c65da97e Remove literals overread in ZSTD_storeSeq() for ~neutral perf 2019-09-20 12:23:25 -07:00
Nick Terrell
fde217df04 Fix bounds check in ZSTD_storeSeq() 2019-09-20 08:25:12 -07:00
Nick Terrell
67b1f5fc72 Fix too strict assert 2019-09-20 01:23:35 -07:00
Nick Terrell
ddab2a94e8 Pass iend into ZSTD_storeSeq() to allow ZSTD_wildcopy() 2019-09-20 00:56:20 -07:00
Nick Terrell
efd37a64ea Optimize decompression and fix wildcopy overread 
* Bump `WILDCOPY_OVERLENGTH` to 16 to fix the wildcopy overread.
* Optimize `ZSTD_wildcopy()` by removing unnecessary branches and
  unrolling the loop.
* Extract `ZSTD_overlapCopy8()` into its own function.
* Add `ZSTD_safecopy()` for `ZSTD_execSequenceEnd()`. It is
  optimized for single long sequences, since that is the important
  case that can end up in `ZSTD_execSequenceEnd()`. Without this
  optimization, decompressing a block with 1 long match goes
  from 5.7 GB/s to 800 MB/s.
* Refactor `ZSTD_execSequenceEnd()`.
* Increase the literal copy shortcut to 16.
* Add a shortcut for offset >= 16.
* Simplify `ZSTD_execSequence()` by pushing more cases into
  `ZSTD_execSequenceEnd()`.
* Delete `ZSTD_execSequenceLong()` since it is exactly the
  same as `ZSTD_execSequence()`.

clang-8 seeds +17.5% on silesia and +21.8% on enwik8.
gcc-9 sees +12% on silesia and +15.5% on enwik8.

TODO: More detailed measurements, and on more datasets.

Crdit to OSS-Fuzz for finding the wildcopy overread.
 2019-09-19 21:07:14 -07:00
Bimba Shrestha
ae6d0e64ae Addressing comments 2019-09-19 15:25:20 -07:00
Yann Collet
bfff5b30a4
Merge pull request #1756 from mgrice/dev 
Improvements in zstd decode performance
 2019-09-18 11:35:50 -07:00
Yann Collet
243200e5bf minor refactor of ZSTD_fast 
- reduced variables lifetime
- more accurate code comments
 2019-09-17 14:02:57 -07:00
Bimba Shrestha
76fea3fb99 Resolving appveyor test failure implicit conversion 2019-09-16 14:02:23 -07:00