Paul Cruz
4dc83ca64c
compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance
2017-07-24 15:14:58 -07:00
Paul Cruz
0ee3f8c2f8
adding more debug
2017-07-24 15:06:11 -07:00
Paul Cruz
8328f8192a
updating debug statements again
2017-07-24 14:40:23 -07:00
Paul Cruz
d3d759301f
changing position of endline for debug
2017-07-24 13:47:39 -07:00
Stella Lau
08a6e9a141
Minor code cleanup
2017-07-24 13:22:00 -07:00
Stella Lau
6eefa32911
Deduplicate code
2017-07-24 12:40:59 -07:00
Stella Lau
8ed9220102
Experiment with eviction policies and minor code cleanup
2017-07-24 12:05:43 -07:00
Paul Cruz
e508f632d6
updated comments and debug statements
2017-07-24 11:01:36 -07:00
Stella Lau
eb16da647d
Minor clean up
2017-07-24 10:20:53 -07:00
Stella Lau
1a188fe864
Fix overflow bug when calculating hash
2017-07-24 10:20:53 -07:00
Stella Lau
0b8fb1703b
Experiment with 64-bit hash insertion policy
2017-07-24 10:20:30 -07:00
Stella Lau
273c17b350
Experiment with 64-bit hash and checksum
2017-07-24 10:19:50 -07:00
Paul Cruz
483d936b87
reduced competition for completion mutex by separating mutex use based on which values is updated
2017-07-23 14:09:16 -07:00
Paul Cruz
880f08d104
change how completion is measured in compression thread
2017-07-23 10:18:54 -07:00
Paul Cruz
08d9e42ec6
removed useless measurements
2017-07-21 18:02:55 -07:00
Paul Cruz
95bef759b3
switched over to model where reading only waits on compression thread
2017-07-21 17:49:39 -07:00
Paul Cruz
6455ec482c
taking the maximum of the completion level reads in order to determine which one was waiting more
2017-07-21 16:05:01 -07:00
Paul Cruz
05fe8dd47c
updating debug statements
2017-07-21 14:06:24 -07:00
Paul Cruz
db109f8fef
measure multiple completion levels during each wait
2017-07-21 13:38:24 -07:00
Paul Cruz
721c6a8b97
added bounding to compression level change
2017-07-21 09:30:24 -07:00
Paul Cruz
e929d3b787
added priority decision making for adapt compression level
2017-07-21 09:26:35 -07:00
Paul Cruz
9259c7afa4
semi working version that stabilizes
2017-07-20 18:45:33 -07:00
Paul Cruz
82e488770c
fixed bug where writeSize could be zero
2017-07-20 16:38:02 -07:00
Paul Cruz
a19916425d
reworked adaptCompressionLevel to only account for completion information
2017-07-20 16:19:16 -07:00
Nick Terrell
7d3ac0710d
[linux] Update patches for v3
2017-07-20 13:33:55 -07:00
Paul Cruz
7ab758a640
changed how completion is actually sampled
2017-07-20 10:53:51 -07:00
Stella Lau
13a01ffb27
Fix off-by-one in size calculations
2017-07-19 17:24:09 -07:00
Stella Lau
2427a154cb
Minor refactoring
2017-07-19 16:56:28 -07:00
Paul Cruz
dcf609f835
make adaptCompressionLevel oscillate less
2017-07-19 16:36:33 -07:00
Paul Cruz
2a22c7915e
call ZSTD_compressBegin() once
2017-07-19 16:00:54 -07:00
Paul Cruz
6767abe652
fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0)
2017-07-19 14:54:15 -07:00
Stella Lau
030264ca51
Experiment with integrating ZSTD_count with findBestMatch
2017-07-19 14:14:26 -07:00
Paul Cruz
42382c1216
added some debug statements, adjusted end condition
2017-07-19 13:30:07 -07:00
Paul Cruz
5a85c57e30
set up new calculations compression completion progress
2017-07-19 11:47:17 -07:00
Paul Cruz
f1ac518b59
split compression into smaller blocks
2017-07-19 11:23:40 -07:00
Paul Cruz
338951cd48
moved compression adapt to avoid warning
2017-07-19 10:23:46 -07:00
Paul Cruz
4497ecf297
change compression level only right before actually performing compression. When waiting, only update waiting statistics.
2017-07-19 10:14:00 -07:00
Paul Cruz
e11bf55d0b
added mechanism for measuring how much of a job has been created
2017-07-19 10:10:47 -07:00
Paul Cruz
559ea4ff25
split up read process into smaller chunks
2017-07-19 09:59:17 -07:00
Paul Cruz
6119cd2164
added additional print for help menu
2017-07-19 09:43:17 -07:00
Stella Lau
4352e09cb0
Avoid recounting match lengths with ZSTD_count
2017-07-18 18:35:25 -07:00
Stella Lau
1fa223859f
Switch to using ZSTD_count instead of function pointer
2017-07-18 18:05:10 -07:00
Paul Cruz
3d7f1afadd
changed createCCtx() to split into initialization and creation
2017-07-18 17:32:36 -07:00
Paul Cruz
2c4e4ddc50
added mutex for stats struct
2017-07-18 15:55:58 -07:00
Paul Cruz
ad66faf16a
added progress check for filewriting, put important shared data behind mutex when being read from/written to
2017-07-18 15:23:11 -07:00
Stella Lau
19258f51c1
Make the meaning of LDM_MEMORY_USAGE consistent across tables
2017-07-18 14:25:39 -07:00
Paul Cruz
a34bc30237
setting up basic readme
2017-07-18 13:31:02 -07:00
Paul Cruz
29c36cf051
rename completion variable, split up fwrite operations in order to track progress
2017-07-18 13:30:29 -07:00
Paul Cruz
ae47eab2fd
changed test cases to use -s setting on the diffs
2017-07-18 12:58:50 -07:00
Stella Lau
fc41a87964
Experiment with using a lag when hashing
2017-07-17 18:13:09 -07:00
Paul Cruz
5af04c57b0
change parameters for compression level adapt
2017-07-17 17:59:50 -07:00
Paul Cruz
b3c9e02bb6
added signal to other threads whenever error occurs
2017-07-17 15:34:58 -07:00
Stella Lau
a00e406231
Remove version archive
2017-07-17 15:17:32 -07:00
Stella Lau
15a041adbf
Add function to get valid entries only from table
2017-07-17 15:16:58 -07:00
Paul Cruz
6be22f1f84
swap buffers instead of copying memory over
2017-07-17 14:39:10 -07:00
Paul Cruz
708238e07e
open file outside of adaptCCtx, pass to the output thread
2017-07-17 14:01:13 -07:00
Stella Lau
4bb42b02c1
Add basic chaining table
2017-07-17 11:53:54 -07:00
Paul Cruz
044e40db5a
removed freeCCtx() calls from createCCtx() so that it is not called twice during errors
2017-07-17 11:19:23 -07:00
Paul Cruz
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
Stella Lau
ca300ce6e0
Decouple hash table from compression function
2017-07-14 17:17:00 -07:00
Paul Cruz
1ab3f06f00
updated tests to use different seeds when executing different tests
2017-07-14 16:29:29 -07:00
Stella Lau
6e443b4960
Move hash table access for own functions
2017-07-14 14:27:55 -07:00
Stella Lau
2d8e6c6608
Add more statistics
2017-07-14 12:31:01 -07:00
Stella Lau
55f960e8db
Add percentages to offset histogram
2017-07-14 11:00:20 -07:00
Stella Lau
4db7f12ef3
Add offset histogram
2017-07-14 10:52:03 -07:00
Paul Cruz
0c8b9436b7
removed goto statements for the most part
2017-07-13 16:38:20 -07:00
Stella Lau
175a6c6029
[ldm] Minor refactoring
2017-07-13 16:16:31 -07:00
Stella Lau
361c06df75
Add min/max offset to stats
2017-07-13 15:29:41 -07:00
Paul Cruz
65a4ce2635
added tests for forced compression level
2017-07-13 14:57:24 -07:00
Paul Cruz
0d9665cef5
added additional tests for performance, allowed force compression level for testing purposes
2017-07-13 14:46:54 -07:00
Stella Lau
2b3c7e4199
[ldm] Make some functions shared
2017-07-13 14:39:35 -07:00
Paul Cruz
9165e97fc6
added some tests for correctness, time, and compression ratio
2017-07-13 13:50:23 -07:00
Stella Lau
9306feb8fa
[ldm] Switch to using lib/common/mem.h and move typedefs to ldm.h
...
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Blame Revision:
2017-07-13 13:44:48 -07:00
Stella Lau
50421d9474
[ldm] Remove old main files
2017-07-13 11:45:00 -07:00
Stella Lau
68c4560701
[ldm] Add TODO and comment for segfaulting in compress function
2017-07-13 10:38:19 -07:00
Paul Cruz
766663f1f1
added altering dictionary size depending on compression level
2017-07-13 10:15:27 -07:00
Stella Lau
92bed4a7e0
[ldm] Add CHAR_OFFSET in hash function and extend header size
2017-07-12 18:47:26 -07:00
Paul Cruz
7c886db0a8
changed to stderr
2017-07-12 17:28:53 -07:00
Paul Cruz
b5b18cf664
changed to malloc, added comment about adaptive compression level, and changed ternary operators
2017-07-12 17:10:58 -07:00
Paul Cruz
954d999abf
fixed up freeCCtx() removed BYTE since it wasn't being used
2017-07-12 16:50:43 -07:00
Paul Cruz
3c16edd26a
added copyright header, removed clean from makefile
2017-07-12 16:40:24 -07:00
Stella Lau
8de82b6eb0
[ldm] Clean up versions
2017-07-12 16:31:31 -07:00
Paul Cruz
74d3a6f5ae
passes tests with adaptive compression level
2017-07-12 16:18:41 -07:00
Paul Cruz
5353d350ae
working with fixed compression level and fixed dictionary size
2017-07-12 16:02:20 -07:00
Stella Lau
8ff8cdb15b
[ldm] Clean up code
2017-07-12 15:12:07 -07:00
Paul Cruz
356ddb649f
working with flush job->src.size and fixed cLevel
2017-07-12 12:21:21 -07:00
Stella Lau
3a48ffd4fd
Fix sumToHash to use hash space more efficiently
2017-07-12 10:53:19 -07:00
Stella Lau
e0d4162464
Minor fix for non-rolling hash
2017-07-12 09:50:24 -07:00
Stella Lau
50502519fb
Switch to using rolling hash only
2017-07-12 09:47:00 -07:00
Stella Lau
583dda17a8
Update rolling hash
2017-07-11 18:13:26 -07:00
Paul Cruz
0a401852c4
added debug statement
2017-07-11 16:50:50 -07:00
Paul Cruz
72a183efad
changed dictionary size, added debugging statements
2017-07-11 15:49:52 -07:00
Paul Cruz
7c54e09347
updated DEBUG statements
2017-07-11 15:15:41 -07:00
Paul Cruz
a3c077b8c6
added error message, updated copying dictionary into the input buffer
2017-07-11 15:00:52 -07:00
Paul Cruz
34afb9b23e
changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue()
2017-07-11 11:50:00 -07:00
Paul Cruz
7ec5928626
fixed an error where -c argument wasn't working for single files
2017-07-11 10:23:25 -07:00
Stella Lau
f6c5d07fe2
Save v3
2017-07-11 09:23:44 -07:00
Stella Lau
6c3673f4c3
Add rolling hash
2017-07-10 22:27:43 -07:00
Paul Cruz
f918545491
made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow
2017-07-10 18:16:42 -07:00
Paul Cruz
01fc7c4244
changed how the detection of the last job works
2017-07-10 16:27:58 -07:00
Paul Cruz
c36552ef8a
dst buffer should use ZSTD_compressBound to determine how much space it needs
2017-07-10 16:10:19 -07:00
Paul Cruz
7aa36df6df
fixed memory leak that was happening when creating jobs
2017-07-10 16:03:09 -07:00
Stella Lau
ef2b728316
Clean up and refactor compress function
2017-07-10 15:48:47 -07:00
Paul Cruz
e410d63d45
made input buffer an internal part of the compression context
2017-07-10 15:37:14 -07:00
Stella Lau
e4155b11d7
Add warning flags to makefile and clean up code to remove warnings
2017-07-10 13:08:19 -07:00
Stella Lau
10a71d9f1c
Add compression context
2017-07-10 12:38:27 -07:00
Paul Cruz
cc7f8e4d71
small changes
2017-07-10 11:10:11 -07:00
Paul Cruz
7e09b508ff
changed name
2017-07-10 11:05:37 -07:00
Paul Cruz
ed72ea5438
removed single from Makefile
2017-07-10 10:58:03 -07:00
Paul Cruz
ced3ec5714
removed scripts
2017-07-10 10:53:02 -07:00
Paul Cruz
82f0d64bee
removed single.c
2017-07-10 10:51:50 -07:00
Paul Cruz
62ebbabd32
updated error checking in each thread
2017-07-10 09:36:22 -07:00
Stella Lau
ae9cf235d6
Add LDM_DCtx
2017-07-10 07:38:09 -07:00
Stella Lau
5432214ee3
Minor refactoring
2017-07-10 06:50:49 -07:00
Stella Lau
b94b468e84
Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm
2017-07-10 06:32:46 -07:00
Stella Lau
474e06ac5b
Minor refactoring
2017-07-10 06:32:29 -07:00
Stella Lau
eb280cd568
Add folder for old versions
2017-07-10 06:32:05 -07:00
Stella Lau
719ccdc5a5
Update mainfile
2017-07-09 22:45:54 -07:00
Stella Lau
acdeb9f302
Add compression statistics
2017-07-07 17:09:28 -07:00
Paul Cruz
c3ae23d459
added ability to compress without specifying out filename
2017-07-07 17:07:05 -07:00
Paul Cruz
7163ffafde
playing around with adapt param
2017-07-07 15:56:00 -07:00
Paul Cruz
1c9d6b2c6b
rewrote time elapsed with UTIL
2017-07-07 15:42:20 -07:00
Paul Cruz
c0c236a28b
changed to using compressCCtx
2017-07-07 15:13:40 -07:00
Stella Lau
4076be09ec
[ldm] Update to hash every position
2017-07-07 14:52:40 -07:00
Stella Lau
7945f9ee47
Fix offset overflow bug
2017-07-07 14:14:01 -07:00
Paul Cruz
11fc0f4119
changed completed -> compressed
2017-07-07 13:55:38 -07:00
Paul Cruz
09d7c6a994
changed completed variables to compressed for clarity
2017-07-07 13:18:55 -07:00
Stella Lau
f791fc27e3
Add header with compress and decompress size
2017-07-07 12:44:29 -07:00
Paul Cruz
8c0eb62920
removed unnecessary comments, uncommented DEBUGLOG for later use
2017-07-07 11:47:16 -07:00
Paul Cruz
70a4153bd3
added ability to force output to stdout, wrote an additional test for this functionality
2017-07-07 11:32:14 -07:00
Paul Cruz
532f439961
cleaned up code for arguments a bit
2017-07-07 10:58:43 -07:00
Paul Cruz
f7e6b358d0
added tests that check to ensure stdout is working
2017-07-07 10:29:06 -07:00
Paul Cruz
4679132f59
updated avg compression rate, also hiding progress bar behind a flag now
2017-07-07 10:25:38 -07:00
Paul Cruz
00bc5df4e0
added compression rate to status bar
2017-07-07 09:35:39 -07:00
Paul Cruz
f351848b76
added data amount
2017-07-06 20:40:00 -07:00
Paul Cruz
2939301023
fixed problem with progress bar not persisting, added time elapsed
2017-07-06 20:30:20 -07:00
Paul Cruz
57ec0232a8
added help menu
2017-07-06 18:09:10 -07:00
Paul Cruz
b6cc084716
added really simple progress update in the corner
2017-07-06 17:48:18 -07:00
Stella Lau
3bbfa1249e
Update compressor and decompressor
2017-07-06 16:47:08 -07:00
Paul Cruz
ff9f2cd057
added some basic logic for altering compression level
2017-07-06 16:06:53 -07:00
Stella Lau
b96ad327a4
Add simple compress and decompress functions
2017-07-06 15:23:15 -07:00
Paul Cruz
a407ccc215
added ability to congregate statistics into single print statement rather than using debug
2017-07-06 13:09:17 -07:00
Igor Vuk
e6e25c9507
Fix typos in README.md
2017-07-06 20:43:14 +02:00
Paul Cruz
f57849b9c6
added ability to set initial compression level
2017-07-06 11:05:51 -07:00
Paul Cruz
592a0d9495
changed to work with std out
2017-07-06 10:49:26 -07:00
Paul Cruz
94fe291b83
small changes
2017-07-06 10:29:16 -07:00
Stella Lau
8aa34a7608
Switch to mmapping files
2017-07-06 07:30:49 -07:00
Paul Cruz
79d4657ce5
small changes
2017-07-05 17:44:36 -07:00
Paul Cruz
6f3ad1b22e
fixed the problem with pipeline tests by changing how jobs move through the threads
2017-07-05 17:24:21 -07:00
Paul Cruz
cc714f3bd3
added print statements and debuglog
2017-07-05 16:54:34 -07:00
Paul Cruz
3f52ca94bf
added more tests, changed makefile
2017-07-05 14:36:09 -07:00
Paul Cruz
faeb6e0b1b
added filenameTable for multiple files
2017-07-05 14:19:56 -07:00
Stella Lau
88f3d8641e
Initial long distance matcher commit
2017-07-05 13:57:07 -07:00
Paul Cruz
f0b9a153f3
added tests to run.sh
2017-07-05 13:23:34 -07:00
Paul Cruz
b42108386a
added some basic parsing for args
2017-07-05 12:20:16 -07:00
Paul Cruz
898c1a5b46
removed references to file size computation and file size function
2017-07-05 11:54:21 -07:00
Paul Cruz
a2680e5b96
removed calculation of file size and replaced with limited number of available jobs
2017-07-05 11:52:55 -07:00
Paul Cruz
dd8a591d5d
moved main logic for job creation into a separate function
2017-07-05 10:48:04 -07:00
Paul Cruz
9ccd55f3a8
free ctx fields when error occurs during creation
2017-07-05 10:20:56 -07:00
Paul Cruz
5df4cb0530
renamed files
2017-07-05 09:57:50 -07:00
Paul Cruz
c9f49198b8
fixed TODOs
2017-07-05 09:49:27 -07:00
Paul Cruz
9a147d8671
removed unnecessary checks for null pointer on free
2017-07-05 09:37:52 -07:00
Paul Cruz
a47ebb1607
removed print statements
2017-07-05 09:23:46 -07:00
Paul Cruz
0b70152a9b
working I believe
2017-07-03 20:05:42 -07:00
Paul Cruz
cd50382c03
fixed some issues with segfaults
2017-07-03 19:28:48 -07:00
Paul Cruz
95ea54b4cf
added code for waiitng for all jobs to finish
2017-07-03 19:24:22 -07:00
Paul Cruz
dd447bb9a7
fixed an error with number of jobs
2017-07-03 19:00:55 -07:00
Paul Cruz
ff9ac637d9
removed unnecessary files
2017-07-03 17:44:40 -07:00
Paul Cruz
dd96efa9ef
added print statements for debugging, fixed long memset by changing to calloc
2017-07-03 17:44:22 -07:00
Paul Cruz
0887e98d4b
finished main portion of code, now need to debug
2017-07-03 17:28:59 -07:00
Paul Cruz
00b5e6c512
continuing work on v2
2017-07-03 14:18:46 -07:00
Paul Cruz
ac79928960
version one complete, can compress a file given input and output names
2017-07-03 14:11:55 -07:00
Nick Terrell
ce91b64f00
[linux-kernel] Update patches for v2
...
* Reduce stack usage of many zstd functions, none use over 388 B anymore.
* Remove an incorrect `const` in `xxhash`.
2017-06-28 22:01:46 -07:00
Nick Terrell
f570609286
[pzstd] Fix deadlock in the case of errors
...
Fixes #720 .
2017-06-22 18:09:42 -07:00
Nick Terrell
aff86be75d
[linux] Write all the patch summaries
2017-06-21 20:23:44 -07:00
Nick Terrell
0a94fead41
[linux] Rename last 2 diffs to patch names
2017-06-21 15:27:07 -07:00
Nick Terrell
f3ad4062ba
[linux] Add summaries to xxhash and zstd patches
2017-06-21 15:27:07 -07:00
Nick Terrell
34b0a0883a
[linux] Rename diffs to patch names
2017-06-21 15:27:07 -07:00
Nick Terrell
b789860db0
[linux] Switch BtrFS compression level to 1
2017-06-21 15:27:02 -07:00
Nick Terrell
631d3664b0
[linux] Add xxhash and zstd in-kernel tests
2017-06-19 19:09:41 -07:00
Nick Terrell
e7093ca18c
[linux] Add libfuzzer targets
2017-06-19 19:09:15 -07:00
Nick Terrell
a48e129d48
[linux] Port libzstd UBSAN fix
2017-06-19 15:48:30 -07:00
Nick Terrell
9ad11bea4e
[linux] Fix decompression memory allocation
2017-06-19 12:32:04 -07:00
Paul Cruz
d75c4ddb1b
added a project support notice for VS2005
2017-06-02 10:28:56 -07:00
Paul Cruz
306701a447
adding zstd/zstd.vcproj because it was ignored during initial commit
2017-06-01 21:25:39 -07:00
Paul Cruz
6dc508539c
Moved VS2005 to the contrib directory
2017-06-01 21:18:48 -07:00
Nick Terrell
53a67ec1a6
[linux-kernel] Fix duplicate symbols when built-in to kernel
2017-05-25 18:34:13 -07:00
Nick Terrell
e63fff9b97
[linux-kernel] Fix clang-format edge case with goto labels
2017-05-25 13:56:46 -07:00
Nick Terrell
74b12f4014
[linux-kernel] Fix some up clang-format edge cases
2017-05-24 13:52:36 -07:00
Nick Terrell
ddc68e174b
[linux-kernel] Fix a weird clang-format corner case
2017-05-17 15:06:41 -07:00
Nick Terrell
a2b21c552f
[linux-kernel] Update tests Makefile
2017-05-17 14:56:37 -07:00
Nick Terrell
8d8e65c6c3
[linux-kernel] Update zstd.diff
2017-05-17 14:52:12 -07:00
Nick Terrell
06fa1b0560
[linux-kernel] Move ZSTD_STATIC_ASSERT to zstd_internal.h
2017-05-17 14:51:50 -07:00
Nick Terrell
67101c6c77
[linux-kernel] clang-format lib/zstd/
2017-05-17 14:40:46 -07:00
Nick Terrell
588579f3a1
[linux-kernel] Rename MEM_* functions to ZSTD_*
2017-05-17 14:39:57 -07:00
Nick Terrell
581e5fbe83
Add clang-format config
2017-05-17 13:49:19 -07:00
Nick Terrell
d1a5790f8e
[linux-kernel] Dual license with GPLv2
2017-05-17 12:08:15 -07:00
Nick Terrell
fb10c91b4f
[linux-kernel] Update zstd.diff
2017-05-16 17:21:22 -07:00
Nick Terrell
358280f107
Remove cross-platform macros and dead code
2017-05-16 17:16:00 -07:00
Nick Terrell
8eb7a32527
[linux-kernel] Fix whitespace edge case with script
2017-05-16 16:55:52 -07:00
Nick Terrell
0ce0782c97
[linux-kernel] Update kernelize.sh script for edge cases
2017-05-16 16:55:03 -07:00
Nick Terrell
0d26da2f25
[linux-kernel] Don't undef current (use curr)
2017-05-16 16:48:58 -07:00
Nick Terrell
f3c9bd014c
[linux-kernel] Remove unused variables in test
2017-05-16 16:13:51 -07:00
Nick Terrell
90a1c94965
[linux-kernel] Update header comments
2017-05-16 16:10:17 -07:00
Nick Terrell
1d42ca8f6d
Add zstd.diff
2017-05-16 16:02:05 -07:00
Nick Terrell
b3ebe9b77c
[linux-kernel] Separate xxhash into its own module
2017-05-16 15:52:35 -07:00
Yann Collet
d47709b6ea
Merge pull request #654 from iburinoc/splittable
...
[RFC] Splittable Format and API
2017-05-08 13:41:56 -07:00
Nick Terrell
cf4f9403fa
[kernel] Update README with SquashFS patch
...
Take patch from PR #682 by @iburinoc and update benchmarks.
2017-05-03 11:07:39 -07:00
Nick Terrell
99972fa9a8
[btrfs] Fix typo in pr_warn() message
2017-05-02 13:57:40 -07:00
Nick Terrell
79a85e1371
[btrfs] Update patch to use ZSTD_init{C,D}Stream()
2017-05-02 13:41:57 -07:00
Sean Purcell
470993c9b1
Add raw seek table construction API and parallel compression example
2017-04-28 12:17:09 -07:00
Yann Collet
ccc0b0eafa
Merge pull request #680 from iburinoc/sqfs-kernel
...
SquashFS linux kernel patch
2017-04-28 10:27:42 -07:00
Sean Purcell
c7e107197a
Fix missing squashfs_finish_page's
2017-04-27 17:19:20 -07:00
Sean Purcell
eeeb201e70
Fix case where pages run out before end of stream
2017-04-27 16:54:05 -07:00
Yann Collet
aca392610b
Merge pull request #679 from terrelln/dev
...
[kernel] Rename ZSTD_create*() to ZSTD_init*()
2017-04-27 16:31:03 -07:00
Sean Purcell
32c658a289
SquashFS linux kernel patch
2017-04-27 15:04:31 -07:00
Nick Terrell
0bd5d25d02
[pzstd] Add logging statements to tests
2017-04-27 09:55:19 -07:00
Nick Terrell
5087c1761f
Rename ZSTD_create*() to ZSTD_init*()
2017-04-26 13:08:20 -07:00
Sean Purcell
11dc940e72
Add parallel processing example for seekable API
2017-04-21 12:23:06 -07:00
Sean Purcell
35186e65b0
Address comments and make sure all prototypes are rendered by gen_html
2017-04-20 16:48:54 -07:00
Sean Purcell
0f7bd772e6
Update seekable API to simplify IO
2017-04-18 16:48:30 -07:00
Sean Purcell
9626cf1ac6
Address @terrelln's comments
2017-04-13 17:48:35 -07:00
Sean Purcell
2785b28e05
Reduce the limit on frame decompressed size to 2 GB
2017-04-12 14:09:13 -07:00
Sean Purcell
5ee1135f30
s/chunk/frame/
2017-04-12 11:15:50 -07:00
Sean Purcell
e80f1d74b3
Address PR comments and minor fixes
2017-04-12 11:15:46 -07:00
Sean Purcell
d048fefef7
Move seekable format content to /contrib
2017-04-11 14:38:56 -07:00
Nick Terrell
b633377d0e
Add BtrFS benchmarks
2017-04-11 12:40:53 -07:00
Nick Terrell
b4dd3378f1
Add BtrFS patch for 4.10 kernel
2017-04-11 11:50:24 -07:00
Nick Terrell
adb1974aec
Switch comments to kernel style + limit to 80 cols
2017-04-10 16:52:59 -07:00
Nick Terrell
bed6c0a4b0
Remove ZSTDLIB_API from decompress.c
2017-04-10 14:49:34 -07:00
Yann Collet
b5b79b3942
Merge pull request #647 from terrelln/linux
...
Add linux kernel modules to contrib/
2017-04-05 07:35:48 -07:00
Nick Terrell
9c7371b3ee
Improve comments and ordering
2017-04-04 19:02:28 -07:00
Nick Terrell
97693aff98
Add a README
2017-04-04 18:39:22 -07:00
Nick Terrell
7f510a7797
Add Kconfig and Makefile diffs
2017-04-04 18:37:55 -07:00
Nick Terrell
60b34b0d1c
fix gitignore
2017-04-04 18:15:21 -07:00
Nick Terrell
bb213798d9
Rename zstd_(de)compress.c to (de)compress.c
2017-04-04 18:09:02 -07:00
Nick Terrell
1b3856463e
Undef current if defined
2017-04-04 17:21:49 -07:00
Nick Terrell
c53bacc881
Fix up some old names
2017-04-04 17:03:48 -07:00
Nick Terrell
2e4dbd2105
Really move the makefile
2017-04-04 17:02:28 -07:00
Nick Terrell
5859101396
Move Makefile to lib/zstd
2017-04-04 17:00:12 -07:00
Nick Terrell
d0bbceac40
Add fs/btrfs/zstd.c
2017-04-04 16:56:52 -07:00
Nick Terrell
2eb623a6eb
Explicitly convert scratchBuffer to unsigned*
2017-04-04 16:38:13 -07:00
Nick Terrell
fd95be0ade
Move sources to lib/zstd/
2017-04-04 16:30:11 -07:00
Nick Terrell
1af700ea43
Remove unused FSE/HUF functions
2017-04-04 16:22:58 -07:00
Nick Terrell
dd62829ccf
Remove more restrict
2017-04-04 15:55:48 -07:00
Nick Terrell
9c257dc268
Fix up xxhash
2017-04-04 15:54:39 -07:00
Nick Terrell
0888251fb1
Switch intptr_t to ptrdiff_t
2017-04-04 15:49:38 -07:00
Nick Terrell
675839254d
Move zstd.h to linux/zstd.h
2017-04-04 15:44:27 -07:00
Nick Terrell
b06507221e
Test module macros
2017-04-04 12:56:44 -07:00
Nick Terrell
b1b582b9fa
Add module macros
2017-04-04 12:56:35 -07:00
Nick Terrell
b5e3e3c9a8
Add zstd kernel module Makefile
2017-04-04 12:10:47 -07:00
Nick Terrell
79298bf187
Add userland test with mock kernel headers
2017-04-04 11:53:22 -07:00
Nick Terrell
33fc0ad56a
Fix ZSTD_createCStream_usingCDict() and ZSTD_compress_usingCDict()
2017-04-04 11:52:49 -07:00
Nick Terrell
87cec8fd56
Fix compilation errors
2017-04-03 16:08:20 -07:00
Nick Terrell
59aadc85dc
Preallocate workSpace for cctx and combine create and init for cstream
2017-04-03 15:23:09 -07:00
Nick Terrell
21b6c53b9f
Working on API
2017-03-31 18:07:32 -07:00
Nick Terrell
e48b135521
Remove STATIC_LINKING_ONLY defines
2017-03-30 14:54:33 -07:00
Nick Terrell
6ce58897da
Use <asm/unaligned.h> for unaligned memory access
2017-03-30 14:53:57 -07:00
Nick Terrell
be7da5d98b
Change stdlib includes to linux includes
2017-03-30 14:53:52 -07:00
Nick Terrell
b3b41d0f6d
Remove platform specific macros
2017-03-30 14:53:23 -07:00
Nick Terrell
1075c12078
Convert all spaces to tabs
2017-03-30 14:49:45 -07:00
Nick Terrell
78063ca2bd
spaces to tabs
2017-03-30 14:47:42 -07:00
Nick Terrell
b1c6bb8702
Copy files into contrib/linux-kernel/
2017-03-30 14:47:35 -07:00
Yann Collet
6851db48e0
created contrib/cleanTabs
2017-03-30 12:42:44 -07:00
Sean Purcell
793e7bb6dd
Add symbolic link checking to pzstd
2017-03-23 12:41:51 -07:00
Yann Collet
7e35b352c6
Merge pull request #602 from iburinoc/doc
...
Add functions missing from manual, and fix parameter alignment
2017-03-14 14:08:41 -07:00
Sean Purcell
dec2b96536
Add functions missing from manual, and fix parameter alignment
2017-03-14 11:24:09 -07:00
Sean Purcell
120df494e9
Update builds to not support legacy v01-v03
2017-03-13 14:44:08 -07:00
Pádraig Brady
38a3428b37
support -Werror=format-security
...
Fedora now enables this option by default, resulting
in the following build failure:
Logging.h: In instantiation of
'void pzstd::Logger::operator()(int, const char*, Args ...)
Pzstd.cpp:413:48: required from here
Logging.h:46:17: error: format not a string literal and no format arguments
[-Werror=format-security]
std::fprintf(out_, fmt, args...);
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
2017-03-05 19:42:51 -08:00
Sean Purcell
822c7187d4
Merge branch 'dev' into doc
2017-02-23 15:23:24 -08:00
Yann Collet
7e224ea03e
Merge pull request #562 from iburinoc/testcorpus
...
Create a tool that generates random, valid, Zstd frames for decoder testing
2017-02-23 15:19:36 -08:00
Sean Purcell
1d1932480e
Move educational_decoder to doc/ and add doc README
...
- Also make some minor bugfixes to educational decoder
2017-02-23 14:34:52 -08:00
Yann Collet
db478ad27e
Merge pull request #558 from facebook/manual
...
Manual
2017-02-23 13:39:36 -08:00
Sean Purcell
3cd8d50c34
Update CLI and link from educational decoder
2017-02-23 13:06:50 -08:00
Yann Collet
508404514c
added manual target to contrib/gen_html/Makefile
2017-02-22 00:57:50 -08:00
Przemyslaw Skibinski
346ce32ade
legacy.c: fix memory leaks
2017-02-21 20:10:21 +01:00
Przemyslaw Skibinski
d8114e5802
zstd_compress.c: fix memory leaks
2017-02-21 18:59:56 +01:00
Przemyslaw Skibinski
684858e7b7
fix memory leaks
2017-02-21 18:17:24 +01:00
Dima Krasner
107c9a4e42
Moved to contrib
2017-02-18 23:30:57 +02:00
Sean Purcell
0f5c95af44
Disambiguate pledgedSrcSize == 0
...
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
Sean Purcell
eb52dbd4fe
Minor changes to educational decoder
2017-02-07 14:44:11 -08:00
Sean Purcell
f191be2fe6
Inlined portions of specification for clarity
2017-02-03 18:04:00 -08:00
Sean Purcell
18ce8b54dd
Switch IO to go through streams
2017-02-03 15:22:52 -08:00
Sean Purcell
823d8c233b
Minor security fixes
2017-02-01 10:41:04 -08:00
Sean Purcell
92ec2ea62f
More const's and readability improvements
2017-01-31 15:57:18 -08:00
Sean Purcell
f5d2f32d4d
Removed circular buffer, matches access destination buffer directly
2017-01-31 15:54:55 -08:00
Sean Purcell
f231626244
Minor fixes according to comments
...
- Add Facebook copyright notice
- Make max size macros more consistent
- Fix some unchecked malloc's
2017-01-30 15:00:19 -08:00
Sean Purcell
5657e0e07d
Added ZSTD_get_decompressed_size
...
Since this implementation handles multiple concatenated frames,
to determine decompressed size we must traverse the entire input,
checking each frame's frame_content_size field
2017-01-30 14:56:29 -08:00
Sean Purcell
9700f92583
Add educational decoder to /contrib
2017-01-30 11:44:43 -08:00
Nick Terrell
5cf84a05e7
Revert unnecessary change to Logging.h
2017-01-27 13:26:44 -08:00
Nick Terrell
58f499c41e
Clean imports and shorten tests
2017-01-27 10:43:18 -08:00
Nick Terrell
9c018cc140
Add BUCK files for Nuclide support
2017-01-27 10:43:12 -08:00
Chocobo1
37a2fb4ce1
Move -std=c++11 cxxflag to PZSTD_CXXFLAGS
...
Fixes the problem that the compiler doesn't enable c++11 mode by default
and the package build system has its own CXXFLAGS
2016-12-26 23:08:23 +08:00
Dimitry Andric
83cc2fb083
Enable using isatty() and nanosleep() on *BSD
2016-12-12 19:24:51 +01:00
Przemyslaw Skibinski
4da53219a0
zstd Manual updated to 1.1.2
2016-12-07 11:18:40 +01:00
Przemyslaw Skibinski
abd9ec0d53
gen_html: comments of type /*= and /**= can be longer than a single line
2016-12-07 11:13:20 +01:00
Yann Collet
379908be3d
fixed zstd.h for manual
2016-12-06 10:36:15 -08:00
Nick Terrell
bcd61586a8
[pzstd] Cast unused parameters to void
2016-11-15 17:46:28 -08:00
Nick Terrell
f147fccd0c
[pzstd] Fix frame size for small files + add logging
2016-11-15 16:39:09 -08:00
Nick Terrell
324c8ab000
[pzstd] Remove gtest dependency from make all
2016-11-14 11:56:28 -08:00
