Commit Graph

440 Commits

Author SHA1 Message Date
Paul Cruz
be92a38d6a decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait 2017-07-26 10:05:10 -07:00
Stella Lau
e9161637b2 Allow parameters to be modified from a separate file 2017-07-25 18:13:27 -07:00
Paul Cruz
0b18d21e03 building on readme, added another help tip in the menu 2017-07-25 17:47:02 -07:00
Paul Cruz
7cc74e0b27 adding more to readme 2017-07-25 16:55:16 -07:00
Paul Cruz
8dbb07d822 updated progress bar with better representation of time, added const 2017-07-25 16:03:43 -07:00
Paul Cruz
9a132707af changing time units to seconds 2017-07-25 15:26:26 -07:00
Stella Lau
629c300118 Rename and remove unneeded files 2017-07-25 15:17:36 -07:00
Paul Cruz
31a9ed9883 updated const values, added more comments 2017-07-25 14:53:40 -07:00
Paul Cruz
5cfbf609a4 removed old debug statements no longer being used 2017-07-25 14:31:48 -07:00
Paul Cruz
0882cd1981 progress bar -- don't print num jobs, time elapsed shown in seconds 2017-07-25 14:26:55 -07:00
Paul Cruz
310c12d07e moved debug statements to a compiler flag 2017-07-25 14:08:39 -07:00
Paul Cruz
e02c79f833 started using decrease cooldown so that compression level would not decrease several times in a row 2017-07-25 11:16:27 -07:00
Paul Cruz
85d7c919f6 created independent function for controlling how completion relates to compression level change 2017-07-25 10:32:14 -07:00
Paul Cruz
6f1e260edd added mechanism for getting rid of spikes 2017-07-25 10:01:10 -07:00
Paul Cruz
700758d676 added help statement for -p, switched it to hide progress bar now that progress bar is default 2017-07-24 16:26:20 -07:00
Paul Cruz
df3754b6ed add quiet option, make progress bar default 2017-07-24 16:19:07 -07:00
Stella Lau
0295a27133 Experiment with not using a checksum 2017-07-24 15:26:44 -07:00
Paul Cruz
4dc83ca64c compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance 2017-07-24 15:14:58 -07:00
Paul Cruz
0ee3f8c2f8 adding more debug 2017-07-24 15:06:11 -07:00
Paul Cruz
8328f8192a updating debug statements again 2017-07-24 14:40:23 -07:00
Paul Cruz
d3d759301f changing position of endline for debug 2017-07-24 13:47:39 -07:00
Stella Lau
08a6e9a141 Minor code cleanup 2017-07-24 13:22:00 -07:00
Stella Lau
6eefa32911 Deduplicate code 2017-07-24 12:40:59 -07:00
Stella Lau
8ed9220102 Experiment with eviction policies and minor code cleanup 2017-07-24 12:05:43 -07:00
Paul Cruz
e508f632d6 updated comments and debug statements 2017-07-24 11:01:36 -07:00
Stella Lau
eb16da647d Minor clean up 2017-07-24 10:20:53 -07:00
Stella Lau
1a188fe864 Fix overflow bug when calculating hash 2017-07-24 10:20:53 -07:00
Stella Lau
0b8fb1703b Experiment with 64-bit hash insertion policy 2017-07-24 10:20:30 -07:00
Stella Lau
273c17b350 Experiment with 64-bit hash and checksum 2017-07-24 10:19:50 -07:00
Paul Cruz
483d936b87 reduced competition for completion mutex by separating mutex use based on which values is updated 2017-07-23 14:09:16 -07:00
Paul Cruz
880f08d104 change how completion is measured in compression thread 2017-07-23 10:18:54 -07:00
Paul Cruz
08d9e42ec6 removed useless measurements 2017-07-21 18:02:55 -07:00
Paul Cruz
95bef759b3 switched over to model where reading only waits on compression thread 2017-07-21 17:49:39 -07:00
Paul Cruz
6455ec482c taking the maximum of the completion level reads in order to determine which one was waiting more 2017-07-21 16:05:01 -07:00
Paul Cruz
05fe8dd47c updating debug statements 2017-07-21 14:06:24 -07:00
Paul Cruz
db109f8fef measure multiple completion levels during each wait 2017-07-21 13:38:24 -07:00
Paul Cruz
721c6a8b97 added bounding to compression level change 2017-07-21 09:30:24 -07:00
Paul Cruz
e929d3b787 added priority decision making for adapt compression level 2017-07-21 09:26:35 -07:00
Paul Cruz
9259c7afa4 semi working version that stabilizes 2017-07-20 18:45:33 -07:00
Paul Cruz
82e488770c fixed bug where writeSize could be zero 2017-07-20 16:38:02 -07:00
Paul Cruz
a19916425d reworked adaptCompressionLevel to only account for completion information 2017-07-20 16:19:16 -07:00
Nick Terrell
7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Paul Cruz
7ab758a640 changed how completion is actually sampled 2017-07-20 10:53:51 -07:00
Stella Lau
13a01ffb27 Fix off-by-one in size calculations 2017-07-19 17:24:09 -07:00
Stella Lau
2427a154cb Minor refactoring 2017-07-19 16:56:28 -07:00
Paul Cruz
dcf609f835 make adaptCompressionLevel oscillate less 2017-07-19 16:36:33 -07:00
Paul Cruz
2a22c7915e call ZSTD_compressBegin() once 2017-07-19 16:00:54 -07:00
Paul Cruz
6767abe652 fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0) 2017-07-19 14:54:15 -07:00
Stella Lau
030264ca51 Experiment with integrating ZSTD_count with findBestMatch 2017-07-19 14:14:26 -07:00
Paul Cruz
42382c1216 added some debug statements, adjusted end condition 2017-07-19 13:30:07 -07:00
Paul Cruz
5a85c57e30 set up new calculations compression completion progress 2017-07-19 11:47:17 -07:00
Paul Cruz
f1ac518b59 split compression into smaller blocks 2017-07-19 11:23:40 -07:00
Paul Cruz
338951cd48 moved compression adapt to avoid warning 2017-07-19 10:23:46 -07:00
Paul Cruz
4497ecf297 change compression level only right before actually performing compression. When waiting, only update waiting statistics. 2017-07-19 10:14:00 -07:00
Paul Cruz
e11bf55d0b added mechanism for measuring how much of a job has been created 2017-07-19 10:10:47 -07:00
Paul Cruz
559ea4ff25 split up read process into smaller chunks 2017-07-19 09:59:17 -07:00
Paul Cruz
6119cd2164 added additional print for help menu 2017-07-19 09:43:17 -07:00
Stella Lau
4352e09cb0 Avoid recounting match lengths with ZSTD_count 2017-07-18 18:35:25 -07:00
Stella Lau
1fa223859f Switch to using ZSTD_count instead of function pointer 2017-07-18 18:05:10 -07:00
Paul Cruz
3d7f1afadd changed createCCtx() to split into initialization and creation 2017-07-18 17:32:36 -07:00
Paul Cruz
2c4e4ddc50 added mutex for stats struct 2017-07-18 15:55:58 -07:00
Paul Cruz
ad66faf16a added progress check for filewriting, put important shared data behind mutex when being read from/written to 2017-07-18 15:23:11 -07:00
Stella Lau
19258f51c1 Make the meaning of LDM_MEMORY_USAGE consistent across tables 2017-07-18 14:25:39 -07:00
Paul Cruz
a34bc30237 setting up basic readme 2017-07-18 13:31:02 -07:00
Paul Cruz
29c36cf051 rename completion variable, split up fwrite operations in order to track progress 2017-07-18 13:30:29 -07:00
Paul Cruz
ae47eab2fd changed test cases to use -s setting on the diffs 2017-07-18 12:58:50 -07:00
Stella Lau
fc41a87964 Experiment with using a lag when hashing 2017-07-17 18:13:09 -07:00
Paul Cruz
5af04c57b0 change parameters for compression level adapt 2017-07-17 17:59:50 -07:00
Paul Cruz
b3c9e02bb6 added signal to other threads whenever error occurs 2017-07-17 15:34:58 -07:00
Stella Lau
a00e406231 Remove version archive 2017-07-17 15:17:32 -07:00
Stella Lau
15a041adbf Add function to get valid entries only from table 2017-07-17 15:16:58 -07:00
Paul Cruz
6be22f1f84 swap buffers instead of copying memory over 2017-07-17 14:39:10 -07:00
Paul Cruz
708238e07e open file outside of adaptCCtx, pass to the output thread 2017-07-17 14:01:13 -07:00
Stella Lau
4bb42b02c1 Add basic chaining table 2017-07-17 11:53:54 -07:00
Paul Cruz
044e40db5a removed freeCCtx() calls from createCCtx() so that it is not called twice during errors 2017-07-17 11:19:23 -07:00
Paul Cruz
50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Stella Lau
ca300ce6e0 Decouple hash table from compression function 2017-07-14 17:17:00 -07:00
Paul Cruz
1ab3f06f00 updated tests to use different seeds when executing different tests 2017-07-14 16:29:29 -07:00
Stella Lau
6e443b4960 Move hash table access for own functions 2017-07-14 14:27:55 -07:00
Stella Lau
2d8e6c6608 Add more statistics 2017-07-14 12:31:01 -07:00
Stella Lau
55f960e8db Add percentages to offset histogram 2017-07-14 11:00:20 -07:00
Stella Lau
4db7f12ef3 Add offset histogram 2017-07-14 10:52:03 -07:00
Paul Cruz
0c8b9436b7 removed goto statements for the most part 2017-07-13 16:38:20 -07:00
Stella Lau
175a6c6029 [ldm] Minor refactoring 2017-07-13 16:16:31 -07:00
Stella Lau
361c06df75 Add min/max offset to stats 2017-07-13 15:29:41 -07:00
Paul Cruz
65a4ce2635 added tests for forced compression level 2017-07-13 14:57:24 -07:00
Paul Cruz
0d9665cef5 added additional tests for performance, allowed force compression level for testing purposes 2017-07-13 14:46:54 -07:00
Stella Lau
2b3c7e4199 [ldm] Make some functions shared 2017-07-13 14:39:35 -07:00
Paul Cruz
9165e97fc6 added some tests for correctness, time, and compression ratio 2017-07-13 13:50:23 -07:00
Stella Lau
9306feb8fa [ldm] Switch to using lib/common/mem.h and move typedefs to ldm.h
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Blame Revision:
2017-07-13 13:44:48 -07:00
Stella Lau
50421d9474 [ldm] Remove old main files 2017-07-13 11:45:00 -07:00
Stella Lau
68c4560701 [ldm] Add TODO and comment for segfaulting in compress function 2017-07-13 10:38:19 -07:00
Paul Cruz
766663f1f1 added altering dictionary size depending on compression level 2017-07-13 10:15:27 -07:00
Stella Lau
92bed4a7e0 [ldm] Add CHAR_OFFSET in hash function and extend header size 2017-07-12 18:47:26 -07:00
Paul Cruz
7c886db0a8 changed to stderr 2017-07-12 17:28:53 -07:00
Paul Cruz
b5b18cf664 changed to malloc, added comment about adaptive compression level, and changed ternary operators 2017-07-12 17:10:58 -07:00
Paul Cruz
954d999abf fixed up freeCCtx() removed BYTE since it wasn't being used 2017-07-12 16:50:43 -07:00
Paul Cruz
3c16edd26a added copyright header, removed clean from makefile 2017-07-12 16:40:24 -07:00
Stella Lau
8de82b6eb0 [ldm] Clean up versions 2017-07-12 16:31:31 -07:00
Paul Cruz
74d3a6f5ae passes tests with adaptive compression level 2017-07-12 16:18:41 -07:00