Commit Graph

8352 Commits

Author SHA1 Message Date
Yann Collet
dd24496951 programs/zstd also automatically generate object dir per conf
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
2020-10-21 23:38:33 -07:00
Yann Collet
01ecad2326 zstd in programs/ can also receive a DEBUGLEVEL value
simplify tests : only set DEBUGLEVEL, not the flags directly
2020-10-21 23:13:46 -07:00
Yann Collet
0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00
Yann Collet
d0436b2a45 automatically detect configuration changes
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
2020-10-21 19:22:45 -07:00
Yann Collet
8a453a34c5 automatic %.h header dependency tracking
also : BUILD_DIR can be manually specified
2020-10-21 17:25:07 -07:00
Yann Collet
2224ec33ed
Merge pull request #2367 from facebook/progressive_build
faster rebuild of zstd
2020-10-21 15:43:14 -07:00
Yann Collet
2b99bc29bf consolidated vpath 2020-10-21 04:01:01 -07:00
Yann Collet
e8eb2939fe store %.o object files into obj/
both static and dynamic libraries have their own object directory
2020-10-21 03:44:38 -07:00
Yann Collet
3e519be965 minor cleaning 2020-10-21 03:22:27 -07:00
Yann Collet
911dbdbb4b build libzstd.so from object files
%.o object files generated for dynamic library
must be different from those generated for static library.

Due to this difference, %.o were so far only generated for the static library.
The dynamic library was rebuilt from %.c source.

This meant that, for every minor change, the entire dynamic library had to be rebuilt.

This is fixed in this PR :
only the modified %.c source get rebuilt.
2020-10-20 22:19:57 -07:00
Yann Collet
684f8ea970 decouple zstd object files from lib/
That was a subtle one :
VPATH is affecting search for both %.c source and %.o object files.
This meant that, when an object file already exists in lib/,
it's used in programs/,
even though programs/ is supposed to generate its own %.o object files.

With the new vpath directive, this is no longer the case :
the search is only activated for %.c source files.
Now, local programs/%.o are always generated
even if equivalent ones are already created in lib/.

It more clearly guarantees that lib/ and programs/ can use different compilation directives
without mixing resulting %.o object files.
2020-10-20 18:16:11 -07:00
Yann Collet
a9f7e77d44 fixed travis tests 2020-10-20 17:08:14 -07:00
senhuang42
8bdb32aebe Add a function for LDM enable check 2020-10-20 13:46:02 -04:00
senhuang42
578e889ec1 Move ldm enable to compressStream2() 2020-10-20 13:04:45 -04:00
Nick Terrell
d6dae2000b
Merge pull request #2365 from senhuang42/move_opt_parser_test_to_long_tests
Move ldm + opt parser no regression test to long tests
2020-10-20 11:34:36 -04:00
senhuang42
d28d8a1d72 Include LDM tables size for CCtx size estimation where relevant 2020-10-20 09:21:30 -04:00
senhuang42
81a2c02d8f Move ldm no regression test to fuzzer longtests 2020-10-19 15:28:46 -04:00
senhuang42
b1c7fc5768 Add compatibility for multithreading 2020-10-19 12:07:06 -04:00
senhuang42
aad436da37 Document ldm enabled by default in zstd.h 2020-10-19 11:02:29 -04:00
senhuang42
df470e176b Add unit test for no cctx requested params change 2020-10-19 10:52:41 -04:00
senhuang42
590f7f55f0 Add ldm enable condition in ZSTD_resetCCtx_internal 2020-10-19 10:26:17 -04:00
senhuang42
4d01979b62 Expose and call ZSTD_ldm_skipRawSeqStoreBytes() 2020-10-16 20:30:00 -04:00
Yann Collet
de985b1c1a fix appveyor test
was mixing `*.o` object files from different compilation settings
2020-10-16 14:31:40 -07:00
Nick Terrell
9327a77c76
Merge pull request #2359 from senhuang42/fuzz_makefile_cleanup
[minor] Fix some /tests/fuzz Makefile annoyances
2020-10-16 13:53:34 -07:00
Yann Collet
a0ec50c2dc
Merge pull request #2355 from senhuang42/change_ldm_mt_config
Reduce --long mode MT jobsize at higher levels
2020-10-16 13:35:50 -07:00
Yann Collet
d380f803b9
Merge pull request #2357 from senhuang42/fix_ldm_nd_behavior
Fix nondeterministic LDM behavior in multithreading
2020-10-16 13:35:28 -07:00
Yann Collet
314c7df170 minor : change test order
to reduce a warning with `clang` on Windows
2020-10-16 13:26:47 -07:00
Yann Collet
eb1959d926 fix .deps list
and minor simplications
2020-10-16 11:08:52 -07:00
Yann Collet
c355d149de support verbose build output with V=1 2020-10-16 00:21:49 -07:00
Yann Collet
b6b87c2b77 track header dependencies during build
modifying a header file will now trigger recompilation of related *.c units
2020-10-16 00:18:16 -07:00
Yann Collet
80cee8d3fe faster rebuild of zstd
Building the zstd CLI costs time.
Some part of it is incompressible, leading to substantial iteration delay when testing code modifications.
That's mainly because all source files from the library must be rebuilt from source every time.

The main reason we don't build the CLI from library object files
is that we can't just build the object directly in the lib/ directory
(which they would by default)
since they use different compilation flags.
Specifically, the CLI enables multithreading, while the library doesn't (by default).

This is solved in this commit, by generating the object files locally.
Now, the CLI and the library can employ different sets of flags, without tripping over each other.
All library object files are generated directly into programs/ dir.
This works because no 2 source files have the same name.

Now, modifying a file doesn't require to recompile the entire lib, just the modified files.
The recipe is also compatible with `-j` parallel build, leading to large build time reductions on multi-core systems.
2020-10-16 00:01:41 -07:00
Yann Collet
ca885a198a
Merge pull request #2361 from senhuang42/fix_travis_minimal_decompressor_macros_test
[travis CI] Change make -j all check to make -j all && make check
2020-10-15 15:35:57 -07:00
senhuang42
adc53db6ed Change make -j all check to make -j all && make check 2020-10-15 15:11:51 -04:00
Yann Collet
056f4c9145
Merge pull request #2360 from facebook/zwrap_parallel_build
fix zwrapbench parallel build
2020-10-15 11:54:15 -07:00
Yann Collet
abf5c0655c fix zlibwrap parallel build
previous recipe would build object files directly within programs/
which could be in competition with other local builds happening in programs/ at the same time.

fixed by generating the relevant object file locally.
2020-10-15 10:56:37 -07:00
senhuang42
42d037bdba Add libregression build target, also fix make clean and .gitignore 2020-10-15 10:34:50 -04:00
senhuang42
f49926edf4 Change cycleLog adjustment to +3 from +4 2020-10-15 09:56:05 -04:00
senhuang42
ee84817fe7 Reset posInSequence when using ZSTD_referenceExternalSequences() 2020-10-14 22:06:08 -04:00
senhuang42
043b934ba2 Pass cRess_t by const const pointer 2020-10-14 20:19:46 -04:00
senhuang42
d0550bb18f Clarify argument names, fix DEBUGLOG() statements 2020-10-14 15:45:43 -04:00
senhuang42
3f99c9b38d Adjust match backwards count args 2020-10-14 15:23:03 -04:00
senhuang42
bf0d559449 Introduce, implement, and call ZSTD_ldm_countBackwardsMatch_2segments() 2020-10-14 12:58:06 -04:00
senhuang42
467e4383b0 Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config 2020-10-14 10:17:50 -04:00
Nick Terrell
8c46c1d851
Merge pull request #2356 from bsdimp/neon
aarch64: use __ARM_NEON instead of __aarch64__ to control use of neon
2020-10-13 15:42:46 -07:00
Yann Collet
1283935ac2
Merge pull request #2281 from likema/fix-aix-51
Fix building on AIX 5.1
2020-10-13 13:09:33 -07:00
Yann Collet
f5d5cd3b40
Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser
Integrate long distance matches into optimal parser
2020-10-13 13:09:07 -07:00
Warner Losh
43c0054405 aarch64: use __ARM_NEON instead of __aarch64__ to control use of neon
There are compilation environments in aarch64 where NEON isn't
available. While these environments could define ZSTD_NO_INTRINSICS,
it's more fail-safe to use the more specific symbol to know if NEON
extensions are available.

__ARM_NEON is the proper symbol, defined in ARM C Language Extensions
Release 2.1 (https://developer.arm.com/documentation/ihi0053/d/). Some
sources suggest __ARM_NEON__, but that's the obsolete spelling from
prior versions of the standard.

Signed-off-by: Warner Losh <imp@bsdimp.com>
2020-10-13 12:12:46 -06:00
senhuang42
a39614de1f Fix incorrect usage of cycleLog() in --patch-from 2020-10-13 13:00:27 -04:00
Nick Terrell
1e47a75abf
Merge pull request #2337 from terrelln/adjust-params
Fix ZSTD_adjustCParams_internal() to handle dictionary logic
2020-10-12 18:42:24 -07:00
Nick Terrell
7e6f91ed84 [minor] Improve docs and add an assert in response to review 2020-10-12 16:43:17 -07:00