compat_symbol_reference no longer needs tests-internal. Do not build
the test at all for newer targets, so that no spurious UNSUPPORTED
result is generated. Use compat_symbol_reference for
__malloc_initialize_hook as well, eliminating the need for -rdynamic.
Reviewed-by: DJ Delorie <dj@redhat.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
In the next release of POSIX, free must preserve errno
<https://www.austingroupbugs.net/view.php?id=385>.
Modify __libc_free to save and restore errno, so that
any internal munmap etc. syscalls do not disturb the caller's errno.
Add a test malloc/tst-free-errno.c (almost all by Bruno Haible),
and document that free preserves errno.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This new variable allows various subsystems in glibc to run all or
some of their tests with MALLOC_CHECK_=3. This patch adds
infrastructure support for this variable as well as an implementation
in malloc/Makefile to allow running some of the tests with
MALLOC_CHECK_=3.
At present some tests in malloc/ have been excluded from the mcheck
tests either because they're specifically testing MALLOC_CHECK_ or
they are failing in master even without the Memory Tagging patches
that prompted this work. Some tests were reviewed and found to need
specific error points that MALLOC_CHECK_ defeats by terminating early
but a thorough review of all tests is needed to bring them into mcheck
coverage.
The following failures are seen in current master:
FAIL: malloc/tst-malloc-fork-deadlock-mcheck
FAIL: malloc/tst-malloc-stats-cancellation-mcheck
FAIL: malloc/tst-malloc-thread-fail-mcheck
FAIL: malloc/tst-realloc-mcheck
FAIL: malloc/tst-reallocarray-mcheck
All of these are due to the Memory Tagging patchset and will be fixed
separately.
This patch adds the ABI-related bits to reflect the new mallinfo2
function, and adds a test case to verify basic functionality.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Adding the test "tst-safe-linking" for testing that Safe-Linking works
as expected. The test checks these 3 main flows:
* tcache protection
* fastbin protection
* malloc_consolidate() correctness
As there is a random chance of 1/16 that of the alignment will remain
correct, the test checks each flow up to 10 times, using different random
values for the pointer corruption. As a result, the chance for a false
failure of a given tested flow is 2**(-40), thus highly unlikely.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
pvalloc is guarantueed to round up the allocation size to the page
size, so applications can assume that the memory region is larger
than the passed-in argument. The alloc_size attribute cannot express
that.
The test case is based on a suggestion from Jakub Jelinek.
This fixes commit 9bf8e29ca1 ("malloc:
make malloc fail with requests larger than PTRDIFF_MAX (BZ#23741)").
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This synchronization method has a lower overhead and makes
it more likely that the signal arrives during one of the critical
functions.
Also test for fork deadlocks explicitly.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
The memusagestat is the only binary that has its own link line which
causes it to be linked against the existing installed C library. It
has been this way since it was originally committed in 1999, but I
don't see any reason as to why. Since we want all the programs we
build locally to be against the new copy of glibc, change the build
to be like all other programs.
This one tests for BZ#23907 where the double free
test didn't check the tcache bin bounds before dereferencing
the bin.
[BZ #23907]
* malloc/tst-tcfree3.c: New.
* malloc/Makefile: Add it.
malloc_stats means to disable cancellation for writes to stderr while
it runs, but it restores stderr->_flags2 with |= instead of =, so what
it actually does is disable cancellation on stderr permanently.
[BZ #22830]
* malloc/malloc.c (__malloc_stats): Restore stderr->_flags2
correctly.
* malloc/tst-malloc-stats-cancellation.c: New test case.
* malloc/Makefile: Add new test case.
When posix_memalign is called with an alignment less than MALLOC_ALIGNMENT
and a requested size close to SIZE_MAX, it falls back to malloc code
(because the alignment of a block returned by malloc is sufficient to
satisfy the call). In this case, an integer overflow in _int_malloc leads
to posix_memalign incorrectly returning successfully.
Upon fixing this and writing a somewhat thorough regression test, it was
discovered that when posix_memalign is called with an alignment larger than
MALLOC_ALIGNMENT (so it uses _int_memalign instead) and a requested size
close to SIZE_MAX, a different integer overflow in _int_memalign leads to
posix_memalign incorrectly returning successfully.
Both integer overflows affect other memory allocation functions that use
_int_malloc (one affected malloc in x86) or _int_memalign as well.
This commit fixes both integer overflows. In addition to this, it adds a
regression test to guard against false successful allocations by the
following memory allocation functions when called with too-large allocation
sizes and, where relevant, various valid alignments:
malloc, realloc, calloc, reallocarray, memalign, posix_memalign,
aligned_alloc, valloc, and pvalloc.
The malloc tcache added in 2.26 will leak all of the elements remaining
in the cache and the cache structure itself when a thread exits. The
defect is that we do not set tcache_shutting_down early enough, and the
thread simply recreates the tcache and places the elements back onto a
new tcache which is subsequently lost as the thread exits (unfreed
memory). The fix is relatively simple, move the setting of
tcache_shutting_down earlier in tcache_thread_freeres. We add a test
case which uses mallinfo and some heuristics to look for unaccounted for
memory usage between the start and end of a thread start/join loop. It
is very reliable at detecting that there is a leak given the number of
iterations. Without the fix the test will consume 122MiB of leaked
memory.
This commit adds fixed-size allocation buffers. The primary use
case is in NSS modules, where dynamically sized data is stored
in a fixed-size buffer provided by the caller.
Other uses include a replacement of mempcpy cascades (which is
safer due to the size checking inherent to allocation buffers).
This is intended as a type-safe alternative to obstacks and
hand-written realloc constructs. The implementation avoids
writing function pointers to the heap.
This patch adds a new build module called 'testsuite'.
IS_IN (testsuite) implies _ISOMAC, as do IS_IN_build and __cplusplus
(which means several ad-hoc tests for __cplusplus can go away).
libc-symbols.h now suppresses almost all of *itself* when _ISOMAC is
defined; in particular, _ISOMAC mode does not get config.h
automatically anymore.
There are still quite a few tests that need to see internal gunk of
one variety or another. For them, we now have 'tests-internal' and
'test-internal-extras'; files in this category will still be compiled
with MODULE_NAME=nonlib, and everything proceeds as it always has.
The bulk of this patch is moving tests from 'tests' to
'tests-internal'. There is also 'tests-static-internal', which has
the same effect on files in 'tests-static', and 'modules-names-tests',
which has the *inverse* effect on files in 'modules-names' (it's
inverted because most of the things in modules-names are *not* tests).
For both of these, the file must appear in *both* the new variable and
the old one.
There is also now a special case for when libc-symbols.h is included
without MODULE_NAME being defined at all. (This happens during the
creation of libc-modules.h, and also when preprocessing Versions
files.) When this happens, IS_IN is set to be always false and
_ISOMAC is *not* defined, which was the status quo, but now it's
explicit.
The remaining changes to C source files in this patch seemed likely to
cause problems in the absence of the main change. They should be
relatively self-explanatory. In a few cases I duplicated a definition
from an internal header rather than move the test to tests-internal;
this was a judgement call each time and I'm happy to change those
however reviewers feel is more appropriate.
* Makerules: New subdir configuration variables 'tests-internal'
and 'test-internal-extras'. Test files in these categories will
still be compiled with MODULE_NAME=nonlib. Test files in the
existing categories (tests, xtests, test-srcs, test-extras) are
now compiled with MODULE_NAME=testsuite.
New subdir configuration variable 'modules-names-tests'. Files
which are in both 'modules-names' and 'modules-names-tests' will
be compiled with MODULE_NAME=testsuite instead of
MODULE_NAME=extramodules.
(gen-as-const-headers): Move to tests-internal.
(do-tests-clean, common-mostlyclean): Support tests-internal.
* Makeconfig (built-modules): Add testsuite.
* Makefile: Change libof-check-installed-headers-c and
libof-check-installed-headers-cxx to 'testsuite'.
* Rules: Likewise. Support tests-internal.
* benchtests/strcoll-inputs/filelist#en_US.UTF-8:
Remove extra-modules.mk.
* config.h.in: Don't check for __OPTIMIZE__ or __FAST_MATH__ here.
* include/libc-symbols.h: Move definitions of _GNU_SOURCE,
PASTE_NAME, PASTE_NAME1, IN_MODULE, IS_IN, and IS_IN_LIB to the
very top of the file and rationalize their order.
If MODULE_NAME is not defined at all, define IS_IN to always be
false, and don't define _ISOMAC.
If any of IS_IN (testsuite), IS_IN_build, or __cplusplus are
true, define _ISOMAC and suppress everything else in this file,
starting with the inclusion of config.h.
Do check for inappropriate definitions of __OPTIMIZE__ and
__FAST_MATH__ here, but only if _ISOMAC is not defined.
Correct some out-of-date commentary.
* include/math.h: If _ISOMAC is defined, undefine NO_LONG_DOUBLE
and _Mlong_double_ before including math.h.
* include/string.h: If _ISOMAC is defined, don't expose
_STRING_ARCH_unaligned. Move a comment to a more appropriate
location.
* include/errno.h, include/stdio.h, include/stdlib.h, include/string.h
* include/time.h, include/unistd.h, include/wchar.h: No need to
check __cplusplus nor use __BEGIN_DECLS/__END_DECLS.
* misc/sys/cdefs.h (__NTHNL): New macro.
* sysdeps/m68k/m680x0/fpu/bits/mathinline.h
(__m81_defun): Use __NTHNL to avoid errors with GCC 6.
* elf/tst-env-setuid-tunables.c: Include config.h with _LIBC
defined, for HAVE_TUNABLES.
* inet/tst-checks-posix.c: No need to define _ISOMAC.
* intl/tst-gettext2.c: Provide own definition of N_.
* math/test-signgam-finite-c99.c: No need to define _ISOMAC.
* math/test-signgam-main.c: No need to define _ISOMAC.
* stdlib/tst-strtod.c: Convert to test-driver. Split locale_test to...
* stdlib/tst-strtod1i.c: ...this new file.
* stdlib/tst-strtod5.c: Convert to test-driver and add copyright notice.
Split tests of __strtod_internal to...
* stdlib/tst-strtod5i.c: ...this new file.
* string/test-string.h: Include stdint.h. Duplicate definition of
inhibit_loop_to_libcall here (from libc-symbols.h).
* string/test-strstr.c: Provide dummy definition of
libc_hidden_builtin_def when including strstr.c.
* sysdeps/ia64/fpu/libm-symbols.h: Suppress entire file in _ISOMAC
mode; no need to test __STRICT_ANSI__ nor __cplusplus as well.
* sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h.
Don't include init-arch.h.
* sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h.
Don't include init-arch.h.
* elf/Makefile: Move tst-ptrguard1-static, tst-stackguard1-static,
tst-tls1-static, tst-tls2-static, tst-tls3-static, loadtest,
unload, unload2, circleload1, neededtest, neededtest2,
neededtest3, neededtest4, tst-tls1, tst-tls2, tst-tls3,
tst-tls6, tst-tls7, tst-tls8, tst-dlmopen2, tst-ptrguard1,
tst-stackguard1, tst-_dl_addr_inside_object, and all of the
ifunc tests to tests-internal.
Don't add $(modules-names) to test-extras.
* inet/Makefile: Move tst-inet6_scopeid_pton to tests-internal.
Add tst-deadline to tests-static-internal.
* malloc/Makefile: Move tst-mallocstate and tst-scratch_buffer to
tests-internal.
* misc/Makefile: Move tst-atomic and tst-atomic-long to tests-internal.
* nptl/Makefile: Move tst-typesizes, tst-rwlock19, tst-sem11,
tst-sem12, tst-sem13, tst-barrier5, tst-signal7, tst-tls3,
tst-tls3-malloc, tst-tls5, tst-stackguard1, tst-sem11-static,
tst-sem12-static, and tst-stackguard1-static to tests-internal.
Link tests-internal with libpthread also.
Don't add $(modules-names) to test-extras.
* nss/Makefile: Move tst-field to tests-internal.
* posix/Makefile: Move bug-regex5, bug-regex20, bug-regex33,
tst-rfc3484, tst-rfc3484-2, and tst-rfc3484-3 to tests-internal.
* stdlib/Makefile: Move tst-strtod1i, tst-strtod3, tst-strtod4,
tst-strtod5i, tst-tls-atexit, and tst-tls-atexit-nodelete to
tests-internal.
* sunrpc/Makefile: Move tst-svc_register to tests-internal.
* sysdeps/powerpc/Makefile: Move test-get_hwcap and
test-get_hwcap-static to tests-internal.
* sysdeps/unix/sysv/linux/Makefile: Move tst-setgetname to
tests-internal.
* sysdeps/x86_64/fpu/Makefile: Add all libmvec test modules to
modules-names-tests.
cppflags-iterator.mk no longer has anything to do with CPPFLAGS; all
it does is set libof-$(foo) for a list of files. extra-modules.mk
does the same thing, but with a different input variable, and doesn't
let the caller control the module. Therefore, this patch gives
cppflags-iterator.mk a better name, removes extra-modules.mk, and
updates all uses of both.
* extra-modules.mk: Delete file.
* cppflags-iterator.mk: Rename to ...
* libof-iterator.mk: ...this. Adjust comments.
* Makerules, extra-lib.mk, benchtests/Makefile, elf/Makefile
* elf/rtld-Rules, iconv/Makefile, locale/Makefile, malloc/Makefile
* nscd/Makefile, sunrpc/Makefile, sysdeps/s390/Makefile:
Use libof-iterator.mk instead of cppflags-iterator.mk or
extra-modules.mk.
* benchtests/strcoll-inputs/filelist#en_US.UTF-8: Remove
extra-modules.mk and cppflags-iterator.mk, add libof-iterator.mk.
Read tunables values from the users using the GLIBC_TUNABLES
environment variable. The value of this variable is a colon-separated
list of name=value pairs. So a typical string would look like this:
GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
* config.make.in (have-loop-to-function): Define.
* elf/Makefile (CFLAGS-dl-tunables.c): Add
-fno-tree-loop-distribute-patterns.
* elf/dl-tunables.c: Include libc-internals.h.
(GLIBC_TUNABLES): New macro.
(tunables_strdup): New function.
(parse_tunables): New function.
(min_strlen): New function.
(__tunables_init): Use the new functions and macro.
(disable_tunable): Disable tunable from GLIBC_TUNABLES.
* malloc/tst-malloc-usable-tunables.c: New test case.
* malloc/tst-malloc-usable-static-tunables.c: New test case.
* malloc/Makefile (tests, tests-static): Add tests.
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* README.tunables: New file.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add HAVE_TUNABLES.
* config.make.in: Add have-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* csu/init-first.c (__libc_init_first): Move
__libc_init_secure earlier...
* csu/init-first.c (LIBC_START_MAIN):... to here.
Include dl-tunables.h, libc-internal.h.
(LIBC_START_MAIN) [!SHARED]: Initialize tunables for static
binaries.
* elf/Makefile (dl-routines): Add dl-tunables.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-support (_dl_nondynamic_init): Unset MALLOC_CHECK_
only when !HAVE_TUNABLES.
* elf/rtld.c (process_envvars): Likewise.
* elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h
(_dl_sysdep_start): Call __tunables_init.
* elf/dl-tunable-types.h: New file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* malloc/tst-malloc-usable-static.c: New test case.
* malloc/Makefile (tests-static): Add it.
* malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK (set_mallopt_check)): New function.
(DL_TUNABLE_CALLBACK_FNDECL): New macro. Use it to define
callback functions.
(ptmalloc_init): Set tunable values.
* scripts/gen-tunables.awk: New file.
* sysdeps/mach/hurd/dl-sysdep.c: Include dl-tunables.h.
(_dl_sysdep_start): Call __tunables_init.
After the removal of __malloc_initialize_hook, newly compiled
Emacs binaries are no longer able to use these interfaces.
malloc_get_state is only used during the Emacs build process,
so we provide a stub implementation only. Existing Emacs binaries
will not call this stub function, but still reference the symbol.
The rewritten tst-mallocstate test constructs a dumped heap
which should approximates what existing Emacs binaries pass
to glibc malloc.
This is a trivial change to add the static tests only to tests-static
and then adding all of tests-static to the tests target to make it
look consistent with some other Makefiles. This avoids having to
duplicate the test names across the two make targets.
* malloc/Makefile (tests): Remove individual static test names
and just add all of tests-static.
Existing interposed mallocs do not define the glibc-internal
fork callbacks (and they should not), so statically interposed
mallocs lead to link failures because the strong reference from
fork pulls in glibc's malloc, resulting in multiple definitions
of malloc-related symbols.
__malloc_initialize_hook is interposed by application code, so
the usual approach to define a compatibility symbol does not work.
This commit adds a new mechanism based on #pragma GCC poison in
<stdc-predef.h>.
This provides a band-aid and addresses the scenario where fork is
called from a signal handler while the process is in the malloc
subsystem (or has acquired the libio list lock). It does not
address the general issue of async-signal-safety of fork;
multi-threaded processes are not covered, and some glibc
subsystems have fork handlers which are not async-signal-safe.
Previously, a thread M invoking fork would acquire locks in this order:
(M1) malloc arena locks (in the registered fork handler)
(M2) libio list lock
A thread F invoking flush (NULL) would acquire locks in this order:
(F1) libio list lock
(F2) individual _IO_FILE locks
A thread G running getdelim would use this order:
(G1) _IO_FILE lock
(G2) malloc arena lock
After executing (M1), (F1), (G1), none of the threads can make progress.
This commit changes the fork lock order to:
(M'1) libio list lock
(M'2) malloc arena locks
It explicitly encodes the lock order in the implementations of fork,
and does not rely on the registration order, thus avoiding the deadlock.
* malloc/Makefile ($(objpfx)tst-malloc-backtrace,
$(objpfx)tst-malloc-thread-exit, $(objpfx)tst-malloc-thread-fail): Use
$(shared-thread-library) instead of hardcoding the path to libpthread.
This test case exercises unusual code paths in allocation functions,
related to allocation failures. Specifically, the test can reveal
the following bugs:
(a) calloc returns non-zero memory on fallback to sysmalloc.
(b) calloc can self-deadlock because it fails to release
the arena lock on certain allocation failures.
(c) pvalloc can dereference a NULL arena pointer.
(a) and (b) appear specific to a faulty downstream backport.
(c) was fixed as part of commit 10ad46bc65.
The test for (a) was inspired by a reproducer supplied by Jeff Layton.
reused_arena can increase the attached thread count of arenas on the
free list. This means that the assertion that the reference count is
zero is incorrect. In this case, the reference count initialization
is incorrect as well and could cause arenas to be put on the free
list too early (while they still have attached threads).
* malloc/arena.c (get_free_list): Remove assert and adjust
reference count handling. Add comment about reused_arena
interaction.
(reused_arena): Add comments abount get_free_list interaction.
* malloc/tst-malloc-thread-exit.c: New file.
* malloc/Makefile (tests): Add tst-malloc-thread-exit.
(tst-malloc-thread-exit): Link against libpthread.
When the malloc subsystem detects some kind of memory corruption,
depending on the configuration it prints the error, a backtrace, a
memory map and then aborts the process. In this process, the
backtrace() call may result in a call to malloc, resulting in
various kinds of problematic behavior.
In one case, the malloc it calls may detect a corruption and call
backtrace again, and a stack overflow may result due to the infinite
recursion. In another case, the malloc it calls may deadlock on an
arena lock with the malloc (or free, realloc, etc.) that detected the
corruption. In yet another case, if the program is linked with
pthreads, backtrace may do a pthread_once initialization, which
deadlocks on itself.
In all these cases, the program exit is not as intended. This is
avoidable by marking the arena that malloc detected a corruption on,
as unusable. The following patch does that. Features of this patch
are as follows:
- A flag is added to the mstate struct of the arena to indicate if the
arena is corrupt.
- The flag is checked whenever malloc functions try to get a lock on
an arena. If the arena is unusable, a NULL is returned, causing the
malloc to use mmap or try the next arena.
- malloc_printerr sets the corrupt flag on the arena when it detects a
corruption
- free does not concern itself with the flag at all. It is not
important since the backtrace workflow does not need free. A free
in a parallel thread may cause another corruption, but that's not
new
- The flag check and set are not atomic and may race. This is fine
since we don't care about contention during the flag check. We want
to make sure that the malloc call in the backtrace does not trip on
itself and all that action happens in the same thread and not across
threads.
I verified that the test case does not show any regressions due to
this patch. I also ran the malloc benchmarks and found an
insignificant difference in timings (< 2%).
* malloc/Makefile (tests): New test case tst-malloc-backtrace.
* malloc/arena.c (arena_lock): Check if arena is corrupt.
(reused_arena): Find a non-corrupt arena.
(heap_trim): Pass arena to unlink.
* malloc/hooks.c (malloc_check_get_size): Pass arena to
malloc_printerr.
(top_check): Likewise.
(free_check): Likewise.
(realloc_check): Likewise.
* malloc/malloc.c (malloc_printerr): Add arena argument.
(unlink): Likewise.
(munmap_chunk): Adjust.
(ARENA_CORRUPTION_BIT): New macro.
(arena_is_corrupt): Likewise.
(set_arena_corrupt): Likewise.
(sysmalloc): Use mmap if there are no usable arenas.
(_int_malloc): Likewise.
(__libc_malloc): Don't fail if arena_get returns NULL.
(_mid_memalign): Likewise.
(__libc_calloc): Likewise.
(__libc_realloc): Adjust for additional argument to
malloc_printerr.
(_int_free): Likewise.
(malloc_consolidate): Likewise.
(_int_realloc): Likewise.
(_int_memalign): Don't touch corrupt arenas.
* malloc/tst-malloc-backtrace.c: New test case.
