At the GNU Tools Cauldron 2016, the state of the current tunables
patchset was considered OK with the addition of a way to select the
frontend to be used for the tunables. That is, to avoid being locked
in to one type of frontend initially, it should be possible to build
tunables with a different frontend with something as simple as a
configure switch.
To that effect, this patch enhances the --enable-tunables option to
accept more values than just 'yes' or 'no'. The current frontend (and
default when enable-tunables is 'yes') is called 'valstring', to
select the frontend where a single environment variable is set to a
colon-separated value string. More such frontends can be added in
future.
* Makeconfig (have-tunables): Check for non-negative instead
of positive.
* configure.ac: Add 'valstring' as a valid value for
--enable-tunables.
* configure: Regenerate.
* elf/Makefile (have-tunables): Check for non-negative instead
of positive.
(CPPFLAGS-dl-tunables.c): Define TUNABLES_FRONTEND for
dl-tunables.c.
* elf/dl-tunables.c (GLIBC_TUNABLES): Define only when
TUNABLES_FRONTEND == TUNABLES_FRONTEND_valstring.
(tunables_strdup): Likewise.
(disable_tunables): Likewise.
(parse_tunables): Likewise.
(__tunables_init): Process GLIBC_TUNABLES envvar only when.
TUNABLES_FRONTEND == TUNABLES_FRONTEND_valstring.
* elf/dl-tunables.h (TUNABLES_FRONTEND_valstring): New macro.
(TUNABLES_FRONTEND_yes): New macro, define as
TUNABLES_FRONTEND_valstring by default.
* manual/install.texi: Document new acceptable values for
--enable-tunables.
* INSTALL: Regenerate.
Read tunables values from the users using the GLIBC_TUNABLES
environment variable. The value of this variable is a colon-separated
list of name=value pairs. So a typical string would look like this:
GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
* config.make.in (have-loop-to-function): Define.
* elf/Makefile (CFLAGS-dl-tunables.c): Add
-fno-tree-loop-distribute-patterns.
* elf/dl-tunables.c: Include libc-internals.h.
(GLIBC_TUNABLES): New macro.
(tunables_strdup): New function.
(parse_tunables): New function.
(min_strlen): New function.
(__tunables_init): Use the new functions and macro.
(disable_tunable): Disable tunable from GLIBC_TUNABLES.
* malloc/tst-malloc-usable-tunables.c: New test case.
* malloc/tst-malloc-usable-static-tunables.c: New test case.
* malloc/Makefile (tests, tests-static): Add tests.
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* README.tunables: New file.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add HAVE_TUNABLES.
* config.make.in: Add have-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* csu/init-first.c (__libc_init_first): Move
__libc_init_secure earlier...
* csu/init-first.c (LIBC_START_MAIN):... to here.
Include dl-tunables.h, libc-internal.h.
(LIBC_START_MAIN) [!SHARED]: Initialize tunables for static
binaries.
* elf/Makefile (dl-routines): Add dl-tunables.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-support (_dl_nondynamic_init): Unset MALLOC_CHECK_
only when !HAVE_TUNABLES.
* elf/rtld.c (process_envvars): Likewise.
* elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h
(_dl_sysdep_start): Call __tunables_init.
* elf/dl-tunable-types.h: New file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* malloc/tst-malloc-usable-static.c: New test case.
* malloc/Makefile (tests-static): Add it.
* malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK (set_mallopt_check)): New function.
(DL_TUNABLE_CALLBACK_FNDECL): New macro. Use it to define
callback functions.
(ptmalloc_init): Set tunable values.
* scripts/gen-tunables.awk: New file.
* sysdeps/mach/hurd/dl-sysdep.c: Include dl-tunables.h.
(_dl_sysdep_start): Call __tunables_init.
The previous commit prevented rtld itself from being built with
-fstack-protector, but this is not quite enough. We identify which
objects belong in rtld via a test link and analysis of the resulting
mapfile. That link is necessarily done against objects that are
stack-protected, so drags in __stack_chk_fail_local, __stack_chk_fail,
and all the libc and libio code they use.
To stop this happening, use --defsym in the test librtld.map-production
link to force the linker to predefine these two symbols (to 0, but it
could be to anything). (In a real link, this would of course be
catastrophic, but these object files are never used for anything else.)
When dynamically linking, ifunc resolvers are called before TLS is
initialized, so they cannot be safely stack-protected.
We avoid disabling stack-protection on large numbers of files by
using __attribute__ ((__optimize__ ("-fno-stack-protector")))
to turn it off just for the resolvers themselves. (We provide
the attribute even when statically linking, because we will later
use it elsewhere too.)
There is at least one use case where during exit a library destructor
might call dlclose() on a valid handle and have it fail with an
assertion. We must allow this case, it is a valid handle, and dlclose()
should not fail with an assert. In the future we might be able to return
an error that the dlclose() could not be completed because the opened
library has already been unloaded and destructors have run as part of
exit processing.
For more details see:
https://www.sourceware.org/ml/libc-alpha/2016-12/msg00859.html
Commit 7a5e3d9d63 (elf: Assume TLS is
initialized in _dl_map_object_from_fd) removed the last call of
_dl_tls_setup, but did not remove the function itself.
The new test driver in <support/test-driver.c> has feature parity with
the old one. The main difference is that its hooking mechanism is
based on functions and function pointers instead of macros. This
commit also implements a new environment variable, TEST_COREDUMPS,
which disables the code which disables coredumps (that is, it enables
them if the invocation environment has not disabled them).
<test-skeleton.c> defines wrapper functions so that it is possible to
use existing macros with the new-style hook functionality.
This commit changes only a few test cases to the new test driver, to
make sure that it works as expected.
Some configurations may use NSS cryptographic routines but have no
static library for those routines. The following changes allow glibc to
be built and tested with --enable-nss-crypt, but without having a static
NSS library. At a high level the change does two things:
(1) Detect at configure time if static NSS crypto libraries are
available. Assumes libfreebl3.a (instead of the existing Fedora
libfreebl.a which is incomplete) which matches libfreebl3.so.
(2) If static NSS crypto libraries are _not_ available then adjust the
way in which we build tst-linkall-static. This includes excluding a
reference to crypt and not linking against libcrypt.a, all of which
will fail otherwise.
Testing assumptions:
* Static library is named libfreebl3.a (not libfreebl.a as is currently
provided in Fedora), matching libfreebl3.so shared link name.
Tested on x86_64 on Fedora with:
(a) --enable-nss-crypt, with no static NSS library support: PASS
(previous FAIL)
(b) --enable-nss-crypt, with faked static NSS library support: PASS
(unsupported)
* Requires changing elf/Makefile to include a stub
/lib64/libfreebl3.a for testing purposes.
(c) --disable-nss-crypt: PASS
(default)
No regressions on x86_64.
For details see:
https://www.sourceware.org/ml/libc-alpha/2016-11/msg00647.html
The first dlopen ("tst-latepthreadmod.so", RTLD_LOCAL | RTLD_LAZY) call
in elf/tst-latepthread.c fails on s390x with "error: dlopen failed:
.../build-s390x/elf/tst-latepthreadmod.so:
undefined symbol: this_function_is_not_defined".
In elf/tst-latepthreadmod.c, this_function_is_not_defined is a sibling
call which leads to a R_390_GLOB_DAT relocation in .rela.dyn instead of
a R_390_JMP_SLOT in .rela.plt.
As RTLD_LAZY skips the JMP_SLOT relocations, but not GLOB_DAT ones,
the dlopen call fails. If elf/tst-latepthreadmod.c is build with
-fno-optimize-sibling-calls, a JMP_SLOT relocation is generated for
this_function_is_not_defined and the test passes.
ChangeLog:
* elf/Makefile (CFLAGS-tst-latepthreadmod.c):
Add -fno-optimize-sibling-calls.
It is no longer needed since commit 6c444ad6e9
(elf: Do not use memalign for TCB/TLS blocks allocation [BZ #17730]).
Applications do not link against ld.so and will use the definition in
libc.so, so there is no ABI impact.
This change moves the main implementation of _dl_catch_error,
_dl_signal_error to libc.so, where TLS variables can be used
directly. This removes a writable function pointer from the
rtld_global variable.
For use during initial relocation, minimal implementations of these
functions are provided in ld.so. These are eventually interposed
by the libc.so implementations. This is implemented by compiling
elf/dl-error-skeleton.c twice, via elf/dl-error.c and
elf/dl-error-minimal.c.
As a side effect of this change, the static version of dl-error.c
no longer includes support for the
_dl_signal_cerror/_dl_receive_error mechanism because it is only
used in ld.so.
The function _dl_addr_inside_object is simplified by removing
the conditional 'reladdr - l->l_phdr[n].p_vaddr >= 0' which is
always true. The function is refactored into it's own object file
and a unit test added to verify the correct behaviour of the
function.
An earlier fix for TLS dropped early initialization of DTV entries for
modules using static TLS, leaving it for __tls_get_addr to set them
up. That worked on platforms that require the GD access model to be
relaxed to LE in the main executable, but it caused a regression on
platforms that allow GD in the main executable, particularly in
statically-linked programs: they use a custom __tls_get_addr that does
not update the DTV, which fails when the DTV early initialization is
not performed.
In static programs, __libc_setup_tls performs the DTV initialization
for the main thread, but the DTV of other threads is set up in
_dl_allocate_tls_init, so that's the fix that matters.
Restoring the initialization in the remaining functions modified by
this patch was just for uniformity. It's not clear that it is ever
needed: even on platforms that allow GD in the main executable, the
dynamically-linked version of __tls_get_addr would set up the DTV
entries, even for static TLS modules, while updating the DTV counter.
for ChangeLog
[BZ #19826]
* elf/dl-tls.c (_dl_allocate_tls_init): Restore DTV early
initialization of static TLS entries.
* elf/dl-reloc.c (_dl_nothread_init_static_tls): Likewise.
* nptl/allocatestack.c (init_one_static_tls): Likewise.
The commit b632bdd3 moved the setting of the DF_1_NODELETE flag earlier
in the dl_open_worker function. However when calling dlopen with both
RTLD_NODELETE and RTLD_NOLOAD, the pointer returned by _dl_map_object is
NULL. This condition is checked just after setting the flag, while it
should be done before. Fix that.
Changelog:
[BZ #19810]
* elf/dl-open.c (dl_open_worker): Set DF_1_NODELETE flag later.
* elf/tst-noload.c: New test case.
* elf/Makefile (tests): Add tst-noload.
Instead, call malloc and explicitly align the pointer.
There is no external location to store the original (unaligned)
pointer, and this commit increases the allocation size to store
the pointer at a fixed location relative to the TCB pointer.
The manual alignment means that some space goes unused which
was previously made available for subsequent allocations.
However, in the TLS_DTV_AT_TP case, the manual alignment code
avoids aligning the pre-TCB to the TLS block alignment. (Even
while using memalign, the allocation had some unused padding
in front.)
This concludes the removal of memalign calls from the TLS code,
and the new tst-tls3-malloc test verifies that only core malloc
routines are used.
Instead of a flag which indicates the pointer can be freed, dtv_t
now includes the pointer which should be freed. Due to padding,
the size of dtv_t does not increase.
To avoid using memalign, the new allocate_dtv_entry function
allocates a sufficiently large buffer so that a sub-buffer
can be found in it which starts with an aligned pointer. Both
the aligned and original pointers are kept, the latter for calling
free later.
The dynamic linker currently uses __libc_memalign for TLS-related
allocations. The goal is to switch to malloc instead. If the minimal
malloc follows the ABI fundamental alignment, we can assume that malloc
provides this alignment, and thus skip explicit alignment in a few
cases as an optimization.
It was requested on libc-alpha that MALLOC_ALIGNMENT should be used,
although this results in wasted space if MALLOC_ALIGNMENT is larger
than the fundamental alignment. (The dynamic linker cannot assume
that the non-minimal malloc will provide an alignment of
MALLOC_ALIGNMENT; the ABI provides _Alignof (max_align_t) only.)
Since _dl_tlsdesc_dynamic is called via PLT, we need to add 8 bytes for
push in the PLT entry to align the stack.
[BZ #20309]
* configure.ac (have-mtls-dialect-gnu2): Set to yes if
-mtls-dialect=gnu2 works.
* configure: Regenerated.
* elf/Makefile [have-mtls-dialect-gnu2 = yes]
(tests): Add tst-gnu2-tls1.
(modules-names): Add tst-gnu2-tls1mod.
($(objpfx)tst-gnu2-tls1): New.
(tst-gnu2-tls1mod.so-no-z-defs): Likewise.
(CFLAGS-tst-gnu2-tls1mod.c): Likewise.
* elf/tst-gnu2-tls1.c: New file.
* elf/tst-gnu2-tls1mod.c: Likewise.
* sysdeps/x86_64/dl-tlsdesc.S (_dl_tlsdesc_dynamic): Add 8
bytes for push in the PLT entry to align the stack.
In a reference to PR ld/19908 make ld.so respect symbol export classes
aka visibility and treat STV_HIDDEN and STV_INTERNAL symbols as local,
preventing such symbols from preempting exported symbols.
According to the ELF gABI[1] neither STV_HIDDEN nor STV_INTERNAL symbols
are supposed to be present in linked binaries:
"A hidden symbol contained in a relocatable object must be either
removed or converted to STB_LOCAL binding by the link-editor when the
relocatable object is included in an executable file or shared object."
"An internal symbol contained in a relocatable object must be either
removed or converted to STB_LOCAL binding by the link-editor when the
relocatable object is included in an executable file or shared object."
however some GNU binutils versions produce such symbols in some cases.
PR ld/19908 is one and we also have this note in scripts/abilist.awk:
so clearly there is linked code out there which contains such symbols
which is prone to symbol table misinterpretation, and it'll be more
productive if we handle this gracefully, under the Robustness Principle:
"be liberal in what you accept, and conservative in what you produce",
especially as this is a simple (STV_HIDDEN|STV_INTERNAL) => STB_LOCAL
mapping.
References:
[1] "System V Application Binary Interface - DRAFT - 24 April 2001",
The Santa Cruz Operation, Inc., "Symbol Table",
<http://www.sco.com/developers/gabi/2001-04-24/ch4.symtab.html>
* sysdeps/generic/ldsodefs.h
(dl_symbol_visibility_binds_local_p): New inline function.
* elf/dl-addr.c (determine_info): Treat hidden and internal
symbols as local.
* elf/dl-lookup.c (do_lookup_x): Likewise.
* elf/dl-reloc.c (RESOLVE_MAP): Likewise.
The EM_BPF number has been officially assigned, though it
has not yet been posted to the gabi webpage yet.
* elf/elf.h (EM_BPF): New.
(EM_NUM): Update.
(R_BPF_NONE, R_BPF_MAP_FD): New.
__libc_memalign in ld.so allocates one page at a time and tries to
optimize consecutive __libc_memalign calls by hoping that the next
mmap is after the current memory allocation.
However, the kernel hands out mmap addresses in top-down order, so
this optimization in practice never happens, with the result that we
have more mmap calls and waste a bunch of space for each __libc_memalign.
This change makes __libc_memalign to mmap one page extra. Worst case,
the kernel never puts a backing page behind it, but best case it allows
__libc_memalign to operate much much better. For elf/tst-align --direct,
it reduces number of mmap calls from 12 to 9.
* elf/dl-minimal.c (__libc_memalign): Mmap one extra page.
This patch fixes the new test tst-dlsym-error build on aarch64
(and possible other architectures as well) due missing strchrnul
definition.
* elf/tst-dlsym-error.c: Include <string.h> for strchrnul.
* elf/dl-lookup.c (_dl_lookup_symbol_x): Report error even if
skip_map != NULL.
* elf/tst-dlsym-error.c: New file.
* elf/Makefile (tests): Add tst-dlsym-error.
(tst-dlsym-error): Link against libdl.
When using sln on some filesystems which return 64-bit inodes,
the stat call might fail during install like so:
.../elf/sln .../elf/symlink.list
/lib32/libc.so.6: invalid destination: Value too large for defined data type
/lib32/ld-linux.so.2: invalid destination: Value too large for defined data type
Makefile:104: recipe for target 'install-symbolic-link' failed
Switch to using stat64 all the time to avoid this.
URL: https://bugs.gentoo.org/576396
In 1999 the project split "localedir" into "localedir" (path to compiled
locale archives) and "msgcatdir" (path to message catalogs). This
predates the 2002 change in the GNU Coding Standard to document the use
of "localedir" for the path to the message catalogs. It appears that
newlib, gcc, and several other projects also used "msgcatdir" at one
point or another in the past, and so it is in line with historical
precedent that glibc would also use "msgcatdir." However, given that the
GNU Coding Standard uses "localedir", we will switch to that for
consistency as a GNU project. Previous uses of --localdir didn't work
anyway (see bug 14259).
I am committing this patch in the understanding that nobody would object
to fixing #14259 as part of aligning our variable usage to the GNU
Coding Standard.
Given that previous "localedir" uses were converted to "complocaledir"
by [1], we can now convert "msgcatdir" to "localedir" and complete the
transition. With an addition to config.make.in we also fix bug 14259 and
allow users to specify the locale dependent data directory with
"--localedir" at configure time. There is still no way to control at
configure time the location of the *compiled* locale directory.
Tested on x86_64 with no regressions.
Tested using "--localedir" to specify alternate locale dependent data
directory and verified with "make install DESTDIR=/tmp/glibc".
[1] 90fe682d30
The following new 386 and X86_64 were added to binutils. They are
non-dynamic relocations, so don't need direct handling in glibc.
But other programs, like elfutils, use the glibc elf.h definitions
for the names and numbers when inspecting ET_REL files.
R_386_GOT32X was proposed in
https://groups.google.com/forum/#!topic/ia32-abi/GbJJskkid4I
X86_64_GOTPCRELX and R_X86_64_REX_GOTPCRELX were proposed in
https://groups.google.com/forum/#!topic/x86-64-abi/n9AWHogmVY0
There also used to be R_X86_64_PC32_BND and R_X86_64_PLT32_BND
but those already got deprecated in
https://groups.google.com/d/msg/x86-64-abi/-hdQyMixt8Y/XFDOvioG85cJ
* elf/elf.h (R_386_GOT32X): New.
(R_386_NUM): Update.
(R_X86_64_GOTPCRELX: New.
(R_X86_64_REX_GOTPCRELX): New.
(R_X86_64_NUM): Update.
The following new 386 and X86_64 were added to binutils. They are
non-dynamic relocations, so don't need direct handling in glibc.
But other programs, like elfutils, use the glibc elf.h definitions
for the names and numbers when inspecting ET_REL files.
R_386_GOT32X was proposed in
https://groups.google.com/forum/#!topic/ia32-abi/GbJJskkid4I
X86_64_GOTPCRELX and R_X86_64_REX_GOTPCRELX were proposed in
https://groups.google.com/forum/#!topic/x86-64-abi/n9AWHogmVY0
There also used to be R_X86_64_PC32_BND and R_X86_64_PLT32_BND
but those already got deprecated in
https://groups.google.com/d/msg/x86-64-abi/-hdQyMixt8Y/XFDOvioG85cJ
* elf/elf.h (R_386_GOT32X): New.
(R_386_NUM): Update.
(R_X86_64_GOTPCRELX: New.
(R_X86_64_REX_GOTPCRELX): New.
(R_X86_64_NUM): Update.
