2001-09-19 10:37:31 +00:00
|
|
|
/* Machine-dependent ELF dynamic relocation inline functions. x86-64 version.
|
2021-01-02 19:32:25 +00:00
|
|
|
Copyright (C) 2001-2021 Free Software Foundation, Inc.
|
2001-09-19 10:37:31 +00:00
|
|
|
This file is part of the GNU C Library.
|
|
|
|
Contributed by Andreas Jaeger <aj@suse.de>.
|
|
|
|
|
|
|
|
The GNU C Library is free software; you can redistribute it and/or
|
|
|
|
modify it under the terms of the GNU Lesser General Public
|
|
|
|
License as published by the Free Software Foundation; either
|
|
|
|
version 2.1 of the License, or (at your option) any later version.
|
|
|
|
|
|
|
|
The GNU C Library is distributed in the hope that it will be useful,
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
|
|
Lesser General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU Lesser General Public
|
2012-02-09 23:18:22 +00:00
|
|
|
License along with the GNU C Library; if not, see
|
Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:
sed -ri '
s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
$(find $(git ls-files) -prune -type f \
! -name '*.po' \
! -name 'ChangeLog*' \
! -path COPYING ! -path COPYING.LIB \
! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
! -path manual/texinfo.tex ! -path scripts/config.guess \
! -path scripts/config.sub ! -path scripts/install-sh \
! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
! -path INSTALL ! -path locale/programs/charmap-kw.h \
! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
! '(' -name configure \
-execdir test -f configure.ac -o -f configure.in ';' ')' \
! '(' -name preconfigure \
-execdir test -f preconfigure.ac ';' ')' \
-print)
and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:
chmod a+x sysdeps/unix/sysv/linux/riscv/configure
# Omit irrelevant whitespace and comment-only changes,
# perhaps from a slightly-different Autoconf version.
git checkout -f \
sysdeps/csky/configure \
sysdeps/hppa/configure \
sysdeps/riscv/configure \
sysdeps/unix/sysv/linux/csky/configure
# Omit changes that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
git checkout -f \
sysdeps/powerpc/powerpc64/ppc-mcount.S \
sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
# Omit change that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
|
|
|
<https://www.gnu.org/licenses/>. */
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
#ifndef dl_machine_h
|
|
|
|
#define dl_machine_h
|
|
|
|
|
|
|
|
#define ELF_MACHINE_NAME "x86_64"
|
|
|
|
|
elf: Fix elf_get_dynamic_info definition
Before to 490e6c62aa31a8a ('elf: Avoid nested functions in the loader
[BZ #27220]'), elf_get_dynamic_info() was defined twice on rtld.c: on
the first dynamic-link.h include and later within _dl_start(). The
former definition did not define DONT_USE_BOOTSTRAP_MAP and it is used
on setup_vdso() (since it is a global definition), while the former does
define DONT_USE_BOOTSTRAP_MAP and it is used on loader self-relocation.
With the commit change, the function is now included and defined once
instead of defined as a nested function. So rtld.c defines without
defining RTLD_BOOTSTRAP and it brokes at least powerpc32.
This patch fixes by moving the get-dynamic-info.h include out of
dynamic-link.h, which then the caller can corirectly set the expected
semantic by defining STATIC_PIE_BOOTSTRAP, RTLD_BOOTSTRAP, and/or
RESOLVE_MAP.
It also required to enable some asserts only for the loader bootstrap
to avoid issues when called from setup_vdso().
As a side note, this is another issues with nested functions: it is
not clear from pre-processed output (-E -dD) how the function will
be build and its semantic (since nested function will be local and
extra C defines may change it).
I checked on x86_64-linux-gnu (w/o --enable-static-pie),
i686-linux-gnu, powerpc64-linux-gnu, powerpc-linux-gnu-power4,
aarch64-linux-gnu, arm-linux-gnu, sparc64-linux-gnu, and
s390x-linux-gnu.
Reviewed-by: Fangrui Song <maskray@google.com>
(cherry picked from commit 4af6982e4c9fc465ffb7a54b794aaaa134241f05)
Resolved conflicts:
elf/rtld.c
2021-10-11 19:01:49 +00:00
|
|
|
#include <assert.h>
|
2001-09-19 10:37:31 +00:00
|
|
|
#include <sys/param.h>
|
2003-05-30 16:12:18 +00:00
|
|
|
#include <sysdep.h>
|
2004-01-13 23:15:57 +00:00
|
|
|
#include <tls.h>
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
#include <dl-tlsdesc.h>
|
elf: Fix dynamic-link.h usage on rtld.c
The 4af6982e4c fix does not fully handle RTLD_BOOTSTRAP usage on
rtld.c due two issues:
1. RTLD_BOOTSTRAP is also used on dl-machine.h on various
architectures and it changes the semantics of various machine
relocation functions.
2. The elf_get_dynamic_info() change was done sideways, previously
to 490e6c62aa get-dynamic-info.h was included by the first
dynamic-link.h include *without* RTLD_BOOTSTRAP being defined.
It means that the code within elf_get_dynamic_info() that uses
RTLD_BOOTSTRAP is in fact unused.
To fix 1. this patch now includes dynamic-link.h only once with
RTLD_BOOTSTRAP defined. The ELF_DYNAMIC_RELOCATE call will now have
the relocation fnctions with the expected semantics for the loader.
And to fix 2. part of 4af6982e4c is reverted (the check argument
elf_get_dynamic_info() is not required) and the RTLD_BOOTSTRAP
pieces are removed.
To reorganize the includes the static TLS definition is moved to
its own header to avoid a circular dependency (it is defined on
dynamic-link.h and dl-machine.h requires it at same time other
dynamic-link.h definition requires dl-machine.h defitions).
Also ELF_MACHINE_NO_REL, ELF_MACHINE_NO_RELA, and ELF_MACHINE_PLT_REL
are moved to its own header. Only ancient ABIs need special values
(arm, i386, and mips), so a generic one is used as default.
The powerpc Elf64_FuncDesc is also moved to its own header, since
csu code required its definition (which would require either include
elf/ folder or add a full path with elf/).
Checked on x86_64, i686, aarch64, armhf, powerpc64, powerpc32,
and powerpc64le.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
(cherry picked from commit d6d89608ac8cf2b37c75debad1fff653f6939f90)
Resolved conflicts:
elf/rtld.c
2021-10-13 12:49:34 +00:00
|
|
|
#include <dl-static-tls.h>
|
|
|
|
#include <dl-machine-rel.h>
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
/* Return nonzero iff ELF header is compatible with the running host. */
|
|
|
|
static inline int __attribute__ ((unused))
|
2012-05-08 16:39:58 +00:00
|
|
|
elf_machine_matches_host (const ElfW(Ehdr) *ehdr)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
|
|
|
return ehdr->e_machine == EM_X86_64;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* Return the link-time address of _DYNAMIC. Conveniently, this is the
|
|
|
|
first element of the GOT. This must be inlined in a function which
|
|
|
|
uses global data. */
|
2012-05-08 16:39:58 +00:00
|
|
|
static inline ElfW(Addr) __attribute__ ((unused))
|
2001-09-19 10:37:31 +00:00
|
|
|
elf_machine_dynamic (void)
|
|
|
|
{
|
2012-09-02 12:22:24 +00:00
|
|
|
/* This produces an IP-relative reloc which is resolved at link time. */
|
|
|
|
extern const ElfW(Addr) _GLOBAL_OFFSET_TABLE_[] attribute_hidden;
|
|
|
|
return _GLOBAL_OFFSET_TABLE_[0];
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* Return the run-time load address of the shared object. */
|
2012-05-08 16:39:58 +00:00
|
|
|
static inline ElfW(Addr) __attribute__ ((unused))
|
2001-09-19 10:37:31 +00:00
|
|
|
elf_machine_load_address (void)
|
|
|
|
{
|
2012-09-02 12:22:24 +00:00
|
|
|
/* Compute the difference between the runtime address of _DYNAMIC as seen
|
|
|
|
by an IP-relative reference, and the link-time address found in the
|
|
|
|
special unrelocated first GOT entry. */
|
2017-10-04 00:48:55 +00:00
|
|
|
extern ElfW(Dyn) _DYNAMIC[] attribute_hidden;
|
|
|
|
return (ElfW(Addr)) &_DYNAMIC - elf_machine_dynamic ();
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Set up the loaded object described by L so its unrelocated PLT
|
|
|
|
entries will jump to the on-demand fixup code in dl-runtime.c. */
|
|
|
|
|
2003-06-18 19:34:34 +00:00
|
|
|
static inline int __attribute__ ((unused, always_inline))
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
elf_machine_runtime_setup (struct link_map *l, struct r_scope_elem *scope[],
|
|
|
|
int lazy, int profile)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
|
|
|
Elf64_Addr *got;
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
extern void _dl_runtime_resolve_fxsave (ElfW(Word)) attribute_hidden;
|
|
|
|
extern void _dl_runtime_resolve_xsave (ElfW(Word)) attribute_hidden;
|
|
|
|
extern void _dl_runtime_resolve_xsavec (ElfW(Word)) attribute_hidden;
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
extern void _dl_runtime_profile_sse (ElfW(Word)) attribute_hidden;
|
|
|
|
extern void _dl_runtime_profile_avx (ElfW(Word)) attribute_hidden;
|
|
|
|
extern void _dl_runtime_profile_avx512 (ElfW(Word)) attribute_hidden;
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
if (l->l_info[DT_JMPREL] && lazy)
|
|
|
|
{
|
|
|
|
/* The GOT entries for functions in the PLT have not yet been filled
|
|
|
|
in. Their initial contents will arrange when called to push an
|
|
|
|
offset into the .rel.plt section, push _GLOBAL_OFFSET_TABLE_[1],
|
2011-12-23 18:59:40 +00:00
|
|
|
and then jump to _GLOBAL_OFFSET_TABLE_[2]. */
|
2001-09-19 10:37:31 +00:00
|
|
|
got = (Elf64_Addr *) D_PTR (l, l_info[DT_PLTGOT]);
|
Update.
2001-12-11 Jakub Jelinek <jakub@redhat.com>
* elf/Makefile (dl-routines): Add conflict.
(rtld-ldscript-in, rtld-ldscript, rtld-parms): Remove.
(ld.so): Add _begin local symbol.
* elf/elf.h (DT_VALTAGIDX, DT_VALNUM, DT_ADDRTAGIDX, DT_ADDRNUM):
Define.
* elf/dl-deps.c (_dl_build_local_scope): New.
(_dl_map_object_deps): If LD_TRACE_PRELINKING, compute local scopes
of all libraries.
* elf/do-rel.h (VALIDX): Define.
(elf_dynamic_do_rel): If ELF_MACHINE_PLT_REL is defined, don't do
lazy binding for RELA. If DT_GNU_PRELINKED, DT_RELACOUNT relocations
can be skipped.
* elf/dl-conflict.c: New file.
* elf/dl-lookup.c (_dl_debug_bindings): New.
(_dl_lookup_symbol): Use _dl_debug_bindings. Reference_name is always
non-NULL.
(_dl_lookup_symbol_skip): Likewise.
(_dl_lookup_versioned_symbol): Likewise.
(_dl_lookup_versioned_symbol_skip): Likewise.
* elf/dl-runtime.c (PLTREL): If ELF_MACHINE_PLT_REL is defined,
define to ElfW(Rel).
* elf/dynamic-link.h (elf_get_dynamic_info): Record selected dynamic
tags in the DT_VALRNGLO..DT_VALRNGHI and DT_ADDRRNGLO..DT_ADDRRNGHI
ranges.
Don't adjust address dynamic tags if l_addr is 0.
* elf/rtld.c (_dl_trace_prelink, _dl_trace_prelink_map): New variables.
(_dl_start): Skip ELF_DYNAMIC_RELOCATE if ld.so is prelinked.
(VALIDX, ADDRIDX): Define.
(_dl_start_final): Initialize _dl_rtld_map's l_map_start and l_map_end.
(dl_main): Print library list for LD_TRACE_PRELINKING.
If prelinking information can be used, skip relocating libraries and
call _dl_resolve_conflicts instead.
(process_envvars): Handle LD_TRACE_PRELINKING envvar.
* elf/dl-load.c (_dl_map_object): Don't create fake libs
if LD_TRACE_PRELINKING.
* include/link.h (struct link_map) [l_info]: Add DT_VALNUM
+ DT_ADDRNUM.
* sysdeps/generic/ldsodefs.h (_dl_trace_prelink_map): New declaration.
(DL_DEBUG_PRELINK): Define.
(_dl_resolve_conflicts): Add prototype.
* sysdeps/alpha/dl-machine.h (elf_machine_runtime_setup): Reinitialize
.plt for prelinked libraries where prelinking info cannot be used.
(elf_machine_rela): If relocating R_ALPHA_JMP_SLOT in .gnu.conflict
section, use RESOLVE_CONFLICT_FIND_MAP to find out reloc's link_map.
* sysdeps/arm/bits/link.h: New file.
* sysdeps/arm/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_ARM_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/i386/bits/link.h: New file.
* sysdeps/i386/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_386_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/powerpc/dl-machine.h (elf_machine_rela): If relocating
conflicts, skip finaladdr computation. Use RESOLVE_CONFLICT_FIND_MAP
to find out map for R_PPC_JMP_SLOT relocs.
* sysdeps/sparc/sparc32/dl-machine.h (VALIDX): Define.
(OPCODE_BA): Define.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
(sparc_fixup_plt): Renamed from elf_machine_fixup_plt.
(elf_machine_fixup_plt): Call sparc_fixup_plt.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Call sparc_fixup_plt for R_SPARC_JMP_SLOT.
* sysdeps/sparc/sparc64/dl-machine.h (VALIDX): Define.
(sparc64_fixup_plt): Fix a typo.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Handle R_SPARC_JMP_SLOT relocs when relocating conflicts.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
* sysdeps/sh/bits/link.h: New file.
* sysdeps/sh/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_SH_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-32/bits/link.h: New file.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-64/bits/link.h: New file.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/x86_64/bits/link.h: New file.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_X86_64_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
2001-12-12 00:21:26 +00:00
|
|
|
/* If a library is prelinked but we have to relocate anyway,
|
|
|
|
we have to be able to undo the prelinking of .got.plt.
|
|
|
|
The prelinker saved us here address of .plt + 0x16. */
|
|
|
|
if (got[1])
|
|
|
|
{
|
|
|
|
l->l_mach.plt = got[1] + l->l_addr;
|
2012-05-08 16:39:58 +00:00
|
|
|
l->l_mach.gotplt = (ElfW(Addr)) &got[3];
|
Update.
2001-12-11 Jakub Jelinek <jakub@redhat.com>
* elf/Makefile (dl-routines): Add conflict.
(rtld-ldscript-in, rtld-ldscript, rtld-parms): Remove.
(ld.so): Add _begin local symbol.
* elf/elf.h (DT_VALTAGIDX, DT_VALNUM, DT_ADDRTAGIDX, DT_ADDRNUM):
Define.
* elf/dl-deps.c (_dl_build_local_scope): New.
(_dl_map_object_deps): If LD_TRACE_PRELINKING, compute local scopes
of all libraries.
* elf/do-rel.h (VALIDX): Define.
(elf_dynamic_do_rel): If ELF_MACHINE_PLT_REL is defined, don't do
lazy binding for RELA. If DT_GNU_PRELINKED, DT_RELACOUNT relocations
can be skipped.
* elf/dl-conflict.c: New file.
* elf/dl-lookup.c (_dl_debug_bindings): New.
(_dl_lookup_symbol): Use _dl_debug_bindings. Reference_name is always
non-NULL.
(_dl_lookup_symbol_skip): Likewise.
(_dl_lookup_versioned_symbol): Likewise.
(_dl_lookup_versioned_symbol_skip): Likewise.
* elf/dl-runtime.c (PLTREL): If ELF_MACHINE_PLT_REL is defined,
define to ElfW(Rel).
* elf/dynamic-link.h (elf_get_dynamic_info): Record selected dynamic
tags in the DT_VALRNGLO..DT_VALRNGHI and DT_ADDRRNGLO..DT_ADDRRNGHI
ranges.
Don't adjust address dynamic tags if l_addr is 0.
* elf/rtld.c (_dl_trace_prelink, _dl_trace_prelink_map): New variables.
(_dl_start): Skip ELF_DYNAMIC_RELOCATE if ld.so is prelinked.
(VALIDX, ADDRIDX): Define.
(_dl_start_final): Initialize _dl_rtld_map's l_map_start and l_map_end.
(dl_main): Print library list for LD_TRACE_PRELINKING.
If prelinking information can be used, skip relocating libraries and
call _dl_resolve_conflicts instead.
(process_envvars): Handle LD_TRACE_PRELINKING envvar.
* elf/dl-load.c (_dl_map_object): Don't create fake libs
if LD_TRACE_PRELINKING.
* include/link.h (struct link_map) [l_info]: Add DT_VALNUM
+ DT_ADDRNUM.
* sysdeps/generic/ldsodefs.h (_dl_trace_prelink_map): New declaration.
(DL_DEBUG_PRELINK): Define.
(_dl_resolve_conflicts): Add prototype.
* sysdeps/alpha/dl-machine.h (elf_machine_runtime_setup): Reinitialize
.plt for prelinked libraries where prelinking info cannot be used.
(elf_machine_rela): If relocating R_ALPHA_JMP_SLOT in .gnu.conflict
section, use RESOLVE_CONFLICT_FIND_MAP to find out reloc's link_map.
* sysdeps/arm/bits/link.h: New file.
* sysdeps/arm/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_ARM_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/i386/bits/link.h: New file.
* sysdeps/i386/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_386_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/powerpc/dl-machine.h (elf_machine_rela): If relocating
conflicts, skip finaladdr computation. Use RESOLVE_CONFLICT_FIND_MAP
to find out map for R_PPC_JMP_SLOT relocs.
* sysdeps/sparc/sparc32/dl-machine.h (VALIDX): Define.
(OPCODE_BA): Define.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
(sparc_fixup_plt): Renamed from elf_machine_fixup_plt.
(elf_machine_fixup_plt): Call sparc_fixup_plt.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Call sparc_fixup_plt for R_SPARC_JMP_SLOT.
* sysdeps/sparc/sparc64/dl-machine.h (VALIDX): Define.
(sparc64_fixup_plt): Fix a typo.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Handle R_SPARC_JMP_SLOT relocs when relocating conflicts.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
* sysdeps/sh/bits/link.h: New file.
* sysdeps/sh/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_SH_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-32/bits/link.h: New file.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-64/bits/link.h: New file.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/x86_64/bits/link.h: New file.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_X86_64_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
2001-12-12 00:21:26 +00:00
|
|
|
}
|
2012-05-08 16:39:58 +00:00
|
|
|
/* Identify this shared object. */
|
|
|
|
*(ElfW(Addr) *) (got + 1) = (ElfW(Addr)) l;
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
/* The got[2] entry contains the address of a function which gets
|
|
|
|
called to get the address of a so far unresolved function and
|
|
|
|
jump to it. The profiling extension of the dynamic linker allows
|
|
|
|
to intercept the calls to collect information. In this case we
|
|
|
|
don't store the address in the GOT so that all future calls also
|
|
|
|
end in this function. */
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (profile))
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE (AVX512F))
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
*(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx512;
|
2020-06-29 23:36:08 +00:00
|
|
|
else if (CPU_FEATURE_USABLE (AVX))
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
*(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx;
|
|
|
|
else
|
|
|
|
*(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_sse;
|
2001-09-19 10:37:31 +00:00
|
|
|
|
2005-01-06 22:40:27 +00:00
|
|
|
if (GLRO(dl_profile) != NULL
|
|
|
|
&& _dl_name_match_p (GLRO(dl_profile), l))
|
2001-09-19 10:37:31 +00:00
|
|
|
/* This is the object we are looking for. Say that we really
|
|
|
|
want profiling and the timers are started. */
|
2002-02-01 01:33:04 +00:00
|
|
|
GL(dl_profile_map) = l;
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
else
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
{
|
|
|
|
/* This function will get called to fix up the GOT entry
|
|
|
|
indicated by the offset on the stack, and then jump to
|
|
|
|
the resolved address. */
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
if (GLRO(dl_x86_cpu_features).xsave_state_size != 0)
|
|
|
|
*(ElfW(Addr) *) (got + 2)
|
2020-06-29 23:36:08 +00:00
|
|
|
= (CPU_FEATURE_USABLE (XSAVEC)
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
? (ElfW(Addr)) &_dl_runtime_resolve_xsavec
|
|
|
|
: (ElfW(Addr)) &_dl_runtime_resolve_xsave);
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
else
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
*(ElfW(Addr) *) (got + 2)
|
|
|
|
= (ElfW(Addr)) &_dl_runtime_resolve_fxsave;
|
Save and restore vector registers in x86-64 ld.so
This patch adds SSE, AVX and AVX512 versions of _dl_runtime_resolve
and _dl_runtime_profile, which save and restore the first 8 vector
registers used for parameter passing. elf_machine_runtime_setup
selects the proper _dl_runtime_resolve or _dl_runtime_profile based
on _dl_x86_cpu_features. It avoids race condition caused by
FOREIGN_CALL macros, which are only used for x86-64.
Performance impact of saving and restoring 8 vector registers are
negligible on Nehalem, Sandy Bridge, Ivy Bridge and Haswell when
ld.so is optimized with SSE2.
[BZ #15128]
* sysdeps/x86_64/Makefile [$(subdir) == elf] (tests): Add
ifuncmain8.
(modules-names): Add ifuncmod8.
($(objpfx)ifuncmain8): New rule.
* sysdeps/x86_64/dl-machine.h: Include <dl-procinfo.h> and
<cpuid.h>.
(elf_machine_runtime_setup): Use _dl_runtime_resolve_sse,
_dl_runtime_resolve_avx, or _dl_runtime_resolve_avx512,
_dl_runtime_profile_sse, _dl_runtime_profile_avx, or
_dl_runtime_profile_avx512, based on HAS_ARCH_FEATURE.
* sysdeps/x86_64/dl-trampoline.S: Rewrite.
* sysdeps/x86_64/dl-trampoline.h: Likewise.
* sysdeps/x86_64/ifuncmain8.c: New file.
* sysdeps/x86_64/ifuncmod8.c: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym (RTLD_SAVESPACE_SSE):
Removed.
* sysdeps/x86_64/nptl/tls.h (__128bits): Removed.
(tcbhead_t): Change rtld_must_xmm_save to __glibc_unused1.
Change rtld_savespace_sse to __glibc_unused2.
(RTLD_CHECK_FOREIGN_CALL): Removed.
(RTLD_ENABLE_FOREIGN_CALL): Likewise.
(RTLD_PREPARE_FOREIGN_CALL): Likewise.
(RTLD_FINALIZE_FOREIGN_CALL): Likewise.
2015-08-25 11:33:54 +00:00
|
|
|
}
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return lazy;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Initial entry point code for the dynamic linker.
|
|
|
|
The C function `_dl_start' is the real entry point;
|
|
|
|
its return value is the user program's entry point. */
|
|
|
|
#define RTLD_START asm ("\n\
|
|
|
|
.text\n\
|
|
|
|
.align 16\n\
|
|
|
|
.globl _start\n\
|
|
|
|
.globl _dl_start_user\n\
|
|
|
|
_start:\n\
|
|
|
|
movq %rsp, %rdi\n\
|
|
|
|
call _dl_start\n\
|
|
|
|
_dl_start_user:\n\
|
|
|
|
# Save the user entry point address in %r12.\n\
|
|
|
|
movq %rax, %r12\n\
|
|
|
|
# See if we were run as a command with the executable file\n\
|
|
|
|
# name as an extra leading argument.\n\
|
2003-09-23 21:31:54 +00:00
|
|
|
movl _dl_skip_args(%rip), %eax\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# Pop the original argument count.\n\
|
|
|
|
popq %rdx\n\
|
|
|
|
# Adjust the stack pointer to skip _dl_skip_args words.\n\
|
|
|
|
leaq (%rsp,%rax,8), %rsp\n\
|
|
|
|
# Subtract _dl_skip_args from argc.\n\
|
|
|
|
subl %eax, %edx\n\
|
|
|
|
# Push argc back on the stack.\n\
|
|
|
|
pushq %rdx\n\
|
|
|
|
# Call _dl_init (struct link_map *main_map, int argc, char **argv, char **env)\n\
|
|
|
|
# argc -> rsi\n\
|
|
|
|
movq %rdx, %rsi\n\
|
2005-01-22 07:55:35 +00:00
|
|
|
# Save %rsp value in %r13.\n\
|
|
|
|
movq %rsp, %r13\n\
|
2014-11-04 23:26:39 +00:00
|
|
|
# And align stack for the _dl_init call. \n\
|
2005-01-22 07:55:35 +00:00
|
|
|
andq $-16, %rsp\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# _dl_loaded -> rdi\n\
|
2003-09-23 21:31:54 +00:00
|
|
|
movq _rtld_local(%rip), %rdi\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# env -> rcx\n\
|
2005-01-22 07:55:35 +00:00
|
|
|
leaq 16(%r13,%rdx,8), %rcx\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# argv -> rdx\n\
|
2005-01-22 07:55:35 +00:00
|
|
|
leaq 8(%r13), %rdx\n\
|
|
|
|
# Clear %rbp to mark outermost frame obviously even for constructors.\n\
|
2005-03-31 10:02:53 +00:00
|
|
|
xorl %ebp, %ebp\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# Call the function to run the initializers.\n\
|
2014-12-21 13:27:31 +00:00
|
|
|
call _dl_init\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# Pass our finalizer function to the user in %rdx, as per ELF ABI.\n\
|
2003-09-23 21:31:54 +00:00
|
|
|
leaq _dl_fini(%rip), %rdx\n\
|
2005-01-22 07:55:35 +00:00
|
|
|
# And make sure %rsp points to argc stored on the stack.\n\
|
|
|
|
movq %r13, %rsp\n\
|
2001-09-19 10:37:31 +00:00
|
|
|
# Jump to the user's entry point.\n\
|
|
|
|
jmp *%r12\n\
|
|
|
|
.previous\n\
|
|
|
|
");
|
|
|
|
|
2002-09-27 07:29:51 +00:00
|
|
|
/* ELF_RTYPE_CLASS_PLT iff TYPE describes relocation of a PLT entry or
|
|
|
|
TLS variable, so undefined references should not be allowed to
|
|
|
|
define the value.
|
2015-03-05 16:26:21 +00:00
|
|
|
ELF_RTYPE_CLASS_COPY iff TYPE should not be allowed to resolve to one
|
2015-03-31 12:15:43 +00:00
|
|
|
of the main executable's symbols, as for a COPY reloc.
|
|
|
|
ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA iff TYPE describes relocation may
|
|
|
|
against protected data whose address be external due to copy relocation.
|
|
|
|
*/
|
2011-09-10 20:50:28 +00:00
|
|
|
#define elf_machine_type_class(type) \
|
2002-09-27 08:29:54 +00:00
|
|
|
((((type) == R_X86_64_JUMP_SLOT \
|
|
|
|
|| (type) == R_X86_64_DTPMOD64 \
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
|| (type) == R_X86_64_DTPOFF64 \
|
|
|
|
|| (type) == R_X86_64_TPOFF64 \
|
|
|
|
|| (type) == R_X86_64_TLSDESC) \
|
2002-09-27 08:29:54 +00:00
|
|
|
* ELF_RTYPE_CLASS_PLT) \
|
2015-03-31 12:15:43 +00:00
|
|
|
| (((type) == R_X86_64_COPY) * ELF_RTYPE_CLASS_COPY) \
|
|
|
|
| (((type) == R_X86_64_GLOB_DAT) * ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA))
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
/* A reloc type used for ld.so cmdline arg lookups to reject PLT entries. */
|
|
|
|
#define ELF_MACHINE_JMP_SLOT R_X86_64_JUMP_SLOT
|
|
|
|
|
2011-10-17 00:58:17 +00:00
|
|
|
/* The relative ifunc relocation. */
|
|
|
|
// XXX This is a work-around for a broken linker. Remove!
|
|
|
|
#define ELF_MACHINE_IRELATIVE R_X86_64_IRELATIVE
|
|
|
|
|
2011-12-23 18:59:40 +00:00
|
|
|
/* We define an initialization function. This is called very early in
|
2001-09-19 10:37:31 +00:00
|
|
|
_dl_sysdep_start. */
|
|
|
|
#define DL_PLATFORM_INIT dl_platform_init ()
|
|
|
|
|
|
|
|
static inline void __attribute__ ((unused))
|
|
|
|
dl_platform_init (void)
|
|
|
|
{
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
#if IS_IN (rtld)
|
2020-07-04 13:35:49 +00:00
|
|
|
/* _dl_x86_init_cpu_features is a wrapper for init_cpu_features which
|
|
|
|
has been called early from __libc_start_main in static executable. */
|
|
|
|
_dl_x86_init_cpu_features ();
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
#else
|
|
|
|
if (GLRO(dl_platform) != NULL && *GLRO(dl_platform) == '\0')
|
|
|
|
/* Avoid an empty string which would disturb us. */
|
|
|
|
GLRO(dl_platform) = NULL;
|
2016-05-13 15:29:22 +00:00
|
|
|
#endif
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
|
2012-05-08 16:39:58 +00:00
|
|
|
static inline ElfW(Addr)
|
2001-09-19 10:37:31 +00:00
|
|
|
elf_machine_fixup_plt (struct link_map *map, lookup_t t,
|
2017-06-14 01:17:25 +00:00
|
|
|
const ElfW(Sym) *refsym, const ElfW(Sym) *sym,
|
2012-05-08 16:39:58 +00:00
|
|
|
const ElfW(Rela) *reloc,
|
|
|
|
ElfW(Addr) *reloc_addr, ElfW(Addr) value)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
|
|
|
return *reloc_addr = value;
|
|
|
|
}
|
|
|
|
|
2011-12-23 18:59:40 +00:00
|
|
|
/* Return the final value of a PLT relocation. On x86-64 the
|
|
|
|
JUMP_SLOT relocation ignores the addend. */
|
2012-05-08 16:39:58 +00:00
|
|
|
static inline ElfW(Addr)
|
|
|
|
elf_machine_plt_value (struct link_map *map, const ElfW(Rela) *reloc,
|
|
|
|
ElfW(Addr) value)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
|
|
|
return value;
|
|
|
|
}
|
|
|
|
|
2005-01-06 22:40:27 +00:00
|
|
|
|
|
|
|
/* Names of the architecture-specific auditing callback functions. */
|
|
|
|
#define ARCH_LA_PLTENTER x86_64_gnu_pltenter
|
|
|
|
#define ARCH_LA_PLTEXIT x86_64_gnu_pltexit
|
|
|
|
|
2001-09-19 10:37:31 +00:00
|
|
|
#endif /* !dl_machine_h */
|
|
|
|
|
2005-01-06 22:40:27 +00:00
|
|
|
#ifdef RESOLVE_MAP
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
/* Perform the relocation specified by RELOC and SYM (which is fully resolved).
|
|
|
|
MAP is the object containing the reloc. */
|
|
|
|
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
static inline void __attribute__((always_inline))
|
|
|
|
elf_machine_rela(struct link_map *map, struct r_scope_elem *scope[],
|
|
|
|
const ElfW(Rela) *reloc, const ElfW(Sym) *sym,
|
|
|
|
const struct r_found_version *version,
|
|
|
|
void *const reloc_addr_arg, int skip_ifunc) {
|
2012-05-08 16:39:58 +00:00
|
|
|
ElfW(Addr) *const reloc_addr = reloc_addr_arg;
|
|
|
|
const unsigned long int r_type = ELFW(R_TYPE) (reloc->r_info);
|
2001-09-19 10:37:31 +00:00
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
# if !defined RTLD_BOOTSTRAP || !defined HAVE_Z_COMBRELOC
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (r_type == R_X86_64_RELATIVE))
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
# if !defined RTLD_BOOTSTRAP && !defined HAVE_Z_COMBRELOC
|
2001-09-19 10:37:31 +00:00
|
|
|
/* This is defined in rtld.c, but nowhere in the static libc.a;
|
|
|
|
make the reference weak so static programs can still link.
|
|
|
|
This declaration cannot be done when compiling rtld.c
|
|
|
|
(i.e. #ifdef RTLD_BOOTSTRAP) because rtld.c contains the
|
|
|
|
common defn for _dl_rtld_map, which is incompatible with a
|
|
|
|
weak decl in the same file. */
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef SHARED
|
2002-02-01 01:33:04 +00:00
|
|
|
weak_extern (GL(dl_rtld_map));
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-02-01 01:33:04 +00:00
|
|
|
if (map != &GL(dl_rtld_map)) /* Already done in rtld itself. */
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2001-09-19 10:37:31 +00:00
|
|
|
*reloc_addr = map->l_addr + reloc->r_addend;
|
|
|
|
}
|
|
|
|
else
|
2012-05-11 00:05:06 +00:00
|
|
|
# endif
|
|
|
|
# if !defined RTLD_BOOTSTRAP
|
|
|
|
/* l_addr + r_addend may be > 0xffffffff and R_X86_64_RELATIVE64
|
|
|
|
relocation updates the whole 64-bit entry. */
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (r_type == R_X86_64_RELATIVE64))
|
2012-05-11 00:05:06 +00:00
|
|
|
*(Elf64_Addr *) reloc_addr = (Elf64_Addr) map->l_addr + reloc->r_addend;
|
|
|
|
else
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (r_type == R_X86_64_NONE))
|
2001-09-19 10:37:31 +00:00
|
|
|
return;
|
|
|
|
else
|
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
2012-05-08 16:39:58 +00:00
|
|
|
const ElfW(Sym) *const refsym = sym;
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
struct link_map *sym_map = RESOLVE_MAP (map, scope, &sym, version,
|
|
|
|
r_type);
|
elf: Unify symbol address run-time calculation [BZ #19818]
Wrap symbol address run-time calculation into a macro and use it
throughout, replacing inline calculations.
There are a couple of variants, most of them different in a functionally
insignificant way. Most calculations are right following RESOLVE_MAP,
at which point either the map or the symbol returned can be checked for
validity as the macro sets either both or neither. In some places both
the symbol and the map has to be checked however.
My initial implementation therefore always checked both, however that
resulted in code larger by as much as 0.3%, as many places know from
elsewhere that no check is needed. I have decided the size growth was
unacceptable.
Having looked closer I realized that it's the map that is the culprit.
Therefore I have modified LOOKUP_VALUE_ADDRESS to accept an additional
boolean argument telling it to access the map without checking it for
validity. This in turn has brought quite nice results, with new code
actually being smaller for i686, and MIPS o32, n32 and little-endian n64
targets, unchanged in size for x86-64 and, unusually, marginally larger
for big-endian MIPS n64, as follows:
i686:
text data bss dec hex filename
152255 4052 192 156499 26353 ld-2.27.9000-base.so
152159 4052 192 156403 262f3 ld-2.27.9000-elf-symbol-value.so
MIPS/o32/el:
text data bss dec hex filename
142906 4396 260 147562 2406a ld-2.27.9000-base.so
142890 4396 260 147546 2405a ld-2.27.9000-elf-symbol-value.so
MIPS/n32/el:
text data bss dec hex filename
142267 4404 260 146931 23df3 ld-2.27.9000-base.so
142171 4404 260 146835 23d93 ld-2.27.9000-elf-symbol-value.so
MIPS/n64/el:
text data bss dec hex filename
149835 7376 408 157619 267b3 ld-2.27.9000-base.so
149787 7376 408 157571 26783 ld-2.27.9000-elf-symbol-value.so
MIPS/o32/eb:
text data bss dec hex filename
142870 4396 260 147526 24046 ld-2.27.9000-base.so
142854 4396 260 147510 24036 ld-2.27.9000-elf-symbol-value.so
MIPS/n32/eb:
text data bss dec hex filename
142019 4404 260 146683 23cfb ld-2.27.9000-base.so
141923 4404 260 146587 23c9b ld-2.27.9000-elf-symbol-value.so
MIPS/n64/eb:
text data bss dec hex filename
149763 7376 408 157547 2676b ld-2.27.9000-base.so
149779 7376 408 157563 2677b ld-2.27.9000-elf-symbol-value.so
x86-64:
text data bss dec hex filename
148462 6452 400 155314 25eb2 ld-2.27.9000-base.so
148462 6452 400 155314 25eb2 ld-2.27.9000-elf-symbol-value.so
[BZ #19818]
* sysdeps/generic/ldsodefs.h (LOOKUP_VALUE_ADDRESS): Add `set'
parameter.
(SYMBOL_ADDRESS): New macro.
[!ELF_FUNCTION_PTR_IS_SPECIAL] (DL_SYMBOL_ADDRESS): Use
SYMBOL_ADDRESS for symbol address calculation.
* elf/dl-runtime.c (_dl_fixup): Likewise.
(_dl_profile_fixup): Likewise.
* elf/dl-symaddr.c (_dl_symbol_address): Likewise.
* elf/rtld.c (dl_main): Likewise.
* sysdeps/aarch64/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/alpha/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/arm/dl-machine.h (elf_machine_rel): Likewise.
(elf_machine_rela): Likewise.
* sysdeps/hppa/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/hppa/dl-symaddr.c (_dl_symbol_address): Likewise.
* sysdeps/i386/dl-machine.h (elf_machine_rel): Likewise.
(elf_machine_rela): Likewise.
* sysdeps/ia64/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/m68k/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/microblaze/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/mips/dl-machine.h (ELF_MACHINE_BEFORE_RTLD_RELOC):
Likewise.
(elf_machine_reloc): Likewise.
(elf_machine_got_rel): Likewise.
* sysdeps/mips/dl-trampoline.c (__dl_runtime_resolve): Likewise.
* sysdeps/nios2/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/powerpc/powerpc32/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/powerpc/powerpc64/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/riscv/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/sh/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/sparc/sparc32/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/sparc/sparc64/dl-machine.h (elf_machine_rela):
Likewise.
* sysdeps/tile/dl-machine.h (elf_machine_rela): Likewise.
* sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2018-04-04 22:09:37 +00:00
|
|
|
ElfW(Addr) value = SYMBOL_ADDRESS (sym_map, sym, true);
|
2001-09-19 10:37:31 +00:00
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
if (sym != NULL
|
2018-02-05 14:07:54 +00:00
|
|
|
&& __glibc_unlikely (ELFW(ST_TYPE) (sym->st_info) == STT_GNU_IFUNC)
|
|
|
|
&& __glibc_likely (sym->st_shndx != SHN_UNDEF)
|
|
|
|
&& __glibc_likely (!skip_ifunc))
|
2016-10-28 16:11:55 +00:00
|
|
|
{
|
|
|
|
# ifndef RTLD_BOOTSTRAP
|
|
|
|
if (sym_map != map
|
|
|
|
&& !sym_map->l_relocated)
|
|
|
|
{
|
|
|
|
const char *strtab
|
|
|
|
= (const char *) D_PTR (map, l_info[DT_STRTAB]);
|
2020-12-28 13:28:49 +00:00
|
|
|
if (sym_map->l_type == lt_executable)
|
|
|
|
_dl_fatal_printf ("\
|
|
|
|
%s: IFUNC symbol '%s' referenced in '%s' is defined in the executable \
|
|
|
|
and creates an unsatisfiable circular dependency.\n",
|
|
|
|
RTLD_PROGNAME, strtab + refsym->st_name,
|
|
|
|
map->l_name);
|
|
|
|
else
|
|
|
|
_dl_error_printf ("\
|
2016-10-28 16:11:55 +00:00
|
|
|
%s: Relink `%s' with `%s' for IFUNC symbol `%s'\n",
|
2020-12-28 13:28:49 +00:00
|
|
|
RTLD_PROGNAME, map->l_name,
|
|
|
|
sym_map->l_name,
|
|
|
|
strtab + refsym->st_name);
|
2016-10-28 16:11:55 +00:00
|
|
|
}
|
|
|
|
# endif
|
|
|
|
value = ((ElfW(Addr) (*) (void)) value) ();
|
|
|
|
}
|
2009-03-12 06:31:25 +00:00
|
|
|
|
2001-09-19 10:37:31 +00:00
|
|
|
switch (r_type)
|
|
|
|
{
|
2013-01-17 04:31:03 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
|
|
|
# ifdef __ILP32__
|
|
|
|
case R_X86_64_SIZE64:
|
|
|
|
/* Set to symbol size plus addend. */
|
|
|
|
*(Elf64_Addr *) (uintptr_t) reloc_addr
|
|
|
|
= (Elf64_Addr) sym->st_size + reloc->r_addend;
|
|
|
|
break;
|
|
|
|
|
|
|
|
case R_X86_64_SIZE32:
|
|
|
|
# else
|
|
|
|
case R_X86_64_SIZE64:
|
|
|
|
# endif
|
|
|
|
/* Set to symbol size plus addend. */
|
|
|
|
value = sym->st_size;
|
|
|
|
# endif
|
2019-02-12 10:30:34 +00:00
|
|
|
/* Fall through. */
|
2001-09-19 10:37:31 +00:00
|
|
|
case R_X86_64_GLOB_DAT:
|
|
|
|
case R_X86_64_JUMP_SLOT:
|
|
|
|
*reloc_addr = value + reloc->r_addend;
|
|
|
|
break;
|
2002-09-27 07:29:51 +00:00
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RESOLVE_CONFLICT_FIND_MAP
|
2002-09-27 07:29:51 +00:00
|
|
|
case R_X86_64_DTPMOD64:
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifdef RTLD_BOOTSTRAP
|
2002-09-27 07:29:51 +00:00
|
|
|
/* During startup the dynamic linker is always the module
|
|
|
|
with index 1.
|
|
|
|
XXX If this relocation is necessary move before RESOLVE
|
|
|
|
call. */
|
|
|
|
*reloc_addr = 1;
|
2009-03-12 06:31:25 +00:00
|
|
|
# else
|
2002-09-27 07:29:51 +00:00
|
|
|
/* Get the information from the link map returned by the
|
|
|
|
resolve function. */
|
|
|
|
if (sym_map != NULL)
|
|
|
|
*reloc_addr = sym_map->l_tls_modid;
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-09-27 07:29:51 +00:00
|
|
|
break;
|
|
|
|
case R_X86_64_DTPOFF64:
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
2002-09-27 07:29:51 +00:00
|
|
|
/* During relocation all TLS symbols are defined and used.
|
|
|
|
Therefore the offset is already correct. */
|
|
|
|
if (sym != NULL)
|
2012-05-10 17:05:40 +00:00
|
|
|
{
|
|
|
|
value = sym->st_value + reloc->r_addend;
|
|
|
|
# ifdef __ILP32__
|
|
|
|
/* This relocation type computes a signed offset that is
|
|
|
|
usually negative. The symbol and addend values are 32
|
|
|
|
bits but the GOT entry is 64 bits wide and the whole
|
|
|
|
64-bit entry is used as a signed quantity, so we need
|
|
|
|
to sign-extend the computed value to 64 bits. */
|
2012-05-10 17:20:51 +00:00
|
|
|
*(Elf64_Sxword *) reloc_addr = (Elf64_Sxword) (Elf32_Sword) value;
|
2012-05-10 17:05:40 +00:00
|
|
|
# else
|
|
|
|
*reloc_addr = value;
|
|
|
|
# endif
|
|
|
|
}
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-09-27 07:29:51 +00:00
|
|
|
break;
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
case R_X86_64_TLSDESC:
|
|
|
|
{
|
|
|
|
struct tlsdesc volatile *td =
|
|
|
|
(struct tlsdesc volatile *)reloc_addr;
|
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
if (! sym)
|
|
|
|
{
|
|
|
|
td->arg = (void*)reloc->r_addend;
|
|
|
|
td->entry = _dl_tlsdesc_undefweak;
|
|
|
|
}
|
|
|
|
else
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
|
|
|
# ifndef SHARED
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
CHECK_STATIC_TLS (map, sym_map);
|
2009-03-12 06:31:25 +00:00
|
|
|
# else
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
if (!TRY_STATIC_TLS (map, sym_map))
|
|
|
|
{
|
|
|
|
td->arg = _dl_make_tlsdesc_dynamic
|
|
|
|
(sym_map, sym->st_value + reloc->r_addend);
|
|
|
|
td->entry = _dl_tlsdesc_dynamic;
|
|
|
|
}
|
|
|
|
else
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
# endif
|
|
|
|
{
|
|
|
|
td->arg = (void*)(sym->st_value - sym_map->l_tls_offset
|
|
|
|
+ reloc->r_addend);
|
|
|
|
td->entry = _dl_tlsdesc_return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
2002-09-27 07:29:51 +00:00
|
|
|
case R_X86_64_TPOFF64:
|
|
|
|
/* The offset is negative, forward from the thread pointer. */
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
2002-09-27 07:29:51 +00:00
|
|
|
if (sym != NULL)
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-10-17 12:10:17 +00:00
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
2002-10-17 19:44:19 +00:00
|
|
|
CHECK_STATIC_TLS (map, sym_map);
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-10-17 12:10:17 +00:00
|
|
|
/* We know the offset of the object the symbol is contained in.
|
|
|
|
It is a negative value which will be added to the
|
|
|
|
thread pointer. */
|
2012-05-10 17:05:40 +00:00
|
|
|
value = (sym->st_value + reloc->r_addend
|
|
|
|
- sym_map->l_tls_offset);
|
|
|
|
# ifdef __ILP32__
|
|
|
|
/* The symbol and addend values are 32 bits but the GOT
|
|
|
|
entry is 64 bits wide and the whole 64-bit entry is used
|
|
|
|
as a signed quantity, so we need to sign-extend the
|
|
|
|
computed value to 64 bits. */
|
|
|
|
*(Elf64_Sxword *) reloc_addr = (Elf64_Sxword) (Elf32_Sword) value;
|
|
|
|
# else
|
|
|
|
*reloc_addr = value;
|
|
|
|
# endif
|
2002-10-17 12:10:17 +00:00
|
|
|
}
|
2002-09-27 07:29:51 +00:00
|
|
|
break;
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2002-09-27 07:29:51 +00:00
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RTLD_BOOTSTRAP
|
2001-09-19 10:37:31 +00:00
|
|
|
case R_X86_64_64:
|
2012-05-11 00:05:06 +00:00
|
|
|
/* value + r_addend may be > 0xffffffff and R_X86_64_64
|
|
|
|
relocation updates the whole 64-bit entry. */
|
|
|
|
*(Elf64_Addr *) reloc_addr = (Elf64_Addr) value + reloc->r_addend;
|
2001-09-19 10:37:31 +00:00
|
|
|
break;
|
2013-01-17 04:31:03 +00:00
|
|
|
# ifndef __ILP32__
|
|
|
|
case R_X86_64_SIZE32:
|
|
|
|
/* Set to symbol size plus addend. */
|
|
|
|
value = sym->st_size;
|
|
|
|
# endif
|
2019-02-12 10:30:34 +00:00
|
|
|
/* Fall through. */
|
2001-09-19 10:37:31 +00:00
|
|
|
case R_X86_64_32:
|
2009-03-12 06:31:25 +00:00
|
|
|
value += reloc->r_addend;
|
|
|
|
*(unsigned int *) reloc_addr = value;
|
|
|
|
|
|
|
|
const char *fmt;
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (value > UINT_MAX))
|
2002-06-25 12:32:52 +00:00
|
|
|
{
|
|
|
|
const char *strtab;
|
|
|
|
|
2009-03-12 06:31:25 +00:00
|
|
|
fmt = "\
|
|
|
|
%s: Symbol `%s' causes overflow in R_X86_64_32 relocation\n";
|
2009-04-26 20:12:37 +00:00
|
|
|
# ifndef RESOLVE_CONFLICT_FIND_MAP
|
2009-03-12 06:31:25 +00:00
|
|
|
print_err:
|
2009-04-26 20:12:37 +00:00
|
|
|
# endif
|
2002-06-25 12:32:52 +00:00
|
|
|
strtab = (const char *) D_PTR (map, l_info[DT_STRTAB]);
|
|
|
|
|
2013-05-29 16:00:20 +00:00
|
|
|
_dl_error_printf (fmt, RTLD_PROGNAME, strtab + refsym->st_name);
|
2002-06-25 12:32:52 +00:00
|
|
|
}
|
2001-09-19 10:37:31 +00:00
|
|
|
break;
|
2009-03-12 06:31:25 +00:00
|
|
|
# ifndef RESOLVE_CONFLICT_FIND_MAP
|
2003-04-24 17:11:08 +00:00
|
|
|
/* Not needed for dl-conflict.c. */
|
2001-09-19 10:37:31 +00:00
|
|
|
case R_X86_64_PC32:
|
2012-05-08 16:39:58 +00:00
|
|
|
value += reloc->r_addend - (ElfW(Addr)) reloc_addr;
|
2009-03-12 06:31:25 +00:00
|
|
|
*(unsigned int *) reloc_addr = value;
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (value != (int) value))
|
2002-06-25 12:32:52 +00:00
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
fmt = "\
|
|
|
|
%s: Symbol `%s' causes overflow in R_X86_64_PC32 relocation\n";
|
|
|
|
goto print_err;
|
2002-06-25 12:32:52 +00:00
|
|
|
}
|
2001-09-19 10:37:31 +00:00
|
|
|
break;
|
|
|
|
case R_X86_64_COPY:
|
|
|
|
if (sym == NULL)
|
|
|
|
/* This can happen in trace mode if an object could not be
|
|
|
|
found. */
|
|
|
|
break;
|
2009-03-12 06:31:25 +00:00
|
|
|
memcpy (reloc_addr_arg, (void *) value,
|
|
|
|
MIN (sym->st_size, refsym->st_size));
|
2018-02-05 14:07:54 +00:00
|
|
|
if (__glibc_unlikely (sym->st_size > refsym->st_size)
|
|
|
|
|| (__glibc_unlikely (sym->st_size < refsym->st_size)
|
2004-03-05 10:29:47 +00:00
|
|
|
&& GLRO(dl_verbose)))
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
2009-03-12 06:31:25 +00:00
|
|
|
fmt = "\
|
|
|
|
%s: Symbol `%s' has different size in shared object, consider re-linking\n";
|
|
|
|
goto print_err;
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
break;
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2009-06-01 06:45:33 +00:00
|
|
|
case R_X86_64_IRELATIVE:
|
|
|
|
value = map->l_addr + reloc->r_addend;
|
2019-11-29 16:55:22 +00:00
|
|
|
if (__glibc_likely (!skip_ifunc))
|
|
|
|
value = ((ElfW(Addr) (*) (void)) value) ();
|
2009-06-01 06:45:33 +00:00
|
|
|
*reloc_addr = value;
|
|
|
|
break;
|
2001-09-19 10:37:31 +00:00
|
|
|
default:
|
|
|
|
_dl_reloc_bad_type (map, r_type, 0);
|
|
|
|
break;
|
2009-03-12 06:31:25 +00:00
|
|
|
# endif
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
static inline void
|
2004-09-24 17:09:04 +00:00
|
|
|
__attribute ((always_inline))
|
2012-05-08 16:39:58 +00:00
|
|
|
elf_machine_rela_relative (ElfW(Addr) l_addr, const ElfW(Rela) *reloc,
|
* elf/dynamic-link.h (elf_machine_rel, elf_machine_rela, elf_machine_rel_relative, elf_machine_rela_relative): Don't assume reloc_addr is aligned. * sysdeps/alpha/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/arm/dl-machine.h (elf_machine_rel, elf_machine_rela, elf_machine_rel_relative, elf_machine_rela_relative): Adjust. * sysdeps/cris/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/hppa/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/i386/dl-machine.h (elf_machine_rel, elf_machine_rela, elf_machine_rel_relative, elf_machine_rela_relative): Adjust. * sysdeps/ia64/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/m68k/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/mips/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/powerpc/powerpc32/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/powerpc/powerpc64/dl-machine.h (elf_machine_rela_relative, elf_machine_rela): Adjust. * sysdeps/s390/s390-32/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/s390/s390-64/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): * sysdeps/sh/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/sparc/sparc32/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/sparc/sparc64/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust. * sysdeps/x86_64/dl-machine.h (elf_machine_rela, elf_machine_rela_relative): Adjust.
2003-07-31 Alexandre Oliva <aoliva@redhat.com>
* elf/dynamic-link.h (elf_machine_rel, elf_machine_rela,
elf_machine_rel_relative, elf_machine_rela_relative): Don't assume
reloc_addr is aligned.
* sysdeps/alpha/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/arm/dl-machine.h (elf_machine_rel, elf_machine_rela,
elf_machine_rel_relative, elf_machine_rela_relative): Adjust.
* sysdeps/cris/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/hppa/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/i386/dl-machine.h (elf_machine_rel, elf_machine_rela,
elf_machine_rel_relative, elf_machine_rela_relative): Adjust.
* sysdeps/ia64/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/m68k/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/mips/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/powerpc/powerpc32/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/powerpc/powerpc64/dl-machine.h
(elf_machine_rela_relative, elf_machine_rela): Adjust.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative):
* sysdeps/sh/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/sparc/sparc32/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/sparc/sparc64/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
* sysdeps/x86_64/dl-machine.h (elf_machine_rela,
elf_machine_rela_relative): Adjust.
2003-07-31 06:33:53 +00:00
|
|
|
void *const reloc_addr_arg)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
2012-05-08 16:39:58 +00:00
|
|
|
ElfW(Addr) *const reloc_addr = reloc_addr_arg;
|
2012-05-11 02:24:54 +00:00
|
|
|
#if !defined RTLD_BOOTSTRAP
|
2012-05-11 00:05:06 +00:00
|
|
|
/* l_addr + r_addend may be > 0xffffffff and R_X86_64_RELATIVE64
|
|
|
|
relocation updates the whole 64-bit entry. */
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_unlikely (ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE64))
|
2012-05-11 00:05:06 +00:00
|
|
|
*(Elf64_Addr *) reloc_addr = (Elf64_Addr) l_addr + reloc->r_addend;
|
|
|
|
else
|
2012-05-11 02:24:54 +00:00
|
|
|
#endif
|
2012-05-11 00:05:06 +00:00
|
|
|
{
|
|
|
|
assert (ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE);
|
|
|
|
*reloc_addr = l_addr + reloc->r_addend;
|
|
|
|
}
|
2001-09-19 10:37:31 +00:00
|
|
|
}
|
|
|
|
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
static inline void
|
2004-09-24 17:09:04 +00:00
|
|
|
__attribute ((always_inline))
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
elf_machine_lazy_rel (struct link_map *map, struct r_scope_elem *scope[],
|
2012-05-08 16:39:58 +00:00
|
|
|
ElfW(Addr) l_addr, const ElfW(Rela) *reloc,
|
2011-10-04 14:10:16 +00:00
|
|
|
int skip_ifunc)
|
2001-09-19 10:37:31 +00:00
|
|
|
{
|
2012-05-08 16:39:58 +00:00
|
|
|
ElfW(Addr) *const reloc_addr = (void *) (l_addr + reloc->r_offset);
|
|
|
|
const unsigned long int r_type = ELFW(R_TYPE) (reloc->r_info);
|
2001-09-19 10:37:31 +00:00
|
|
|
|
|
|
|
/* Check for unexpected PLT reloc type. */
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_likely (r_type == R_X86_64_JUMP_SLOT))
|
Update.
2001-12-11 Jakub Jelinek <jakub@redhat.com>
* elf/Makefile (dl-routines): Add conflict.
(rtld-ldscript-in, rtld-ldscript, rtld-parms): Remove.
(ld.so): Add _begin local symbol.
* elf/elf.h (DT_VALTAGIDX, DT_VALNUM, DT_ADDRTAGIDX, DT_ADDRNUM):
Define.
* elf/dl-deps.c (_dl_build_local_scope): New.
(_dl_map_object_deps): If LD_TRACE_PRELINKING, compute local scopes
of all libraries.
* elf/do-rel.h (VALIDX): Define.
(elf_dynamic_do_rel): If ELF_MACHINE_PLT_REL is defined, don't do
lazy binding for RELA. If DT_GNU_PRELINKED, DT_RELACOUNT relocations
can be skipped.
* elf/dl-conflict.c: New file.
* elf/dl-lookup.c (_dl_debug_bindings): New.
(_dl_lookup_symbol): Use _dl_debug_bindings. Reference_name is always
non-NULL.
(_dl_lookup_symbol_skip): Likewise.
(_dl_lookup_versioned_symbol): Likewise.
(_dl_lookup_versioned_symbol_skip): Likewise.
* elf/dl-runtime.c (PLTREL): If ELF_MACHINE_PLT_REL is defined,
define to ElfW(Rel).
* elf/dynamic-link.h (elf_get_dynamic_info): Record selected dynamic
tags in the DT_VALRNGLO..DT_VALRNGHI and DT_ADDRRNGLO..DT_ADDRRNGHI
ranges.
Don't adjust address dynamic tags if l_addr is 0.
* elf/rtld.c (_dl_trace_prelink, _dl_trace_prelink_map): New variables.
(_dl_start): Skip ELF_DYNAMIC_RELOCATE if ld.so is prelinked.
(VALIDX, ADDRIDX): Define.
(_dl_start_final): Initialize _dl_rtld_map's l_map_start and l_map_end.
(dl_main): Print library list for LD_TRACE_PRELINKING.
If prelinking information can be used, skip relocating libraries and
call _dl_resolve_conflicts instead.
(process_envvars): Handle LD_TRACE_PRELINKING envvar.
* elf/dl-load.c (_dl_map_object): Don't create fake libs
if LD_TRACE_PRELINKING.
* include/link.h (struct link_map) [l_info]: Add DT_VALNUM
+ DT_ADDRNUM.
* sysdeps/generic/ldsodefs.h (_dl_trace_prelink_map): New declaration.
(DL_DEBUG_PRELINK): Define.
(_dl_resolve_conflicts): Add prototype.
* sysdeps/alpha/dl-machine.h (elf_machine_runtime_setup): Reinitialize
.plt for prelinked libraries where prelinking info cannot be used.
(elf_machine_rela): If relocating R_ALPHA_JMP_SLOT in .gnu.conflict
section, use RESOLVE_CONFLICT_FIND_MAP to find out reloc's link_map.
* sysdeps/arm/bits/link.h: New file.
* sysdeps/arm/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_ARM_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/i386/bits/link.h: New file.
* sysdeps/i386/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_386_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/powerpc/dl-machine.h (elf_machine_rela): If relocating
conflicts, skip finaladdr computation. Use RESOLVE_CONFLICT_FIND_MAP
to find out map for R_PPC_JMP_SLOT relocs.
* sysdeps/sparc/sparc32/dl-machine.h (VALIDX): Define.
(OPCODE_BA): Define.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
(sparc_fixup_plt): Renamed from elf_machine_fixup_plt.
(elf_machine_fixup_plt): Call sparc_fixup_plt.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Call sparc_fixup_plt for R_SPARC_JMP_SLOT.
* sysdeps/sparc/sparc64/dl-machine.h (VALIDX): Define.
(sparc64_fixup_plt): Fix a typo.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Handle R_SPARC_JMP_SLOT relocs when relocating conflicts.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
* sysdeps/sh/bits/link.h: New file.
* sysdeps/sh/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_SH_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-32/bits/link.h: New file.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-64/bits/link.h: New file.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/x86_64/bits/link.h: New file.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_X86_64_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
2001-12-12 00:21:26 +00:00
|
|
|
{
|
2018-02-05 14:07:54 +00:00
|
|
|
/* Prelink has been deprecated. */
|
|
|
|
if (__glibc_likely (map->l_mach.plt == 0))
|
Update.
2001-12-11 Jakub Jelinek <jakub@redhat.com>
* elf/Makefile (dl-routines): Add conflict.
(rtld-ldscript-in, rtld-ldscript, rtld-parms): Remove.
(ld.so): Add _begin local symbol.
* elf/elf.h (DT_VALTAGIDX, DT_VALNUM, DT_ADDRTAGIDX, DT_ADDRNUM):
Define.
* elf/dl-deps.c (_dl_build_local_scope): New.
(_dl_map_object_deps): If LD_TRACE_PRELINKING, compute local scopes
of all libraries.
* elf/do-rel.h (VALIDX): Define.
(elf_dynamic_do_rel): If ELF_MACHINE_PLT_REL is defined, don't do
lazy binding for RELA. If DT_GNU_PRELINKED, DT_RELACOUNT relocations
can be skipped.
* elf/dl-conflict.c: New file.
* elf/dl-lookup.c (_dl_debug_bindings): New.
(_dl_lookup_symbol): Use _dl_debug_bindings. Reference_name is always
non-NULL.
(_dl_lookup_symbol_skip): Likewise.
(_dl_lookup_versioned_symbol): Likewise.
(_dl_lookup_versioned_symbol_skip): Likewise.
* elf/dl-runtime.c (PLTREL): If ELF_MACHINE_PLT_REL is defined,
define to ElfW(Rel).
* elf/dynamic-link.h (elf_get_dynamic_info): Record selected dynamic
tags in the DT_VALRNGLO..DT_VALRNGHI and DT_ADDRRNGLO..DT_ADDRRNGHI
ranges.
Don't adjust address dynamic tags if l_addr is 0.
* elf/rtld.c (_dl_trace_prelink, _dl_trace_prelink_map): New variables.
(_dl_start): Skip ELF_DYNAMIC_RELOCATE if ld.so is prelinked.
(VALIDX, ADDRIDX): Define.
(_dl_start_final): Initialize _dl_rtld_map's l_map_start and l_map_end.
(dl_main): Print library list for LD_TRACE_PRELINKING.
If prelinking information can be used, skip relocating libraries and
call _dl_resolve_conflicts instead.
(process_envvars): Handle LD_TRACE_PRELINKING envvar.
* elf/dl-load.c (_dl_map_object): Don't create fake libs
if LD_TRACE_PRELINKING.
* include/link.h (struct link_map) [l_info]: Add DT_VALNUM
+ DT_ADDRNUM.
* sysdeps/generic/ldsodefs.h (_dl_trace_prelink_map): New declaration.
(DL_DEBUG_PRELINK): Define.
(_dl_resolve_conflicts): Add prototype.
* sysdeps/alpha/dl-machine.h (elf_machine_runtime_setup): Reinitialize
.plt for prelinked libraries where prelinking info cannot be used.
(elf_machine_rela): If relocating R_ALPHA_JMP_SLOT in .gnu.conflict
section, use RESOLVE_CONFLICT_FIND_MAP to find out reloc's link_map.
* sysdeps/arm/bits/link.h: New file.
* sysdeps/arm/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_ARM_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/i386/bits/link.h: New file.
* sysdeps/i386/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_386_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/powerpc/dl-machine.h (elf_machine_rela): If relocating
conflicts, skip finaladdr computation. Use RESOLVE_CONFLICT_FIND_MAP
to find out map for R_PPC_JMP_SLOT relocs.
* sysdeps/sparc/sparc32/dl-machine.h (VALIDX): Define.
(OPCODE_BA): Define.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
(sparc_fixup_plt): Renamed from elf_machine_fixup_plt.
(elf_machine_fixup_plt): Call sparc_fixup_plt.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Call sparc_fixup_plt for R_SPARC_JMP_SLOT.
* sysdeps/sparc/sparc64/dl-machine.h (VALIDX): Define.
(sparc64_fixup_plt): Fix a typo.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Handle R_SPARC_JMP_SLOT relocs when relocating conflicts.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
* sysdeps/sh/bits/link.h: New file.
* sysdeps/sh/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_SH_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-32/bits/link.h: New file.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-64/bits/link.h: New file.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/x86_64/bits/link.h: New file.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_X86_64_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
2001-12-12 00:21:26 +00:00
|
|
|
*reloc_addr += l_addr;
|
|
|
|
else
|
|
|
|
*reloc_addr =
|
|
|
|
map->l_mach.plt
|
2012-05-08 16:39:58 +00:00
|
|
|
+ (((ElfW(Addr)) reloc_addr) - map->l_mach.gotplt) * 2;
|
Update.
2001-12-11 Jakub Jelinek <jakub@redhat.com>
* elf/Makefile (dl-routines): Add conflict.
(rtld-ldscript-in, rtld-ldscript, rtld-parms): Remove.
(ld.so): Add _begin local symbol.
* elf/elf.h (DT_VALTAGIDX, DT_VALNUM, DT_ADDRTAGIDX, DT_ADDRNUM):
Define.
* elf/dl-deps.c (_dl_build_local_scope): New.
(_dl_map_object_deps): If LD_TRACE_PRELINKING, compute local scopes
of all libraries.
* elf/do-rel.h (VALIDX): Define.
(elf_dynamic_do_rel): If ELF_MACHINE_PLT_REL is defined, don't do
lazy binding for RELA. If DT_GNU_PRELINKED, DT_RELACOUNT relocations
can be skipped.
* elf/dl-conflict.c: New file.
* elf/dl-lookup.c (_dl_debug_bindings): New.
(_dl_lookup_symbol): Use _dl_debug_bindings. Reference_name is always
non-NULL.
(_dl_lookup_symbol_skip): Likewise.
(_dl_lookup_versioned_symbol): Likewise.
(_dl_lookup_versioned_symbol_skip): Likewise.
* elf/dl-runtime.c (PLTREL): If ELF_MACHINE_PLT_REL is defined,
define to ElfW(Rel).
* elf/dynamic-link.h (elf_get_dynamic_info): Record selected dynamic
tags in the DT_VALRNGLO..DT_VALRNGHI and DT_ADDRRNGLO..DT_ADDRRNGHI
ranges.
Don't adjust address dynamic tags if l_addr is 0.
* elf/rtld.c (_dl_trace_prelink, _dl_trace_prelink_map): New variables.
(_dl_start): Skip ELF_DYNAMIC_RELOCATE if ld.so is prelinked.
(VALIDX, ADDRIDX): Define.
(_dl_start_final): Initialize _dl_rtld_map's l_map_start and l_map_end.
(dl_main): Print library list for LD_TRACE_PRELINKING.
If prelinking information can be used, skip relocating libraries and
call _dl_resolve_conflicts instead.
(process_envvars): Handle LD_TRACE_PRELINKING envvar.
* elf/dl-load.c (_dl_map_object): Don't create fake libs
if LD_TRACE_PRELINKING.
* include/link.h (struct link_map) [l_info]: Add DT_VALNUM
+ DT_ADDRNUM.
* sysdeps/generic/ldsodefs.h (_dl_trace_prelink_map): New declaration.
(DL_DEBUG_PRELINK): Define.
(_dl_resolve_conflicts): Add prototype.
* sysdeps/alpha/dl-machine.h (elf_machine_runtime_setup): Reinitialize
.plt for prelinked libraries where prelinking info cannot be used.
(elf_machine_rela): If relocating R_ALPHA_JMP_SLOT in .gnu.conflict
section, use RESOLVE_CONFLICT_FIND_MAP to find out reloc's link_map.
* sysdeps/arm/bits/link.h: New file.
* sysdeps/arm/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_ARM_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/i386/bits/link.h: New file.
* sysdeps/i386/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(ELF_MACHINE_NO_RELA): Only define if RTLD_BOOTSTRAP.
(ELF_MACHINE_PLT_REL): Define.
(elf_machine_rela, elf_machine_rela_relative): New.
(elf_machine_lazy_rel): Reinitialize R_386_JUMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/powerpc/dl-machine.h (elf_machine_rela): If relocating
conflicts, skip finaladdr computation. Use RESOLVE_CONFLICT_FIND_MAP
to find out map for R_PPC_JMP_SLOT relocs.
* sysdeps/sparc/sparc32/dl-machine.h (VALIDX): Define.
(OPCODE_BA): Define.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
(sparc_fixup_plt): Renamed from elf_machine_fixup_plt.
(elf_machine_fixup_plt): Call sparc_fixup_plt.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Call sparc_fixup_plt for R_SPARC_JMP_SLOT.
* sysdeps/sparc/sparc64/dl-machine.h (VALIDX): Define.
(sparc64_fixup_plt): Fix a typo.
(elf_machine_rela): Set value to 0 if relocating conflicts.
Handle R_SPARC_JMP_SLOT relocs when relocating conflicts.
(elf_machine_runtime_setup): Reinitialize .plt for prelinked
libraries where prelinking info cannot be used.
* sysdeps/sh/bits/link.h: New file.
* sysdeps/sh/dl-machine.h (elf_machine_runtime_setup): Save original
content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_SH_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-32/bits/link.h: New file.
* sysdeps/s390/s390-32/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/s390/s390-64/bits/link.h: New file.
* sysdeps/s390/s390-64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_390_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
* sysdeps/x86_64/bits/link.h: New file.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Save original content of .got[1].
(elf_machine_lazy_rel): Reinitialize R_X86_64_JMP_SLOT address instead
of adjusting it if prelinked and prelinking cannot be used.
2001-12-12 00:21:26 +00:00
|
|
|
}
|
2014-02-10 13:45:42 +00:00
|
|
|
else if (__glibc_likely (r_type == R_X86_64_TLSDESC))
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
{
|
2021-02-11 11:29:23 +00:00
|
|
|
const Elf_Symndx symndx = ELFW (R_SYM) (reloc->r_info);
|
|
|
|
const ElfW (Sym) *symtab = (const void *)D_PTR (map, l_info[DT_SYMTAB]);
|
|
|
|
const ElfW (Sym) *sym = &symtab[symndx];
|
|
|
|
const struct r_found_version *version = NULL;
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
|
2021-02-11 11:29:23 +00:00
|
|
|
if (map->l_info[VERSYMIDX (DT_VERSYM)] != NULL)
|
|
|
|
{
|
|
|
|
const ElfW (Half) *vernum =
|
|
|
|
(const void *)D_PTR (map, l_info[VERSYMIDX (DT_VERSYM)]);
|
|
|
|
version = &map->l_versions[vernum[symndx] & 0x7fff];
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Always initialize TLS descriptors completely at load time, in
|
|
|
|
case static TLS is allocated for it that requires locking. */
|
elf: Avoid nested functions in the loader [BZ #27220]
dynamic-link.h is included more than once in some elf/ files (rtld.c,
dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested
functions. This harms readability and the nested functions usage
is the biggest obstacle prevents Clang build (Clang doesn't support GCC
nested functions).
The key idea for unnesting is to add extra parameters (struct link_map
*and struct r_scope_elm *[]) to RESOLVE_MAP,
ELF_MACHINE_BEFORE_RTLD_RELOC, ELF_DYNAMIC_RELOCATE, elf_machine_rel[a],
elf_machine_lazy_rel, and elf_machine_runtime_setup. (This is inspired
by Stan Shebs' ppc64/x86-64 implementation in the
google/grte/v5-2.27/master which uses mixed extra parameters and static
variables.)
Future simplification:
* If mips elf_machine_runtime_setup no longer needs RESOLVE_GOTSYM,
elf_machine_runtime_setup can drop the `scope` parameter.
* If TLSDESC no longer need to be in elf_machine_lazy_rel,
elf_machine_lazy_rel can drop the `scope` parameter.
Tested on aarch64, i386, x86-64, powerpc64le, powerpc64, powerpc32,
sparc64, sparcv9, s390x, s390, hppa, ia64, armhf, alpha, and mips64.
In addition, tested build-many-glibcs.py with {arc,csky,microblaze,nios2}-linux-gnu
and riscv64-linux-gnu-rv64imafdc-lp64d.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 490e6c62aa31a8aa5c4a059f6e646ede121edf0a)
2021-10-07 18:55:02 +00:00
|
|
|
elf_machine_rela (map, scope, reloc, sym, version, reloc_addr, skip_ifunc);
|
Introduce TLS descriptors for i386 and x86_64.
* include/inline-hashtab.h: New file, copied from 2005's
libiberty, with fix for memory leak imported afterwards by
Glauber de Oliveira Costa.
* elf/tlsdeschtab.h: New file.
* elf/dl-reloc.c (_dl_try_allocate_static_tls): Extract from...
(_dl_allocate_static_tls): ... here. Rearrange failure path.
(CHECK_STATIC_TLS): Move to...
* elf/dynamic-link.h: ... this file.
(TRY_STATIC_TLS): New macro.
* elf/dl-conflict.c (CHECK_STATIC_TLS, TRY_STATIC_TLS): Override.
* elf/elf.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL,
R_386_TLS_DESC): Define.
(R_X86_64_PC64, R_X86_GOTOFF64, R_X86_64_GOTPC32): Merge from
binutils.
(R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL,
R_X86_64_TLSDESC): Define.
(R_386_NUM, R_X86_64_NUM): Adjust.
* sysdeps/i386/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/i386/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/i386/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_type_class): Mark R_386_TLS_DESC as PLT class.
(elf_machine_rel): Handle R_386_TLS_DESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
(elf_machine_lazy_rela): Likewise.
* sysdeps/i386/dl-tls.h (struct dl_tls_index): Name it.
* sysdeps/i386/dl-tlsdesc.S: New file.
* sysdeps/i386/dl-tlsdesc.h: New file.
* sysdeps/i386/tlsdesc.c: New file.
* sysdeps/i386/tlsdesc.sym: New file.
* sysdeps/i386/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table.
* sysdeps/x86_64/Makefile (sysdep-dl-routines, sysdep_routines,
systep-rtld-routines): Add tlsdesc and dl-tlsdesc for elf subdir.
(gen-as-const-headers): Add tlsdesc.sym to csu subdir.
* sysdeps/x86_64/dl-lookupcfg.h: New file. Introduce _dl_unmap to
release tlsdesc_table.
* sysdeps/x86_64/dl-machine.h: Include dl-tlsdesc.h.
(elf_machine_runtime_setup): Set up lazy TLSDESC GOT entry.
(elf_machine_type_class): Mark R_X86_64_TLSDESC as PLT class.
(elf_machine_rel): Handle R_X86_64_TLSDESC.
(elf_machine_rela): Likewise.
(elf_machine_lazy_rel): Likewise.
* sysdeps/x86_64/dl-tls.h (struct dl_tls_index): Name it.
(__tls_get_addr): Do not declare for non-shared compiles.
* sysdeps/x86_64/dl-tlsdesc.S: New file.
* sysdeps/x86_64/dl-tlsdesc.h: New file.
* sysdeps/x86_64/tlsdesc.c: New file.
* sysdeps/x86_64/tlsdesc.sym: New file.
* sysdeps/x86_64/bits/linkmap.h (struct link_map_machine): Add
tlsdesc_table for both 32- and 64-bit structs.
2008-05-13 05:41:30 +00:00
|
|
|
}
|
2014-02-10 13:45:42 +00:00
|
|
|
else if (__glibc_unlikely (r_type == R_X86_64_IRELATIVE))
|
2009-06-01 06:45:33 +00:00
|
|
|
{
|
2012-05-08 16:39:58 +00:00
|
|
|
ElfW(Addr) value = map->l_addr + reloc->r_addend;
|
2014-02-10 13:45:42 +00:00
|
|
|
if (__glibc_likely (!skip_ifunc))
|
2012-05-08 16:39:58 +00:00
|
|
|
value = ((ElfW(Addr) (*) (void)) value) ();
|
2009-06-01 06:45:33 +00:00
|
|
|
*reloc_addr = value;
|
|
|
|
}
|
2001-09-19 10:37:31 +00:00
|
|
|
else
|
|
|
|
_dl_reloc_bad_type (map, r_type, 1);
|
|
|
|
}
|
|
|
|
|
2005-01-06 22:40:27 +00:00
|
|
|
#endif /* RESOLVE_MAP */
|