Linux vDSO initialization code the internal function pointers require a
lot of duplicated boilerplate over different architectures. This patch
aims to simplify not only the code but the required definition to enable
a vDSO symbol.
The changes are:
1. Consolidate all init-first.c on only one implementation and enable
the symbol based on HAVE_*_VSYSCALL existence.
2. Set the HAVE_*_VSYSCALL to the architecture expected names string.
3. Add a new internal implementation, get_vdso_mangle_symbol, which
returns a mangled function pointer.
Currently the clock_gettime, clock_getres, gettimeofday, getcpu, and time
are handled in an arch-independent way, powerpc still uses some
arch-specific vDSO symbol handled in a specific init-first implementation.
Checked on aarch64-linux-gnu, arm-linux-gnueabihf, i386-linux-gnu,
mips64-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu,
sparc64-linux-gnu, and x86_64-linux-gnu.
* sysdeps/powerpc/powerpc32/backtrace.c (is_sigtramp_address,
is_sigtramp_address_rt): Use HAVE_SIGTRAMP_{RT}32 instead of SHARED.
* sysdeps/powerpc/powerpc64/backtrace.c (is_sigtramp_address):
Likewise.
* sysdeps/unix/sysv/linux/aarch64/init-first.c: Remove file.
* sysdeps/unix/sysv/linux/aarch64/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/arm/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/arm/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/mips/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/mips/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/i386/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/riscv/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/s390/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/s390/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/sparc/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/aarch64/sysdep.h
(HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Define value based on kernel exported
name.
* sysdeps/unix/sysv/linux/arm/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/mips/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/sysdep.h
(HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETCPU_VSYSCALL, HAVE_TIME_VSYSCALL, HAVE_GET_TBFREQ,
HAVE_SIGTRAMP_RT64, HAVE_SIGTRAMP_32, HAVE_SIGTRAMP_RT32i,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/riscv/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL,
HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/s390/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL,
HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/dl-vdso.h (VDSO_NAME, VDSO_HASH): Define to
invalid names if architecture does not define them.
(get_vdso_mangle_symbol): New symbol.
* sysdeps/unix/sysv/linux/init-first.c: New file.
* sysdeps/unix/sysv/linux/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/init-first.c (gettimeofday,
clock_gettime, clock_getres, getcpu, time): Remove declaration.
(__libc_vdso_platform_setup_arch): Likewise and use
get_vdso_mangle_symbol to setup vDSO symbols.
(sigtramp_rt64, sigtramp32, sigtramp_rt32, get_tbfreq): Add
attribute_hidden.
* sysdeps/unix/sysv/linux/powerpc/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sysdep-vdso.h (VDSO_SYMBOL): Remove
definition.
This patch assumes static vDSO is supported as default, it is now supported
on all current architectures that support vDSO. It allows removing both
ALWAYS_USE_VSYSCALL define, which an architecture requires to explicit define
and USE_VSYSCALL (which defines vDSO only for shared or if architecture defines
ALWAYS_USE_VSYSCALL).
Checked with a build against all affected ABIs.
[BZ #19767]
* sysdeps/unix/sysv/linux/aarch64/sysdep.h (ALWAYS_USE_VSYSCALL):
Remove definition.
* sysdeps/unix/sysv/linux/arm/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/riscv/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Remove #if USE_VSYSCALL.
* sysdeps/unix/sysv/linux/sysdep-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sysdep.h (ALWAYS_USE_VSYSCALL,
USE_VSYSCALL): Remove defitions.
All the required code already existed, and some of it was already
running.
AT_SYSINFO_EHDR is processed if NEED_DL_SYSINFO_DSO is defined, but it
looks like it always is. The call to setup_vdso is also unconditional,
so all that was left to do was setup the function pointers and use
them. This patch just deletes some #ifdef to enable that.
[BZ #19767]
* nptl/Makefile (tests-static): Add tst-cond11-static.
(tests): Likewise.
* nptl/tst-cond11-static.c: New File.
* sysdeps/unix/sysv/linux/Makefile (tests-static): Add
tst-affinity-static.
(tests): Likewise.
* sysdeps/unix/sysv/linux/sysdep-vdso.h: Check USE_VSYSCALL
instead of SHARED.
* sysdeps/unix/sysv/linux/sysdep.h (ALWAYS_USE_VSYSCALL): New.
(USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/tst-affinity-static.c: New file.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Check USE_VSYSCALL
instead of SHARED.
* sysdeps/unix/sysv/linux/x86_64/init-first.c: Don't check
SHARED.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (ALWAYS_USE_VSYSCALL):
New.
This patch adds a field to ucontext_t to save shadow stack:
1. getcontext and swapcontext are updated to save the caller's shadow
stack pointer and return addresses.
2. setcontext and swapcontext are updated to restore shadow stack and
jump to new context directly.
3. makecontext is updated to allocate a new shadow stack and set the
caller's return address to __start_context.
Since makecontext allocates a new shadow stack when making a new
context and kernel allocates a new shadow stack for clone/fork/vfork
syscalls, we track the current shadow stack base. In setcontext and
swapcontext, if the target shadow stack base is the same as the current
shadow stack base, we unwind the shadow stack. Otherwise it is a stack
switch and we look for a restore token.
We enable shadow stack at run-time only if program and all used shared
objects, including dlopened ones, are shadow stack enabled, which means
that they must be compiled with GCC 8 or above and glibc 2.28 or above.
We need to save and restore shadow stack only if shadow stack is enabled.
When caller of getcontext, setcontext, swapcontext and makecontext is
compiled with smaller ucontext_t, shadow stack won't be enabled at
run-time. We check if shadow stack is enabled before accessing the
extended field in ucontext_t.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* sysdeps/unix/sysv/linux/x86/sys/ucontext.h (ucontext_t): Add
__ssp.
* sysdeps/unix/sysv/linux/x86_64/__start_context.S: Include
<asm/prctl.h> and "ucontext_i.h" when shadow stack is enabled.
(__push___start_context): New.
* sysdeps/unix/sysv/linux/x86_64/getcontext.S: Include
<asm/prctl.h>.
(__getcontext): Record the current shadow stack base. Save the
caller's shadow stack pointer and base.
* sysdeps/unix/sysv/linux/x86_64/makecontext.c: Include
<pthread.h>, <libc-pointer-arith.h> and <sys/prctl.h>.
(__push___start_context): New prototype.
(__makecontext): Call __push___start_context to allocate a new
shadow stack, push __start_context onto the new stack as well
as the new shadow stack.
* sysdeps/unix/sysv/linux/x86_64/setcontext.S: Include
<asm/prctl.h>.
(__setcontext): Restore the target shadow stack.
* sysdeps/unix/sysv/linux/x86_64/swapcontext.S: Include
<asm/prctl.h>.
(__swapcontext): Record the current shadow stack base. Save
the caller's shadow stack pointer and base. Restore the target
shadow stack.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(STACK_SIZE_TO_SHADOW_STACK_SIZE_SHIFT): New.
* sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym (oSSP): New.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h: Always include
<dl-sysdep.h>. Test for value of RTLD_PRIVATE_ERRNO instead of
testing whether it is defined.
Current GLIBC has two ways to implement the single thread optimization
on syscalls to avoid calling the cancellation path: either by using
global variables (__{libc,pthread}_multiple_thread) or by accessing
the TCB field (defined by TLS_MULTIPLE_THREADS_IN_TCB). Both the
variables and the macros to acces its value are defined in the
architecture sysdep-cancel.h header.
This patch consolidates its definition on only one header,
sysdeps/unix/sysv/linux/sysdep-cancel.h, and adds a new define
(SINGLE_THREAD_BY_GLOBAL) which the architecture defines if it prefer
to use the global variables instead of the TCB field. This is an
optimization, so if the architecture does not define it, the TCB
method will be used as default.
Checked on x86_64-linux-gnu and on a build with major touched
ABIs (aarch64-linux-gnu, alpha-linux-gnu, arm-linux-gnueabihf,
hppa-linux-gnu, i686-linux-gnu, m68k-linux-gnu, microblaze-linux-gnu,
mips-linux-gnu, mips64-linux-gnu, powerpc-linux-gnu,
powerpc64le-linux-gnu, s390-linux-gnu, s390x-linux-gnu, sh4-linux-gnu,
sparcv9-linux-gnu, sparc64-linux-gnu, tilegx-linux-gnu).
* sysdeps/unix/sysv/linux/aarch64/sysdep-cancel.h: Remove file.
* sysdeps/unix/sysv/linux/alpha/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/arm/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/mips/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/nios2/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/sh/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/tile/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h
(SINGLE_THREAD_BY_GLOBAL): Define.
* sysdeps/unix/sysv/linux/aarch64/sysdep.h (SINGLE_THREAD_BY_GLOBAL):
Likewise.
* sysdeps/unix/sysv/linux/alpha/sysdep.h (SINGLE_THREAD_BY_GLOBAL):
Likewise.
* sysdeps/unix/sysv/linux/arm/sysdep.h (SINGLE_THREAD_BY_GLOBAL):
Likewise.
* sysdeps/unix/sysv/linux/hppa/sysdep.h (SINGLE_THREAD_BY_GLOBAL):
Likewise.
* sysdeps/unix/sysv/linux/microblaze/sysdep.h
(SINGLE_THREAD_BY_GLOBAL): Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (SINGLE_THREAD_BY_GLOBAL):
Likewise.
The problem for x32 is the {INTERNAL,INLINE}_SYSCALL C macros explicit
cast the arguments to 'long int', thus passing as 32 bits arguments
that should be passed to 64 bits.
Previous x32 implementation uses the auto-generated syscalls from
assembly macros (syscalls.list), so the {INTERNAL,INLINE}_SYSCALL
macros are never used with 64 bit argument in x32 (which are
internally broken for this ILP).
To fix it I used a strategy similar to MIPS64n32 (although both
ABI differs for some syscalls on how top pass 64-bits arguments)
where argument types for kernel call are defined using GCC extension
'typeof' with a arithmetic operation. This allows 64-bits arguments
to be defined while 32-bits argument will still passed as 32-bits.
I also cleanup the {INLINE,INTERNAL}_SYSCALL definition by defining
'inline_syscallX' instead of constructing the argument passing using
macros (it adds some readability) and removed the ununsed
INTERNAL_SYSCALL_NCS_TYPES define (since the patch idea is exactly to
avoid requiric explicit types passing).
Tested on x86_64 and x32.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(INTERNAL_SYSCALL_NCS_TYPES): Remove define.
(LOAD_ARGS_0): Likewise.
(LOAD_ARGS_1): Likewise.
(LOAD_ARGS_2): Likewise.
(LOAD_ARGS_3): Likewise.
(LOAD_ARGS_4): Likewise.
(LOAD_ARGS_5): Likewise.
(LOAD_ARGS_6): Likewise.
(LOAD_REGS_0): Likewise.
(LOAD_REGS_1): Likewise.
(LOAD_REGS_2): Likewise.
(LOAD_REGS_3): Likewise.
(LOAD_REGS_4): Likewise.
(LOAD_REGS_5): Likewise.
(LOAD_REGS_6): Likewise.
(ASM_ARGS_0): Likewise.
(ASM_ARGS_1): Likewise.
(ASM_ARGS_2): Likewise.
(ASM_ARGS_3): Likewise.
(ASM_ARGS_4): Likewise.
(ASM_ARGS_5): Likewise.
(ASM_ARGS_6): Likewise.
(LOAD_ARGS_TYPES_1): Likewise.
(LOAD_ARGS_TYPES_2): Likewise.
(LOAD_ARGS_TYPES_3): Likewise.
(LOAD_ARGS_TYPES_4): Likewise.
(LOAD_ARGS_TYPES_5): Likewise.
(LOAD_ARGS_TYPES_6): Likewise.
(LOAD_REGS_TYPES_1): Likewise.
(LOAD_REGS_TYPES_2): Likewise.
(LOAD_REGS_TYPES_3): Likewise.
(LOAD_REGS_TYPES_4): Likewise.
(LOAD_REGS_TYPES_5): Likewise.
(LOAD_REGS_TYPES_6): Likewise.
(TYPEFY): New define.
(ARGIFY): Likewise.
(internal_syscall0): Likewise.
(internal_syscall1): Likewise.
(internal_syscall2): Likewise.
(internal_syscall3): Likewise.
(internal_syscall4): Likewise.
(internal_syscall5): Likewise.
(internal_syscall6): Likewise.
* sysdeps/unix/sysv/linux/x86_64/x32/times.c
(INTERNAL_SYSCALL_NCS): Remove define.
(internal_syscall1): Add define.
The kernel interface for p{readv,writev}{64}v is
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
unsigned long pos_l, unsigned long pos_h)
Except for targets which define __ARCH_WANT_COMPAT_SYS_PREADV64 and
__ARCH_WANT_COMPAT_SYS_PWRITEV64,
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
off64_t pos)
is used for p{readv,writev}{64}v. X32 is the only such target. The
LO_HI_LONG macro is used to pass offset to the pos_l and pos_h pair.
Since pos_h is ignored when size of offset == sizeof of pos_l, x86-64
has
#define LO_HI_LONG(val) (val)
But the kernel interface for p{readv,writev}{64}v2 is
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
unsigned long pos_l, unsigned long pos_h, int flags)
Except for targets which define __ARCH_WANT_COMPAT_SYS_PREADV64V2 and
__ARCH_WANT_COMPAT_SYS_PWRITEV64V2,
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
off64_t pos, int flags)
is used for p{readv,writev}{64}v2. X32 is the only such target. Update
x86-64 LO_HI_LONG to pass 0 as the high part of the offset argument for
p{readv,writev}{64}v2 and define a different LO_HI_LONG for x32 to only
pass one argument for offset.
Tested on x32 and x86-64.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (LO_HI_LONG): Pass
0 as the high part of offset.
* sysdeps/unix/sysv/linux/x86_64/x32/sysdep.h (LO_HI_LONG): New.
Define LO_HI_LONG to skip pos_h since it is ignored by kernel:
static inline loff_t pos_from_hilo(unsigned long high, unsigned long low)
{
#define HALF_LONG_BITS (BITS_PER_LONG / 2)
return (((loff_t)high << HALF_LONG_BITS) << HALF_LONG_BITS) | low;
}
where size of loff_t == size of long.
[BZ #20349]
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (LO_HI_LONG): New.
X86-64 system calls use a different calling convention, which clobbers
CC, %r11 an %rcx registers. Define REGISTERS_CLOBBERED_BY_SYSCALL for
x86-64 inline asm statements.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(REGISTERS_CLOBBERED_BY_SYSCALL): New.
(INTERNAL_SYSCALL_NCS): Use it.
(INTERNAL_SYSCALL_NCS_TYPES): Likewise.
This patch consolidates the sched_getcpu implementations across all
arches (except tile, which requires its own). This patch removes
the powerpc, x86_64 and x32 specific files and change the default
linux one to use INLINE_VSYSCALL where possible (for ports that
implements it).
This patch consolidate the Linux vDSO define and usage across all ports
that uses it. The common vDSO definitions and calling through
{INLINE/INTERNAL}_VSYSCALL macros are moved to a common header
sysdep-vdso.h and vDSO name declaration and prototype is defined
using a common macro.
Also PTR_{MANGLE,DEMANGLE} is added to ports that does not use them
for vDSO calls (aarch64, powerpc, s390, and tile) and thus it will
reflect in code changes. For ports that already implement pointer
mangling/demangling in vDSO system (i386, x32, x86_64) this patch
is mainly a code refactor.
Checked on x32, x86_64, x32, ppc64le, and aarch64.
(_libc_vdso_platform_setup): Mangle function pointers before storing
them.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (INLINE_VSYSCALL):
Demangle vdso pointer before use.
(INTERNAL_VSYSCALL): Likewise.
* elf/cache.c (primes): Mark as const.
Noted by Roland McGrath.
returning.
(PTR_DEMANGLE): Real definition now that it's not the same as
PRT_MANGLE anymore.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h: Likewise.
* string/strerror_l.c: New file.
* string/Makefile (routines): Add strerror_l.
* string/string.h: Declare strerror_l.
* string/Versions: Export strerror_l for GLIBC_2.6.
* nscd/connections.c (nscd_run): Don't define MAXKEYLEN here.
* elf/rtld.c: Define __pointer_chk_guard_local and if necessary
__pointer_chk_guard.
(_rtld_global_ro): Initialize _dl_pointer_guard.
(dl_main): Initialize __pointer_chk_guard_local and either
__pointer_chk_guard or TLS value if necessary.
(process_envvars): Recognize and handle LD_POINTER_GUARD.
* sysdeps/generic/ldsodefs.h (rtld_global_ro): Add _dl_pointer_guard.
* sysdeps/i386/__longjmp.S: Use PTR_DEMANGLE for PC if defined.
* sysdeps/x86_64/__longjmp.S: Likewise.
* sysdeps/i386/bsd-_setjmp.S: Use PTR_MANGLE for PC if defined.
* sysdeps/i386/bsd-_setjmp.S: Likewise.
* sysdeps/i386/setjmp.S: Likewise.
[IS_IN_rtld]: Avoid call to __sigjmp_save.
* sysdeps/i386/setjmp.S: Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h: Define PTR_MANGLE and
PTR_DEMANGLE.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h: Likewise.
* sysdeps/i386/elf/setjmp.S: Removed.
* sysdeps/i386/elf/bsd-setjmp.S: Removed.
