In the glibc the gettimeofday can use vDSO (on power and x86 the
USE_IFUNC_GETTIMEOFDAY is defined), gettimeofday syscall or 'default'
___gettimeofday() from ./time/gettime.c (as a fallback).
In this patch the last function (___gettimeofday) has been refactored and
moved to ./sysdeps/unix/sysv/linux/gettimeofday.c to be Linux specific.
The new __gettimeofday64 explicit 64 bit function for getting 64 bit time from
the kernel (by internally calling __clock_gettime64) has been introduced.
Moreover, a 32 bit version - __gettimeofday has been refactored to internally
use __gettimeofday64.
The __gettimeofday is now supposed to be used on systems still supporting 32
bit time (__TIMESIZE != 64) - hence the necessary check for time_t potential
overflow and conversion of struct __timeval64 to 32 bit struct timespec.
The iFUNC vDSO direct call optimization has been removed from both i686 and
powerpc32 (USE_IFUNC_GETTIMEOFDAY is not defined for those architectures
anymore). The Linux kernel does not provide a y2038 safe implementation of
gettimeofday neither it plans to provide it in the future, clock_gettime64
should be used instead. Keeping support for this optimization would require
to handle another build permutation (!__ASSUME_TIME64_SYSCALLS &&
USE_IFUNC_GETTIMEOFDAY) which adds more complexity and has limited use
(since the idea is to eventually have a y2038 safe glibc build).
Build tests:
./src/scripts/build-many-glibcs.py glibcs
Run-time tests:
- Run specific tests on ARM/x86 32bit systems (qemu):
https://github.com/lmajewski/meta-y2038 and run tests:
https://github.com/lmajewski/y2038-tests/commits/master
Above tests were performed with Y2038 redirection applied as well as without
to test proper usage of both __gettimeofday64 and __gettimeofday.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
[Including some commit message improvement]
With all Linux ABIs using the expected Linux kABI to indicate
syscalls errors, the INTERNAL_SYSCALL_DECL is an empty declaration
on all ports.
This patch removes the 'err' argument on INTERNAL_SYSCALL* macro
and remove the INTERNAL_SYSCALL_DECL usage.
Checked with a build against all affected ABIs.
The __suseconds64_t type is supposed to be the 64 bit type across all
architectures.
It would be mostly used internally in the glibc - however, when passed to
Linux kernel (very unlikely), if necessary, it shall be converted to 32
bit type (i.e. __suseconds_t)
Build tests:
./src/scripts/build-many-glibcs.py glibcs
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This patch moves the vDSO setup from libc to loader code, just after
the vDSO link_map setup. For static case the initialization
is moved to _dl_non_dynamic_init instead.
Instead of using the mangled pointer, the vDSO data is set as
attribute_relro (on _rtld_global_ro for shared or _dl_vdso_* for
static). It is read-only even with partial relro.
It fixes BZ#24967 now that the vDSO pointer is setup earlier than
malloc interposition is called.
Also, vDSO calls should not be a problem for static dlopen as
indicated by BZ#20802. The vDSO pointer would be zero-initialized
and the syscall will be issued instead.
Checked on x86_64-linux-gnu, i686-linux-gnu, aarch64-linux-gnu,
arm-linux-gnueabihf, powerpc64le-linux-gnu, powerpc64-linux-gnu,
powerpc-linux-gnu, s390x-linux-gnu, sparc64-linux-gnu, and
sparcv9-linux-gnu. I also run some tests on mips.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The IFUNC bypass to vDSO is used when USE_IFUNC_TIME is set.
Currently powerpc and x86 defines it. Otherwise the generic
implementation is used, which calls clock_gettime.
Checked on powerpc64le-linux-gnu, powerpc64-linux-gnu,
powerpc-linux-gnu-power4, x86_64-linux-gnu, and i686-linux-gnu.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The IFUNC bypass to vDSO is used when USE_IFUNC_GETTIMEOFDAY is set.
Currently aarch64, powerpc*, and x86 defines it. Otherwise the
generic implementation is used, which calls clock_gettime.
Checked on aarch64-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, powerpc-linux-gnu-power4, x86_64-linux-gnu,
and i686-linux-gnu.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Add the missing time and clock_getres vDSO symbol names on x86.
For time, the iFUNC already uses expected name so it affects only
the static build.
The clock_getres is a new implementation added on Linux 5.3
(f66501dc53e72).
Checked on x86-linux-gnu and i686-linux-gnu.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
As indicated on libc-help [1] the ec138c67cb commit broke 32-bit
builds when configured with --enable-kernel=5.1 or higher. The
scenario 10 from [2] might also occur in this configuration and
INLINE_VSYSCALL will try to use the vDSO symbol and
HAVE_CLOCK_GETTIME64_VSYSCALL does not set HAVE_VSYSCALL prior its
usage.
Also, there is no easy way to just enable the code to use one
vDSO symbol since the macro INLINE_VSYSCALL is redefined if
HAVE_VSYSCALL is set.
Instead of adding more pre-processor handling and making the code
even more convoluted, this patch removes the requirement of defining
HAVE_VSYSCALL before including sysdep-vdso.h to enable vDSO usage.
The INLINE_VSYSCALL is now expected to be issued inside a
HAVE_*_VSYSCALL check, since it will try to use the internal vDSO
pointers.
Both clock_getres and clock_gettime vDSO code for time64_t were
removed since there is no vDSO setup code for the symbol (an
architecture can not set HAVE_CLOCK_GETTIME64_VSYSCALL).
Checked on i686-linux-gnu (default and with --enable-kernel=5.1),
x86_64-linux-gnu, aarch64-linux-gnu, and powerpc64le-linux-gnu.
I also checked against a build to mips64-linux-gnu and
sparc64-linux-gnu.
[1] https://sourceware.org/ml/libc-help/2019-12/msg00014.html
[2] https://sourceware.org/ml/libc-alpha/2019-12/msg00142.html
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Also make the public prototype of gettimeofday declare its second
argument with type "void *" unconditionally, consistent with POSIX.
It is also consistent with POSIX.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, powerpc-linux-gnu, and aarch64-linux-gnu.
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Consolidate generic gettimeofday implementation to use clock_gettime.
Linux ports that still provide gettimeofday through vDSO are not
changed.
Remove sysdeps/unix/clock_gettime.c, which implemented clock_gettime
using gettimeofday; new OS ports must provide a real implementation of
clock_gettime.
Rename sysdeps/mach/gettimeofday.c to sysdeps/mach/clock_gettime.c and
convert into an implementation of clock_gettime. It only supports
CLOCK_REALTIME; Mach does not appear to have any support for monotonic
clocks. It uses __host_get_time, which provides at best microsecond
resolution. Hurd is currently using sysdeps/posix/clock_getres.c for
clock_getres; its output for CLOCK_REALTIME is based on
sysconf (_SC_CLK_TCK), and I do not know whether that gives the
correct result.
Unlike settimeofday, there are no known uses of gettimeofday's
vestigial "get time zone" feature that are not bugs. (The per-process
timezone support in localtime and friends is unrelated, and the
programs that set the kernel's offset between the hardware clock and
UTC do not need to read it back.) Therefore, this feature is dummied
out. Henceforth, if gettimeofday's "struct timezone" argument is not
NULL, it will write zeroes to both fields. Any program that is
actually looking at this data will thus think it is running in UTC,
which is probably more correct than whatever it was doing before.
[__]gettimeofday no longer has any internal callers, so we can now
remove its internal prototype and PLT bypass aliases. The
__gettimeofday@GLIBC_2.0 export remains, in case it is used by any
third-party code.
It also allows to simplify the arch-specific implementation on x86 and
powerpc to remove the hack to disable the internal route to non iFUNC
variant for internal symbol.
This patch also fixes a missing optimization on aarch64, powerpc, and
x86 where the code used on static build do not use the vDSO.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, powerpc-linux-gnu, and aarch64-linux-gnu.
Co-authored-by: Zack Weinberg <zackw@panix.com>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
As for gettimeofday, time will be implemented based on clock_gettime
on all platforms and internal code should use clock_gettime
directly. In addition to removing a layer of indirection, this will
allow us to remove the PLT-bypass gunk for gettimeofday.
The changed code always assumes __clock_gettime (CLOCK_REALTIME)
or __clock_gettime (CLOCK_REALTIME_COARSE) (for Linux case) cannot
fail, using the same rationale for gettimeofday change. And internal
helper was added (time_now).
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, and powerpc-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Add a new macro __STATFS_MATCHES_STATFS64 that specifies if fsblkcnt_t
matches fsblkcnt64_t and if fsfilcnt_t matches fsfilcnt64_t.
As we don't have the padding we also need to update the overflow checker
to not access the undefined members.
Linux 5.3 adds a PTRACE_GET_SYSCALL_INFO constant, with an associated
structure and PTRACE_SYSCALL_INFO_* constants.
This patch adds these to sys/ptrace.h in glibc
(PTRACE_GET_SYSCALL_INFO in each architecture version, the rest in
bits/ptrace-shared.h). As with previous such constants and associated
structures, the glibc version of the structure is named struct
__ptrace_syscall_info.
Tested for x86_64, and with build-many-glibcs.py.
Linux vDSO initialization code the internal function pointers require a
lot of duplicated boilerplate over different architectures. This patch
aims to simplify not only the code but the required definition to enable
a vDSO symbol.
The changes are:
1. Consolidate all init-first.c on only one implementation and enable
the symbol based on HAVE_*_VSYSCALL existence.
2. Set the HAVE_*_VSYSCALL to the architecture expected names string.
3. Add a new internal implementation, get_vdso_mangle_symbol, which
returns a mangled function pointer.
Currently the clock_gettime, clock_getres, gettimeofday, getcpu, and time
are handled in an arch-independent way, powerpc still uses some
arch-specific vDSO symbol handled in a specific init-first implementation.
Checked on aarch64-linux-gnu, arm-linux-gnueabihf, i386-linux-gnu,
mips64-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu,
sparc64-linux-gnu, and x86_64-linux-gnu.
* sysdeps/powerpc/powerpc32/backtrace.c (is_sigtramp_address,
is_sigtramp_address_rt): Use HAVE_SIGTRAMP_{RT}32 instead of SHARED.
* sysdeps/powerpc/powerpc64/backtrace.c (is_sigtramp_address):
Likewise.
* sysdeps/unix/sysv/linux/aarch64/init-first.c: Remove file.
* sysdeps/unix/sysv/linux/aarch64/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/arm/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/arm/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/mips/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/mips/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/i386/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/riscv/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/s390/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/s390/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/sparc/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise.
* sysdeps/unix/sysv/linux/aarch64/sysdep.h
(HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Define value based on kernel exported
name.
* sysdeps/unix/sysv/linux/arm/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/mips/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/sysdep.h
(HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETCPU_VSYSCALL, HAVE_TIME_VSYSCALL, HAVE_GET_TBFREQ,
HAVE_SIGTRAMP_RT64, HAVE_SIGTRAMP_32, HAVE_SIGTRAMP_RT32i,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/riscv/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL,
HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/s390/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL,
HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL,
HAVE_GETTIMEOFDAY_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL,
HAVE_GETCPU_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/dl-vdso.h (VDSO_NAME, VDSO_HASH): Define to
invalid names if architecture does not define them.
(get_vdso_mangle_symbol): New symbol.
* sysdeps/unix/sysv/linux/init-first.c: New file.
* sysdeps/unix/sysv/linux/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/init-first.c (gettimeofday,
clock_gettime, clock_getres, getcpu, time): Remove declaration.
(__libc_vdso_platform_setup_arch): Likewise and use
get_vdso_mangle_symbol to setup vDSO symbols.
(sigtramp_rt64, sigtramp32, sigtramp_rt32, get_tbfreq): Add
attribute_hidden.
* sysdeps/unix/sysv/linux/powerpc/libc-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sysdep-vdso.h (VDSO_SYMBOL): Remove
definition.
This patch assumes static vDSO is supported as default, it is now supported
on all current architectures that support vDSO. It allows removing both
ALWAYS_USE_VSYSCALL define, which an architecture requires to explicit define
and USE_VSYSCALL (which defines vDSO only for shared or if architecture defines
ALWAYS_USE_VSYSCALL).
Checked with a build against all affected ABIs.
[BZ #19767]
* sysdeps/unix/sysv/linux/aarch64/sysdep.h (ALWAYS_USE_VSYSCALL):
Remove definition.
* sysdeps/unix/sysv/linux/arm/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/riscv/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h
(ALWAYS_USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (ALWAYS_USE_VSYSCALL):
Likewise.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Remove #if USE_VSYSCALL.
* sysdeps/unix/sysv/linux/sysdep-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/sysdep.h (ALWAYS_USE_VSYSCALL,
USE_VSYSCALL): Remove defitions.
Rename lll_timedlock to lll_clocklock and add clockid
parameter to indicate the clock that the abstime parameter should
be measured against in preparation for adding
pthread_mutex_clocklock.
The name change mirrors the naming for the exposed pthread functions:
timed => absolute timeout measured against CLOCK_REALTIME (or clock
specified by attribute in the case of pthread_cond_timedwait.)
clock => absolute timeout measured against clock specified in preceding
parameter.
* sysdeps/nptl/lowlevellock.h (lll_clocklock): Rename from
lll_timedlock and add clockid parameter. (__lll_clocklock): Rename
from __lll_timedlock and add clockid parameter.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_clocklock):
Likewise.
* nptl/lll_timedlock_wait.c (__lll_clocklock_wait): Rename from
__lll_timedlock_wait and add clockid parameter. Use __clock_gettime
rather than __gettimeofday so that clockid can be used. This means
that conversion from struct timeval is no longer required.
* sysdeps/sparc/sparc32/lowlevellock.c (lll_clocklock_wait):
Likewise.
* sysdeps/sparc/sparc32/lll_timedlock_wait.c: Update comment to
refer to __lll_clocklock_wait rather than __lll_timedlock_wait.
* nptl/pthread_mutex_timedlock.c (lll_clocklock_elision): Rename
from lll_timedlock_elision, add clockid parameter and use
meaningful names for other parameters. (__pthread_mutex_timedlock):
Pass CLOCK_REALTIME where necessary to lll_clocklock and
lll_clocklock_elision.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h
(lll_clocklock_elision): Rename from lll_timedlock_elision and add
clockid parameter. (__lll_clocklock_elision): Rename from
__lll_timedlock_elision and add clockid parameter.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/elision-timed.c
(__lll_lock_elision): Call __lll_clocklock_elision rather than
__lll_timedlock_elision. (EXTRAARG): Add clockid parameter.
(LLL_LOCK): Likewise.
* sysdeps/unix/sysv/linux/s390/elision-timed.c: Likewise.
* sysdeps/unix/sysv/linux/x86/elision-timed.c: Likewise.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This patch removes the arch-specific x86 assembly implementation for
low level locking and consolidate both 64 bits and 32 bits in a
single implementation.
Different than other architectures, x86 lll_trylock, lll_lock, and
lll_unlock implements a single-thread optimization to avoid atomic
operation, using cmpxchgl instead. This patch implements by using
the new single-thread.h definitions in a generic way, although using
the previous semantic.
The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use
atomic operations plus calls to lll_lock_wait*.
For __lll_lock_wait_private and __lll_lock_wait the generic implemtation
there is no indication that assembly implementation is required
performance-wise.
Checked on x86_64-linux-gnu and i686-linux-gnu.
* sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro.
(lll_trylock): Call __lll_trylock.
* sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file.
* sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file.
* sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include
lowlevellock-futex.h.
One group of warnings seen with -Wextra is warnings for static or
inline not at the start of a declaration (-Wold-style-declaration).
This patch fixes various such cases for inline, ensuring it comes at
the start of the declaration (after any static). A common case of the
fix is "static inline <type> __always_inline"; the definition of
__always_inline starts with __inline, so the natural change is to
"static __always_inline <type>". Other cases of the warning may be
harder to fix (one pattern is a function definition that gets
rewritten to be static by an including file, "#define funcname static
wrapped_funcname" or similar), but it seems worth fixing these cases
with inline anyway.
Tested for x86_64.
* elf/dl-load.h (_dl_postprocess_loadcmd): Use __always_inline
before return type, without separate inline.
* elf/dl-tunables.c (maybe_enable_malloc_check): Likewise.
* elf/dl-tunables.h (tunable_is_name): Likewise.
* malloc/malloc.c (do_set_trim_threshold): Likewise.
(do_set_top_pad): Likewise.
(do_set_mmap_threshold): Likewise.
(do_set_mmaps_max): Likewise.
(do_set_mallopt_check): Likewise.
(do_set_perturb_byte): Likewise.
(do_set_arena_test): Likewise.
(do_set_arena_max): Likewise.
(do_set_tcache_max): Likewise.
(do_set_tcache_count): Likewise.
(do_set_tcache_unsorted_limit): Likewise.
* nis/nis_subr.c (count_dots): Likewise.
* nptl/allocatestack.c (advise_stack_range): Likewise.
* sysdeps/ieee754/dbl-64/s_sin.c (do_cos): Likewise.
(do_sin): Likewise.
(reduce_sincos): Likewise.
(do_sincos): Likewise.
* sysdeps/unix/sysv/linux/x86/elision-conf.c
(do_set_elision_enable): Likewise.
(TUNABLE_CALLBACK_FNDECL): Likewise.
All the required code already existed, and some of it was already
running.
AT_SYSINFO_EHDR is processed if NEED_DL_SYSINFO_DSO is defined, but it
looks like it always is. The call to setup_vdso is also unconditional,
so all that was left to do was setup the function pointers and use
them. This patch just deletes some #ifdef to enable that.
[BZ #19767]
* nptl/Makefile (tests-static): Add tst-cond11-static.
(tests): Likewise.
* nptl/tst-cond11-static.c: New File.
* sysdeps/unix/sysv/linux/Makefile (tests-static): Add
tst-affinity-static.
(tests): Likewise.
* sysdeps/unix/sysv/linux/sysdep-vdso.h: Check USE_VSYSCALL
instead of SHARED.
* sysdeps/unix/sysv/linux/sysdep.h (ALWAYS_USE_VSYSCALL): New.
(USE_VSYSCALL): Likewise.
* sysdeps/unix/sysv/linux/tst-affinity-static.c: New file.
* sysdeps/unix/sysv/linux/x86/libc-vdso.h: Check USE_VSYSCALL
instead of SHARED.
* sysdeps/unix/sysv/linux/x86_64/init-first.c: Don't check
SHARED.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (ALWAYS_USE_VSYSCALL):
New.
To determine whether the default time_t interfaces are 32-bit
and so need conversions, or are 64-bit and so are compatible
with the internal 64-bit type without conversions, a macro
giving the size of the default time_t is also required.
This macro is called __TIMESIZE.
This macro can then be used instead of __WORDSIZE in msq-pad.h
and shm-pad.h files, which in turn allows removing their x86
variants, and in sem-pad.h files but keeping the x86 variant.
This patch was tested by running 'make check' on branch master
then applying this patch and running 'make check' again, and
checking that both 'make check' yield identical results.
This was done on x86_64-linux-gnu and i686-linux-gnu.
* bits/timesize.h: New file.
* stdlib/Makefile (headers): Add bits/timesize.h.
* sysdeps/unix/sysv/linux/bits/msq-pad.h
(__MSQ_PAD_AFTER_TIME): Use __TIMESIZE instead of __WORDSIZE.
* sysdeps/unix/sysv/linux/bits/sem-pad.h
(__SEM_PAD_AFTER_TIME): Likewise.
* sysdeps/unix/sysv/linux/bits/shm-pad.h
(__SHM_PAD_AFTER_TIME): Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/msq-pad.h
(__MSQ_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/sem-pad.h
(__SEM_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/shm-pad.h
(__SHM_PAD_BEFORE_TIME, __SHM_PAD_BETWEEN_TIME_AND_SEGSZ): Likewise.
* sysdeps/unix/sysv/linux/mips/bits/msq-pad.h
(__MSQ_PAD_AFTER_TIME, __MSQ_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/msq-pad.h
(__MSQ_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/sem-pad.h
(__SEM_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/shm-pad.h
(__SHM_PAD_BEFORE_TIME, __SHM_PAD_BETWEEN_TIME_AND_SEGSZ): Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/msq-pad.h
(__MSQ_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/sem-pad.h
(__SEM_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/shm-pad.h
(__SHM_PAD_BEFORE_TIME): Likewise.
* sysdeps/unix/sysv/linux/x86/bits/msq-pad.h: Delete file.
* sysdeps/unix/sysv/linux/x86/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/timesize.h: New file.
After my patch to move SHMLBA to its own header, the bits/shm.h
headers for architectures using the Linux kernel still vary in a few
ways: the use of __syscall_ulong_t; whether padding for 32-bit systems
is present before or after time fields, or missing altogether (mips,
x32); whether shm_segsz is before or after the time fields; whether,
if after time fields, there is extra padding before shm_segsz.
This patch arranges for a single header to be used. __syscall_ulong_t
is safe to use everywhere, while bits/shm-pad.h is added with new
macros __SHM_PAD_AFTER_TIME, __SHM_PAD_BEFORE_TIME,
__SHM_SEGSZ_AFTER_TIME and __SHM_PAD_BETWEEN_TIME_AND_SEGSZ to
describe the differences.
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/Makefile (sysdep_headers): Add
bits/shm-pad.h.
* sysdeps/unix/sysv/linux/bits/shm.h: Include <bits/shm-pad.h>.
(shmatt_t): Define as __syscall_ulong_t.
(__SHM_PAD_TIME): New macro, depending on [__SHM_PAD_BEFORE_TIME]
and [__SHM_PAD_AFTER_TIME].
(struct shmid_ds): Define time fields using __SHM_PAD_TIME.
Define shm_segsz and associated padding based on
[__SHM_SEGSZ_AFTER_TIME] and [__SHM_PAD_BETWEEN_TIME_AND_SEGSZ].
Use __syscall_ulong_t instead of unsigned long int.
[__USE_MISC] (struct shminfo): Use __syscall_ulong_t instead of
unsigned long int.
[__USE_MISC] (struct shm_info): Likewise.
* sysdeps/unix/sysv/linux/bits/shm-pad.h: New file.
* sysdeps/unix/sysv/linux/hppa/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/mips/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/shm-pad.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/shm.h: Remove.
* sysdeps/unix/sysv/linux/mips/bits/shm.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/shm.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/shm.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/shm.h: Likewise.
One difference between bits/shm.h headers for architectures using the
Linux kernel is the definition of SHMLBA. This was noted in
<https://sourceware.org/ml/libc-alpha/2018-09/msg00175.html> as a
reason why even a new architecture (C-SKY) might need its own
bits/shm.h; thus, splitting it out of bits/shm.h can allow less
duplication of headers for new architectures.
This patch moves that definition to its own header, bits/shmlba.h, to
allow more sharing of headers between architectures. That move allows
the arm, ia64 and sh variants of bits/shm.h to be removed, as they had
no other significant differences from the generic bits/shm.h; powerpc
and x86 have their own bits/shm.h but do not need to get their own
bits/shmlba.h because they use the same SHMLBA as the generic header.
Other architectures with their own bits/shm.h get their own
bits/shmlba.h without being able to remove their own bits/shm.h until
the generic one has been adapted to be able to handle more
architectures (where, in addition to the differences seen for
bits/msq.h and bits/sem.h, the position of shm_segsz in struct
shmid_ds also depends on the architecture).
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/Makefile (sysdep_headers): Add
bits/shmlba.h.
* sysdeps/unix/sysv/linux/bits/shm.h: Include <bits/shmlba.h>.
(SHMLBA): Remove macro.
(__getpagesize): Remove function declaration.
* sysdeps/unix/sysv/linux/hppa/bits/shm.h: Include
<bits/shmlba.h>.
(SHMLBA): Remove macro.
* sysdeps/unix/sysv/linux/mips/bits/shm.h: Include
<bits/shmlba.h>.
(SHMLBA): Remove macro.
* sysdeps/unix/sysv/linux/powerpc/bits/shm.h: Include
<bits/shmlba.h>.
(SHMLBA): Remove macro.
(__getpagesize): Remove function declaration.
* sysdeps/unix/sysv/linux/sparc/bits/shm.h: Include
<bits/shmlba.h>.
(SHMLBA): Remove macro.
(__getshmlba): Remove function declaration.
* sysdeps/unix/sysv/linux/x86/bits/shm.h: Include <bits/shmlba.h>.
(SHMLBA): Remove macro.
(__getpagesize): Remove function declaration.
* sysdeps/unix/sysv/linux/arm/bits/shm.h: Remove file.
* sysdeps/unix/sysv/linux/ia64/bits/shm.h: Likewise.
* sysdeps/unix/sysv/linux/sh/bits/shm.h: Likewise.
* sysdeps/unix/sysv/linux/bits/shmlba.h: New file.
* sysdeps/unix/sysv/linux/arm/bits/shmlba.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/shmlba.h: Likewise.
* sysdeps/unix/sysv/linux/ia64/bits/shmlba.h: Likewise.
* sysdeps/unix/sysv/linux/mips/bits/shmlba.h: Likewise.
* sysdeps/unix/sysv/linux/sh/bits/shmlba.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/shmlba.h: Likewise.
The race leads either to pthread_mutex_destroy returning EBUSY
or triggering an assertion (See description in bugzilla).
This patch is fixing the race by ensuring that the elision path is
used in all cases if elision is enabled by the GLIBC_TUNABLES framework.
The __kind variable in struct __pthread_mutex_s is accessed concurrently.
Therefore we are now using the atomic macros.
The new testcase tst-mutex10 is triggering the race on s390x and intel.
Presumably also on power, but I don't have access to a power machine
with lock-elision. At least the code for power is the same as on the other
two architectures.
ChangeLog:
[BZ #23275]
* nptl/tst-mutex10.c: New File.
* nptl/Makefile (tests): Add tst-mutex10.
(tst-mutex10-ENV): New variable.
* sysdeps/unix/sysv/linux/s390/force-elision.h: (FORCE_ELISION):
Ensure that elision path is used if elision is available.
* sysdeps/unix/sysv/linux/powerpc/force-elision.h (FORCE_ELISION):
Likewise.
* sysdeps/unix/sysv/linux/x86/force-elision.h: (FORCE_ELISION):
Likewise.
* nptl/pthreadP.h (PTHREAD_MUTEX_TYPE, PTHREAD_MUTEX_TYPE_ELISION)
(PTHREAD_MUTEX_PSHARED): Use atomic_load_relaxed.
* nptl/pthread_mutex_consistent.c (pthread_mutex_consistent): Likewise.
* nptl/pthread_mutex_getprioceiling.c (pthread_mutex_getprioceiling):
Likewise.
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock_full)
(__pthread_mutex_cond_lock_adjust): Likewise.
* nptl/pthread_mutex_setprioceiling.c (pthread_mutex_setprioceiling):
Likewise.
* nptl/pthread_mutex_timedlock.c (__pthread_mutex_timedlock): Likewise.
* nptl/pthread_mutex_trylock.c (__pthread_mutex_trylock): Likewise.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_full): Likewise.
* sysdeps/nptl/bits/thread-shared-types.h (struct __pthread_mutex_s):
Add comments.
* nptl/pthread_mutex_destroy.c (__pthread_mutex_destroy):
Use atomic_load_relaxed and atomic_store_relaxed.
* nptl/pthread_mutex_init.c (__pthread_mutex_init):
Use atomic_store_relaxed.
The bits/sem.h headers for architectures using the Linux kernel vary
in a few ways:
* x32 uses __syscall_ulong_t instead of unsigned long int.
* The x86 header uses padding after time fields unconditionally
(including for both x86_64 ABIs), not just for 32-bit time (unlike
in msqid_ds where there is only padding for 32-bit time). Because
this padding is present for x32, and is __syscall_ulong_t there, it
does have to be __syscall_ulong_t, not unsigned long int.
* The MIPS header never uses padding around time fields, even when
32-bit (unlike in msqid_ds where it has endian-dependent padding for
32-bit time).
* Some older 32-bit big-endian architectures have padding before
rather than after time fields, although the preferred generic
approach is padding after the time fields independent of endianness.
(There are also insubstantial differences such as use of unsigned int
for padding instead of unsigned long int, which makes no difference to
layout since the padding fields using unsigned int are only present on
32-bit architectures.)
For the first, __syscall_ulong_t can be used in the generic version as
it's the same as unsigned long int everywhere except x32. For the
other differences, this patch adds macros __SEM_PAD_BEFORE_TIME and
__SEM_PAD_AFTER_TIME in a new bits/sem-pad.h header, so that header is
the only one needing to be provided on architectures with differences
in this area, and everything else can go in a single common bits/sem.h
header.
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/Makefile (sysdep_headers): Add
bits/sem-pad.h.
* sysdeps/unix/sysv/linux/bits/sem.h: Include <bits/sem-pad.h>
instead of <bits/wordsize.h>.
(__SEM_PAD_TIME): New macro, depending on [__SEM_PAD_BEFORE_TIME]
and [__SEM_PAD_AFTER_TIME].
(struct semid_ds): Define time fields using __SEM_PAD_TIME. Use
__syscall_ulong_t instead of unsigned long int.
* sysdeps/unix/sysv/linux/bits/sem-pad.h: New file.
* sysdeps/unix/sysv/linux/hppa/bits/sem-pad.h: Likewise.
* sysdeps/unix/sysv/linux/mips/bits/sem-pad.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/sem-pad.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/sem-pad.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/sem-pad.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/sem.h: Remove.
* sysdeps/unix/sysv/linux/mips/bits/sem.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/sem.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/sem.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/sem.h: Likewise.
The bits/msq.h headers for architectures using the Linux kernel vary
in a few ways:
* x32 uses __syscall_ulong_t instead of unsigned long int.
* x32 has 64-bit time_t, so no padding around time fields despite
__WORDSIZE == 32.
* Some older 32-bit big-endian architectures have padding before
rather than after time fields, although the preferred generic
approach is padding after the time fields independent of endianness.
(There are also insubstantial differences such as use of unsigned int
for padding instead of unsigned long int, which makes no difference to
layout since the padding fields using unsigned int are only present on
32-bit architectures.)
For the first, __syscall_ulong_t can be used in the generic version as
it's the same as unsigned long int everywhere except x32. For the
other two differences, this patch adds macros __MSQ_PAD_BEFORE_TIME
and __MSQ_PAD_AFTER_TIME in a new bits/msq-pad.h header, so that
header is the only one needing to be provided on architectures with
differences in this area, and everything else can go in a single
common bits/msq.h header. Once we have __TIMESIZE, the generic
bits/msq-pad.h can change to use that instead of __WORDSIZE, at which
point the x86 version of bits/msq-pad.h won't be needed either.
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/Makefile (sysdep_headers): Add
bits/msq-pad.h.
* sysdeps/unix/sysv/linux/bits/msq.h: Include <bits/msq-pad.h>
instead of <bits/wordsize.h>.
(msgqnum_t): Define as __syscall_ulong_t.
(msglen_t): Likewise.
(__MSQ_PAD_TIME): New macro, depending on [__MSQ_PAD_BEFORE_TIME]
and [__MSQ_PAD_AFTER_TIME].
(struct msqid_ds): Define time fields using __MSQ_PAD_TIME. Use
__syscall_ulong_t instead of unsigned long int.
* sysdeps/unix/sysv/linux/bits/msq-pad.h: New file.
* sysdeps/unix/sysv/linux/hppa/bits/msq-pad.h: Likewise.
* sysdeps/unix/sysv/linux/mips/bits/msq-pad.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/msq-pad.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/msq-pad.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/msq-pad.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/msq.h: Remove.
* sysdeps/unix/sysv/linux/mips/bits/msq.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/msq.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/msq.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/msq.h: Likewise.
Since RTM intrinsics are supported in GCC 4.9, we can use them in
pthread mutex lock elision.
* sysdeps/unix/sysv/linux/x86/Makefile (CFLAGS-elision-lock.c):
Add -mrtm.
(CFLAGS-elision-unlock.c): Likewise.
(CFLAGS-elision-timed.c): Likewise.
(CFLAGS-elision-trylock.c): Likewise.
* sysdeps/unix/sysv/linux/x86/hle.h: Rewritten.
Continuing bits/mman.h unification between architectures using the
Linux kernel, this patch arranges for the common set of MAP_* flags to
be used by two more architectures. That common set is moved to
bits/mman-map-flags-generic.h, which is included by bits/mman.h, to
allow architectures to use that common set even if they also have
architecture-specific additions to it. As well as the generic
bits/mman.h, the versions for x86 and ia64 are also then made to
include bits/mman-map-flags-generic.h, so while they still need
architecture-specific bits/mman.h (for MAP_32BIT and MAP_GROWSUP
respectively), they do not need to duplicate the generic flag
definitions in there.
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/bits/mman-map-flags-generic.h: New
file. Most contents moved from ....
* sysdeps/unix/sysv/linux/bits/mman.h: ... here. Move contents to
and include <bits/mman-map-flags-generic.h>.
* sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc]
(sysdep_headers): Add bits/mman-map-flags-generic.h.
* sysdeps/unix/sysv/linux/ia64/bits/mman.h: Include
<bits/mman-map-flags-generic.h>.
[__USE_MISC] (MAP_GROWSUP): Only define this macro, not other
macros defined in <bits/mman-map-flags-generic.h>.
* sysdeps/unix/sysv/linux/x86/bits/mman.h: Include
<bits/mman-map-flags-generic.h>.
[__USE_MISC] (MAP_32BIT): Only define this macro, not other macros
defined in <bits/mman-map-flags-generic.h>.
This patch continues the process of unifying sys/procfs.h headers for
architectures using the Linux kernel.
A bits/procfs-id.h header is added to define __pr_uid_t and __pr_gid_t
for the types of pr_uid and pr_gid; the default version of this header
uses unsigned int. On some architectures, sys/procfs.h has copies of
32-bit structures for 64-bit builds; those move into a
bits/procfs-extra.h header (they can't go in bits/procfs.h because
they have to come *after* other declarations from sys/procfs.h).
Given appropriate versions of these headers, six more architectures
can then move to providing only bits/procfs*.h without duplicating the
rest of the contents of sys/procfs.h. Only alpha needs a further
bits/ header to be added before it can stop having its own
sys/procfs.h.
Tested for x86_64 and x86, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/sys/procfs.h: Include
<bits/procfs-id.h> and <bits/procfs-extra.h>.
(struct elf_prpsinfo): Use __pr_uid_t and __pr_gid_t as types of
pr_uid and pr_gid.
* sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc]
(sysdep_headers): Add bits/procfs-id.h and bits/procfs-extra.h.
* sysdeps/unix/sysv/linux/bits/procfs-extra.h: New file.
* sysdeps/unix/sysv/linux/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/arm/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/arm/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/s390/bits/procfs-extra.h: Likewise.
* sysdeps/unix/sysv/linux/s390/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/s390/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/sh/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/sh/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/procfs-extra.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/procfs-id.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/arm/sys/procfs.h: Remove file.
* sysdeps/unix/sysv/linux/m68k/sys/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/s390/sys/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/sh/sys/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sys/procfs.h: Likewise.
* sysdeps/unix/sysv/linux/x86/sys/procfs.h: Likewise.
GNU_PROPERTY_X86_FEATURE_1_AND may not be the first property item. We
need to check each property item until we reach the end of the property
or find GNU_PROPERTY_X86_FEATURE_1_AND.
This patch adds 2 tests. The first test checks if IBT is enabled and
the second test reads the output from the first test to check if IBT
is is enabled. The second second test fails if IBT isn't enabled
properly.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
[BZ #23467]
* sysdeps/unix/sysv/linux/x86/Makefile (tests): Add
tst-cet-property-1 and tst-cet-property-2 if CET is enabled.
(CFLAGS-tst-cet-property-1.o): New.
(ASFLAGS-tst-cet-property-dep-2.o): Likewise.
($(objpfx)tst-cet-property-2): Likewise.
($(objpfx)tst-cet-property-2.out): Likewise.
* sysdeps/unix/sysv/linux/x86/tst-cet-property-1.c: New file.
* sysdeps/unix/sysv/linux/x86/tst-cet-property-2.c: Likewise.
* sysdeps/unix/sysv/linux/x86/tst-cet-property-dep-2.S: Likewise.
* sysdeps/x86/dl-prop.h (_dl_process_cet_property_note): Parse
each property item until GNU_PROPERTY_X86_FEATURE_1_AND is found.
Verify that setcontext works with gaps above and below the newly
allocated shadow stack.
* sysdeps/unix/sysv/linux/x86/Makefile (tests): Add
tst-cet-setcontext-1 if CET is enabled.
(CFLAGS-tst-cet-setcontext-1.c): Add -mshstk.
* sysdeps/unix/sysv/linux/x86/tst-cet-setcontext-1.c: New file.
This patch adds a field to ucontext_t to save shadow stack:
1. getcontext and swapcontext are updated to save the caller's shadow
stack pointer and return addresses.
2. setcontext and swapcontext are updated to restore shadow stack and
jump to new context directly.
3. makecontext is updated to allocate a new shadow stack and set the
caller's return address to __start_context.
Since makecontext allocates a new shadow stack when making a new
context and kernel allocates a new shadow stack for clone/fork/vfork
syscalls, we track the current shadow stack base. In setcontext and
swapcontext, if the target shadow stack base is the same as the current
shadow stack base, we unwind the shadow stack. Otherwise it is a stack
switch and we look for a restore token.
We enable shadow stack at run-time only if program and all used shared
objects, including dlopened ones, are shadow stack enabled, which means
that they must be compiled with GCC 8 or above and glibc 2.28 or above.
We need to save and restore shadow stack only if shadow stack is enabled.
When caller of getcontext, setcontext, swapcontext and makecontext is
compiled with smaller ucontext_t, shadow stack won't be enabled at
run-time. We check if shadow stack is enabled before accessing the
extended field in ucontext_t.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* sysdeps/unix/sysv/linux/x86/sys/ucontext.h (ucontext_t): Add
__ssp.
* sysdeps/unix/sysv/linux/x86_64/__start_context.S: Include
<asm/prctl.h> and "ucontext_i.h" when shadow stack is enabled.
(__push___start_context): New.
* sysdeps/unix/sysv/linux/x86_64/getcontext.S: Include
<asm/prctl.h>.
(__getcontext): Record the current shadow stack base. Save the
caller's shadow stack pointer and base.
* sysdeps/unix/sysv/linux/x86_64/makecontext.c: Include
<pthread.h>, <libc-pointer-arith.h> and <sys/prctl.h>.
(__push___start_context): New prototype.
(__makecontext): Call __push___start_context to allocate a new
shadow stack, push __start_context onto the new stack as well
as the new shadow stack.
* sysdeps/unix/sysv/linux/x86_64/setcontext.S: Include
<asm/prctl.h>.
(__setcontext): Restore the target shadow stack.
* sysdeps/unix/sysv/linux/x86_64/swapcontext.S: Include
<asm/prctl.h>.
(__swapcontext): Record the current shadow stack base. Save
the caller's shadow stack pointer and base. Restore the target
shadow stack.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h
(STACK_SIZE_TO_SHADOW_STACK_SIZE_SHIFT): New.
* sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym (oSSP): New.
CET arch_prctl bits should be defined in <asm/prctl.h> from Linux kernel
header files. Add x86 <include/asm/prctl.h> for pre-CET kernel header
files.
Note: sysdeps/unix/sysv/linux/x86/include/asm/prctl.h should be removed
if <asm/prctl.h> from the required kernel header files contains CET
arch_prctl bits.
/* CET features:
IBT: GNU_PROPERTY_X86_FEATURE_1_IBT
SHSTK: GNU_PROPERTY_X86_FEATURE_1_SHSTK
*/
/* Return CET features in unsigned long long *addr:
features: addr[0].
shadow stack base address: addr[1].
shadow stack size: addr[2].
*/
# define ARCH_CET_STATUS 0x3001
/* Disable CET features in unsigned int features. */
# define ARCH_CET_DISABLE 0x3002
/* Lock all CET features. */
# define ARCH_CET_LOCK 0x3003
/* Allocate a new shadow stack with unsigned long long *addr:
IN: requested shadow stack size: *addr.
OUT: allocated shadow stack address: *addr.
*/
# define ARCH_CET_ALLOC_SHSTK 0x3004
/* Return legacy region bitmap info in unsigned long long *addr:
address: addr[0].
size: addr[1].
*/
# define ARCH_CET_LEGACY_BITMAP 0x3005
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* sysdeps/unix/sysv/linux/x86/include/asm/prctl.h: New file.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Include
<sys/prctl.h> and <asm/prctl.h>.
(get_cet_status): Call arch_prctl with ARCH_CET_STATUS.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Include <sys/prctl.h>
and <asm/prctl.h>.
(dl_cet_allocate_legacy_bitmap): Call arch_prctl with
ARCH_CET_LEGACY_BITMAP.
(dl_cet_disable_cet): Call arch_prctl with ARCH_CET_DISABLE.
(dl_cet_lock_cet): Call arch_prctl with ARCH_CET_LOCK.
* sysdeps/x86/libc-start.c: Include <startup.h>.
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
Save and restore shadow stack pointer in setjmp and longjmp to support
shadow stack in Intel CET. Use feature_1 in tcbhead_t to check if
shadow stack is enabled before saving and restoring shadow stack pointer.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* sysdeps/i386/__longjmp.S: Include <jmp_buf-ssp.h>.
(__longjmp): Restore shadow stack pointer if shadow stack is
enabled, SHADOW_STACK_POINTER_OFFSET is defined and __longjmp
isn't defined for __longjmp_cancel.
* sysdeps/i386/bsd-_setjmp.S: Include <jmp_buf-ssp.h>.
(_setjmp): Save shadow stack pointer if shadow stack is enabled
and SHADOW_STACK_POINTER_OFFSET is defined.
* sysdeps/i386/bsd-setjmp.S: Include <jmp_buf-ssp.h>.
(setjmp): Save shadow stack pointer if shadow stack is enabled
and SHADOW_STACK_POINTER_OFFSET is defined.
* sysdeps/i386/setjmp.S: Include <jmp_buf-ssp.h>.
(__sigsetjmp): Save shadow stack pointer if shadow stack is
enabled and SHADOW_STACK_POINTER_OFFSET is defined.
* sysdeps/unix/sysv/linux/i386/____longjmp_chk.S: Include
<jmp_buf-ssp.h>.
(____longjmp_chk): Restore shadow stack pointer if shadow stack
is enabled and SHADOW_STACK_POINTER_OFFSET is defined.
* sysdeps/unix/sysv/linux/x86/Makefile (gen-as-const-headers):
Remove jmp_buf-ssp.sym.
* sysdeps/unix/sysv/linux/x86_64/____longjmp_chk.S: Include
<jmp_buf-ssp.h>.
(____longjmp_chk): Restore shadow stack pointer if shadow stack
is enabled and SHADOW_STACK_POINTER_OFFSET is defined.
* sysdeps/x86/Makefile (gen-as-const-headers): Add
jmp_buf-ssp.sym.
* sysdeps/x86/jmp_buf-ssp.sym: New dummy file.
* sysdeps/x86_64/__longjmp.S: Include <jmp_buf-ssp.h>.
(__longjmp): Restore shadow stack pointer if shadow stack is
enabled, SHADOW_STACK_POINTER_OFFSET is defined and __longjmp
isn't defined for __longjmp_cancel.
* sysdeps/x86_64/setjmp.S: Include <jmp_buf-ssp.h>.
(__sigsetjmp): Save shadow stack pointer if shadow stack is
enabled and SHADOW_STACK_POINTER_OFFSET is defined.
The pad array in struct pthread_unwind_buf is used by setjmp to save
shadow stack register. We assert that size of struct pthread_unwind_buf
is no less than offset of shadow stack pointer + shadow stack pointer
size.
Since functions, like LIBC_START_MAIN, START_THREAD_DEFN as well as
these with thread cancellation, call setjmp, but never return after
__libc_unwind_longjmp, __libc_unwind_longjmp, which is defined as
__libc_longjmp on x86, doesn't need to restore shadow stack register.
__libc_longjmp, which is a private interface for thread cancellation
implementation in libpthread, is changed to call __longjmp_cancel,
instead of __longjmp. __longjmp_cancel is a new internal function
in libc, which is similar to __longjmp, but doesn't restore shadow
stack register.
The compatibility longjmp and siglongjmp in libpthread.so are changed
to call __libc_siglongjmp, instead of __libc_longjmp, so that they will
restore shadow stack register.
Tested with build-many-glibcs.py.
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* nptl/pthread_create.c (START_THREAD_DEFN): Clear previous
handlers after setjmp.
* setjmp/longjmp.c (__libc_longjmp): Don't define alias if
defined.
* sysdeps/unix/sysv/linux/x86/setjmpP.h: Include
<libc-pointer-arith.h>.
(_JUMP_BUF_SIGSET_BITS_PER_WORD): New.
(_JUMP_BUF_SIGSET_NSIG): Changed to 96.
(_JUMP_BUF_SIGSET_NWORDS): Changed to use ALIGN_UP and
_JUMP_BUF_SIGSET_BITS_PER_WORD.
* sysdeps/x86/Makefile (sysdep_routines): Add __longjmp_cancel.
* sysdeps/x86/__longjmp_cancel.S: New file.
* sysdeps/x86/longjmp.c: Likewise.
* sysdeps/x86/nptl/pt-longjmp.c: Likewise.
This patch adds the PTRACE_SECCOMP_GET_METADATA constant from Linux
4.16 to all relevant sys/ptrace.h files. A type struct
__ptrace_seccomp_metadata, analogous to other such types, is also
added.
Tested for x86_64, and with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): New enum value and macro.
* sysdeps/unix/sysv/linux/bits/ptrace-shared.h
(struct __ptrace_seccomp_metadata): New type.
* sysdeps/unix/sysv/linux/aarch64/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/arm/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/ia64/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/powerpc/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/s390/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/sparc/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/tile/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
* sysdeps/unix/sysv/linux/x86/sys/ptrace.h
(PTRACE_SECCOMP_GET_METADATA): Likewise.
This patch adds the MAP_SYNC macro from Linux 4.15 to various
bits/mman.h headers. Note that this is *not* added to all
architectures: in Linux 4.15, this macro is only in
asm-generic/mman.h, and only some architectures' asm/mman.h include
the asm-generic file - the architectures not using the asm-generic
file will need their own values of MAP_SYNC allocated to support this
functionality (some of them also already have conflicting mmap flags
so the value there will have to be different from the generic
0x80000). Specifically, for glibc architectures, alpha hppa mips
powerpc sparc tile lack allocations of values for MAP_SYNC.
Tested for x86_64.
* sysdeps/unix/sysv/linux/aarch64/bits/mman.h [__USE_MISC]
(MAP_SYNC): New macro.
* sysdeps/unix/sysv/linux/arm/bits/mman.h [__USE_MISC] (MAP_SYNC):
Likewise.
* sysdeps/unix/sysv/linux/ia64/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/m68k/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/microblaze/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/nios2/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/riscv/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/s390/bits/mman.h [__USE_MISC]
(MAP_SYNC): Likewise.
* sysdeps/unix/sysv/linux/sh/bits/mman.h [__USE_MISC] (MAP_SYNC):
Likewise.
* sysdeps/unix/sysv/linux/x86/bits/mman.h [__USE_MISC] (MAP_SYNC):
Likewise.
In commit cba595c350 and commit
f81ddabffd, ABI compatibility with
applications was broken by increasing the size of the on-stack
allocated __pthread_unwind_buf_t beyond the oringal size.
Applications only have the origianl space available for
__pthread_unwind_register, and __pthread_unwind_next to use,
any increase in the size of __pthread_unwind_buf_t causes these
functions to write beyond the original structure into other
on-stack variables leading to segmentation faults in common
applications like vlc. The only workaround is to version those
functions which operate on the old sized objects, but this must
happen in glibc 2.28.
Thank you to Andrew Senkevich, H.J. Lu, and Aurelien Jarno, for
submitting reports and tracking the issue down.
The commit reverts the above mentioned commits and testing on
x86_64 shows that the ABI compatibility is restored. A tst-cleanup1
regression test linked with an older glibc now passes when run
with the newly built glibc. Previously a tst-cleanup1 linked with
an older glibc would segfault when run with an affected glibc build.
Tested on x86_64 with no regressions.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
