Make glibc MTE-safe on systems where MTE is available. This allows
using heap tagging with an LD_PRELOADed malloc implementation that
enables MTE. We don't document this as guaranteed contract yet, so
glibc may not be MTE safe when HWCAP2_MTE is set (older glibcs
certainly aren't). This is mainly for testing and debugging.
The HWCAP flag is not exposed in public headers until Linux adds it
to its uapi. The HWCAP value reservation will be in Linux 5.9.
Use PROT_READ and PROT_WRITE according to the load segment p_flags
when adding PROT_BTI.
This is before processing relocations which may drop PROT_BTI in
case of textrels. Executable stacks are not protected via PROT_BTI
either. PROT_BTI is hardening in case memory corruption happened,
it's value is reduced if there is writable and executable memory
available so missing it on such memory is fine, but we should
respect the p_flags and should not drop PROT_WRITE.
Add a line that was missing from a previous commit.
Without increasing str, the null-byte is not validated, and
_dl_string_platform returns -1.
Fixes: d2ba3677da ("powerpc: Add support for POWER10")
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Upstream GCC 11 development is now building the ibm128 runtime
support (in libgcc) without a .gnu.attributes section on ppc64le.
Ensure we have one to replace by building one ibm128 file in
libc and libm with attributes.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
When e.g. an LD_PRELOAD fails, _dl_signal_exception/error longjmps, but TLS
is not initialized yet, let along signal state. We thus mustn't look at
them within __longjmp.
* sysdeps/mach/hurd/i386/____longjmp_chk.S,__longjmp.S: Check for
initialized value of %gs, and that sigstate is non-NULL.
Optimize strlen using a mix of scalar and SIMD code. On modern micro
architectures large strings are 2.6 times faster than existing
strlen_asimd and 35% faster than the new MTE version of strlen.
On a random strlen benchmark using small sizes the speedup is 7% vs
strlen_asimd and 40% vs the MTE strlen. This fixes the main strlen
regressions on Cortex-A53 and other cores with a simple Neon unit.
Rename __strlen_generic to __strlen_mte, and select strlen_asimd when
MTE is not enabled (this is waiting on support for a HWCAP_MTE bit).
This fixes big-endian bug 25824. Passes GLIBC regression tests.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
The kernel ABI is not finalized, and there are now various proposals
to change the size of struct rseq, which would make the glibc ABI
dependent on the version of the kernels used for building glibc.
This is of course not acceptable.
This reverts commit 48699da1c4 ("elf:
Support at least 32-byte alignment in static dlopen"), commit
8f4632deb3 ("Linux: rseq registration
tests"), commit 6e29cb3f61 ("Linux: Use
rseq in sched_getcpu if available"), and commit
0c76fc3c2b ("Linux: Perform rseq
registration at C startup and thread creation"), resolving the conflicts
introduced by the ARC port and the TLS static surplus changes.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The arm string/tst-memmove-overflow XFAIL has been added in commit
eca1b23332 ("arm: XFAIL string/tst-memmove-overflow due to bug 25620")
as a way to reproduce the reported bug.
Now that this bug has been fixed in commits 79a4fa341b ("arm:
CVE-2020-6096: fix memcpy and memmove for negative length [BZ #25620]")
and beea361050 ("arm: CVE-2020-6096: Fix multiarch memcpy for negative
length [BZ #25620]"), let's remove the XFAIL.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Add a new memcpy using 128-bit Q registers - this is faster on modern
cores and reduces codesize. Similar to the generic memcpy, small cases
include copies up to 32 bytes. 64-128 byte copies are split into two
cases to improve performance of 64-96 byte copies. Large copies align
the source rather than the destination.
bench-memcpy-random is ~9% faster than memcpy_falkor on Neoverse N1,
so make this memcpy the default on N1 (on Centriq it is 15% faster than
memcpy_falkor).
Passes GLIBC regression tests.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Given almost all uses of ENTRY are for string/memory functions,
align ENTRY to a cacheline to simplify things.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Sun RPC was removed from glibc. This includes rpcgen program, librpcsvc,
and Sun RPC headers. Also test for bug #20790 was removed
(test for rpcgen).
Backward compatibility for old programs is kept only for architectures
and ABIs that have been added in or before version 2.28.
libtirpc is mature enough, librpcsvc and rpcgen are provided in
rpcsvc-proto project.
NOTE: libnsl code depends on Sun RPC (installed libnsl headers use
installed Sun RPC headers), thus --enable-obsolete-rpc was a dependency
for --enable-obsolete-nsl (removed in a previous commit).
The arc ABI list file has to be updated because the port was added
with the sunrpc symbols
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Support usable check for all CPU features with the following changes:
1. Change struct cpu_features to
struct cpuid_features
{
struct cpuid_registers cpuid;
struct cpuid_registers usable;
};
struct cpu_features
{
struct cpu_features_basic basic;
struct cpuid_features features[COMMON_CPUID_INDEX_MAX];
unsigned int preferred[PREFERRED_FEATURE_INDEX_MAX];
...
};
so that there is a usable bit for each cpuid bit.
2. After the cpuid bits have been initialized, copy the known bits to the
usable bits. EAX/EBX from INDEX_1 and EAX from INDEX_7 aren't used for
CPU feature detection.
3. Clear the usable bits which require OS support.
4. If the feature is supported by OS, copy its cpuid bit to its usable
bit.
5. Replace HAS_CPU_FEATURE and CPU_FEATURES_CPU_P with CPU_FEATURE_USABLE
and CPU_FEATURE_USABLE_P to check if a feature is usable.
6. Add DEPR_FPU_CS_DS for INDEX_7_EBX_13.
7. Unset MPX feature since it has been deprecated.
The results are
1. If the feature is known and doesn't requre OS support, its usable bit
is copied from the cpuid bit.
2. Otherwise, its usable bit is copied from the cpuid bit only if the
feature is known to supported by OS.
3. CPU_FEATURE_USABLE/CPU_FEATURE_USABLE_P are used to check if the
feature can be used.
4. HAS_CPU_FEATURE/CPU_FEATURE_CPU_P are used to check if CPU supports
the feature.
Since
commit 430388d5dc
Author: H.J. Lu <hjl.tools@gmail.com>
Date: Fri Aug 3 08:04:49 2018 -0700
x86: Don't include <init-arch.h> in assembly codes
removed all usages of <init-arch.h> from assembly codes, we can remove
__ASSEMBLER__ check in init-arch.h.
Since
commit c867597bff
Author: H.J. Lu <hjl.tools@gmail.com>
Date: Wed Jun 8 13:57:50 2016 -0700
X86-64: Remove previous default/SSE2/AVX2 memcpy/memmove
removed the only usage of __x86_prefetchw, we can remove the unused
__x86_prefetchw.
A big shoutout to Cupertino Miranda <cmiranda@synopsys.com> for his
valuable contribution in initial bringup and debugging on Linux and
later in solving pesky unwinding/cancelation failures in testsuite.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Teach the linker that __mcount_internal, __sigjmp_save_symbol,
__syscall_error and __GI_exit do not use r2, so that it does not need to
recover r2 after the call.
Test at configure time if the assembler supports @notoc and define
USE_PPC64_NOTOC.
Make the instructions for syscall list generation match Makefile and
refer to `update-syscall-lists'; there has been no `update-arch-syscall'
target. Also use single quotes around the command to stick to the ASCII
character set.
Fixes 4cf0d22305 ("Linux: Add tables with system call numbers").
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
To provide a y2038 safe interface a new symbol __shmctl64 is added
and __shmctl is change to call it instead (it adds some extra buffer
copying for the 32 bit time_t implementation).
Two new structures are added:
1. kernel_shmid64_ds: used internally only on 32-bit architectures
to issue the syscall. A handful of architectures (hppa, i386,
mips, powerpc32, and sparc32) require specific implementations
due to their kernel ABI.
2. shmid_ds64: this is only for __TIMESIZE != 64 to use along with
the 64-bit shmctl. It is different than the kernel struct because
the exported 64-bit time_t might require different alignment
depending on the architecture ABI.
So the resulting implementation does:
1. For 64-bit architectures it assumes shmid_ds already contains
64-bit time_t fields and will result in just the __shmctl symbol
using the __shmctl64 code. The shmid_ds argument is passed as-is
to the syscall.
2. For 32-bit architectures with default 64-bit time_t (newer ABIs
such riscv32 or arc), it will also result in only one exported
symbol but with the required high/low time handling.
3. Finally for 32-bit architecture with both 32-bit and 64-bit time_t
support we follow the already set way to provide one symbol with
64-bit time_t support and implement the 32-bit time_t support
using of the 64-bit one.
The default 32-bit symbol will allocate and copy the shmid_ds
over multiple buffers, but this should be deprecated in favor
of the __shmctl64 anyway.
Checked on i686-linux-gnu and x86_64-linux-gnu. I also did some sniff
tests on powerpc, powerpc64, mips, mips64, armhf, sparcv9, and
sparc64.
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Each architecture overrides the struct msqid_ds which its required
kernel ABI one.
Checked on x86_64-linux-gnu and some bases sysvipc tests on hppa,
mips, mipsle, mips64, mips64le, sparc64, sparcv9, powerpc64le,
powerpc64, and powerpc.
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This will allow us to have architectures specify their own version.
Not semantic changes expected. Checked with a build against the
all affected ABIs.
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
To provide a y2038 safe interface a new symbol __msgctl64 is added
and __msgctl is change to call it instead (it adds some extra buffer
coping for the 32 bit time_t implementation).
Two new structures are added:
1. kernel_msqid64_ds: used internally only on 32-bit architectures
to issue the syscall. A handful of architectures (hppa, i386, mips,
powerpc32, and sparc32) require specific implementations due to
their kernel ABI.
2. msqid_ds64: this is only for __TIMESIZE != 64 to use along with
the 64-bit msgctl. It is different than the kernel struct because
the exported 64-bit time_t might require different alignment
depending on the architecture ABI.
So the resulting implementation does:
1. For 64-bit architectures it assumes msqid_ds already contains
64-bit time_t fields and will result in just the __msgctl symbol
using the __msgctl64 code. The msgid_ds argument is passed as-is
to the syscall.
2. For 32-bit architectures with default 64-bit time_t (newer ABIs
such riscv32 or arc), it will also result in only one exported
symbol but with the required high/low time handling.
3. Finally for 32-bit architecture with both 32-bit and 64-bit time_t
support we follow the already set way to provide one symbol with
64-bit time_t support and implement the 32-bit time_t support using
the 64-bit time_t.
The default 32-bit symbol will allocate and copy the msqid_ds
over multiple buffers, but this should be deprecated in favor
of the __msgctl64 anyway.
Checked on i686-linux-gnu and x86_64-linux-gnu. I also did some sniff
tests on powerpc, powerpc64, mips, mips64, armhf, sparcv9, and
sparc64.
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Each architecture overrides the struct msqid_ds which its required
kernel ABI one.
Checked on x86_64-linux-gnu and some bases sysvipc tests on hppa,
mips, mipsle, mips64, mips64le, sparc64, sparcv9, powerpc64le,
powerpc64, and powerpc.
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
This will allow us to have architectures specify their own version.
Not semantic changes expected. Checked with a build against the
all affected ABIs.
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Different than others 64-bit time_t syscalls, the SysIPC interface
does not provide a new set of syscall for y2038 safeness. Instead it
uses unused fields in semid_ds structure to return the high bits for
the timestamps.
To provide a y2038 safe interface a new symbol __semctl64 is added
and __semctl is change to call it instead (it adds some extra buffer
copying for the 32 bit time_t implementation).
Two new structures are added:
1. kernel_semid64_ds: used internally only on 32-bit architectures
to issue the syscall. A handful of architectures (hppa, i386,
mips, powerpc32, sparc32) require specific implementations due
their kernel ABI.
2. semid_ds64: this is only for __TIMESIZE != 64 to use along with
the 64-bit semctl. It is different than the kernel struct because
the exported 64-bit time_t might require different alignment
depending on the architecture ABI.
So the resulting implementation does:
1. For 64-bit architectures it assumes semid_ds already contains
64-bit time_t fields and will result in just the __semctl symbol
using the __semctl64 code. The semid_ds argument is passed as-is
to the syscall.
2. For 32-bit architectures with default 64-bit time_t (newer ABIs
such riscv32 or arc), it will also result in only one exported
symbol but with the required high/low handling.
It might be possible to optimize it further to avoid the
kernel_semid64_ds to semun transformation if the exported ABI
for the architectures matches the expected kernel ABI, but the
implementation is already complex enough and don't think this
should be a hotspot in any case.
3. Finally for 32-bit architecture with both 32-bit and 64-bit time_t
support we follow the already set way to provide one symbol with
64-bit time_t support and implement the 32-bit time_t support
using the 64-bit one.
The default 32-bit symbol will allocate and copy the semid_ds
over multiple buffers, but this should be deprecated in favor
of the __semctl64 anyway.
Checked on i686-linux-gnu and x86_64-linux-gnu. I also did some sniff
tests on powerpc, powerpc64, mips, mips64, armhf, sparcv9, and
sparc64.
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Alistair Francis <alistair.francis@wdc.com>
Tested-by: Vineet Gupta <vgupta@synopsys.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
On some targets static TLS surplus area can be used opportunistically
for dynamically loaded modules such that the TLS access then becomes
faster (TLSDESC and powerpc TLS optimization). However we don't want
all surplus TLS to be used for this optimization because dynamically
loaded modules with initial-exec model TLS can only use surplus TLS.
The new contract for surplus static TLS use is:
- libc.so can have up to 192 bytes of IE TLS,
- other system libraries together can have up to 144 bytes of IE TLS.
- Some "optional" static TLS is available for opportunistic use.
The optional TLS is now tunable: rtld.optional_static_tls, so users
can directly affect the allocated static TLS size. (Note that module
unloading with dlclose does not reclaim static TLS. After the optional
TLS runs out, TLS access is no longer optimized to use static TLS.)
The default setting of rtld.optional_static_tls is 512 so the surplus
TLS is 3*192 + 4*144 + 512 = 1664 by default, the same as before.
Fixes BZ #25051.
Tested on aarch64-linux-gnu and x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The new static TLS surplus size computation is
surplus_tls = 192 * (nns-1) + 144 * nns + 512
where nns is controlled via the rtld.nns tunable. This commit
accounts audit modules too so nns = rtld.nns + audit modules.
rtld.nns should only include the namespaces required by the
application, namespaces for audit modules are accounted on top
of that so audit modules don't use up the static TLS that is
reserved for the application. This allows loading many audit
modules without tuning rtld.nns or using up static TLS, and it
fixes
FAIL: elf/tst-auditmany
Note that DL_NNS is currently a hard upper limit for nns, and
if rtld.nns + audit modules go over the limit that's a fatal
error. By default rtld.nns is 4 which allows 12 audit modules.
Counting the audit modules is based on existing audit string
parsing code, we cannot use GLRO(dl_naudit) before the modules
are actually loaded.
TLS_STATIC_SURPLUS is 1664 bytes currently which is not enough to
support DL_NNS (== 16) number of dynamic link namespaces, if we
assume 192 bytes of TLS are reserved for libc use and 144 bytes
are reserved for other system libraries that use IE TLS.
A new tunable is introduced to control the number of supported
namespaces and to adjust the surplus static TLS size as follows:
surplus_tls = 192 * (rtld.nns-1) + 144 * rtld.nns + 512
The default is rtld.nns == 4 and then the surplus TLS size is the
same as before, so the behaviour is unchanged by default. If an
application creates more namespaces than the rtld.nns setting
allows, then it is not guaranteed to work, but the limit is not
checked. So existing usage will continue to work, but in the
future if an application creates more than 4 dynamic link
namespaces then the tunable will need to be set.
In this patch DL_NNS is a fixed value and provides a maximum to
the rtld.nns setting.
Static linking used fixed 2048 bytes surplus TLS, this is changed
so the same contract is used as for dynamic linking. With static
linking DL_NNS == 1 so rtld.nns tunable is forced to 1, so by
default the surplus TLS is reduced to 144 + 512 = 656 bytes. This
change is not expected to cause problems.
Tested on aarch64-linux-gnu and x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
this means that *always* libnsl is only built as shared library for
backward compatibility and the NSS modules libnss_nis and libnss_nisplus
are not built at all, libnsl's headers aren't installed.
This compatibility is kept only for architectures and ABIs that have
been added in or before version 2.28.
Replacement implementations based on TIRPC, which additionally support
IPv6, are available from <https://github.com/thkukuk/>.
This change does not affect libnss_compat which does not depended
on libnsl since 2.27 and thus can be used without NIS.
libnsl code depends on Sun RPC, e.g. on --enable-obsolete-rpc (installed
libnsl headers use installed Sun RPC headers), which will be removed in
the following commit.
RETURN_ADDRESS is used at several places in glibc to mean a valid
code address of the call site, but with pac-ret it may contain a
pointer authentication code (PAC), so its definition is adjusted.
This is gcc PR target/94891: __builtin_return_address should not
expose signed pointers to user code where it can cause ABI issues.
In glibc RETURN_ADDRESS is only changed if it is built with pac-ret.
There is no detection for the specific gcc issue because it is
hard to test and the additional xpac does not cause problems.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Currently gcc -pg -mbranch-protection=pac-ret passes signed return
address to _mcount, so _mcount now has to always strip pac from the
frompc since that's from user code that may be built with pac-ret.
This is gcc PR target/94791: signed pointers should not escape and get
passed across extern call boundaries, since that's an ABI break, but
because existing gcc has this issue we work it around in glibc until
that is resolved. This is compatible with a fixed gcc and it is a nop
on systems without PAuth support. The bug was introduced in gcc-7 with
-msign-return-address=non-leaf|all support which in gcc-9 got renamed
to -mbranch-protection=pac-ret|pac-ret+leaf|standard.
strip_pac uses inline asm instead of __builtin_aarch64_xpaclri since
that is not a documented api and not available in all supported gccs.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Use return address signing in assembly files for functions that save
LR when pac-ret is enabled in the compiler.
The GNU property note for PAC-RET is not meaningful to the dynamic
linker so it is not strictly required, but it may be used to track
the security property of binaries. (The PAC-RET property is only set
if BTI is set too because BTI implies working GNU property support.)
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Return address signing requires unwinder support, which is
present in libgcc since >=gcc-7, however due to bugs the
support may be broken in <gcc-10 (and similarly there may
be issues in custom unwinders), so pac-ret is not always
safe to use. So in assembly code glibc should only use
pac-ret if the compiler uses it too. Unfortunately there
is no predefined feature macro for it set by the compiler
so pac-ret is inferred from the code generation.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
When glibc is built with branch protection (i.e. with a gcc configured
with --enable-standard-branch-protection), all glibc binaries should
be BTI compatible and marked as such.
It is easy to link BTI incompatible objects by accident and this is
silent currently which is usually not the expectation, so this is
changed into a link error. (There is no linker flag for failing on
BTI incompatible inputs so all warnings are turned into fatal errors
outside the test system when building glibc with branch protection.)
Unfortunately, outlined atomic functions are not BTI compatible in
libgcc (PR libgcc/96001), so to build glibc with current gcc use
'CC=gcc -mno-outline-atomics', this should be fixed in libgcc soon
and then glibc can be built and tested without such workarounds.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
