2009-03-13 23:53:18 +00:00
|
|
|
/* Initialize CPU feature data.
|
|
|
|
|
This file is part of the GNU C Library.
|
2021-01-02 19:32:25 +00:00
|
|
|
Copyright (C) 2008-2021 Free Software Foundation, Inc.
|
2009-03-13 23:53:18 +00:00
|
|
|
|
|
|
|
|
The GNU C Library is free software; you can redistribute it and/or
|
|
|
|
|
modify it under the terms of the GNU Lesser General Public
|
|
|
|
|
License as published by the Free Software Foundation; either
|
|
|
|
|
version 2.1 of the License, or (at your option) any later version.
|
|
|
|
|
|
|
|
|
|
The GNU C Library is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
|
|
|
Lesser General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU Lesser General Public
|
2012-02-09 23:18:22 +00:00
|
|
|
License along with the GNU C Library; if not, see
|
Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:
sed -ri '
s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
$(find $(git ls-files) -prune -type f \
! -name '*.po' \
! -name 'ChangeLog*' \
! -path COPYING ! -path COPYING.LIB \
! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
! -path manual/texinfo.tex ! -path scripts/config.guess \
! -path scripts/config.sub ! -path scripts/install-sh \
! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
! -path INSTALL ! -path locale/programs/charmap-kw.h \
! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
! '(' -name configure \
-execdir test -f configure.ac -o -f configure.in ';' ')' \
! '(' -name preconfigure \
-execdir test -f preconfigure.ac ';' ')' \
-print)
and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:
chmod a+x sysdeps/unix/sysv/linux/riscv/configure
# Omit irrelevant whitespace and comment-only changes,
# perhaps from a slightly-different Autoconf version.
git checkout -f \
sysdeps/csky/configure \
sysdeps/hppa/configure \
sysdeps/riscv/configure \
sysdeps/unix/sysv/linux/csky/configure
# Omit changes that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
git checkout -f \
sysdeps/powerpc/powerpc64/ppc-mcount.S \
sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
# Omit change that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
|
|
|
<https://www.gnu.org/licenses/>. */
|
2009-03-13 23:53:18 +00:00
|
|
|
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
#include <dl-hwcap.h>
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
#include <libc-pointer-arith.h>
|
2020-10-09 13:06:56 +00:00
|
|
|
#include <get-isa-level.h>
|
2020-09-18 14:55:14 +00:00
|
|
|
#include <cacheinfo.h>
|
|
|
|
|
#include <dl-cacheinfo.h>
|
2021-02-01 19:00:38 +00:00
|
|
|
#include <dl-minsigstacksize.h>
|
2009-03-13 23:53:18 +00:00
|
|
|
|
tunables: Add IFUNC selection and cache sizes
The current IFUNC selection is based on microbenchmarks in glibc. It
should give the best performance for most workloads. But other choices
may have better performance for a particular workload or on the hardware
which wasn't available at the selection was made. The environment
variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
where the feature name is case-sensitive and has to match the ones in
cpu-features.h. It can be used by glibc developers to override the
IFUNC selection to tune for a new processor or improve performance for
a particular workload. It isn't intended for normal end users.
NOTE: the IFUNC selection may change over time. Please check all
multiarch implementations when experimenting.
Also, GLIBC_TUNABLES=glibc.tune.x86_non_temporal_threshold=NUMBER is
provided to set threshold to use non temporal store to NUMBER,
GLIBC_TUNABLES=glibc.tune.x86_data_cache_size=NUMBER to set data cache
size, GLIBC_TUNABLES=glibc.tune.x86_shared_cache_size=NUMBER to set
shared cache size.
* elf/dl-tunables.list (tune): Add ifunc,
x86_non_temporal_threshold,
x86_data_cache_size and x86_shared_cache_size.
* manual/tunables.texi: Document glibc.tune.ifunc,
glibc.tune.x86_data_cache_size, glibc.tune.x86_shared_cache_size
and glibc.tune.x86_non_temporal_threshold.
* sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
* sysdeps/x86/cpu-tunables.c: Likewise.
* sysdeps/x86/cacheinfo.c
(init_cacheinfo): Check and get data cache size, shared cache
size and non temporal threshold from cpu_features.
* sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
New.
[HAVE_TUNABLES] Include <unistd.h>.
[HAVE_TUNABLES] Include <elf/dl-tunables.h>.
[HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
[HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
IFUNC selection, data cache size, shared cache size and non
temporal threshold.
* sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
shared_cache_size and non_temporal_threshold.
2017-06-20 15:33:29 +00:00
|
|
|
#if HAVE_TUNABLES
|
2017-06-21 17:20:24 +00:00
|
|
|
extern void TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *)
|
tunables: Add IFUNC selection and cache sizes
The current IFUNC selection is based on microbenchmarks in glibc. It
should give the best performance for most workloads. But other choices
may have better performance for a particular workload or on the hardware
which wasn't available at the selection was made. The environment
variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
where the feature name is case-sensitive and has to match the ones in
cpu-features.h. It can be used by glibc developers to override the
IFUNC selection to tune for a new processor or improve performance for
a particular workload. It isn't intended for normal end users.
NOTE: the IFUNC selection may change over time. Please check all
multiarch implementations when experimenting.
Also, GLIBC_TUNABLES=glibc.tune.x86_non_temporal_threshold=NUMBER is
provided to set threshold to use non temporal store to NUMBER,
GLIBC_TUNABLES=glibc.tune.x86_data_cache_size=NUMBER to set data cache
size, GLIBC_TUNABLES=glibc.tune.x86_shared_cache_size=NUMBER to set
shared cache size.
* elf/dl-tunables.list (tune): Add ifunc,
x86_non_temporal_threshold,
x86_data_cache_size and x86_shared_cache_size.
* manual/tunables.texi: Document glibc.tune.ifunc,
glibc.tune.x86_data_cache_size, glibc.tune.x86_shared_cache_size
and glibc.tune.x86_non_temporal_threshold.
* sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
* sysdeps/x86/cpu-tunables.c: Likewise.
* sysdeps/x86/cacheinfo.c
(init_cacheinfo): Check and get data cache size, shared cache
size and non temporal threshold from cpu_features.
* sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
New.
[HAVE_TUNABLES] Include <unistd.h>.
[HAVE_TUNABLES] Include <elf/dl-tunables.h>.
[HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
[HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
IFUNC selection, data cache size, shared cache size and non
temporal threshold.
* sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
shared_cache_size and non_temporal_threshold.
2017-06-20 15:33:29 +00:00
|
|
|
attribute_hidden;
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
|
|
|
|
|
# if CET_ENABLED
|
|
|
|
|
extern void TUNABLE_CALLBACK (set_x86_ibt) (tunable_val_t *)
|
|
|
|
|
attribute_hidden;
|
|
|
|
|
extern void TUNABLE_CALLBACK (set_x86_shstk) (tunable_val_t *)
|
|
|
|
|
attribute_hidden;
|
|
|
|
|
# endif
|
tunables: Add IFUNC selection and cache sizes
The current IFUNC selection is based on microbenchmarks in glibc. It
should give the best performance for most workloads. But other choices
may have better performance for a particular workload or on the hardware
which wasn't available at the selection was made. The environment
variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
where the feature name is case-sensitive and has to match the ones in
cpu-features.h. It can be used by glibc developers to override the
IFUNC selection to tune for a new processor or improve performance for
a particular workload. It isn't intended for normal end users.
NOTE: the IFUNC selection may change over time. Please check all
multiarch implementations when experimenting.
Also, GLIBC_TUNABLES=glibc.tune.x86_non_temporal_threshold=NUMBER is
provided to set threshold to use non temporal store to NUMBER,
GLIBC_TUNABLES=glibc.tune.x86_data_cache_size=NUMBER to set data cache
size, GLIBC_TUNABLES=glibc.tune.x86_shared_cache_size=NUMBER to set
shared cache size.
* elf/dl-tunables.list (tune): Add ifunc,
x86_non_temporal_threshold,
x86_data_cache_size and x86_shared_cache_size.
* manual/tunables.texi: Document glibc.tune.ifunc,
glibc.tune.x86_data_cache_size, glibc.tune.x86_shared_cache_size
and glibc.tune.x86_non_temporal_threshold.
* sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
* sysdeps/x86/cpu-tunables.c: Likewise.
* sysdeps/x86/cacheinfo.c
(init_cacheinfo): Check and get data cache size, shared cache
size and non temporal threshold from cpu_features.
* sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
New.
[HAVE_TUNABLES] Include <unistd.h>.
[HAVE_TUNABLES] Include <elf/dl-tunables.h>.
[HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
[HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
IFUNC selection, data cache size, shared cache size and non
temporal threshold.
* sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
shared_cache_size and non_temporal_threshold.
2017-06-20 15:33:29 +00:00
|
|
|
#endif
|
|
|
|
|
|
2018-07-17 11:16:19 +00:00
|
|
|
#if CET_ENABLED
|
|
|
|
|
# include <dl-cet.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-07-26 17:49:19 +00:00
|
|
|
static void
|
2021-06-05 13:42:20 +00:00
|
|
|
update_active (struct cpu_features *cpu_features)
|
2018-07-26 17:49:19 +00:00
|
|
|
{
|
2021-06-05 13:42:20 +00:00
|
|
|
/* Copy the cpuid bits to active bits for CPU featuress whose usability
|
2020-08-05 20:51:56 +00:00
|
|
|
in user space can be detected without additonal OS support. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE3);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, PCLMULQDQ);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSSE3);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CMPXCHG16B);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE4_1);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE4_2);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, MOVBE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, POPCNT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AES);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, OSXSAVE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, TSC);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CX8);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CMOV);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CLFSH);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, MMX);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FXSR);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE2);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, HTT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, BMI1);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, HLE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, BMI2);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, ERMS);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RDSEED);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, ADX);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CLFLUSHOPT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CLWB);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SHA);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, PREFETCHWT1);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, OSPKE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, WAITPKG);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, GFNI);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RDPID);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RDRAND);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, CLDEMOTE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, MOVDIRI);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, MOVDIR64B);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FSRM);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RTM_ALWAYS_ABORT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SERIALIZE);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, TSXLDTRK);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, LAHF64_SAHF64);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, LZCNT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SSE4A);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, PREFETCHW);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, TBM);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RDTSCP);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, WBNOINVD);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FZLRM);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FSRS);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FSRCS);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, PTWRITE);
|
2018-12-03 13:54:43 +00:00
|
|
|
|
2021-06-30 17:47:06 +00:00
|
|
|
if (!CPU_FEATURES_CPU_P (cpu_features, RTM_ALWAYS_ABORT))
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, RTM);
|
2021-06-30 17:47:06 +00:00
|
|
|
|
2021-06-23 21:27:58 +00:00
|
|
|
#if CET_ENABLED
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, IBT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, SHSTK);
|
2021-06-23 21:27:58 +00:00
|
|
|
#endif
|
|
|
|
|
|
2016-03-22 14:46:56 +00:00
|
|
|
/* Can we call xgetbv? */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, OSXSAVE))
|
|
|
|
|
{
|
|
|
|
|
unsigned int xcrlow;
|
|
|
|
|
unsigned int xcrhigh;
|
|
|
|
|
asm ("xgetbv" : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
|
|
|
|
|
/* Is YMM and XMM state usable? */
|
2019-02-25 13:19:19 +00:00
|
|
|
if ((xcrlow & (bit_YMM_state | bit_XMM_state))
|
|
|
|
|
== (bit_YMM_state | bit_XMM_state))
|
2016-03-22 14:46:56 +00:00
|
|
|
{
|
|
|
|
|
/* Determine if AVX is usable. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, AVX))
|
2016-10-17 23:35:34 +00:00
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_SET (cpu_features, AVX);
|
2016-10-17 23:35:34 +00:00
|
|
|
/* The following features depend on AVX being usable. */
|
|
|
|
|
/* Determine if AVX2 is usable. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, AVX2))
|
2020-06-29 23:36:08 +00:00
|
|
|
{
|
|
|
|
|
CPU_FEATURE_SET (cpu_features, AVX2);
|
|
|
|
|
|
|
|
|
|
/* Unaligned load with 256-bit AVX registers are faster
|
|
|
|
|
on Intel/AMD processors with AVX2. */
|
|
|
|
|
cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load]
|
|
|
|
|
|= bit_arch_AVX_Fast_Unaligned_Load;
|
|
|
|
|
}
|
2020-10-08 15:33:45 +00:00
|
|
|
/* Determine if AVX-VNNI is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX_VNNI);
|
2016-10-17 23:35:34 +00:00
|
|
|
/* Determine if FMA is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FMA);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if VAES is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, VAES);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if VPCLMULQDQ is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, VPCLMULQDQ);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if XOP is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, XOP);
|
2020-06-18 12:34:15 +00:00
|
|
|
/* Determine if F16C is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, F16C);
|
2016-10-17 23:35:34 +00:00
|
|
|
}
|
|
|
|
|
|
2016-03-22 14:46:56 +00:00
|
|
|
/* Check if OPMASK state, upper 256-bit of ZMM0-ZMM15 and
|
|
|
|
|
ZMM16-ZMM31 state are enabled. */
|
|
|
|
|
if ((xcrlow & (bit_Opmask_state | bit_ZMM0_15_state
|
2019-02-25 13:19:19 +00:00
|
|
|
| bit_ZMM16_31_state))
|
|
|
|
|
== (bit_Opmask_state | bit_ZMM0_15_state | bit_ZMM16_31_state))
|
2016-03-22 14:46:56 +00:00
|
|
|
{
|
|
|
|
|
/* Determine if AVX512F is usable. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, AVX512F))
|
|
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_SET (cpu_features, AVX512F);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512CD is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512CD);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512ER is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512ER);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512PF is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512PF);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512VL is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512VL);
|
2016-03-22 14:46:56 +00:00
|
|
|
/* Determine if AVX512DQ is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512DQ);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512BW is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512BW);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_4FMAPS is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_4FMAPS);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_4VNNIW is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_4VNNIW);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_BITALG is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_BITALG);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_IFMA is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_IFMA);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_VBMI is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_VBMI);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_VBMI2 is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_VBMI2);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if is AVX512_VNNI usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_VNNI);
|
2018-12-03 13:54:43 +00:00
|
|
|
/* Determine if AVX512_VPOPCNTDQ is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features,
|
2020-06-29 23:36:08 +00:00
|
|
|
AVX512_VPOPCNTDQ);
|
2020-06-17 13:34:46 +00:00
|
|
|
/* Determine if AVX512_VP2INTERSECT is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features,
|
2020-06-29 23:36:08 +00:00
|
|
|
AVX512_VP2INTERSECT);
|
2020-06-17 13:34:46 +00:00
|
|
|
/* Determine if AVX512_BF16 is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_BF16);
|
2020-10-08 15:24:47 +00:00
|
|
|
/* Determine if AVX512_FP16 is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AVX512_FP16);
|
2016-03-22 14:46:56 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
|
2020-06-25 22:12:57 +00:00
|
|
|
/* Are XTILECFG and XTILEDATA states usable? */
|
|
|
|
|
if ((xcrlow & (bit_XTILECFG_state | bit_XTILEDATA_state))
|
|
|
|
|
== (bit_XTILECFG_state | bit_XTILEDATA_state))
|
|
|
|
|
{
|
|
|
|
|
/* Determine if AMX_BF16 is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AMX_BF16);
|
2020-06-25 22:12:57 +00:00
|
|
|
/* Determine if AMX_TILE is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AMX_TILE);
|
2020-06-25 22:12:57 +00:00
|
|
|
/* Determine if AMX_INT8 is usable. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, AMX_INT8);
|
2020-06-25 22:12:57 +00:00
|
|
|
}
|
|
|
|
|
|
2020-08-05 20:51:56 +00:00
|
|
|
/* These features are usable only when OSXSAVE is enabled. */
|
|
|
|
|
CPU_FEATURE_SET (cpu_features, XSAVE);
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, XSAVEOPT);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, XSAVEC);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, XGETBV_ECX_1);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, XFD);
|
2020-06-29 23:36:08 +00:00
|
|
|
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
/* For _dl_runtime_resolve, set xsave_state_size to xsave area
|
|
|
|
|
size + integer register save size and align it to 64 bytes. */
|
2018-12-03 13:54:43 +00:00
|
|
|
if (cpu_features->basic.max_cpuid >= 0xd)
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
{
|
|
|
|
|
unsigned int eax, ebx, ecx, edx;
|
|
|
|
|
|
|
|
|
|
__cpuid_count (0xd, 0, eax, ebx, ecx, edx);
|
|
|
|
|
if (ebx != 0)
|
|
|
|
|
{
|
|
|
|
|
unsigned int xsave_state_full_size
|
|
|
|
|
= ALIGN_UP (ebx + STATE_SAVE_OFFSET, 64);
|
|
|
|
|
|
|
|
|
|
cpu_features->xsave_state_size
|
|
|
|
|
= xsave_state_full_size;
|
|
|
|
|
cpu_features->xsave_state_full_size
|
|
|
|
|
= xsave_state_full_size;
|
|
|
|
|
|
|
|
|
|
/* Check if XSAVEC is available. */
|
2018-12-03 13:54:43 +00:00
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, XSAVEC))
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
{
|
|
|
|
|
unsigned int xstate_comp_offsets[32];
|
|
|
|
|
unsigned int xstate_comp_sizes[32];
|
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
|
|
xstate_comp_offsets[0] = 0;
|
|
|
|
|
xstate_comp_offsets[1] = 160;
|
|
|
|
|
xstate_comp_offsets[2] = 576;
|
|
|
|
|
xstate_comp_sizes[0] = 160;
|
|
|
|
|
xstate_comp_sizes[1] = 256;
|
|
|
|
|
|
|
|
|
|
for (i = 2; i < 32; i++)
|
|
|
|
|
{
|
|
|
|
|
if ((STATE_SAVE_MASK & (1 << i)) != 0)
|
|
|
|
|
{
|
|
|
|
|
__cpuid_count (0xd, i, eax, ebx, ecx, edx);
|
|
|
|
|
xstate_comp_sizes[i] = eax;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
ecx = 0;
|
|
|
|
|
xstate_comp_sizes[i] = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (i > 2)
|
|
|
|
|
{
|
|
|
|
|
xstate_comp_offsets[i]
|
|
|
|
|
= (xstate_comp_offsets[i - 1]
|
|
|
|
|
+ xstate_comp_sizes[i -1]);
|
|
|
|
|
if ((ecx & (1 << 1)) != 0)
|
|
|
|
|
xstate_comp_offsets[i]
|
|
|
|
|
= ALIGN_UP (xstate_comp_offsets[i], 64);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Use XSAVEC. */
|
|
|
|
|
unsigned int size
|
|
|
|
|
= xstate_comp_offsets[31] + xstate_comp_sizes[31];
|
|
|
|
|
if (size)
|
|
|
|
|
{
|
|
|
|
|
cpu_features->xsave_state_size
|
|
|
|
|
= ALIGN_UP (size + STATE_SAVE_OFFSET, 64);
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_SET (cpu_features, XSAVEC);
|
x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
(get_common_indeces): Set xsave_state_size, xsave_state_full_size
and bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 18:00:08 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2015-11-30 16:53:37 +00:00
|
|
|
}
|
2020-06-17 13:34:46 +00:00
|
|
|
|
|
|
|
|
/* Determine if PKU is usable. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, OSPKE))
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_SET (cpu_features, PKU);
|
2020-09-15 12:49:27 +00:00
|
|
|
|
|
|
|
|
/* Determine if Key Locker instructions are usable. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, AESKLE))
|
|
|
|
|
{
|
|
|
|
|
CPU_FEATURE_SET (cpu_features, AESKLE);
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, KL);
|
|
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, WIDE_KL);
|
2020-09-15 12:49:27 +00:00
|
|
|
}
|
2020-10-09 13:06:56 +00:00
|
|
|
|
|
|
|
|
cpu_features->isa_1 = get_isa_level (cpu_features);
|
2020-06-29 23:36:08 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
get_extended_indices (struct cpu_features *cpu_features)
|
|
|
|
|
{
|
|
|
|
|
unsigned int eax, ebx, ecx, edx;
|
|
|
|
|
__cpuid (0x80000000, eax, ebx, ecx, edx);
|
|
|
|
|
if (eax >= 0x80000001)
|
|
|
|
|
__cpuid (0x80000001,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_80000001].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000001].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000001].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000001].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
if (eax >= 0x80000007)
|
|
|
|
|
__cpuid (0x80000007,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_80000007].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000007].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000007].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000007].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
if (eax >= 0x80000008)
|
|
|
|
|
__cpuid (0x80000008,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_80000008].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000008].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000008].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_80000008].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
get_common_indices (struct cpu_features *cpu_features,
|
|
|
|
|
unsigned int *family, unsigned int *model,
|
|
|
|
|
unsigned int *extended_model, unsigned int *stepping)
|
|
|
|
|
{
|
|
|
|
|
if (family)
|
|
|
|
|
{
|
|
|
|
|
unsigned int eax;
|
|
|
|
|
__cpuid (1, eax,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_1].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_1].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_1].cpuid.edx);
|
|
|
|
|
cpu_features->features[CPUID_INDEX_1].cpuid.eax = eax;
|
2020-06-29 23:36:08 +00:00
|
|
|
*family = (eax >> 8) & 0x0f;
|
|
|
|
|
*model = (eax >> 4) & 0x0f;
|
|
|
|
|
*extended_model = (eax >> 12) & 0xf0;
|
|
|
|
|
*stepping = eax & 0x0f;
|
|
|
|
|
if (*family == 0x0f)
|
|
|
|
|
{
|
|
|
|
|
*family += (eax >> 20) & 0xff;
|
|
|
|
|
*model += *extended_model;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (cpu_features->basic.max_cpuid >= 7)
|
|
|
|
|
{
|
|
|
|
|
__cpuid_count (7, 0,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_7].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
__cpuid_count (7, 1,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_7_ECX_1].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7_ECX_1].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7_ECX_1].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_7_ECX_1].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (cpu_features->basic.max_cpuid >= 0xd)
|
|
|
|
|
__cpuid_count (0xd, 1,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_D_ECX_1].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_D_ECX_1].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_D_ECX_1].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_D_ECX_1].cpuid.edx);
|
2020-06-29 23:36:08 +00:00
|
|
|
|
2021-02-04 18:39:34 +00:00
|
|
|
if (cpu_features->basic.max_cpuid >= 0x14)
|
|
|
|
|
__cpuid_count (0x14, 0,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_14_ECX_0].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_14_ECX_0].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_14_ECX_0].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_14_ECX_0].cpuid.edx);
|
|
|
|
|
|
2020-09-15 12:49:27 +00:00
|
|
|
if (cpu_features->basic.max_cpuid >= 0x19)
|
|
|
|
|
__cpuid_count (0x19, 0,
|
2020-12-25 15:30:46 +00:00
|
|
|
cpu_features->features[CPUID_INDEX_19].cpuid.eax,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_19].cpuid.ebx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_19].cpuid.ecx,
|
|
|
|
|
cpu_features->features[CPUID_INDEX_19].cpuid.edx);
|
2021-02-01 19:00:38 +00:00
|
|
|
|
|
|
|
|
dl_check_minsigstacksize (cpu_features);
|
2009-06-30 11:39:09 +00:00
|
|
|
}
|
|
|
|
|
|
2018-12-03 13:54:43 +00:00
|
|
|
_Static_assert (((index_arch_Fast_Unaligned_Load
|
|
|
|
|
== index_arch_Fast_Unaligned_Copy)
|
|
|
|
|
&& (index_arch_Fast_Unaligned_Load
|
|
|
|
|
== index_arch_Prefer_PMINUB_for_stringop)
|
|
|
|
|
&& (index_arch_Fast_Unaligned_Load
|
|
|
|
|
== index_arch_Slow_SSE4_2)
|
|
|
|
|
&& (index_arch_Fast_Unaligned_Load
|
|
|
|
|
== index_arch_Fast_Rep_String)
|
|
|
|
|
&& (index_arch_Fast_Unaligned_Load
|
|
|
|
|
== index_arch_Fast_Copy_Backward)),
|
|
|
|
|
"Incorrect index_arch_Fast_Unaligned_Load");
|
|
|
|
|
|
2015-08-13 10:37:47 +00:00
|
|
|
static inline void
|
|
|
|
|
init_cpu_features (struct cpu_features *cpu_features)
|
2009-03-13 23:53:18 +00:00
|
|
|
{
|
2015-08-13 10:37:47 +00:00
|
|
|
unsigned int ebx, ecx, edx;
|
2010-04-04 07:25:46 +00:00
|
|
|
unsigned int family = 0;
|
|
|
|
|
unsigned int model = 0;
|
2018-12-03 13:54:43 +00:00
|
|
|
unsigned int stepping = 0;
|
2010-04-04 07:25:46 +00:00
|
|
|
enum cpu_features_kind kind;
|
2009-03-13 23:53:18 +00:00
|
|
|
|
2015-08-18 14:59:49 +00:00
|
|
|
#if !HAS_CPUID
|
2015-08-13 11:52:50 +00:00
|
|
|
if (__get_cpuid_max (0, 0) == 0)
|
|
|
|
|
{
|
|
|
|
|
kind = arch_kind_other;
|
|
|
|
|
goto no_cpuid;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-12-03 13:54:43 +00:00
|
|
|
__cpuid (0, cpu_features->basic.max_cpuid, ebx, ecx, edx);
|
2009-03-13 23:53:18 +00:00
|
|
|
|
|
|
|
|
/* This spells out "GenuineIntel". */
|
|
|
|
|
if (ebx == 0x756e6547 && ecx == 0x6c65746e && edx == 0x49656e69)
|
|
|
|
|
{
|
2018-12-03 13:54:43 +00:00
|
|
|
unsigned int extended_model;
|
2015-11-30 16:53:37 +00:00
|
|
|
|
2010-04-04 07:25:46 +00:00
|
|
|
kind = arch_kind_intel;
|
2009-03-13 23:53:18 +00:00
|
|
|
|
2018-08-01 11:57:34 +00:00
|
|
|
get_common_indices (cpu_features, &family, &model, &extended_model,
|
2016-12-19 10:20:31 +00:00
|
|
|
&stepping);
|
2009-06-30 11:39:09 +00:00
|
|
|
|
2018-07-26 17:49:19 +00:00
|
|
|
get_extended_indices (cpu_features);
|
|
|
|
|
|
2021-06-05 13:42:20 +00:00
|
|
|
update_active (cpu_features);
|
2020-06-29 23:36:08 +00:00
|
|
|
|
2015-11-30 16:53:37 +00:00
|
|
|
if (family == 0x06)
|
2010-01-12 19:22:03 +00:00
|
|
|
{
|
2010-04-04 07:25:46 +00:00
|
|
|
model += extended_model;
|
2010-05-27 18:14:18 +00:00
|
|
|
switch (model)
|
2010-01-12 19:22:03 +00:00
|
|
|
{
|
2010-08-25 17:07:37 +00:00
|
|
|
case 0x1c:
|
|
|
|
|
case 0x26:
|
|
|
|
|
/* BSF is slow on Atom. */
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Slow_BSF]
|
2016-03-10 13:26:46 +00:00
|
|
|
|= bit_arch_Slow_BSF;
|
2010-08-25 17:07:37 +00:00
|
|
|
break;
|
|
|
|
|
|
2015-12-15 19:46:54 +00:00
|
|
|
case 0x57:
|
|
|
|
|
/* Knights Landing. Enable Silvermont optimizations. */
|
|
|
|
|
|
2020-05-21 20:36:54 +00:00
|
|
|
case 0x7a:
|
|
|
|
|
/* Unaligned load versions are faster than SSSE3
|
|
|
|
|
on Goldmont Plus. */
|
|
|
|
|
|
2016-04-15 12:22:53 +00:00
|
|
|
case 0x5c:
|
|
|
|
|
case 0x5f:
|
|
|
|
|
/* Unaligned load versions are faster than SSSE3
|
|
|
|
|
on Goldmont. */
|
|
|
|
|
|
|
|
|
|
case 0x4c:
|
2020-05-21 20:36:54 +00:00
|
|
|
case 0x5a:
|
|
|
|
|
case 0x75:
|
2016-04-15 12:22:53 +00:00
|
|
|
/* Airmont is a die shrink of Silvermont. */
|
|
|
|
|
|
2013-06-14 18:46:15 +00:00
|
|
|
case 0x37:
|
2015-01-24 01:27:09 +00:00
|
|
|
case 0x4a:
|
|
|
|
|
case 0x4d:
|
2015-01-24 02:52:45 +00:00
|
|
|
case 0x5d:
|
2013-06-14 18:46:15 +00:00
|
|
|
/* Unaligned load versions are faster than SSSE3
|
|
|
|
|
on Silvermont. */
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Fast_Unaligned_Load]
|
2016-03-10 13:26:46 +00:00
|
|
|
|= (bit_arch_Fast_Unaligned_Load
|
2016-03-28 11:39:48 +00:00
|
|
|
| bit_arch_Fast_Unaligned_Copy
|
2016-03-10 13:26:46 +00:00
|
|
|
| bit_arch_Prefer_PMINUB_for_stringop
|
|
|
|
|
| bit_arch_Slow_SSE4_2);
|
2013-06-14 18:46:15 +00:00
|
|
|
break;
|
|
|
|
|
|
2020-05-21 20:36:54 +00:00
|
|
|
case 0x86:
|
|
|
|
|
case 0x96:
|
|
|
|
|
case 0x9c:
|
|
|
|
|
/* Enable rep string instructions, unaligned load, unaligned
|
|
|
|
|
copy, pminub and avoid SSE 4.2 on Tremont. */
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Fast_Rep_String]
|
2020-05-21 20:36:54 +00:00
|
|
|
|= (bit_arch_Fast_Rep_String
|
|
|
|
|
| bit_arch_Fast_Unaligned_Load
|
|
|
|
|
| bit_arch_Fast_Unaligned_Copy
|
|
|
|
|
| bit_arch_Prefer_PMINUB_for_stringop
|
|
|
|
|
| bit_arch_Slow_SSE4_2);
|
|
|
|
|
break;
|
|
|
|
|
|
2011-06-03 11:01:25 +00:00
|
|
|
default:
|
|
|
|
|
/* Unknown family 0x06 processors. Assuming this is one
|
2011-10-21 02:43:15 +00:00
|
|
|
of Core i3/i5/i7 processors if AVX is available. */
|
2017-03-17 18:38:13 +00:00
|
|
|
if (!CPU_FEATURES_CPU_P (cpu_features, AVX))
|
2011-06-03 11:01:25 +00:00
|
|
|
break;
|
2019-02-12 10:30:34 +00:00
|
|
|
/* Fall through. */
|
2011-06-03 11:01:25 +00:00
|
|
|
|
2010-01-12 19:22:03 +00:00
|
|
|
case 0x1a:
|
|
|
|
|
case 0x1e:
|
|
|
|
|
case 0x1f:
|
|
|
|
|
case 0x25:
|
2010-11-12 08:48:52 +00:00
|
|
|
case 0x2c:
|
2010-01-12 19:22:03 +00:00
|
|
|
case 0x2e:
|
|
|
|
|
case 0x2f:
|
2016-04-01 22:08:48 +00:00
|
|
|
/* Rep string instructions, unaligned load, unaligned copy,
|
2011-07-19 21:11:54 +00:00
|
|
|
and pminub are fast on Intel Core i3, i5 and i7. */
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Fast_Rep_String]
|
2016-03-10 13:26:46 +00:00
|
|
|
|= (bit_arch_Fast_Rep_String
|
|
|
|
|
| bit_arch_Fast_Unaligned_Load
|
2016-03-28 11:39:48 +00:00
|
|
|
| bit_arch_Fast_Unaligned_Copy
|
2016-03-10 13:26:46 +00:00
|
|
|
| bit_arch_Prefer_PMINUB_for_stringop);
|
2010-01-12 19:22:03 +00:00
|
|
|
break;
|
2018-10-11 18:18:40 +00:00
|
|
|
}
|
2016-12-19 10:20:31 +00:00
|
|
|
|
2018-10-11 18:18:40 +00:00
|
|
|
/* Disable TSX on some Haswell processors to avoid TSX on kernels that
|
|
|
|
|
weren't updated with the latest microcode package (which disables
|
|
|
|
|
broken feature by default). */
|
|
|
|
|
switch (model)
|
|
|
|
|
{
|
2016-12-19 10:20:31 +00:00
|
|
|
case 0x3f:
|
|
|
|
|
/* Xeon E7 v3 with stepping >= 4 has working TSX. */
|
|
|
|
|
if (stepping >= 4)
|
|
|
|
|
break;
|
2019-02-12 10:30:34 +00:00
|
|
|
/* Fall through. */
|
2016-12-19 10:20:31 +00:00
|
|
|
case 0x3c:
|
|
|
|
|
case 0x45:
|
|
|
|
|
case 0x46:
|
|
|
|
|
/* Disable Intel TSX on Haswell processors (except Xeon E7 v3
|
|
|
|
|
with stepping >= 4) to avoid TSX on kernels that weren't
|
|
|
|
|
updated with the latest microcode package (which disables
|
|
|
|
|
broken feature by default). */
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_UNSET (cpu_features, RTM);
|
2016-12-19 10:20:31 +00:00
|
|
|
break;
|
2010-01-12 19:22:03 +00:00
|
|
|
}
|
|
|
|
|
}
|
2016-03-22 14:46:56 +00:00
|
|
|
|
2016-09-06 15:50:55 +00:00
|
|
|
|
2017-04-18 15:27:22 +00:00
|
|
|
/* Since AVX512ER is unique to Xeon Phi, set Prefer_No_VZEROUPPER
|
2017-04-18 21:01:45 +00:00
|
|
|
if AVX512ER is available. Don't use AVX512 to avoid lower CPU
|
|
|
|
|
frequency if AVX512ER isn't available. */
|
2017-04-18 15:27:22 +00:00
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, AVX512ER))
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Prefer_No_VZEROUPPER]
|
2017-04-18 15:27:22 +00:00
|
|
|
|= bit_arch_Prefer_No_VZEROUPPER;
|
2017-04-18 21:01:45 +00:00
|
|
|
else
|
2021-02-26 13:36:59 +00:00
|
|
|
{
|
|
|
|
|
cpu_features->preferred[index_arch_Prefer_No_AVX512]
|
|
|
|
|
|= bit_arch_Prefer_No_AVX512;
|
|
|
|
|
|
|
|
|
|
/* Avoid RTM abort triggered by VZEROUPPER inside a
|
|
|
|
|
transactionally executing RTM region. */
|
|
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, RTM))
|
|
|
|
|
cpu_features->preferred[index_arch_Prefer_No_VZEROUPPER]
|
|
|
|
|
|= bit_arch_Prefer_No_VZEROUPPER;
|
|
|
|
|
|
|
|
|
|
/* Since to compare 2 32-byte strings, 256-bit EVEX strcmp
|
|
|
|
|
requires 2 loads, 3 VPCMPs and 2 KORDs while AVX2 strcmp
|
|
|
|
|
requires 1 load, 2 VPCMPEQs, 1 VPMINU and 1 VPMOVMSKB,
|
|
|
|
|
AVX2 strcmp is faster than EVEX strcmp. */
|
|
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX2))
|
|
|
|
|
cpu_features->preferred[index_arch_Prefer_AVX2_STRCMP]
|
|
|
|
|
|= bit_arch_Prefer_AVX2_STRCMP;
|
|
|
|
|
}
|
x86-64: Add Avoid_Short_Distance_REP_MOVSB
commit 3ec5d83d2a237d39e7fd6ef7a0bc8ac4c171a4a5
Author: H.J. Lu <hjl.tools@gmail.com>
Date: Sat Jan 25 14:19:40 2020 -0800
x86-64: Avoid rep movsb with short distance [BZ #27130]
introduced some regressions on Intel processors without Fast Short REP
MOV (FSRM). Add Avoid_Short_Distance_REP_MOVSB to avoid rep movsb with
short distance only on Intel processors with FSRM. bench-memmove-large
on Skylake server shows that cycles of __memmove_evex_unaligned_erms
improves for the following data size:
before after Improvement
length=4127, align1=3, align2=0: 479.38 349.25 27%
length=4223, align1=9, align2=5: 405.62 333.25 18%
length=8223, align1=3, align2=0: 786.12 496.38 37%
length=8319, align1=9, align2=5: 727.50 501.38 31%
length=16415, align1=3, align2=0: 1436.88 840.00 41%
length=16511, align1=9, align2=5: 1375.50 836.38 39%
length=32799, align1=3, align2=0: 2890.00 1860.12 36%
length=32895, align1=9, align2=5: 2891.38 1931.88 33%
2021-07-23 03:26:25 +00:00
|
|
|
|
|
|
|
|
/* Avoid avoid short distance REP MOVSB on processor with FSRM. */
|
|
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, FSRM))
|
|
|
|
|
cpu_features->preferred[index_arch_Avoid_Short_Distance_REP_MOVSB]
|
|
|
|
|
|= bit_arch_Avoid_Short_Distance_REP_MOVSB;
|
2009-03-13 23:53:18 +00:00
|
|
|
}
|
2018-12-12 14:24:17 +00:00
|
|
|
/* This spells out "AuthenticAMD" or "HygonGenuine". */
|
|
|
|
|
else if ((ebx == 0x68747541 && ecx == 0x444d4163 && edx == 0x69746e65)
|
|
|
|
|
|| (ebx == 0x6f677948 && ecx == 0x656e6975 && edx == 0x6e65476e))
|
2009-03-13 23:53:18 +00:00
|
|
|
{
|
2018-12-03 13:54:43 +00:00
|
|
|
unsigned int extended_model;
|
2015-11-30 16:53:37 +00:00
|
|
|
|
2010-04-04 07:25:46 +00:00
|
|
|
kind = arch_kind_amd;
|
2009-03-13 23:53:18 +00:00
|
|
|
|
2018-08-01 11:57:34 +00:00
|
|
|
get_common_indices (cpu_features, &family, &model, &extended_model,
|
2016-12-19 10:20:31 +00:00
|
|
|
&stepping);
|
2011-03-05 04:30:08 +00:00
|
|
|
|
2018-07-26 17:49:19 +00:00
|
|
|
get_extended_indices (cpu_features);
|
2011-03-05 04:30:08 +00:00
|
|
|
|
2021-06-05 13:42:20 +00:00
|
|
|
update_active (cpu_features);
|
2016-01-14 14:36:02 +00:00
|
|
|
|
2020-12-25 15:30:46 +00:00
|
|
|
ecx = cpu_features->features[CPUID_INDEX_1].cpuid.ecx;
|
2020-06-29 23:36:08 +00:00
|
|
|
|
|
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX))
|
2016-06-07 15:00:21 +00:00
|
|
|
{
|
2020-12-25 15:30:46 +00:00
|
|
|
/* Since the FMA4 bit is in CPUID_INDEX_80000001 and
|
2016-06-07 15:00:21 +00:00
|
|
|
FMA4 requires AVX, determine if FMA4 is usable here. */
|
2021-06-05 13:42:20 +00:00
|
|
|
CPU_FEATURE_SET_ACTIVE (cpu_features, FMA4);
|
2016-06-07 15:00:21 +00:00
|
|
|
}
|
|
|
|
|
|
2016-01-14 14:36:02 +00:00
|
|
|
if (family == 0x15)
|
|
|
|
|
{
|
|
|
|
|
/* "Excavator" */
|
|
|
|
|
if (model >= 0x60 && model <= 0x7f)
|
2018-07-06 13:55:36 +00:00
|
|
|
{
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Fast_Unaligned_Load]
|
2016-03-28 11:39:48 +00:00
|
|
|
|= (bit_arch_Fast_Unaligned_Load
|
|
|
|
|
| bit_arch_Fast_Copy_Backward);
|
2018-07-06 13:55:36 +00:00
|
|
|
|
|
|
|
|
/* Unaligned AVX loads are slower.*/
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load]
|
|
|
|
|
&= ~bit_arch_AVX_Fast_Unaligned_Load;
|
2018-07-06 13:55:36 +00:00
|
|
|
}
|
2016-01-14 14:36:02 +00:00
|
|
|
}
|
2009-03-13 23:53:18 +00:00
|
|
|
}
|
2020-04-24 04:55:38 +00:00
|
|
|
/* This spells out "CentaurHauls" or " Shanghai ". */
|
|
|
|
|
else if ((ebx == 0x746e6543 && ecx == 0x736c7561 && edx == 0x48727561)
|
|
|
|
|
|| (ebx == 0x68532020 && ecx == 0x20206961 && edx == 0x68676e61))
|
|
|
|
|
{
|
|
|
|
|
unsigned int extended_model, stepping;
|
|
|
|
|
|
|
|
|
|
kind = arch_kind_zhaoxin;
|
|
|
|
|
|
|
|
|
|
get_common_indices (cpu_features, &family, &model, &extended_model,
|
|
|
|
|
&stepping);
|
|
|
|
|
|
|
|
|
|
get_extended_indices (cpu_features);
|
|
|
|
|
|
2021-06-05 13:42:20 +00:00
|
|
|
update_active (cpu_features);
|
2020-06-29 23:36:08 +00:00
|
|
|
|
2020-04-24 04:55:38 +00:00
|
|
|
model += extended_model;
|
|
|
|
|
if (family == 0x6)
|
|
|
|
|
{
|
|
|
|
|
if (model == 0xf || model == 0x19)
|
|
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX2);
|
2020-04-24 04:55:38 +00:00
|
|
|
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Slow_SSE4_2]
|
|
|
|
|
|= bit_arch_Slow_SSE4_2;
|
2020-04-24 04:55:38 +00:00
|
|
|
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load]
|
|
|
|
|
&= ~bit_arch_AVX_Fast_Unaligned_Load;
|
2020-04-24 04:55:38 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
else if (family == 0x7)
|
|
|
|
|
{
|
2020-06-17 13:34:46 +00:00
|
|
|
if (model == 0x1b)
|
|
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX2);
|
2020-04-24 04:55:38 +00:00
|
|
|
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_Slow_SSE4_2]
|
|
|
|
|
|= bit_arch_Slow_SSE4_2;
|
|
|
|
|
|
|
|
|
|
cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load]
|
|
|
|
|
&= ~bit_arch_AVX_Fast_Unaligned_Load;
|
|
|
|
|
}
|
|
|
|
|
else if (model == 0x3b)
|
|
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX2);
|
2020-04-24 04:55:38 +00:00
|
|
|
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load]
|
|
|
|
|
&= ~bit_arch_AVX_Fast_Unaligned_Load;
|
|
|
|
|
}
|
|
|
|
|
}
|
2020-04-24 04:55:38 +00:00
|
|
|
}
|
2009-03-13 23:53:18 +00:00
|
|
|
else
|
2016-03-22 14:46:56 +00:00
|
|
|
{
|
|
|
|
|
kind = arch_kind_other;
|
2018-08-01 11:57:34 +00:00
|
|
|
get_common_indices (cpu_features, NULL, NULL, NULL, NULL);
|
2021-06-05 13:42:20 +00:00
|
|
|
update_active (cpu_features);
|
2016-03-22 14:46:56 +00:00
|
|
|
}
|
2010-04-04 07:25:46 +00:00
|
|
|
|
2015-08-27 16:06:26 +00:00
|
|
|
/* Support i586 if CX8 is available. */
|
2016-03-22 14:46:56 +00:00
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, CX8))
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_I586] |= bit_arch_I586;
|
2015-08-27 16:06:26 +00:00
|
|
|
|
|
|
|
|
/* Support i686 if CMOV is available. */
|
2016-03-22 14:46:56 +00:00
|
|
|
if (CPU_FEATURES_CPU_P (cpu_features, CMOV))
|
2020-06-17 13:34:46 +00:00
|
|
|
cpu_features->preferred[index_arch_I686] |= bit_arch_I686;
|
2015-08-27 16:06:26 +00:00
|
|
|
|
2015-08-18 14:59:49 +00:00
|
|
|
#if !HAS_CPUID
|
2015-08-13 11:52:50 +00:00
|
|
|
no_cpuid:
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-12-03 13:54:43 +00:00
|
|
|
cpu_features->basic.kind = kind;
|
|
|
|
|
cpu_features->basic.family = family;
|
|
|
|
|
cpu_features->basic.model = model;
|
|
|
|
|
cpu_features->basic.stepping = stepping;
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
|
2020-09-18 14:55:14 +00:00
|
|
|
dl_init_cacheinfo (cpu_features);
|
|
|
|
|
|
tunables: Add IFUNC selection and cache sizes
The current IFUNC selection is based on microbenchmarks in glibc. It
should give the best performance for most workloads. But other choices
may have better performance for a particular workload or on the hardware
which wasn't available at the selection was made. The environment
variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
where the feature name is case-sensitive and has to match the ones in
cpu-features.h. It can be used by glibc developers to override the
IFUNC selection to tune for a new processor or improve performance for
a particular workload. It isn't intended for normal end users.
NOTE: the IFUNC selection may change over time. Please check all
multiarch implementations when experimenting.
Also, GLIBC_TUNABLES=glibc.tune.x86_non_temporal_threshold=NUMBER is
provided to set threshold to use non temporal store to NUMBER,
GLIBC_TUNABLES=glibc.tune.x86_data_cache_size=NUMBER to set data cache
size, GLIBC_TUNABLES=glibc.tune.x86_shared_cache_size=NUMBER to set
shared cache size.
* elf/dl-tunables.list (tune): Add ifunc,
x86_non_temporal_threshold,
x86_data_cache_size and x86_shared_cache_size.
* manual/tunables.texi: Document glibc.tune.ifunc,
glibc.tune.x86_data_cache_size, glibc.tune.x86_shared_cache_size
and glibc.tune.x86_non_temporal_threshold.
* sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
* sysdeps/x86/cpu-tunables.c: Likewise.
* sysdeps/x86/cacheinfo.c
(init_cacheinfo): Check and get data cache size, shared cache
size and non temporal threshold from cpu_features.
* sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
New.
[HAVE_TUNABLES] Include <unistd.h>.
[HAVE_TUNABLES] Include <elf/dl-tunables.h>.
[HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
[HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
IFUNC selection, data cache size, shared cache size and non
temporal threshold.
* sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
shared_cache_size and non_temporal_threshold.
2017-06-20 15:33:29 +00:00
|
|
|
#if HAVE_TUNABLES
|
2017-06-21 17:20:24 +00:00
|
|
|
TUNABLE_GET (hwcaps, tunable_val_t *, TUNABLE_CALLBACK (set_hwcaps));
|
2021-03-19 13:15:37 +00:00
|
|
|
|
|
|
|
|
bool disable_xsave_features = false;
|
|
|
|
|
|
|
|
|
|
if (!CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE))
|
|
|
|
|
{
|
|
|
|
|
/* These features are usable only if OSXSAVE is usable. */
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XSAVE);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XSAVEOPT);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XSAVEC);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XGETBV_ECX_1);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XFD);
|
|
|
|
|
|
|
|
|
|
disable_xsave_features = true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (disable_xsave_features
|
|
|
|
|
|| (!CPU_FEATURE_USABLE_P (cpu_features, XSAVE)
|
|
|
|
|
&& !CPU_FEATURE_USABLE_P (cpu_features, XSAVEC)))
|
|
|
|
|
{
|
|
|
|
|
/* Clear xsave_state_size if both XSAVE and XSAVEC aren't usable. */
|
|
|
|
|
cpu_features->xsave_state_size = 0;
|
|
|
|
|
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX2);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX_VNNI);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, FMA);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, VAES);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, VPCLMULQDQ);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, XOP);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, F16C);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512F);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512CD);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512ER);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512PF);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512VL);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512DQ);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512BW);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_4FMAPS);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_4VNNIW);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_BITALG);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_IFMA);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_VBMI);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_VBMI2);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_VNNI);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_VPOPCNTDQ);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_VP2INTERSECT);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_BF16);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AVX512_FP16);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AMX_BF16);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AMX_TILE);
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, AMX_INT8);
|
|
|
|
|
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, FMA4);
|
|
|
|
|
}
|
|
|
|
|
|
2020-09-18 14:55:14 +00:00
|
|
|
#elif defined SHARED
|
|
|
|
|
/* Reuse dl_platform, dl_hwcap and dl_hwcap_mask for x86. The
|
|
|
|
|
glibc.cpu.hwcap_mask tunable is initialized already, so no
|
|
|
|
|
need to do this. */
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
GLRO(dl_hwcap_mask) = HWCAP_IMPORTANT;
|
2017-06-01 17:02:03 +00:00
|
|
|
#endif
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
|
2017-05-22 19:29:16 +00:00
|
|
|
#ifdef __x86_64__
|
2017-09-11 15:18:11 +00:00
|
|
|
GLRO(dl_hwcap) = HWCAP_X86_64;
|
2018-12-03 13:54:43 +00:00
|
|
|
if (cpu_features->basic.kind == arch_kind_intel)
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
{
|
2017-10-19 15:28:09 +00:00
|
|
|
const char *platform = NULL;
|
|
|
|
|
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX512CD))
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX512ER))
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX512PF))
|
2017-10-19 15:28:09 +00:00
|
|
|
platform = "xeon_phi";
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, AVX512BW)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, AVX512VL))
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
GLRO(dl_hwcap) |= HWCAP_X86_AVX512_1;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2017-10-19 15:28:09 +00:00
|
|
|
if (platform == NULL
|
2020-06-29 23:36:08 +00:00
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, AVX2)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, FMA)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, BMI1)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, BMI2)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, LZCNT)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, MOVBE)
|
|
|
|
|
&& CPU_FEATURE_USABLE_P (cpu_features, POPCNT))
|
2017-10-19 15:28:09 +00:00
|
|
|
platform = "haswell";
|
|
|
|
|
|
|
|
|
|
if (platform != NULL)
|
|
|
|
|
GLRO(dl_platform) = platform;
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
}
|
2017-05-22 19:29:16 +00:00
|
|
|
#else
|
2017-09-11 15:18:11 +00:00
|
|
|
GLRO(dl_hwcap) = 0;
|
2020-06-29 23:36:08 +00:00
|
|
|
if (CPU_FEATURE_USABLE_P (cpu_features, SSE2))
|
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup. They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object. dl_platform and
dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.
On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker. On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated. On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
[BZ #21391]
* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
Only call init_cpu_features.
[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
* sysdeps/i386/dl-procinfo.h: Removed.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include
<sysdeps/x86/dl-procinfo.h>.
(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
Include <sysdeps/x86/dl-procinfo.h> instead of
<sysdeps/generic/dl-procinfo.h>.
* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
(bit_cpu_MOVBE): Likewise.
(bit_cpu_BMI1): Likewise.
(bit_cpu_BMI2): Likewise.
(index_cpu_BMI1): Likewise.
(index_cpu_BMI2): Likewise.
(index_cpu_LZCNT): Likewise.
(index_cpu_MOVBE): Likewise.
(index_cpu_POPCNT): Likewise.
(reg_BMI1): Likewise.
(reg_BMI2): Likewise.
(reg_LZCNT): Likewise.
(reg_MOVBE): Likewise.
(reg_POPCNT): Likewise.
* sysdeps/x86/dl-hwcap.h: New file.
* sysdeps/x86/dl-procinfo.h: Likewise.
* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
(_dl_x86_platforms): Likewise.
2017-05-03 20:42:42 +00:00
|
|
|
GLRO(dl_hwcap) |= HWCAP_X86_SSE2;
|
|
|
|
|
|
|
|
|
|
if (CPU_FEATURES_ARCH_P (cpu_features, I686))
|
|
|
|
|
GLRO(dl_platform) = "i686";
|
|
|
|
|
else if (CPU_FEATURES_ARCH_P (cpu_features, I586))
|
|
|
|
|
GLRO(dl_platform) = "i586";
|
|
|
|
|
#endif
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
|
|
|
|
|
#if CET_ENABLED
|
|
|
|
|
# if HAVE_TUNABLES
|
|
|
|
|
TUNABLE_GET (x86_ibt, tunable_val_t *,
|
|
|
|
|
TUNABLE_CALLBACK (set_x86_ibt));
|
|
|
|
|
TUNABLE_GET (x86_shstk, tunable_val_t *,
|
|
|
|
|
TUNABLE_CALLBACK (set_x86_shstk));
|
|
|
|
|
# endif
|
|
|
|
|
|
|
|
|
|
/* Check CET status. */
|
|
|
|
|
unsigned int cet_status = get_cet_status ();
|
|
|
|
|
|
2021-01-27 04:48:45 +00:00
|
|
|
if ((cet_status & GNU_PROPERTY_X86_FEATURE_1_IBT) == 0)
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, IBT)
|
|
|
|
|
if ((cet_status & GNU_PROPERTY_X86_FEATURE_1_SHSTK) == 0)
|
|
|
|
|
CPU_FEATURE_UNSET (cpu_features, SHSTK)
|
|
|
|
|
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
if (cet_status)
|
|
|
|
|
{
|
2020-04-28 17:05:25 +00:00
|
|
|
GL(dl_x86_feature_1) = cet_status;
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
|
|
|
|
|
# ifndef SHARED
|
|
|
|
|
/* Check if IBT and SHSTK are enabled by kernel. */
|
|
|
|
|
if ((cet_status & GNU_PROPERTY_X86_FEATURE_1_IBT)
|
|
|
|
|
|| (cet_status & GNU_PROPERTY_X86_FEATURE_1_SHSTK))
|
|
|
|
|
{
|
|
|
|
|
/* Disable IBT and/or SHSTK if they are enabled by kernel, but
|
|
|
|
|
disabled by environment variable:
|
|
|
|
|
|
2018-08-02 18:19:19 +00:00
|
|
|
GLIBC_TUNABLES=glibc.cpu.hwcaps=-IBT,-SHSTK
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
*/
|
|
|
|
|
unsigned int cet_feature = 0;
|
2021-01-27 04:48:45 +00:00
|
|
|
if (!CPU_FEATURE_USABLE (IBT))
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
cet_feature |= GNU_PROPERTY_X86_FEATURE_1_IBT;
|
2021-01-27 04:48:45 +00:00
|
|
|
if (!CPU_FEATURE_USABLE (SHSTK))
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
cet_feature |= GNU_PROPERTY_X86_FEATURE_1_SHSTK;
|
|
|
|
|
|
|
|
|
|
if (cet_feature)
|
|
|
|
|
{
|
|
|
|
|
int res = dl_cet_disable_cet (cet_feature);
|
|
|
|
|
|
|
|
|
|
/* Clear the disabled bits in dl_x86_feature_1. */
|
|
|
|
|
if (res == 0)
|
2020-04-28 17:05:25 +00:00
|
|
|
GL(dl_x86_feature_1) &= ~cet_feature;
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Lock CET if IBT or SHSTK is enabled in executable. Don't
|
2020-04-28 17:05:25 +00:00
|
|
|
lock CET if IBT or SHSTK is enabled permissively. */
|
|
|
|
|
if (GL(dl_x86_feature_control).ibt != cet_permissive
|
|
|
|
|
&& GL(dl_x86_feature_control).shstk != cet_permissive)
|
x86: Support IBT and SHSTK in Intel CET [BZ #21598]
Intel Control-flow Enforcement Technology (CET) instructions:
https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-en
forcement-technology-preview.pdf
includes Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).
GNU_PROPERTY_X86_FEATURE_1_IBT is added to GNU program property to
indicate that all executable sections are compatible with IBT when
ENDBR instruction starts each valid target where an indirect branch
instruction can land. Linker sets GNU_PROPERTY_X86_FEATURE_1_IBT on
output only if it is set on all relocatable inputs.
On an IBT capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable IBT and
lock IBT if GNU_PROPERTY_X86_FEATURE_1_IBT is set on the executable.
2. When loading an executable with an interpreter, enable IBT if
GNU_PROPERTY_X86_FEATURE_1_IBT is set on the interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_IBT isn't set on the executable,
disable IBT.
b. Lock IBT.
3. If IBT is enabled, when loading a shared object without
GNU_PROPERTY_X86_FEATURE_1_IBT:
a. If legacy interwork is allowed, then mark all pages in executable
PT_LOAD segments in legacy code page bitmap. Failure of legacy code
page bitmap allocation causes an error.
b. If legacy interwork isn't allowed, it causes an error.
GNU_PROPERTY_X86_FEATURE_1_SHSTK is added to GNU program property to
indicate that all executable sections are compatible with SHSTK where
return address popped from shadow stack always matches return address
popped from normal stack. Linker sets GNU_PROPERTY_X86_FEATURE_1_SHSTK
on output only if it is set on all relocatable inputs.
On a SHSTK capable processor, the following steps should be taken:
1. When loading an executable without an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on the executable.
2. When loading an executable with an interpreter, enable SHSTK if
GNU_PROPERTY_X86_FEATURE_1_SHSTK is set on interpreter.
a. If GNU_PROPERTY_X86_FEATURE_1_SHSTK isn't set on the executable
or any shared objects loaded via the DT_NEEDED tag, disable SHSTK.
b. Otherwise lock SHSTK.
3. After SHSTK is enabled, it is an error to load a shared object
without GNU_PROPERTY_X86_FEATURE_1_SHSTK.
To enable CET support in glibc, --enable-cet is required to configure
glibc. When CET is enabled, both compiler and assembler must support
CET. Otherwise, it is a configure-time error.
To support CET run-time control,
1. _dl_x86_feature_1 is added to the writable ld.so namespace to indicate
if IBT or SHSTK are enabled at run-time. It should be initialized by
init_cpu_features.
2. For dynamic executables:
a. A l_cet field is added to struct link_map to indicate if IBT or
SHSTK is enabled in an ELF module. _dl_process_pt_note or
_rtld_process_pt_note is called to process PT_NOTE segment for
GNU program property and set l_cet.
b. _dl_open_check is added to check IBT and SHSTK compatibilty when
dlopening a shared object.
3. Replace i386 _dl_runtime_resolve and _dl_runtime_profile with
_dl_runtime_resolve_shstk and _dl_runtime_profile_shstk, respectively if
SHSTK is enabled.
CET run-time control can be changed via GLIBC_TUNABLES with
$ export GLIBC_TUNABLES=glibc.tune.x86_shstk=[permissive|on|off]
$ export GLIBC_TUNABLES=glibc.tune.x86_ibt=[permissive|on|off]
1. permissive: SHSTK is disabled when dlopening a legacy ELF module.
2. on: IBT or SHSTK are always enabled, regardless if there are IBT or
SHSTK bits in GNU program property.
3. off: IBT or SHSTK are always disabled, regardless if there are IBT or
SHSTK bits in GNU program property.
<cet.h> from CET-enabled GCC is automatically included by assembly codes
to add GNU_PROPERTY_X86_FEATURE_1_IBT and GNU_PROPERTY_X86_FEATURE_1_SHSTK
to GNU program property. _CET_ENDBR is added at the entrance of all
assembly functions whose address may be taken. _CET_NOTRACK is used to
insert NOTRACK prefix with indirect jump table to support IBT. It is
defined as notrack when _CET_NOTRACK is defined in <cet.h>.
[BZ #21598]
* configure.ac: Add --enable-cet.
* configure: Regenerated.
* elf/Makefille (all-built-dso): Add a comment.
* elf/dl-load.c (filebuf): Moved before "dynamic-link.h".
Include <dl-prop.h>.
(_dl_map_object_from_fd): Call _dl_process_pt_note on PT_NOTE
segment.
* elf/dl-open.c: Include <dl-prop.h>.
(dl_open_worker): Call _dl_open_check.
* elf/rtld.c: Include <dl-prop.h>.
(dl_main): Call _rtld_process_pt_note on PT_NOTE segment. Call
_rtld_main_check.
* sysdeps/generic/dl-prop.h: New file.
* sysdeps/i386/dl-cet.c: Likewise.
* sysdeps/unix/sysv/linux/x86/cpu-features.c: Likewise.
* sysdeps/unix/sysv/linux/x86/dl-cet.h: Likewise.
* sysdeps/x86/cet-tunables.h: Likewise.
* sysdeps/x86/check-cet.awk: Likewise.
* sysdeps/x86/configure: Likewise.
* sysdeps/x86/configure.ac: Likewise.
* sysdeps/x86/dl-cet.c: Likewise.
* sysdeps/x86/dl-procruntime.c: Likewise.
* sysdeps/x86/dl-prop.h: Likewise.
* sysdeps/x86/libc-start.h: Likewise.
* sysdeps/x86/link_map.h: Likewise.
* sysdeps/i386/dl-trampoline.S (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
(_dl_runtime_resolve_shstk): New.
(_dl_runtime_profile_shstk): Likewise.
* sysdeps/linux/x86/Makefile (sysdep-dl-routines): Add dl-cet
if CET is enabled.
(CFLAGS-.o): Add -fcf-protection if CET is enabled.
(CFLAGS-.os): Likewise.
(CFLAGS-.op): Likewise.
(CFLAGS-.oS): Likewise.
(asm-CPPFLAGS): Add -fcf-protection -include cet.h if CET
is enabled.
(tests-special): Add $(objpfx)check-cet.out.
(cet-built-dso): New.
(+$(cet-built-dso:=.note)): Likewise.
(common-generated): Add $(cet-built-dso:$(common-objpfx)%=%.note).
($(objpfx)check-cet.out): New.
(generated): Add check-cet.out.
* sysdeps/x86/cpu-features.c: Include <dl-cet.h> and
<cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New prototype.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
(init_cpu_features): Call get_cet_status to check CET status
and update dl_x86_feature_1 with CET status. Call
TUNABLE_CALLBACK (set_x86_ibt) and TUNABLE_CALLBACK
(set_x86_shstk). Disable and lock CET in libc.a.
* sysdeps/x86/cpu-tunables.c: Include <cet-tunables.h>.
(TUNABLE_CALLBACK (set_x86_ibt)): New function.
(TUNABLE_CALLBACK (set_x86_shstk)): Likewise.
* sysdeps/x86/sysdep.h (_CET_NOTRACK): New.
(_CET_ENDBR): Define if not defined.
(ENTRY): Add _CET_ENDBR.
* sysdeps/x86/dl-tunables.list (glibc.tune): Add x86_ibt and
x86_shstk.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve): Add
_CET_ENDBR.
(_dl_runtime_profile): Likewise.
2018-07-16 21:08:15 +00:00
|
|
|
dl_cet_lock_cet ();
|
|
|
|
|
}
|
|
|
|
|
# endif
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2020-07-04 13:35:49 +00:00
|
|
|
|
|
|
|
|
#ifndef SHARED
|
|
|
|
|
/* NB: In libc.a, call init_cacheinfo. */
|
|
|
|
|
init_cacheinfo ();
|
|
|
|
|
#endif
|
2009-07-29 22:22:28 +00:00
|
|
|
}
|
