mirror of
https://sourceware.org/git/glibc.git
synced 2024-12-04 02:40:06 +00:00
5e1122827a
1080 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
H.J. Lu
|
5e1122827a |
x86-64: Rename wmemset.h to ifunc-wmemset.h
No code changes. * sysdeps/x86_64/multiarch/wmemset.c: Include ifunc-wmemset.h instead of wmemset.h. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/multiarch/wmemset.h: Renamed to ... * sysdeps/x86_64/multiarch/ifunc-wmemset.h: This. |
||
|
H.J. Lu
|
2e87c7d158 |
x86-64: Fold ifunc-sse4_1.h into wcsnlen.c
Since ifunc-sse4_1.h is included only by wcsnlen.c, we can fold it into wcsnlen.c. No code changes in wcsnlen.o. 2017-06-07 H.J. Lu <hongjiu.lu@intel.com> * sysdeps/x86_64/multiarch/ifunc-sse4_1.h: Removed and folded into ... * sysdeps/x86_64/multiarch/wcsnlen.c: Here. Don't include ifunc-sse4_1.h. |
||
|
H.J. Lu
|
d4cc385c6e |
x86-64: Move wcsnlen.S to multiarch/wcsnlen-sse4_1.S
Since wcsnlen.S uses pminud which is the part of SSE4.1, move wcsnlen.S to multiarch/wcsnlen-sse4_1.S. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wcsnlen-sse4_1 and wcsnlen-c. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Test __wcsnlen_sse4_1 and __wcsnlen_sse2. * sysdeps/x86_64/multiarch/ifunc-sse4_1.h: New file. * sysdeps/x86_64/multiarch/wcsnlen-c.c: Likewise. * sysdeps/x86_64/multiarch/wcsnlen-sse4_1.S: Likewise. * sysdeps/x86_64/multiarch/wcsnlen.c: Likewise. * sysdeps/x86_64/wcsnlen.S: Removed. |
||
Stefan Liebler
|
12d2dd7060 |
Optimize generic spinlock code and use C11 like atomic macros.
This patch optimizes the generic spinlock code. The type pthread_spinlock_t is a typedef to volatile int on all archs. Passing a volatile pointer to the atomic macros which are not mapped to the C11 atomic builtins can lead to extra stores and loads to stack if such a macro creates a temporary variable by using "__typeof (*(mem)) tmp;". Thus, those macros which are used by spinlock code - atomic_exchange_acquire, atomic_load_relaxed, atomic_compare_exchange_weak - have to be adjusted. According to the comment from Szabolcs Nagy, the type of a cast expression is unqualified (see http://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_423.htm): __typeof ((__typeof (*(mem)) *(mem)) tmp; Thus from spinlock perspective the variable tmp is of type int instead of type volatile int. This patch adjusts those macros in include/atomic.h. With this construct GCC >= 5 omits the extra stores and loads. The atomic macros are replaced by the C11 like atomic macros and thus the code is aligned to it. The pthread_spin_unlock implementation is now using release memory order instead of sequentially consistent memory order. The issue with passed volatile int pointers applies to the C11 like atomic macros as well as the ones used before. I've added a glibc_likely hint to the first atomic exchange in pthread_spin_lock in order to return immediately to the caller if the lock is free. Without the hint, there is an additional jump if the lock is free. I've added the atomic_spin_nop macro within the loop of plain reads. The plain reads are also realized by C11 like atomic_load_relaxed macro. The new define ATOMIC_EXCHANGE_USES_CAS determines if the first try to acquire the spinlock in pthread_spin_lock or pthread_spin_trylock is an exchange or a CAS. This is defined in atomic-machine.h for all architectures. The define SPIN_LOCK_READS_BETWEEN_CMPXCHG is now removed. There is no technical reason for throwing in a CAS every now and then, and so far we have no evidence that it can improve performance. If that would be the case, we have to adjust other spin-waiting loops elsewhere, too! Using a CAS loop without plain reads is not a good idea on many targets and wasn't used by one. Thus there is now no option to do so. Architectures are now using the generic spinlock automatically if they do not provide an own implementation. Thus the pthread_spin_lock.c files in sysdeps folder are deleted. ChangeLog: * NEWS: Mention new spinlock implementation. * include/atomic.h: (__atomic_val_bysize): Cast type to omit volatile qualifier. (atomic_exchange_acq): Likewise. (atomic_load_relaxed): Likewise. (ATOMIC_EXCHANGE_USES_CAS): Check definition. * nptl/pthread_spin_init.c (pthread_spin_init): Use atomic_store_relaxed. * nptl/pthread_spin_lock.c (pthread_spin_lock): Use C11-like atomic macros. * nptl/pthread_spin_trylock.c (pthread_spin_trylock): Likewise. * nptl/pthread_spin_unlock.c (pthread_spin_unlock): Use atomic_store_release. * sysdeps/aarch64/nptl/pthread_spin_lock.c: Delete File. * sysdeps/arm/nptl/pthread_spin_lock.c: Likewise. * sysdeps/hppa/nptl/pthread_spin_lock.c: Likewise. * sysdeps/m68k/nptl/pthread_spin_lock.c: Likewise. * sysdeps/microblaze/nptl/pthread_spin_lock.c: Likewise. * sysdeps/mips/nptl/pthread_spin_lock.c: Likewise. * sysdeps/nios2/nptl/pthread_spin_lock.c: Likewise. * sysdeps/aarch64/atomic-machine.h (ATOMIC_EXCHANGE_USES_CAS): Define. * sysdeps/alpha/atomic-machine.h: Likewise. * sysdeps/arm/atomic-machine.h: Likewise. * sysdeps/i386/atomic-machine.h: Likewise. * sysdeps/ia64/atomic-machine.h: Likewise. * sysdeps/m68k/coldfire/atomic-machine.h: Likewise. * sysdeps/m68k/m680x0/m68020/atomic-machine.h: Likewise. * sysdeps/microblaze/atomic-machine.h: Likewise. * sysdeps/mips/atomic-machine.h: Likewise. * sysdeps/powerpc/powerpc32/atomic-machine.h: Likewise. * sysdeps/powerpc/powerpc64/atomic-machine.h: Likewise. * sysdeps/s390/atomic-machine.h: Likewise. * sysdeps/sparc/sparc32/atomic-machine.h: Likewise. * sysdeps/sparc/sparc32/sparcv9/atomic-machine.h: Likewise. * sysdeps/sparc/sparc64/atomic-machine.h: Likewise. * sysdeps/tile/tilegx/atomic-machine.h: Likewise. * sysdeps/tile/tilepro/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/hppa/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/m68k/coldfire/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/nios2/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/sh/atomic-machine.h: Likewise. * sysdeps/x86_64/atomic-machine.h: Likewise. |
||
|
H.J. Lu
|
935971ba6b |
x86-64: Optimize memcmp/wmemcmp with AVX2 and MOVBE
Optimize x86-64 memcmp/wmemcmp with AVX2. It uses vector compare as much as possible. It is as fast as SSE4 memcmp for size <= 16 bytes and up to 2X faster for size > 16 bytes on Haswell and Skylake. Select AVX2 memcmp/wmemcmp on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. NB: It uses TZCNT instead of BSF since TZCNT produces the same result as BSF for non-zero input. TZCNT is faster than BSF and is executed as BSF if machine doesn't support TZCNT. Key features: 1. For size from 2 to 7 bytes, load as big endian with movbe and bswap to avoid branches. 2. Use overlapping compare to avoid branch. 3. Use vector compare when size >= 4 bytes for memcmp or size >= 8 bytes for wmemcmp. 4. If size is 8 * VEC_SIZE or less, unroll the loop. 5. Compare 4 * VEC_SIZE at a time with the aligned first memory area. 6. Use 2 vector compares when size is 2 * VEC_SIZE or less. 7. Use 4 vector compares when size is 4 * VEC_SIZE or less. 8. Use 8 vector compares when size is 8 * VEC_SIZE or less. * sysdeps/x86/cpu-features.h (index_cpu_MOVBE): New. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memcmp-avx2 and wmemcmp-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Test __memcmp_avx2 and __wmemcmp_avx2. * sysdeps/x86_64/multiarch/memcmp-avx2.S: New file. * sysdeps/x86_64/multiarch/wmemcmp-avx2.S: Likewise. * sysdeps/x86_64/multiarch/memcmp.S: Use __memcmp_avx2 on AVX 2 machines if AVX unaligned load is fast and vzeroupper is preferred. * sysdeps/x86_64/multiarch/wmemcmp.S: Use __wmemcmp_avx2 on AVX 2 machines if AVX unaligned load is fast and vzeroupper is preferred. |
||
|
H.J. Lu
|
ef9c4cb6c7 |
x86-64: Optimize wmemset with SSE2/AVX2/AVX512
The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise. |
||
|
H.J. Lu
|
30cb625a21 |
x86-64: Update strlen.S to support wcslen/wcsnlen
The difference between strlen and wcslen is byte vs int. We can replace pminub and pcmpeqb with pminud and pcmpeqd to turn strlen into wcslen. * sysdeps/x86_64/strlen.S (PMINU): New. (PCMPEQ): Likewise. (SHIFT_RETURN): Likewise. (FIND_ZERO): Replace pcmpeqb with PCMPEQ. (strlen): Add SHIFT_RETURN before ret. Replace pcmpeqb and pminub with PCMPEQ and PMINU. * sysdeps/x86_64/wcsnlen.S: New file. |
||
|
H.J. Lu
|
7395928b95 |
x86_64: Remove redundant REX bytes from memrchr.S
By x86-64 specification, 32-bit destination registers are zero-extended to 64 bits. There is no need to use 64-bit registers when only the lower 32 bits are non-zero. Also 2 instructions in: mov %rdi, %rcx and $15, %rcx jz L(length_less16_offset0) mov %rdi, %rcx <<< redundant and $15, %rcx <<< redundant are redundant. * sysdeps/x86_64/memrchr.S (__memrchr): Use 32-bit registers for the lower 32 bits. Remove redundant instructions. |
||
|
H.J. Lu
|
4f26ef1b67 |
x86_64: Remove redundant REX bytes from memchr.S
By x86-64 specification, 32-bit destination registers are zero-extended to 64 bits. There is no need to use 64-bit registers when only the lower 32 bits are non-zero. * sysdeps/x86_64/memchr.S (MEMCHR): Use 32-bit registers for the lower 32 bits. |
||
|
H.J. Lu
|
1f655beb08 |
x86_64: Remove L(return_null) from rawmemchr.S
L(return_null) is unused. * sysdeps/x86_64/rawmemchr.S (L(return_null)): Removed. |
||
|
H.J. Lu
|
402bf06952 |
x86: Optimize SSE2 memchr overflow calculation
SSE2 memchr computes "edx + ecx - 16" where ecx is less than 16. Use "edx - (16 - ecx)", instead of satured math, to avoid possible addition overflow. This replaces add %ecx, %edx sbb %eax, %eax or %eax, %edx sub $16, %edx with neg %ecx add $16, %ecx sub %ecx, %edx It is the same for x86_64, except for rcx/rdx, instead of ecx/edx. * sysdeps/i386/i686/multiarch/memchr-sse2.S (MEMCHR): Use "edx + ecx - 16" to avoid possible addition overflow. * sysdeps/x86_64/memchr.S (memchr): Likewise. |
||
|
H.J. Lu
|
a7fbedff76 |
Correct comments in x86_64/multiarch/memcmp.S
* sysdeps/x86_64/multiarch/memcmp.S (__GI_memcmp): Correct comments. |
||
Zack Weinberg
|
7c3018f9e4 |
Suppress internal declarations for most of the testsuite.
This patch adds a new build module called 'testsuite'.
IS_IN (testsuite) implies _ISOMAC, as do IS_IN_build and __cplusplus
(which means several ad-hoc tests for __cplusplus can go away).
libc-symbols.h now suppresses almost all of *itself* when _ISOMAC is
defined; in particular, _ISOMAC mode does not get config.h
automatically anymore.
There are still quite a few tests that need to see internal gunk of
one variety or another. For them, we now have 'tests-internal' and
'test-internal-extras'; files in this category will still be compiled
with MODULE_NAME=nonlib, and everything proceeds as it always has.
The bulk of this patch is moving tests from 'tests' to
'tests-internal'. There is also 'tests-static-internal', which has
the same effect on files in 'tests-static', and 'modules-names-tests',
which has the *inverse* effect on files in 'modules-names' (it's
inverted because most of the things in modules-names are *not* tests).
For both of these, the file must appear in *both* the new variable and
the old one.
There is also now a special case for when libc-symbols.h is included
without MODULE_NAME being defined at all. (This happens during the
creation of libc-modules.h, and also when preprocessing Versions
files.) When this happens, IS_IN is set to be always false and
_ISOMAC is *not* defined, which was the status quo, but now it's
explicit.
The remaining changes to C source files in this patch seemed likely to
cause problems in the absence of the main change. They should be
relatively self-explanatory. In a few cases I duplicated a definition
from an internal header rather than move the test to tests-internal;
this was a judgement call each time and I'm happy to change those
however reviewers feel is more appropriate.
* Makerules: New subdir configuration variables 'tests-internal'
and 'test-internal-extras'. Test files in these categories will
still be compiled with MODULE_NAME=nonlib. Test files in the
existing categories (tests, xtests, test-srcs, test-extras) are
now compiled with MODULE_NAME=testsuite.
New subdir configuration variable 'modules-names-tests'. Files
which are in both 'modules-names' and 'modules-names-tests' will
be compiled with MODULE_NAME=testsuite instead of
MODULE_NAME=extramodules.
(gen-as-const-headers): Move to tests-internal.
(do-tests-clean, common-mostlyclean): Support tests-internal.
* Makeconfig (built-modules): Add testsuite.
* Makefile: Change libof-check-installed-headers-c and
libof-check-installed-headers-cxx to 'testsuite'.
* Rules: Likewise. Support tests-internal.
* benchtests/strcoll-inputs/filelist#en_US.UTF-8:
Remove extra-modules.mk.
* config.h.in: Don't check for __OPTIMIZE__ or __FAST_MATH__ here.
* include/libc-symbols.h: Move definitions of _GNU_SOURCE,
PASTE_NAME, PASTE_NAME1, IN_MODULE, IS_IN, and IS_IN_LIB to the
very top of the file and rationalize their order.
If MODULE_NAME is not defined at all, define IS_IN to always be
false, and don't define _ISOMAC.
If any of IS_IN (testsuite), IS_IN_build, or __cplusplus are
true, define _ISOMAC and suppress everything else in this file,
starting with the inclusion of config.h.
Do check for inappropriate definitions of __OPTIMIZE__ and
__FAST_MATH__ here, but only if _ISOMAC is not defined.
Correct some out-of-date commentary.
* include/math.h: If _ISOMAC is defined, undefine NO_LONG_DOUBLE
and _Mlong_double_ before including math.h.
* include/string.h: If _ISOMAC is defined, don't expose
_STRING_ARCH_unaligned. Move a comment to a more appropriate
location.
* include/errno.h, include/stdio.h, include/stdlib.h, include/string.h
* include/time.h, include/unistd.h, include/wchar.h: No need to
check __cplusplus nor use __BEGIN_DECLS/__END_DECLS.
* misc/sys/cdefs.h (__NTHNL): New macro.
* sysdeps/m68k/m680x0/fpu/bits/mathinline.h
(__m81_defun): Use __NTHNL to avoid errors with GCC 6.
* elf/tst-env-setuid-tunables.c: Include config.h with _LIBC
defined, for HAVE_TUNABLES.
* inet/tst-checks-posix.c: No need to define _ISOMAC.
* intl/tst-gettext2.c: Provide own definition of N_.
* math/test-signgam-finite-c99.c: No need to define _ISOMAC.
* math/test-signgam-main.c: No need to define _ISOMAC.
* stdlib/tst-strtod.c: Convert to test-driver. Split locale_test to...
* stdlib/tst-strtod1i.c: ...this new file.
* stdlib/tst-strtod5.c: Convert to test-driver and add copyright notice.
Split tests of __strtod_internal to...
* stdlib/tst-strtod5i.c: ...this new file.
* string/test-string.h: Include stdint.h. Duplicate definition of
inhibit_loop_to_libcall here (from libc-symbols.h).
* string/test-strstr.c: Provide dummy definition of
libc_hidden_builtin_def when including strstr.c.
* sysdeps/ia64/fpu/libm-symbols.h: Suppress entire file in _ISOMAC
mode; no need to test __STRICT_ANSI__ nor __cplusplus as well.
* sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h.
Don't include init-arch.h.
* sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h.
Don't include init-arch.h.
* elf/Makefile: Move tst-ptrguard1-static, tst-stackguard1-static,
tst-tls1-static, tst-tls2-static, tst-tls3-static, loadtest,
unload, unload2, circleload1, neededtest, neededtest2,
neededtest3, neededtest4, tst-tls1, tst-tls2, tst-tls3,
tst-tls6, tst-tls7, tst-tls8, tst-dlmopen2, tst-ptrguard1,
tst-stackguard1, tst-_dl_addr_inside_object, and all of the
ifunc tests to tests-internal.
Don't add $(modules-names) to test-extras.
* inet/Makefile: Move tst-inet6_scopeid_pton to tests-internal.
Add tst-deadline to tests-static-internal.
* malloc/Makefile: Move tst-mallocstate and tst-scratch_buffer to
tests-internal.
* misc/Makefile: Move tst-atomic and tst-atomic-long to tests-internal.
* nptl/Makefile: Move tst-typesizes, tst-rwlock19, tst-sem11,
tst-sem12, tst-sem13, tst-barrier5, tst-signal7, tst-tls3,
tst-tls3-malloc, tst-tls5, tst-stackguard1, tst-sem11-static,
tst-sem12-static, and tst-stackguard1-static to tests-internal.
Link tests-internal with libpthread also.
Don't add $(modules-names) to test-extras.
* nss/Makefile: Move tst-field to tests-internal.
* posix/Makefile: Move bug-regex5, bug-regex20, bug-regex33,
tst-rfc3484, tst-rfc3484-2, and tst-rfc3484-3 to tests-internal.
* stdlib/Makefile: Move tst-strtod1i, tst-strtod3, tst-strtod4,
tst-strtod5i, tst-tls-atexit, and tst-tls-atexit-nodelete to
tests-internal.
* sunrpc/Makefile: Move tst-svc_register to tests-internal.
* sysdeps/powerpc/Makefile: Move test-get_hwcap and
test-get_hwcap-static to tests-internal.
* sysdeps/unix/sysv/linux/Makefile: Move tst-setgetname to
tests-internal.
* sysdeps/x86_64/fpu/Makefile: Add all libmvec test modules to
modules-names-tests.
|
||
|
H.J. Lu
|
1432d38ea0 |
x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very early during startup. They are used by dynamic linker to determine platform and build an array of hardware capability names, which are added to search path when loading shared object. dl_platform and dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686 platforms were supported and only SSE2 capability was used. On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and processor capabilities is obsolete since all information is available in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from dl_x86_cpu_features in dynamic linker. On i386, the available plaforms are changed to i586 and i686 since i386 has been deprecated. On x86-64, the available plaforms are haswell, which is for Haswell class processors with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512 ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL. [BZ #21391] * sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]: Only call init_cpu_features. [!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed. * sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise. * sysdeps/i386/dl-procinfo.h: Removed. * sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include <sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include <sysdeps/x86/dl-procinfo.h>. (_dl_procinfo): Replace _DL_HWCAP_COUNT with 32. * sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]: Include <sysdeps/x86/dl-procinfo.h> instead of <sysdeps/generic/dl-procinfo.h>. * sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>. (init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask. * sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New. (bit_cpu_MOVBE): Likewise. (bit_cpu_BMI1): Likewise. (bit_cpu_BMI2): Likewise. (index_cpu_BMI1): Likewise. (index_cpu_BMI2): Likewise. (index_cpu_LZCNT): Likewise. (index_cpu_MOVBE): Likewise. (index_cpu_POPCNT): Likewise. (reg_BMI1): Likewise. (reg_BMI2): Likewise. (reg_LZCNT): Likewise. (reg_MOVBE): Likewise. (reg_POPCNT): Likewise. * sysdeps/x86/dl-hwcap.h: New file. * sysdeps/x86/dl-procinfo.h: Likewise. * sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New. (_dl_x86_platforms): Likewise. |
||
|
H.J. Lu
|
4cb334c4d6 |
x86: Use AVX2 memcpy/memset on Skylake server [BZ #21396]
On Skylake server, AVX512 load/store instructions in memcpy/memset may lead to lower CPU turbo frequency in certain situations. Use of AVX2 in memcpy/memset has been observed to have improved overall performance in many workloads due to the higher frequency. Since AVX512ER is unique to Xeon Phi, this patch sets Prefer_No_AVX512 if AVX512ER isn't available so that AVX2 versions of memcpy/memset are used on Skylake server. [BZ #21396] * sysdeps/x86/cpu-features.c (init_cpu_features): Set Prefer_No_AVX512 if AVX512ER isn't available. * sysdeps/x86/cpu-features.h (bit_arch_Prefer_No_AVX512): New. (index_arch_Prefer_No_AVX512): Likewise. * sysdeps/x86_64/multiarch/memcpy.S (__new_memcpy): Don't use AVX512 version if Prefer_No_AVX512 is set. * sysdeps/x86_64/multiarch/memcpy_chk.S (__memcpy_chk): Likewise. * sysdeps/x86_64/multiarch/memmove.S (__libc_memmove): Likewise. * sysdeps/x86_64/multiarch/memmove_chk.S (__memmove_chk): Likewise. * sysdeps/x86_64/multiarch/mempcpy.S (__mempcpy): Likewise. * sysdeps/x86_64/multiarch/mempcpy_chk.S (__mempcpy_chk): Likewise. * sysdeps/x86_64/multiarch/memset.S (memset): Likewise. * sysdeps/x86_64/multiarch/memset_chk.S (__memset_chk): Likewise. |
||
|
H.J. Lu
|
fda19e0438 |
Add sysdeps/x86/dl-procinfo.c
Add sysdeps/x86/dl-procinfo.c for x86 version of processor capability information to reduce duplication between i386 and x86_64 dl-procinfo.c. * sysdeps/i386/dl-procinfo.c: Include <sysdeps/x86/dl-procinfo.c>. * sysdeps/x86_64/dl-procinfo.c: Likewise. * sysdeps/x86/dl-procinfo.c: New file. |
||
Adhemerval Zanella
|
a358c80530 |
Remove CALL_THREAD_FCT macro
This patch removes CALL_THREAD_FCT macro usage and its defition for x86. For 32 bits it usage is only for force 16 stack alignment, however stack is already explicit aligned in clone syscall. For 64 bits and x32 it just a function call and there is no need to code it with inline assembly. Checked on i686-linux-gnu, x86_64-linux-gnu, and x86_64-linux-gnu-x32. * nptl/pthread_create.c (START_THREAD_DEFN): Remove CALL_THREAD_FCT macro usage. * sysdeps/i386/nptl/tls.h (CALL_THREAD_FCT): Remove definition. * sysdeps/x86_64/nptl/tls.h (CALL_THREAD_FCT): Likewise. * sysdeps/x86_64/32/nptl/tls.h: Remove file. |
||
|
H.J. Lu
|
c15f8eb50c |
x86-64: Improve branch predication in _dl_runtime_resolve_avx512_opt [BZ #21258]
On Skylake server, _dl_runtime_resolve_avx512_opt is used to preserve
the first 8 vector registers. The code layout is
if only %xmm0 - %xmm7 registers are used
preserve %xmm0 - %xmm7 registers
if only %ymm0 - %ymm7 registers are used
preserve %ymm0 - %ymm7 registers
preserve %zmm0 - %zmm7 registers
Branch predication always executes the fallthrough code path to preserve
%zmm0 - %zmm7 registers speculatively, even though only %xmm0 - %xmm7
registers are used. This leads to lower CPU frequency on Skylake
server. This patch changes the fallthrough code path to preserve
%xmm0 - %xmm7 registers instead:
if whole %zmm0 - %zmm7 registers are used
preserve %zmm0 - %zmm7 registers
if only %ymm0 - %ymm7 registers are used
preserve %ymm0 - %ymm7 registers
preserve %xmm0 - %xmm7 registers
Tested on Skylake server.
[BZ #21258]
* sysdeps/x86_64/dl-trampoline.S (_dl_runtime_resolve_opt):
Define only if _dl_runtime_resolve is defined to
_dl_runtime_resolve_sse_vex.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_opt):
Fallthrough to _dl_runtime_resolve_sse_vex.
|
||
Mike Frysinger
|
fbe355fbd1 |
x86_64: fix static build of __mempcpy_chk for compilers defaulting to PIC/PIE
When glibc is compiled with gcc 6.2 that has been configured with to default to PIC/PIE, the static version of __mempcpy_chk is not built, as the test is done on PIC instead of SHARED. Fix the test to check for SHARED, like it is done for similar functions like __memcpy_chk. 2017-03-12 Mike Frysinger <vapier@gentoo.org> * sysdeps/x86_64/mempcpy_chk.S (__mempcpy_chk): Check for SHARED instead of PIC. |
||
Florian Weimer
|
2d6ab5df3b | Document and fix --enable-bind-now [BZ #21015] | ||
|
Zack Weinberg
|
9090848d06 |
Narrowing the visibility of libc-internal.h even further.
posix/wordexp-test.c used libc-internal.h for PTR_ALIGN_DOWN; similar
to what was done with libc-diag.h, I have split the definitions of
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN
to a new header, libc-pointer-arith.h.
It then occurred to me that the remaining declarations in libc-internal.h
are mostly to do with early initialization, and probably most of the
files including it, even in the core code, don't need it anymore. Indeed,
only 19 files actually need what remains of libc-internal.h. 23 others
need libc-diag.h instead, and 12 need libc-pointer-arith.h instead.
No file needs more than one of them, and 16 don't need any of them!
So, with this patch, libc-internal.h stops including libc-diag.h as
well as losing the pointer arithmetic macros, and all including files
are adjusted.
* include/libc-pointer-arith.h: New file. Define
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and
PTR_ALIGN_DOWN here.
* include/libc-internal.h: Definitions of above macros
moved from here. Don't include libc-diag.h anymore either.
* posix/wordexp-test.c: Include stdint.h and libc-pointer-arith.h.
Don't include libc-internal.h.
* debug/pcprofile.c, elf/dl-tunables.c, elf/soinit.c, io/openat.c
* io/openat64.c, misc/ptrace.c, nptl/pthread_clock_gettime.c
* nptl/pthread_clock_settime.c, nptl/pthread_cond_common.c
* string/strcoll_l.c, sysdeps/nacl/brk.c
* sysdeps/unix/clock_settime.c
* sysdeps/unix/sysv/linux/i386/get_clockfreq.c
* sysdeps/unix/sysv/linux/ia64/get_clockfreq.c
* sysdeps/unix/sysv/linux/powerpc/get_clockfreq.c
* sysdeps/unix/sysv/linux/sparc/sparc64/get_clockfreq.c:
Don't include libc-internal.h.
* elf/get-dynamic-info.h, iconv/loop.c
* iconvdata/iso-2022-cn-ext.c, locale/weight.h, locale/weightwc.h
* misc/reboot.c, nis/nis_table.c, nptl_db/thread_dbP.h
* nscd/connections.c, resolv/res_send.c, soft-fp/fmadf4.c
* soft-fp/fmasf4.c, soft-fp/fmatf4.c, stdio-common/vfscanf.c
* sysdeps/ieee754/dbl-64/e_lgamma_r.c
* sysdeps/ieee754/dbl-64/k_rem_pio2.c
* sysdeps/ieee754/flt-32/e_lgammaf_r.c
* sysdeps/ieee754/flt-32/k_rem_pio2f.c
* sysdeps/ieee754/ldbl-128/k_tanl.c
* sysdeps/ieee754/ldbl-128ibm/k_tanl.c
* sysdeps/ieee754/ldbl-96/e_lgammal_r.c
* sysdeps/ieee754/ldbl-96/k_tanl.c, sysdeps/nptl/futex-internal.h:
Include libc-diag.h instead of libc-internal.h.
* elf/dl-load.c, elf/dl-reloc.c, locale/programs/locarchive.c
* nptl/nptl-init.c, string/strcspn.c, string/strspn.c
* malloc/malloc.c, sysdeps/i386/nptl/tls.h
* sysdeps/nacl/dl-map-segments.h, sysdeps/x86_64/atomic-machine.h
* sysdeps/unix/sysv/linux/spawni.c
* sysdeps/x86_64/nptl/tls.h:
Include libc-pointer-arith.h instead of libc-internal.h.
* elf/get-dynamic-info.h, sysdeps/nacl/dl-map-segments.h
* sysdeps/x86_64/atomic-machine.h:
Add multiple include guard.
|
||
|
Zack Weinberg
|
963394a22b |
Allow direct use of math_ldbl.h in testsuite.
A few 'long double'-related tests include math_private.h just for their variety of math_ldbl.h, which contains macros for assembling and disassembling the binary representation of 'long double'. math_ldbl.h insists on being included from math_private.h, but if we relax this restriction (and fix some portability sloppiness) we can use it directly and not have to expose all of math_private.h to the testsuite. * sysdeps/generic/math_private.h: Use __BIG_ENDIAN and __LITTLE_ENDIAN, not BIG_ENDIAN and LITTLE_ENDIAN. * sysdeps/generic/math_ldbl.h * sysdeps/ia64/fpu/math_ldbl.h * sysdeps/ieee754/ldbl-128/math_ldbl.h * sysdeps/ieee754/ldbl-128ibm/math_ldbl.h * sysdeps/ieee754/ldbl-96/math_ldbl.h * sysdeps/powerpc/fpu/math_ldbl.h * sysdeps/x86_64/fpu/math_ldbl.h: Allow direct inclusion. Use uintNN_t instead of u_intNN_t. Use __BIG_ENDIAN and __LITTLE_ENDIAN, not BIG_ENDIAN and LITTLE_ENDIAN. Include endian.h and/or stdint.h if necessary. Add copyright notices. * sysdeps/ieee754/ldbl-128ibm/math_ldbl.h (ldbl_canonicalize_int): Don't use EXTRACT_WORDS64. * sysdeps/ieee754/ldbl-96/test-canonical-ldbl-96.c * sysdeps/ieee754/ldbl-96/test-totalorderl-ldbl-96.c * sysdeps/ieee754/ldbl-128ibm/test-canonical-ldbl-128ibm.c * sysdeps/ieee754/ldbl-128ibm/test-totalorderl-ldbl-128ibm.c: Include math_ldbl.h, not math_private.h. |
||
Joseph Myers
|
92061bb033 |
Run libm tests separately for each function.
At present, libm tests for each function get built into a single executable (for each floating point type, for each of normal / inline / finite-math-only functions, plus vector variants) and run together, resulting in a single PASS or FAIL (for each of those nine variants plus vector variants). Building this executable involves reading over 50 MB of libm-test-*.c sources. This patch arranges for tests of each function to be run separately from the makefiles instead. There are 121 functions being tested for each (type, variant pair) (actually 126, but run as 121 from the Makefile because each of the pairs (exp10, pow10), (isfinite, finite), (lgamma, gamma), (remainder, drem), (scalbn, ldexp), shares a table of test results and so is run together), so 1089 separate tests run from the Makefile, plus 48 vector tests on x86_64 (six functions for eight vector variants). Each test only involves a libm-test-<func>.c file of no more than about 4 MB, rather than all such files taking about 50 MB. With tests run separately, test summaries will indicate which functions actually have problems (of course, those problems may just be out-of-date libm-test-ulps files if the file hasn't been updated for the architecture in question recently). All the .c files for the 1089+48 tests are generated automatically from the Makefiles. Various checked-in boilerplate .c files are removed as no longer needed. CFLAGS definitions for the different kinds of tests are generated using makefile iterators to apply target-specific variable settings. libm-have-vector-test.h is no longer needed; the list of functions to test for each vector type is now in the sysdeps Makefile. This should reduce the amount of boilerplate needed for float128 testing support; test-float128.h will still be needed, but not various .c files or Makefile CFLAGS definitions. The logic for creating dependencies on libm-test-support-*.o files should also render <https://sourceware.org/ml/libc-alpha/2017-02/msg00279.html> unnecessary. Tested for x86_64 and x86. * math/Makefile (libm-tests-generated): Remove variable. (libm-tests-base-normal): New variable. (libm-tests-base-finite): Likewise. (libm-tests-base-inline): Likewise. (libm-tests-base): Likewise. (libm-tests-normal): Likewise. (libm-tests-finite): Likewise. (libm-tests-inline): Likewise. (libm-tests-vector): Likewise. (libm-tests): Define in terms of these new variables. (libm-tests-for-type): New variable. (libm-tests.o): Move definition. (tests): Move addition of $(libm-tests). (generated): Update for new and removed libm test files. ($(objpfx)libm-test.c): Remove target. ($(objpfx)libm-have-vector-test.h): Likewise. (CFLAGS-test-double-vlen2.c): Remove variable. (CFLAGS-test-double-vlen4.c): Likewise. (CFLAGS-test-double-vlen8.c): Likewise. (CFLAGS-test-float-vlen4.c): Likewise. (CFLAGS-test-float-vlen8.c): Likewise. (CFLAGS-test-float-vlen16.c): Likewise. (CFLAGS-test-float.c): Likewise. (CFLAGS-test-float-finite.c): Likewise. (CFLAGS-libm-test-support-float.c): Likewise. (CFLAGS-test-double.c): Likewise. (CFLAGS-test-double-finite.c): Likewise. (CFLAGS-libm-test-support-double.c): Likewise. (CFLAGS-test-ldouble.c): Likewise. (CFLAGS-test-ldouble-finite.c): Likewise. (CFLAGS-libm-test-support-ldouble.c): Likewise. (libm-test-inline-cflags): New variable. (CFLAGS-test-ifloat.c): Remove variable. (CFLAGS-test-idouble.c): Likewise. (CFLAGS-test-ildouble.c): Likewise. ($(addprefix $(objpfx), $(libm-tests.o))): Move target and update dependencies. ($(foreach t,$(libm-tests-normal),$(objpfx)$(t).c)): New rule. ($(foreach t,$(libm-tests-finite),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(libm-tests-inline),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(libm-tests-vector),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(types),$(objpfx)libm-test-support-$(t).c)): Likewise. (dependencies on libm-test-support-*.o): Remove. ($(foreach f,$(libm-test-funcs-all),$(objpfx)$(o)-$(f).o)): New rules using iterators. ($(addprefix $(objpfx),$(call libm-tests-for-type,$(o)))): Likewise. ($(objpfx)libm-test-support-$(o).o): Likewise. ($(addprefix $(objpfx),$(filter-out $(tests-static) $(libm-vec-tests),$(tests)))): Filter out $(libm-tests-vector) instead. ($(addprefix $(objpfx), $(libm-vec-tests))): Use iterator to define rule instead. * math/README.libm-test: Update. * math/libm-test-acos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-acosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-asin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-asinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atan2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cabs.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cacos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cacosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-canonicalize.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-carg.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-casin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-casinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-catan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-catanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cbrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ccos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ccosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ceil.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cexp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cimag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-clog.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-clog10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-conj.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-copysign.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cpow.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cproj.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-creal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csqrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ctan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ctanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-erf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-erfc.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-expm1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fabs.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fdim.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-floor.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmax.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmaxmag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fminmag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmod.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fpclassify.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-frexp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fromfp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fromfpx.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-getpayload.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-hypot.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ilogb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iscanonical.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iseqsig.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isfinite.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isgreater.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isgreaterequal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isinf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isless.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-islessequal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-islessgreater.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isnan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isnormal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-issignaling.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-issubnormal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isunordered.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iszero.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-j0.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-j1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-jn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lgamma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llogb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llrint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llround.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log1p.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-logb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lrint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lround.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-modf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nearbyint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextafter.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextdown.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nexttoward.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextup.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-pow.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-remainder.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-remquo.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-rint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-round.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-roundeven.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalbln.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalbn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-setpayload.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-setpayloadsig.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-signbit.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-significand.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sincos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sqrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tgamma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-totalorder.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-totalordermag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-trunc.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ufromfp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ufromfpx.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-y0.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-y1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-yn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-driver.c: Do not include libm-have-vector-test.h. (HAVE_VECTOR): Remove macro. (START): Do not call HAVE_VECTOR. * math/test-double-vlen2.h (FUNC_TEST): Remove macro. * math/test-double-vlen4.h (FUNC_TEST): Remove macro. * math/test-double-vlen8.h (FUNC_TEST): Remove macro. * math/test-float-vlen16.h (FUNC_TEST): Remove macro. * math/test-float-vlen4.h (FUNC_TEST): Remove macro. * math/test-float-vlen8.h (FUNC_TEST): Remove macro. * math/test-math-vector.h (FUNC_TEST): New macro. (WRAPPER_DECL): Rename to WRAPPER_DECL_f. * sysdeps/x86_64/fpu/Makefile (double-vlen2-funcs): New variable. (double-vlen4-funcs): Likewise. (double-vlen4-avx2-funcs): Likewise. (double-vlen8-funcs): Likewise. (float-vlen4-funcs): Likewise. (float-vlen8-funcs): Likewise. (float-vlen8-avx2-funcs): Likewise. (float-vlen16-funcs): Likewise. (CFLAGS-test-double-vlen4-avx2.c): Remove variable. (CFLAGS-test-float-vlen8-avx2.c): Likewise. * sysdeps/x86_64/fpu/test-double-vlen4.h (TEST_VECTOR_cos): Remove macro. (TEST_VECTOR_sin): Likewise. (TEST_VECTOR_sincos): Likewise. (TEST_VECTOR_log): Likewise. (TEST_VECTOR_exp): Likewise. (TEST_VECTOR_pow): Likewise. * sysdeps/x86_64/fpu/test-double-vlen8.h (TEST_VECTOR_cos): Likewise. (TEST_VECTOR_sin): Likewise. (TEST_VECTOR_sincos): Likewise. (TEST_VECTOR_log): Likewise. (TEST_VECTOR_exp): Likewise. (TEST_VECTOR_pow): Likewise. * sysdeps/x86_64/fpu/test-float-vlen16.h (TEST_VECTOR_cosf): Likewise. (TEST_VECTOR_sinf): Likewise. (TEST_VECTOR_sincosf): Likewise. (TEST_VECTOR_logf): Likewise. (TEST_VECTOR_expf): Likewise. (TEST_VECTOR_powf): Likewise. * sysdeps/x86_64/fpu/test-float-vlen8.h (TEST_VECTOR_cosf): Likewise. (TEST_VECTOR_sinf): Likewise. (TEST_VECTOR_sincosf): Likewise. (TEST_VECTOR_logf): Likewise. (TEST_VECTOR_expf): Likewise. (TEST_VECTOR_powf): Likewise. * math/gen-libm-have-vector-test.sh: Remove file. * math/libm-test.inc: Likewise. * math/libm-test-support-double.c: Likewise. * math/libm-test-support-float.c: Likewise. * math/libm-test-support-ldouble.c: Likewise. * math/test-double-finite.c: Likewise.: Likewise. * math/test-double.c: Likewise. * math/test-float-finite.c: Likewise. * math/test-float.c: Likewise. * math/test-idouble.c: Likewise. * math/test-ifloat.c: Likewise. * math/test-ildouble.c: Likewise. * math/test-ldouble-finite.c: Likewise. * math/test-ldouble.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen2.h: Likewise. * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen16.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen4.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen4.h: Likewise. * sysdeps/x86_64/fpu/test-float-vlen8-avx2.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen8.c: Likewise. |
||
|
Joseph Myers
|
2c51dfd05d |
Move tests of catan, catanh to auto-libm-test-*.
This patch moves tests of catan and catanh with finite inputs (other than the divide-by-zero cases producing an exact infinity) to using the auto-libm-test machinery. Each of auto-libm-test-out-catan and auto-libm-test-out-catanh takes about three seconds to generate on my system (so in fact it wasn't necessary after all to defer the move to auto-libm-test-* until the output files were split up by function). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of catan and catanh. * math/auto-libm-test-out-catan: New generated file. * math/auto-libm-test-out-catanh: Likewise. * math/libm-test-catan.inc (catan_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs, except divide-by-zero cases, to auto-libm-test-in. * math/libm-test-catanh.inc (catanh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add catan and catanh. (libm-test-funcs-noauto): Remove catan and catanh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise. |
||
|
Joseph Myers
|
fa2a3dd7a3 |
Move tests of casin, casinh to auto-libm-test-*.
This patch moves tests of casin and casinh with finite inputs to using the auto-libm-test machinery. Each of auto-libm-test-out-casin and auto-libm-test-out-casinh takes about 38 minutes to generate on my system because of MPC slowness on special cases that appear in the tests (with MPC 1.0.3; I don't know to what extent current MPC master might speed it up). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of casin and casinh. * math/auto-libm-test-out-casin: New generated file. * math/auto-libm-test-out-casinh: Likewise. * math/libm-test-casin.inc (casin_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs to auto-libm-test-in. * math/libm-test-casinh.inc (casinh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add casin and casinh. (libm-test-funcs-noauto): Remove casin and casinh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise. |
||
|
Joseph Myers
|
6b8303a383 |
Move tests of cacos, cacosh to auto-libm-test-*.
This patch moves tests of cacos and cacosh with finite inputs to using the auto-libm-test machinery. Each of auto-libm-test-out-cacos and auto-libm-test-out-cacosh takes about 80 minutes to generate on my system because of MPC slowness on special cases that appear in the tests (with MPC 1.0.3; I don't know to what extent current MPC master might speed it up). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of cacos and cacosh. * math/auto-libm-test-out-cacos: New generated file. * math/auto-libm-test-out-cacosh: Likewise. * math/libm-test-cacos.inc (cacos_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs to auto-libm-test-in. * math/libm-test-cacosh.inc (cacosh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add cacos and cacosh. (libm-test-funcs-noauto): Remove cacos and cacosh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise. |
||
|
Joseph Myers
|
f7a51347a4 |
Revert header inclusion changes that break math/ testing on x86_64.
Revert: 2017-02-16 Zack Weinberg <zackw@panix.com> * sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h. Don't include init-arch.h. |
||
|
Zack Weinberg
|
ceaa98897c |
Add missing header files throughout the testsuite.
* crypt/md5.h: Test _LIBC with #if defined, not #if. * dirent/opendir-tst1.c: Include sys/stat.h. * dirent/tst-fdopendir.c: Include sys/stat.h. * dirent/tst-fdopendir2.c: Include stdlib.h. * dirent/tst-scandir.c: Include stdbool.h. * elf/tst-auditmod1.c: Include link.h and stddef.h. * elf/tst-tls15.c: Include stdlib.h. * elf/tst-tls16.c: Include stdlib.h. * elf/tst-tls17.c: Include stdlib.h. * elf/tst-tls18.c: Include stdlib.h. * iconv/tst-iconv6.c: Include endian.h. * iconvdata/bug-iconv11.c: Include limits.h. * io/test-utime.c: Include stdint.h. * io/tst-faccessat.c: Include sys/stat.h. * io/tst-fchmodat.c: Include sys/stat.h. * io/tst-fchownat.c: Include sys/stat.h. * io/tst-fstatat.c: Include sys/stat.h. * io/tst-futimesat.c: Include sys/stat.h. * io/tst-linkat.c: Include sys/stat.h. * io/tst-mkdirat.c: Include sys/stat.h and stdbool.h. * io/tst-mkfifoat.c: Include sys/stat.h and stdbool.h. * io/tst-mknodat.c: Include sys/stat.h and stdbool.h. * io/tst-openat.c: Include stdbool.h. * io/tst-readlinkat.c: Include sys/stat.h. * io/tst-renameat.c: Include sys/stat.h. * io/tst-symlinkat.c: Include sys/stat.h. * io/tst-unlinkat.c: Include stdbool.h. * libio/bug-memstream1.c: Include stdlib.h. * libio/bug-wmemstream1.c: Include stdlib.h. * libio/tst-fwrite-error.c: Include stdlib.h. * libio/tst-memstream1.c: Include stdlib.h. * libio/tst-memstream2.c: Include stdlib.h. * libio/tst-memstream3.c: Include stdlib.h. * malloc/tst-interpose-aux.c: Include stdint.h. * misc/tst-preadvwritev-common.c: Include sys/stat.h. * nptl/tst-basic7.c: Include limits.h. * nptl/tst-cancel25.c: Include pthread.h, not pthreadP.h. * nptl/tst-cancel4.c: Include stddef.h, limits.h, and sys/stat.h. * nptl/tst-cancel4_1.c: Include stddef.h. * nptl/tst-cancel4_2.c: Include stddef.h. * nptl/tst-cond16.c: Include limits.h. Use sysconf(_SC_PAGESIZE) instead of __getpagesize. * nptl/tst-cond18.c: Include limits.h. Use sysconf(_SC_PAGESIZE) instead of __getpagesize. * nptl/tst-cond4.c: Include stdint.h. * nptl/tst-cond6.c: Include stdint.h. * nptl/tst-stack2.c: Include limits.h. * nptl/tst-stackguard1.c: Include stddef.h. * nptl/tst-tls4.c: Include stdint.h. Don't include tls.h. * nptl/tst-tls4moda.c: Include stddef.h. Don't include stdio.h, unistd.h, or tls.h. * nptl/tst-tls4modb.c: Include stddef.h. Don't include stdio.h, unistd.h, or tls.h. * nptl/tst-tls5.h: Include stddef.h. Don't include stdlib.h or tls.h. * posix/tst-getaddrinfo2.c: Include stdio.h. * posix/tst-getaddrinfo5.c: Include stdio.h. * posix/tst-pathconf.c: Include sys/stat.h. * posix/tst-posix_fadvise-common.c: Include stdint.h. * posix/tst-preadwrite-common.c: Include sys/stat.h. * posix/tst-regex.c: Include stdint.h. Don't include spawn.h or spawn_int.h. * posix/tst-regexloc.c: Don't include spawn.h or spawn_int.h. * posix/tst-vfork3.c: Include sys/stat.h. * resolv/tst-bug18665-tcp.c: Include stdlib.h. * resolv/tst-res_hconf_reorder.c: Include stdlib.h. * resolv/tst-resolv-search.c: Include stdlib.h. * stdio-common/tst-fmemopen2.c: Include stdint.h. * stdio-common/tst-vfprintf-width-prec.c: Include stdlib.h. * stdlib/test-canon.c: Include sys/stat.h. * stdlib/tst-tls-atexit.c: Include stdbool.h. * string/test-memchr.c: Include stdint.h. * string/tst-cmp.c: Include stdint.h. * sysdeps/pthread/tst-timer.c: Include stdint.h. * sysdeps/unix/sysv/linux/tst-sync_file_range.c: Include stdint.h. * sysdeps/wordsize-64/tst-writev.c: Include limits.h and stdint.h. * sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/tst-auditmod10b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod3b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod4b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod5b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod6b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod6c.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod7b.c: Include link.h and stddef.h. * time/clocktest.c: Include stdint.h. * time/tst-posixtz.c: Include stdint.h. * timezone/tst-timezone.c: Include stdint.h. |
||
|
Joseph Myers
|
10303eb74b |
Move most libmvec test contents from .c to .h files.
The libmvec tests put substantive, architecture-specific contents in .c files such as test-double-vlen4.c, so making those files architecture-specific and causing issues for generating such files automatically when splitting up tests by function. This patch moves all the substantive contents to .h files, so the .c files only include the .h file and then libm-test.c. This allows for automatic generation of per-function .c files in future. The .h files in turn #include or #include_next the architecture-independent file and add the architecture-specific definitions to that. (Splitting by function should in fact allow the TEST_VECTOR_* macros to be replaced by sysdeps makefile information on which functions to test in each case, removing the need for gen-libm-have-vector-test.sh as well as removing the need for some of the architecture-specific headers.) Tested for x86_64. * sysdeps/x86_64/fpu/test-double-vlen2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen2.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen4-avx2.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen4.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen4.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen8.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen8.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen16.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen16.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen4.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen4.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen8-avx2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen8-avx2.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen8.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen8.h: ... here. New file. |
||
|
H.J. Lu
|
3403a17fea |
x86-64: Verify that _dl_runtime_resolve preserves vector registers
On x86-64, _dl_runtime_resolve must preserve the first 8 vector registers. Add 3 _dl_runtime_resolve tests to verify that SSE, AVX and AVX512 registers are preserved. * sysdeps/x86_64/Makefile (tests): Add tst-sse, tst-avx and tst-avx512. (test-extras): Add tst-avx-aux and tst-avx512-aux. (extra-test-objs): Add tst-avx-aux.o and tst-avx512-aux.o. (modules-names): Add tst-ssemod, tst-avxmod and tst-avx512mod. ($(objpfx)tst-sse): New rule. ($(objpfx)tst-avx): Likewise. ($(objpfx)tst-avx512): Likewise. (CFLAGS-tst-avx-aux.c): New. (CFLAGS-tst-avxmod.c): Likewise. (CFLAGS-tst-avx512-aux.c): Likewise. (CFLAGS-tst-avx512mod.c): Likewise. * sysdeps/x86_64/tst-avx-aux.c: New file. * sysdeps/x86_64/tst-avx.c: Likewise. * sysdeps/x86_64/tst-avx512-aux.c: Likewise. * sysdeps/x86_64/tst-avx512.c: Likewise. * sysdeps/x86_64/tst-avx512mod.c: Likewise. * sysdeps/x86_64/tst-avxmod.c: Likewise. * sysdeps/x86_64/tst-sse.c: Likewise. * sysdeps/x86_64/tst-ssemod.c: Likewise. |
||
|
Adhemerval Zanella
|
f2d7f23a30 |
Remove i686, x86_64, and powerpc strtok implementations
Based on comments on previous attempt to address BZ#16640 [1], the idea is not support invalid use of strtok (the original bug report proposal). This leader to a new strtok optimized strtok implementation [2]. The idea of this patch is to fix BZ#16640 to align all the implementations to a same contract. However, with newer strtok code it is better to get remove the old assembly ones instead of fix them. For x86 is a gain in all cases since the new implementation can potentially use sse2/sse42 implementation for strspn and strcspn. This shows a better performance on both i686 and x86_64 using the string benchtests. On powerpc64 the gains are mixed, where only for larger inputs or keys some gains are showns (based on benchtest it seems that it shows some gains for keys larger than 10 and inputs larger than 32). I would prefer to remove the optimized implementation based on first code simplicity and second because some more gain could be optimized using a better optimized strcspn/strspn code (as for x86). However if powerpc arch maintainers prefer I can send a v2 with the assembly code adjusted instead. Checked on x86_64-linux-gnu, i686-linux-gnu, and powerpc64le-linux-gnu. [BZ #16640] * sysdeps/i386/i686/strtok.S: Remove file. * sysdeps/i386/i686/strtok_r.S: Likewise. * sysdeps/i386/strtok.S: Likewise. * sysdeps/i386/strtok_r.S: Likewise. * sysdeps/powerpc/powerpc64/strtok.S: Likewise. * sysdeps/powerpc/powerpc64/strtok_r.S: Likewise. * sysdeps/x86_64/strtok.S: Likewise. * sysdeps/x86_64/strtok_r.S: Likewise. [1] https://sourceware.org/ml/libc-alpha/2016-10/msg00411.html [2] https://sourceware.org/ml/libc-alpha/2016-12/msg00461.html |
||
|
H.J. Lu
|
6fab532b47 |
Allow IFUNC relocation against unrelocated shared library
IFUNC relocation against definition in unrelocated shared library will lead to segfault when the IFUNC function is called. This patch allows such IFUNC relocations with a warning. This isn't a real fix for https://sourceware.org/bugzilla/show_bug.cgi?id=21041 It simply allows the program to load. The program will segfault when longjmp is called. * sysdeps/i386/dl-machine.h (elf_machine_rel): Replace _dl_fatal_printf with _dl_error_printf for IFUNC relocation against unrelocated shared library. * sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise. |
||
|
H.J. Lu
|
02b78ff749 |
Add VZEROUPPER to memset-vec-unaligned-erms.S [BZ #21081]
Since memset-vec-unaligned-erms.S has VDUP_TO_VEC0_AND_SET_RETURN at
function entry, memset optimized for AVX2 and AVX512 will always use
ymm/zmm register. VZEROUPPER should be placed before ret in
L(stosb):
movq %rdx, %rcx
movzbl %sil, %eax
movq %rdi, %rdx
rep stosb
movq %rdx, %rax
ret
since it can be reached from
L(stosb_more_2x_vec):
cmpq $REP_STOSB_THRESHOLD, %rdx
ja L(stosb)
[BZ #21081]
* sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(L(stosb)): Add VZEROUPPER before ret.
|
||
|
Adhemerval Zanella
|
8dad72997a |
Fix x86 strncat optimized implementation for large sizes
Similar to BZ#19387, BZ#21014, and BZ#20971, both x86 sse2 strncat
optimized assembly implementations do not handle the size overflow
correctly.
The x86_64 one is in fact an issue with strcpy-sse2-unaligned, but
that is triggered also with strncat optimized implementation.
This patch uses a similar strategy used on
|
||
|
Joseph Myers
|
bfff8b1bec | Update copyright dates with scripts/update-copyrights. | ||
|
Adhemerval Zanella
|
3daef2c8ee |
Fix x86_64 memchr for large input sizes
Current optimized memchr for x86_64 does for input arguments pointers
module 64 in range of [49,63] if there is no searchr char in the rest
of 64-byte block a pointer addition which might overflow:
* sysdeps/x86_64/memchr.S
77 .p2align 4
78 L(unaligned_no_match):
79 add %rcx, %rdx
Add (uintptr_t)s % 16 to n in %rdx.
80 sub $16, %rdx
81 jbe L(return_null)
This patch fixes by adding a saturated math that sets a maximum pointer
value if it overflows (UINTPTR_MAX).
Checked on x86_64-linux-gnu and powerpc64-linux-gnu.
[BZ# 19387]
* sysdeps/x86_64/memchr.S (memchr): Avoid overflow in pointer
addition.
* string/test-memchr.c (do_test): Remove alignment limitation.
(test_main): Add test that trigger BZ# 19387.
|
||
Nick Alcock
|
de6591238b |
Do not stack-protect ifunc resolvers [BZ #7065]
When dynamically linking, ifunc resolvers are called before TLS is
initialized, so they cannot be safely stack-protected.
We avoid disabling stack-protection on large numbers of files by
using __attribute__ ((__optimize__ ("-fno-stack-protector")))
to turn it off just for the resolvers themselves. (We provide
the attribute even when statically linking, because we will later
use it elsewhere too.)
|
||
|
Nick Alcock
|
fcd942370f |
x86_64: tst-quad1pie, tst-quad2pie: compile with -fPIE [BZ #7065]
With stack protection enabled, these files have external symbol references for the first time, so the fact that they are not compiled with -fPIE and are then linked into a -pie binary starts to hurt. |
||
|
Joseph Myers
|
0a2546cdaa |
Fix x86, x86_64 fmax, fmin sNaN handling, add tests (bug 20947).
Various fmax and fmin function implementations mishandle sNaN arguments: (a) When both arguments are NaNs, the return value should be a qNaN, but sometimes it is an sNaN if at least one argument is an sNaN. (b) Under TS 18661-1 semantics, if either argument is an sNaN then the result should be a qNaN (whereas if one argument is a qNaN and the other is not a NaN, the result should be the non-NaN argument). Various implementations treat sNaNs like qNaNs here. This patch fixes the x86 and x86_64 versions (ignoring float and double for 32-bit x86 given the inability to reliably avoid the sNaN turning into a qNaN before it gets to the called function). Tests of sNaN inputs to these functions are added. Note on architecture versions I haven't changed for this issue: AArch64 already gets this right (it uses a hardware instruction with the correct semantics for both quiet and signaling NaNs) and does not need changes. It's possible Alpha, IA64, SPARC might need changes (this would be shown by the testsuite if so). Tested for x86_64 and x86 (both i686 and i586 builds, to cover the different x86 implementations). [BZ #20947] * sysdeps/i386/fpu/s_fmaxl.S (__fmaxl): Add the arguments when either is a signaling NaN. * sysdeps/i386/fpu/s_fminl.S (__fminl): Likewise. Make code follow fmaxl more closely. * sysdeps/i386/i686/fpu/s_fmaxl.S (__fmaxl): Add the arguments when either is a signaling NaN. * sysdeps/i386/i686/fpu/s_fminl.S (__fminl): Likewise. * sysdeps/x86_64/fpu/s_fmax.S (__fmax): Likewise. * sysdeps/x86_64/fpu/s_fmaxf.S (__fmaxf): Likewise. * sysdeps/x86_64/fpu/s_fmaxl.S (__fmaxl): Likewise. * sysdeps/x86_64/fpu/s_fmin.S (__fmin): Likewise. * sysdeps/x86_64/fpu/s_fminf.S (__fminf): Likewise. * sysdeps/x86_64/fpu/s_fminl.S (__fminl): Likewise. * math/libm-test.inc (fmax_test_data): Add tests of sNaN inputs. (fmin_test_data): Likewise. |
||
|
Joseph Myers
|
a91fd168a0 |
Fix x86_64/x86 powl handling of sNaN arguments (bug 20916).
The x86_64/x86 powl implementations mishandle sNaN arguments, both by returning sNaN in some cases (instead of doing arithmetic on the arguments to produce the result when NaN arguments result in NaN results) and by treating sNaN the same as qNaN for arguments (1, sNaN) and (sNaN, 0), contrary to TS 18661-1 which requires those cases to return qNaN instead of 1. This patch makes the x86_64/x86 powl implementations follow TS 18661-1 semantics for sNaN arguments; sNaN tests are also added for pow. Given the problems with testing float and double sNaN arguments on 32-bit x86 (sNaN tests disabled because the compiler may convert unnecessarily to a qNaN when passing arguments), no changes are made to the powf and pow implementations there. Tested for x86_64 and x86. [BZ #20916] * sysdeps/i386/fpu/e_powl.S (__ieee754_powl): Do not return 1 for arguments (sNaN, 0) or (1, sNaN). Do arithmetic on NaN arguments to compute result. * sysdeps/x86_64/fpu/e_powl.S (__ieee754_powl): Likewise. * math/libm-test.inc (pow_test_data): Add tests of sNaN arguments. |
||
|
Florian Weimer
|
b04beebf07 |
ld.so: Remove __libc_memalign
It is no longer needed since commit
|
||
|
Florian Weimer
|
9e78f6f6e7 |
Implement _dl_catch_error, _dl_signal_error in libc.so [BZ #16628]
This change moves the main implementation of _dl_catch_error, _dl_signal_error to libc.so, where TLS variables can be used directly. This removes a writable function pointer from the rtld_global variable. For use during initial relocation, minimal implementations of these functions are provided in ld.so. These are eventually interposed by the libc.so implementations. This is implemented by compiling elf/dl-error-skeleton.c twice, via elf/dl-error.c and elf/dl-error-minimal.c. As a side effect of this change, the static version of dl-error.c no longer includes support for the _dl_signal_cerror/_dl_receive_error mechanism because it is only used in ld.so. |
||
|
H.J. Lu
|
c9070e6305 |
X86_64: Don't use PLT nor GOT in static archives [BZ #20750]
There is no need to use PLT nor GOT in static archives to branch to a function, regardless whether static archives is compiled with PIC or not. When static archives are used to create dynamic executable, PLT/GOT may be used. The resulting executable still works correctly. [BZ #20750] * sysdeps/x86_64/sysdep.h (JUMPTARGET): Check SHARED instead of PIC. |
||
|
Adhemerval Zanella
|
c579f48edb |
Remove cached PID/TID in clone
This patch remove the PID cache and usage in current GLIBC code. Current
usage is mainly used a performance optimization to avoid the syscall,
however it adds some issues:
- The exposed clone syscall will try to set pid/tid to make the new
thread somewhat compatible with current GLIBC assumptions. This cause
a set of issue with new workloads and usecases (such as BZ#17214 and
[1]) as well for new internal usage of clone to optimize other algorithms
(such as clone plus CLONE_VM for posix_spawn, BZ#19957).
- The caching complexity also added some bugs in the past [2] [3] and
requires more effort of each port to handle such requirements (for
both clone and vfork implementation).
- Caching performance gain in mainly on getpid and some specific
code paths. The getpid performance leverage is questionable [4],
either by the idea of getpid being a hotspot as for the getpid
implementation itself (if it is indeed a justifiable hotspot a
vDSO symbol could let to a much more simpler solution).
Other usage is mainly for non usual code paths, such as pthread
cancellation signal and handling.
For thread creation (on stack allocation) the code simplification in fact
adds some performance gain due the no need of transverse the stack cache
and invalidate each element pid.
Other thread usages will require a direct getpid syscall, such as
cancellation/setxid signal, thread cancellation, thread fail path (at
create_thread), and thread signal (pthread_kill and pthread_sigqueue).
However these are hardly usual hotspots and I think adding a syscall is
justifiable.
It also simplifies both the clone and vfork arch-specific implementation.
And by review each fork implementation there are some discrepancies that
this patch also solves:
- microblaze clone/vfork does not set/reset the pid/tid field
- hppa uses the default vfork implementation that fallback to fork.
Since vfork is deprecated I do not think we should bother with it.
The patch also removes the TID caching in clone. My understanding for
such semantic is try provide some pthread usage after a user program
issue clone directly (as done by thread creation with CLONE_PARENT_SETTID
and pthread tid member). However, as stated before in multiple discussions
threads, GLIBC provides clone syscalls without further supporting all this
semantics.
I ran a full make check on x86_64, x32, i686, armhf, aarch64, and powerpc64le.
For sparc32, sparc64, and mips I ran the basic fork and vfork tests from
posix/ folder (on a qemu system). So it would require further testing
on alpha, hppa, ia64, m68k, nios2, s390, sh, and tile (I excluded microblaze
because it is already implementing the patch semantic regarding clone/vfork).
[1] https://codereview.chromium.org/800183004/
[2] https://sourceware.org/ml/libc-alpha/2006-07/msg00123.html
[3] https://sourceware.org/bugzilla/show_bug.cgi?id=15368
[4] http://yarchive.net/comp/linux/getpid_caching.html
* sysdeps/nptl/fork.c (__libc_fork): Remove pid cache setting.
* nptl/allocatestack.c (allocate_stack): Likewise.
(__reclaim_stacks): Likewise.
(setxid_signal_thread): Obtain pid through syscall.
* nptl/nptl-init.c (sigcancel_handler): Likewise.
(sighandle_setxid): Likewise.
* nptl/pthread_cancel.c (pthread_cancel): Likewise.
* sysdeps/unix/sysv/linux/pthread_kill.c (__pthread_kill): Likewise.
* sysdeps/unix/sysv/linux/pthread_sigqueue.c (pthread_sigqueue):
Likewise.
* sysdeps/unix/sysv/linux/createthread.c (create_thread): Likewise.
* sysdeps/unix/sysv/linux/getpid.c: Remove file.
* nptl/descr.h (struct pthread): Change comment about pid value.
* nptl/pthread_getattr_np.c (pthread_getattr_np): Remove thread
pid assert.
* sysdeps/unix/sysv/linux/pthread-pids.h (__pthread_initialize_pids):
Do not set pid value.
* nptl_db/td_ta_thr_iter.c (iterate_thread_list): Remove thread
pid cache check.
* nptl_db/td_thr_validate.c (td_thr_validate): Likewise.
* sysdeps/aarch64/nptl/tcb-offsets.sym: Remove pid offset.
* sysdeps/alpha/nptl/tcb-offsets.sym: Likewise.
* sysdeps/arm/nptl/tcb-offsets.sym: Likewise.
* sysdeps/hppa/nptl/tcb-offsets.sym: Likewise.
* sysdeps/i386/nptl/tcb-offsets.sym: Likewise.
* sysdeps/ia64/nptl/tcb-offsets.sym: Likewise.
* sysdeps/m68k/nptl/tcb-offsets.sym: Likewise.
* sysdeps/microblaze/nptl/tcb-offsets.sym: Likewise.
* sysdeps/mips/nptl/tcb-offsets.sym: Likewise.
* sysdeps/nios2/nptl/tcb-offsets.sym: Likewise.
* sysdeps/powerpc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/s390/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sh/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sparc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/tile/nptl/tcb-offsets.sym: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym: Likewise.
* sysdeps/unix/sysv/linux/aarch64/clone.S: Remove pid and tid caching.
* sysdeps/unix/sysv/linux/alpha/clone.S: Likewise.
* sysdeps/unix/sysv/linux/arm/clone.S: Likewise.
* sysdeps/unix/sysv/linux/hppa/clone.S: Likewise.
* sysdeps/unix/sysv/linux/i386/clone.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/clone2.S: Likewise.
* sysdeps/unix/sysv/linux/mips/clone.S: Likewise.
* sysdeps/unix/sysv/linux/nios2/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sh/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/tile/clone.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/aarch64/vfork.S: Remove pid set and reset.
* sysdeps/unix/sysv/linux/alpha/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/arm/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/i386/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/m68k/clone.S: Likewise.
* sysdeps/unix/sysv/linux/m68k/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/mips/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/nios2/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sh/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/tile/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/tst-clone2.c (f): Remove direct pthread
struct access.
(clone_test): Remove function.
(do_test): Rewrite to take in consideration pid is not cached anymore.
|
||
Aurelien Jarno
|
380ec16d62 |
x86_64: fix static build of __memcpy_chk for compilers defaulting to PIC/PIE
When glibc is compiled with gcc 6.2 that has been configured with to default to PIC/PIE, the static version of __memcpy_chk is not built, as the test is done on PIC instead of SHARED. Fix the test to check for SHARED, like it is done for similar functions like memmove_chk. Changelog: * sysdeps/x86_64/memcpy_chk.S (__memcpy_chk): Check for SHARED instead of PIC. |
||
|
Joseph Myers
|
799131036e |
Do not hardcode platform names in manual/libm-err-tab.pl (bug 14139).
manual/libm-err-tab.pl hardcodes a list of names for particular platforms (mapping from sysdeps directory name to friendly name for the manual). This goes against the principle of keeping information about individual platforms in their corresponding sysdeps directory, and the list is also very out-of-date regarding supported platforms and their corresponding sysdeps directories. This patch fixes this by adding a libm-test-ulps-name file alongside each libm-test-ulps file. The script then gets the friendly name from that file, which is required to exist, so it no longer needs to allow for the mapping being missing. Tested for x86_64. [BZ #14139] * manual/libm-err-tab.pl (%pplatforms): Initialize to empty. (find_files): Obtain platform name from libm-test-ulps-name and store in %pplatforms. (canonicalize_platform): Remove. (print_platforms): Use $pplatforms directly. (by_platforms): Do not allow for platforms missing from %pplatforms. * sysdeps/aarch64/libm-test-ulps-name: New file. * sysdeps/alpha/fpu/libm-test-ulps-name: Likewise. * sysdeps/arm/libm-test-ulps-name: Likewise. * sysdeps/generic/libm-test-ulps-name: Likewise. * sysdeps/hppa/fpu/libm-test-ulps-name: Likewise. * sysdeps/i386/fpu/libm-test-ulps-name: Likewise. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps-name: Likewise. * sysdeps/ia64/fpu/libm-test-ulps-name: Likewise. * sysdeps/m68k/coldfire/fpu/libm-test-ulps-name: Likewise. * sysdeps/m68k/m680x0/fpu/libm-test-ulps-name: Likewise. * sysdeps/microblaze/libm-test-ulps-name: Likewise. * sysdeps/mips/mips32/libm-test-ulps-name: Likewise. * sysdeps/mips/mips64/libm-test-ulps-name: Likewise. * sysdeps/nios2/libm-test-ulps-name: Likewise. * sysdeps/powerpc/fpu/libm-test-ulps-name: Likewise. * sysdeps/powerpc/nofpu/libm-test-ulps-name: Likewise. * sysdeps/s390/fpu/libm-test-ulps-name: Likewise. * sysdeps/sh/libm-test-ulps-name: Likewise. * sysdeps/sparc/fpu/libm-test-ulps-name: Likewise. * sysdeps/tile/libm-test-ulps-name: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps-name: Likewise. |
||
|
H.J. Lu
|
0e6d3adc60 |
Check IFUNC definition in unrelocated shared library [BZ #20019]
Calling an IFUNC function defined in unrelocated shared library may lead to segfault. This patch issues an error message to request relinking the shared library if it references IFUNC function defined in the unrelocated shared library. [BZ #20019] * sysdeps/i386/dl-machine.h (elf_machine_rel): Check IFUNC definition in unrelocated shared library. * sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise. |
||
|
Joseph Myers
|
f280fa6d17 |
Use __builtin_fma more in dbl-64 code.
sysdeps/ieee754/dbl-64/dla.h can use a macro DLA_FMS for more efficient double-width operations when fused multiply-subtract is supported. However, this macro is only defined for x86_64, conditional on architecture-specific __FMA4__. This patch makes the code use __builtin_fma conditional on __FP_FAST_FMA, as used elsewhere in glibc. Tested for x86_64, x86 and powerpc. On powerpc (where this is causing fused operations to be used where they weren't previously) I see an increase from 1ulp to 2ulp in the imaginary part of clog10: testing double (without inline functions) Failure: Test: Imaginary part of: clog10 (0x1.7a858p+0 - 0x6.d940dp-4 i) Result: is: -1.2237865208199886e-01 -0x1.f5435146bb61ap-4 should be: -1.2237865208199888e-01 -0x1.f5435146bb61cp-4 difference: 2.7755575615628914e-17 0x1.0000000000000p-55 ulp : 2.0000 max.ulp : 1.0000 Maximal error of real part of: clog10 is : 3 ulp accepted: 3 ulp Maximal error of imaginary part of: clog10 is : 2 ulp accepted: 1 ulp This is actually resulting from atan2 becoming *more* accurate (atan2 (-0x6.d940dp-4, 0x1.7a858p+0) should ideally be -0x1.208cd6e841554p-2 but was -0x1.208cd6e841555p-2 from a powerpc libm built before this change, and is -0x1.208cd6e841554p-2 from a powerpc libm built after this change). Since these functions are not expected to be correctly rounding by glibc's accuracy goals, neither result is a problem, but this does imply that some of this code, although designed to be correctly rounding, is not in fact correctly rounding (possibly because of GCC creating fused operations where the code does not expect it, something we've only disabled for specific functions where it was found to cause large errors). (Of course as previously discussed I think we should remove the slow cases where an error analysis shows this wouldn't increase the errors much above 0.5ulp; it's only functions such as cratan2 that are expected to be correctly rounding, not atan2.) * sysdeps/ieee754/dbl-64/dla.h [__FP_FAST_FMA] (DLA_FMS): Define macro to use __builtin_fma. * sysdeps/x86_64/fpu/dla.h: Remove file. |
||
|
Joseph Myers
|
ec94343f59 |
Add femode_t functions.
TS 18661-1 defines a type femode_t to represent the set of dynamic floating-point control modes (such as the rounding mode and trap enablement modes), and functions fegetmode and fesetmode to manipulate those modes (without affecting other state such as the raised exception flags) and a corresponding macro FE_DFL_MODE. This patch series implements those interfaces for glibc. This first patch adds the architecture-independent pieces, the x86 and x86_64 implementations, and the <bits/fenv.h> and ABI baseline updates for all architectures so glibc keeps building and passing the ABI tests on all architectures. Subsequent patches add the fegetmode and fesetmode implementations for other architectures. femode_t is generally an integer type - the same type as fenv_t, or as the single element of fenv_t where fenv_t is a structure containing a single integer (or the single relevant element, where it has elements for both status and control registers) - except where architecture properties or consistency with the fenv_t implementation indicate otherwise. FE_DFL_MODE follows FE_DFL_ENV in whether it's a magic pointer value (-1 cast to const femode_t *), a value that can be distinguished from valid pointers by its high bits but otherwise contains a representation of the desired register contents, or a pointer to a constant variable (the powerpc case; __fe_dfl_mode is added as an exported constant object, an alias to __fe_dfl_env). Note that where architectures (that share a register between control and status bits) gain definitions of new floating-point control or status bits in future, the implementations of fesetmode for those architectures may need updating (depending on whether the new bits are control or status bits and what the implementation does with previously unknown bits), just like existing implementations of <fenv.h> functions that take care not to touch reserved bits may need updating when the set of reserved bits changes. (As any new bits are outside the scope of ISO C, that's just a quality-of-implementation issue for supporting them, not a conformance issue.) As with fenv_t, femode_t should properly include any software DFP rounding mode (and for both fenv_t and femode_t I'd consider that fragment of DFP support appropriate for inclusion in glibc even in the absence of the rest of libdfp; hardware DFP rounding modes should already be included if the definitions of which bits are status / control bits are correct). Tested for x86_64, x86, mips64 (hard float, and soft float to test the fallback version), arm (hard float) and powerpc (hard float, soft float and e500). Other architecture versions are untested. * math/fegetmode.c: New file. * math/fesetmode.c: Likewise. * sysdeps/i386/fpu/fegetmode.c: Likewise. * sysdeps/i386/fpu/fesetmode.c: Likewise. * sysdeps/x86_64/fpu/fegetmode.c: Likewise. * sysdeps/x86_64/fpu/fesetmode.c: Likewise. * math/fenv.h: Update comment on inclusion of <bits/fenv.h>. [__GLIBC_USE (IEC_60559_BFP_EXT)] (fegetmode): New function declaration. [__GLIBC_USE (IEC_60559_BFP_EXT)] (fesetmode): Likewise. * bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/aarch64/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/alpha/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/arm/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/hppa/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/ia64/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/m68k/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/microblaze/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/mips/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/nios2/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/powerpc/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (__fe_dfl_mode): New variable declaration. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/s390/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/sh/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/sparc/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/tile/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * sysdeps/x86/fpu/bits/fenv.h [__GLIBC_USE (IEC_60559_BFP_EXT)] (femode_t): New typedef. [__GLIBC_USE (IEC_60559_BFP_EXT)] (FE_DFL_MODE): New macro. * manual/arith.texi (FE_DFL_MODE): Document macro. (fegetmode): Document function. (fesetmode): Likewise. * math/Versions (fegetmode): New libm symbol at version GLIBC_2.25. (fesetmode): Likewise. * math/Makefile (libm-support): Add fegetmode and fesetmode. (tests): Add test-femode and test-femode-traps. * math/test-femode-traps.c: New file. * math/test-femode.c: Likewise. * sysdeps/powerpc/fpu/fenv_const.c (__fe_dfl_mode): Declare as alias for __fe_dfl_env. * sysdeps/powerpc/nofpu/fenv_const.c (__fe_dfl_mode): Likewise. * sysdeps/powerpc/powerpc32/e500/nofpu/fenv_const.c (__fe_dfl_mode): Likewise. * sysdeps/powerpc/Versions (__fe_dfl_mode): New libm symbol at version GLIBC_2.25. * sysdeps/nacl/libm.abilist: Update. * sysdeps/unix/sysv/linux/aarch64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/alpha/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/arm/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/hppa/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/i386/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/ia64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/m68k/coldfire/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/m68k/m680x0/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/microblaze/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/mips/mips32/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/mips/mips64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/nios2/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc64/libm-le.abilist: Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/s390/s390-64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/sh/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/sparc/sparc32/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/sparc/sparc64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/tile/tilegx/tilegx32/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/tile/tilegx/tilegx64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/tile/tilepro/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/x86_64/64/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/x86_64/x32/libm.abilist: Likewise. |
||
|
H.J. Lu
|
fb0f7a6755 |
X86-64: Add _dl_runtime_resolve_avx[512]_{opt|slow} [BZ #20508]
There is transition penalty when SSE instructions are mixed with 256-bit AVX or 512-bit AVX512 load instructions. Since _dl_runtime_resolve_avx and _dl_runtime_profile_avx512 save/restore 256-bit YMM/512-bit ZMM registers, there is transition penalty when SSE instructions are used with lazy binding on AVX and AVX512 processors. To avoid SSE transition penalty, if only the lower 128 bits of the first 8 vector registers are non-zero, we can preserve %xmm0 - %xmm7 registers with the zero upper bits. For AVX and AVX512 processors which support XGETBV with ECX == 1, we can use XGETBV with ECX == 1 to check if the upper 128 bits of YMM registers or the upper 256 bits of ZMM registers are zero. We can restore only the non-zero portion of vector registers with AVX/AVX512 load instructions which will zero-extend upper bits of vector registers. This patch adds _dl_runtime_resolve_sse_vex which saves and restores XMM registers with 128-bit AVX store/load instructions. It is used to preserve YMM/ZMM registers when only the lower 128 bits are non-zero. _dl_runtime_resolve_avx_opt and _dl_runtime_resolve_avx512_opt are added and used on AVX/AVX512 processors supporting XGETBV with ECX == 1 so that we store and load only the non-zero portion of vector registers. This avoids SSE transition penalty caused by _dl_runtime_resolve_avx and _dl_runtime_profile_avx512 when only the lower 128 bits of vector registers are used. _dl_runtime_resolve_avx_slow is added and used for AVX processors which don't support XGETBV with ECX == 1. Since there is no SSE transition penalty on AVX512 processors which don't support XGETBV with ECX == 1, _dl_runtime_resolve_avx512_slow isn't provided. [BZ #20495] [BZ #20508] * sysdeps/x86/cpu-features.c (init_cpu_features): For Intel processors, set Use_dl_runtime_resolve_slow and set Use_dl_runtime_resolve_opt if XGETBV suports ECX == 1. * sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt): New. (bit_arch_Use_dl_runtime_resolve_slow): Likewise. (index_arch_Use_dl_runtime_resolve_opt): Likewise. (index_arch_Use_dl_runtime_resolve_slow): Likewise. * sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup): Use _dl_runtime_resolve_avx512_opt and _dl_runtime_resolve_avx_opt if Use_dl_runtime_resolve_opt is set. Use _dl_runtime_resolve_slow if Use_dl_runtime_resolve_slow is set. * sysdeps/x86_64/dl-trampoline.S: Include <cpu-features.h>. (_dl_runtime_resolve_opt): New. Defined for AVX and AVX512. (_dl_runtime_resolve): Add one for _dl_runtime_resolve_sse_vex. * sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx_slow): New. (_dl_runtime_resolve_opt): Likewise. (_dl_runtime_profile): Define only if _dl_runtime_profile is defined. |