Commit Graph

175 Commits

Author SHA1 Message Date
H.J. Lu
95b93c6e0d x86: Add sysdeps/x86/sysdep.h
Add a new header file, sysdeps/x86/sysdep.h, for common assembly code
macros between i386 and x86-64.  Tested on i686 and x86-64.  There are
no differences in outputs of "readelf -a" and "objdump -dw" on all glibc
shared objects before and after the patch.

	* sysdeps/i386/sysdep.h: Include <sysdeps/x86/sysdep.h> instead
	of <sysdeps/generic/sysdep.h>.
	(ALIGNARG): Removed.
	(ASM_SIZE_DIRECTIVE): Likewise.
	(ENTRY): Likewise.
	(END): Likewise.
	(ENTRY_CHK): Likewise.
	(END_CHK): Likewise.
	(syscall_error): Likewise.
	(mcount): Likewise.
	(PSEUDO_END): Likewise.
	(L): Likewise.
	(atom_text_section): Likewise.
	* sysdeps/x86/sysdep.h: New file.
	* sysdeps/x86_64/sysdep.h: Include <sysdeps/x86/sysdep.h> instead
	of <sysdeps/generic/sysdep.h>.
	(ALIGNARG): Removed.
	(ASM_SIZE_DIRECTIVE): Likewise.
	(ENTRY): Likewise.
	(END): Likewise.
	(ENTRY_CHK): Likewise.
	(END_CHK): Likewise.
	(syscall_error): Likewise.
	(mcount): Likewise.
	(PSEUDO_END): Likewise.
	(L): Likewise.
	(atom_text_section): Likewise.
2017-11-01 05:37:26 -07:00
H.J. Lu
4ad5106e3b sysdeps/x86/libc-start.c: Add /* !SHARED */
* sysdeps/x86/libc-start.c: Add /* !SHARED */.
2017-10-30 13:40:28 -07:00
H.J. Lu
fe326df7b0 Reformat sysdeps/x86/libc-start.c
* sysdeps/x86/libc-start.c: Reformat.
2017-10-30 13:01:18 -07:00
Joseph Myers
91c3985c23 Update x86 fix-fp-int-compare-invalid.h for GCC 8.
The glibc implementation of iseqsig relies on ordered comparison
operators raising the "invalid" exception for quiet NaN operands, with
a workaround on platforms where a GCC bug means that exception is not
raised.  For x86, that bug has now been fixed for GCC 8, so this patch
disables the workaround in that case.  If and when the corresponding
bugs for powerpc and s390 are fixed, the headers for those platforms
should of course be updated similarly.

Tested for x86_64 and x86, including with GCC mainline.  Note that
other failures appear with GCC mainline because of spurious use of
ordered comparison instructions for unordered operations
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82692>.

	* sysdeps/x86/fpu/fix-fp-int-compare-invalid.h
	(FIX_COMPARE_INVALID): Define to 0 if [__GNUC_PREREQ (8, 0)].
2017-10-24 00:33:08 +00:00
Joseph Myers
797ba44ba2 Add bits/floatn.h defines for more _FloatN / _FloatNx types.
The bits/floatn.h header currently only has defines relating to
_Float128.  This patch adds defines relating to other _FloatN /
_FloatNx types.

The approach taken is to add defines for all _FloatN / _FloatNx types
known to GCC, and to put them in a common bits/floatn-common.h header
included at the end of all the individual bits/floatn.h headers.  If
in future some defines become different for different glibc
configurations, they will move out into the separate bits/floatn.h
headers.

Some defines are expected always to be the same across glibc ports.
Corresponding defines are nevertheless put in this header.  The intent
is that where there are conditionals (in headers or in non-installed
files) that can just repeat the same or nearly the same logic for each
floating-point type, they should do so, even if in fact the cases for
some types could be unconditionally present or absent because the same
conditionals are true or false for all glibc configurations.  This
should make the glibc code with such conditionals easier to read,
because the reader can just see that the same conditionals are
repeated for each type, rather than seeing different conditionals for
different types and needing to reason, at each location with such
differences, why those differences are indeed correct there.  (Cases
involving per-format rather than per-type logic are more likely still
to need differences in how they handle different types.)

Having such defines and conditionals also helps in incremental
preparation for adding _Float32 / _Float64 / _Float32x / _Float64x
function aliases.  I intend subsequent patches to add such
conditionals corresponding to those already present for _Float128, as
well as making more architecture-specific function implementations use
common macros to define aliases in preparation for adding such _FloatN
/ _FloatNx aliases.

Tested for x86_64.

	* bits/floatn-common.h: New file.
	* math/Makefile (headers): Add bits/floatn-common.h.
	* bits/floatn.h: Include <bits/floatn-common.h>.
	* sysdeps/ia64/bits/floatn.h: Likewise.
	* sysdeps/ieee754/ldbl-128/bits/floatn.h: Likewise.
	* sysdeps/mips/ieee754/bits/floatn.h: Likewise.
	* sysdeps/powerpc/bits/floatn.h: Likewise.
	* sysdeps/x86/bits/floatn.h: Likewise.
2017-10-20 21:42:51 +00:00
H.J. Lu
b52b0d793d x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers.  It simplifies _dl_runtime_resolve and supports
different calling conventions.  ld.so code size is reduced by more than
1 KB.  However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.

Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:

                             Before    After     Change

Westmere (SSE)/fxsave         345      866       151%
IvyBridge (AVX)/xsave         420      643       53%
Haswell (AVX)/xsave           713      1252      75%
Skylake (AVX+MPX)/xsavec      559      719       28%
Skylake (AVX512+MPX)/xsavec   145      272       87%
Ryzen (AVX)/xsavec            280      553       97%

This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases.  With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.

On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises.  On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.

	[BZ #21265]
	* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
	New.
	* sysdeps/x86/cpu-features.c: Include <libc-pointer-arith.h>.
	(get_common_indeces): Set xsave_state_size, xsave_state_full_size
	and bit_arch_XSAVEC_Usable if needed.
	(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
	and bit_arch_Use_dl_runtime_resolve_opt.
	* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
	Removed.
	(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
	(bit_arch_Prefer_No_AVX512): Updated.
	(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
	(bit_arch_XSAVEC_Usable): New.
	(STATE_SAVE_OFFSET): Likewise.
	(STATE_SAVE_MASK): Likewise.
	[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
	(cpu_features): Add xsave_state_size and xsave_state_full_size.
	(index_arch_Use_dl_runtime_resolve_opt): Removed.
	(index_arch_Use_dl_runtime_resolve_slow): Likewise.
	(index_arch_XSAVEC_Usable): New.
	* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
	Support XSAVEC_Usable.  Remove Use_dl_runtime_resolve_slow.
	* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
	is enabled.
	* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
	Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
	_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
	_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
	with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
	_dl_runtime_resolve_xsavec.
	* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
	Removed.
	(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
	instead of VEC_SIZE.
	(REGISTER_SAVE_BND0): Removed.
	(REGISTER_SAVE_BND1): Likewise.
	(REGISTER_SAVE_BND3): Likewise.
	(REGISTER_SAVE_RAX): Always defined to 0.
	(VMOV): Removed.
	(_dl_runtime_resolve_avx): Likewise.
	(_dl_runtime_resolve_avx_slow): Likewise.
	(_dl_runtime_resolve_avx_opt): Likewise.
	(_dl_runtime_resolve_avx512): Likewise.
	(_dl_runtime_resolve_avx512_opt): Likewise.
	(_dl_runtime_resolve_sse): Likewise.
	(_dl_runtime_resolve_sse_vex): Likewise.
	(USE_FXSAVE): New.
	(_dl_runtime_resolve_fxsave): Likewise.
	(USE_XSAVE): Likewise.
	(_dl_runtime_resolve_xsave): Likewise.
	(USE_XSAVEC): Likewise.
	(_dl_runtime_resolve_xsavec): Likewise.
	* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
	Removed.
	(_dl_runtime_resolve_avx512_opt): Likewise.
	(_dl_runtime_resolve_avx): Likewise.
	(_dl_runtime_resolve_avx_opt): Likewise.
	(_dl_runtime_resolve_sse): Likewise.
	(_dl_runtime_resolve_sse_vex): Likewise.
	(_dl_runtime_resolve_fxsave): New.
	(_dl_runtime_resolve_xsave): Likewise.
	(_dl_runtime_resolve_xsavec): Likewise.
2017-10-20 11:00:34 -07:00
H.J. Lu
4d916f0f12 x86-64: Don't set GLRO(dl_platform) to NULL [BZ #22299]
Since ld.so expands $PLATFORM with GLRO(dl_platform), don't set
GLRO(dl_platform) to NULL.

	[BZ #22299]
	* sysdeps/x86/cpu-features.c (init_cpu_features): Don't set
	GLRO(dl_platform) to NULL.
	* sysdeps/x86_64/Makefile (tests): Add tst-platform-1.
	(modules-names): Add tst-platformmod-1 and
	x86_64/tst-platformmod-2.
	(CFLAGS-tst-platform-1.c): New.
	(CFLAGS-tst-platformmod-1.c): Likewise.
	(CFLAGS-tst-platformmod-2.c): Likewise.
	(LDFLAGS-tst-platformmod-2.so): Likewise.
	($(objpfx)tst-platform-1): Likewise.
	($(objpfx)tst-platform-1.out): Likewise.
	(tst-platform-1-ENV): Likewise.
	($(objpfx)x86_64/tst-platformmod-2.os): Likewise.
	* sysdeps/x86_64/tst-platform-1.c: New file.
	* sysdeps/x86_64/tst-platformmod-1.c: Likewise.
	* sysdeps/x86_64/tst-platformmod-2.c: Likewise.
2017-10-19 08:28:26 -07:00
Adhemerval Zanella
4e17c78e4a Add common ifunc-init.h header
This patch moves the generic definition from x86_64 init-arch
to a common header ifunc-init.h.  No functional changes is expected.

Checked on a x86_64-linux-gnu build.

	* sysdeps/generic/ifunc-init.h: New file.
	* sysdeps/x86/init-arch.h: Use generic ifunc-init.h.
2017-10-17 12:01:22 -02:00
Wilco Dijkstra
4d3693ec1c Remove ancient __signbit inlines
Remove __signbit inlines from mathinline.h.  Math.h already uses
the builtin when supported, so additional inlines are only used
on pre 4.0 GCCs.  Similarly remove ancient copysign and fabs
inlines.

	* sysdeps/alpha/fpu/bits/mathinline.h: Delete file.
	* sysdeps/ia64/fpu/bits/mathinline.h: Delete file.
	* sysdeps/m68k/coldfire/fpu/bits/mathinline.h: Delete file.
	* sysdeps/m68k/m680x0/fpu/bits/mathinline.h: (__signbitf): Remove.
	(__signbit): Remove.
	(__signbitl): Remove.
	* sysdeps/powerpc/bits/mathinline.h (__signbitf): Remove.
	(__signbit): Remove.
	(__signbitl): Remove.
	* sysdeps/s390/fpu/bits/mathinline.h: (__signbitf): Remove.
	(__signbit): Remove.
	(__signbitl): Remove
	* sysdeps/sparc/fpu/bits/mathinline.h (__signbitf): Remove.
	(__signbit): Remove.
	(__signbitl): Remove.
	* sysdeps/tile/bits/mathinline.h: Delete file.
	* sysdeps/x86/fpu/bits/mathinline.h (__signbitf): Remove.
	(__signbit): Remove.
	(__signbitl): Remove.
2017-09-28 19:52:13 +01:00
Wilco Dijkstra
1e6d07234f Simplify C99 isgreater macros
Simplify the C99 isgreater macros.  Although some support was added
in GCC 2.97, not all targets added support until GCC 3.1.  Therefore
only use the builtins in math.h from GCC 3.1 onwards, and defer to
generic macros otherwise.  Improve the generic isunordered macro
to use compares rather than call fpclassify twice - this is not only
faster but also correct for signaling NaNs.

	* math/math.h: Improve handling of C99 isgreater macros.
	* sysdeps/alpha/fpu/bits/mathinline.h: Remove isgreater macros.
	* sysdeps/m68k/m680x0/fpu/bits/mathinline.h: Likewise.
	* sysdeps/powerpc/bits/mathinline.h: Likewise.
	* sysdeps/sparc/fpu/bits/mathinline.h: Likewise.
	* sysdeps/x86/fpu/bits/mathinline.h: Likewise.
2017-09-28 19:43:54 +01:00
H.J. Lu
ef8adeb041 x86: Add MathVec_Prefer_No_AVX512 to cpu-features [BZ #21967]
AVX512 functions in mathvec are used on machines with AVX512.  An AVX2
wrapper is also provided and it can be used when the AVX512 version
isn't profitable.  MathVec_Prefer_No_AVX512 is addded to cpu-features.
If glibc.tune.hwcaps=MathVec_Prefer_No_AVX512 is set in GLIBC_TUNABLES
environment variable, the AVX2 wrapper will be used.

Tested on x86-64 machines with and without AVX512.  Also verified
glibc.tune.hwcaps=MathVec_Prefer_No_AVX512 on AVX512 machine.

	[BZ #21967]
	* sysdeps/x86/cpu-features.h (bit_arch_MathVec_Prefer_No_AVX512):
	New.
	(index_arch_MathVec_Prefer_No_AVX512): Likewise.
	* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
	Handle MathVec_Prefer_No_AVX512.
	* sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h
	(IFUNC_SELECTOR): Return AVX2 version if MathVec_Prefer_No_AVX512
	is set.
2017-09-12 07:54:47 -07:00
H.J. Lu
45ff34638f x86: Add x86_64 to x86-64 HWCAP [BZ #22093]
Before glibc 2.26, ld.so set dl_platform to "x86_64" and searched the
"x86_64" subdirectory when loading a shared library.  ld.so in glibc
2.26 was changed to set dl_platform to "haswell" or "xeon_phi", based
on supported ISAs.  This led to shared library loading failure for
shared libraries placed under the "x86_64" subdirectory.

This patch adds "x86_64" to x86-64 dl_hwcap so that ld.so will always
search the "x86_64" subdirectory when loading a shared library.

NB: We can't set x86-64 dl_platform to "x86-64" since ld.so will skip
the "haswell" and "xeon_phi" subdirectories on "haswell" and "xeon_phi"
machines.

Tested on i686 and x86-64.

	[BZ #22093]
	* sysdeps/x86/cpu-features.c (init_cpu_features): Initialize
	GLRO(dl_hwcap) to HWCAP_X86_64 for x86-64.
	* sysdeps/x86/dl-hwcap.h (HWCAP_COUNT): Updated.
	(HWCAP_IMPORTANT): Likewise.
	(HWCAP_X86_64): New enum.
	(HWCAP_X86_AVX512_1): Updated.
	* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): Add "x86_64".
	* sysdeps/x86_64/Makefile (tests): Add tst-x86_64-1.
	(modules-names): Add x86_64/tst-x86_64mod-1.
	(LDFLAGS-tst-x86_64mod-1.so): New.
	($(objpfx)tst-x86_64-1): Likewise.
	($(objpfx)x86_64/tst-x86_64mod-1.os): Likewise.
	(tst-x86_64-1-clean): Likewise.
	* sysdeps/x86_64/tst-x86_64-1.c: New file.
	* sysdeps/x86_64/tst-x86_64mod-1.c: Likewise.
2017-09-11 08:18:32 -07:00
Samuel Thibault
1946d950f2 hurd: fix libm link
* sysdeps/x86/fpu/include/bits/fenv.h [NO_HIDDEN]: Redirect
	__feraiseexcept_renamed to feraiseexcept instead of
	__GI_feraiseexcept.
2017-09-03 05:32:10 +02:00
Joseph Myers
a60eca2e55 Simplify HUGE_VAL definitions.
There are various bits/huge_val*.h headers to define HUGE_VAL and
related macros.  All of them use __builtin_huge_val etc. for GCC 3.3
and later.  Then there are various fallbacks, such as using a large
hex float constant for GCC 2.96 and later, or using unions (with or
without compound literals) to construct the bytes of an infinity, with
this last being the reason for having architecture-specific files.
Supporting TS 18661-3 _FloatN / _FloatNx types that have the same
format as other supported types will mean adding more such macros;
needing to add more headers for them doesn't seem very desirable.

The fallbacks based on bytes of the representation of an infinity do
not meet the standard requirements for a constant expression.  At
least one of them is also wrong: sysdeps/sh/bits/huge_val.h is
producing a mixed-endian representation which does not match what GCC
does.

This patch eliminates all those headers, defining the macros directly
in math.h.  For GCC 3.3 and later, the built-in functions are used as
now.  For other compilers, a large constant 1e10000 (with appropriate
suffix) is used.  This is like the fallback for GCC 2.96 and later,
but without using hex floats (which have no apparent advantage here).
It is unambiguously valid standard C for all floating-point formats
with infinities, which covers all formats supported by glibc or likely
to be supported by glibc in future (C90 DR#025 said that if a
floating-point format represents infinities, all real values lie
within the range of representable values, so the constraints for
constant expressions are not violated), but may generate compiler
warnings and wouldn't handle the TS 18661-1 FENV_ROUND pragma
correctly.  If someone is actually using a compiler with glibc that
does not claim to be GCC 3.3 or later, but which has a better way to
define the HUGE_VAL macros, we can always add compiler conditionals in
with alternative definitions.

I intend to make similar changes for INF and NAN.  The SNAN macros
already just use __builtin_nans etc. with no fallback for compilers
not claiming to be GCC 3.3 or later.

Tested for x86_64.

	* math/math.h: Do not include bits/huge_val.h, bits/huge_valf.h,
	bits/huge_vall.h or bits/huge_val_flt128.h.
	(HUGE_VAL): Define directly here.
	[__USE_ISOC99] (HUGE_VALF): Likewise.
	[__USE_ISOC99] (HUGE_VALL): Likewise.
	[__HAVE_FLOAT128 && __GLIBC_USE (IEC_60559_TYPES_EXT)]
	(HUGE_VAL_F128): Likewise.
	* math/Makefile (headers): Remove bits/huge_val.h,
	bits/huge_valf.h, bits/huge_vall.h and bits/huge_val_flt128.h.
	* bits/huge_val.h: Remove.
	* bits/huge_val_flt128.h: Likewise.
	* bits/huge_valf.h: Likewise.
	* bits/huge_vall.h: Likewise.
	* sysdeps/ia64/bits/huge_vall.h: Likewise.
	* sysdeps/ieee754/bits/huge_val.h: Likewise.
	* sysdeps/ieee754/bits/huge_valf.h: Likewise.
	* sysdeps/m68k/m680x0/bits/huge_vall.h: Likewise.
	* sysdeps/sh/bits/huge_val.h: Likewise.
	* sysdeps/sparc/bits/huge_vall.h: Likewise.
	* sysdeps/x86/bits/huge_vall.h: Likewise.
2017-08-31 15:50:50 +00:00
H.J. Lu
7ab70c98e8 x86: Remove assembly versions of index_cpu_*/index_arch_*
Since assembly versions of HAS_CPU_FEATURE and HAS_ARCH_FEATURE have
been removed,  assembly versions of index_cpu_* and index_arch_* can
also be removed.

Tested on i686 and x86-64 with and without --disable-multi-arch.

	* sysdeps/x86/cpu-features.h [__ASSEMBLER__]
	(index_cpu_*, index_arch_*): Removed.
2017-08-25 10:51:04 -07:00
H.J. Lu
73322d5ff6 x86: Add IBT/SHSTK bits to cpu-features
Add IBT/SHSTK bits to cpu-features for Shadow Stack in Intel Control-flow
Enforcement Technology (CET) instructions:

https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf

	* sysdeps/x86/cpu-features.h (bit_cpu_BIT): New.
	(bit_cpu_SHSTK): Likewise.
	(index_cpu_IBT): Likewise.
	(index_cpu_SHSTK): Likewise.
	(reg_IBT): Likewise.
	(reg_SHSTK): Likewise.
	* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
	Handle index_cpu_IBT and index_cpu_SHSTK.
2017-08-14 05:54:38 -07:00
H.J. Lu
64d4dea6cd x86: Remove assembly versions of HAS_CPU_FEATURE/HAS_ARCH_FEATURE
Since all x86 IFUNC selectors are implemented in C, assembly versions of
HAS_CPU_FEATURE and HAS_ARCH_FEATURE can be removed.

	* sysdeps/x86/cpu-features.h [__ASSEMBLER__]
	(LOAD_RTLD_GLOBAL_RO_RDX, HAS_FEATURE, LOAD_FUNC_GOT_EAX,
	HAS_CPU_FEATURE, HAS_ARCH_FEATURE): Removed.
2017-08-04 13:38:20 -07:00
H.J. Lu
d2cf37c0a2 x86-64: Use _dl_runtime_resolve_opt only with AVX512F [BZ #21871]
On AVX machines with XGETBV (ECX == 1) like Skylake processors,

(gdb) disass _dl_runtime_resolve_avx_opt
Dump of assembler code for function _dl_runtime_resolve_avx_opt:
   0x0000000000015890 <+0>:	push   %rax
   0x0000000000015891 <+1>:	push   %rcx
   0x0000000000015892 <+2>:	push   %rdx
   0x0000000000015893 <+3>:	mov    $0x1,%ecx
   0x0000000000015898 <+8>:	xgetbv
   0x000000000001589b <+11>:	mov    %eax,%r11d
   0x000000000001589e <+14>:	pop    %rdx
   0x000000000001589f <+15>:	pop    %rcx
   0x00000000000158a0 <+16>:	pop    %rax
   0x00000000000158a1 <+17>:	and    $0x4,%r11d
   0x00000000000158a5 <+21>:	bnd je 0x16200 <_dl_runtime_resolve_sse_vex>
End of assembler dump.

is slower than:

(gdb) disass _dl_runtime_resolve_avx_slow
Dump of assembler code for function _dl_runtime_resolve_avx_slow:
   0x0000000000015850 <+0>:	vorpd  %ymm0,%ymm1,%ymm8
   0x0000000000015854 <+4>:	vorpd  %ymm2,%ymm3,%ymm9
   0x0000000000015858 <+8>:	vorpd  %ymm4,%ymm5,%ymm10
   0x000000000001585c <+12>:	vorpd  %ymm6,%ymm7,%ymm11
   0x0000000000015860 <+16>:	vorpd  %ymm8,%ymm9,%ymm9
   0x0000000000015865 <+21>:	vorpd  %ymm10,%ymm11,%ymm10
   0x000000000001586a <+26>:	vpcmpeqd %xmm8,%xmm8,%xmm8
   0x000000000001586f <+31>:	vorpd  %ymm9,%ymm10,%ymm10
   0x0000000000015874 <+36>:	vptest %ymm10,%ymm8
   0x0000000000015879 <+41>:	bnd jae 0x158b0 <_dl_runtime_resolve_avx>
   0x000000000001587c <+44>:	vzeroupper
   0x000000000001587f <+47>:	bnd jmpq 0x16200 <_dl_runtime_resolve_sse_vex>
End of assembler dump.
(gdb)

since xgetbv takes much more cycles than single cycle operations like
vpord/vvpcmpeq/ptest.  _dl_runtime_resolve_opt should be used only with
AVX512 where AVX512 instructions lead to lower CPU frequency on Skylake
server.

	[BZ #21871]
	* sysdeps/x86/cpu-features.c (init_cpu_features): Set
	bit_arch_Use_dl_runtime_resolve_opt only with AVX512F.
2017-08-04 11:14:33 -07:00
Joseph Myers
24ab7723b8 Consistently use uintN_t not u_intN_t in libm.
This patch changes libm code to make consistent use of C99 uintN_t
types instead of sometimes using those and sometimes using the older
nonstandard u_intN_t names.  This makes sense as a cleanup in its own
right, and also facilitates merges to GCC's libquadmath (which gets
the types from stdint.h and so may not have u_intN_t available at
all).

Tested for x86_64, and with build-many-glibcs.py.

	* math/s_nextafter.c (__nextafter): Use uintN_t instead of
	u_intN_t.
	* math/s_nexttowardf.c (__nexttowardf): Likewise.
	* sysdeps/generic/math_private.h (ieee_double_shape_type):
	Likewise.
	(ieee_float_shape_type): Likewise.
	* sysdeps/i386/fpu/s_fpclassifyl.c (__fpclassifyl): Likewise.
	* sysdeps/i386/fpu/s_isnanl.c (__isnanl): Likewise.
	* sysdeps/i386/fpu/s_nextafterl.c (__nextafterl): Likewise.
	* sysdeps/i386/fpu/s_nexttoward.c (__nexttoward): Likewise.
	* sysdeps/i386/fpu/s_nexttowardf.c (__nexttowardf): Likewise.
	* sysdeps/ieee754/dbl-64/e_acosh.c (__ieee754_acosh): Likewise.
	* sysdeps/ieee754/dbl-64/e_cosh.c (__ieee754_cosh): Likewise.
	* sysdeps/ieee754/dbl-64/e_fmod.c (__ieee754_fmod): Likewise.
	* sysdeps/ieee754/dbl-64/e_gamma_r.c (__ieee754_gamma_r):
	Likewise.
	* sysdeps/ieee754/dbl-64/e_hypot.c (__ieee754_hypot): Likewise.
	* sysdeps/ieee754/dbl-64/e_jn.c (__ieee754_jn): Likewise.
	(__ieee754_yn): Likewise.
	* sysdeps/ieee754/dbl-64/e_log10.c (__ieee754_log10): Likewise.
	* sysdeps/ieee754/dbl-64/e_log2.c (__ieee754_log2): Likewise.
	* sysdeps/ieee754/dbl-64/e_rem_pio2.c (__ieee754_rem_pio2):
	Likewise.
	* sysdeps/ieee754/dbl-64/e_sinh.c (__ieee754_sinh): Likewise.
	* sysdeps/ieee754/dbl-64/s_ceil.c (__ceil): Likewise.
	* sysdeps/ieee754/dbl-64/s_copysign.c (__copysign): Likewise.
	* sysdeps/ieee754/dbl-64/s_erf.c (__erf): Likewise.
	(__erfc): Likewise.
	* sysdeps/ieee754/dbl-64/s_expm1.c (__expm1): Likewise.
	* sysdeps/ieee754/dbl-64/s_finite.c (FINITE): Likewise.
	* sysdeps/ieee754/dbl-64/s_floor.c (__floor): Likewise.
	* sysdeps/ieee754/dbl-64/s_fpclassify.c (__fpclassify): Likewise.
	* sysdeps/ieee754/dbl-64/s_isnan.c (__isnan): Likewise.
	* sysdeps/ieee754/dbl-64/s_issignaling.c (__issignaling):
	Likewise.
	* sysdeps/ieee754/dbl-64/s_llrint.c (__llrint): Likewise.
	* sysdeps/ieee754/dbl-64/s_llround.c (__llround): Likewise.
	* sysdeps/ieee754/dbl-64/s_lrint.c (__lrint): Likewise.
	* sysdeps/ieee754/dbl-64/s_lround.c (__lround): Likewise.
	* sysdeps/ieee754/dbl-64/s_modf.c (__modf): Likewise.
	* sysdeps/ieee754/dbl-64/s_nextup.c (__nextup): Likewise.
	* sysdeps/ieee754/dbl-64/s_remquo.c (__remquo): Likewise.
	* sysdeps/ieee754/dbl-64/s_round.c (__round): Likewise.
	* sysdeps/ieee754/dbl-64/s_trunc.c (__trunc): Likewise.
	* sysdeps/ieee754/dbl-64/wordsize-64/s_issignaling.c
	(__issignaling): Likewise.
	* sysdeps/ieee754/flt-32/e_atan2f.c (__ieee754_atan2f): Likewise.
	* sysdeps/ieee754/flt-32/e_fmodf.c (__ieee754_fmodf): Likewise.
	* sysdeps/ieee754/flt-32/e_gammaf_r.c (__ieee754_gammaf_r):
	Likewise.
	* sysdeps/ieee754/flt-32/e_jnf.c (__ieee754_ynf): Likewise.
	* sysdeps/ieee754/flt-32/e_log10f.c (__ieee754_log10f): Likewise.
	* sysdeps/ieee754/flt-32/e_powf.c (__ieee754_powf): Likewise.
	* sysdeps/ieee754/flt-32/e_rem_pio2f.c (__ieee754_rem_pio2f):
	Likewise.
	* sysdeps/ieee754/flt-32/e_remainderf.c (__ieee754_remainderf):
	Likewise.
	* sysdeps/ieee754/flt-32/e_sqrtf.c (__ieee754_sqrtf): Likewise.
	* sysdeps/ieee754/flt-32/s_ceilf.c (__ceilf): Likewise.
	* sysdeps/ieee754/flt-32/s_copysignf.c (__copysignf): Likewise.
	* sysdeps/ieee754/flt-32/s_erff.c (__erff): Likewise.
	(__erfcf): Likewise.
	* sysdeps/ieee754/flt-32/s_expm1f.c (__expm1f): Likewise.
	* sysdeps/ieee754/flt-32/s_finitef.c (FINITEF): Likewise.
	* sysdeps/ieee754/flt-32/s_floorf.c (__floorf): Likewise.
	* sysdeps/ieee754/flt-32/s_fpclassifyf.c (__fpclassifyf):
	Likewise.
	* sysdeps/ieee754/flt-32/s_isnanf.c (__isnanf): Likewise.
	* sysdeps/ieee754/flt-32/s_issignalingf.c (__issignalingf):
	Likewise.
	* sysdeps/ieee754/flt-32/s_llrintf.c (__llrintf): Likewise.
	* sysdeps/ieee754/flt-32/s_llroundf.c (__llroundf): Likewise.
	* sysdeps/ieee754/flt-32/s_lrintf.c (__lrintf): Likewise.
	* sysdeps/ieee754/flt-32/s_lroundf.c (__lroundf): Likewise.
	* sysdeps/ieee754/flt-32/s_modff.c (__modff): Likewise.
	* sysdeps/ieee754/flt-32/s_remquof.c (__remquof): Likewise.
	* sysdeps/ieee754/flt-32/s_roundf.c (__roundf): Likewise.
	* sysdeps/ieee754/ldbl-128/e_acoshl.c (__ieee754_acoshl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_atan2l.c (__ieee754_atan2l):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_atanhl.c (__ieee754_atanhl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_fmodl.c (__ieee754_fmodl): Likewise.
	* sysdeps/ieee754/ldbl-128/e_gammal_r.c (__ieee754_gammal_r):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_hypotl.c (__ieee754_hypotl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_jnl.c (__ieee754_jnl): Likewise.
	(__ieee754_ynl): Likewise.
	* sysdeps/ieee754/ldbl-128/e_powl.c (__ieee754_powl): Likewise.
	* sysdeps/ieee754/ldbl-128/e_rem_pio2l.c (__ieee754_rem_pio2l):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_remainderl.c (__ieee754_remainderl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/e_sinhl.c (__ieee754_sinhl): Likewise.
	* sysdeps/ieee754/ldbl-128/k_cosl.c (__kernel_cosl): Likewise.
	* sysdeps/ieee754/ldbl-128/k_sincosl.c (__kernel_sincosl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/k_sinl.c (__kernel_sinl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_ceill.c (__ceill): Likewise.
	* sysdeps/ieee754/ldbl-128/s_copysignl.c (__copysignl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_erfl.c (__erfcl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_fabsl.c (__fabsl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_finitel.c (__finitel): Likewise.
	* sysdeps/ieee754/ldbl-128/s_floorl.c (__floorl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_fpclassifyl.c (__fpclassifyl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_frexpl.c (__frexpl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_isnanl.c (__isnanl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_issignalingl.c (__issignalingl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_llrintl.c (__llrintl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_llroundl.c (__llroundl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_lrintl.c (__lrintl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_lroundl.c (__lroundl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_modfl.c (__modfl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_nearbyintl.c (__nearbyintl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_nextafterl.c (__nextafterl):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_nexttoward.c (__nexttoward):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_nexttowardf.c (__nexttowardf):
	Likewise.
	* sysdeps/ieee754/ldbl-128/s_nextupl.c (__nextupl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_remquol.c (__remquol): Likewise.
	* sysdeps/ieee754/ldbl-128/s_rintl.c (__rintl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_roundl.c (__roundl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_tanhl.c (__tanhl): Likewise.
	* sysdeps/ieee754/ldbl-128/s_truncl.c (__truncl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/e_fmodl.c (__ieee754_fmodl):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/e_gammal_r.c (__ieee754_gammal_r):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/e_powl.c (__ieee754_powl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/e_rem_pio2l.c (__ieee754_rem_pio2l):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/e_remainderl.c
	(__ieee754_remainderl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/k_cosl.c (__kernel_cosl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/k_sinl.c (__kernel_sinl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_fabsl.c (__fabsl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_fpclassifyl.c (___fpclassifyl):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_modfl.c (__modfl): Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_nexttowardf.c (__nexttowardf):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_remquol.c (__remquol): Likewise.
	* sysdeps/ieee754/ldbl-96/e_acoshl.c (__ieee754_acoshl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_asinl.c (__ieee754_asinl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_atanhl.c (__ieee754_atanhl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_coshl.c (__ieee754_coshl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_gammal_r.c (__ieee754_gammal_r):
	Likewise.
	* sysdeps/ieee754/ldbl-96/e_hypotl.c (__ieee754_hypotl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_j0l.c (__ieee754_j0l): Likewise.
	(__ieee754_y0l): Likewise.
	(pzero): Likewise.
	(qzero): Likewise.
	* sysdeps/ieee754/ldbl-96/e_j1l.c (__ieee754_j1l): Likewise.
	(__ieee754_y1l): Likewise.
	(pone): Likewise.
	(qone): Likewise.
	* sysdeps/ieee754/ldbl-96/e_jnl.c (__ieee754_jnl): Likewise.
	(__ieee754_ynl): Likewise.
	* sysdeps/ieee754/ldbl-96/e_lgammal_r.c (sin_pi): Likewise.
	(__ieee754_lgammal_r): Likewise.
	* sysdeps/ieee754/ldbl-96/e_rem_pio2l.c (__ieee754_rem_pio2l):
	Likewise.
	* sysdeps/ieee754/ldbl-96/e_sinhl.c (__ieee754_sinhl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_copysignl.c (__copysignl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_erfl.c (__erfl): Likewise.
	(__erfcl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_frexpl.c (__frexpl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_issignalingl.c (__issignalingl):
	Likewise.
	* sysdeps/ieee754/ldbl-96/s_llrintl.c (__llrintl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_llroundl.c (__llroundl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_lrintl.c (__lrintl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_lroundl.c (__lroundl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_modfl.c (__modfl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_nexttoward.c (__nexttoward): Likewise.
	* sysdeps/ieee754/ldbl-96/s_nexttowardf.c (__nexttowardf):
	Likewise.
	* sysdeps/ieee754/ldbl-96/s_nextupl.c (__nextupl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_remquol.c (__remquol): Likewise.
	* sysdeps/ieee754/ldbl-96/s_roundl.c (__roundl): Likewise.
	* sysdeps/ieee754/ldbl-96/s_tanhl.c (__tanhl): Likewise.
	* sysdeps/ieee754/ldbl-opt/s_nexttowardfd.c (__nldbl_nexttowardf):
	Likewise.
	* sysdeps/m68k/m680x0/fpu/e_pow.c (s(__ieee754_pow)): Likewise.
	* sysdeps/m68k/m680x0/fpu/s_fpclassifyl.c (__fpclassifyl):
	Likewise.
	* sysdeps/m68k/m680x0/fpu/s_llrint.c (__llrint): Likewise.
	* sysdeps/m68k/m680x0/fpu/s_llrintf.c (__llrintf): Likewise.
	* sysdeps/m68k/m680x0/fpu/s_llrintl.c (__llrintl): Likewise.
	* sysdeps/m68k/m680x0/fpu/s_nextafterl.c (__nextafterl): Likewise.
	* sysdeps/x86/fpu/powl_helper.c (__powl_helper): Likewise.
2017-08-03 19:55:04 +00:00
Gabriel F. T. Gomes
8466ee1cb7 float128: Add signbit alternative for old compilers
In math/math.h, __MATH_TG will expand signbit to __builtin_signbit*,
e.g.: __builtin_signbitf128, before GCC 6.  However, there has never
been a __builtin_signbitf128 in GCC and the type-generic builtin is
only available since GCC 6.  For older GCC, this patch defines
__builtin_signbitf128 to __signbitf128, so that the internal function
is used instead of the non-existent builtin.

This patch also changes the implementation of __signbitf128, because
it was reusing the implementation of __signbitl from ldbl-128, which
calls __builtin_signbitl.  Using the long double version of the
builtin is not correct on machines where _Float128 is ABI-distinct
from long double (i.e.: ia64, powerpc64le, x86, x86_84).  The new
implementation does not rely on builtins when being built with GCC
versions older than 6.0.

The new code does not currently affect powerpc64le builds, because
only GCC 6.2 fulfills the requirements from configure.  It might
affect powerpc64le builds if those requirements are backported to
older versions of the compiler.  The new code affects x86_64 builds,
since glibc is supposed to build correctly with older versions of GCC.

Tested for powerpc64le and x86_64.

	* include/math.h (__signbitf128): Define as hidden.
	* sysdeps/ieee754/float128/s_signbitf128.c (__signbitf128):
	Reimplement without builtins.
	* sysdeps/ia64/bits/floatn.h [!__GNUC_PREREQ (6, 0)]
	(__builtin_signbitf128): Define to __signbitf128.
	* sysdeps/powerpc/bits/floatn.h: Likewise.
	* sysdeps/x86/bits/floatn.h: Likewise.
2017-06-30 18:34:29 -03:00
Joseph Myers
c86ed71d63 Add float128 support for x86_64, x86.
This patch enables float128 support for x86_64 and x86.  All GCC
versions that can build glibc provide the required support, but since
GCC 6 and before don't provide __builtin_nanq / __builtin_nansq, sNaN
tests and some tests of NaN payloads need to be disabled with such
compilers (this does not affect the generated glibc binaries at all,
just the tests).  bits/floatn.h declares float128 support to be
available for GCC versions that provide the required libgcc support
(4.3 for x86_64, 4.4 for i386 GNU/Linux, 4.5 for i386 GNU/Hurd);
compilation-only support was present some time before then, but not
really useful without the libgcc functions.

fenv_private.h needed updating to avoid trying to put _Float128 values
in registers.  I make no assertion of optimality of the
math_opt_barrier / math_force_eval definitions for this case; they are
simply intended to be sufficient to work correctly.

Tested for x86_64 and x86, with GCC 7 and GCC 6.  (Testing for x32 was
compilation tests only with build-many-glibcs.py to verify the ABI
baseline updates.  I have not done any testing for Hurd, although the
float128 support is enabled there as for GNU/Linux.)

	* sysdeps/i386/Implies: Add ieee754/float128.
	* sysdeps/x86_64/Implies: Likewise.
	* sysdeps/x86/bits/floatn.h: New file.
	* sysdeps/x86/float128-abi.h: Likewise.
	* manual/math.texi (Mathematics): Document support for _Float128
	on x86_64 and x86.
	* sysdeps/i386/fpu/fenv_private.h: Include <bits/floatn.h>.
	(math_opt_barrier): Do not put _Float128 values in floating-point
	registers.
	(math_force_eval): Likewise.
	[__x86_64__] (SET_RESTORE_ROUNDF128): New macro.
	* sysdeps/x86/fpu/Makefile [$(subdir) = math] (CPPFLAGS): Append
	to Makefile variable.
	* sysdeps/x86/fpu/e_sqrtf128.c: New file.
	* sysdeps/x86/fpu/sfp-machine.h: Likewise.  Based on libgcc.
	* sysdeps/x86/math-tests.h: New file.
	* math/libm-test-support.h (XFAIL_FLOAT128_PAYLOAD): New macro.
	* math/libm-test-getpayload.inc (getpayload_test_data): Use
	XFAIL_FLOAT128_PAYLOAD.
	* math/libm-test-setpayload.inc (setpayload_test_data): Likewise.
	* math/libm-test-totalorder.inc (totalorder_test_data): Likewise.
	* math/libm-test-totalordermag.inc (totalordermag_test_data):
	Likewise.
	* sysdeps/unix/sysv/linux/i386/libc.abilist: Update.
	* sysdeps/unix/sysv/linux/i386/libm.abilist: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/64/libc.abilist: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/64/libm.abilist: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/x32/libm.abilist: Likewise.
	* sysdeps/i386/fpu/libm-test-ulps: Likewise.
	* sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise.
	* sysdeps/x86_64/fpu/libm-test-ulps: Likewise.
2017-06-26 22:02:24 +00:00
Joseph Myers
16000c8d04 Avoid localplt issues from x86 fereaiseexcept inline.
Building for x86_64 with float128 support, I get a localplt test
failure from lrintf128 calling feraiseexcept.

The problem is that an inline optimized version of feraiseexcept calls
__feraiseexcept_renamed in cases where it doesn't completely expand
inline, and that in turn is redirected to feraiseexcept for a library
call, so meaning the redirection of feraiseexcept to
__GI_feraiseexcept inside libm is lost for that call.

This patch fixes the problem by moving the redirect to an internal
header in the _LIBC case, with the internal header using
__GI_feraiseexcept where appropriate.

Tested for x86_64 (in conjunction with float128 patches).

	* sysdeps/x86/fpu/bits/fenv.h [_LIBC] (__feraiseexcept_renamed):
	Do not declare.
	* sysdeps/x86/fpu/include/bits/fenv.h [_LIBC &&
	__USE_EXTERN_INLINES] (__feraiseexcept_renamed): Declare here,
	redirected to __GI_feraiseexcept if [SHARED && IS_IN (libm)].
2017-06-23 20:04:23 +00:00
H.J. Lu
03feacb562 x86: Rename glibc.tune.ifunc to glibc.tune.hwcaps
Rename glibc.tune.ifunc to glibc.tune.hwcaps and move it to
sysdeps/x86/dl-tunables.list since it is x86 specicifc.  Also
change type of data_cache_size, data_cache_size and
non_temporal_threshold to unsigned long int to match size_t.
Remove usage DEFAULT_STRLEN from cpu-tunables.c.

	* elf/dl-tunables.list (glibc.tune.ifunc): Removed.
	* sysdeps/x86/dl-tunables.list (glibc.tune.hwcaps): New.
	Remove security_level on all fields.
	* manual/tunables.texi: Replace ifunc with hwcaps.
	* sysdeps/x86/cpu-features.c (TUNABLE_CALLBACK (set_ifunc)):
	Renamed to ..
	(TUNABLE_CALLBACK (set_hwcaps)): This.
	(init_cpu_features): Updated.
	* sysdeps/x86/cpu-features.h (cpu_features): Change type of
	data_cache_size, data_cache_size and non_temporal_threshold to
	unsigned long int.
	* sysdeps/x86/cpu-tunables.c (DEFAULT_STRLEN): Removed.
	(TUNABLE_CALLBACK (set_ifunc)): Renamed to ...
	(TUNABLE_CALLBACK (set_hwcaps)): This.  Update comments.  Don't
	use DEFAULT_STRLEN.
2017-06-21 10:21:37 -07:00
H.J. Lu
da69a35566 Move x86 specific tunables to x86/dl-tunables.list
* elf/dl-tunables.list: Move x86 specific tunables to ...
	* sysdeps/x86/dl-tunables.list: Here.  New file.
2017-06-20 14:03:31 -07:00
H.J. Lu
905947c304 tunables: Add IFUNC selection and cache sizes
The current IFUNC selection is based on microbenchmarks in glibc.  It
should give the best performance for most workloads.  But other choices
may have better performance for a particular workload or on the hardware
which wasn't available at the selection was made.  The environment
variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
where the feature name is case-sensitive and has to match the ones in
cpu-features.h.  It can be used by glibc developers to override the
IFUNC selection to tune for a new processor or improve performance for
a particular workload.  It isn't intended for normal end users.

NOTE: the IFUNC selection may change over time.  Please check all
multiarch implementations when experimenting.

Also, GLIBC_TUNABLES=glibc.tune.x86_non_temporal_threshold=NUMBER is
provided to set threshold to use non temporal store to NUMBER,
GLIBC_TUNABLES=glibc.tune.x86_data_cache_size=NUMBER to set data cache
size, GLIBC_TUNABLES=glibc.tune.x86_shared_cache_size=NUMBER to set
shared cache size.

	* elf/dl-tunables.list (tune): Add ifunc,
	x86_non_temporal_threshold,
	x86_data_cache_size and x86_shared_cache_size.
	* manual/tunables.texi: Document glibc.tune.ifunc,
	glibc.tune.x86_data_cache_size, glibc.tune.x86_shared_cache_size
	and glibc.tune.x86_non_temporal_threshold.
	* sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
	* sysdeps/x86/cpu-tunables.c: Likewise.
	* sysdeps/x86/cacheinfo.c
	(init_cacheinfo): Check and get data cache size, shared cache
	size and non temporal threshold from cpu_features.
	* sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
	New.
	[HAVE_TUNABLES] Include <unistd.h>.
	[HAVE_TUNABLES] Include <elf/dl-tunables.h>.
	[HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
	[HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
	IFUNC selection, data cache size, shared cache size and non
	temporal threshold.
	* sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
	shared_cache_size and non_temporal_threshold.
2017-06-20 08:37:28 -07:00
Zack Weinberg
09a596cc2c Remove bits/string.h.
These machine-dependent inline string functions have never been on by
default, and even if they were a good idea at the time they were
introduced, they haven't really been touched in ten to fifteen years
and probably aren't a good idea on current-gen processors.  Current
thinking is that this class of optimization is best left to the
compiler.

	* bits/string.h, string/bits/string.h
	* sysdeps/aarch64/bits/string.h
	* sysdeps/m68k/m680x0/m68020/bits/string.h
	* sysdeps/s390/bits/string.h, sysdeps/sparc/bits/string.h
	* sysdeps/x86/bits/string.h: Delete file.

	* string/string.h: Don't include bits/string.h.
	* string/bits/string3.h: Rename to bits/string_fortified.h.
	No need to undef various symbols that the removed headers
	might have defined as macros.
	* string/Makefile (headers): Remove bits/string.h, change
	bits/string3.h to bits/string_fortified.h.
	* string/string-inlines.c: Update commentary.  Remove definitions
	of various macros that nothing looks at anymore.  Don't directly
	include bits/string.h. Set _STRING_INLINE_unaligned here, based on
	compiler-predefined macros.
	* string/strncat.c: If STRNCAT is not defined, or STRNCAT_PRIMARY
	_is_ defined, provide internal hidden alias __strncat.
	* include/string.h: Declare internal hidden alias __strncat.
	Only forward __stpcpy to __builtin_stpcpy if __NO_STRING_INLINES is
	not defined.
	* include/bits/string3.h: Rename to bits/string_fortified.h,
	update to match above.

	* sysdeps/i386/string-inlines.c: Define compat symbols for
	everything formerly defined by sysdeps/x86/bits/string.h.
	Make existing definitions into compat symbols as well.
	Remove some no-longer-necessary messing around with macros.

	* sysdeps/powerpc/powerpc32/power4/multiarch/mempcpy.c
	* sysdeps/powerpc/powerpc64/multiarch/mempcpy.c
	* sysdeps/powerpc/powerpc64/multiarch/stpcpy.c
	* sysdeps/s390/multiarch/mempcpy.c
	No need to define _HAVE_STRING_ARCH_mempcpy.
	Do define __NO_STRING_INLINES and NO_MEMPCPY_STPCPY_REDIRECT.

	* sysdeps/i386/i686/multiarch/strncat-c.c
	* sysdeps/s390/multiarch/strncat-c.c
	* sysdeps/x86_64/multiarch/strncat-c.c
	Define STRNCAT_PRIMARY.  Don't change definition of libc_hidden_def.
2017-06-20 08:21:24 -04:00
Siddhesh Poyarekar
511c5a1087 Make LD_HWCAP_MASK usable for static binaries
The LD_HWCAP_MASK environment variable was ignored in static binaries,
which is inconsistent with the behaviour of dynamically linked
binaries.  This seems to have been because of the inability of
ld_hwcap_mask being read early enough to influence anything but now
that it is in tunables, the mask is usable in static binaries as well.

This feature is important for aarch64, which relies on HWCAP_CPUID
being masked out to disable multiarch.  A sanity test on x86_64 shows
that there are no failures.  Likewise for aarch64.

	* elf/dl-hwcaps.h [HAVE_TUNABLES]: Always read hwcap_mask.
	* sysdeps/sparc/sparc32/dl-machine.h [HAVE_TUNABLES]:
	Likewise.
	* sysdeps/x86/cpu-features.c (init_cpu_features): Always set
	up hwcap and hwcap_mask.
2017-06-07 11:11:40 +05:30
Siddhesh Poyarekar
ff08fc59e3 tunables: Use glibc.tune.hwcap_mask tunable instead of _dl_hwcap_mask
Drop _dl_hwcap_mask when building with tunables.  This completes the
transition of hwcap_mask reading from _dl_hwcap_mask to tunables.

	* elf/dl-hwcaps.h: New file.
	* elf/dl-hwcaps.c: Include it.
	(_dl_important_hwcaps)[HAVE_TUNABLES]: Read and update
	glibc.tune.hwcap_mask.
	* elf/dl-cache.c: Include dl-hwcaps.h.
	(_dl_load_cache_lookup)[HAVE_TUNABLES]: Read
	glibc.tune.hwcap_mask.
	* sysdeps/sparc/sparc32/dl-machine.h: Likewise.
	* elf/dl-support.c (_dl_hwcap2)[HAVE_TUNABLES]: Drop
	_dl_hwcap_mask.
	* elf/rtld.c (rtld_global_ro)[HAVE_TUNABLES]: Drop
	_dl_hwcap_mask.
	(process_envvars)[HAVE_TUNABLES]: Likewise.
	* sysdeps/generic/ldsodefs.h (rtld_global_ro)[HAVE_TUNABLES]:
	Likewise.
	* sysdeps/x86/cpu-features.c (init_cpu_features): Don't
	initialize dl_hwcap_mask when tunables are enabled.
2017-06-07 11:11:38 +05:30
H.J. Lu
48e7bc7a55 x86: Don't use dl_x86_cpu_features in cacheinfo.c
Since cpu_features is available, use it instead of dl_x86_cpu_features.

	* sysdeps/x86/cacheinfo.c (intel_check_word): Accept cpu_features
	and use it instead of dl_x86_cpu_features.
	(handle_intel): Replace maxidx with cpu_features.  Pass
	cpu_features to intel_check_word.
	(__cache_sysconf): Pass cpu_features to handle_intel.
	(init_cacheinfo): Likewise.  Use cpu_features instead of
	dl_x86_cpu_features.
2017-06-05 16:20:11 -07:00
H.J. Lu
935971ba6b x86-64: Optimize memcmp/wmemcmp with AVX2 and MOVBE
Optimize x86-64 memcmp/wmemcmp with AVX2.  It uses vector compare as
much as possible.  It is as fast as SSE4 memcmp for size <= 16 bytes
and up to 2X faster for size > 16 bytes on Haswell and Skylake.  Select
AVX2 memcmp/wmemcmp on AVX2 machines where vzeroupper is preferred and
AVX unaligned load is fast.

NB: It uses TZCNT instead of BSF since TZCNT produces the same result
as BSF for non-zero input.  TZCNT is faster than BSF and is executed
as BSF if machine doesn't support TZCNT.

Key features:

1. For size from 2 to 7 bytes, load as big endian with movbe and bswap
   to avoid branches.
2. Use overlapping compare to avoid branch.
3. Use vector compare when size >= 4 bytes for memcmp or size >= 8
   bytes for wmemcmp.
4. If size is 8 * VEC_SIZE or less, unroll the loop.
5. Compare 4 * VEC_SIZE at a time with the aligned first memory area.
6. Use 2 vector compares when size is 2 * VEC_SIZE or less.
7. Use 4 vector compares when size is 4 * VEC_SIZE or less.
8. Use 8 vector compares when size is 8 * VEC_SIZE or less.

	* sysdeps/x86/cpu-features.h (index_cpu_MOVBE): New.
	* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
	memcmp-avx2 and wmemcmp-avx2.
	* sysdeps/x86_64/multiarch/ifunc-impl-list.c
	(__libc_ifunc_impl_list): Test __memcmp_avx2 and __wmemcmp_avx2.
	* sysdeps/x86_64/multiarch/memcmp-avx2.S: New file.
	* sysdeps/x86_64/multiarch/wmemcmp-avx2.S: Likewise.
	* sysdeps/x86_64/multiarch/memcmp.S: Use __memcmp_avx2 on AVX
	2 machines if AVX unaligned load is fast and vzeroupper is
	preferred.
	* sysdeps/x86_64/multiarch/wmemcmp.S: Use __wmemcmp_avx2 on AVX
	2 machines if AVX unaligned load is fast and vzeroupper is
	preferred.
2017-06-05 12:52:55 -07:00
H.J. Lu
9cd30491dd x86: Add macros to implement ifunce selection in C
These macros are used to implement ifunc selection in C.  To implement
an ifunc function, foo, which returns the address of __foo_sse2 or
__foo_avx2:

   __foo_avx2:

   #define foo __redirect_foo
   #define __foo __redirect___foo
   #include <foo.h>
   #undef foo
   #undef __foo
   #define SYMBOL_NAME foo
   #include <init-arch.h>

   extern __typeof (REDIRECT_NAME) OPTIMIZE (sse2) attribute_hidden;
   extern __typeof (REDIRECT_NAME) OPTIMIZE (avx2) attribute_hidden;

   static inline void *
   foo_selector (void)
   {
     if (use AVX2)
      return OPTIMIZE (avx2);

     return OPTIMIZE (sse2);
   }

   libc_ifunc_redirected (__redirect_foo, foo, foo_selector ());

	* sysdeps/x86/init-arch.h (PASTER1): New.
	(EVALUATOR1): Likewise.
	(PASTER2): Likewise.
	(EVALUATOR2): Likewise.
	(REDIRECT_NAME): Likewise.
	(OPTIMIZE): Likewise.
	(IFUNC_SELECTOR): Likewise.
2017-06-05 08:28:13 -07:00
H.J. Lu
808fd9e6fe x86: Update __x86_shared_non_temporal_threshold
__x86_shared_non_temporal_threshold was set to 6 times of per-core
shared cache size, based on the large memcpy micro benchmark in glibc
on a 8-core processor.  For a processor with more than 8 cores, the
threshold is too low.  Set __x86_shared_non_temporal_threshold to the
3/4 of the total shared cache size so that it is unchanged on 8-core
processors.  On processors with less than 8 cores, the threshold is
lower.

	* sysdeps/x86/cacheinfo.c (__x86_shared_non_temporal_threshold):
	Set to the 3/4 of the total shared cache size.
2017-06-02 17:32:37 -07:00
Siddhesh Poyarekar
4158ba082c Delay initialization of CPU features struct in static binaries
Allow the CPU features structure set up to be overridden by tunables
by delaying it to until after tunables are initialized.  The
initialization is already delayed in dynamically linked glibc, it is
only in static binaries that the initialization is set early to allow
it to influence IFUNC relocations that happen in libc-start.  It is a
bit too early however and there is a good place between tunables
initialization and IFUNC relocations where this can be done.

Verified that this does not regress the testsuite.

	* csu/libc-start.c [!ARCH_INIT_CPU_FEATURES]: Define
	ARCH_INIT_CPU_FEATURES.
	(LIBC_START_MAIN): Call it.
	* sysdeps/unix/sysv/linux/aarch64/libc-start.c
	(__libc_start_main): Remove.
	(ARCH_INIT_CPU_FEATURES): New macro.
	* sysdeps/x86/libc-start.c (__libc_start_main): Remove.
	(ARCH_INIT_CPU_FEATURES): New macro.
2017-05-31 06:38:33 +05:30
H.J. Lu
9c450f6f6f x86: Don't include cacheinfo.c in ld.so
Since cacheinfo.c isn't used by ld.so, there is no need to include it
in ld.so.

	* sysdeps/x86/cacheinfo.c: Skip if not in libc.
2017-05-24 06:33:43 -07:00
H.J. Lu
7c1d722554 x86: Use __get_cpu_features to get cpu_features
Remove is_intel, is_amd and max_cpuid macros.  Use __get_cpu_features
to get cpu_features instead.

	* sysdeps/x86/cacheinfo.c (is_intel): Removed.
	(is_amd): Likewise.
	(max_cpuid): Likewise.
	(__cache_sysconf): Use __get_cpu_features to get cpu_features.
	(init_cacheinfo): Likewise.
2017-05-24 06:28:52 -07:00
Adhemerval Zanella
eab380d8ec Move shared pthread definitions to common headers
This patch removes all the replicated pthread definition accross the
architectures and consolidates it on shared headers.  The new
organization is as follow:

  * Architecture specific definition (such as pthread types sizes) are
    place in the new pthreadtypes-arch.h header in arch specific path.

  * All shared structure definition are moved to a common NPTL header
    at sysdeps/nptl/bits/pthreadtypes.h (with now includes the arch
    specific one for internal definitions).

  * Also, for C11 future thread support, both mutex and condition
    definition are placed in a common header at
    sysdeps/nptl/bits/thread-shared-types.h.

It is also a refactor patch without expected functional changes.
Checked with a build for all major ABI (aarch64-linux-gnu, alpha-linux-gnu,
arm-linux-gnueabi, i386-linux-gnu, ia64-linux-gnu,
m68k-linux-gnu, microblaze-linux-gnu, mips{64}-linux-gnu, nios2-linux-gnu,
powerpc{64le}-linux-gnu, s390{x}-linux-gnu, sparc{64}-linux-gnu,
tile{pro,gx}-linux-gnu, and x86_64-linux-gnu).

	* posix/Makefile (headers): Add pthreadtypes-arch.h and
	thread-shared-types.h.
	* sysdeps/aarch64/nptl/bits/pthreadtypes-arch.h: New file: arch
	specific thread definition.
	* sysdeps/alpha/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/arm/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/hppa/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/ia64/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/m68k/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/microblaze/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/mips/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/nios2/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/powerpc/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/s390/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/sh/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/sparc/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/tile/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/x86/nptl/bits/pthreadtypes-arch.h: Likewise.
	* sysdeps/nptl/bits/thread-shared-types.h: New file: shared
	thread definition between POSIX and C11.
	* sysdeps/aarch64/nptl/bits/pthreadtypes.h.: Remove file.
	* sysdeps/alpha/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/arm/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/hppa/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/m68k/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/microblaze/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/mips/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/nios2/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/ia64/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/powerpc/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/s390/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/sh/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/sparc/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/tile/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/x86/nptl/bits/pthreadtypes.h: Likewise.
	* sysdeps/nptl/bits/pthreadtypes.h: New file: common thread
	definitions shared across all architectures.
2017-05-09 17:49:17 -03:00
H.J. Lu
1432d38ea0 x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
early during startup.  They are used by dynamic linker to determine
platform and build an array of hardware capability names, which are
added to search path when loading shared object.  dl_platform and
dl_hwcap are unused on x86-64.  On i386, i386, i486, i586 and i686
platforms were supported and only SSE2 capability was used.

On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
processor capabilities is obsolete since all information is available
in dl_x86_cpu_features.  This patch sets dl_platform and dl_hwcap from
dl_x86_cpu_features in dynamic linker.  On i386, the available plaforms
are changed to i586 and i686 since i386 has been deprecated.  On x86-64,
the available plaforms are haswell, which is for Haswell class processors
with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
AVX512PF.  A capability, avx512_1, is also added to x86-64 for AVX512
ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.

	[BZ #21391]
	* sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]:
	Only call init_cpu_features.
	[!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed.
	* sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise.
	* sysdeps/i386/dl-procinfo.h: Removed.
	* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include
	<sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>.  Include
	<sysdeps/x86/dl-procinfo.h>.
	(_dl_procinfo): Replace _DL_HWCAP_COUNT with 32.
	* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]:
	Include <sysdeps/x86/dl-procinfo.h> instead of
	 <sysdeps/generic/dl-procinfo.h>.
	* sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>.
	(init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask.
	* sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New.
	(bit_cpu_MOVBE): Likewise.
	(bit_cpu_BMI1): Likewise.
	(bit_cpu_BMI2): Likewise.
	(index_cpu_BMI1): Likewise.
	(index_cpu_BMI2): Likewise.
	(index_cpu_LZCNT): Likewise.
	(index_cpu_MOVBE): Likewise.
	(index_cpu_POPCNT): Likewise.
	(reg_BMI1): Likewise.
	(reg_BMI2): Likewise.
	(reg_LZCNT): Likewise.
	(reg_MOVBE): Likewise.
	(reg_POPCNT): Likewise.
	* sysdeps/x86/dl-hwcap.h: New file.
	* sysdeps/x86/dl-procinfo.h: Likewise.
	* sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New.
	(_dl_x86_platforms): Likewise.
2017-05-03 13:44:35 -07:00
H.J. Lu
4cb334c4d6 x86: Use AVX2 memcpy/memset on Skylake server [BZ #21396]
On Skylake server, AVX512 load/store instructions in memcpy/memset may
lead to lower CPU turbo frequency in certain situations.  Use of AVX2
in memcpy/memset has been observed to have improved overall performance
in many workloads due to the higher frequency.

Since AVX512ER is unique to Xeon Phi, this patch sets Prefer_No_AVX512
if AVX512ER isn't available so that AVX2 versions of memcpy/memset are
used on Skylake server.

	[BZ #21396]
	* sysdeps/x86/cpu-features.c (init_cpu_features): Set
	Prefer_No_AVX512 if AVX512ER isn't available.
	* sysdeps/x86/cpu-features.h (bit_arch_Prefer_No_AVX512): New.
	(index_arch_Prefer_No_AVX512): Likewise.
	* sysdeps/x86_64/multiarch/memcpy.S (__new_memcpy): Don't use
	AVX512 version if Prefer_No_AVX512 is set.
	* sysdeps/x86_64/multiarch/memcpy_chk.S (__memcpy_chk):
	Likewise.
	* sysdeps/x86_64/multiarch/memmove.S (__libc_memmove): Likewise.
	* sysdeps/x86_64/multiarch/memmove_chk.S (__memmove_chk):
	Likewise.
	* sysdeps/x86_64/multiarch/mempcpy.S (__mempcpy): Likewise.
	* sysdeps/x86_64/multiarch/mempcpy_chk.S (__mempcpy_chk):
	Likewise.
	* sysdeps/x86_64/multiarch/memset.S (memset): Likewise.
	* sysdeps/x86_64/multiarch/memset_chk.S (__memset_chk):
	Likewise.
2017-04-18 14:01:45 -07:00
H.J. Lu
1c53cb49de x86: Set Prefer_No_VZEROUPPER if AVX512ER is available
AVX512ER won't be implemented in any Xeon processors and will be in
all Xeon Phi processors.  Don't check CPU model number when setting
Prefer_No_VZEROUPPER for Xeon Phi.  Instead, set Prefer_No_VZEROUPPER
if AVX512ER is available.  It works with current and future Xeon Phi
and non-Xeon Phi processors.

	* sysdeps/x86/cpu-features.c (init_cpu_features): Set
	Prefer_No_VZEROUPPER if AVX512ER is available.
	* sysdeps/x86/cpu-features.h
	(bit_cpu_AVX512PF): New.
	(bit_cpu_AVX512ER): Likewise.
	(bit_cpu_AVX512CD): Likewise.
	(bit_cpu_AVX512BW): Likewise.
	(bit_cpu_AVX512VL): Likewise.
	(index_cpu_AVX512PF): Likewise.
	(index_cpu_AVX512ER): Likewise.
	(index_cpu_AVX512CD): Likewise.
	(index_cpu_AVX512BW): Likewise.
	(index_cpu_AVX512VL): Likewise.
	(reg_AVX512PF): Likewise.
	(reg_AVX512ER): Likewise.
	(reg_AVX512CD): Likewise.
	(reg_AVX512BW): Likewise.
	(reg_AVX512VL): Likewise.
2017-04-18 08:27:32 -07:00
Adhemerval Zanella
38efe8c5a5 Consolidate pthreadtype.h placementConsolidate pthreadtype.h placement
This patch moves all arch specific pthreadtypes.h to a similar path
for all architectures (sysdeps/unix/sysv/<arch>/bits).  No functional
or build change is expected.  The idea is mainly to organize the
header placement for all architectures.

Checked with a build for all major ABI (aarch64-linux-gnu, alpha-linux-gnu,
arm-linux-gnueabi, i386-linux-gnu, ia64-linux-gnu,
m68k-linux-gnu, microblaze-linux-gnu [1], mips{64}-linux-gnu, nios2-linux-gnu,
powerpc{64le}-linux-gnu, s390{x}-linux-gnu, sparc{64}-linux-gnu,
tile{pro,gx}-linux-gnu, and x86_64-linux-gnu).

	* sysdeps/unix/sysv/linux/x86/Implies: New file.
	* sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h: Move to ...
	* sysdeps/alpha/nptl/bits/pthreadtypes.h: ... here.
	* sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h: Move to ...
	* sysdeps/powerpc/nptl/bits/pthreadtypes.h: ... here.
	* sysdeps/x86/bits/pthreadtypes.h: Move to ...
	* sysdeps/x86/nptl/bits/pthreadtypes.h: ... here.
2017-04-10 17:33:10 -03:00
H.J. Lu
fda19e0438 Add sysdeps/x86/dl-procinfo.c
Add sysdeps/x86/dl-procinfo.c for x86 version of processor capability
information to reduce duplication between i386 and x86_64 dl-procinfo.c.

	* sysdeps/i386/dl-procinfo.c: Include
	<sysdeps/x86/dl-procinfo.c>.
	* sysdeps/x86_64/dl-procinfo.c: Likewise.
	* sysdeps/x86/dl-procinfo.c: New file.
2017-04-10 12:01:45 -07:00
H.J. Lu
bf7730194f Check if SSE is available with HAS_CPU_FEATURE
Similar to other CPU feature checks, check if SSE is available with
HAS_CPU_FEATURE.

	* sysdeps/i386/fpu/fclrexcpt.c (__feclearexcept): Use
	HAS_CPU_FEATURE to check for SSE.
	* sysdeps/i386/fpu/fedisblxcpt.c (fedisableexcept): Likewise.
	* sysdeps/i386/fpu/feenablxcpt.c (feenableexcept): Likewise.
	* sysdeps/i386/fpu/fegetenv.c (__fegetenv): Likewise.
	* sysdeps/i386/fpu/fegetmode.c (fegetmode): Likewise.
	* sysdeps/i386/fpu/feholdexcpt.c (__feholdexcept): Likewise.
	* sysdeps/i386/fpu/fesetenv.c (__fesetenv): Likewise.
	* sysdeps/i386/fpu/fesetmode.c (fesetmode): Likewise.
	* sysdeps/i386/fpu/fesetround.c (__fesetround): Likewise.
	* sysdeps/i386/fpu/feupdateenv.c (__feupdateenv): Likewise.
	* sysdeps/i386/fpu/fgetexcptflg.c (__fegetexceptflag): Likewise.
	* sysdeps/i386/fpu/fsetexcptflg.c (__fesetexceptflag): Likewise.
	* sysdeps/i386/fpu/ftestexcept.c (fetestexcept): Likewise.
	* sysdeps/i386/setfpucw.c (__setfpucw): Likewise.
	* sysdeps/x86/cpu-features.h (bit_cpu_SSE): New.
	(index_cpu_SSE): Likewise.
	(reg_SSE): Likewise.
2017-04-07 07:44:59 -07:00
H.J. Lu
b170d2e7ab Use CPU_FEATURES_CPU_P to check if AVX is available
Don't use bit_cpu_AVX directly.

	* sysdeps/x86/cpu-features.c (init_cpu_features): Check AVX with
	CPU_FEATURES_CPU_P.
2017-03-17 11:38:13 -07:00
Joseph Myers
2072f5c34e Remove C++ namespace handling from glibc headers.
glibc headers include some code (not particularly consistent or
systematic) to put various declarations in C++ namespaces std and
__c99, if _GLIBCPP_USE_NAMESPACES is defined.

As noted in <https://gcc.gnu.org/ml/libstdc++/2017-03/msg00025.html>,
this macro was removed from libstdc++ in 2000.  I don't expect
compilation with such old versions of libstdc++ to work with current
glibc headers anyway (whereas old *binaries* are expected to stay
working with current glibc); this patch (which should be a no-op with
any libstdc++ version postdating that removal) removes all this code
from the glibc headers.

The begin-end-check.pl test, whose comments say it is about checking
these namespace macro calls, is also removed.  The code in that test
would have covered __BEGIN_DECLS / __END_DECLS as well, but if those
weren't properly matched it would show up with the
check-installed-headers-cxx tests, so I don't think there is an actual
use for keeping begin-end-check.pl with the namespace code removed.

Tested for x86_64 and x86 (testsuite, and that installed stripped
shared libraries are unchanged by the patch).

	* misc/sys/cdefs.h (__BEGIN_NAMESPACE_STD): Remove macro.
	(__END_NAMESPACE_STD): Likewise.
	(__USING_NAMESPACE_STD): Likewise.
	(__BEGIN_NAMESPACE_C99): Likewise.
	(__END_NAMESPACE_C99): Likewise.
	(__USING_NAMESPACE_C99): Likewise.
	* math/math.h (_Mdouble_BEGIN_NAMESPACE): Do not define and
	undefine macro.
	(_Mdouble_END_NAMESPACE): Likewise.
	* ctype/ctype.h: Do not handle C++ namespaces.
	* libio/bits/stdio-ldbl.h: Likewise.
	* libio/stdio.h: Likewise.
	* locale/locale.h: Likewise.
	* math/bits/mathcalls.h: Likewise.
	* setjmp/setjmp.h: Likewise.
	* signal/signal.h: Likewise.
	* stdlib/bits/stdlib-float.h: Likewise.
	* stdlib/bits/stdlib-ldbl.h: Likewise.
	* stdlib/stdlib.h: Likewise.
	* string/string.h: Likewise.
	* sysdeps/x86/fpu/bits/mathinline.h: Likewise.
	* time/bits/types/clock_t.h: Likewise.
	* time/bits/types/struct_tm.h: Likewise.
	* time/bits/types/time_t.h: Likewise.
	* time/time.h: Likewise.
	* wcsmbs/bits/wchar-ldbl.h: Likewise.
	* wcsmbs/uchar.h: Likewise.
	* wcsmbs/wchar.h: Likewise.
	[_GLIBCPP_USE_NAMESPACES] (wint_t): Remove conditional definition.
	* wctype/wctype.h: Do not handle C++ namespaces.
	* scripts/begin-end-check.pl: Remove.
	* Makefile (installed-headers): Likewise.
	(tests-special): Do not add $(objpfx)begin-end-check.out.
	($(objpfx)begin-end-check.out): Remove.
2017-03-16 13:31:57 +00:00
Joseph Myers
ffe308e4fc Fix test-math-vector-sincos.h aliasing.
x86_64 libmvec tests have been failing to build lately with GCC
mainline with -Wuninitialized errors, and Markus Trippelsdorf traced
this to an aliasing issue
<https://sourceware.org/ml/libc-alpha/2017-03/msg00169.html>.

This patch fixes the aliasing issue, so that the vectors-of-pointers
are initialized using a union instead of pointer casts.  This also
fixes the testsuite build failures with GCC mainline.

Tested for x86_64 (full testsuite with GCC 6; testsuite build with GCC
mainline with build-many-glibcs.py).

	* sysdeps/x86/fpu/test-math-vector-sincos.h (INIT_VEC_PTRS_LOOP):
	Use a union when storing pointers.
	(VECTOR_WRAPPER_fFF_2): Do not take address of integer vector and
	cast result when passing to INIT_VEC_PTRS_LOOP.
	(VECTOR_WRAPPER_fFF_3): Likewise.
	(VECTOR_WRAPPER_fFF_4): Likewise.
2017-03-15 17:32:46 +00:00
H.J. Lu
52ac22365a Use index_cpu_RTM and reg_RTM to clear the bit_cpu_RTM bit
* sysdeps/x86/cpu-features.c (init_cpu_features): Use
	index_cpu_RTM and reg_RTM to clear the bit_cpu_RTM bit.
2017-02-17 11:53:26 -08:00
Torvald Riegel
cc25c8b4c1 New pthread rwlock that is more scalable.
This replaces the pthread rwlock with a new implementation that uses a
more scalable algorithm (primarily through not using a critical section
anymore to make state changes).  The fast path for rdlock acquisition and
release is now basically a single atomic read-modify write or CAS and a few
branches.  See nptl/pthread_rwlock_common.c for details.

	* nptl/DESIGN-rwlock.txt: Remove.
	* nptl/lowlevelrwlock.sym: Remove.
	* nptl/Makefile: Add new tests.
	* nptl/pthread_rwlock_common.c: New file.  Contains the new rwlock.
	* nptl/pthreadP.h (PTHREAD_RWLOCK_PREFER_READER_P): Remove.
	(PTHREAD_RWLOCK_WRPHASE, PTHREAD_RWLOCK_WRLOCKED,
	PTHREAD_RWLOCK_RWAITING, PTHREAD_RWLOCK_READER_SHIFT,
	PTHREAD_RWLOCK_READER_OVERFLOW, PTHREAD_RWLOCK_WRHANDOVER,
	PTHREAD_RWLOCK_FUTEX_USED): New.
	* nptl/pthread_rwlock_init.c (__pthread_rwlock_init): Adapt to new
	implementation.
	* nptl/pthread_rwlock_rdlock.c (__pthread_rwlock_rdlock_slow): Remove.
	(__pthread_rwlock_rdlock): Adapt.
	* nptl/pthread_rwlock_timedrdlock.c
	(pthread_rwlock_timedrdlock): Adapt.
	* nptl/pthread_rwlock_timedwrlock.c
	(pthread_rwlock_timedwrlock): Adapt.
	* nptl/pthread_rwlock_trywrlock.c (pthread_rwlock_trywrlock): Adapt.
	* nptl/pthread_rwlock_tryrdlock.c (pthread_rwlock_tryrdlock): Adapt.
	* nptl/pthread_rwlock_unlock.c (pthread_rwlock_unlock): Adapt.
	* nptl/pthread_rwlock_wrlock.c (__pthread_rwlock_wrlock_slow): Remove.
	(__pthread_rwlock_wrlock): Adapt.
	* nptl/tst-rwlock10.c: Adapt.
	* nptl/tst-rwlock11.c: Adapt.
	* nptl/tst-rwlock17.c: New file.
	* nptl/tst-rwlock18.c: New file.
	* nptl/tst-rwlock19.c: New file.
	* nptl/tst-rwlock2b.c: New file.
	* nptl/tst-rwlock8.c: Adapt.
	* nptl/tst-rwlock9.c: Adapt.
	* sysdeps/aarch64/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/arm/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/hppa/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/ia64/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/m68k/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/microblaze/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/mips/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/nios2/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/s390/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/sh/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/sparc/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/tile/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h
	(pthread_rwlock_t): Adapt.
	* sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h
	(pthread_rwlock_t): Adapt.
	* sysdeps/x86/bits/pthreadtypes.h (pthread_rwlock_t): Adapt.
	* nptl/nptl-printers.py (): Adapt.
	* nptl/nptl_lock_constants.pysym: Adapt.
	* nptl/test-rwlock-printers.py: Adapt.
	* nptl/test-rwlockattr-printers.c: Adapt.
	* nptl/test-rwlockattr-printers.py: Adapt.
2017-01-10 11:50:17 +01:00
Joseph Myers
bfff8b1bec Update copyright dates with scripts/update-copyrights. 2017-01-01 00:14:16 +00:00
Torvald Riegel
ed19993b5b New condvar implementation that provides stronger ordering guarantees.
This is a new implementation for condition variables, required
after http://austingroupbugs.net/view.php?id=609 to fix bug 13165.  In
essence, we need to be stricter in which waiters a signal or broadcast
is required to wake up; this couldn't be solved using the old algorithm.
ISO C++ made a similar clarification, so this also fixes a bug in
current libstdc++, for example.

We can't use the old algorithm anymore because futexes do not guarantee
to wake in FIFO order.  Thus, when we wake, we can't simply let any
waiter grab a signal, but we need to ensure that one of the waiters
happening before the signal is woken up.  This is something the previous
algorithm violated (see bug 13165).

There's another issue specific to condvars: ABA issues on the underlying
futexes.  Unlike mutexes that have just three states, or semaphores that
have no tokens or a limited number of them, the state of a condvar is
the *order* of the waiters.  A waiter on a semaphore can grab a token
whenever one is available; a condvar waiter must only consume a signal
if it is eligible to do so as determined by the relative order of the
waiter and the signal.
Therefore, this new algorithm maintains two groups of waiters: Those
eligible to consume signals (G1), and those that have to wait until
previous waiters have consumed signals (G2).  Once G1 is empty, G2
becomes the new G1.  64b counters are used to avoid ABA issues.

This condvar doesn't yet use a requeue optimization (ie, on a broadcast,
waking just one thread and requeueing all others on the futex of the
mutex supplied by the program).  I don't think doing the requeue is
necessarily the right approach (but I haven't done real measurements
yet):
* If a program expects to wake many threads at the same time and make
that scalable, a condvar isn't great anyway because of how it requires
waiters to operate mutually exclusive (due to the mutex usage).  Thus, a
thundering herd problem is a scalability problem with or without the
optimization.  Using something like a semaphore might be more
appropriate in such a case.
* The scalability problem is actually at the mutex side; the condvar
could help (and it tries to with the requeue optimization), but it
should be the mutex who decides how that is done, and whether it is done
at all.
* Forcing all but one waiter into the kernel-side wait queue of the
mutex prevents/avoids the use of lock elision on the mutex.  Thus, it
prevents the only cure against the underlying scalability problem
inherent to condvars.
* If condvars use short critical sections (ie, hold the mutex just to
check a binary flag or such), which they should do ideally, then forcing
all those waiter to proceed serially with kernel-based hand-off (ie,
futex ops in the mutex' contended state, via the futex wait queues) will
be less efficient than just letting a scalable mutex implementation take
care of it.  Our current mutex impl doesn't employ spinning at all, but
if critical sections are short, spinning can be much better.
* Doing the requeue stuff requires all waiters to always drive the mutex
into the contended state.  This leads to each waiter having to call
futex_wake after lock release, even if this wouldn't be necessary.

	[BZ #13165]
	* nptl/pthread_cond_broadcast.c (__pthread_cond_broadcast): Rewrite to
	use new algorithm.
	* nptl/pthread_cond_destroy.c (__pthread_cond_destroy): Likewise.
	* nptl/pthread_cond_init.c (__pthread_cond_init): Likewise.
	* nptl/pthread_cond_signal.c (__pthread_cond_signal): Likewise.
	* nptl/pthread_cond_wait.c (__pthread_cond_wait): Likewise.
	(__pthread_cond_timedwait): Move here from pthread_cond_timedwait.c.
	(__condvar_confirm_wakeup, __condvar_cancel_waiting,
	__condvar_cleanup_waiting, __condvar_dec_grefs,
	__pthread_cond_wait_common): New.
	(__condvar_cleanup): Remove.
	* npt/pthread_condattr_getclock.c (pthread_condattr_getclock): Adapt.
	* npt/pthread_condattr_setclock.c (pthread_condattr_setclock):
	Likewise.
	* npt/pthread_condattr_getpshared.c (pthread_condattr_getpshared):
	Likewise.
	* npt/pthread_condattr_init.c (pthread_condattr_init): Likewise.
	* nptl/tst-cond1.c: Add comment.
	* nptl/tst-cond20.c (do_test): Adapt.
	* nptl/tst-cond22.c (do_test): Likewise.
	* sysdeps/aarch64/nptl/bits/pthreadtypes.h (pthread_cond_t): Adapt
	structure.
	* sysdeps/arm/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/ia64/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/m68k/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/microblaze/nptl/bits/pthreadtypes.h (pthread_cond_t):
	Likewise.
	* sysdeps/mips/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/nios2/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/s390/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/sh/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/tile/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h (pthread_cond_t):
	Likewise.
	* sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h (pthread_cond_t):
	Likewise.
	* sysdeps/x86/bits/pthreadtypes.h (pthread_cond_t): Likewise.
	* sysdeps/nptl/internaltypes.h (COND_NWAITERS_SHIFT): Remove.
	(COND_CLOCK_BITS): Adapt.
	* sysdeps/nptl/pthread.h (PTHREAD_COND_INITIALIZER): Adapt.
	* nptl/pthreadP.h (__PTHREAD_COND_CLOCK_MONOTONIC_MASK,
	__PTHREAD_COND_SHARED_MASK): New.
	* nptl/nptl-printers.py (CLOCK_IDS): Remove.
	(ConditionVariablePrinter, ConditionVariableAttributesPrinter): Adapt.
	* nptl/nptl_lock_constants.pysym: Adapt.
	* nptl/test-cond-printers.py: Adapt.
	* sysdeps/unix/sysv/linux/hppa/internaltypes.h (cond_compat_clear,
	cond_compat_check_and_clear): Adapt.
	* sysdeps/unix/sysv/linux/hppa/pthread_cond_timedwait.c: Remove file ...
	* sysdeps/unix/sysv/linux/hppa/pthread_cond_wait.c
	(__pthread_cond_timedwait): ... and move here.
	* nptl/DESIGN-condvar.txt: Remove file.
	* nptl/lowlevelcond.sym: Likewise.
	* nptl/pthread_cond_timedwait.c: Likewise.
	* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i586/pthread_cond_broadcast.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i586/pthread_cond_signal.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i586/pthread_cond_timedwait.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i586/pthread_cond_wait.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i686/pthread_cond_broadcast.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i686/pthread_cond_signal.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i686/pthread_cond_timedwait.S: Likewise.
	* sysdeps/unix/sysv/linux/i386/i686/pthread_cond_wait.S: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: Likewise.
	* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: Likewise.
2016-12-31 14:56:47 +01:00
Andrew Senkevich
2702856bf4 Disable TSX on some Haswell processors.
Patch disables Intel TSX on some Haswell processors to avoid TSX
on kernels that weren't updated with the latest microcode package
(which disables broken feature by default).

    * sysdeps/x86/cpu-features.c (get_common_indeces): Add
    stepping identification.
    (init_cpu_features): Add handle of Haswell.
2016-12-19 14:15:57 +03:00