Optimize strlen/strnlen/wcslen/wcsnlen with AVX2 to check 32 bytes with
a single vector compare instruction. It is as fast as SSE2 versions for
size <= 16 bytes and up to 1X faster for or size > 16 bytes on Haswell.
Select AVX2 version on AVX2 machines where vzeroupper is preferred and
AVX unaligned load is fast.
NB: It uses TZCNT instead of BSF since TZCNT produces the same result
as BSF for non-zero input. TZCNT is faster than BSF and is executed
as BSF if machine doesn't support TZCNT.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
strlen-sse2, strnlen-sse2, strlen-avx2, strnlen-avx2,
wcslen-sse2, wcslen-avx2 and wcsnlen-avx2.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list): Add tests for __strlen_avx2,
__strlen_sse2, __strnlen_avx2, __strnlen_sse2, __wcslen_avx2,
__wcslen_sse2 and __wcsnlen_avx2.
* sysdeps/x86_64/multiarch/strlen-avx2.S: New file.
* sysdeps/x86_64/multiarch/strlen-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/strlen.c: Likewise.
* sysdeps/x86_64/multiarch/strnlen-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/strnlen-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/strnlen.c: Likewise.
* sysdeps/x86_64/multiarch/wcslen-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/wcslen-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/wcslen.c: Likewise.
* sysdeps/x86_64/multiarch/wcsnlen-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/wcsnlen.c (OPTIMIZE (avx2)): New.
(IFUNC_SELECTOR): Return OPTIMIZE (avx2) on AVX2 machines where
vzeroupper is preferred and AVX unaligned load is fast.
SSE2 memchr is extended to support wmemchr. AVX2 memchr/rawmemchr/wmemchr
are added to search 32 bytes with a single vector compare instruction.
AVX2 memchr/rawmemchr/wmemchr are as fast as SSE2 memchr/rawmemchr/wmemchr
for small sizes and up to 1.5X faster for larger sizes on Haswell and
Skylake. Select AVX2 memchr/rawmemchr/wmemchr on AVX2 machines where
vzeroupper is preferred and AVX unaligned load is fast.
NB: It uses TZCNT instead of BSF since TZCNT produces the same result
as BSF for non-zero input. TZCNT is faster than BSF and is executed
as BSF if machine doesn't support TZCNT.
* sysdeps/x86_64/memchr.S (MEMCHR): New. Depending on if
USE_AS_WMEMCHR is defined.
(PCMPEQ): Likewise.
(memchr): Renamed to ...
(MEMCHR): This. Support wmemchr if USE_AS_WMEMCHR is defined.
Replace pcmpeqb with PCMPEQ.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
memchr-sse2, rawmemchr-sse2, memchr-avx2, rawmemchr-avx2,
wmemchr-sse4_1, wmemchr-avx2 and wmemchr-c.
* sysdeps/x86_64/multiarch/ifunc-avx2.h: New file.
* sysdeps/x86_64/multiarch/memchr-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/memchr-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/memchr.c: Likewise.
* sysdeps/x86_64/multiarch/rawmemchr-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/rawmemchr-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/rawmemchr.c: Likewise.
* sysdeps/x86_64/multiarch/wmemchr-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/wmemchr-sse2.S: Likewise.
* sysdeps/x86_64/multiarch/wmemchr.c: Likewise.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list): Test __memchr_avx2, __memchr_sse2,
__rawmemchr_avx2, __rawmemchr_sse2, __wmemchr_avx2 and
__wmemchr_sse2.
Previously, the implementation would conditionally exit based on the
status argument, which GCC did not know about. This leads to
false uninitialized variable warnings when data is accessed after a
TEST_VERIFY_EXIT failure (from code which would never execute).
1 << 31 is undefined, so replace it with a cleaner check. Also remove
magic numbers in comments.
* sysdeps/unix/sysv/linux/aarch64/dl-procinfo.h: Remove
mention of magic numbers in comments.
(_dl_procinfo): Fix undefined behavior
This patch corrects the conformtest expectations for sys/wait.h for
XPG4. This does not change the test results for this header at the
makefile level since it fixes some spurious failures for this header
while introducing other failures previously hidden by incorrect
expectations. However, it shows up issues with stdlib.h for XPG4
defining W* names that are not permitted in this case (previously
wrongly allowed; the W* reservation is UX-shaded in XPG4.2 /
XSI-shaded in POSIX before 2008); that test is thus XFAILed.
Tested for x86_64.
* conform/data/sys/wait.h-data (WIFCONTINUED): Do not expect for
[XPG4].
(WEXITED): Likewise.
(WSTOPPED): Likewise.
(WCONTINUED): Likewise.
(WNOHANG): Likewise.
(WNOWAIT): Likewise.
(idtype_t): Likewise.
(P_ALL): Likewise.
(P_PID): Likewise.
(P_PGID): Likewise.
(id_t): Likewise.
(siginfo_t): Likewise.
(wait3): Likewise.
(waitid): Likewise.
(signal.h): Do not allow header for [XPG4].
(sys/resource.h): Likewise.
(si_*): Do not allow for [XPG4].
(W*): Likewise.
(P_*): Likewise.
(BUS_): Likewise.
(CLD_): Likewise.
(FPE_): Likewise.
(ILL_): Likewise.
(POLL_): Likewise.
(SEGV_): Likewise.
(SI_): Likewise.
(TRAP_): Likewise.
* conform/Makefile (test-xfail-XPG4/stdlib.h/conform): New
variable.
There are 2 minimal strtoul implementations in ld.so:
1. __strtoul_internal in elf/dl-minimal.c.
2. tunables_strtoul in elf/dl-tunables.c.
This patch adds _dl_strtoul to replace them. Tested builds with and
without --enable-tunables.
[BZ #21528]
* elf/dl-minimal.c (__strtoul_internal): Removed.
(strtoul): Likewise.
* elf/dl-misc.c (_dl_strtoul): New function.
* elf/dl-tunables.c (tunables_strtoul): Removed.
(tunable_initialize): Replace tunables_strtoul with _dl_strtoul.
* elf/rtld.c (process_envvars): Likewise.
* sysdeps/unix/sysv/linux/dl-librecon.h (_dl_osversion_init):
Likewise.
* sysdeps/generic/ldsodefs.h (_dl_strtoul): New prototype.
wint_t is a little finicky because it might be defined by stddef.h, which
belongs to the compiler.
In addition to the _types_, a bunch of other declarations shared between
wctype.h and wchar.h are factored out to their own header.
* libio/bits/types/FILE.h, libio/bits/types/__FILE.h
* wcsmbs/bits/types/mbstate_t.h, wcsmbs/bits/types/__mbstate_t.h
* wcsmbs/bits/types/wint_t.h: New single-type definition files.
* wctype/bits/wctype-wchar.h: New file holding declarations shared
between wctype.h and wchar.h.
* libio/Makefile, wcsmbs/Makefile, wctype/Makefile:
Install them.
* include/bits/types/FILE.h, include/bits/types/__FILE.h
* include/bits/types/mbstate_t.h, include/bits/types/__mbstate_t.h
* include/bits/types/wint_t.h, include/bits/wcsmbs-wchar.h:
New wrappers.
* include/stdio.h, include/wchar.h, include/wctype.h:
No need to handle __need macros.
* grp/grp.h, gshadow/gshadow.h, hurd/hurd.h, iconv/gconv.h
* libio/stdio.h, mach/mach.h, misc/mntent.h, pwd/pwd.h
* shadow/shadow.h, stdio-common/printf.h, wcsmbs/uchar.h
* wcsmbs/wchar.h, wctype/wctype.h
* sysdeps/generic/_G_config.h, sysdeps/unix/sysv/linux/_G_config.h
Use the new files instead of __need macros.
This patch adds tests for len == 0 and tests for positions close to the
beginning, which are equivalent to positions close to the end for memchr.
* string/test-memrchr.c (test_main): Add tests for len == 0
and tests for positions close to the beginning, which are
equivalent to positions close to the end for memchr.
Add strfromf128 to stdlib when _Float128 support is enabled.
* stdio-common/printf-parsemb.c (__parse_one_specmb): Initialize
spec->info.is_binary128 to zero.
* stdio-common/printf.h (printf_info): Add new member is_binary128
to indicate that the number being converted to string is compatible
with the IEC 60559 binary128 format.
* stdio-common/printf_fp.c (__printf_fp_l): Add code to deal with
_Float128 numbers.
* stdio-common/printf_fphex.c: Include ieee754_float128.h and
ldbl-128/printf_fphex_macros.h
(__printf_fphex): Add code to deal with _Float128 numbers.
* stdio-common/printf_size.c (__printf_size): Likewise.
* stdio-common/vfprintf.c (process_arg): Initialize member
info.is_binary128 to zero.
* stdlib/fpioconst.h (FLT128_MAX_10_EXP_LOG): New macro.
* stdlib/stdlib.h: Include bits/floatn.h for _Float128 support.
(strfromf128): New declaration.
* stdlib/strfrom-skeleton.c (STRFROM): Set member info.is_binary128
to one.
* sysdeps/ieee754/float128/Makefile: Add strfromf128.
* sysdeps/ieee754/float128/Versions: Likewise.
* sysdeps/ieee754/float128/strfromf128.c: New file.
This patch refactors the macro PRINT_FPHEX_LONG_DOUBLE from the file
sysdeps/ieee754/ldbl-128/printf_fphex.c into a function-like macro to
enable its use for both long double and _Float128, when they are
ABI-distinct.
* sysdeps/ieee754/ldbl-128/printf_fphex.c: Include
ldbl-128/printf_fphex_macros.h for the definition of PRINT_FPHEX.
(PRINT_FPHEX_LONG_DOUBLE): Define based on PRINT_FPHEX.
* sysdeps/ieee754/ldbl-128/printf_fphex_macros.h
(PRINT_FPHEX): New function-like macro that can be used for long
double, as well as for _Float128
In __printf_fp_l, __printf_fphex, and __printf_size the blocks of code that are
used to read a double or long double argument, check for special values and
convert to multiprecision are similar. When adding float128 support to libc,
more code would be duplicated to deal with the extra type. This patch moves
the repetitive code to a macro which is now used by double and long double and
will be used for float128 when support is added, thus avoiding more
duplication.
Tested for powerpc64le and s390x.
* stdio-common/printf_fp.c (PRINTF_FP_FETCH): New macro.
(__printf_fp_l): Use the new macro to avoid duplicating code.
* stdio-common/printf_fphex.c (PRINTF_FPHEX_FETCH): New macro.
(__printf_fphex): Use the new macro to avoid duplicating code.
* stdio-common/printf_size.c (PRINTF_SIZE_FETCH): New macro.
(__printf_size): Use the new macro to avoid duplicating code.
Reuse the code for __mpn_extract_long_double to implement
__mpn_extract_float128.
* include/gmp.h: Include bits/floatn.h
(__mpn_extract_float128): Declare when __HAVE_DISTINCT_FLOAT128 is 1.
* stdlib/gmp-impl.h: Also check if alloca is not defined before
including stack-alloc.h. It could have been defined by other header
which not necessarily defines HAVE_ALLOCA.
* sysdeps/ieee754/float128/Makefile: New file.
* sysdeps/ieee754/float128/float1282mpn.c: New file.
* sysdeps/ieee754/float128/float128_private.h: Include gmp.h before
redefining __mpn_extract_long_double to __mpn_extract_float128, then
redefine __mpn_extract_long_double to __mpn_extract_float128.
* sysdeps/ieee754/ldbl-128/ldbl2mpn.c: Replace long double with
_Float128 to allow float128_private.h overrides.
Update the power of ten tables used by the common implementation when long
double is not the most expressive real type.
* stdlib/fpioconst.h: Include bits/floatn.h.
(FPIOCONST_HAVE_EXTENDED_RANGE): New macro for testing how big the
power of ten table should be.
(FPIOCONST_POW10_ARRAY_SIZE): Use larger table if above is true.
* stdlib/fpioconst.c (__tens): Use FPIOCONST_HAVE_EXTENDED_RANGE
to include larger tables when _Float128 support is enabled.
(_fpioconst_pow10): Likewise.
Various include/bits/types/*.h files do
where the path specified is relative to the toplevel glibc source
directory.
That has the wrong number of ../ components to achieve the desired
effect; it actually searches relative to include/ for a file that does
not exist there, then goes on to search the #include <> paths
specified with -I, eventually finding the desired file via such a path
(e.g. sysdeps/nptl/) with the right number of directory components.
Before that it searches include/../.. because of the -Iinclude,
meaning that an appropriately named file outside the glibc source tree
can affect the build.
This patch changes all those files to do #include <path> without the
../../, as some such files already do.
Tested for x86_64.
* include/bits/types/clock_t.h: Use #include <path> instead of
#include "../../path".
* include/bits/types/clockid_t.h: Likewise.
* include/bits/types/struct_iovec.h: Likewise.
* include/bits/types/struct_itimerspec.h: Likewise.
* include/bits/types/struct_osockaddr.h: Likewise.
* include/bits/types/struct_sigstack.h: Likewise.
* include/bits/types/struct_timespec.h: Likewise.
* include/bits/types/struct_timeval.h: Likewise.
* include/bits/types/struct_tm.h: Likewise.
* include/bits/types/time_t.h: Likewise.
* include/bits/types/timer_t.h: Likewise.
I observed a failure of timezone/tst-timezone with the same symptoms
as reported in bug 14096.
I was unable to reproduce the failure on testing again. However, from
inspection, the cause is as follows. tst-timezone uses time zones
compiled from the northamerica file, but has no dependency on any such
zones, so with a parallel build it is possible that they do not get
compiled until after tst-timezone is run. This patch adds a
dependency on the compiled America/New_York zone (the one used as a
makefile target to cause such zones to be compiled, rather than one
which is actually used in that test) to fix the race.
Tested for x86_64.
[BZ #14096]
* timezone/Makefile ($(objpfx)tst-timezone.out): Depend on
America/New_York.
Since ifunc-sse4_1.h is included only by wcsnlen.c, we can fold it
into wcsnlen.c. No code changes in wcsnlen.o.
2017-06-07 H.J. Lu <hongjiu.lu@intel.com>
* sysdeps/x86_64/multiarch/ifunc-sse4_1.h: Removed and folded
into ...
* sysdeps/x86_64/multiarch/wcsnlen.c: Here. Don't include
ifunc-sse4_1.h.
`ptsname_r' is declared in stdlib.h to only accept a `nonnull'
second argument and therefore GCC may choose to make optimizations
based on the assumption that this argument is NULL. This means
that potentially, GCC can optimize away the NULL check at some
point in the future. Since this is a programming interface, we
might as well remove the NULL check ourselves.
This also warrants a change to the `ptsname_r' manual page that
must be submitted to the corresponding mailing list.
In addition, remove the NULL buffer test in login/tst-ptsname.c.
Add support for routines in dl-procinfo.h to show string versions of
HWCAP entries when a program is invoked with the LD_SHOW_AUXV
environment variable set and also to aid in path resolution for
ldconfig.
* sysdeps/unix/sysv/linux/aarch64/dl-procinfo.c
(_dl_aarch64_cap_flags): New array.
* sysdeps/unix/sysv/linux/aarch64/dl-procinfo.h
(_dl_hwcap_string, _dl_string_hwcap, _dl_procinfo): Implement
functions.
The LD_HWCAP_MASK environment variable was ignored in static binaries,
which is inconsistent with the behaviour of dynamically linked
binaries. This seems to have been because of the inability of
ld_hwcap_mask being read early enough to influence anything but now
that it is in tunables, the mask is usable in static binaries as well.
This feature is important for aarch64, which relies on HWCAP_CPUID
being masked out to disable multiarch. A sanity test on x86_64 shows
that there are no failures. Likewise for aarch64.
* elf/dl-hwcaps.h [HAVE_TUNABLES]: Always read hwcap_mask.
* sysdeps/sparc/sparc32/dl-machine.h [HAVE_TUNABLES]:
Likewise.
* sysdeps/x86/cpu-features.c (init_cpu_features): Always set
up hwcap and hwcap_mask.
Now that LD_HWCAP_MASK (or glibc.tune.hwcap_mask) is read early enough
to influence cpu feature check in aarch64, use it to influence
multiarch selection. Setting LD_HWCAP_MASK such that it clears
HWCAP_CPUID will now disable multiarch for the binary.
HWCAP_CPUID is also now set in HWCAP_IMPORTANT so that it is set by
default. With this patch, this feature is only usable with
dyanmically linked binaries because LD_HWCAP_MASK is not read for
static binaries. A future patch fixes that.
* sysdeps/unix/sysv/linux/aarch64/cpu-features.c
(init_cpu_features): Use glibc.tune.hwcap_mask.
* sysdeps/unix/sysv/linux/aarch64/dl-procinfo.h: New file.
Drop _dl_hwcap_mask when building with tunables. This completes the
transition of hwcap_mask reading from _dl_hwcap_mask to tunables.
* elf/dl-hwcaps.h: New file.
* elf/dl-hwcaps.c: Include it.
(_dl_important_hwcaps)[HAVE_TUNABLES]: Read and update
glibc.tune.hwcap_mask.
* elf/dl-cache.c: Include dl-hwcaps.h.
(_dl_load_cache_lookup)[HAVE_TUNABLES]: Read
glibc.tune.hwcap_mask.
* sysdeps/sparc/sparc32/dl-machine.h: Likewise.
* elf/dl-support.c (_dl_hwcap2)[HAVE_TUNABLES]: Drop
_dl_hwcap_mask.
* elf/rtld.c (rtld_global_ro)[HAVE_TUNABLES]: Drop
_dl_hwcap_mask.
(process_envvars)[HAVE_TUNABLES]: Likewise.
* sysdeps/generic/ldsodefs.h (rtld_global_ro)[HAVE_TUNABLES]:
Likewise.
* sysdeps/x86/cpu-features.c (init_cpu_features): Don't
initialize dl_hwcap_mask when tunables are enabled.
The dl-procinfo.h for linux/s390 and linux/i386 don't have include
guards, which causes them to fail since addition of LD_HWCAP_MASK to
tunables. Add _DL_I386_PROCINFO_H guard to avoid redefining
_dl_procinfo on multiple includes and also allow the subsequent
include of another dl-procinfo.h to work.
Verified with a build test on i686.
* sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Add include
guard.
* sysdeps/unix/sysv/linux/s390/dl-procinfo.h: Likewise.
Add LD_HWCAP_MASK to tunables in preparation of it being removed from
rtld.c. This allows us to read LD_HWCAP_MASK much earlier so that it
can influence IFUNC resolution in aarch64.
This patch does not actually do anything other than read the
LD_HWCAP_MASK variable and add the tunables way to set the
LD_HWCAP_MASK, i.e. via the glibc.tune.hwcap_mask tunable. In a
follow-up patch, the _dl_hwcap_mask will be replaced with
glibc.tune.hwcap_mask to complete the transition.
* elf/dl-tunables.list: Add glibc.tune.hwcap_mask.
* scripts/gen-tunables.awk: Include dl-procinfo.h.
* manual/tunables.texi: Document glibc.tune.hwcap_mask.
The TUNABLE_SET_VALUE and family of macros (and my later attempt to
add a TUNABLE_GET) never quite went together very well because the
overall interface was not clearly defined. This patch is an attempt
to do just that.
This patch consolidates the API to two simple sets of macros,
TUNABLE_GET* and TUNABLE_SET*. If TUNABLE_NAMESPACE is defined,
TUNABLE_GET takes just the tunable name, type and a (optionally NULL)
callback function to get the value of the tunable. The callback
function, if non-NULL, is called if the tunable was externally set
(i.e. via GLIBC_TUNABLES or any future mechanism). For example:
val = TUNABLE_GET (check, int32_t, check_callback)
returns the value of the glibc.malloc.check tunable (assuming
TUNABLE_NAMESPACE is set to malloc) as an int32_t into VAL after
calling check_callback.
Likewise, TUNABLE_SET can be used to set the value of the tunable,
although this is currently possible only in the dynamic linker before
it relocates itself. For example:
TUNABLE_SET (check, int32_t, 2)
will set glibc.malloc.check to 2. Of course, this is not possible
since we set (or read) glibc.malloc.check long after it is relocated.
To access or set a tunable outside of TUNABLE_NAMESPACE, use the
TUNABLE_GET_FULL and TUNABLE_SET_FULL macros, which have the following
prototype:
TUNABLE_GET_FULL (glibc, tune, hwcap_mask, uint64_t, NULL)
TUNABLE_SET_FULL (glibc, tune, hwcap_mask, uint64_t, 0xffff)
In future the tunable list may get split into mutable and immutable
tunables where mutable tunables can be modified by the library and
userspace after relocation as well and TUNABLE_SET will be more useful
than it currently is. However whenever we actually do that split, we
will have to ensure that the mutable tunables are protected with
locks.
* elf/Versions (__tunable_set_val): Rename to __tunable_get_val.
* elf/dl-tunables.c: Likewise.
(do_tunable_update_val): New function.
(__tunable_set_val): New function.
(__tunable_get_val): Call CB only if the tunable was externally
initialized.
(tunables_strtoul): Replace strval with initialized.
* elf/dl-tunables.h (strval): Replace with a bool initialized.
(TUNABLE_ENUM_NAME, TUNABLE_ENUM_NAME1): Adjust names to
prevent collision.
(__tunable_set_val): New function.
(TUNABLE_GET, TUNABLE_GET_FULL): New macros.
(TUNABLE_SET, TUNABLE_SET_FULL): Likewise.
(TUNABLE_SET_VAL): Remove.
(TUNABLE_SET_VAL_WITH_CALLBACK): Likewise.
* README.tunables: Document the new macros.
* malloc/arena.c (ptmalloc_init): Adjust.
Since wcsnlen.S uses pminud which is the part of SSE4.1, move wcsnlen.S
to multiarch/wcsnlen-sse4_1.S.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
wcsnlen-sse4_1 and wcsnlen-c.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list): Test __wcsnlen_sse4_1 and
__wcsnlen_sse2.
* sysdeps/x86_64/multiarch/ifunc-sse4_1.h: New file.
* sysdeps/x86_64/multiarch/wcsnlen-c.c: Likewise.
* sysdeps/x86_64/multiarch/wcsnlen-sse4_1.S: Likewise.
* sysdeps/x86_64/multiarch/wcsnlen.c: Likewise.
* sysdeps/x86_64/wcsnlen.S: Removed.
This patch removes the s390 specific implementation of spinlock code
and is now using the generic one.
ChangeLog:
* sysdeps/s390/nptl/pthread_spin_init.c: Delete File.
* sysdeps/s390/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/s390/nptl/pthread_spin_trylock.c: Likewise.
* sysdeps/s390/nptl/pthread_spin_unlock.c: Likewise.
This patch optimizes the generic spinlock code.
The type pthread_spinlock_t is a typedef to volatile int on all archs.
Passing a volatile pointer to the atomic macros which are not mapped to the
C11 atomic builtins can lead to extra stores and loads to stack if such
a macro creates a temporary variable by using "__typeof (*(mem)) tmp;".
Thus, those macros which are used by spinlock code - atomic_exchange_acquire,
atomic_load_relaxed, atomic_compare_exchange_weak - have to be adjusted.
According to the comment from Szabolcs Nagy, the type of a cast expression is
unqualified (see http://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_423.htm):
__typeof ((__typeof (*(mem)) *(mem)) tmp;
Thus from spinlock perspective the variable tmp is of type int instead of
type volatile int. This patch adjusts those macros in include/atomic.h.
With this construct GCC >= 5 omits the extra stores and loads.
The atomic macros are replaced by the C11 like atomic macros and thus
the code is aligned to it. The pthread_spin_unlock implementation is now
using release memory order instead of sequentially consistent memory order.
The issue with passed volatile int pointers applies to the C11 like atomic
macros as well as the ones used before.
I've added a glibc_likely hint to the first atomic exchange in
pthread_spin_lock in order to return immediately to the caller if the lock is
free. Without the hint, there is an additional jump if the lock is free.
I've added the atomic_spin_nop macro within the loop of plain reads.
The plain reads are also realized by C11 like atomic_load_relaxed macro.
The new define ATOMIC_EXCHANGE_USES_CAS determines if the first try to acquire
the spinlock in pthread_spin_lock or pthread_spin_trylock is an exchange
or a CAS. This is defined in atomic-machine.h for all architectures.
The define SPIN_LOCK_READS_BETWEEN_CMPXCHG is now removed.
There is no technical reason for throwing in a CAS every now and then,
and so far we have no evidence that it can improve performance.
If that would be the case, we have to adjust other spin-waiting loops
elsewhere, too! Using a CAS loop without plain reads is not a good idea
on many targets and wasn't used by one. Thus there is now no option to
do so.
Architectures are now using the generic spinlock automatically if they
do not provide an own implementation. Thus the pthread_spin_lock.c files
in sysdeps folder are deleted.
ChangeLog:
* NEWS: Mention new spinlock implementation.
* include/atomic.h:
(__atomic_val_bysize): Cast type to omit volatile qualifier.
(atomic_exchange_acq): Likewise.
(atomic_load_relaxed): Likewise.
(ATOMIC_EXCHANGE_USES_CAS): Check definition.
* nptl/pthread_spin_init.c (pthread_spin_init):
Use atomic_store_relaxed.
* nptl/pthread_spin_lock.c (pthread_spin_lock):
Use C11-like atomic macros.
* nptl/pthread_spin_trylock.c (pthread_spin_trylock):
Likewise.
* nptl/pthread_spin_unlock.c (pthread_spin_unlock):
Use atomic_store_release.
* sysdeps/aarch64/nptl/pthread_spin_lock.c: Delete File.
* sysdeps/arm/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/hppa/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/m68k/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/microblaze/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/mips/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/nios2/nptl/pthread_spin_lock.c: Likewise.
* sysdeps/aarch64/atomic-machine.h (ATOMIC_EXCHANGE_USES_CAS): Define.
* sysdeps/alpha/atomic-machine.h: Likewise.
* sysdeps/arm/atomic-machine.h: Likewise.
* sysdeps/i386/atomic-machine.h: Likewise.
* sysdeps/ia64/atomic-machine.h: Likewise.
* sysdeps/m68k/coldfire/atomic-machine.h: Likewise.
* sysdeps/m68k/m680x0/m68020/atomic-machine.h: Likewise.
* sysdeps/microblaze/atomic-machine.h: Likewise.
* sysdeps/mips/atomic-machine.h: Likewise.
* sysdeps/powerpc/powerpc32/atomic-machine.h: Likewise.
* sysdeps/powerpc/powerpc64/atomic-machine.h: Likewise.
* sysdeps/s390/atomic-machine.h: Likewise.
* sysdeps/sparc/sparc32/atomic-machine.h: Likewise.
* sysdeps/sparc/sparc32/sparcv9/atomic-machine.h: Likewise.
* sysdeps/sparc/sparc64/atomic-machine.h: Likewise.
* sysdeps/tile/tilegx/atomic-machine.h: Likewise.
* sysdeps/tile/tilepro/atomic-machine.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/atomic-machine.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/coldfire/atomic-machine.h: Likewise.
* sysdeps/unix/sysv/linux/nios2/atomic-machine.h: Likewise.
* sysdeps/unix/sysv/linux/sh/atomic-machine.h: Likewise.
* sysdeps/x86_64/atomic-machine.h: Likewise.
Since cpu_features is available, use it instead of dl_x86_cpu_features.
* sysdeps/x86/cacheinfo.c (intel_check_word): Accept cpu_features
and use it instead of dl_x86_cpu_features.
(handle_intel): Replace maxidx with cpu_features. Pass
cpu_features to intel_check_word.
(__cache_sysconf): Pass cpu_features to handle_intel.
(init_cacheinfo): Likewise. Use cpu_features instead of
dl_x86_cpu_features.
Optimize x86-64 memcmp/wmemcmp with AVX2. It uses vector compare as
much as possible. It is as fast as SSE4 memcmp for size <= 16 bytes
and up to 2X faster for size > 16 bytes on Haswell and Skylake. Select
AVX2 memcmp/wmemcmp on AVX2 machines where vzeroupper is preferred and
AVX unaligned load is fast.
NB: It uses TZCNT instead of BSF since TZCNT produces the same result
as BSF for non-zero input. TZCNT is faster than BSF and is executed
as BSF if machine doesn't support TZCNT.
Key features:
1. For size from 2 to 7 bytes, load as big endian with movbe and bswap
to avoid branches.
2. Use overlapping compare to avoid branch.
3. Use vector compare when size >= 4 bytes for memcmp or size >= 8
bytes for wmemcmp.
4. If size is 8 * VEC_SIZE or less, unroll the loop.
5. Compare 4 * VEC_SIZE at a time with the aligned first memory area.
6. Use 2 vector compares when size is 2 * VEC_SIZE or less.
7. Use 4 vector compares when size is 4 * VEC_SIZE or less.
8. Use 8 vector compares when size is 8 * VEC_SIZE or less.
* sysdeps/x86/cpu-features.h (index_cpu_MOVBE): New.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
memcmp-avx2 and wmemcmp-avx2.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list): Test __memcmp_avx2 and __wmemcmp_avx2.
* sysdeps/x86_64/multiarch/memcmp-avx2.S: New file.
* sysdeps/x86_64/multiarch/wmemcmp-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/memcmp.S: Use __memcmp_avx2 on AVX
2 machines if AVX unaligned load is fast and vzeroupper is
preferred.
* sysdeps/x86_64/multiarch/wmemcmp.S: Use __wmemcmp_avx2 on AVX
2 machines if AVX unaligned load is fast and vzeroupper is
preferred.
The difference between strlen and wcslen is byte vs int. We can
replace pminub and pcmpeqb with pminud and pcmpeqd to turn strlen
into wcslen.
* sysdeps/x86_64/strlen.S (PMINU): New.
(PCMPEQ): Likewise.
(SHIFT_RETURN): Likewise.
(FIND_ZERO): Replace pcmpeqb with PCMPEQ.
(strlen): Add SHIFT_RETURN before ret. Replace pcmpeqb and
pminub with PCMPEQ and PMINU.
* sysdeps/x86_64/wcsnlen.S: New file.
By x86-64 specification, 32-bit destination registers are zero-extended
to 64 bits. There is no need to use 64-bit registers when only the lower
32 bits are non-zero. Also 2 instructions in:
mov %rdi, %rcx
and $15, %rcx
jz L(length_less16_offset0)
mov %rdi, %rcx <<< redundant
and $15, %rcx <<< redundant
are redundant.
* sysdeps/x86_64/memrchr.S (__memrchr): Use 32-bit registers for
the lower 32 bits. Remove redundant instructions.
The kernel interface for p{readv,writev}{64}v is
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
unsigned long pos_l, unsigned long pos_h)
Except for targets which define __ARCH_WANT_COMPAT_SYS_PREADV64 and
__ARCH_WANT_COMPAT_SYS_PWRITEV64,
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
off64_t pos)
is used for p{readv,writev}{64}v. X32 is the only such target. The
LO_HI_LONG macro is used to pass offset to the pos_l and pos_h pair.
Since pos_h is ignored when size of offset == sizeof of pos_l, x86-64
has
#define LO_HI_LONG(val) (val)
But the kernel interface for p{readv,writev}{64}v2 is
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
unsigned long pos_l, unsigned long pos_h, int flags)
Except for targets which define __ARCH_WANT_COMPAT_SYS_PREADV64V2 and
__ARCH_WANT_COMPAT_SYS_PWRITEV64V2,
(unsigned long fd, {const }struct iovec *iov, unsigned long vlen,
off64_t pos, int flags)
is used for p{readv,writev}{64}v2. X32 is the only such target. Update
x86-64 LO_HI_LONG to pass 0 as the high part of the offset argument for
p{readv,writev}{64}v2 and define a different LO_HI_LONG for x32 to only
pass one argument for offset.
Tested on x32 and x86-64.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h (LO_HI_LONG): Pass
0 as the high part of offset.
* sysdeps/unix/sysv/linux/x86_64/x32/sysdep.h (LO_HI_LONG): New.
Various bits/signum.h headers define SIG_HOLD if __USE_UNIX98. That
should be __USE_XOPEN, as this macro is in XPG4. This patch fixes the
conditionals accordingly. Because of other header bugs, this does not
allow any XFAILs to be removed (however, the XPG4/signal.h/conform
XFAIL only depends on a few such straightforward header bugs, not on
the more complicated to fix ucontext_t issues, as ucontext_t isn't
included in signal.h in XPG4).
Tested for x86_64.
[BZ #21538]
* bits/signum.h (SIG_HOLD): Define if [__USE_XOPEN], not
[__USE_UNIX98].
* sysdeps/unix/bsd/bits/signum.h (SIG_HOLD): Likewise.
* sysdeps/unix/sysv/linux/alpha/bits/signum.h (SIG_HOLD):
Likewise.
* sysdeps/unix/sysv/linux/bits/signum.h (SIG_HOLD): Likewise.
* sysdeps/unix/sysv/linux/hppa/bits/signum.h (SIG_HOLD): Likewise.
* sysdeps/unix/sysv/linux/mips/bits/signum.h (SIG_HOLD): Likewise.
* sysdeps/unix/sysv/linux/sparc/bits/signum.h (SIG_HOLD):
Likewise.
glibc defines the stack_t type with the tag struct sigaltstack. This
is not permitted by POSIX; sigaltstack is only reserved with file
scope in the namespace of ordinary identifiers, not the tag namespace,
and in the case where stack_t is obtained from ucontext.h rather than
signal.h, it's not reserved with file scope at all.
This patch removes the tag accordingly and updates uses in glibc of
struct sigaltstack. This is similar to the removal of the "struct
siginfo" tag a few years ago: C++ name mangling changes are an
unavoidable consequence. A NEWS item is added to note the changed
mangling. There is inevitably some risk of breaking builds of
anything that relies on the struct sigaltstack name (though the first
few hits I looked at from codesearch.debian.net generally seemed to
involve code that could use the stack_t name conditionally, so
depending on how they determine the conditionals they may work with
glibc not defining the struct tag anyway).
Tested for x86_64 and x86, and with build-many-glibcs.py.
[BZ #21517]
* bits/types/stack_t.h (stack_t): Remove struct tag.
* sysdeps/unix/sysv/linux/bits/types/stack_t.h (stack_t):
Likewise.
* sysdeps/unix/sysv/linux/mips/bits/types/stack_t.h (stack_t):
Likewise.
* debug/segfault.c (install_handler): Use stack_t instead of
struct sigaltstack.
* hurd/hurd/signal.h (struct hurd_sigstate): Likewise.
* hurd/trampoline.c (_hurd_setup_sighandler): Likewise.
* include/signal.h (__sigaltstack): Likwise.
* signal/sigaltstack.c (__sigaltstack): Likewise.
* signal/signal.h (sigaltstack): Likewise.
* sysdeps/mach/hurd/i386/signal-defines.sym
(SIGALTSTACK__SS_SP__OFFSET): Likewise.
(SIGALTSTACK__SS_SIZE__OFFSET): Likewise.
(SIGALTSTACK__SS_FLAGS__OFFSET): Likewise.
* sysdeps/mach/hurd/sigaltstack.c (__sigaltstack): Likewise.
* sysdeps/mach/hurd/sigstack.c (sigstack): Likewise.
* sysdeps/unix/sysv/linux/alpha/sys/procfs.h (struct
elf_prstatus): Likewise.
* sysdeps/unix/sysv/linux/hppa/____longjmp_chk.c (CHECK_SP):
Likewise.
* sysdeps/unix/sysv/linux/ia64/sys/procfs.h (struct elf_prstatus):
Likewise.
* sysdeps/unix/sysv/linux/m68k/____longjmp_chk.c (CHECK_SP):
Likewise.
* sysdeps/unix/sysv/linux/powerpc/sys/procfs.h (struct
elf_prstatus): Likewise.
* sysdeps/unix/sysv/linux/sh/sys/procfs.h (struct elf_prstatus):
Likewise.
* sysdeps/unix/sysv/linux/sys/procfs.h (struct elf_prstatus):
Likewise.
bench-memchr.c is shared with bench-memrchr.c. This patch adds some
tests for positions close to the beginning for memrchr, which are
equivalent to positions close to the end for memchr.
* benchtests/bench-memchr.c (do_test): Print out both length
and position.
(test_main): Also test the position close to the beginning for
memrchr.
__x86_shared_non_temporal_threshold was set to 6 times of per-core
shared cache size, based on the large memcpy micro benchmark in glibc
on a 8-core processor. For a processor with more than 8 cores, the
threshold is too low. Set __x86_shared_non_temporal_threshold to the
3/4 of the total shared cache size so that it is unchanged on 8-core
processors. On processors with less than 8 cores, the threshold is
lower.
* sysdeps/x86/cacheinfo.c (__x86_shared_non_temporal_threshold):
Set to the 3/4 of the total shared cache size.
The @errno macro is extended to render the canonical error string in
every documented errno error. Redundant entries and "???" are
removed. Sixty-six errors now at least contain the error string as
the description, where no description (or "???") existed before.
* manual/errno.texi: Remove redundant error strings.
* manual/macros.texi (@errno): Render the error string in
every description.
signal.h defines the sigevent structure and constants if defined
__USE_POSIX199309 || defined __USE_XOPEN_EXTENDED. The
__USE_XOPEN_EXTENDED condition is incorrect; this structure does not
appear in XSI standards before Unix98 (which implies
__USE_POSIX199309). This patch fixes the conditionals accordingly
(the existing conditional is correct for siginfo_t, just not for
sigevent).
Tested for x86_64. This does not allow any conform/ XFAILs to be
removed because of other signal.h namespace bugs.
[BZ #21543]
* signal/signal.h: Only include <bits/types/sigevent_t.h> and
<bits/sigevent-consts.h> if [__USE_POSIX199309], not if
[__USE_XOPEN_EXTENDED].
This patch adds the missing compiler flags for correct pthread cancellation
on some architectures for the p{read,write}v2 implementation (52bd938169).
Checked on x86_64-linux-gnu and powerpc64le-linux-gnu.
* misc/Makefile (CFLAGS-preadv2.c): New rule.
(CFLAGS-preadv64v2.c): Likewise.
(CFLAGS-pwritev2.c): Likewise.
(CFLAGS-pwritev64v2.c): Likewise.
A further simplification could eliminate the canon variable in
gaih_inet and replace it with canonbuf. However, canonbuf is
used as a flag in the nscd code, which makes this somewhat
non-straightforward.
