This patch makes files using __ASSUME_* macros include
<kernel-features.h> explicitly, rather than relying on some other
header (such as tls.h, lowlevellock.h or pthreadP.h) to include it
implicitly. (I omitted cases where I've already posted or am testing
the patch that stops the file from needing __ASSUME_* at all.) This
accords with the general principle of making source files include the
headers for anything they use, and also helps make it safe to remove
<kernel-features.h> includes from any file that doesn't use
__ASSUME_* (some of those may be stray includes left behind after
increasing the minimum kernel version, others may never have been
needed or may have become obsolete after some other change).
Tested x86_64 that the disassembly of installed shared libraries is
unchanged by this patch.
* nptl/pthread_cond_wait.c: Include <kernel-features.h>.
* nptl/pthread_rwlock_timedrdlock.c: Likewise.
* nptl/pthread_rwlock_timedwrlock.c: Likewise.
* nptl/sysdeps/unix/sysv/linux/lowlevelrobustlock.c: Likewise.
* nscd/nscd.c: Likewise.
* sysdeps/i386/nptl/tcb-offsets.sym: Likewise.
* sysdeps/powerpc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sh/nptl/tcb-offsets.sym: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym: Likewise.
This patch removes conditionals on __ASSUME_SOCK_CLOEXEC, and on
SOCK_CLOEXEC being defined, in Linux-specific code, now that all
supported Linux kernel versions can be assumed to have this
functionality. (The macro is also used in OS-independent code and is
not defined for Hurd.)
Tested x86_64 that the disassembly of installed shared libraries is
unchanged by this patch.
* nptl/sysdeps/unix/sysv/linux/mq_notify.c: Do not include
<kernel-features.h>.
(init_mq_netlink): Remove conditional have_sock_cloexec
definitions. Remove code conditional on have_sock_cloexec < 0.
(init_mq_netlink) [!SOCK_CLOEXEC]: Remove conditional code.
(init_mq_netlink) [!__ASSUME_SOCK_CLOEXEC]: Likewise.
* sysdeps/unix/sysv/linux/opensock.c: Do not include
<kernel-features.h>.
(__opensock) [SOCK_CLOEXEC]: Make code unconditional.
(__opensock) [!__ASSUME_SOCK_CLOEXEC]: Remove conditional code.
This patch adds ifunc tests for x86_64 memset_chk and memset. It also
defines HAS_AVX2 with AVX2_Usable since AVX2 may not be usable even if
processor has AVX2.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list):
Add tests for memset_chk and memset.
* sysdeps/x86_64/multiarch/init-arch.h (HAS_AVX2): Defined
with AVX2_Usable.
This fixes an off-by-one argc interpretation error (argv[0] is included
in the count) in stdlib/tst-qsort2.c causing a segfault if the program
is called with one command-line argument only (argv[2] is NULL in that
case).
This patch removes __ASSUME_F_GETOWN_EX now it can be assumed to be
true unconditionally.
Tested x86_64 that disassembly of installed shared libraries is
unchanged by this patch.
* sysdeps/unix/sysv/linux/kernel-features.h
(__ASSUME_F_GETOWN_EX): Remove macro.
* sysdeps/unix/sysv/linux/fcntl.c: Do not include
<kernel-features.h>.
(miss_F_GETOWN_EX): Remove variable or macro.
(do_fcntl): Do not check miss_F_GETOWN_EX.
(do_fcntl) [!__ASSUME_F_GETOWN_EX]: Remove conditional code.
This patch removes __ASSUME_AT_RANDOM now it can be assumed to be true
unconditionally.
Tested x86_64 that the disassembly of installed shared libraries is
unchanged by this patch.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_AT_RANDOM):
Remove macro.
* sysdeps/unix/sysv/linux/dl-osinfo.h (_dl_setup_stack_chk_guard)
[!__ASSUME_AT_RANDOM]: Remove conditional code.
(_dl_setup_pointer_guard) [!__ASSUME_AT_RANDOM]: Likewise.
This patch removes the __ASSUME_ADJ_OFFSET_SS_READ macro (and
conditionals on whether ADJ_OFFSET_SS_READ is defined), now it can be
unconditionally assumed to be true and ADJ_OFFSET_SS_READ can be
assumed to be defined.
Tested x86_64 that the disassembly of installed shared libraries is
unchanged by this patch.
* sysdeps/unix/sysv/linux/kernel-features.h
(__ASSUME_ADJ_OFFSET_SS_READ): Remove macro.
* sysdeps/unix/sysv/linux/adjtime.c (ADJTIME)
[ADJ_OFFSET_SS_READ]: Make code unconditional.
(ADJTIME) [!ADJ_OFFSET_SS_READ]: Remove conditional code.
This fixes the calculation of R_ARM_TLS_DESC relocations for lazy global
symbol references, i.e. created with `-z lazy' in effect with the static
linker, where immediate resolution is requested with LD_BIND_NOW.
This patch cleans up for __ASSUME_ATFCTS now always being true for the
supported Linux kernel versions by removing conditional code in
sysdeps/unix/sysv/linux. Several fchownat.c files that were only
present because of differences in the fallback syscalls used
(depending on the architecture-specific names of chown-related
syscalls for 32-bit uids) are removed. Files that looks like they
could be replaced by syscalls.list entries have the standard "Consider
moving to syscalls.list." comment (see bug 14138) added. Conditionals
on the relevant __NR_* syscall numbers being defined are also removed,
since my analysis indicated that the relevant syscalls are always
defined for all relevant kernel versions using any affected file.
Much of the removed fallback code had unbounded stack allocations, so
this reduces the number of cases to consider for anyone reviewing uses
of alloca and VLAs in glibc.
There remain tests of __ASSUME_ATFCTS in io/openat.c (to determine
whether to define __have_atfcts) and sysdeps/posix/getcwd.c (which
also uses __have_atfcts); thus, the definition of __ASSUME_ATFCTS
remains in kernel-features.h. The logical condition relevant there is
whether openat64_not_cancel_3 is known to work. Hurd doesn't use this
version of getcwd at all, so the conditionals in getcwd.c are always
true in glibc. However, this code is also used in gnulib. So the
best way to deal with the conditionals there may be for gnulib people
to deal with merging all relevant changes in both directions between
the glibc and gnulib versions of this file, at the end of which the
openat conditionals should be in whatever form is best for gnulib, and
hardcoded in the _LIBC case to having openat supported.
Tested by comparing before-and-after disassembly of installed
(stripped) shared libraries, on x86_64 and x86. On x86 the patch made
no change to the disassembly; on x86_64, the only changes were in
readlinkat, where formerly the return value from the readlinkat
syscall was stored in an int variable before being converted to
ssize_t for the return, and now the return value is returned directly
without truncation to int. I think it's clearly correct not to
truncate the return value (although I also think the truncation would
not have been a user-visible bug because the kernel would never have
returned a value it could have affected).
* include/fcntl.h (__atfct_seterrno): Remove prototype.
(__atfct_seterrno_2): Likewise.
* sysdeps/unix/sysv/linux/alpha/dl-fxstatat64.c: Do not include
<kernel-features.h>.
(__ASSUME_ATFCTS): Do not undefine and redefine.
* sysdeps/unix/sysv/linux/alpha/fxstatat.c [__ASSUME_ATFCTS]
(__have_atfcts): Remove conditional definition.
(__fxstatat([__NR_fstatat64]: Make code unconditional.
(__fxstatat) [!__ASSUME_ATFCTS]: Remove conditional code and code
unreachable if [__ASSUME_ATFCTS].
* sysdeps/unix/sysv/linux/dl-fxstatat64.c (__ASSUME_ATFCTS): Do
not undefine and redefine.
* sysdeps/unix/sysv/linux/faccessat.c: Do not include
<kernel-features.h>.
(faccessat) [__NR_faccessat]: Make code unconditional.
(faccessat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/fchmodat.c: Do not include
<kernel-features.h>.
(fchmodat) [__NR_fchmodat]: Make code unconditional.
(fchmodat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/fchownat.c: Do not include
<kernel-features.h>.
(fchownat) [__NR_fchownat]: Make code unconditional.
(fchownat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/futimesat.c: Do not include
<kernel-features.h>.
(futimesat) [__NR_futimesat]: Make code unconditional.
(futimesat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/fxstatat.c: Do not include
<kernel-features.h>.
(__fxstatat) [__NR_newfstatat]: Make code unconditional.
(__fxstatat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/fxstatat64.c: Do not include
<kernel-features.h>.
(__fxstatat64) [__NR_fstatat64]: Make code unconditional.
(__fxstatat64) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/i386/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/i386/fxstatat.c: Do not include
<kernel-features.h>.
(__fxstatat) [__NR_fstatat64]: Make code unconditional.
(__fxstatat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/linkat.c: Do not include
<kernel-features.h>.
(linkat) [__NR_linkat]: Make code unconditional.
(linkat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/m68k/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/mips/mips64/fxstatat64.c: Do not include
<kernel-features.h>.
(__fxstatat64) [__NR_newfstatat]: Make code unconditional.
(__fxstatat64) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/mkdirat.c: Do not include
<kernel-features.h>.
(mkdirat) [__NR_mkdirat]: Make code unconditional.
(mkdirat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/openat.c: Do not include
<kernel-features.h>.
[!__ASSUME_ATFCTS] (__atfct_seterrno): Remove function.
[!__ASSUME_ATFCTS] (__have_atfcts): Remove variable.
(OPENAT_NOT_CANCEL) [__NR_openat]: Make code unconditional.
(OPENAT_NOT_CANCEL) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/powerpc/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/readlinkat.c: Do not include
<kernel-features.h>.
(readlinkat) [__NR_readlinkat]: Make code unconditional.
(readlinkat) [!__ASSUME_ATFCTS]: Remove conditional code. Return
result of INLINE_SYSCALL directly, not via int variable.
* sysdeps/unix/sysv/linux/renameat.c: Do not include
<kernel-features.h>.
[!__ASSUME_ATFCTS] (__atfct_seterrno_2): Remove function.
(renameat) [__NR_renameat]: Make code unconditional.
(renameat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/s390/s390-32/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/sh/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/sparc/sparc32/fchownat.c: Remove file.
* sysdeps/unix/sysv/linux/sparc/sparc64/dl-fxstatat64.c
(__ASSUME_ATFCTS): Do not undefine and redefine.
* sysdeps/unix/sysv/linux/symlinkat.c: Do not include
<kernel-features.h>.
(symlinkat) [__NR_symlinkat]: Make code unconditional.
(symlinkat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/unlinkat.c: Do not include
<kernel-features.h>.
(unlinkat) [__NR_unlinkat]: Make code unconditional.
(unlinkat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/wordsize-64/dl-fxstatat64.c
(__ASSUME_ATFCTS): Do not undefine and redefine.
* sysdeps/unix/sysv/linux/wordsize-64/fxstatat.c: Do not include
<kernel-features.h>.
(__fxstatat) [__NR_newfstatat]: Make code unconditional.
(__fxstatat) [!__ASSUME_ATFCTS]: Remove conditional code.
* sysdeps/unix/sysv/linux/xmknodat.c: Do not include
<kernel-features.h>.
(__xmknodat) [__NR_mknodat]: Make code unconditional.
(__xmknodat) [!__ASSUME_ATFCTS]: Remove conditional code.
Errno is not set and the testcases will fail.
Now the scalbln-aliases are removed in i386/m68
and the wrappers are used when calling the scalbln-functions.
On ia64 only scalblnf has its own implementation.
For scalbln and scalblnl the ieee754/dbl-64 and ieee754/ldbl-96 are used, thus
the wrappers are needed, too.
Since long is 4 bytes for x32, we should use 3 bytes for __pad1 when
a long __pad1 is replaced by a byte __rwelision and __pad1.
* sysdeps/x86/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Use
3 bytes for __pad1 for x32.
(__PTHREAD_RWLOCK_ELISION_EXTRA): Likewise.
In this patch we take advantage of HSW memory bandwidth, manage to
reduce miss branch prediction by avoiding using branch instructions and
force destination to be aligned with avx & avx2 instruction.
The CPU2006 403.gcc benchmark indicates this patch improves performance
from 26% to 59%.
* sysdeps/x86_64/multiarch/Makefile: Add memset-avx2.
* sysdeps/x86_64/multiarch/memset-avx2.S: New file.
* sysdeps/x86_64/multiarch/memset.S: Likewise.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
* sysdeps/x86_64/multiarch/rtld-memset.S: Likewise.
Due to my bad review suggestion for the fix for BZ #15089 a check
was removed from systrim to prevent sbrk being called with a zero
argument. Add the check back to avoid this useless work.
ChangeLog:
2014-06-19 Will Newton <will.newton@linaro.org>
* malloc/malloc.c (systrim): If extra is zero then return
early.
Implementation of strchr for AArch64. Speedups taken from micro-bench
show the improvements relative to the standard C code.
The use of LD1 means we have identical code for both big- and
little-endian systems.
This patch fixes __ieee754_logl (-LDBL_MAX) on x86_64 and x86 not to
subtract 1 from its argument and so cause spurious overflow in
FE_DOWNWARD mode. (For any argument strictly less than -1, it doesn't
matter whether or not 1 is subtracted before computing log1p, as long
as the result doesn't overflow to -Inf.)
Tested x86_64 and x86. (This particular case lacks test coverage,
since the testsuite doesn't cover -lieee, but it will be covered by
tests after the following patch to test pow in all rounding modes,
which was the context in which this bug was found.)
[BZ #17022]
* sysdeps/i386/fpu/e_logl.S (__ieee754_logl): Do not subtract 1
from arguments -2 or below.
* sysdeps/i386/i686/fpu/e_logl.S (__ieee754_logl): Likewise.
* sysdeps/x86_64/fpu/e_logl.S (__ieee754_logl): Likewise.
The glibc makefiles have a standard variable, $(rtld-prefix), to run
the dynamic linker with a default --library-path option; this is used
as the basis of lots of other variables for running programs compiled
with the newly built library.
A few places however use $(elf-objpfx)ld.so or
$(elf-objpfx)${rtld-installed-name} directly, with such a
--library-path option. This patch makes such places use
$(rtld-prefix) instead. I'm not aware of any significance in these
cases to the choice of ld.so or ${rtld-installed-name} when running
the dynamic linker, or to whether $(patsubst
%,:%,$(sysdep-library-path)) is included in the library-path as it is
in $(rtld-prefix) and just one of the places being changed.
Tested x86_64.
* elf/Makefile ($(objpfx)tst-unused-dep.out): Use $(rtld-prefix).
* iconvdata/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Likewise.
* sysdeps/s390/s390-64/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Likewise.
localedata/ChangeLog:
* Makefile (LOCALEDEF): Use $(rtld-prefix).
This patch fixes few failures in nearbyintl() where the fraction part is
close to 0.5.i The new tests added report few extra failures in
nearbyint_downward and nearbyint_towardzero which is a known issue.
Fixes#17031.
Add the missing fallback file for elide.h to fix non x86 builds.
Sorry about that. This is just a noop macro file that makes
all elision code to be optimized out.
With the recent tuning the C version of rwlocks is basically the same
performance as the x86 assembler version for uncontended locks (with a
a few cycles near the run-to-run variability). For others it should not
matter anyways.
So remove the assembler code and use the C version like other
architectures.
This patch relies on the C version of the rwlocks posted earlier.
With C rwlocks it is very straight forward to do adaptive elision
using TSX. It is based on the infrastructure added earlier
for mutexes, but uses its own elision macros. The macros
are fairly general purpose and could be used for other
elision purposes too.
This version is much cleaner than the earlier assembler based
version, and in particular implements adaptation which makes
it safer.
I changed the behavior slightly to not require any changes
in the test suite and fully conform to all expected
behaviors (generally at the cost of not eliding in
various situations). In particular this means the timedlock
variants are not elided. Nested trylock aborts.
One difference of the C versions to the assembler wr/rdlock
is that the C compiler saves some registers which are unnecessary
for the fast path in the prologue of the functions. Split the
uncontended fast path out into a separate function. Only when contention is
detected is the full featured function called. This makes
the fast path code (nearly) identical to the assembler version,
and gives uncontended performance within a few cycles.
v2: Rename some functions and add space.
The implementation of __get_nprocs uses a stactic variable to cache
the value of the current number of processors. The caching breaks when
'time (NULL) == 0':
$ cat nproc.c
#include <stdio.h>
#include <time.h>
#include <sys/time.h>
int main(int argc, char *argv[])
{
time_t t;
struct timeval tv = {0, 0};
printf("settimeofday({0, 0}, NULL) = %d\n", settimeofday(&tv, NULL));
t = time(NULL);
printf("Time: %d, CPUs: %d\n", (unsigned int)t, get_nprocs());
return 0;
}
$ gcc -O3 nproc.c
$ ./a.out
settimeofday({0, 0}, NULL) = -1
Time: 1401311578, CPUs: 4
$ sudo ./a.out
settimeofday({0, 0}, NULL) = 0
Time: 0, CPUs: 0
The problem is with the condition used to check whether a cached
value should be returned or not:
static int cached_result;
static time_t timestamp;
time_t now = time (NULL);
time_t prev = timestamp;
atomic_read_barrier ();
if (now == prev)
return cached_result;
This patch fixes the problem by ensuring that 'cached_result' has
been set at least once before returning it.
