Remove the 4 uses of atomic_and and atomic_or with atomic_fetch_and_acquire
and atomic_fetch_or_acquire. This is preserves existing implied semantics,
however relaxed MO on FUTEX_OWNER_DIED accesses may be correct.
Passes regress on AArch64.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.
I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah. I don't
know why I run into these diagnostics whereas others evidently do not.
remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
Update
commit 49302b8fdf
Author: H.J. Lu <hjl.tools@gmail.com>
Date: Thu Nov 11 06:54:01 2021 -0800
Avoid extra load with CAS in __pthread_mutex_clocklock_common [BZ #28537]
Replace boolean CAS with value CAS to avoid the extra load.
and
commit 0b82747dc4
Author: H.J. Lu <hjl.tools@gmail.com>
Date: Thu Nov 11 06:31:51 2021 -0800
Avoid extra load with CAS in __pthread_mutex_lock_full [BZ #28537]
Replace boolean CAS with value CAS to avoid the extra load.
by moving assignment out of the CAS condition.
Linux added FUTEX_LOCK_PI2 to support clock selection
(commit bf22a6976897977b0a3f1aeba6823c959fc4fdae). With the new
flag we can now proper support CLOCK_MONOTONIC for
pthread_mutex_clocklock with Priority Inheritance. If kernel
does not support, EINVAL is returned instead.
The difference is the futex operation will be issued and the kernel
will advertise the missing support (instead of hard-code error
return).
Checked on x86_64-linux-gnu and i686-linux-gnu on Linux 5.14, 5.11,
and 4.15.
This patch uses the new futex PI operation provided by Linux v5.14
when it is required.
The futex_lock_pi64() is moved to futex-internal.c (since it used on
two different places and its code size might be large depending of the
kernel configuration) and clockid is added as an argument.
Co-authored-by: Kurt Kanzenbach <kurt@linutronix.de>
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The remaining symbols are mostly used by libthread_db.
__pthread_get_minstack has to remain exported even though unused.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
A new build flag, _TIME_BITS, enables the usage of the newer 64-bit
time symbols for legacy ABI (where 32-bit time_t is default). The 64
bit time support is only enabled if LFS (_FILE_OFFSET_BITS=64) is
also used.
Different than LFS support, the y2038 symbols are added only for the
required ABIs (armhf, csky, hppa, i386, m68k, microblaze, mips32,
mips64-n32, nios2, powerpc32, sparc32, s390-32, and sh). The ABIs with
64-bit time support are unchanged, both for symbol and types
redirection.
On Linux the full 64-bit time support requires a minimum of kernel
version v5.1. Otherwise, the 32-bit fallbacks are used and might
results in error with overflow return code (EOVERFLOW).
The i686-gnu does not yet support 64-bit time.
This patch exports following rediretions to support 64-bit time:
* libc:
adjtime
adjtimex
clock_adjtime
clock_getres
clock_gettime
clock_nanosleep
clock_settime
cnd_timedwait
ctime
ctime_r
difftime
fstat
fstatat
futimens
futimes
futimesat
getitimer
getrusage
gettimeofday
gmtime
gmtime_r
localtime
localtime_r
lstat_time
lutimes
mktime
msgctl
mtx_timedlock
nanosleep
nanosleep
ntp_gettime
ntp_gettimex
ppoll
pselec
pselect
pthread_clockjoin_np
pthread_cond_clockwait
pthread_cond_timedwait
pthread_mutex_clocklock
pthread_mutex_timedlock
pthread_rwlock_clockrdlock
pthread_rwlock_clockwrlock
pthread_rwlock_timedrdlock
pthread_rwlock_timedwrlock
pthread_timedjoin_np
recvmmsg
sched_rr_get_interval
select
sem_clockwait
semctl
semtimedop
sem_timedwait
setitimer
settimeofday
shmctl
sigtimedwait
stat
thrd_sleep
time
timegm
timerfd_gettime
timerfd_settime
timespec_get
utime
utimensat
utimes
utimes
wait3
wait4
* librt:
aio_suspend
mq_timedreceive
mq_timedsend
timer_gettime
timer_settime
* libanl:
gai_suspend
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
The symbol was moved using scripts/move-symbol-to-libc.py.
The __pthread_mutex_timedlock@@GLIBC_PRIVATE export is no longer
needed, so it is removed with this commit.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
The symbols were moved using scripts/move-symbol-to-libc.py.
The symbol aliasing follows pthread_cond_timedwait et al.
Missing hidden prototypes had to be added to nptl/pthreadP.h
for consistency.
The elision interfaces are closely aligned between the targets that
implement them, so declare them in the generic <lowlevellock.h>
file.
Empty .c stubs are provided, so that fewer makefile updates
under sysdeps are needed. Also simplify initialization via
__libc_early_init.
The symbols __lll_clocklock_elision, __lll_lock_elision,
__lll_trylock_elision, __lll_unlock_elision, __pthread_force_elision
move into libc. For the time being, non-hidden references are used
from libpthread to access them, but once that part of libpthread
is moved into libc, hidden symbols will be used again. (Hidden
references seem desirable to reduce the likelihood of transactions
aborts.)
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
Some futex-internal calls require additional check for EOVERFLOW (as
indicated by [1] [2] [3]). For both mutex and rwlock code, EOVERFLOW is
handle as ETIMEDOUT; since it indicate to the caller that the blocking
operation could not be issued.
For mutex it avoids a possible issue where PTHREAD_MUTEX_ROBUST_* might
assume EOVERFLOW indicate futex has succeed, and for PTHREAD_MUTEX_PP_*
it avoid a potential busy infinite loop. For rwlock and semaphores, it
also avoids potential busy infinite loops.
Checked on x86_64-linux-gnu and i686-linux-gnu, although EOVERFLOW
won't be possible with current usage (since all timeouts on 32-bit
architectures with 32-bit time_t support will be in the range of
32-bit time_t).
[1] https://sourceware.org/pipermail/libc-alpha/2020-November/120079.html
[2] https://sourceware.org/pipermail/libc-alpha/2020-November/120080.html
[3] https://sourceware.org/pipermail/libc-alpha/2020-November/120127.html
The 878fe624d4 changed lll_futex_timed_wait, which expects a relative
timeout, with a __futex_abstimed_wait64, which expects an absolute
timeout. However the code still passes a relative timeout.
Also, the PTHREAD_PRIO_PROTECT support for clocks different than
CLOCK_REALTIME was broken since the inclusion of
pthread_mutex_clocklock (9d20e22e46) since lll_futex_timed_wait
always use CLOCK_REALTIME.
This patch fixes by removing the relative time calculation. It
also adds some xtests that tests both thread and inter-process
usage.
Checked on x86_64-linux-gnu.
Linux futex FUTEX_LOCK_PI operation only supports CLOCK_REALTIME,
so pthread_mutex_clocklock operation with priority aware mutexes
may fail depending of the input timeout.
Also, it is not possible to convert a CLOCK_MONOTONIC to a
CLOCK_REALTIME due the possible wall clock time change which might
invalid the requested timeout.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
The idea is to make NPTL implementation to use on the functions
provided by futex-internal.h.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
The idea is to make NPTL implementation to use on the functions
provided by futex-internal.h.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
For non null timeouts, the __futex_clocklock_wait64 creates an a
relative timeout by subtracting the current time from the input
argument. The same behavior can be obtained with FUTEX_WAIT_BITSET
without the need to calculate the relative timeout. Besides consolidate
the code it also avoid the possible relative timeout issues [1].
The __futex_abstimed_wait64 needs also to return EINVAL syscall
errors.
Checked on x86_64-linux-gnu and i686-linux-gnu.
[1] https://sourceware.org/pipermail/libc-alpha/2020-November/119881.html
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Most systems are SMP, so optimizing for the UP case is no longer
approriate. A dynamic check based on the kernel identification
has been only implemented for i386 anyway.
To disable adaptive mutexes on sh, define DEFAULT_ADAPTIVE_COUNT
as zero for this architecture.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The commit:
"y2038: nptl: Convert pthread_mutex_{clock|timed}lock to support 64 bit"
SHA1: 29e9874a04
introduced support for 64 bit timeouts. Unfortunately, it was missing the
code for bitset - i.e. lll_futex_clock_wait_bitset C preprocessor macro
was used. As a result the 64 bit struct __timespec64 was coerced to 32
bit struct timespec and regression visible as timeout was observed
(nptl/tst-robust10 on s390).
Reported-by: Stefan Liebler <stli@linux.ibm.com>
Tested-by: Stefan Liebler <stli@linux.ibm.com>
The pthread_mutex_clocklock and pthread_mutex_timedlock have been converted
to support 64 bit time.
This change uses:
- New __futex_clocklock_wait64 (instead of lll_timedwait)
from ./sysdeps/nptl/futex-helpers.c and
- New __futex_clocklock64 function (instead of lll_clocklock)
- New futex_lock_pi64
defined in sysdeps/nptl/futex-internal.h
The pthread_mutex_{clock|timed}lock only accepts absolute time.
Moreover, there is no need to check for NULL passed as *abstime pointer to the
syscalls as those calls have exported symbols marked with __nonull attribute
for abstime.
Some architectures - namely x86, powerpc and s390 - do support lock elision.
For those - adjustments have been made in arch specific elision-*.c files
to use __futex_clocklock64 instead of lll_clocklock.
The __lll_lock_elision (aliased to __lll_clocklock_elision in e.g.
sysdeps/unix/sysv/linux/s390/elision-timed.c) just uses, in this patch
provided, __futex_clocklock64.
For systems with __TIMESIZE != 64 && __WORDSIZE == 32:
- Conversions between 64 bit time to 32 bit are necessary
- Redirection to pthread_mutex_{clock|timed}lock will provide support for 64
bit time
Build tests:
./src/scripts/build-many-glibcs.py glibcs
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This patch adds the generic futex_lock_pi and futex_unlock_pi to wrap
around the syscall machinery required to issue the syscall calls. It
simplifies a bit the futex code required to implement PI mutexes.
No function changes, checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
To help y2038 work avoid duplicate all the logic of nanosleep on
non cancellable version, the patch replace it with a new futex
operation, lll_timedwait. The changes are:
- Add a expected value for __lll_clocklock_wait, so it can be used
to wait for generic values.
- Remove its internal atomic operation and move the logic to
__lll_clocklock. It makes __lll_clocklock_wait even more generic
and __lll_clocklock slight faster on fast-path (since it won't
require a function call anymore).
- Add lll_timedwait, which uses __lll_clocklock_wait, to replace both
__pause_nocancel and __nanosleep_nocancel.
It also allows remove the sparc32 __lll_clocklock_wait implementation
(since it is similar to the generic one).
Checked on x86_64-linux-gnu, sparcv9-linux-gnu, and i686-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Since gettimeofday will shortly be implemented in terms of
clock_gettime on all platforms, internal code should use clock_gettime
directly; in addition to removing a layer of indirection, this will
allow us to remove the PLT-bypass gunk for gettimeofday. (We can't
quite do that yet, but it'll be coming later in this patch series.)
In many cases, the changed code does fewer conversions.
The changed code always assumes __clock_gettime (CLOCK_REALTIME)
cannot fail. Most of the call sites were assuming gettimeofday could
not fail, but a few places were checking for errors. POSIX says
clock_gettime can only fail if the clock constant is invalid or
unsupported, and CLOCK_REALTIME is the one and only clock constant
that's required to be supported. For consistency I grepped the entire
source tree for any other places that checked for errors from
__clock_gettime (CLOCK_REALTIME), found one, and changed it too.
(For the record, POSIX also says gettimeofday can never fail.)
(It would be nice if we could declare that GNU systems will always
support CLOCK_MONOTONIC as well as CLOCK_REALTIME; there are several
places where we are using CLOCK_REALTIME where _MONOTONIC would be
more appropriate, and/or trying to use _MONOTONIC and then falling
back to _REALTIME. But the Hurd doesn't support CLOCK_MONOTONIC yet,
and it looks like adding it would involve substantial changes to
gnumach's internals and API. Oh well.)
A few Hurd-specific files were changed to use __host_get_time instead
of __clock_gettime, as this seemed tidier. We also assume this cannot
fail. Skimming the code in gnumach leads me to believe the only way
it could fail is if __mach_host_self also failed, and our
Hurd-specific code consistently assumes that can't happen, so I'm
going with that.
With the exception of support/support_test_main.c, test cases are not
modified, mainly because I didn't want to have to figure out which
test cases were testing gettimeofday specifically.
The definition of GETTIME in sysdeps/generic/memusage.h had a typo and
was not reading tv_sec at all. I fixed this. It appears nobody has been
generating malloc traces on a machine that doesn't have a superseding
definition.
There are a whole bunch of places where the code could be simplified
by factoring out timespec subtraction and/or comparison logic, but I
want to keep this patch as mechanical as possible.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, powerpc-linux-gnu, and aarch64-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
The valid_nanoseconds () static inline function has been introduced to
check if nanoseconds value is in the correct range - greater or equal to
zero and less than 1000000000.
The explicit #include <time.h> has been added to files where it was
missing.
The __syscall_slong_t type for ns has been used to avoid issues on x32.
Tested with:
- scripts/build-many-glibcs.py
- make PARALLELMFLAGS="-j12" && make PARALLELMFLAGS="-j12" xcheck on x86_64
Rename lll_timedlock to lll_clocklock and add clockid
parameter to indicate the clock that the abstime parameter should
be measured against in preparation for adding
pthread_mutex_clocklock.
The name change mirrors the naming for the exposed pthread functions:
timed => absolute timeout measured against CLOCK_REALTIME (or clock
specified by attribute in the case of pthread_cond_timedwait.)
clock => absolute timeout measured against clock specified in preceding
parameter.
* sysdeps/nptl/lowlevellock.h (lll_clocklock): Rename from
lll_timedlock and add clockid parameter. (__lll_clocklock): Rename
from __lll_timedlock and add clockid parameter.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_clocklock):
Likewise.
* nptl/lll_timedlock_wait.c (__lll_clocklock_wait): Rename from
__lll_timedlock_wait and add clockid parameter. Use __clock_gettime
rather than __gettimeofday so that clockid can be used. This means
that conversion from struct timeval is no longer required.
* sysdeps/sparc/sparc32/lowlevellock.c (lll_clocklock_wait):
Likewise.
* sysdeps/sparc/sparc32/lll_timedlock_wait.c: Update comment to
refer to __lll_clocklock_wait rather than __lll_timedlock_wait.
* nptl/pthread_mutex_timedlock.c (lll_clocklock_elision): Rename
from lll_timedlock_elision, add clockid parameter and use
meaningful names for other parameters. (__pthread_mutex_timedlock):
Pass CLOCK_REALTIME where necessary to lll_clocklock and
lll_clocklock_elision.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h
(lll_clocklock_elision): Rename from lll_timedlock_elision and add
clockid parameter. (__lll_clocklock_elision): Rename from
__lll_timedlock_elision and add clockid parameter.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/elision-timed.c
(__lll_lock_elision): Call __lll_clocklock_elision rather than
__lll_timedlock_elision. (EXTRAARG): Add clockid parameter.
(LLL_LOCK): Likewise.
* sysdeps/unix/sysv/linux/s390/elision-timed.c: Likewise.
* sysdeps/unix/sysv/linux/x86/elision-timed.c: Likewise.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
In preparation for adding POSIX clockwait variants of timedwait functions,
add a clockid_t parameter to futex_abstimed_wait functions and pass
CLOCK_REALTIME from all callers for the time being.
Replace lll_futex_timed_wait_bitset with lll_futex_clock_wait_bitset
which takes a clockid_t parameter rather than the magic clockbit.
* sysdeps/nptl/lowlevellock-futex.h,
sysdeps/unix/sysv/linux/lowlevellock-futex.h: Replace
lll_futex_timed_wait_bitset with lll_futex_clock_wait_bitset that
takes a clockid rather than a special clockbit.
* sysdeps/nptl/lowlevellock-futex.h: Add
lll_futex_supported_clockid so that client functions can check
whether their clockid parameter is valid even if they don't
ultimately end up calling lll_futex_clock_wait_bitset.
* sysdeps/nptl/futex-internal.h,
sysdeps/unix/sysv/linux/futex-internal.h
(futex_abstimed_wait, futex_abstimed_wait_cancelable): Add
clockid_t parameter to indicate which clock the absolute time
passed should be measured against. Pass that clockid onto
lll_futex_clock_wait_bitset. Add invalid clock as reason for
returning -EINVAL.
* sysdeps/nptl/futex-internal.h,
sysdeps/unix/sysv/linux/futex-internal.h: Introduce
futex_abstimed_supported_clockid so that client functions can check
whether their clockid parameter is valid even if they don't
ultimately end up calling futex_abstimed_wait.
* nptl/pthread_cond_wait.c (__pthread_cond_wait_common): Remove
code to calculate relative timeout for
__PTHREAD_COND_CLOCK_MONOTONIC_MASK and just pass CLOCK_MONOTONIC
or CLOCK_REALTIME as required to futex_abstimed_wait_cancelable.
* nptl/pthread_rwlock_common (__pthread_rwlock_rdlock_full)
(__pthread_wrlock_full), nptl/sem_waitcommon (do_futex_wait): Pass
additional CLOCK_REALTIME to futex_abstimed_wait_cancelable.
* nptl/pthread_mutex_timedlock.c (__pthread_mutex_timedlock):
Switch to lll_futex_clock_wait_bitset and pass CLOCK_REALTIME
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This patch assumes realtime clock support for nptl and thus removes
all the associated code.
For __pthread_mutex_timedlock the fallback usage for the case where
lll_futex_timed_wait_bitset it not set define is also removed. The
generic lowlevellock-futex.h always define it, so for NPTL code the
check always yield true.
Checked on x86_64-linux-gnu and i686-linux-gnu.
* nptl/nptl-init.c (__have_futex_clock_realtime,
__have_futex_clock_realtime): Remove definition.
(__pthread_initialize_minimal_internal): Remove FUTEX_CLOCK_REALTIME
check test for !__ASSUME_FUTEX_CLOCK_REALTIME.
* nptl/pthread_mutex_timedlock.c (__pthread_mutex_timedlock): Assume
__ASSUME_FUTEX_CLOCK_REALTIME support.
* sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/kernel-features.h
(__ASSUME_FUTEX_CLOCK_REALTIME): Remove.
* sysdeps/nptl/lowlevellock-futex.h (lll_futex_timed_wait_bitset):
Adjust comment.
This patch does not have any functionality change, we only provide a spin
count tunes for pthread adaptive spin mutex. The tunable
glibc.pthread.mutex_spin_count tunes can be used by system administrator to
squeeze system performance according to different hardware capabilities and
workload characteristics.
The maximum value of spin count is limited to 32767 to avoid the overflow
of mutex->__data.__spins variable with the possible type of short in
pthread_mutex_lock ().
The default value of spin count is set to 100 with the reference to the
previous number of times of spinning via trylock. This value would be
architecture-specific and can be tuned with kinds of benchmarks to fit most
cases in future.
I would extend my appreciation sincerely to H.J.Lu for his help to refine
this patch series.
* manual/tunables.texi (POSIX Thread Tunables): New node.
* nptl/Makefile (libpthread-routines): Add pthread_mutex_conf.
* nptl/nptl-init.c: Include pthread_mutex_conf.h
(__pthread_initialize_minimal_internal) [HAVE_TUNABLES]: Call
__pthread_tunables_init.
* nptl/pthreadP.h (MAX_ADAPTIVE_COUNT): Remove.
(max_adaptive_count): Define.
* nptl/pthread_mutex_conf.c: New file.
* nptl/pthread_mutex_conf.h: New file.
* sysdeps/generic/adaptive_spin_count.h: New file.
* sysdeps/nptl/dl-tunables.list: New file.
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock): Use
max_adaptive_count () not MAX_ADAPTIVE_COUNT.
* nptl/pthread_mutex_timedlock.c (__pthrad_mutex_timedlock):
Likewise.
Suggested-by: Andi Kleen <andi.kleen@intel.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Signed-off-by: Kemi.wang <kemi.wang@intel.com>
The race leads either to pthread_mutex_destroy returning EBUSY
or triggering an assertion (See description in bugzilla).
This patch is fixing the race by ensuring that the elision path is
used in all cases if elision is enabled by the GLIBC_TUNABLES framework.
The __kind variable in struct __pthread_mutex_s is accessed concurrently.
Therefore we are now using the atomic macros.
The new testcase tst-mutex10 is triggering the race on s390x and intel.
Presumably also on power, but I don't have access to a power machine
with lock-elision. At least the code for power is the same as on the other
two architectures.
ChangeLog:
[BZ #23275]
* nptl/tst-mutex10.c: New File.
* nptl/Makefile (tests): Add tst-mutex10.
(tst-mutex10-ENV): New variable.
* sysdeps/unix/sysv/linux/s390/force-elision.h: (FORCE_ELISION):
Ensure that elision path is used if elision is available.
* sysdeps/unix/sysv/linux/powerpc/force-elision.h (FORCE_ELISION):
Likewise.
* sysdeps/unix/sysv/linux/x86/force-elision.h: (FORCE_ELISION):
Likewise.
* nptl/pthreadP.h (PTHREAD_MUTEX_TYPE, PTHREAD_MUTEX_TYPE_ELISION)
(PTHREAD_MUTEX_PSHARED): Use atomic_load_relaxed.
* nptl/pthread_mutex_consistent.c (pthread_mutex_consistent): Likewise.
* nptl/pthread_mutex_getprioceiling.c (pthread_mutex_getprioceiling):
Likewise.
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock_full)
(__pthread_mutex_cond_lock_adjust): Likewise.
* nptl/pthread_mutex_setprioceiling.c (pthread_mutex_setprioceiling):
Likewise.
* nptl/pthread_mutex_timedlock.c (__pthread_mutex_timedlock): Likewise.
* nptl/pthread_mutex_trylock.c (__pthread_mutex_trylock): Likewise.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_full): Likewise.
* sysdeps/nptl/bits/thread-shared-types.h (struct __pthread_mutex_s):
Add comments.
* nptl/pthread_mutex_destroy.c (__pthread_mutex_destroy):
Use atomic_load_relaxed and atomic_store_relaxed.
* nptl/pthread_mutex_init.c (__pthread_mutex_init):
Use atomic_store_relaxed.
Fix the typo in the fallback path in __pthread_mutex_timedlock ()
whic hcalls lll_futex_timed_wait (). This is only useful for cases
where the patch is being backported to older distributions where
only lll_futex_timed_wait () is available.
This patch consolidates all the non cancellable nanosleep calls to use
the __nanosleep_nocancel identifier. For non cancellable targets it will
be just a macro to call the default respective symbol while on Linux
will be a internal one.
Checked on x86_64-linux-gnu, x86_64-linux-gnu-x32, and i686-linux-gnu.
* nptl/pthread_mutex_timedlock.c (__pthread_mutex_timedlock): Replace
nanosleep_not_cancel with __nanosleep_nocancel.
* sysdeps/generic/not-cancel.h (nanosleep_not_cancel): Remove macro.
(__nanosleep_nocancel): New macro.
* sysdeps/unix/sysv/linux/nanosleep.c (__nanosleep_nocancel): New
function.
* sysdeps/unix/sysv/linux/not-cancel.h (nanosleep_not_cancel): Remove
macro.
(__nanosleep_nocancel): New prototype.
65810f0ef0 fixed a robust mutex bug but
introduced BZ 21778: if the CAS used to try to acquire a lock fails, the
expected value is not updated, which breaks other cases in the loce
acquisition loop. The fix is to simply update the expected value with
the value returned by the CAS, which ensures that behavior is as if the
first case with the CAS never happened (if the CAS fails).
This is a regression introduced in the last release.
Tested on x86_64, i686, ppc64, ppc64le, s390x, aarch64, armv7hl.
Any changes to the per-thread list of robust mutexes currently acquired as
well as the pending-operations entry are not simply sequential code but
basically concurrent with any actions taken by the kernel when it tries
to clean up after a crash. This is not quite like multi-thread concurrency
but more like signal-handler concurrency.
This patch fixes latent bugs by adding compiler barriers where necessary so
that it is ensured that the kernel crash handling sees consistent data.
This is meant to be easy to backport, so we do not use C11-style signal
fences yet.
* nptl/descr.h (ENQUEUE_MUTEX_BOTH, DEQUEUE_MUTEX): Add compiler
barriers and comments.
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock_full): Likewise.
* nptl/pthread_mutex_timedlock.c (pthread_mutex_timedlock): Likewise.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_full): Likewise.
lll_robust_unlock on i386 and x86_64 first sets the futex word to
FUTEX_WAITERS|0 before calling __lll_unlock_wake, which will set the
futex word to 0. If the thread is killed between these steps, then the
futex word will be FUTEX_WAITERS|0, and the kernel (at least current
upstream) will not set it to FUTEX_OWNER_DIED|FUTEX_WAITERS because 0 is
not equal to the TID of the crashed thread.
The lll_robust_lock assembly code on i386 and x86_64 is not prepared to
deal with this case because the fastpath tries to only CAS 0 to TID and
not FUTEX_WAITERS|0 to TID; the slowpath simply waits until it can CAS 0
to TID or the futex_word has the FUTEX_OWNER_DIED bit set.
This issue is fixed by removing the custom x86 assembly code and using
the generic C code instead. However, instead of adding more duplicate
code to the custom x86 lowlevellock.h, the code of the lll_robust* functions
is inlined into the single call sites that exist for each of these functions
in the pthread_mutex_* functions. The robust mutex paths in the latter
have been slightly reorganized to make them simpler.
This patch is meant to be easy to backport, so C11-style atomics are not
used.
[BZ #20985]
* nptl/Makefile: Adapt.
* nptl/pthread_mutex_cond_lock.c (LLL_ROBUST_MUTEX_LOCK): Remove.
(LLL_ROBUST_MUTEX_LOCK_MODIFIER): New.
* nptl/pthread_mutex_lock.c (LLL_ROBUST_MUTEX_LOCK): Remove.
(LLL_ROBUST_MUTEX_LOCK_MODIFIER): New.
(__pthread_mutex_lock_full): Inline lll_robust* functions and adapt.
* nptl/pthread_mutex_timedlock.c (pthread_mutex_timedlock): Inline
lll_robust* functions and adapt.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_full): Likewise.
* sysdeps/nptl/lowlevellock.h (__lll_robust_lock_wait,
__lll_robust_lock, lll_robust_cond_lock, __lll_robust_timedlock_wait,
__lll_robust_timedlock, __lll_robust_unlock): Remove.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h (lll_robust_lock,
lll_robust_cond_lock, lll_robust_timedlock, lll_robust_unlock): Remove.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (lll_robust_lock,
lll_robust_cond_lock, lll_robust_timedlock, lll_robust_unlock): Remove.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h (__lll_robust_lock_wait,
__lll_robust_lock, lll_robust_cond_lock, __lll_robust_timedlock_wait,
__lll_robust_timedlock, __lll_robust_unlock): Remove.
* nptl/lowlevelrobustlock.c: Remove file.
* nptl/lowlevelrobustlock.sym: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevelrobustlock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevelrobustlock.S: Likewise.
Assume that Thread 1 waits to acquire a robust mutex using futexes to
block (and thus sets the FUTEX_WAITERS flag), and is unblocked when this
mutex is released. If Thread 2 concurrently acquires the lock and is
killed, Thread 1 can recover from the died owner but fail to restore the
FUTEX_WAITERS flag. This can lead to a Thread 3 that also blocked using
futexes at the same time as Thread 1 to not get woken up because
FUTEX_WAITERS is not set anymore.
The fix for this is to ensure that we continue to preserve the
FUTEX_WAITERS flag whenever we may have set it or shared it with another
thread. This is the same requirement as in the algorithm for normal
mutexes, only that the robust mutexes need additional handling for died
owners and thus preserving the FUTEX_WAITERS flag cannot be done just in
the futex slowpath code.
[BZ #20973]
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock_full): Fix lost
wake-up in robust mutexes.
* nptl/pthread_mutex_timedlock.c (pthread_mutex_timedlock): Likewise.
Error checking mutexes are not supposed to be subject to lock elision.
That would defeat the error checking nature of the mutex because lock
elision doesn't record ownership.
This mostly automatically-generated patch converts 113 function
definitions in glibc from old-style K&R to prototype-style. Following
my other recent such patches, this one deals with the case of function
definitions in files that either contain assertions or where grep
suggested they might contain assertions - and thus where it isn't
possible to use a simple object code comparison as a sanity check on
the correctness of the patch, because line numbers are changed.
A few such automatically-generated changes needed to be supplemented
by manual changes for the result to compile. openat64 had a prototype
declaration with "..." but an old-style definition in
sysdeps/unix/sysv/linux/dl-openat64.c, and "..." needed adding to the
generated prototype in the definition (I've filed
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68024> for diagnosing
such cases in GCC; the old state was undefined behavior not requiring
a diagnostic, but one seems a good idea). In addition, as Florian has
noted regparm attribute mismatches between declaration and definition
are only diagnosed for prototype definitions, and five functions
needed internal_function added to their definitions (in the case of
__pthread_mutex_cond_lock, via the macro definition of
__pthread_mutex_lock) to compile on i386.
After this patch is in, remaining old-style definitions are probably
most readily fixed manually before we can turn on
-Wold-style-definition for all builds.
Tested for x86_64 and x86 (testsuite).
* crypt/md5-crypt.c (__md5_crypt_r): Convert to prototype-style
function definition.
* crypt/sha256-crypt.c (__sha256_crypt_r): Likewise.
* crypt/sha512-crypt.c (__sha512_crypt_r): Likewise.
* debug/backtracesyms.c (__backtrace_symbols): Likewise.
* elf/dl-minimal.c (_itoa): Likewise.
* hurd/hurdmalloc.c (malloc): Likewise.
(free): Likewise.
(realloc): Likewise.
* inet/inet6_option.c (inet6_option_space): Likewise.
(inet6_option_init): Likewise.
(inet6_option_append): Likewise.
(inet6_option_alloc): Likewise.
(inet6_option_next): Likewise.
(inet6_option_find): Likewise.
* io/ftw.c (FTW_NAME): Likewise.
(NFTW_NAME): Likewise.
(NFTW_NEW_NAME): Likewise.
(NFTW_OLD_NAME): Likewise.
* libio/iofwide.c (_IO_fwide): Likewise.
* libio/strops.c (_IO_str_init_static_internal): Likewise.
(_IO_str_init_static): Likewise.
(_IO_str_init_readonly): Likewise.
(_IO_str_overflow): Likewise.
(_IO_str_underflow): Likewise.
(_IO_str_count): Likewise.
(_IO_str_seekoff): Likewise.
(_IO_str_pbackfail): Likewise.
(_IO_str_finish): Likewise.
* libio/wstrops.c (_IO_wstr_init_static): Likewise.
(_IO_wstr_overflow): Likewise.
(_IO_wstr_underflow): Likewise.
(_IO_wstr_count): Likewise.
(_IO_wstr_seekoff): Likewise.
(_IO_wstr_pbackfail): Likewise.
(_IO_wstr_finish): Likewise.
* locale/programs/localedef.c (normalize_codeset): Likewise.
* locale/programs/locarchive.c (add_locale_to_archive): Likewise.
(add_locales_to_archive): Likewise.
(delete_locales_from_archive): Likewise.
* malloc/malloc.c (__libc_mallinfo): Likewise.
* math/gen-auto-libm-tests.c (init_fp_formats): Likewise.
* misc/tsearch.c (__tfind): Likewise.
* nptl/pthread_attr_destroy.c (__pthread_attr_destroy): Likewise.
* nptl/pthread_attr_getdetachstate.c
(__pthread_attr_getdetachstate): Likewise.
* nptl/pthread_attr_getguardsize.c (pthread_attr_getguardsize):
Likewise.
* nptl/pthread_attr_getinheritsched.c
(__pthread_attr_getinheritsched): Likewise.
* nptl/pthread_attr_getschedparam.c
(__pthread_attr_getschedparam): Likewise.
* nptl/pthread_attr_getschedpolicy.c
(__pthread_attr_getschedpolicy): Likewise.
* nptl/pthread_attr_getscope.c (__pthread_attr_getscope):
Likewise.
* nptl/pthread_attr_getstack.c (__pthread_attr_getstack):
Likewise.
* nptl/pthread_attr_getstackaddr.c (__pthread_attr_getstackaddr):
Likewise.
* nptl/pthread_attr_getstacksize.c (__pthread_attr_getstacksize):
Likewise.
* nptl/pthread_attr_init.c (__pthread_attr_init_2_1): Likewise.
(__pthread_attr_init_2_0): Likewise.
* nptl/pthread_attr_setdetachstate.c
(__pthread_attr_setdetachstate): Likewise.
* nptl/pthread_attr_setguardsize.c (pthread_attr_setguardsize):
Likewise.
* nptl/pthread_attr_setinheritsched.c
(__pthread_attr_setinheritsched): Likewise.
* nptl/pthread_attr_setschedparam.c
(__pthread_attr_setschedparam): Likewise.
* nptl/pthread_attr_setschedpolicy.c
(__pthread_attr_setschedpolicy): Likewise.
* nptl/pthread_attr_setscope.c (__pthread_attr_setscope):
Likewise.
* nptl/pthread_attr_setstack.c (__pthread_attr_setstack):
Likewise.
* nptl/pthread_attr_setstackaddr.c (__pthread_attr_setstackaddr):
Likewise.
* nptl/pthread_attr_setstacksize.c (__pthread_attr_setstacksize):
Likewise.
* nptl/pthread_condattr_setclock.c (pthread_condattr_setclock):
Likewise.
* nptl/pthread_create.c (__find_in_stack_list): Likewise.
* nptl/pthread_getattr_np.c (pthread_getattr_np): Likewise.
* nptl/pthread_mutex_cond_lock.c (__pthread_mutex_lock): Define to
use internal_function.
* nptl/pthread_mutex_init.c (__pthread_mutex_init): Convert to
prototype-style function definition.
* nptl/pthread_mutex_lock.c (__pthread_mutex_lock): Likewise.
(__pthread_mutex_cond_lock_adjust): Likewise. Use
internal_function.
* nptl/pthread_mutex_timedlock.c (pthread_mutex_timedlock):
Convert to prototype-style function definition.
* nptl/pthread_mutex_trylock.c (__pthread_mutex_trylock):
Likewise.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt):
Likewise.
(__pthread_mutex_unlock): Likewise.
* nptl_db/td_ta_clear_event.c (td_ta_clear_event): Likewise.
* nptl_db/td_ta_set_event.c (td_ta_set_event): Likewise.
* nptl_db/td_thr_clear_event.c (td_thr_clear_event): Likewise.
* nptl_db/td_thr_event_enable.c (td_thr_event_enable): Likewise.
* nptl_db/td_thr_set_event.c (td_thr_set_event): Likewise.
* nss/makedb.c (process_input): Likewise.
* posix/fnmatch.c (__strchrnul): Likewise.
(__wcschrnul): Likewise.
(fnmatch): Likewise.
* posix/fnmatch_loop.c (FCT): Likewise.
* posix/glob.c (globfree): Likewise.
(__glob_pattern_type): Likewise.
(__glob_pattern_p): Likewise.
* posix/regcomp.c (re_compile_pattern): Likewise.
(re_set_syntax): Likewise.
(re_compile_fastmap): Likewise.
(regcomp): Likewise.
(regerror): Likewise.
(regfree): Likewise.
* posix/regexec.c (regexec): Likewise.
(re_match): Likewise.
(re_search): Likewise.
(re_match_2): Likewise.
(re_search_2): Likewise.
(re_search_stub): Likewise. Use internal_function
(re_copy_regs): Likewise.
(re_set_registers): Convert to prototype-style function
definition.
(prune_impossible_nodes): Likewise. Use internal_function.
* resolv/inet_net_pton.c (inet_net_pton): Convert to
prototype-style function definition.
(inet_net_pton_ipv4): Likewise.
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Likewise.
* sysdeps/pthread/aio_cancel.c (aio_cancel): Likewise.
* sysdeps/pthread/aio_suspend.c (aio_suspend): Likewise.
* sysdeps/pthread/timer_delete.c (timer_delete): Likewise.
* sysdeps/unix/sysv/linux/dl-openat64.c (openat64): Likewise.
Make variadic.
* time/strptime_l.c (localtime_r): Convert to prototype-style
function definition.
* wcsmbs/mbsnrtowcs.c (__mbsnrtowcs): Likewise.
* wcsmbs/mbsrtowcs_l.c (__mbsrtowcs_l): Likewise.
* wcsmbs/wcsnrtombs.c (__wcsnrtombs): Likewise.
* wcsmbs/wcsrtombs.c (__wcsrtombs): Likewise.
