And __pthread_mutexattr_destroy as a compat symbol (so no
GLIBC_2.34 symbol version is added for it).
The symbols were moved using scripts/move-symbol-to-libc.py.
The symbols were moved using scripts/move-symbol-to-libc.py.
__pthread_mutex_trylock is used to implement mtx_timedlock,
which still resides in libpthread, so add a GLIBC_2.34 version
for it, to match the existing GLIBC_2.0 version.
The symbols were moved using scripts/move-symbol-to-libc.py.
The symbol aliasing follows pthread_cond_timedwait et al.
Missing hidden prototypes had to be added to nptl/pthreadP.h
for consistency.
Improvements compared to POWER9 version:
1. Take into account first 16B comparison for aligned strings
The previous version compares the first 16B and increments r4 by the number
of bytes until the address is 16B-aligned, then starts doing aligned loads at
that address. For aligned strings, this causes the first 16B to be compared
twice, because the increment is 0. Here we calculate the next 16B-aligned
address differently, which avoids that issue.
2. Use simple comparisons for the first ~192 bytes
The main loop is good for big strings, but comparing 16B each time is better
for smaller strings. So after aligning the address to 16 Bytes, we check
more 176B in 16B chunks. There may be some overlaps with the main loop for
unaligned strings, but we avoid using the more aggressive strategy too soon,
and also allow the loop to start at a 64B-aligned address. This greatly
benefits smaller strings and avoids overlapping checks if the string is
already aligned at a 64B boundary.
3. Reduce dependencies between load blocks caused by address calculation on loop
Doing a precise time tracing on the code showed many loads in the loop were
stalled waiting for updates to r4 from previous code blocks. This
implementation avoids that as much as possible by using 2 registers (r4 and
r5) to hold addresses to be used by different parts of the code.
Also, the previous code aligned the address to 16B, then to 64B by doing a
few 48B loops (if needed) until the address was aligned. The main loop could
not start until that 48B loop had finished and r4 was updated with the
current address. Here we calculate the address used by the loop very early,
so it can start sooner.
The main loop now uses 2 pointers 128B apart to make pointer updates less
frequent, and also unrolls 1 iteration to guarantee there is enough time
between iterations to update the pointers, reducing stalled cycles.
4. Use new P10 instructions
lxvp is used to load 32B with a single instruction, reducing contention in
the load queue.
vextractbm allows simplifying the tail code for the loop, replacing
vbpermq and avoiding having to generate a permute control vector.
Reviewed-by: Paul E Murphy <murphyp@linux.ibm.com>
Reviewed-by: Raphael M Zinsly <rzinsly@linux.ibm.com>
Reviewed-by: Lucas A. M. Magalhaes <lamm@linux.ibm.com>
The symbol was moved using scripts/move-symbol-to-libc.py.
There is no new symbol version because of the compatibility symbol
status. The __pthread_atfork reference in nptl/Versions was unused.
This is required for GCC versions before 10 which default to -fcommon.
Fixes commit 442e8a40da ("nptl: Move part
of TCB initialization from libpthread to __tls_init_tp").
The signal handler is exported as __nptl_setxid_sighandler, so
that the libpthread initialization code can install it. This
is sufficient for now because it is guarantueed to happen before
the first pthread_create call.
Onl pthread_cond_clockwait did not have a forwarder, so it needs
a new symbol version.
Some complications arise due to the need to supply hidden aliases,
GLIBC_PRIVATE exports (for the C11 condition variable implementation
that still remains in libpthread) and 64-bit time_t stubs.
pthread_cond_broadcast, pthread_cond_signal, pthread_cond_timedwait,
pthread_cond_wait, pthread_cond_clockwait have been moved using
scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This is complicated because of a second compilation of
nptl/pthread_mutex_lock.c via nptl/pthread_mutex_cond_lock.c.
PTHREAD_MUTEX_VERSIONS is introduced to suppress symbol versions
in that case.
The symbols __pthread_mutex_lock, __pthread_mutex_unlock,
__pthread_mutex_init, __pthread_mutex_destroy, pthread_mutex_lock,
pthread_mutex_unlock, pthread_mutex_init, pthread_mutex_destroy
have been moved using scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The current approach is to do this optimizations at a higher level,
in generic code, so that single-threaded cases can be specifically
targeted.
Furthermore, using IS_IN (libc) as a compile-time indicator that
all locks are private is no longer correct once process-shared lock
implementations are moved into libc.
The generic <lowlevellock.h> is not compatible with assembler code
(obviously), so it's necessary to remove two long-unused #includes.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This is optimization is similar in spirit to the SINGLE_THREAD_P check
in the malloc implementation. Doing this in generic code allows us
to prioritize those cases which are likely to occur in single-threaded
programs (normal and recursive mutexes).
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This is in preparation of moving the mutex code into libc.
__pthread_tunables_init is now called via __pthread_early_init.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The forwarders were only used internally, so new symbol versions
are needed. All symbols are moved at once because the forwarders
are no-ops if libpthread is not loaded, leading to inconsistencies
in case of a partial migration.
The symbols __pthread_rwlock_rdlock, __pthread_rwlock_unlock,
__pthread_rwlock_wrlock, pthread_rwlock_rdlock,
pthread_rwlock_unlock, pthread_rwlock_wrlock have been moved using
scripts/move-symbol-to-libc.py.
The __ symbol variants are turned into compat symbols, which is why they
do not receive a GLIBC_2.34 version.
The symbol was moved using scripts/move-symbol-to-libc.py.
tss_delete (still in libpthread) uses the __pthread_key_create
alias, so that is now exported under GLIBC_PRIVATE.
This initalization should only happen once for the main thread's TCB.
At present, auditors can achieve this by not linking against
libpthread. If libpthread becomes part of libc, doing this
initialization in libc would happen for every audit namespace,
or too late (if it happens from the main libc only). That's why
moving this code into ld.so seems the right thing to do, right after
the TCB initialization.
For !__ASSUME_SET_ROBUST_LIST ports, this also moves the symbol
__set_robust_list_avail into ld.so, as __nptl_set_robust_list_avail.
It also turned into a proper boolean flag.
Inline the __pthread_initialize_pids function because it seems no
longer useful as a separate function.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
TLS_INIT_TP is processor-specific, so it is not a good place to
put thread library initialization code (it would have to be repeated
for all CPUs). Introduce __tls_init_tp as a separate function,
to be called immediately after TLS_INIT_TP. Move the existing
stack list setup code for NPTL to this function.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Calling free directly may end up freeing a pointer allocated by the
dynamic loader using malloc from libc.so in the base namespace using
the allocator from libc.so in a secondary namespace, which results in
crashes.
This commit redirects the free call through GLRO and the dynamic
linker, to reach the correct namespace. It also cleans up the dlerror
handling along the way, so that pthread_setspecific is no longer
needed (which avoids triggering bug 24774).
Commit 9e78f6f6e7 ("Implement
_dl_catch_error, _dl_signal_error in libc.so [BZ #16628]") has the
side effect that distinct namespaces, as created by dlmopen, now have
separate implementations of the rtld exception mechanism. This means
that the call to _dl_catch_error from libdl in a secondary namespace
does not actually install an exception handler because the
thread-local variable catch_hook in the libc.so copy in the secondary
namespace is distinct from that of the base namepace. As a result, a
dlsym/dlopen/... failure in a secondary namespace terminates the process
with a dynamic linker error because it looks to the exception handler
mechanism as if no handler has been installed.
This commit restores GLRO (dl_catch_error) and uses it to set the
handler in the base namespace.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
No new symbol version is required because there was a forwarder.
The symbol has been moved using scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
No new symbol version is required because there was a forwarder.
The symbol has been moved using scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The pthread_exit symbol was moved using
scripts/move-symbol-to-libc.py. No new symbol version is needed
because there was a forwarder.
The new tests nptl/tst-pthread_exit-nothreads and
nptl/tst-pthread_exit-nothreads-static exercise the scenario
that pthread_exit is called without libpthread having been linked in.
This is not possible for the generic code, so these tests do not
live in sysdeps/pthread for now.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
It's necessary to stub out __libc_disable_asynccancel and
__libc_enable_asynccancel via rtld-stubbed-symbols because the new
direct references to the unwinder result in symbol conflicts when the
rtld exception handling from libc is linked in during the construction
of librtld.map.
unwind-forcedunwind.c is merged into unwind-resume.c. libc now needs
the functions that were previously only used in libpthread.
The GLIBC_PRIVATE exports of __libc_longjmp and __libc_siglongjmp are
no longer needed, so switch them to hidden symbols.
The symbol __pthread_unwind_next has been moved using
scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerva Zanella <adhemerval.zanella@linaro.org>
And also the fork generation counter, __fork_generation. This
eliminates the need for __fork_generation_pointer.
call_once remains in libpthread and calls the exported __pthread_once
symbol.
pthread_once and __pthread_once have been moved using
scripts/move-symbol-to-libc.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This internal symbol is used as part of the longjmp implementation.
Rename the file from nptl/pt-cleanup.c to nptl/pthread_cleanup_upto.c
so that the pt-* files remain restricted to libpthread.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The definitions in libc are sufficient, the forwarders are no longer
needed.
The symbols have been moved using scripts/move-symbol-to-libc.py.
s390-linux-gnu and s390x-linux-gnu need a new version placeholder
to keep the GLIBC_2.19 symbol version in libpthread.
Tested on i386-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu,
x86_64-linux-gnu. Built with build-many-glibcs.py.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This affects _pthread_cleanup_pop, _pthread_cleanup_pop_restore,
_pthread_cleanup_push, _pthread_cleanup_push_defer. The symbols
have been moved using scripts/move-symbol-to-libc.py.
No new symbol versions are added because the symbols are turned into
compatibility symbols at the same time.
__pthread_cleanup_pop and __pthread_cleanup_push are added as
GLIBC_PRIVATE symbols because they are also used internally, for
glibc's own cancellation handling.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
It is still used internally. Since unwinding is now available
unconditionally, avoid indirect calls through function pointers loaded
from the stack by inlining the non-cancellation cleanup code. This
avoids a regression in security hardening.
The out-of-line __libc_cleanup_routine implementation is no longer
needed because the inline definition is now static __always_inline.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
