Current allocate_stack logic for create stacks is to first mmap all
the required memory with the desirable memory and then mprotect the
guard area with PROT_NONE if required. Although it works as expected,
it pessimizes the allocation because it requires the kernel to actually
increase commit charge (it counts against the available physical/swap
memory available for the system).
The only issue is to actually check this change since side-effects are
really Linux specific and to actually account them it would require a
kernel specific tests to parse the system wide information. On the kernel
I checked /proc/self/statm does not show any meaningful difference for
vmm and/or rss before and after thread creation. I could only see
really meaningful information checking on system wide /proc/meminfo
between thread creation: MemFree, MemAvailable, and Committed_AS shows
large difference without the patch. I think trying to use these
kind of information on a testcase is fragile.
The BZ#18988 reports shows that the commit pages are easily seen with
mlockall (MCL_FUTURE) (with lock all pages that become mapped in the
process) however a more straighfoward testcase shows that pthread_create
could be faster using this patch:
--
static const int inner_count = 256;
static const int outer_count = 128;
static
void *thread1(void *arg)
{
return NULL;
}
static
void *sleeper(void *arg)
{
pthread_t ts[inner_count];
for (int i = 0; i < inner_count; i++)
pthread_create (&ts[i], &a, thread1, NULL);
for (int i = 0; i < inner_count; i++)
pthread_join (ts[i], NULL);
return NULL;
}
int main(void)
{
pthread_attr_init(&a);
pthread_attr_setguardsize(&a, 1<<20);
pthread_attr_setstacksize(&a, 1134592);
pthread_t ts[outer_count];
for (int i = 0; i < outer_count; i++)
pthread_create(&ts[i], &a, sleeper, NULL);
for (int i = 0; i < outer_count; i++)
pthread_join(ts[i], NULL);
assert(r == 0);
}
return 0;
}
--
On x86_64 (4.4.0-45-generic, gcc 5.4.0) running the small benchtests
I see:
$ time ./test
real 0m3.647s
user 0m0.080s
sys 0m11.836s
While with the patch I see:
$ time ./test
real 0m0.696s
user 0m0.040s
sys 0m1.152s
So I added a pthread_create benchtest (thread_create) which check
the thread creation latency. As for the simple benchtests, I saw
improvements in thread creation on all architectures I tested the
change.
Checked on x86_64-linux-gnu, i686-linux-gnu, aarch64-linux-gnu,
arm-linux-gnueabihf, powerpc64le-linux-gnu, sparc64-linux-gnu,
and sparcv9-linux-gnu.
[BZ #18988]
* benchtests/thread_create-inputs: New file.
* benchtests/thread_create-source.c: Likewise.
* support/xpthread_attr_setguardsize.c: Likewise.
* support/Makefile (libsupport-routines): Add
xpthread_attr_setguardsize object.
* support/xthread.h: Add xpthread_attr_setguardsize prototype.
* benchtests/Makefile (bench-pthread): Add thread_create.
* nptl/allocatestack.c (allocate_stack): Call mmap with PROT_NONE and
then mprotect the required area.
This patch removes the COLORING_INCREMENT define and usage on allocatestack.c.
It has not been used since 564cd8b67e (glibc-2.3.3) by any architecture.
The idea is to simplify the code by removing obsolete code.
* nptl/allocatestack.c [COLORING_INCREMENT] (nptl_ncreated): Remove.
(allocate_stack): Remove COLORING_INCREMENT usage.
* nptl/stack-aliasing.h (COLORING_INCREMENT). Likewise.
* sysdeps/i386/i686/stack-aliasing.h (COLORING_INCREMENT): Likewise.
In _dl_nothread_init_static_tls() and init_one_static_tls() we must not
touch the DTV of other threads since we do not have ownership of them.
The DTV need not be initialized at this point anyway since only LD/GD
accesses will use them. If LD/GD accesses occur they will take care to
initialize their own thread's DTV.
Concurrency comments were removed from the patch since they need to be
reworked along with a full description of DTV ownership and when it is
or is not safe to modify these structures.
Alexandre Oliva's original patch and discussion:
https://sourceware.org/ml/libc-alpha/2016-09/msg00512.html
This patch remove the PID cache and usage in current GLIBC code. Current
usage is mainly used a performance optimization to avoid the syscall,
however it adds some issues:
- The exposed clone syscall will try to set pid/tid to make the new
thread somewhat compatible with current GLIBC assumptions. This cause
a set of issue with new workloads and usecases (such as BZ#17214 and
[1]) as well for new internal usage of clone to optimize other algorithms
(such as clone plus CLONE_VM for posix_spawn, BZ#19957).
- The caching complexity also added some bugs in the past [2] [3] and
requires more effort of each port to handle such requirements (for
both clone and vfork implementation).
- Caching performance gain in mainly on getpid and some specific
code paths. The getpid performance leverage is questionable [4],
either by the idea of getpid being a hotspot as for the getpid
implementation itself (if it is indeed a justifiable hotspot a
vDSO symbol could let to a much more simpler solution).
Other usage is mainly for non usual code paths, such as pthread
cancellation signal and handling.
For thread creation (on stack allocation) the code simplification in fact
adds some performance gain due the no need of transverse the stack cache
and invalidate each element pid.
Other thread usages will require a direct getpid syscall, such as
cancellation/setxid signal, thread cancellation, thread fail path (at
create_thread), and thread signal (pthread_kill and pthread_sigqueue).
However these are hardly usual hotspots and I think adding a syscall is
justifiable.
It also simplifies both the clone and vfork arch-specific implementation.
And by review each fork implementation there are some discrepancies that
this patch also solves:
- microblaze clone/vfork does not set/reset the pid/tid field
- hppa uses the default vfork implementation that fallback to fork.
Since vfork is deprecated I do not think we should bother with it.
The patch also removes the TID caching in clone. My understanding for
such semantic is try provide some pthread usage after a user program
issue clone directly (as done by thread creation with CLONE_PARENT_SETTID
and pthread tid member). However, as stated before in multiple discussions
threads, GLIBC provides clone syscalls without further supporting all this
semantics.
I ran a full make check on x86_64, x32, i686, armhf, aarch64, and powerpc64le.
For sparc32, sparc64, and mips I ran the basic fork and vfork tests from
posix/ folder (on a qemu system). So it would require further testing
on alpha, hppa, ia64, m68k, nios2, s390, sh, and tile (I excluded microblaze
because it is already implementing the patch semantic regarding clone/vfork).
[1] https://codereview.chromium.org/800183004/
[2] https://sourceware.org/ml/libc-alpha/2006-07/msg00123.html
[3] https://sourceware.org/bugzilla/show_bug.cgi?id=15368
[4] http://yarchive.net/comp/linux/getpid_caching.html
* sysdeps/nptl/fork.c (__libc_fork): Remove pid cache setting.
* nptl/allocatestack.c (allocate_stack): Likewise.
(__reclaim_stacks): Likewise.
(setxid_signal_thread): Obtain pid through syscall.
* nptl/nptl-init.c (sigcancel_handler): Likewise.
(sighandle_setxid): Likewise.
* nptl/pthread_cancel.c (pthread_cancel): Likewise.
* sysdeps/unix/sysv/linux/pthread_kill.c (__pthread_kill): Likewise.
* sysdeps/unix/sysv/linux/pthread_sigqueue.c (pthread_sigqueue):
Likewise.
* sysdeps/unix/sysv/linux/createthread.c (create_thread): Likewise.
* sysdeps/unix/sysv/linux/getpid.c: Remove file.
* nptl/descr.h (struct pthread): Change comment about pid value.
* nptl/pthread_getattr_np.c (pthread_getattr_np): Remove thread
pid assert.
* sysdeps/unix/sysv/linux/pthread-pids.h (__pthread_initialize_pids):
Do not set pid value.
* nptl_db/td_ta_thr_iter.c (iterate_thread_list): Remove thread
pid cache check.
* nptl_db/td_thr_validate.c (td_thr_validate): Likewise.
* sysdeps/aarch64/nptl/tcb-offsets.sym: Remove pid offset.
* sysdeps/alpha/nptl/tcb-offsets.sym: Likewise.
* sysdeps/arm/nptl/tcb-offsets.sym: Likewise.
* sysdeps/hppa/nptl/tcb-offsets.sym: Likewise.
* sysdeps/i386/nptl/tcb-offsets.sym: Likewise.
* sysdeps/ia64/nptl/tcb-offsets.sym: Likewise.
* sysdeps/m68k/nptl/tcb-offsets.sym: Likewise.
* sysdeps/microblaze/nptl/tcb-offsets.sym: Likewise.
* sysdeps/mips/nptl/tcb-offsets.sym: Likewise.
* sysdeps/nios2/nptl/tcb-offsets.sym: Likewise.
* sysdeps/powerpc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/s390/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sh/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sparc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/tile/nptl/tcb-offsets.sym: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym: Likewise.
* sysdeps/unix/sysv/linux/aarch64/clone.S: Remove pid and tid caching.
* sysdeps/unix/sysv/linux/alpha/clone.S: Likewise.
* sysdeps/unix/sysv/linux/arm/clone.S: Likewise.
* sysdeps/unix/sysv/linux/hppa/clone.S: Likewise.
* sysdeps/unix/sysv/linux/i386/clone.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/clone2.S: Likewise.
* sysdeps/unix/sysv/linux/mips/clone.S: Likewise.
* sysdeps/unix/sysv/linux/nios2/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sh/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/tile/clone.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/aarch64/vfork.S: Remove pid set and reset.
* sysdeps/unix/sysv/linux/alpha/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/arm/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/i386/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/m68k/clone.S: Likewise.
* sysdeps/unix/sysv/linux/m68k/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/mips/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/nios2/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sh/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/tile/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/tst-clone2.c (f): Remove direct pthread
struct access.
(clone_test): Remove function.
(do_test): Rewrite to take in consideration pid is not cached anymore.
An earlier fix for TLS dropped early initialization of DTV entries for
modules using static TLS, leaving it for __tls_get_addr to set them
up. That worked on platforms that require the GD access model to be
relaxed to LE in the main executable, but it caused a regression on
platforms that allow GD in the main executable, particularly in
statically-linked programs: they use a custom __tls_get_addr that does
not update the DTV, which fails when the DTV early initialization is
not performed.
In static programs, __libc_setup_tls performs the DTV initialization
for the main thread, but the DTV of other threads is set up in
_dl_allocate_tls_init, so that's the fix that matters.
Restoring the initialization in the remaining functions modified by
this patch was just for uniformity. It's not clear that it is ever
needed: even on platforms that allow GD in the main executable, the
dynamically-linked version of __tls_get_addr would set up the DTV
entries, even for static TLS modules, while updating the DTV counter.
for ChangeLog
[BZ #19826]
* elf/dl-tls.c (_dl_allocate_tls_init): Restore DTV early
initialization of static TLS entries.
* elf/dl-reloc.c (_dl_nothread_init_static_tls): Likewise.
* nptl/allocatestack.c (init_one_static_tls): Likewise.
Instead of a flag which indicates the pointer can be freed, dtv_t
now includes the pointer which should be freed. Due to padding,
the size of dtv_t does not increase.
To avoid using memalign, the new allocate_dtv_entry function
allocates a sufficiently large buffer so that a sub-buffer
can be found in it which starts with an aligned pointer. Both
the aligned and original pointers are kept, the latter for calling
free later.
On arches that set _STACK_GROWS_UP, the stacktop variable is declared
and set, but never actually used. Refactor the code a bit so that the
variable is only declared/set under _STACK_GROWS_DOWN settings.
This adds new functions for futex operations, starting with wait,
abstimed_wait, reltimed_wait, wake. They add documentation and error
checking according to the current draft of the Linux kernel futex manpage.
Waiting with absolute or relative timeouts is split into separate functions.
This allows for removing a few cases of code duplication in pthreads code,
which uses absolute timeouts; also, it allows us to put platform-specific
code to go from an absolute to a relative timeout into the platform-specific
futex abstractions..
Futex operations that can be canceled are also split out into separate
functions suffixed by "_cancelable".
There are separate versions for both Linux and NaCl; while they currently
differ only slightly, my expectation is that the separate versions of
lowlevellock-futex.h will eventually be merged into futex-internal.h
when we get to move the lll_ functions over to the new futex API.
for ChangeLog
[BZ #17090]
[BZ #17620]
[BZ #17621]
[BZ #17628]
* NEWS: Update.
* elf/dl-tls.c (_dl_update_slotinfo): Clean up outdated DTV
entries with Static TLS too. Skip entries past the end of the
allocated DTV, from Alan Modra.
(tls_get_addr_tail): Update to glibc_likely/unlikely. Move
Static TLS DTV entry set up from...
(_dl_allocate_tls_init): ... here (fix modid assertion), ...
* elf/dl-reloc.c (_dl_nothread_init_static_tls): ... here...
* nptl/allocatestack.c (init_one_static_tls): ... and here...
* elf/dlopen.c (dl_open_worker): Drop l_tls_modid upper bound
for Static TLS.
* elf/tlsdeschtab.h (map_generation): Return size_t. Check
that the slot we find is associated with the given map before
using its generation count.
* nptl_db/db_info.c: Include ldsodefs.h.
(rtld_global, dtv_slotinfo_list, dtv_slotinfo): New typedefs.
* nptl_db/structs.def (DB_RTLD_VARIABLE): New macro.
(DB_MAIN_VARIABLE, DB_RTLD_GLOBAL_FIELD): Likewise.
(link_map::l_tls_offset): New struct field.
(dtv_t::counter): Likewise.
(rtld_global): New struct.
(_rtld_global): New rtld variable.
(dl_tls_dtv_slotinfo_list): New rtld global field.
(dtv_slotinfo_list): New struct.
(dtv_slotinfo): Likewise.
* nptl_db/td_symbol_list.c: Drop gnu/lib-names.h include.
(td_lookup): Rename to...
(td_mod_lookup): ... this. Use new mod parameter instead of
LIBPTHREAD_SO.
* nptl_db/td_thr_tlsbase.c: Include link.h.
(dtv_slotinfo_list, dtv_slotinfo): New functions.
(td_thr_tlsbase): Check DTV generation. Compute Static TLS
addresses even if the DTV is out of date or missing them.
* nptl_db/fetch-value.c (_td_locate_field): Do not refuse to
index zero-length arrays.
* nptl_db/thread_dbP.h: Include gnu/lib-names.h.
(td_lookup): Make it a macro implemented in terms of...
(td_mod_lookup): ... this declaration.
* nptl_db/db-symbols.awk (DB_RTLD_VARIABLE): Override.
(DB_MAIN_VARIABLE): Likewise.
If a call to the set*id functions fails in a multi-threaded program,
the abort introduced in commit 13f7fe35ae
was triggered.
We address by checking that all calls to set*id on all threads give
the same result, and only abort if we see success followed by failure
(or vice versa).
This patch introduces two new convenience functions to set the default
thread attributes used for creating threads. This allows a programmer
to set the default thread attributes just once in a process and then
run pthread_create without additional attributes.
(__free_stack_cache): Removed. Change callers to call __free_stacks.
* init.c (nptl_freeres): New function.
(pthread_functions): Initialize ptr_freeres to nptl_freeres.
* pthreadP.h: Don't declare __free_stack_cache. Declare __free_stacks.
* sysdeps/pthread/unwind-forcedunwind.c (libgcc_s_handle): New
variable.
(pthread_cancel_init): Depend in libgcc_s_handle for decision to
load DSO. Assign last.
(__unwind_freeres): New function.
* allocatestack.c (__reclaim_stacks): Reset in_flight_stack later
for better debugging. No need to use stack_list_add here.
Use __asm __volatile (__lll_acq_instr ::: "memory") instead of
atomic_full_barrier.
2007-07-31 Jakub Jelinek <jakub@redhat.com>
* allocatestack.c (stack_cache_lock): Change type to int.
(get_cached_stack, allocate_stack, __deallocate_stack,
__make_stacks_executable, __find_thread_by_id, __nptl_setxid,
__pthread_init_static_tls, __wait_lookup_done): Add LLL_PRIVATE
as second argument to lll_lock and lll_unlock macros on
stack_cache_lock.
* pthread_create.c (__find_in_stack_list): Likewise.
(start_thread): Similarly with pd->lock. Use lll_robust_dead
macro instead of lll_robust_mutex_dead, pass LLL_SHARED to it
as second argument.
* descr.h (struct pthread): Change lock and setxid_futex field
type to int.
* old_pthread_cond_broadcast.c (__pthread_cond_broadcast_2_0): Use
LLL_LOCK_INITIALIZER instead of LLL_MUTEX_LOCK_INITIALIZER.
* old_pthread_cond_signal.c (__pthread_cond_signal_2_0): Likewise.
* old_pthread_cond_timedwait.c (__pthread_cond_timedwait_2_0):
Likewise.
* old_pthread_cond_wait.c (__pthread_cond_wait_2_0): Likewise.
* pthread_cond_init.c (__pthread_cond_init): Likewise.
* pthreadP.h (__attr_list_lock): Change type to int.
* pthread_attr_init.c (__attr_list_lock): Likewise.
* pthread_barrier_destroy.c (pthread_barrier_destroy): Pass
ibarrier->private ^ FUTEX_PRIVATE_FLAG as second argument to
lll_{,un}lock.
* pthread_barrier_wait.c (pthread_barrier_wait): Likewise and
also for lll_futex_{wake,wait}.
* pthread_barrier_init.c (pthread_barrier_init): Make iattr
a pointer to const.
* pthread_cond_broadcast.c (__pthread_cond_broadcast): Pass
LLL_SHARED as second argument to lll_{,un}lock.
* pthread_cond_destroy.c (__pthread_cond_destroy): Likewise.
* pthread_cond_signal.c (__pthread_cond_singal): Likewise.
* pthread_cond_timedwait.c (__pthread_cond_timedwait): Likewise.
* pthread_cond_wait.c (__condvar_cleanup, __pthread_cond_wait):
Likewise.
* pthread_getattr_np.c (pthread_getattr_np): Add LLL_PRIVATE
as second argument to lll_{,un}lock macros on pd->lock.
* pthread_getschedparam.c (__pthread_getschedparam): Likewise.
* pthread_setschedparam.c (__pthread_setschedparam): Likewise.
* pthread_setschedprio.c (pthread_setschedprio): Likewise.
* tpp.c (__pthread_tpp_change_priority, __pthread_current_priority):
Likewise.
* sysdeps/pthread/createthread.c (do_clone, create_thread):
Likewise.
* pthread_once.c (once_lock): Change type to int.
(__pthread_once): Pass LLL_PRIVATE as second argument to
lll_{,un}lock macros on once_lock.
* pthread_rwlock_rdlock.c (__pthread_rwlock_rdlock): Use
lll_{,un}lock macros instead of lll_mutex_{,un}lock, pass
rwlock->__data.__shared as second argument to them and similarly
for lll_futex_w*.
* pthread_rwlock_timedrdlock.c (pthread_rwlock_timedrdlock):
Likewise.
* pthread_rwlock_timedwrlock.c (pthread_rwlock_timedwrlock):
Likewise.
* pthread_rwlock_tryrdlock.c (__pthread_rwlock_tryrdlock): Likewise.
* pthread_rwlock_trywrlock.c (__pthread_rwlock_trywrlock): Likewise.
* pthread_rwlock_unlock.c (__pthread_rwlock_unlock): Likewise.
* pthread_rwlock_wrlock.c (__pthread_rwlock_wrlock): Likewise.
* sem_close.c (sem_close): Pass LLL_PRIVATE as second argument
to lll_{,un}lock macros on __sem_mappings_lock.
* sem_open.c (check_add_mapping): Likewise.
(__sem_mappings_lock): Change type to int.
* semaphoreP.h (__sem_mappings_lock): Likewise.
* pthread_mutex_lock.c (LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK,
LLL_ROBUST_MUTEX_LOCK): Use lll_{,try,robust_}lock macros
instead of lll_*mutex_*, pass LLL_SHARED as last
argument.
(__pthread_mutex_lock): Use lll_unlock instead of lll_mutex_unlock,
pass LLL_SHARED as last argument.
* sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c (LLL_MUTEX_LOCK,
LLL_MUTEX_TRYLOCK, LLL_ROBUST_MUTEX_LOCK): Use
lll_{cond_,cond_try,robust_cond}lock macros instead of lll_*mutex_*,
pass LLL_SHARED as last argument.
* pthread_mutex_timedlock.c (pthread_mutex_timedlock): Use
lll_{timed,try,robust_timed,un}lock instead of lll_*mutex*, pass
LLL_SHARED as last argument.
* pthread_mutex_trylock.c (__pthread_mutex_trylock): Similarly.
* pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt):
Similarly.
* sysdeps/pthread/bits/libc-lock.h (__libc_lock_lock,
__libc_lock_lock_recursive, __libc_lock_unlock,
__libc_lock_unlock_recursive): Pass LLL_PRIVATE as second
argument to lll_{,un}lock.
* sysdeps/pthread/bits/stdio-lock.h (_IO_lock_lock,
_IO_lock_unlock): Likewise.
* sysdeps/unix/sysv/linux/fork.c (__libc_fork): Don't use
compound literal.
* sysdeps/unix/sysv/linux/unregister-atfork.c (__unregister_atfork):
Pass LLL_PRIVATE as second argument to lll_{,un}lock macros on
__fork_lock.
* sysdeps/unix/sysv/linux/register-atfork.c (__register_atfork,
free_mem): Likewise.
(__fork_lock): Change type to int.
* sysdeps/unix/sysv/linux/fork.h (__fork_lock): Likewise.
* sysdeps/unix/sysv/linux/sem_post.c (__new_sem_post): Pass
isem->private ^ FUTEX_PRIVATE_FLAG as second argument to
lll_futex_wake.
* sysdeps/unix/sysv/linux/sem_timedwait.c (sem_timedwait): Likewise.
* sysdeps/unix/sysv/linux/sem_wait.c (__new_sem_wait): Likewise.
* sysdeps/unix/sysv/linux/lowlevellock.c (__lll_lock_wait_private):
New function.
(__lll_lock_wait, __lll_timedlock_wait): Add private argument and
pass it through to lll_futex_*wait, only compile in when
IS_IN_libpthread.
* sysdeps/unix/sysv/linux/lowlevelrobustlock.c
(__lll_robust_lock_wait, __lll_robust_timedlock_wait): Add private
argument and pass it through to lll_futex_*wait.
* sysdeps/unix/sysv/linux/alpha/lowlevellock.h: Renamed all
lll_mutex_* resp. lll_robust_mutex_* macros to lll_* resp.
lll_robust_*. Renamed all __lll_mutex_* resp. __lll_robust_mutex_*
inline functions to __lll_* resp. __lll_robust_*.
(LLL_MUTEX_LOCK_INITIALIZER): Remove.
(lll_mutex_dead): Add private argument.
(__lll_lock_wait_private): New prototype.
(__lll_lock_wait, __lll_robust_lock_wait, __lll_lock_timedwait,
__lll_robust_lock_timedwait): Add private argument to prototypes.
(__lll_lock): Add private argument, if it is constant LLL_PRIVATE,
call __lll_lock_wait_private, otherwise pass private to
__lll_lock_wait.
(__lll_robust_lock, __lll_cond_lock, __lll_timedlock,
__lll_robust_timedlock): Add private argument, pass it to
__lll_*wait functions.
(__lll_unlock): Add private argument, if it is constant LLL_PRIVATE,
call __lll_unlock_wake_private, otherwise pass private to
__lll_unlock_wake.
(__lll_robust_unlock): Add private argument, pass it to
__lll_robust_unlock_wake.
(lll_lock, lll_robust_lock, lll_cond_lock, lll_timedlock,
lll_robust_timedlock, lll_unlock, lll_robust_unlock): Add private
argument, pass it through to __lll_* inline function.
(__lll_mutex_unlock_force, lll_mutex_unlock_force): Remove.
(lll_lock_t): Remove.
(__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake,
__lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait,
lll_cond_wake, lll_cond_broadcast): Remove.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Allow including
the header from assembler. Renamed all lll_mutex_* resp.
lll_robust_mutex_* macros to lll_* resp. lll_robust_*.
(LOCK, FUTEX_CMP_REQUEUE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE): Define.
(LLL_MUTEX_LOCK_INITIALIZER, LLL_MUTEX_LOCK_INITIALIZER_LOCKED,
LLL_MUTEX_LOCK_INITIALIZER_WAITERS): Remove.
(__lll_mutex_lock_wait, __lll_mutex_timedlock_wait,
__lll_mutex_unlock_wake, __lll_lock_wait, __lll_unlock_wake):
Remove prototype.
(__lll_trylock_asm, __lll_lock_asm_start, __lll_unlock_asm): Define.
(lll_robust_trylock, lll_cond_trylock): Use LLL_LOCK_INITIALIZER*
rather than LLL_MUTEX_LOCK_INITIALIZER* macros.
(lll_trylock): Likewise, use __lll_trylock_asm, pass
MULTIPLE_THREADS_OFFSET as another asm operand.
(lll_lock): Add private argument, use __lll_lock_asm_start, pass
MULTIPLE_THREADS_OFFSET as last asm operand, call
__lll_lock_wait_private if private is constant LLL_PRIVATE,
otherwise pass private as another argument to __lll_lock_wait.
(lll_robust_lock, lll_cond_lock, lll_robust_cond_lock,
lll_timedlock, lll_robust_timedlock): Add private argument, pass
private as another argument to __lll_*lock_wait call.
(lll_unlock): Add private argument, use __lll_unlock_asm, pass
MULTIPLE_THREADS_OFFSET as another asm operand, call
__lll_unlock_wake_private if private is constant LLL_PRIVATE,
otherwise pass private as another argument to __lll_unlock_wake.
(lll_robust_unlock): Add private argument, pass private as another
argument to __lll_unlock_wake.
(lll_robust_dead): Add private argument, use __lll_private_flag
macro.
(lll_islocked): Use LLL_LOCK_INITIALIZER instead of
LLL_MUTEX_LOCK_INITIALIZER.
(lll_lock_t): Remove.
(LLL_LOCK_INITIALIZER_WAITERS): Define.
(__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake,
__lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait,
lll_cond_wake, lll_cond_broadcast): Remove.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S: Revert
2007-05-2{3,9} changes.
* sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_PRIVATE_FUTEX_WAIT): Define.
(LOAD_FUTEX_WAIT): Rewritten.
(LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't
define.
(__lll_lock_wait_private, __lll_unlock_wake_private): New functions.
(__lll_mutex_lock_wait): Rename to ...
(__lll_lock_wait): ... this. Take futex addr from %edx instead of
%ecx, %ecx is now private argument. Don't compile in for libc.so.
(__lll_mutex_timedlock_wait): Rename to ...
(__lll_timedlock_wait): ... this. Use __NR_gettimeofday. %esi
contains private argument. Don't compile in for libc.so.
(__lll_mutex_unlock_wake): Rename to ...
(__lll_unlock_wake): ... this. %ecx contains private argument.
Don't compile in for libc.so.
(__lll_timedwait_tid): Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/lowlevelrobustlock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_FUTEX_WAIT): Define.
(LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't
define.
(__lll_robust_mutex_lock_wait): Rename to ...
(__lll_robust_lock_wait): ... this. Futex addr is now in %edx
argument, %ecx argument contains private. Use LOAD_FUTEX_WAIT
macro.
(__lll_robust_mutex_timedlock_wait): Rename to ...
(__lll_robust_timedlock_wait): ... this. Use __NR_gettimeofday.
%esi argument contains private, use LOAD_FUTEX_WAIT macro.
* sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S: Include
lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(pthread_barrier_wait): Rename __lll_mutex_* to __lll_*, pass
PRIVATE(%ebx) ^ LLL_SHARED as private argument in %ecx to
__lll_lock_wait and __lll_unlock_wake, pass MUTEX(%ebx) address
to __lll_lock_wait in %edx.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_REQUEUE,
FUTEX_CMP_REQUEUE, EINVAL, LOCK): Don't define.
(__pthread_cond_broadcast): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE, EINVAL, LOCK): Don't define.
(__pthread_cond_signal): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_cond_timedwait): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_wait, __condvar_w_cleanup): Rename __lll_mutex_*
to __lll_*, pass cond_lock address in %edx rather than %ecx to
__lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_rdlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebx) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebp) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebp) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_unlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%edi) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%edi) in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebx) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register.
* sysdeps/unix/sysv/linux/i386/pthread_once.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG): Don't
define.
* sysdeps/unix/sysv/linux/i386/i486/sem_post.S: Include lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAKE): Don't define.
* sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, SYS_gettimeofday, FUTEX_WAIT): Don't define.
(sem_timedwait): Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S: Include
lowlevellock.h.
(LOCK): Don't define.
* sysdeps/unix/sysv/linux/i386/i486/sem_wait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
* sysdeps/unix/sysv/linux/powerpc/sem_post.c: Wake only when there
are waiters.
* sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Revert
2007-05-2{3,9} changes.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_PRIVATE_FUTEX_WAIT): Define.
(LOAD_FUTEX_WAIT): Rewritten.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define.
(__lll_lock_wait_private, __lll_unlock_wake_private): New functions.
(__lll_mutex_lock_wait): Rename to ...
(__lll_lock_wait): ... this. %esi is now private argument.
Don't compile in for libc.so.
(__lll_mutex_timedlock_wait): Rename to ...
(__lll_timedlock_wait): ... this. %esi contains private argument.
Don't compile in for libc.so.
(__lll_mutex_unlock_wake): Rename to ...
(__lll_unlock_wake): ... this. %esi contains private argument.
Don't compile in for libc.so.
* sysdeps/unix/sysv/linux/x86_64/lowlevelrobustlock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_FUTEX_WAIT): Define.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define.
(__lll_robust_mutex_lock_wait): Rename to ...
(__lll_robust_lock_wait): ... this. %esi argument contains private.
Use LOAD_FUTEX_WAIT macro.
(__lll_robust_mutex_timedlock_wait): Rename to ...
(__lll_robust_timedlock_wait): ... this. %esi argument contains
private, use LOAD_FUTEX_WAIT macro.
* sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S: Include
lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(pthread_barrier_wait): Rename __lll_mutex_* to __lll_*, pass
PRIVATE(%rdi) ^ LLL_SHARED as private argument in %esi to
__lll_lock_wait and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_REQUEUE,
FUTEX_CMP_REQUEUE, EINVAL, LOCK): Don't define.
(__pthread_cond_broadcast): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE, EINVAL, LOCK): Don't define.
(__pthread_cond_signal): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_timedwait): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_wait, __condvar_cleanup): Rename __lll_mutex_*
to __lll_*, pass LLL_SHARED in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_rdlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_unlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_once.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG): Don't
define.
* sysdeps/unix/sysv/linux/x86_64/sem_post.S: Include lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAKE): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_trywait.S: Include
lowlevellock.h.
(LOCK): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_wait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
* sysdeps/unix/sysv/linux/sparc/internaltypes.h: New file.
* sysdeps/unix/sysv/linux/sparc/pthread_barrier_destroy.c: New file.
* sysdeps/unix/sysv/linux/sparc/pthread_barrier_init.c: New file.
* sysdeps/unix/sysv/linux/sparc/pthread_barrier_wait.c: New file.
* sysdeps/unix/sysv/linux/sparc/sparc32/lowlevellock.c
(__lll_lock_wait_private): New function.
(__lll_lock_wait, __lll_timedlock_wait): Add private argument, pass
it to lll_futex_*wait. Don't compile in for libc.so.
* sysdeps/unix/sysv/linux/sparc/sparc32/pthread_barrier_init.c:
Remove.
* sysdeps/unix/sysv/linux/sparc/sparc32/pthread_barrier_wait.c
(struct sparc_pthread_barrier): Remove.
(pthread_barrier_wait): Use union sparc_pthread_barrier instead of
struct sparc_pthread_barrier. Pass
ibarrier->s.pshared ? LLL_SHARED : LLL_PRIVATE to lll_{,un}lock
and lll_futex_wait macros.
* sysdeps/unix/sysv/linux/sparc/sparc32/sparcv9/pthread_barrier_init.c:
Remove.
* sysdeps/unix/sysv/linux/sparc/sparc32/sparcv9/pthread_barrier_wait.c:
Include sparc pthread_barrier_wait.c instead of generic one.
char array resp. pointer.
* iconvdata/iso-2022-kr.c (BODY): Make buf unsigned char instead of
char array.
* iconvdata/cns11643.h (cns11643_to_ucs4): Change first argument
to const unsigned char **.
(ucs4_to_cns11643): Change second argument to unsigned char *.
* iconvdata/euc-tw.c (BODY): Change endp type to
const unsigned char *.
* iconvdata/iso-ir-165.h (ucs4_to_isoir165): Change second argument
to unsigned char *.
* iconvdata/ibm1008_420.c (LOOP_NEED_FLAGS): Don't define.
* iconvdata/iso-2022-cn.c (BODY): Change buf to unsigned char array.
* iconvdata/iso-2022-cn-ext.c (BODY): Change buf, tmpbuf, tmp
types to unsigned char pointers/arrays instead of char.
* iconvdata/jis0201.h (ucs4_to_jisx0201): Change second argument
to unsigned char *.
* iconvdata/jis0208.h (ucs4_to_jisx0208): Likewise.
* iconvdata/jis0212.h: Include assert.h.
(ucs4_to_jisx0212): Change second argument to unsigned char *.
assert that if cp[0] is not '\0', cp[1] is not '\0' either instead
of trying to handle that.
* iconvdata/euc-kr.c (euckr_from_ucs4): Initialize also cp[1] to
shut up a warning.
* iconvdata/euc-jp-ms.c (from_ucs4_lat1, from_ucs4_greek,
from_ucs4_cjk, from_ucs4_cjkcpt, from_ucs4_extra): Change type to
two dimensional const unsigned char arrays.
(BODY): Cast "" to (const unsigned char *) for assignment to cp.
Initialize endp to inptr to shut up a warning.
