While ppc has the more important string functions in assembly,
there are still a few generic routines used.
Use the Power 6 CMPB insn for testing of zeros.
Checked on powerpc64le-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
While arm has the more important string functions in assembly,
there are still a few generic routines used.
Use the UQSUB8 insn for testing of zeros.
Checked on armv7-linux-gnueabihf
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
While alpha has the more important string functions in assembly,
there are still a few for find the generic routines are used.
Use the CMPBGE insn, via the builtin, for testing of zeros. Use a
simplified expansion of __builtin_ctz when the insn isn't available.
Checked on alpha-linux-gnu.
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Use UXOR,SBZ to test for a zero byte within a word. While we can
get semi-decent code out of asm-goto, we would do slightly better
with a compiler builtin.
For index_zero et al, sequential testing of bytes is less expensive than
any tricks that involve a count-leading-zeros insn that we don't have.
Checked on hppa-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
GCC's combine pass cannot merge (x >> c | y << (32 - c)) into a
double-word shift unless (1) the subtract is in the same basic block
and (2) the result of the subtract is used exactly once. Neither
condition is true for any use of MERGE.
By forcing the use of a double-word shift, we not only reduce
contention on SAR, but also allow the setting of SAR to be hoisted
outside of a loop.
Checked on hppa-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
It also cleanups the multiple inclusion by leaving the ifunc
implementation to undef the weak_alias and libc_hidden_def.
Co-authored-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
New algorithm read the first aligned address and mask off the
unwanted bytes (this strategy is similar to arch-specific
implementations used on powerpc, sparc, and sh).
The loop now read word-aligned address and check using the has_eq
macro.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc-linux-gnu,
and powerpc64-linux-gnu by removing the arch-specific assembly
implementation and disabling multi-arch (it covers both LE and BE
for 64 and 32 bits).
Co-authored-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
New algorithm now calls strchrnul.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc-linux-gnu,
and powerpc64-linux-gnu by removing the arch-specific assembly
implementation and disabling multi-arch (it covers both LE and BE
for 64 and 32 bits).
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
New algorithm read the first aligned address and mask off the unwanted
bytes (this strategy is similar to arch-specific implementations used
on powerpc, sparc, and sh).
The loop now read word-aligned address and check using the has_zero_eq
function.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64-linux-gnu,
and powerpc-linux-gnu by removing the arch-specific assembly
implementation and disabling multi-arch (it covers both LE and BE
for 64 and 32 bits).
Co-authored-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
New algorithm read the first aligned address and mask off the
unwanted bytes (this strategy is similar to arch-specific
implementations used on powerpc, sparc, and sh).
The loop now read word-aligned address and check using the has_zero
macro.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc-linux-gnu,
and powercp64-linux-gnu by removing the arch-specific assembly
implementation and disabling multi-arch (it covers both LE and BE
for 64 and 32 bits).
Co-authored-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
This patch adds generic string find and detection meant to be used in
generic vectorized string implementation. The idea is to decompose the
basic string operation so each architecture can reimplement if it
provides any specialized hardware instruction.
The 'string-misc.h' provides miscellaneous functions:
- extractbyte: extracts the byte from an specific index.
- repeat_bytes: setup an word by replicate the argument on each byte.
The 'string-fza.h' provides zero byte detection functions:
- find_zero_low, find_zero_all, find_eq_low, find_eq_all,
find_zero_eq_low, find_zero_eq_all, and find_zero_ne_all
The 'string-fzb.h' provides boolean zero byte detection functions:
- has_zero: determine if any byte within a word is zero.
- has_eq: determine byte equality between two words.
- has_zero_eq: determine if any byte within a word is zero along with
byte equality between two words.
The 'string-fzi.h' provides positions for string-fza.h results:
- index_first: return index of first zero byte within a word.
- index_last: return index of first byte different between two words.
The 'string-fzc.h' provides a combined version of fza and fzi:
- index_first_zero_eq: return index of first zero byte within a word or
first byte different between two words.
- index_first_zero_ne: return index of first zero byte within a word or
first byte equal between two words.
- index_last_zero: return index of last zero byte within a word.
- index_last_eq: return index of last byte different between two words.
The 'string-shift.h' provides a way to mask off parts of a work based on
some alignmnet (to handle unaligned arguments):
- shift_find, shift_find_last.
Co-authored-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
It moves OP_T_THRES out of memcopy.h to its own header and adjust
each architecture that redefines it.
Checked with a build and check with run-built-tests=no for all major
Linux ABIs.
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
It moves the op_t definition out to an specific header, adds
the attribute 'may-alias', and cleanup its duplicated definitions.
Checked with a build and check with run-built-tests=no for all major
Linux ABIs.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Improve SVE memcpy by copying 2 vectors if the size is small enough.
This improves performance of random memcpy by ~9% on Neoverse V1, and
33-64 byte copies are ~16% faster.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
This file is not used today since we end up using
sysdeps/i386/htl/machine-sp.h. Getting the stack pointer does not need
to be hurd specific and can go into sysdeps/<arch>.
Message-Id: <Y9tpWs2WOgE/Duiq@jupiter.tail36e24.ts.net>
This adds a special SHM_ANON value that can be passed into shm_open ()
in place of a name. When called in this way, shm_open () will create a
new anonymous shared memory file. The file will be created in the same
way that other shared memory files are created (i.e., under /dev/shm/),
except that it is not given a name and therefore cannot be reached from
the file system, nor by other calls to shm_open (). This is accomplished
by utilizing O_TMPFILE.
This is intended to be compatible with FreeBSD's API of the same name.
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
Message-Id: <20230130125216.6254-4-bugaevc@gmail.com>
This is a flag that causes open () to create a new, unnamed file in the
same filesystem as the given directory. The file descriptor can be
simply used in the creating process as a temporary file, or shared with
children processes via fork (), or sent over a Unix socket. The file can
be left anonymous, in which case it will be deleted from the backing
file system once all copies of the file descriptor are closed, or given
a permanent name with a linkat () call, such as the following:
int fd = open ("/tmp", O_TMPFILE | O_RDWR, 0700);
/* Do something with the file... */
linkat (fd, "", AT_FDCWD, "/tmp/filename", AT_EMPTY_PATH);
In between creating the file and linking it to the file system, it is
possible to set the file content, mode, ownership, author, and other
attributes, so that the file visibly appears in the file system (perhaps
replacing another file) atomically, with all of its attributes already
set up.
The Hurd support for O_TMPFILE directly exposes the dir_mkfile RPC to
user programs. Previously, dir_mkfile was used by glibc internally, in
particular for implementing tmpfile (), but not exposed to user programs
through a Unix-level API.
O_TMPFILE was initially introduced by Linux. This implementation is
intended to be compatible with the Linux implementation, except that the
O_EXCL flag is not given the special meaning when used together with
O_TMPFILE, unlike on Linux.
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
Message-Id: <20230130125216.6254-3-bugaevc@gmail.com>
Add an optimization to avoid calling clone3 when glibc detects that
there is no kernel support. It also adds __ASSUME_CLONE3, which allows
skipping this optimization and issuing the clone3 syscall directly.
It does not handle the the small window between 5.3 and 5.5 for
posix_spawn (CLONE_CLEAR_SIGHAND was added in 5.5).
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
It follow the internal signature:
extern int clone3 (struct clone_args *__cl_args, size_t __size,
int (*__func) (void *__arg), void *__arg);
Checked on aarch64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The clone3 flag resets all signal handlers of the child not set to
SIG_IGN to SIG_DFL. It allows to skip most of the sigaction calls
to setup child signal handling, where previously a posix_spawn
had to issue 2 times NSIG sigaction calls (one to obtain the current
disposition and another to set either SIG_DFL or SIG_IGN).
With POSIX_SPAWN_SETSIGDEF the child will setup the signal for the case
where the disposition is SIG_IGN.
The code must handle the fallback where clone3 is not available. This is
done by splitting __clone_internal_fallback from __clone_internal.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
All internal callers of __clone3 should provide an already aligned
stack. Removing the stack alignment in __clone3 is a net gain: it
simplifies the internal function contract (mask/unmask signals) along
with the arch-specific code.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Different than kernel, clone3 returns EINVAL for NULL struct
clone_args or function pointer. This is similar to clone
interface that return EINVAL for NULL function argument.
It also clean up the Linux clone3.h interface, since it not
currently exported.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
There is no need to issue another sigaction if the disposition is
already SIG_DFL.
Checked on x86_64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Occurs when `src` has no null-term.
Two cases:
1) Zero-length check is doing:
```
test %rdx, %rdx
jl L(zero_len)
```
which doesn't actually check zero (was at some point `decq` and the
flag never got updated).
The fix is just make the flag `jle` i.e:
```
test %rdx, %rdx
jle L(zero_len)
```
2) Length check in page-cross case checking if we should continue is
doing:
```
cmpq %r8, %rdx
jb L(page_cross_small)
```
which means we will continue searching for null-term if length ends at
the end of a page and there was no null-term in `src`.
The fix is to make the flag:
```
cmpq %r8, %rdx
jbe L(page_cross_small)
```
All AMD architectures cache details will be computed based on
__cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be
zeroed out for future architectures.
Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com>
Use shrn for narrowing the mask which simplifies code and speeds up small
strings. Unroll the first search loop to improve performance on large
strings.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Optimize strnlen using the shrn instruction and improve the main loop.
Small strings are around 10% faster, large strings are 40% faster on
modern CPUs.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Simplify calculation of the mask using shrn. Unroll the main loop.
Small strings are 20% faster on modern CPUs.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Use shrn for the mask, merge tst+bne into cbnz, and tweak code alignment.
Performance improves slightly as a result.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
The patch suppress the same warnings from 87c266d758,
that shows issues for microblaze, mips soft-fp, nios2, and or1k.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Fix the following issues with built-in function use in
sysdeps/ieee754/ldbl-128 and sysdeps/ieee754/float128:
* fabsl used __builtin_fabsf128 unconditionally, breaking the build
with GCC 6 for several architectures; it should use __builtin_fabsl
with an appropriate redirection in float128_private.h. (I'm not
particularly concerned with building glibc with GCC 6; rather, I
want to be able to run the tgmath.h tests with GCC 6, which is a
significantly different case for tgmath.h compared to GCC 7 and
later because of the lack of _FloatN / _FloatNx support in the
compiler, and at present running the tests with a compiler means
building glibc with that compiler.)
* Some (conditional) uses of built-in functions had been added to
ldbl-128 without appropriate float128_private.h remapping (there was
remapping for the macros controlling whether the built-in functions
are used, just not for the functions themselves).
* s_llrintl.c called __builtin_round not __builtin_llrintl, which is
obviously wrong.
Tested with build-many-glibcs.py for aarch64-linux-gnu, GCC 6 (where
it fixes the glibc build) and GCC 12, and with the glibc testsuite for
x86_64.
The minimum non_temporal_threshold is 0x4040. non_temporal_threshold may
be set to less than the minimum value when the shared cache size isn't
available (e.g., in an emulator) or by the tunable. Add checks for
minimum and maximum of non_temporal_threshold.
This fixes BZ #29953.
If the value changes between sem_wait's read and the gsync_wait call,
the kernel will return KERN_INVALID_ARGUMENT, which we have to interpret
as the value having already changed.
This fixes applications (e.g. libgo) seeing sem_wait erroneously return
KERN_INVALID_ARGUMENT.
