AuroraMiddleware/glibc
1
0
Fork 0
mirror of https://sourceware.org/git/glibc.git synced 2024-11-08 22:30:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
be77d5ae41
glibc/sysdeps/aarch64/memchr.S

129 lines
3.0 KiB
ArmAsm
Raw Normal View History

An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
/* memchr - find a character in a memory zone
Update copyright dates with scripts/update-copyrights
2024-01-01 18:12:26 +00:00
Copyright (C) 2015-2024 Free Software Foundation, Inc.
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library. If not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
<https://www.gnu.org/licenses/>. */
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
#include <sysdep.h>
/* Assumptions:
*
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
* ARMv8-a, AArch64, Advanced SIMD.
* MTE compatible.
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
*/
aarch64: Optimized memchr specific to AmpereComputing emag This version uses general register based memory instruction to load data, because vector register based is slightly slower in emag. Character-matching is performed on 16-byte (both size and alignment) memory block in parallel each iteration. * sysdeps/aarch64/memchr.S (__memchr): Rename to MEMCHR. [!MEMCHR](MEMCHR): Set to __memchr. * sysdeps/aarch64/multiarch/Makefile (sysdep_routines): Add memchr_generic and memchr_nosimd. * sysdeps/aarch64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add memchr ifuncs. * sysdeps/aarch64/multiarch/memchr.c: New file. * sysdeps/aarch64/multiarch/memchr_generic.S: Likewise. * sysdeps/aarch64/multiarch/memchr_nosimd.S: Likewise.
2018-08-09 08:38:03 +00:00
#ifndef MEMCHR
# define MEMCHR __memchr
#endif
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
#define srcin x0
#define chrin w1
#define cntin x2
#define result x0
#define src x3
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
#define cntrem x4
#define synd x5
#define shift x6
#define tmp x7
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
#define vrepchr v0
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
#define qdata q1
#define vdata v1
#define vhas_chr v2
aarch64: Optimize string functions with shrn instruction We found that string functions were using AND+ADDP to find the nibble/syndrome mask but there is an easier opportunity through `SHRN dst.8b, src.8h, 4` (shift right every 2 bytes by 4 and narrow to 1 byte) and has same latency on all SIMD ARMv8 targets as ADDP. There are also possible gaps for memcmp but that's for another patch. We see 10-20% savings for small-mid size cases (<=128) which are primary cases for general workloads.
2022-06-27 16:12:13 +00:00
#define vend v3
#define dend d3
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
/*
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
Core algorithm:
aarch64: Optimize string functions with shrn instruction We found that string functions were using AND+ADDP to find the nibble/syndrome mask but there is an easier opportunity through `SHRN dst.8b, src.8h, 4` (shift right every 2 bytes by 4 and narrow to 1 byte) and has same latency on all SIMD ARMv8 targets as ADDP. There are also possible gaps for memcmp but that's for another patch. We see 10-20% savings for small-mid size cases (<=128) which are primary cases for general workloads.
2022-06-27 16:12:13 +00:00
For each 16-byte chunk we calculate a 64-bit nibble mask value with four bits
per byte. We take 4 bits of every comparison byte with shift right and narrow
by 4 instruction. Since the bits in the nibble mask reflect the order in
which things occur in the original string, counting leading zeros identifies
exactly which byte matched. */
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
aarch64: Optimized memchr specific to AmpereComputing emag This version uses general register based memory instruction to load data, because vector register based is slightly slower in emag. Character-matching is performed on 16-byte (both size and alignment) memory block in parallel each iteration. * sysdeps/aarch64/memchr.S (__memchr): Rename to MEMCHR. [!MEMCHR](MEMCHR): Set to __memchr. * sysdeps/aarch64/multiarch/Makefile (sysdep_routines): Add memchr_generic and memchr_nosimd. * sysdeps/aarch64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add memchr ifuncs. * sysdeps/aarch64/multiarch/memchr.c: New file. * sysdeps/aarch64/multiarch/memchr_generic.S: Likewise. * sysdeps/aarch64/multiarch/memchr_nosimd.S: Likewise.
2018-08-09 08:38:03 +00:00
ENTRY (MEMCHR)
aarch64: use PTR_ARG and SIZE_ARG instead of DELOUSE DELOUSE was added to asm code to make them compatible with non-LP64 ABIs, but it is an unfortunate name and the code was not compatible with ABIs where pointer and size_t are different. Glibc currently only supports the LP64 ABI so these macros are not really needed or tested, but for now the name is changed to be more meaningful instead of removing them completely. Some DELOUSE macros were dropped: clone, strlen and strnlen used it unnecessarily. The out of tree ILP32 patches are currently not maintained and will likely need a rework to rebase them on top of the time64 changes.
2020-12-17 10:03:05 +00:00
PTR_ARG (0)
SIZE_ARG (2)
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
bic src, srcin, 15
cbz cntin, L(nomatch)
ld1 {vdata.16b}, [src]
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
dup vrepchr.16b, chrin
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
lsl shift, srcin, 2
aarch64: Optimize string functions with shrn instruction We found that string functions were using AND+ADDP to find the nibble/syndrome mask but there is an easier opportunity through `SHRN dst.8b, src.8h, 4` (shift right every 2 bytes by 4 and narrow to 1 byte) and has same latency on all SIMD ARMv8 targets as ADDP. There are also possible gaps for memcmp but that's for another patch. We see 10-20% savings for small-mid size cases (<=128) which are primary cases for general workloads.
2022-06-27 16:12:13 +00:00
shrn vend.8b, vhas_chr.8h, 4 /* 128->64 */
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
fmov synd, dend
lsr synd, synd, shift
cbz synd, L(start_loop)
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
rbit synd, synd
clz synd, synd
cmp cntin, synd, lsr 2
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
add result, srcin, synd, lsr 2
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
csel result, result, xzr, hi
ret
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
.p2align 3
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
L(start_loop):
sub tmp, src, srcin
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
add tmp, tmp, 17
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
subs cntrem, cntin, tmp
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
b.lo L(nomatch)
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
/* Make sure that it won't overread by a 16-byte chunk */
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
tbz cntrem, 4, L(loop32_2)
sub src, src, 16
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
.p2align 4
L(loop32):
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
ldr qdata, [src, 32]!
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
umaxp vend.16b, vhas_chr.16b, vhas_chr.16b /* 128->64 */
fmov synd, dend
cbnz synd, L(end)
L(loop32_2):
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
ldr qdata, [src, 16]
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
subs cntrem, cntrem, 32
b.lo L(end_2)
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
umaxp vend.16b, vhas_chr.16b, vhas_chr.16b /* 128->64 */
fmov synd, dend
cbz synd, L(loop32)
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
L(end_2):
add src, src, 16
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
L(end):
aarch64: Optimize string functions with shrn instruction We found that string functions were using AND+ADDP to find the nibble/syndrome mask but there is an easier opportunity through `SHRN dst.8b, src.8h, 4` (shift right every 2 bytes by 4 and narrow to 1 byte) and has same latency on all SIMD ARMv8 targets as ADDP. There are also possible gaps for memcmp but that's for another patch. We see 10-20% savings for small-mid size cases (<=128) which are primary cases for general workloads.
2022-06-27 16:12:13 +00:00
shrn vend.8b, vhas_chr.8h, 4 /* 128->64 */
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
sub cntrem, src, srcin
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
fmov synd, dend
AArch64: Optimize memchr Optimize the main loop - large strings are 40% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 13:50:59 +00:00
sub cntrem, cntin, cntrem
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
#ifndef __AARCH64EB__
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
rbit synd, synd
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
#endif
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
clz synd, synd
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
cmp cntrem, synd, lsr 2
add result, src, synd, lsr 2
csel result, result, xzr, hi
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
ret
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
L(nomatch):
mov result, 0
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
ret
aarch64: MTE compatible memchr Add support for MTE to memchr. Regression tested with xcheck and benchmarked with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Co-authored-by: Gabor Kertesz <gabor.kertesz@arm.com>
2020-06-09 16:06:03 +00:00
aarch64: Optimized memchr specific to AmpereComputing emag This version uses general register based memory instruction to load data, because vector register based is slightly slower in emag. Character-matching is performed on 16-byte (both size and alignment) memory block in parallel each iteration. * sysdeps/aarch64/memchr.S (__memchr): Rename to MEMCHR. [!MEMCHR](MEMCHR): Set to __memchr. * sysdeps/aarch64/multiarch/Makefile (sysdep_routines): Add memchr_generic and memchr_nosimd. * sysdeps/aarch64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add memchr ifuncs. * sysdeps/aarch64/multiarch/memchr.c: New file. * sysdeps/aarch64/multiarch/memchr_generic.S: Likewise. * sysdeps/aarch64/multiarch/memchr_nosimd.S: Likewise.
2018-08-09 08:38:03 +00:00
END (MEMCHR)
weak_alias (MEMCHR, memchr)
An optimized memchr was missing for AArch64. This version is similar to strchr and is significantly faster than the C version. 2016-11-04 Wilco Dijkstra <wdijkstr@arm.com> Kevin Petit <kevin.petit@arm.com> * sysdeps/aarch64/memchr.S (__memchr): New file.
2016-11-04 14:37:10 +00:00
libc_hidden_builtin_def (memchr)
Reference in New Issue Copy Permalink