AuroraMiddleware/glibc
1
0
Fork 0
mirror of https://sourceware.org/git/glibc.git synced 2024-11-22 21:10:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
642933158e
glibc/sysdeps/x86_64/multiarch/Makefile

185 lines
3.6 KiB
Makefile
Raw Normal View History

Add SSE4.2 support for strcmp and strncmp on x86-64.
2009-06-23 03:38:41 +00:00
ifeq ($(subdir),string)
Improved st{r,p}{,n}cpy for SSE2 and SSSE3 on x86-64
2011-06-24 19:14:22 +00:00
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
memchr-avx2 \
memchr-avx2-rtm \
memchr-evex \
x86_64: Implement evex512 version of memchr, rawmemchr and wmemchr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - memchr function using 512 bit vectors. - rawmemchr function using 512 bit vectors. - wmemchr function using 512 bit vectors. Code size data: memchr-evex.o 762 byte memchr-evex512.o 576 byte (-24%) rawmemchr-evex.o 461 byte rawmemchr-evex512.o 412 byte (-11%) wmemchr-evex.o 794 byte wmemchr-evex512.o 552 byte (-30%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-18 13:48:07 +00:00
memchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memchr-evex-rtm \
memchr-sse2 \
memcmp-avx2-movbe \
memcmp-avx2-movbe-rtm \
memcmp-evex-movbe \
memcmp-sse2 \
memcmpeq-avx2 \
memcmpeq-avx2-rtm \
memcmpeq-evex \
memcmpeq-sse2 \
memmove-avx-unaligned-erms \
memmove-avx-unaligned-erms-rtm \
memmove-avx512-no-vzeroupper \
memmove-avx512-unaligned-erms \
x86: Move mem{p}{mov|cpy}_{chk_}erms to its own file The primary memmove_{impl}_unaligned_erms implementations don't interact with this function. Putting them in same file both wastes space and unnecessarily bloats a hot code section.
2022-06-29 23:07:15 +00:00
memmove-erms \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memmove-evex-unaligned-erms \
memmove-sse2-unaligned-erms \
memmove-ssse3 \
memrchr-avx2 \
memrchr-avx2-rtm \
memrchr-evex \
memrchr-sse2 \
memset-avx2-unaligned-erms \
memset-avx2-unaligned-erms-rtm \
memset-avx512-no-vzeroupper \
memset-avx512-unaligned-erms \
x86: Move and slightly improve memset_erms Implementation wise: 1. Remove the VZEROUPPER as memset_{impl}_unaligned_erms does not use the L(stosb) label that was previously defined. 2. Don't give the hotpath (fallthrough) to zero size. Code positioning wise: Move memset_{chk}_erms to its own file. Leaving it in between the memset_{impl}_unaligned both adds unnecessary complexity to the file and wastes space in a relatively hot cache section.
2022-06-29 23:07:05 +00:00
memset-erms \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memset-evex-unaligned-erms \
memset-sse2-unaligned-erms \
rawmemchr-avx2 \
rawmemchr-avx2-rtm \
rawmemchr-evex \
x86_64: Implement evex512 version of memchr, rawmemchr and wmemchr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - memchr function using 512 bit vectors. - rawmemchr function using 512 bit vectors. - wmemchr function using 512 bit vectors. Code size data: memchr-evex.o 762 byte memchr-evex512.o 576 byte (-24%) rawmemchr-evex.o 461 byte rawmemchr-evex512.o 412 byte (-11%) wmemchr-evex.o 794 byte wmemchr-evex512.o 552 byte (-30%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-18 13:48:07 +00:00
rawmemchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
rawmemchr-evex-rtm \
rawmemchr-sse2 \
stpcpy-avx2 \
stpcpy-avx2-rtm \
stpcpy-evex \
stpcpy-sse2 \
stpcpy-sse2-unaligned \
stpncpy-avx2 \
stpncpy-avx2-rtm \
stpncpy-evex \
stpncpy-sse2-unaligned \
x86: Add AVX2 optimized str{n}casecmp geometric_mean(N=40) of all benchmarks AVX2 / SSE42: .702 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:12 +00:00
strcasecmp_l-avx2 \
strcasecmp_l-avx2-rtm \
x86: Add EVEX optimized str{n}casecmp geometric_mean(N=40) of all benchmarks EVEX / SSE42: .621 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:13 +00:00
strcasecmp_l-evex \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strcasecmp_l-sse2 \
strcasecmp_l-sse4_2 \
strcat-avx2 \
strcat-avx2-rtm \
strcat-evex \
strcat-sse2 \
strcat-sse2-unaligned \
strchr-avx2 \
strchr-avx2-rtm \
strchr-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
strchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strchr-sse2 \
strchr-sse2-no-bsf \
strchrnul-avx2 \
strchrnul-avx2-rtm \
strchrnul-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
strchrnul-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strchrnul-sse2 \
strcmp-avx2 \
strcmp-avx2-rtm \
strcmp-evex \
strcmp-sse2 \
strcmp-sse2-unaligned \
strcmp-sse4_2 \
strcpy-avx2 \
strcpy-avx2-rtm \
strcpy-evex \
strcpy-sse2 \
strcpy-sse2-unaligned \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strcspn-sse4 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strlen-avx2 \
strlen-avx2-rtm \
strlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
strlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strlen-sse2 \
x86: Add AVX2 optimized str{n}casecmp geometric_mean(N=40) of all benchmarks AVX2 / SSE42: .702 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:12 +00:00
strncase_l-avx2 \
strncase_l-avx2-rtm \
x86: Add EVEX optimized str{n}casecmp geometric_mean(N=40) of all benchmarks EVEX / SSE42: .621 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:13 +00:00
strncase_l-evex \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strncase_l-sse2 \
strncase_l-sse4_2 \
strncat-avx2 \
strncat-avx2-rtm \
strncat-evex \
strncat-sse2-unaligned \
strncmp-avx2 \
strncmp-avx2-rtm \
strncmp-evex \
strncmp-sse2 \
strncmp-sse4_2 \
strncpy-avx2 \
strncpy-avx2-rtm \
strncpy-evex \
strncpy-sse2-unaligned \
strnlen-avx2 \
strnlen-avx2-rtm \
strnlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
strnlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strnlen-sse2 \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strpbrk-sse4 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strrchr-avx2 \
strrchr-avx2-rtm \
strrchr-evex \
x86_64: Implement evex512 version of strrchr and wcsrchr Changes from v1: Use vec api for register. Replace VPCMP with VPCMPEQ Restructure and remove 1 unconditional jump. Change page cross logic to use sall. This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strrchr function using 512 bit vectors. - wcsrchr function using 512 bit vectors. Code size data: strrchr-evex.o 879 byte strrchr-evex512.o 601 byte (-32%) wcsrchr-evex.o 882 byte wcsrchr-evex512.o 572 byte (-35%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-09 14:57:29 +00:00
strrchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strrchr-sse2 \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strspn-sse4 \
x86_64: Add strstr function with 512-bit EVEX Adding a 512-bit EVEX version of strstr. The algorithm works as follows: (1) We spend a few cycles at the begining to peek into the needle. We locate an edge in the needle (first occurance of 2 consequent distinct characters) and also store the first 64-bytes into a zmm register. (2) We search for the edge in the haystack by looking into one cache line of the haystack at a time. This avoids having to read past a page boundary which can cause a seg fault. (3) If an edge is found in the haystack we first compare the first 64-bytes of the needle (already stored in a zmm register) before we proceed with a full string compare performed byte by byte. Benchmarking results: (old = strstr_sse2_unaligned, new = strstr_avx512) Geometric mean of all benchmarks: new / old = 0.66 Difficult skiptable(0) : new / old = 0.02 Difficult skiptable(1) : new / old = 0.01 Difficult 2-way : new / old = 0.25 Difficult testing first 2 : new / old = 1.26 Difficult skiptable(0) : new / old = 0.05 Difficult skiptable(1) : new / old = 0.06 Difficult 2-way : new / old = 0.26 Difficult testing first 2 : new / old = 1.05 Difficult skiptable(0) : new / old = 0.42 Difficult skiptable(1) : new / old = 0.24 Difficult 2-way : new / old = 0.21 Difficult testing first 2 : new / old = 1.04 Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-06-06 19:17:43 +00:00
strstr-avx512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strstr-sse2-unaligned \
varshift \
# sysdep_routines
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
CFLAGS-strcspn-sse4.c += -msse4
CFLAGS-strpbrk-sse4.c += -msse4
CFLAGS-strspn-sse4.c += -msse4
x86_64: Add strstr function with 512-bit EVEX Adding a 512-bit EVEX version of strstr. The algorithm works as follows: (1) We spend a few cycles at the begining to peek into the needle. We locate an edge in the needle (first occurance of 2 consequent distinct characters) and also store the first 64-bytes into a zmm register. (2) We search for the edge in the haystack by looking into one cache line of the haystack at a time. This avoids having to read past a page boundary which can cause a seg fault. (3) If an edge is found in the haystack we first compare the first 64-bytes of the needle (already stored in a zmm register) before we proceed with a full string compare performed byte by byte. Benchmarking results: (old = strstr_sse2_unaligned, new = strstr_avx512) Geometric mean of all benchmarks: new / old = 0.66 Difficult skiptable(0) : new / old = 0.02 Difficult skiptable(1) : new / old = 0.01 Difficult 2-way : new / old = 0.25 Difficult testing first 2 : new / old = 1.26 Difficult skiptable(0) : new / old = 0.05 Difficult skiptable(1) : new / old = 0.06 Difficult 2-way : new / old = 0.26 Difficult testing first 2 : new / old = 1.05 Difficult skiptable(0) : new / old = 0.42 Difficult skiptable(1) : new / old = 0.24 Difficult 2-way : new / old = 0.21 Difficult testing first 2 : new / old = 1.04 Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-06-06 19:17:43 +00:00
CFLAGS-strstr-avx512.c += -mavx512f -mavx512vl -mavx512dq -mavx512bw -mbmi -mbmi2 -O3
Add SSE4.2 support for strcmp and strncmp on x86-64.
2009-06-23 03:38:41 +00:00
endif
Optimized wcschr and wcscpy for x86-64 and x86-32
2011-12-17 19:39:23 +00:00
ifeq ($(subdir),wcsmbs)
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
wcschr-avx2 \
wcschr-avx2-rtm \
wcschr-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
wcschr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcschr-sse2 \
wcscmp-avx2 \
wcscmp-avx2-rtm \
wcscmp-evex \
wcscmp-sse2 \
wcscpy-ssse3 \
wcslen-avx2 \
wcslen-avx2-rtm \
wcslen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
wcslen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcslen-sse2 \
wcslen-sse4_1 \
wcsncmp-avx2 \
wcsncmp-avx2-rtm \
wcsncmp-evex \
wcsnlen-avx2 \
wcsnlen-avx2-rtm \
wcsnlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
wcsnlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsnlen-sse4_1 \
wcsrchr-avx2 \
wcsrchr-avx2-rtm \
wcsrchr-evex \
x86_64: Implement evex512 version of strrchr and wcsrchr Changes from v1: Use vec api for register. Replace VPCMP with VPCMPEQ Restructure and remove 1 unconditional jump. Change page cross logic to use sall. This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strrchr function using 512 bit vectors. - wcsrchr function using 512 bit vectors. Code size data: strrchr-evex.o 879 byte strrchr-evex512.o 601 byte (-32%) wcsrchr-evex.o 882 byte wcsrchr-evex512.o 572 byte (-35%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-09 14:57:29 +00:00
wcsrchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsrchr-sse2 \
wmemchr-avx2 \
wmemchr-avx2-rtm \
wmemchr-evex \
x86_64: Implement evex512 version of memchr, rawmemchr and wmemchr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - memchr function using 512 bit vectors. - rawmemchr function using 512 bit vectors. - wmemchr function using 512 bit vectors. Code size data: memchr-evex.o 762 byte memchr-evex512.o 576 byte (-24%) rawmemchr-evex.o 461 byte rawmemchr-evex512.o 412 byte (-11%) wmemchr-evex.o 794 byte wmemchr-evex512.o 552 byte (-30%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-18 13:48:07 +00:00
wmemchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wmemchr-evex-rtm \
wmemchr-sse2 \
wmemcmp-avx2-movbe \
wmemcmp-avx2-movbe-rtm \
wmemcmp-evex-movbe \
x86: Optimize memcmp SSE2 in memcmp.S New code save size (-303 bytes) and has significantly better performance. geometric_mean(N=20) of page cross cases New / Original: 0.634 Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-04-15 17:27:59 +00:00
wmemcmp-sse2 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
# sysdep_routines
Optimized wcschr and wcscpy for x86-64 and x86-32
2011-12-17 19:39:23 +00:00
endif
x86-64: Optimize wmemset with SSE2/AVX2/AVX512 The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise.
2017-06-05 18:09:48 +00:00
ifeq ($(subdir),debug)
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
memcpy_chk-nonshared \
memmove_chk-nonshared \
mempcpy_chk-nonshared \
memset_chk-nonshared \
wmemset_chk-nonshared \
# sysdep_routines
x86-64: Optimize wmemset with SSE2/AVX2/AVX512 The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise.
2017-06-05 18:09:48 +00:00
endif
Reference in New Issue Copy Permalink