AuroraMiddleware/glibc
1
0
Fork 0
mirror of https://sourceware.org/git/glibc.git synced 2024-11-22 13:00:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
c19457aec6
glibc/sysdeps/x86_64/multiarch/Makefile

200 lines
3.8 KiB
Makefile
Raw Normal View History

Add SSE4.2 support for strcmp and strncmp on x86-64.
2009-06-23 03:38:41 +00:00
ifeq ($(subdir),string)
Improved st{r,p}{,n}cpy for SSE2 and SSSE3 on x86-64
2011-06-24 19:14:22 +00:00
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
memchr-avx2 \
memchr-avx2-rtm \
memchr-evex \
memchr-evex-rtm \
Apply the Makefile sorting fix Apply the Makefile sorting fix generated by sort-makefile-lines.py.
2024-02-15 19:19:56 +00:00
memchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memchr-sse2 \
memcmp-avx2-movbe \
memcmp-avx2-movbe-rtm \
memcmp-evex-movbe \
memcmp-sse2 \
memcmpeq-avx2 \
memcmpeq-avx2-rtm \
memcmpeq-evex \
memcmpeq-sse2 \
memmove-avx-unaligned-erms \
memmove-avx-unaligned-erms-rtm \
memmove-avx512-no-vzeroupper \
memmove-avx512-unaligned-erms \
x86: Move mem{p}{mov|cpy}_{chk_}erms to its own file The primary memmove_{impl}_unaligned_erms implementations don't interact with this function. Putting them in same file both wastes space and unnecessarily bloats a hot code section.
2022-06-29 23:07:15 +00:00
memmove-erms \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memmove-evex-unaligned-erms \
memmove-sse2-unaligned-erms \
memmove-ssse3 \
memrchr-avx2 \
memrchr-avx2-rtm \
memrchr-evex \
memrchr-sse2 \
memset-avx2-unaligned-erms \
memset-avx2-unaligned-erms-rtm \
memset-avx512-no-vzeroupper \
memset-avx512-unaligned-erms \
x86: Move and slightly improve memset_erms Implementation wise: 1. Remove the VZEROUPPER as memset_{impl}_unaligned_erms does not use the L(stosb) label that was previously defined. 2. Don't give the hotpath (fallthrough) to zero size. Code positioning wise: Move memset_{chk}_erms to its own file. Leaving it in between the memset_{impl}_unaligned both adds unnecessary complexity to the file and wastes space in a relatively hot cache section.
2022-06-29 23:07:05 +00:00
memset-erms \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
memset-evex-unaligned-erms \
memset-sse2-unaligned-erms \
rawmemchr-avx2 \
rawmemchr-avx2-rtm \
rawmemchr-evex \
rawmemchr-evex-rtm \
Apply the Makefile sorting fix Apply the Makefile sorting fix generated by sort-makefile-lines.py.
2024-02-15 19:19:56 +00:00
rawmemchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
rawmemchr-sse2 \
stpcpy-avx2 \
stpcpy-avx2-rtm \
stpcpy-evex \
stpcpy-sse2 \
stpcpy-sse2-unaligned \
stpncpy-avx2 \
stpncpy-avx2-rtm \
stpncpy-evex \
stpncpy-sse2-unaligned \
x86: Add AVX2 optimized str{n}casecmp geometric_mean(N=40) of all benchmarks AVX2 / SSE42: .702 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:12 +00:00
strcasecmp_l-avx2 \
strcasecmp_l-avx2-rtm \
x86: Add EVEX optimized str{n}casecmp geometric_mean(N=40) of all benchmarks EVEX / SSE42: .621 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:13 +00:00
strcasecmp_l-evex \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strcasecmp_l-sse2 \
strcasecmp_l-sse4_2 \
strcat-avx2 \
strcat-avx2-rtm \
strcat-evex \
strcat-sse2 \
strcat-sse2-unaligned \
strchr-avx2 \
strchr-avx2-rtm \
strchr-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
strchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strchr-sse2 \
strchr-sse2-no-bsf \
strchrnul-avx2 \
strchrnul-avx2-rtm \
strchrnul-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
strchrnul-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strchrnul-sse2 \
strcmp-avx2 \
strcmp-avx2-rtm \
strcmp-evex \
strcmp-sse2 \
strcmp-sse2-unaligned \
strcmp-sse4_2 \
strcpy-avx2 \
strcpy-avx2-rtm \
strcpy-evex \
strcpy-sse2 \
strcpy-sse2-unaligned \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strcspn-sse4 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strlen-avx2 \
strlen-avx2-rtm \
strlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
strlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strlen-sse2 \
x86: Add AVX2 optimized str{n}casecmp geometric_mean(N=40) of all benchmarks AVX2 / SSE42: .702 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:12 +00:00
strncase_l-avx2 \
strncase_l-avx2-rtm \
x86: Add EVEX optimized str{n}casecmp geometric_mean(N=40) of all benchmarks EVEX / SSE42: .621 All string/memory tests pass. Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-03-24 23:56:13 +00:00
strncase_l-evex \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strncase_l-sse2 \
strncase_l-sse4_2 \
strncat-avx2 \
strncat-avx2-rtm \
strncat-evex \
strncat-sse2-unaligned \
strncmp-avx2 \
strncmp-avx2-rtm \
strncmp-evex \
strncmp-sse2 \
strncmp-sse4_2 \
strncpy-avx2 \
strncpy-avx2-rtm \
strncpy-evex \
strncpy-sse2-unaligned \
strnlen-avx2 \
strnlen-avx2-rtm \
strnlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
strnlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strnlen-sse2 \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strpbrk-sse4 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strrchr-avx2 \
strrchr-avx2-rtm \
strrchr-evex \
x86_64: Implement evex512 version of strrchr and wcsrchr Changes from v1: Use vec api for register. Replace VPCMP with VPCMPEQ Restructure and remove 1 unconditional jump. Change page cross logic to use sall. This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strrchr function using 512 bit vectors. - wcsrchr function using 512 bit vectors. Code size data: strrchr-evex.o 879 byte strrchr-evex512.o 601 byte (-32%) wcsrchr-evex.o 882 byte wcsrchr-evex512.o 572 byte (-35%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-09 14:57:29 +00:00
strrchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strrchr-sse2 \
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
strspn-sse4 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
strstr-sse2-unaligned \
varshift \
# sysdep_routines
x86: Rename generic functions with unique postfix for clarity No functions are changed. It just renames generic implementations from '{func}_sse2' to '{func}_generic'. This is just because the postfix "_sse2" was overloaded and was used for files that had hand-optimized sse2 assembly implementations and files that just redirected back to the generic implementation. Full xcheck passed on x86_64.
2022-06-16 22:07:12 +00:00
CFLAGS-strcspn-sse4.c += -msse4
CFLAGS-strpbrk-sse4.c += -msse4
CFLAGS-strspn-sse4.c += -msse4
Add SSE4.2 support for strcmp and strncmp on x86-64.
2009-06-23 03:38:41 +00:00
endif
Optimized wcschr and wcscpy for x86-64 and x86-32
2011-12-17 19:39:23 +00:00
ifeq ($(subdir),wcsmbs)
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcpcpy-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcpcpy-evex \
wcpcpy-generic \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcpncpy-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcpncpy-evex \
wcpncpy-generic \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcscat-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcscat-evex \
wcscat-generic \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcschr-avx2 \
wcschr-avx2-rtm \
wcschr-evex \
x86_64: Implement evex512 version of strchrnul, strchr and wcschr This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strchrnul function using 512 bit vectors. - strchr function using 512 bit vectors. - wcschr function using 512 bit vectors. Code size data: strchrnul-evex.o 599 byte strchrnul-evex512.o 569 byte (-5%) strchr-evex.o 639 byte strchr-evex512.o 595 byte (-7%) wcschr-evex.o 644 byte wcschr-evex512.o 607 byte (-6%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-07-26 20:54:56 +00:00
wcschr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcschr-sse2 \
wcscmp-avx2 \
wcscmp-avx2-rtm \
wcscmp-evex \
wcscmp-sse2 \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcscpy-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcscpy-evex \
wcscpy-generic \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcscpy-ssse3 \
wcslen-avx2 \
wcslen-avx2-rtm \
wcslen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
wcslen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcslen-sse2 \
wcslen-sse4_1 \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcsncat-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcsncat-evex \
wcsncat-generic \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsncmp-avx2 \
wcsncmp-avx2-rtm \
wcsncmp-evex \
x86: Add avx2 optimized functions for the wchar_t strcpy family Implemented: wcscat-avx2 (+ 744 bytes wcscpy-avx2 (+ 539 bytes) wcpcpy-avx2 (+ 577 bytes) wcsncpy-avx2 (+1108 bytes) wcpncpy-avx2 (+1214 bytes) wcsncat-avx2 (+1085 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-avx2 -> 0.975 wcscpy-avx2 -> 0.591 wcpcpy-avx2 -> 0.698 wcsncpy-avx2 -> 0.730 wcpncpy-avx2 -> 0.711 wcsncat-avx2 -> 0.954 Code Size Changes: This change increase the size of libc.so by ~5.5kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.2kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:41 +00:00
wcsncpy-avx2 \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wcsncpy-evex \
wcsncpy-generic \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsnlen-avx2 \
wcsnlen-avx2-rtm \
wcsnlen-evex \
x86_64: Implement evex512 version of strlen, strnlen, wcslen and wcsnlen This patch implements following evex512 version of string functions. Perf gain for evex512 version is up to 50% as compared to evex, depending on length and alignment. Placeholder function, not used by any processor at the moment. - String length function using 512 bit vectors. - String N length using 512 bit vectors. - Wide string length using 512 bit vectors. - Wide string N length using 512 bit vectors. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-02-28 00:39:47 +00:00
wcsnlen-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsnlen-sse4_1 \
wcsrchr-avx2 \
wcsrchr-avx2-rtm \
wcsrchr-evex \
x86_64: Implement evex512 version of strrchr and wcsrchr Changes from v1: Use vec api for register. Replace VPCMP with VPCMPEQ Restructure and remove 1 unconditional jump. Change page cross logic to use sall. This patch implements following evex512 version of string functions. evex512 version takes up to 30% less cycle as compared to evex, depending on length and alignment. - strrchr function using 512 bit vectors. - wcsrchr function using 512 bit vectors. Code size data: strrchr-evex.o 879 byte strrchr-evex512.o 601 byte (-32%) wcsrchr-evex.o 882 byte wcsrchr-evex512.o 572 byte (-35%) Placeholder function, not used by any processor at the moment. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
2022-08-09 14:57:29 +00:00
wcsrchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wcsrchr-sse2 \
wmemchr-avx2 \
wmemchr-avx2-rtm \
wmemchr-evex \
wmemchr-evex-rtm \
x86: Add evex optimized functions for the wchar_t strcpy family Implemented: wcscat-evex (+ 905 bytes) wcscpy-evex (+ 674 bytes) wcpcpy-evex (+ 709 bytes) wcsncpy-evex (+1358 bytes) wcpncpy-evex (+1467 bytes) wcsncat-evex (+1213 bytes) Performance Changes: Times are from N = 10 runs of the benchmark suite and are reported as geometric mean of all ratios of New Implementation / Best Old Implementation. Best Old Implementation was determined with the highest ISA implementation. wcscat-evex -> 0.991 wcscpy-evex -> 0.587 wcpcpy-evex -> 0.695 wcsncpy-evex -> 0.719 wcpncpy-evex -> 0.694 wcsncat-evex -> 0.979 Code Size Changes: This change increase the size of libc.so by ~6.3kb bytes. For reference the patch optimizing the normal strcpy family functions decreases libc.so by ~5.7kb. Full check passes on x86-64 and build succeeds for all ISA levels w/ and w/o multiarch.
2022-11-09 01:38:40 +00:00
wmemchr-evex512 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
wmemchr-sse2 \
wmemcmp-avx2-movbe \
wmemcmp-avx2-movbe-rtm \
wmemcmp-evex-movbe \
x86: Optimize memcmp SSE2 in memcmp.S New code save size (-303 bytes) and has significantly better performance. geometric_mean(N=20) of page cross cases New / Original: 0.634 Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
2022-04-15 17:27:59 +00:00
wmemcmp-sse2 \
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
# sysdep_routines
Optimized wcschr and wcscpy for x86-64 and x86-32
2011-12-17 19:39:23 +00:00
endif
x86-64: Optimize wmemset with SSE2/AVX2/AVX512 The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise.
2017-06-05 18:09:48 +00:00
ifeq ($(subdir),debug)
x86_64/multiarch: Sort sysdep_routines and put one entry per line
2022-02-05 19:52:33 +00:00
sysdep_routines += \
memcpy_chk-nonshared \
memmove_chk-nonshared \
mempcpy_chk-nonshared \
memset_chk-nonshared \
wmemset_chk-nonshared \
# sysdep_routines
x86-64: Optimize wmemset with SSE2/AVX2/AVX512 The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise.
2017-06-05 18:09:48 +00:00
endif
Reference in New Issue Copy Permalink