H.J. Lu
3c9a4cd16c
Don't set %rcx twice before "rep movsb"
...
* sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S (MEMCPY):
Don't set %rcx twice before "rep movsb".
2016-03-22 08:36:16 -07:00
H.J. Lu
fee9eb6200
Group AVX512 functions in .text.avx512 section
...
* sysdeps/x86_64/multiarch/memcpy-avx512-no-vzeroupper.S:
Replace .text with .text.avx512.
* sysdeps/x86_64/multiarch/memset-avx512-no-vzeroupper.S:
Likewise.
2016-03-06 16:48:11 -08:00
H.J. Lu
14a1d7cc4c
x86-64: Fix memcpy IFUNC selection
...
Chek Fast_Unaligned_Load, instead of Slow_BSF, and also check for
Fast_Copy_Backward to enable __memcpy_ssse3_back. Existing selection
order is updated with following selection order:
1. __memcpy_avx_unaligned if AVX_Fast_Unaligned_Load bit is set.
2. __memcpy_sse2_unaligned if Fast_Unaligned_Load bit is set.
3. __memcpy_sse2 if SSSE3 isn't available.
4. __memcpy_ssse3_back if Fast_Copy_Backward bit it set.
5. __memcpy_ssse3
[BZ #18880 ]
* sysdeps/x86_64/multiarch/memcpy.S: Check Fast_Unaligned_Load,
instead of Slow_BSF, and also check for Fast_Copy_Backward to
enable __memcpy_ssse3_back.
2016-03-04 08:39:07 -08:00
Andrew Senkevich
df782dc690
Fixed build with assembler w/o AVX-512 support.
...
* sysdeps/x86_64/multiarch/ifunc-impl-list.c: Fixed build with
assembler not supporting AVX-512.
2016-01-19 14:34:53 +03:00
Andrew Senkevich
214a44f394
Fixed typos in __memcpy_chk.
...
* sysdeps/x86_64/multiarch/memcpy_chk.S: Fixed typos.
2016-01-16 14:42:26 +03:00
Andrew Senkevich
72276d6e88
Added memcpy/memmove family optimized with AVX512 for KNL hardware.
...
Added AVX512 implementations of memcpy, mempcpy, memmove, memcpy_chk,
mempcpy_chk, memmove_chk.
It shows average improvement more than 30% over AVX versions on KNL
hardware (performance results in the thread
<https://sourceware.org/ml/libc-alpha/2016-01/msg00258.html >).
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Added new files.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c: Added new tests.
* sysdeps/x86_64/multiarch/memcpy-avx512-no-vzeroupper.S: New file.
* sysdeps/x86_64/multiarch/mempcpy-avx512-no-vzeroupper.S: Likewise.
* sysdeps/x86_64/multiarch/memmove-avx512-no-vzeroupper.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy.S: Added new IFUNC branch.
* sysdeps/x86_64/multiarch/memcpy_chk.S: Likewise.
* sysdeps/x86_64/multiarch/memmove.c: Likewise.
* sysdeps/x86_64/multiarch/memmove_chk.c: Likewise.
* sysdeps/x86_64/multiarch/mempcpy.S: Likewise.
* sysdeps/x86_64/multiarch/mempcpy_chk.S: Likewise.
2016-01-16 00:49:45 +03:00
Joseph Myers
f7a9f785e5
Update copyright dates with scripts/update-copyrights.
2016-01-04 16:05:18 +00:00
Andrew Senkevich
83d776f979
Added memset optimized with AVX512 for KNL hardware.
...
It shows improvement up to 28% over AVX2 memset (performance results
attached at <https://sourceware.org/ml/libc-alpha/2015-12/msg00052.html >).
* sysdeps/x86_64/multiarch/memset-avx512-no-vzeroupper.S: New file.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Added new file.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c: Added new tests.
* sysdeps/x86_64/multiarch/memset.S: Added new IFUNC branch.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
* sysdeps/x86/cpu-features.h (bit_Prefer_No_VZEROUPPER,
index_Prefer_No_VZEROUPPER): New.
* sysdeps/x86/cpu-features.c (init_cpu_features): Set the
Prefer_No_VZEROUPPER for Knights Landing.
2015-12-19 02:47:28 +03:00
Joseph Myers
c871b9b096
Remove -mavx2 configure tests.
...
There are configure tests for the -mavx2 compiler option. AVX2
support was added in GCC 4.7, so these tests are now obsolete; this
patch removes them.
Tested for x86_64 and x86 (testsuite, and that installed stripped
shared libraries are unchanged by the patch).
* sysdeps/i386/configure.ac (libc_cv_cc_avx2): Remove configure
test.
* sysdeps/i386/configure: Regenerated.
* sysdeps/x86_64/configure.ac (libc_cv_cc_avx2): Remove configure
test.
* sysdeps/x86_64/configure: Regenerated.
* config.h.in (HAVE_AVX2_SUPPORT): Remove #undef.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
memset-avx2 unconditionally instead of conditionally on
[$(config-cflags-avx2) = yes].
* sysdeps/x86_64/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list) [HAVE_AVX2_SUPPORT]: Make code
unconditional.
* sysdeps/x86_64/multiarch/memset.S [HAVE_AVX2_SUPPORT]: Likewise.
* sysdeps/x86_64/multiarch/memset_chk.S
[IS_IN (libc) && SHARED && HAVE_AVX2_SUPPORT]: Change conditional
to [IS_IN (libc) && SHARED].
2015-10-28 13:29:03 +00:00
Joseph Myers
1b12cd7f4d
Remove configure tests for AVX support.
...
GCC added support for -mavx and -msse2avx in version 4.4. Thus the
configure tests for this support are obsolete, and this patch removes
them.
Tested for x86_64 and x86 (testsuite, and that installed stripped
shared libraries are unchanged by this patch).
* sysdeps/i386/configure.ac (libc_cv_cc_avx): Remove configure
test.
(libc_cv_cc_sse2avx): Likewise.
* sysdeps/i386/configure: Regenerated.
* sysdeps/i386/i686/multiarch/Makefile
[$(subdir)$(config-cflags-avx) = mathyes]: Change conditional to
[$(subdir) = math].
* sysdeps/i386/i686/multiarch/s_fma-fma.c [HAVE_AVX_SUPPORT]: Make
code unconditional.
* sysdeps/i386/i686/multiarch/s_fma.c [HAVE_AVX_SUPPORT]:
Likewise.
* sysdeps/i386/i686/multiarch/s_fmaf-fma.c [HAVE_AVX_SUPPORT]:
Likewise.
* sysdeps/i386/i686/multiarch/s_fmaf.c [HAVE_AVX_SUPPORT]:
Likewise.
* sysdeps/x86_64/configure.ac (libc_cv_cc_avx): Remove configure
test.
(libc_cv_cc_sse2avx): Likewise.
* sysdeps/x86_64/configure: Regenerated.
* sysdeps/x86_64/Makefile [$(config-cflags-avx) = yes]: Make code
unconditional.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_profile)
[HAVE_AVX_SUPPORT || HAVE_AVX512_ASM_SUPPORT]: Make code
unconditional.
(_dl_runtime_profile)
[!(HAVE_AVX_SUPPORT || HAVE_AVX512_ASM_SUPPORT)]: Remove
conditional code.
* sysdeps/x86_64/fpu/multiarch/Makefile
[$(config-cflags-sse2avx) = yes]: Make code unconditional.
* sysdeps/x86_64/fpu/multiarch/e_atan2.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/fpu/multiarch/e_log.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_atan.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_fma.c [HAVE_AVX_SUPPORT]:
Likewise.
* sysdeps/x86_64/fpu/multiarch/s_fmaf.c [HAVE_AVX_SUPPORT]:
Likewise.
* sysdeps/x86_64/fpu/multiarch/s_sin.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_tan.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
* sysdeps/x86_64/multiarch/strcmp.S [HAVE_AVX_SUPPORT]: Likewise.
* config.h.in (HAVE_AVX_SUPPORT): Remove #undef.
(HAVE_SSE2AVX_SUPPORT): Likewise.
2015-10-08 15:59:32 +00:00
Joseph Myers
3b7aa5bf59
Remove configure tests for SSE4 support.
...
GCC added support for -msse4 in version 4.3. Thus the configure tests
for it are obsolete, and this patch removes them.
Tested for x86_64 and x86 (testsuite, and that installed stripped
shared libraries are unchanged by this patch).
* sysdeps/i386/configure.ac (libc_cv_cc_sse4): Remove configure
test.
* sysdeps/i386/configure: Regenerated.
* sysdeps/i386/i686/multiarch/Makefile
[$(config-cflags-sse4) = yes]: Make code unconditional.
* sysdeps/i386/i686/multiarch/strcspn.S [HAVE_SSE4_SUPPORT]:
Likewise.
* sysdeps/i386/i686/multiarch/strspn.S [HAVE_SSE4_SUPPORT]:
Likewise.
* sysdeps/x86_64/configure.ac (libc_cv_cc_sse4): Remove configure
test.
* sysdeps/x86_64/configure: Regenerated.
* sysdeps/x86_64/multiarch/Makefile [$(config-cflags-sse4) = yes]:
Make code unconditional.
* sysdeps/x86_64/multiarch/strcspn.S [HAVE_SSE4_SUPPORT]:
Likewise.
* sysdeps/x86_64/multiarch/strspn.S [HAVE_SSE4_SUPPORT]: Likewise.
* config.h.in (HAVE_SSE4_SUPPORT): Remove #undef.
2015-10-06 20:47:40 +00:00
H.J. Lu
2339c6f4bd
Remove x86-64 rtld-xxx.c and rtld-xxx.S
...
Since ld.so preserves vector registers now, we can use the regular,
non-ifunc string and memory functions in ld.so.
* sysdeps/x86_64/rtld-memcmp.c: Removed.
* sysdeps/x86_64/rtld-memset.S: Likewise.
* sysdeps/x86_64/rtld-strchr.S: Likewise.
* sysdeps/x86_64/rtld-strlen.S: Likewise.
* sysdeps/x86_64/multiarch/rtld-memcmp.c: Likewise.
* sysdeps/x86_64/multiarch/rtld-memset.S: Likewise.
2015-08-25 08:50:06 -07:00
H.J. Lu
daa4db69fc
Remove the unused IFUNC files
...
sysdeps/i386/i686/multiarch/strcasestr-c.c became unused after
commit 1818483b15584b18eb4d
2015-08-20 12:47:20 -07:00
H.J. Lu
1ae6c72dc1
Move x86_64 init-arch.h to sysdeps/x86/init-arch.h
...
Move sysdeps/x86_64/multiarch/init-arch.h to sysdeps/x86/init-arch.h
which can be used for both i386 and x86_64.
* sysdeps/i386/i686/multiarch/init-arch.h: Removed.
* sysdeps/unix/sysv/linux/x86/init-arch.h: Likewise.
* sysdeps/x86_64/cacheinfo.c: Include <init-arch.h> instead
of "multiarch/init-arch.h".
* sysdeps/x86_64/multiarch/init-arch.h: Renamed to ...
* sysdeps/x86/init-arch.h: This.
2015-08-20 04:29:23 -07:00
H.J. Lu
0b5395f052
Update x86_64 multiarch functions for <cpu-features.h>
...
This patch updates x86_64 multiarch functions to use the newly defined
HAS_CPU_FEATURE, HAS_ARCH_FEATURE and LOAD_RTLD_GLOBAL_RO_RDX from
<cpu-features.h>.
* sysdeps/x86_64/fpu/multiarch/e_asin.c: Replace HAS_XXX with
HAS_CPU_FEATURE/HAS_ARCH_FEATURE (XXX).
* sysdeps/x86_64/fpu/multiarch/e_atan2.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/e_log.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/e_pow.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_atan.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_fma.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_fmaf.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_sin.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_tan.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_ceil.S: Use
LOAD_RTLD_GLOBAL_RO_RDX and HAS_CPU_FEATURE (SSE4_1).
* sysdeps/x86_64/fpu/multiarch/s_ceilf.S: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_floor.S: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_floorf.S: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_nearbyint.S : Likewise.
* sysdeps/x86_64/fpu/multiarch/s_nearbyintf.S: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_rintf.S: Likewise.
* sysdeps/x86_64/fpu/multiarch/s_rintf.S : Likewise.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c: Likewise.
* sysdeps/x86_64/multiarch/sched_cpucount.c: Likewise.
* sysdeps/x86_64/multiarch/strstr.c: Likewise.
* sysdeps/x86_64/multiarch/memmove.c: Likewise.
* sysdeps/x86_64/multiarch/memmove_chk.c: Likewise.
* sysdeps/x86_64/multiarch/test-multiarch.c: Likewise.
* sysdeps/x86_64/multiarch/memcmp.S: Remove __init_cpu_features
call. Add LOAD_RTLD_GLOBAL_RO_RDX. Replace HAS_XXX with
HAS_CPU_FEATURE/HAS_ARCH_FEATURE (XXX).
* sysdeps/x86_64/multiarch/memcpy.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy_chk.S: Likewise.
* sysdeps/x86_64/multiarch/mempcpy.S: Likewise.
* sysdeps/x86_64/multiarch/mempcpy_chk.S: Likewise.
* sysdeps/x86_64/multiarch/memset.S: Likewise.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
* sysdeps/x86_64/multiarch/strcat.S: Likewise.
* sysdeps/x86_64/multiarch/strchr.S: Likewise.
* sysdeps/x86_64/multiarch/strcmp.S: Likewise.
* sysdeps/x86_64/multiarch/strcpy.S: Likewise.
* sysdeps/x86_64/multiarch/strcspn.S: Likewise.
* sysdeps/x86_64/multiarch/strspn.S: Likewise.
* sysdeps/x86_64/multiarch/wcscpy.S: Likewise.
* sysdeps/x86_64/multiarch/wmemcmp.S: Likewise.
2015-08-13 03:41:30 -07:00
H.J. Lu
e2e4f56056
Add _dl_x86_cpu_features to rtld_global
...
This patch adds _dl_x86_cpu_features to rtld_global in x86 ld.so
and initializes it early before __libc_start_main is called so that
cpu_features is always available when it is used and we can avoid
calling __init_cpu_features in IFUNC selectors.
* sysdeps/i386/dl-machine.h: Include <cpu-features.c>.
(dl_platform_init): Call init_cpu_features.
* sysdeps/i386/dl-procinfo.c (_dl_x86_cpu_features): New.
* sysdeps/i386/i686/cacheinfo.c
(DISABLE_PREFERRED_MEMORY_INSTRUCTION): Removed.
* sysdeps/i386/i686/multiarch/Makefile (aux): Remove init-arch.
* sysdeps/i386/i686/multiarch/Versions: Removed.
* sysdeps/i386/i686/multiarch/ifunc-defines.sym (KIND_OFFSET):
Removed.
* sysdeps/i386/ldsodefs.h: Include <cpu-features.h>.
* sysdeps/unix/sysv/linux/x86/Makefile
(libpthread-sysdep_routines): Remove init-arch.
* sysdeps/unix/sysv/linux/x86_64/dl-procinfo.c: Include
<sysdeps/x86_64/dl-procinfo.c> instead of
sysdeps/generic/dl-procinfo.c>.
* sysdeps/x86/Makefile [$(subdir) == csu] (gen-as-const-headers):
Add cpu-features-offsets.sym and rtld-global-offsets.sym.
[$(subdir) == elf] (sysdep-dl-routines): Add dl-get-cpu-features.
[$(subdir) == elf] (tests): Add tst-get-cpu-features.
[$(subdir) == elf] (tests-static): Add
tst-get-cpu-features-static.
* sysdeps/x86/Versions: New file.
* sysdeps/x86/cpu-features-offsets.sym: Likewise.
* sysdeps/x86/cpu-features.c: Likewise.
* sysdeps/x86/cpu-features.h: Likewise.
* sysdeps/x86/dl-get-cpu-features.c: Likewise.
* sysdeps/x86/libc-start.c: Likewise.
* sysdeps/x86/rtld-global-offsets.sym: Likewise.
* sysdeps/x86/tst-get-cpu-features-static.c: Likewise.
* sysdeps/x86/tst-get-cpu-features.c: Likewise.
* sysdeps/x86_64/dl-procinfo.c: Likewise.
* sysdeps/x86_64/cacheinfo.c (__cpuid_count): Removed.
Assume USE_MULTIARCH is defined and don't check it.
(is_intel): Replace __cpu_features with GLRO(dl_x86_cpu_features).
(is_amd): Likewise.
(max_cpuid): Likewise.
(intel_check_word): Likewise.
(__cache_sysconf): Don't call __init_cpu_features.
(__x86_preferred_memory_instruction): Removed.
(init_cacheinfo): Don't call __init_cpu_features. Replace
__cpu_features with GLRO(dl_x86_cpu_features).
* sysdeps/x86_64/dl-machine.h: <cpu-features.c>.
(dl_platform_init): Call init_cpu_features.
* sysdeps/x86_64/ldsodefs.h: Include <cpu-features.h>.
* sysdeps/x86_64/multiarch/Makefile (aux): Remove init-arch.
* sysdeps/x86_64/multiarch/Versions: Removed.
* sysdeps/x86_64/multiarch/cacheinfo.c: Likewise.
* sysdeps/x86_64/multiarch/init-arch.c: Likewise.
* sysdeps/x86_64/multiarch/ifunc-defines.sym (KIND_OFFSET):
Removed.
* sysdeps/x86_64/multiarch/init-arch.h: Rewrite.
2015-08-13 03:41:22 -07:00
H.J. Lu
3b8d2eb7f8
Compile {memcpy,strcmp}-sse2-unaligned.S only for libc
...
{memcpy,strcmp}-sse2-unaligned.S aren't needed in ld.so.
* sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: Compile
only for libc.
* sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S: Likewise.
2015-08-05 08:28:37 -07:00
Andrew Senkevich
5fe2a126d1
This patch adds detection of availability for AVX512F and AVX512DQ ISAs.
...
* sysdeps/x86_64/multiarch/init-arch.h (bit_AVX512F_Usable,
bit_AVX512DQ_Usable, bit_Opmask_state, bit_ZMM0_15_state,
bit_ZMM16_31_state): New macro.
* sysdeps/x86_64/multiarch/init-arch.c (__init_cpu_features):
Check and set bit_AVX512F_Usable, bit_AVX512DQ_Usable.
2015-06-08 14:07:59 +03:00
Ondřej Bílka
0f4840be25
Use strspn/strcspn/strpbrk ifunc in internal calls.
...
To make a strtok faster and improve performance in general we need to do one
additional change.
A comment:
/* It doesn't make sense to send libc-internal strcspn calls through a PLT.
The speedup we get from using SSE4.2 instruction is likely eaten away
by the indirect call in the PLT. */
Does not make sense at all because nobody bothered to check it. Gap
between these implementations is quite big, when haystack is empty a
sse2 is around 40 cycles slower because it needs to populate a lookup
table and difference only increases with size. That is much bigger than
plt slowdown which is few cycles.
Even benchtest show a gap which also may be reverse by branch
misprediction but my internal benchmark shown.
simple_strspn stupid_strspn __strspn_sse42 __strspn_sse2
Length 0, alignment 0, acc len 6: 18.6562 35.2344 17.0469 61.6719
Length 6, alignment 0, acc len 6: 59.5469 72.5781 16.4219 73.625
This patch also handles strpbrk which is implemented by including a
x86_64/multiarch/strcspn.S file.
* sysdeps/x86_64/multiarch/strspn.S: Remove plt indirection.
* sysdeps/x86_64/multiarch/strcspn.S: Likewise.
2015-05-12 20:18:51 +02:00
H.J. Lu
5f3d0b78e0
Use AVX unaligned memcpy only if AVX2 is available
...
memcpy with unaligned 256-bit AVX register loads/stores are slow on older
processorsl like Sandy Bridge. This patch adds bit_AVX_Fast_Unaligned_Load
and sets it only when AVX2 is available.
[BZ #17801 ]
* sysdeps/x86_64/multiarch/init-arch.c (__init_cpu_features):
Set the bit_AVX_Fast_Unaligned_Load bit for AVX2.
* sysdeps/x86_64/multiarch/init-arch.h (bit_AVX_Fast_Unaligned_Load):
New.
(index_AVX_Fast_Unaligned_Load): Likewise.
(HAS_AVX_FAST_UNALIGNED_LOAD): Likewise.
* sysdeps/x86_64/multiarch/memcpy.S (__new_memcpy): Check the
bit_AVX_Fast_Unaligned_Load bit instead of the bit_AVX_Usable bit.
* sysdeps/x86_64/multiarch/memcpy_chk.S (__memcpy_chk): Likewise.
* sysdeps/x86_64/multiarch/mempcpy.S (__mempcpy): Likewise.
* sysdeps/x86_64/multiarch/mempcpy_chk.S (__mempcpy_chk): Likewise.
* sysdeps/x86_64/multiarch/memmove.c (__libc_memmove): Replace
HAS_AVX with HAS_AVX_FAST_UNALIGNED_LOAD.
* sysdeps/x86_64/multiarch/memmove_chk.c (__memmove_chk): Likewise.
2015-01-30 15:37:58 -08:00
H.J. Lu
972af9e8dd
Also treat model numbers 0x5a/0x5d as Silvermont
2015-01-23 18:52:45 -08:00
H.J. Lu
ede0236c86
Treat model numbers 0x4a/0x4d as Silvermont
...
* sysdeps/x86_64/multiarch/init-arch.c (__init_cpu_features):
Treat model numbers 0x4a/0x4d as Intel Silvermont architecture.
2015-01-23 18:08:10 -08:00
Joseph Myers
b168057aaa
Update copyright dates with scripts/update-copyrights.
2015-01-02 16:29:47 +00:00
Siddhesh Poyarekar
4f41c682f3
Remove NOT_IN_libc
...
Replace with !IS_IN (libc). This completes the transition from
the IS_IN/NOT_IN macros to the IN_MODULE macro set.
The generated code is unchanged on x86_64.
* stdlib/isomac.c (fmt): Replace NOT_IN_libc with IN_MODULE.
(get_null_defines): Adjust.
* sunrpc/Makefile: Adjust comment.
* Makerules (CPPFLAGS-nonlib): Remove NOT_IN_libc.
* elf/Makefile (CPPFLAGS-sotruss-lib): Likewise.
(CFLAGS-interp.c): Likewise.
(CFLAGS-ldconfig.c): Likewise.
(CPPFLAGS-.os): Likewise.
* elf/rtld-Rules (rtld-CPPFLAGS): Likewise.
* extra-lib.mk (CPPFLAGS-$(lib)): Likewise.
* extra-modules.mk (extra-modules.mk): Likewise.
* iconv/Makefile (CPPFLAGS-iconvprogs): Likewise.
* locale/Makefile (CPPFLAGS-locale_programs): Likewise.
* malloc/Makefile (CPPFLAGS-memusagestat): Likewise.
* nscd/Makefile (CPPFLAGS-nscd): Likewise.
* nss/Makefile (CPPFLAGS-nss_test1): Likewise.
* stdlib/Makefile (CFLAGS-tst-putenvmod.c): Likewise.
* sysdeps/gnu/Makefile ($(objpfx)errlist-compat.c): Likewise.
* sysdeps/unix/sysv/linux/Makefile (CPPFLAGS-lddlibc4): Likewise.
* iconvdata/Makefile (CPPFLAGS): Likewise.
(cpp-srcs-left): Add libof for all iconvdata routines.
* bits/stdio-lock.h: Replace NOT_IN_libc with IS_IN.
* include/assert.h: Likewise.
* include/ctype.h: Likewise.
* include/errno.h: Likewise.
* include/libc-symbols.h: Likewise.
* include/math.h: Likewise.
* include/netdb.h: Likewise.
* include/resolv.h: Likewise.
* include/stdio.h: Likewise.
* include/stdlib.h: Likewise.
* include/string.h: Likewise.
* include/sys/stat.h: Likewise.
* include/wctype.h: Likewise.
* intl/l10nflist.c: Likewise.
* libidn/idn-stub.c: Likewise.
* libio/libioP.h: Likewise.
* nptl/libc_multiple_threads.c: Likewise.
* nptl/pthreadP.h: Likewise.
* posix/regex_internal.h: Likewise.
* resolv/res_hconf.c: Likewise.
* sysdeps/arm/armv7/multiarch/memcpy.S: Likewise.
* sysdeps/arm/memmove.S: Likewise.
* sysdeps/arm/sysdep.h: Likewise.
* sysdeps/generic/_itoa.h: Likewise.
* sysdeps/generic/symbol-hacks.h: Likewise.
* sysdeps/gnu/errlist.awk: Likewise.
* sysdeps/gnu/errlist.c: Likewise.
* sysdeps/i386/i586/memcpy.S: Likewise.
* sysdeps/i386/i586/memset.S: Likewise.
* sysdeps/i386/i686/memcpy.S: Likewise.
* sysdeps/i386/i686/memmove.S: Likewise.
* sysdeps/i386/i686/mempcpy.S: Likewise.
* sysdeps/i386/i686/memset.S: Likewise.
* sysdeps/i386/i686/multiarch/bcopy.S: Likewise.
* sysdeps/i386/i686/multiarch/bzero.S: Likewise.
* sysdeps/i386/i686/multiarch/memchr-sse2-bsf.S: Likewise.
* sysdeps/i386/i686/multiarch/memchr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/memchr.S: Likewise.
* sysdeps/i386/i686/multiarch/memcmp-sse4.S: Likewise.
* sysdeps/i386/i686/multiarch/memcmp-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/memcmp.S: Likewise.
* sysdeps/i386/i686/multiarch/memcpy-ssse3-rep.S: Likewise.
* sysdeps/i386/i686/multiarch/memcpy-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/memcpy.S: Likewise.
* sysdeps/i386/i686/multiarch/memcpy_chk.S: Likewise.
* sysdeps/i386/i686/multiarch/memmove.S: Likewise.
* sysdeps/i386/i686/multiarch/memmove_chk.S: Likewise.
* sysdeps/i386/i686/multiarch/mempcpy.S: Likewise.
* sysdeps/i386/i686/multiarch/mempcpy_chk.S: Likewise.
* sysdeps/i386/i686/multiarch/memrchr-c.c: Likewise.
* sysdeps/i386/i686/multiarch/memrchr-sse2-bsf.S: Likewise.
* sysdeps/i386/i686/multiarch/memrchr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/memrchr.S: Likewise.
* sysdeps/i386/i686/multiarch/memset-sse2-rep.S: Likewise.
* sysdeps/i386/i686/multiarch/memset-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/memset.S: Likewise.
* sysdeps/i386/i686/multiarch/memset_chk.S: Likewise.
* sysdeps/i386/i686/multiarch/rawmemchr.S: Likewise.
* sysdeps/i386/i686/multiarch/strcat-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/strcat-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/strcat.S: Likewise.
* sysdeps/i386/i686/multiarch/strchr-sse2-bsf.S: Likewise.
* sysdeps/i386/i686/multiarch/strchr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/strchr.S: Likewise.
* sysdeps/i386/i686/multiarch/strcmp-sse4.S: Likewise.
* sysdeps/i386/i686/multiarch/strcmp-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/strcmp.S: Likewise.
* sysdeps/i386/i686/multiarch/strcpy-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/strcpy-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/strcpy.S: Likewise.
* sysdeps/i386/i686/multiarch/strcspn.S: Likewise.
* sysdeps/i386/i686/multiarch/strlen-sse2-bsf.S: Likewise.
* sysdeps/i386/i686/multiarch/strlen-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/strlen.S: Likewise.
* sysdeps/i386/i686/multiarch/strnlen.S: Likewise.
* sysdeps/i386/i686/multiarch/strrchr-sse2-bsf.S: Likewise.
* sysdeps/i386/i686/multiarch/strrchr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/strrchr.S: Likewise.
* sysdeps/i386/i686/multiarch/strspn.S: Likewise.
* sysdeps/i386/i686/multiarch/wcschr-c.c: Likewise.
* sysdeps/i386/i686/multiarch/wcschr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/wcschr.S: Likewise.
* sysdeps/i386/i686/multiarch/wcscmp-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/wcscmp.S: Likewise.
* sysdeps/i386/i686/multiarch/wcscpy-c.c: Likewise.
* sysdeps/i386/i686/multiarch/wcscpy-ssse3.S: Likewise.
* sysdeps/i386/i686/multiarch/wcscpy.S: Likewise.
* sysdeps/i386/i686/multiarch/wcslen-c.c: Likewise.
* sysdeps/i386/i686/multiarch/wcslen-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/wcslen.S: Likewise.
* sysdeps/i386/i686/multiarch/wcsrchr-c.c: Likewise.
* sysdeps/i386/i686/multiarch/wcsrchr-sse2.S: Likewise.
* sysdeps/i386/i686/multiarch/wcsrchr.S: Likewise.
* sysdeps/i386/i686/multiarch/wmemcmp-c.c: Likewise.
* sysdeps/i386/i686/multiarch/wmemcmp.S: Likewise.
* sysdeps/ia64/fpu/libm-symbols.h: Likewise.
* sysdeps/nptl/bits/libc-lock.h: Likewise.
* sysdeps/nptl/bits/libc-lockP.h: Likewise.
* sysdeps/nptl/bits/stdio-lock.h: Likewise.
* sysdeps/posix/closedir.c: Likewise.
* sysdeps/posix/opendir.c: Likewise.
* sysdeps/posix/readdir.c: Likewise.
* sysdeps/posix/rewinddir.c: Likewise.
* sysdeps/powerpc/novmx-sigjmp.c: Likewise.
* sysdeps/powerpc/powerpc32/__longjmp.S: Likewise.
* sysdeps/powerpc/powerpc32/bsd-_setjmp.S: Likewise.
* sysdeps/powerpc/powerpc32/fpu/__longjmp.S: Likewise.
* sysdeps/powerpc/powerpc32/fpu/setjmp.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/bzero.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memchr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memcmp-ppc32.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memcmp.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memcpy-ppc32.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memcpy.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memmove.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/mempcpy.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memrchr-ppc32.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memrchr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memset-ppc32.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/memset.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/rawmemchr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strcasecmp.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strcasecmp_l.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strchr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strchrnul.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strlen-ppc32.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strlen.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strncase.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strncase_l.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strncmp-ppc32.S: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strncmp.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/strnlen.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcschr-ppc32.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcschr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcscpy-ppc32.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcscpy.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcsrchr-ppc32.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wcsrchr.c: Likewise.
* sysdeps/powerpc/powerpc32/power4/multiarch/wordcopy.c: Likewise.
* sysdeps/powerpc/powerpc32/power6/memset.S: Likewise.
* sysdeps/powerpc/powerpc32/setjmp.S: Likewise.
* sysdeps/powerpc/powerpc64/__longjmp.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/bzero.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memcmp-ppc64.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memcmp.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memcpy-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memcpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memmove-ppc64.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memmove.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/mempcpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memrchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memset-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/memset.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/rawmemchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/stpcpy-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/stpcpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/stpncpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcasecmp.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcasecmp_l.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcat.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strchrnul.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcmp-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcmp.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcpy-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strcspn.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strlen-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strlen.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncase.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncase_l.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncat.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncmp-ppc64.S: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncmp.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncpy-ppc64.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strncpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strnlen.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strpbrk.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strrchr-ppc64.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strrchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strspn-ppc64.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/strspn.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/wcschr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/wcscpy.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/wcsrchr.c: Likewise.
* sysdeps/powerpc/powerpc64/multiarch/wordcopy.c: Likewise.
* sysdeps/powerpc/powerpc64/setjmp.S: Likewise.
* sysdeps/s390/s390-32/multiarch/ifunc-resolve.c: Likewise.
* sysdeps/s390/s390-32/multiarch/memcmp.S: Likewise.
* sysdeps/s390/s390-32/multiarch/memcpy.S: Likewise.
* sysdeps/s390/s390-32/multiarch/memset.S: Likewise.
* sysdeps/s390/s390-64/multiarch/ifunc-resolve.c: Likewise.
* sysdeps/s390/s390-64/multiarch/memcmp.S: Likewise.
* sysdeps/s390/s390-64/multiarch/memcpy.S: Likewise.
* sysdeps/s390/s390-64/multiarch/memset.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memcpy-niagara1.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memcpy-niagara2.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memcpy-niagara4.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memcpy.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memset-niagara1.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memset-niagara4.S: Likewise.
* sysdeps/sparc/sparc64/multiarch/memset.S: Likewise.
* sysdeps/unix/alpha/sysdep.S: Likewise.
* sysdeps/unix/alpha/sysdep.h: Likewise.
* sysdeps/unix/make-syscalls.sh: Likewise.
* sysdeps/unix/sysv/linux/aarch64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/aarch64/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/alpha/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/alpha/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/arm/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/arm/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/getpid.c: Likewise.
* sysdeps/unix/sysv/linux/hppa/nptl/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/hppa/nptl/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/i386/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/ia64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/ia64/sysdep.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/lowlevellock-futex.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/bits/m68k-vdso.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/m68k/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/microblaze/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/microblaze/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/mips/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/not-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/longjmp_chk.c: Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sh/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/sh/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/sh/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/sh/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/sh/vfork.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/brk.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/tile/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/tile/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/tile/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/tile/waitpid.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep-cancel.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sysdep.h: Likewise.
* sysdeps/wordsize-32/symbol-hacks.h: Likewise.
* sysdeps/x86_64/memcpy.S: Likewise.
* sysdeps/x86_64/memmove.c: Likewise.
* sysdeps/x86_64/memset.S: Likewise.
* sysdeps/x86_64/multiarch/init-arch.h: Likewise.
* sysdeps/x86_64/multiarch/memcmp-sse4.S: Likewise.
* sysdeps/x86_64/multiarch/memcmp-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/memcmp.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy-ssse3-back.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy.S: Likewise.
* sysdeps/x86_64/multiarch/memcpy_chk.S: Likewise.
* sysdeps/x86_64/multiarch/memmove.c: Likewise.
* sysdeps/x86_64/multiarch/mempcpy.S: Likewise.
* sysdeps/x86_64/multiarch/mempcpy_chk.S: Likewise.
* sysdeps/x86_64/multiarch/memset-avx2.S: Likewise.
* sysdeps/x86_64/multiarch/memset.S: Likewise.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
* sysdeps/x86_64/multiarch/strcat-sse2-unaligned.S: Likewise.
* sysdeps/x86_64/multiarch/strcat-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/strcat.S: Likewise.
* sysdeps/x86_64/multiarch/strchr-sse2-no-bsf.S: Likewise.
* sysdeps/x86_64/multiarch/strchr.S: Likewise.
* sysdeps/x86_64/multiarch/strcmp-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/strcmp.S: Likewise.
* sysdeps/x86_64/multiarch/strcpy-sse2-unaligned.S: Likewise.
* sysdeps/x86_64/multiarch/strcpy-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/strcpy.S: Likewise.
* sysdeps/x86_64/multiarch/strcspn.S: Likewise.
* sysdeps/x86_64/multiarch/strspn.S: Likewise.
* sysdeps/x86_64/multiarch/wcscpy-c.c: Likewise.
* sysdeps/x86_64/multiarch/wcscpy-ssse3.S: Likewise.
* sysdeps/x86_64/multiarch/wcscpy.S: Likewise.
* sysdeps/x86_64/multiarch/wmemcmp-c.c: Likewise.
* sysdeps/x86_64/multiarch/wmemcmp.S: Likewise.
* sysdeps/x86_64/strcmp.S: Likewise.
2014-11-24 15:03:45 +05:30
Andreas Schwab
3574f2fdf3
Fix misdetected Slow_SSE4_2 cpu feature bit (bug 17501)
2014-10-27 10:44:28 +01:00
Ling Ma
05f3633da4
Improve 64bit memcpy performance for Haswell CPU with AVX instruction
...
In this patch we take advantage of HSW memory bandwidth, manage to
reduce miss branch prediction by avoiding using branch instructions and
force destination to be aligned with avx instruction.
The CPU2006 403.gcc benchmark indicates this patch improves performance
from 2% to 10%.
2014-07-30 08:02:35 -07:00
H.J. Lu
f2fef657d8
Enable AVX2 optimized memset only if -mavx2 works
...
* config.h.in (HAVE_AVX2_SUPPORT): New #undef.
* sysdeps/i386/configure.ac: Set HAVE_AVX2_SUPPORT and
config-cflags-avx2.
* sysdeps/x86_64/configure.ac: Likewise.
* sysdeps/i386/configure: Regenerated.
* sysdeps/x86_64/configure: Likewise.
* sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add
memset-avx2 only if config-cflags-avx2 is yes.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list):
Tests for memset_chk and memset only if HAVE_AVX2_SUPPORT is
defined.
* sysdeps/x86_64/multiarch/memset.S: Define multiple versions
only if HAVE_AVX2_SUPPORT is defined.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
2014-07-14 07:58:27 -07:00
H.J. Lu
d92d8f8a42
Add ifunc tests for x86_64 memset_chk and memset
...
This patch adds ifunc tests for x86_64 memset_chk and memset. It also
defines HAS_AVX2 with AVX2_Usable since AVX2 may not be usable even if
processor has AVX2.
* sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list):
Add tests for memset_chk and memset.
* sysdeps/x86_64/multiarch/init-arch.h (HAS_AVX2): Defined
with AVX2_Usable.
2014-06-20 14:52:29 -07:00
H.J. Lu
224c6c51c5
Remove sysdeps/x86_64/multiarch/rtld-strlen.S
...
Since there is no sysdeps/x86_64/multiarch/strlen.S,
sysdeps/x86_64/rtld-strlen.S will be used.
* sysdeps/x86_64/multiarch/rtld-strlen.S: Removed.
2014-06-20 08:10:07 -07:00
Ling Ma
5c74e47cd6
Add x86_64 memset optimized for AVX2
...
In this patch we take advantage of HSW memory bandwidth, manage to
reduce miss branch prediction by avoiding using branch instructions and
force destination to be aligned with avx & avx2 instruction.
The CPU2006 403.gcc benchmark indicates this patch improves performance
from 26% to 59%.
* sysdeps/x86_64/multiarch/Makefile: Add memset-avx2.
* sysdeps/x86_64/multiarch/memset-avx2.S: New file.
* sysdeps/x86_64/multiarch/memset.S: Likewise.
* sysdeps/x86_64/multiarch/memset_chk.S: Likewise.
* sysdeps/x86_64/multiarch/rtld-memset.S: Likewise.
2014-06-19 15:14:08 -07:00
Carlos O'Donell
8f1df5cf9d
Fix -Wundef warning for FEATURE_INDEX_1.
...
Define FEATURE_INDEX_1 and FEATURE_INDEX_MAX as macros
for use by both assembly and C code. This fixes the
-Wundef error for cases where FEATURE_INDEX_1 was not
defined but used the correct value of 0 for an undefined
macro.
2014-05-03 00:25:21 -04:00
Sihai Yao
f9281df995
Detect if AVX2 is usable
...
This patch checks and sets bit_AVX2_Usable in __cpu_features.feature.
* sysdeps/x86_64/multiarch/ifunc-defines.sym (COMMON_CPUID_INDEX_7):
New.
* sysdeps/x86_64/multiarch/init-arch.c (__init_cpu_features):
Check and set bit_AVX2_Usable.
* sysdeps/x86_64/multiarch/init-arch.h (bit_AVX2_Usable): New
macro.
(bit_AVX2): Likewise.
(index_AVX2_Usable): Likewise.
(CPUID_AVX2): Likewise.
(HAS_AVX2): Likewise.
2014-04-17 08:00:21 -07:00
Allan McRae
d4697bc93d
Update copyright notices with scripts/update-copyrights
2014-01-01 22:00:23 +10:00
Allan McRae
6f8e37ebf8
Update file name in x86_64 ifunc list
...
File name update missed in commit 584b18eb
2013-12-16 13:00:39 +10:00
Ondřej Bílka
584b18eb4d
Add strstr with unaligned loads. Fixes bug 12100.
...
A sse42 version of strstr used pcmpistr instruction which is quite
ineffective. A faster way is look for pairs of characters which is uses
sse2, is faster than pcmpistr and for real strings a pairs we look for
are relatively rare.
For linear time complexity we use buy or rent technique which switches
to two-way algorithm when superlinear behaviour is detected.
2013-12-14 20:08:13 +01:00
Ondřej Bílka
e7044ea76b
Use p2align instead ALIGN
2013-10-08 15:46:48 +02:00
Ondřej Bílka
dc1a95c730
Faster strrchr.
2013-09-26 19:23:01 +02:00
Ondřej Bílka
5905e7b3e2
Faster strchr implementation.
2013-09-11 17:07:38 +02:00
Ondřej Bílka
8f02859f17
Add unaligned strcmp.
2013-09-03 16:27:10 +02:00
Ondřej Bílka
382466e04e
Fix typos.
2013-08-30 18:08:59 +02:00
Ondřej Bílka
0186c6e97e
Fix rawmemchr regression on bulldozer.
2013-08-30 10:14:37 +02:00
Ondřej Bílka
c0c3f78afb
Fix typos.
2013-08-21 19:48:48 +02:00
Liubov Dmitrieva
6308fd9a46
Skip SSE4.2 versions on Intel Silvermont
...
SSE2/SSSE3 versions are faster than SSE4.2 versions on Intel Silvermont.
2013-06-28 15:31:40 -07:00
Liubov Dmitrieva
11b8a0e1d7
Fix buffers overrun in x86_64 memcmp-ssse3.S
2013-06-26 12:31:51 -07:00
Liubov Dmitrieva
d086fc7ba0
Set fast unaligned load flag for new Intel microarchitecture
...
I have small patch for new Intel Silvermont machines.
http://newsroom.intel.com/community/intel_newsroom/blog/2013/05/06/intel-launches-low-power-high-performance-silvermont-microarchitecture
I checked this on my machine and see that strcpy, ... unaligned
versions are faster than ssse3 versions.
2013-06-14 20:46:15 +02:00
Ondrej Bilka
2d48b41c8f
Faster memcpy on x64.
...
We add new memcpy version that uses unaligned loads which are fast
on modern processors. This allows second improvement which is avoiding
computed jump which is relatively expensive operation.
Tests available here:
http://kam.mff.cuni.cz/~ondra/memcpy_profile_result27_04_13.tar.bz2
2013-05-20 08:24:41 +02:00
Ondrej Bilka
37bb363f03
Faster strlen on x64.
2013-03-18 07:39:12 +01:00
Ondrej Bilka
80f844c9d8
Remove Prefer_SSE_for_memop on x64
2013-03-11 15:39:08 +01:00
Ondrej Bilka
87bd9bc4bd
Revert " * sysdeps/x86_64/strlen.S: Replace with new SSE2 based implementation"
...
This reverts commit b79188d717
2013-03-06 22:27:18 +01:00
Ondrej Bilka
b79188d717
* sysdeps/x86_64/strlen.S: Replace with new SSE2 based implementation
...
which is faster on all x86_64 architectures.
Tested on AMD, Intel Nehalem, SNB, IVB.
2013-03-06 21:54:01 +01:00
Roland McGrath
f1d70dad53
Remove lots of inline keywords.
2013-02-07 14:44:18 -08:00
H.J. Lu
afec409af9
Change __x86_64 prefix in cache size to __x86
2013-01-05 16:00:38 -08:00
H.J. Lu
5d7dd1ca84
Add HAS_RTM
2013-01-03 09:38:20 -08:00
Joseph Myers
568035b787
Update copyright notices with scripts/update-copyrights.
2013-01-02 19:05:09 +00:00
Pino Toscano
94558d30b1
test-multiarch: terminate printf output with newline
2012-11-22 11:34:03 +01:00
H.J. Lu
f62c8abcfb
Compile x86 rtld with -mno-sse -mno-mmx
2012-11-02 18:43:27 -07:00
H.J. Lu
ac49ecaf9d
Add x86-64 __libc_ifunc_impl_list
2012-10-11 16:41:12 -07:00
H.J. Lu
9a387d1f78
Use IFUNC memmove/memset in x86-64 bcopy/bzero
...
Also add separate tests for bcopy and bzero.
2012-10-11 13:58:16 -07:00
H.J. Lu
0569936773
Define HAS_FMA with bit_FMA_Usable
2012-10-02 05:05:17 -07:00
H.J. Lu
31ed415328
Don't define x86-64 __strncmp_ssse3 in libc.a
2012-09-27 07:43:03 -07:00
Roland McGrath
7312ca90dc
Clean up x86_64/multiarch/strstr-c.c include order.
2012-08-15 11:38:57 -07:00
Roland McGrath
9a0a54864b
Clean up x86_64/multiarch/memmove.c include order.
2012-08-15 11:26:02 -07:00
H.J. Lu
f85fa27058
Avoid DWARF definition DIE on ifunc symbols
2012-08-09 16:04:37 -07:00
Carlos O'Donell
1a0994f535
BZ#14059: Fix AVX and FMA4 detection.
...
Fix AVX and FMA4 detection by following the guidelines
set out by Intel and AMD for detecting these features.
2012-05-17 06:59:28 -07:00
H.J. Lu
70bc83b910
Load pointers into RAX_LP in strcmp-sse42.S
2012-05-15 09:59:31 -07:00
H.J. Lu
9bc0b730a6
Load cache sizes into R*_LP in memcpy-ssse3.S
2012-05-15 09:58:28 -07:00
H.J. Lu
6d2850e7f5
Load cache sizes into R*_LP in memcpy-ssse3-back.S
2012-05-15 09:56:17 -07:00
H.J. Lu
8a17f34979
Load cache size into R8_LP
2012-05-15 09:35:43 -07:00
Paul Eggert
59ba27a63a
Replace FSF snail mail address with URLs.
2012-02-09 23:18:22 +00:00
Ulrich Drepper
08cf777f9e
Really fix AVX tests
...
There is no problem with strcmp, it doesn't use the YMM registers.
The math routines might since gcc perhaps generates such code.
Introduce bit_YMM_USBALE and use it in the math routines.
2012-01-26 09:45:54 -05:00
Ulrich Drepper
afc5ed09cb
Reset bit_AVX in __cpu_features is OS support is missing
2012-01-26 07:45:14 -05:00
Liubov Dmitrieva
15db4de19d
Fix overrun in destination buffer
2011-12-23 12:02:15 -05:00
Ulrich Drepper
370a7d88f7
WP fixes
2011-12-17 14:41:05 -05:00
Ulrich Drepper
1d3e4b618a
Optimized wcschr and wcscpy for x86-64 and x86-32
2011-12-17 14:39:23 -05:00
Ulrich Drepper
aff2453df7
Fix more warnings
2011-12-03 21:49:35 -05:00
Ulrich Drepper
34372fc6d3
Fix test of non-ASCII locales in x86-64 strcasecmp et.al.
2011-11-01 16:46:23 -04:00
Ulrich Drepper
52e4b9eb62
More cleanups of x86-64 strstr
2011-10-28 19:01:48 -04:00
Ulrich Drepper
fd52bc6dc4
Clean up x86-64 strcasestr
...
Actually describe in the C code what is going on.
2011-10-28 18:18:04 -04:00
Ulrich Drepper
e0016b11d6
Add AVX optimized versions for some x86-64 math functions
2011-10-25 21:34:55 -04:00
Ulrich Drepper
618280a192
Optimize x86-64 SSE4.2+ strcmp a bit more
2011-10-25 14:50:31 -04:00
Ulrich Drepper
09229f3e1b
Fix WS
2011-10-23 14:57:28 -04:00
Liubov Dmitrieva
ce7dd29f28
Optimized strnlen and wcscmp for x86-64
2011-10-23 14:56:04 -04:00
Ulrich Drepper
c196fed8f0
Fix compilation problems in x86-64 init-arch
2011-10-21 20:47:20 -04:00
Ulrich Drepper
ed72b6545f
Check for FMA4 support and generate appropriate fma functions
2011-10-20 22:43:15 -04:00
Ulrich Drepper
8d4f46c613
Move fma routines to right place
2011-10-20 21:55:41 -04:00
Ulrich Drepper
855d156018
Optimize x86-64 rawmemchr and add test
2011-10-19 22:22:29 -04:00
Ulrich Drepper
d9a4d2ab27
Add optimized str{,n}casecmp for AVX on x86-64
2011-10-19 12:42:38 -04:00
Ulrich Drepper
2d1f3a4db6
Fix WS
2011-10-15 11:11:12 -04:00
Liubov Dmitrieva
be13f7bff6
Optimized memcmp and wmemcmp for x86-64 and x86-32
2011-10-15 11:10:08 -04:00
Liubov Dmitrieva
093ecf9299
Improve 64 bit memchr, memrchr, rawmemchr with SSE2
2011-10-07 11:49:10 -04:00
Ulrich Drepper
ceaa0c5dc3
Move Atom-optimized code out of the way and together
2011-09-06 21:53:03 -04:00
Ulrich Drepper
6d18b67f4d
Fix whitespaces
2011-09-05 21:42:12 -04:00
Liubov Dmitrieva
a5f524e479
Add Atom-optimized strchr and strrchr for x86-64
2011-09-05 21:34:03 -04:00
Andreas Schwab
8c1a459f9a
Fix inline strncat/strncmp on x86
2011-08-04 14:59:25 -04:00
Ulrich Drepper
21137f89c5
Fix overflow bug is optimized strncat for x86-64
2011-07-21 12:32:36 -04:00
Ulrich Drepper
8002999481
Fix whitespaces
2011-07-19 17:27:09 -04:00
Liubov Dmitrieva
99710781cc
Improve 64 bit strcat functions with SSE2/SSSE3
2011-07-19 17:11:54 -04:00
H.J. Lu
8912479f9e
Improved st{r,p}{,n}cpy for SSE2 and SSSE3 on x86-64
2011-06-24 15:14:22 -04:00
H.J. Lu
0b1cbaaef5
Optimized st{r,p}{,n}cpy for SSE2/SSSE3 on x86-32
2011-06-24 14:15:32 -04:00
H.J. Lu
3d29045b5e
Assume Intel Core i3/i5/i7 processor if AVX is available
2011-06-03 07:01:25 -04:00
Mike Frysinger
4c559bcdf3
Fix static linking with checking x86/x86-64 memcpy.
2011-04-17 22:20:47 -04:00
H.J. Lu
0354e35501
Work around old buggy program which cannot cope with memcpy semantics.
2011-04-01 19:38:21 -04:00
H.J. Lu
c97a1282a4
Handle page boundaries in x86 SSE4.2 strncmp.
2011-03-21 05:35:38 -04:00
Harsha Jagasia
7e4ba49cd3
Enable SSE2 memset for AMD'supcoming Orochi processor.
...
This patch enables SSE2 memset for AMD's upcoming Orochi processor.
This patch also fixes the following bug:
For misaligned blocks larger than > 144 Bytes, memset branches into
the integer code path depending on the value of misalignment even if
the startup code chooses the SSE2 code path upfront, when multiarch
is enabled.
2011-03-04 23:30:08 -05:00
Roland McGrath
a0bf67cca2
Fix some warning nits.
2011-02-04 10:53:51 -08:00
H.J. Lu
13b695749a
Support Intel processor model 6 and model 0x2.
2010-11-12 03:48:52 -05:00
H.J. Lu
8ca52c6e3b
Fix one exit path in x86-64 SSE4.2 str{,n}casecmp.
2010-11-10 03:05:37 -05:00
H.J. Lu
ff02d5280b
Use IFUNC on x86-64 memset
2010-11-08 03:41:34 -05:00
Richard Li
dbf3a06904
Fix x86-64 strchr propagation of search byte into all bytes of SSE register
2010-10-25 14:13:17 -04:00
Jakub Jelinek
5e908464b9
Implement accurate fma.
2010-10-13 22:27:03 -04:00
Jakub Jelinek
9ff8d36f27
Correct implementation of fmaf.
2010-10-11 09:27:05 -04:00
Ulrich Drepper
015a4c6193
Re-enable all strncasecmp versions.
2010-09-20 20:18:00 -07:00
Ulrich Drepper
8ffcee4a04
Fix limit detection in x86-64 SSE2 strncasecmp.
2010-09-20 14:02:23 -07:00
Ulrich Drepper
9ea3de11f1
Move slow Atom code to separate section.
2010-08-26 22:17:03 -07:00
H.J. Lu
623aac7f84
Unroll x86-64 strlen
2010-08-26 22:09:34 -07:00
H.J. Lu
b416a90085
Missing comma in last commit.
2010-08-26 13:18:46 -07:00
Roland McGrath
8b2b771538
Clean up warnings in new x86_64/multiarch code.
2010-08-25 12:13:08 -07:00
H.J. Lu
e73015f2d6
Unroll 32bit SSE strlen and handle slow bsf
2010-08-25 10:07:37 -07:00
Ulrich Drepper
1cdfe7242f
Add missing copyright year updated and pretty printing.
2010-08-24 11:42:19 -07:00
Richard Henderson
73f27d5e72
Clean up SSE variable shifts
2010-08-24 11:35:01 -07:00
Ulrich Drepper
9da4bb316f
Fix two typos in x86-64 SSE4.2 strncasecmp implementation.
2010-08-19 09:20:44 -07:00
Ulrich Drepper
1feccb6caf
Fix fourth parameter of SSE4.2 strcmp for x86-64.
2010-08-15 20:46:09 -07:00
Ulrich Drepper
e9f82e0d1d
Add optimized strncasecmp versions for x86-64.
2010-08-14 22:04:01 -07:00
Ulrich Drepper
ca6bb004eb
Fix x86-64 build without multiarch.
2010-08-14 14:56:32 -07:00
Ulrich Drepper
73507d3ae0
Add support for SSSE3 and SSE4.2 versions of strcasecmp on x86-64.
2010-07-31 21:41:09 -07:00
Ulrich Drepper
66f6765a47
Pretty printing x86-64 SSE4.3 strcmp.
2010-07-30 12:54:37 -07:00
Ulrich Drepper
fe36dd025e
Fix tolower operation in strcasestr.
2010-07-30 00:09:07 -07:00
Ulrich Drepper
880113d91e
Avoid compiling unneeded file in ld.so.
2010-07-27 21:12:59 -07:00
Ulrich Drepper
8e96b93aa7
Speed up x86-64 strcasestr a bit moew.
...
Using the new SSE4.2 instructions is cool but not really the fastest.
Some older SSE instructions can do the trick faster.
2010-07-24 08:34:44 -07:00
Andreas Schwab
f6a31e0eb6
Add strcasestr-nonascii to i386 build
2010-07-21 07:26:18 -07:00
Ulrich Drepper
d02dc4ba08
Fix non-ASCII case of SSE4.2 strcasstr.
2010-07-16 16:00:22 -07:00
Ulrich Drepper
cc9f2e47a0
Speed up SSE4.2 strcasestr by avoiding indirect function call.
2010-07-16 15:37:38 -07:00
H.J. Lu
6fb8cbcb58
Improve 64bit memcpy/memmove for Atom, Core 2 and Core i7
...
This patch includes optimized 64bit memcpy/memmove for Atom, Core 2 and
Core i7. It improves memcpy by up to 3X on Atom, up to 4X on Core 2 and
up to 1X on Core i7. It also improves memmove by up to 3X on Atom, up to
4X on Core 2 and up to 2X on Core i7.
2010-06-30 08:26:11 -07:00
H.J. Lu
3c88fe1e3a
Incorrect x86 CPU family and model check.
2010-05-27 11:14:18 -07:00
H.J. Lu
df87f54923
Check DATA_CACHE_SIZE_HALF
2010-04-14 22:18:27 -07:00
H.J. Lu
dd37cd1a12
Optimie x86-64 SSE4 memcmp for unaligned data.
2010-04-14 17:53:44 -07:00
H.J. Lu
404a6e3201
x86-64 SSE4 optimized memcmp
...
This is 64bit SSE4 optimized memcmp. It improves memcmp by upto 3X
on Intel Core i7.
2010-04-14 00:12:53 -07:00
Ulrich Drepper
bbbdd77809
Update x86-64 cpu multiarch selection header.
2010-04-13 19:17:10 -07:00
Ulrich Drepper
22f4f44b67
Fix concurrent handling of __cpu_features.
2010-04-04 00:25:46 -07:00
H.J. Lu
7d9335ecd7
Don't define __strpbrk_sse42 in static library
2010-03-24 12:16:24 -07:00
H.J. Lu
5a7af22fbb
Unroll the loop x86-64 SSE4.2 strlen.
2010-01-13 07:51:48 -08:00
H.J. Lu
3af48cbdfa
Optimize 32bit memset/memcpy with SSE2/SSSE3.
2010-01-12 11:22:03 -08:00
H.J. Lu
2510d01ddb
Define bit_SSE2 and index_SSE2.
2009-12-13 15:23:02 -08:00
H.J. Lu
51ddd2c01e
Define bit_XXX and index_XXX.
...
This patch defines bit_XXX and index_XXX and use them to check processor
feature in assembly code. It can prevent typos in processor feature
check.
2009-12-13 09:47:02 -08:00
Ulrich Drepper
823bc6da65
Fix whitespaces.
2009-10-22 22:50:00 -07:00
H.J. Lu
001659f4d5
Implement SSE4.2 optimized strchr and strrchr.
2009-10-22 22:47:12 -07:00
Roland McGrath
b0f3a2e43f
Clean up unnecessary libc_hidden_builtin_def fiddling in x86 multiarch definitions.
2009-10-06 20:01:23 -07:00
Roland McGrath
9d6982d5d2
Clean up x86 multiarch HAS_FOO macros.
2009-10-06 19:59:03 -07:00
Jakub Jelinek
22bb992d51
Fix strstr/strcasestr/fma/fmaf on x86_64.
2009-09-02 19:43:04 -07:00
H.J. Lu
5a4eb7282e
Remove ENABLE_SSSE3_ON_ATOM.
...
It turns that SSSE3 isn't slow on Atom. The problem is bsf. This patch
removes ENABLE_SSSE3_ON_ATOM.
2009-08-28 14:54:46 -07:00
Ulrich Drepper
8e436522e1
Move SSE4.2 functions together.
2009-08-08 09:38:32 -07:00
Ulrich Drepper
0fda545d5f
Add SSSE3-optimized implementation of str{,n}cmp for x86-64.
2009-08-07 22:51:02 -07:00
Ulrich Drepper
57b378ac89
Avoid warning through fake initialization.
2009-08-07 16:19:54 -07:00
H.J. Lu
02cea47161
Add x86 32-bit SSE4.2 string functions.
...
This patch adds 32bit SSE4.2 string functions. It uses -16L instead of
0xfffffffffffffff0L, which works for both 32bit and 64bit long. Tested
on 32bit Core i7 and Core 2.
2009-08-04 12:13:43 -07:00
H.J. Lu
6f6f1215f6
Support multiarch for i686.
...
This patch adds multiarch support when configured for i686. I modified
some x86-64 functions to support 32bit. I will contribute 32bit SSE string
and memory functions later.
2009-07-31 11:53:35 -07:00
Ulrich Drepper
78c4ef475d
Add support for x86-64 fma instruction.
...
Use it to implement fma and fmaf, if possible.
2009-07-29 15:26:06 -07:00
Ulrich Drepper
9a1d2d4555
Prepare use if IFUNC functions outside libc.so.
...
We use a callback function into libc.so to get access to the data
structure with the information and have special versions of the test
macros which automatically use this function.
2009-07-29 15:22:28 -07:00
Ulrich Drepper
e83c1a8a72
Refine testing for xmm/ymm register use in x86-64 ld.so.
...
The test now takes the callgraph into account. Only code called
during runtime relocation is affected by the limitation. We now
determine the affected object files as closely as possible from
the outside. This allowed to remove some the specializations
for some of the string functions as they are only used in other
code paths.
2009-07-27 13:40:27 -07:00
Ulrich Drepper
16d2ea4c82
Make sure no code in ld.so uses xmm/ymm registers on x86-64.
...
This patch introduces a test to make sure no function modifies the
xmm/ymm registers. With the exception of the auditing functions.
The test is probably too pessimistic. All code linked into ld.so
is checked. Perhaps at some point the callgraph starting from
_dl_fixup and _dl_profile_fixup is checked and we can start using
faster SSE-using functions in parts of ld.so.
2009-07-26 16:10:00 -07:00
H.J. Lu
7956a3d27c
Add SSE2 support to str{,n}cmp for x86-64.
2009-07-26 13:32:28 -07:00
H.J. Lu
4e5b5821bf
Some some optimizations for x86-64 strcmp.
2009-07-25 19:15:14 -07:00
Ulrich Drepper
29e92fa5cd
Optimize x86-64 SSE4.2 strcmp.
...
The file contained some code which was never used. Don't compile it
in.
2009-07-25 12:02:47 -07:00
Ulrich Drepper
d28797e426
Perform test for Arom x86-64 in central place and handle it.
...
There will be more than one function which, in multiarch mode, wants
to use SSSE3. We should not test in each of them for Atoms with
slow SSSE3. Instead, disable the SSSE3 bit in the startup code for
such machines.
2009-07-23 13:15:17 -07:00
Ulrich Drepper
ae612b04cc
Minor cleanups in x86-64 strstr.
2009-07-21 07:52:12 -07:00
Ulrich Drepper
a8f895ebe1
Better check for optimization in new x86-64 strstr/strcasestr.
2009-07-20 21:18:28 -07:00
H.J. Lu
2b7a8664fa
SSE4.2 strstr/strcasestr for x86-64.
...
This patch implements SSE4.2 strstr/strcasestr, using Knuth-Morris-Pratt
string searching algorithm.
2009-07-20 21:06:50 -07:00
Ulrich Drepper
cea4329592
Minor cleanups in recently added files.
2009-07-03 03:23:01 -07:00
Ulrich Drepper
d6485c981b
Align functions to 16-byte boundary.
...
Some of the new multi-arch string functions for x86-64 were
not aligned to 16 byte boundarie,s possibly creating unnecessary
cache line misses and delays.
2009-07-03 03:01:57 -07:00
H.J. Lu
06e51c8f3d
Add SSE4.2 support for strcspn, strpbrk, and strspn on x86-64.
2009-07-03 02:48:56 -07:00
Ulrich Drepper
af263b8154
Whitespace fixes in last patch.
2009-07-02 03:43:05 -07:00
H.J. Lu
ab6a873fe0
SSSE3 strcpy/stpcpy for x86-64
...
This patch adds SSSE3 strcpy/stpcpy. I got up to 4X speed up on Core 2
and Core i7. I disabled it on Atom since SSSE3 version is slower for
shorter (<64byte) data.
2009-07-02 03:39:03 -07:00
Ulrich Drepper
b38a2e2e64
Fix little checkin problem in last patch.
2009-06-30 04:41:38 -07:00
H.J. Lu
0181291385
Determine and store processor family and model on x86-64.
2009-06-30 04:39:09 -07:00
Ulrich Drepper
059215ae21
Clean up whitespaces in last patch.
2009-06-22 20:39:37 -07:00
H.J. Lu
772f4e6a1b
Add SSE4.2 support for strcmp and strncmp on x86-64.
2009-06-22 20:38:41 -07:00
Ulrich Drepper
b77c932329
Add SSE4.2 optimized rawmemchr implementation for x86-64.
2009-06-05 16:54:50 -07:00
Ulrich Drepper
6f9eea15bf
Forgot some more cleanups for the SSE4.2 strlen on x86-64.
2009-06-05 11:51:59 -07:00
Ulrich Drepper
f85a9e72e2
Add missing cleanups from SSE4.2 x86-64 strlen.
2009-06-05 11:39:45 -07:00
Ulrich Drepper
3ab2d57a4d
Optimize x86-64 strlen for SSE4.2.
...
The SSE4.2 implementation is used in the DSO only. The patch also adds
some infrastructure to be used in similar code later one.
2009-06-05 11:32:00 -07:00
Ulrich Drepper
8ea2372936
Fix up sched_cpucount in x86-64.
...
Now that static executables can handle IFUNC functions don't exclude
optimization for sched_cpucount for !SHARED.
2009-05-31 23:46:42 -07:00
Ulrich Drepper
963cb6fcb4
Simplify CPUID value handling.
...
SO far Intel and AMD use exactly the same bits meaning the same
things in CPUID index 1. Simplify the code. Should an architecture
come along which doesn't use the same semantics then it must use a
different index value than COMMON_CPUID_INDEX_1.
2009-05-31 17:52:05 -07:00
Ulrich Drepper
425ce2edb9
* config.h.in (USE_MULTIARCH): Define.
...
* configure.in: Handle --enable-multi-arch.
* elf/dl-runtime.c (_dl_fixup): Handle STT_GNU_IFUNC.
(_dl_fixup_profile): Likewise.
* elf/do-lookup.c (dl_lookup_x): Likewise.
* sysdeps/x86_64/dl-machine.h: Handle STT_GNU_IFUNC.
* elf/elf.h (STT_GNU_IFUNC): Define.
* include/libc-symbols.h (libc_ifunc): Define.
* sysdeps/x86_64/cacheinfo.c: If USE_MULTIARCH is defined, use the
framework in init-arch.h to get CPUID values.
* sysdeps/x86_64/multiarch/Makefile: New file.
* sysdeps/x86_64/multiarch/init-arch.c: New file.
* sysdeps/x86_64/multiarch/init-arch.h: New file.
* sysdeps/x86_64/multiarch/sched_cpucount.c: New file.
* config.make.in (experimental-malloc): Define.
* configure.in: Handle --enable-experimental-malloc.
* malloc/Makefile: Handle experimental-malloc flag.
* malloc/malloc.c: Implement PER_THREAD and ATOMIC_FASTBINS features.
* malloc/arena.c: Likewise.
* malloc/hooks.c: Likewise.
* malloc/malloc.h: Define M_ARENA_TEST and M_ARENA_MAX.
2009-03-13 23:53:18 +00:00
