This patch introduces the new arch13 ifunc variant for memmove.
For the forward or non-overlapping case it is just using memcpy.
For the backward case it relies on the new instruction mvcrl.
The instruction copies up to 256 bytes at once.
In case of an overlap, it copies the bytes like copying them
one by one starting from right to left.
ChangeLog:
* sysdeps/s390/ifunc-memcpy.h (HAVE_MEMMOVE_ARCH13, MEMMOVE_ARCH13
HAVE_MEMMOVE_IFUNC_AND_ARCH13_SUPPORT): New defines.
* sysdeps/s390/memcpy-z900.S: Add arch13 memmove implementation.
* sysdeps/s390/memmove.c (memmove): Add arch13 variant in
ifunc selector.
* sysdeps/s390/multiarch/ifunc-impl-list.c
(__libc_ifunc_impl_list): Add ifunc variant for arch13 memmove.
* sysdeps/s390/multiarch/ifunc-resolve.h (S390_STFLE_BITS_ARCH13_MIE3,
S390_IS_ARCH13_MIE3): New defines.
The ifunc macros s390_vx_libc* are no longer used and
can be removed as all users are now relying on
s390_libc_ifunc_expr.
The same applies to s390_libc_ifunc. The macro
s390_libc_ifunc_init is now renamed to
s390_libc_ifunc_expr_stfle_init and the users are
adjusted accordingly.
ChangeLog:
* sysdeps/s390/multiarch/ifunc-resolve.h
(s390_vx_libc_ifunc, s390_vx_libc_ifunc_redirected,
s390_vx_libc_ifunc2, s390_vx_libc_ifunc_init,
s390_vx_libc_ifunc2_redirected, s390_libc_ifunc):
Delete macro definition.
(s390_libc_ifunc_init): Rename to
s390_libc_ifunc_expr_stfle_init.
* sysdeps/s390/bzero: Use
s390_libc_ifunc_expr_stfle_init instead of
s390_libc_ifunc_init.
* sysdeps/s390/memcmp.c: Likewise.
* sysdeps/s390/memcpy.c: Likewise.
* sysdeps/s390/mempcpy.c: Likewise.
* sysdeps/s390/memset.c: Likewise.
The renaming of hwcap arguments in ifunc-resolvers is needed
in order to prepare for further commits which refactors
ifunc handling for memset, memcmp, and memcpy. Now you are able
to use s390_libc_ifunc_init which stores the stfle bits
within the expression for an ifunc-resolver generated by
s390_libc_ifunc_expr.
ChangeLog:
* sysdeps/s390/multiarch/ifunc-resolve.h
(s390_libc_ifunc_init, s390_libc_ifunc,
s390_vx_libc_ifunc2_redirected): Use hwcap instead of dl_hwcap.
This patch adds s390_libc_ifunc_expr macro which uses the __ifunc base macro
in include/libc-symbols.h and lets the user define a generic expression to
choose the correct ifunc variant. Furthermore as the base macro is used,
the ifunc resolver functions are now also using inhibit_stack_protector.
S390 needs its own version due to the hwcap argument of the ifunc resolver.
This new macro is now used in iconv code in 8bit-generic.c instead of using
gcc attribute ifunc directly.
ChangeLog:
* sysdeps/s390/multiarch/ifunc-resolve.h
(s390_libc_ifunc_expr_init, s390_libc_ifunc_expr): New Define.
* sysdeps/s390/multiarch/8bit-generic.c
(__to_generic, __from_generic): Use s390_libc_ifunc_expr to
define ifunc resolvers.
This patch adjusts the s390 specific ifunc helper macros in ifunc-resolve.h to
use the common __ifunc macro, which uses gcc attribute ifunc to get rid of the
false debuginfo. Therefore the redirection construct is applied where needed.
Perhaps in future we can switch some of the internal symbols __GI_* from the
fallback variant to the ifunc function. But this change is not
straightforward due to a segmentation fault while linking libc.so with older
binutils on s390.
ChangeLog:
[BZ #20478]
* sysdeps/s390/multiarch/ifunc-resolve.h
(s390_vx_libc_ifunc2, s390_libc_ifunc): Use __ifunc from libc-symbols.h
to create ifunc symbols.
(s390_vx_libc_ifunc_init, s390_vx_libc_ifunc_redirected
, s390_vx_libc_ifunc2_redirected, s390_libc_ifunc_init): New define.
* sysdeps/s390/multiarch/memchr.c: Redirect ifunced function in header
for using it as type for ifunc function.
* sysdeps/s390/multiarch/mempcpy.c: Likewise.
* sysdeps/s390/multiarch/rawmemchr.c: Likewise.
* sysdeps/s390/multiarch/stpcpy.c: Likewise.
* sysdeps/s390/multiarch/stpncpy.c: Likewise.
* sysdeps/s390/multiarch/strcat.c: Likewise.
* sysdeps/s390/multiarch/strchr.c: Likewise.
* sysdeps/s390/multiarch/strcmp.c: Likewise.
* sysdeps/s390/multiarch/strcpy.c: Likewise.
* sysdeps/s390/multiarch/strcspn.c: Likewise.
* sysdeps/s390/multiarch/strlen.c: Likewise.
* sysdeps/s390/multiarch/strncmp.c: Likewise.
* sysdeps/s390/multiarch/strncpy.c: Likewise.
* sysdeps/s390/multiarch/strnlen.c: Likewise.
* sysdeps/s390/multiarch/strpbrk.c: Likewise.
* sysdeps/s390/multiarch/strrchr.c: Likewise.
* sysdeps/s390/multiarch/strspn.c: Likewise.
* sysdeps/s390/multiarch/wcschr.c: Likewise.
* sysdeps/s390/multiarch/wcscmp.c: Likewise.
* sysdeps/s390/multiarch/wcspbrk.c: Likewise.
* sysdeps/s390/multiarch/wcsspn.c: Likewise.
* sysdeps/s390/multiarch/wmemchr.c: Likewise.
* sysdeps/s390/multiarch/wmemset.c: Likewise.
* sysdeps/s390/s390-32/multiarch/memcmp.c: Likewise.
* sysdeps/s390/s390-32/multiarch/memcpy.c: Likewise.
* sysdeps/s390/s390-32/multiarch/memset.c: Likewise.
* sysdeps/s390/s390-64/multiarch/memcmp.c: Likewise.
* sysdeps/s390/s390-64/multiarch/memcpy.c: Likewise.
* sysdeps/s390/s390-64/multiarch/memset.c: Likewise.
On s390, the memcpy, memcmp, memset functions are IFUNC symbols,
which are created with s390_libc_ifunc-macro.
This macro creates a __GI_ symbol which is set to the
ifunced symbol. Thus calls within libc.so to e.g. memcpy
result in a call to *ABS*+0x954c0@plt stub and afterwards
to the resolved memcpy-ifunc-variant.
This patch sets the __GI_ symbol to the default-ifunc-variant
to avoid the plt call. The __GI_ symbols are now created at the
default variant of ifunced function.
ChangeLog:
* sysdeps/s390/multiarch/ifunc-resolve.h (s390_libc_ifunc):
Remove __GI_ symbol.
* sysdeps/s390/s390-32/multiarch/memcmp-s390.S: Add __GI_memcmp symbol.
* sysdeps/s390/s390-64/multiarch/memcmp-s390x.S: Likewise.
* sysdeps/s390/s390-32/multiarch/memcpy-s390.S: Add __GI_memcpy symbol.
* sysdeps/s390/s390-64/multiarch/memcpy-s390x.S: Likewise.
* sysdeps/s390/s390-32/multiarch/memset-s390.S: Add __GI_memset symbol.
* sysdeps/s390/s390-64/multiarch/memset-s390x.S: Likewise.
This patch introduces a s390 specific ifunc resolver macro for 32/64bit,
which chooses <func>_vx with vector instructions if HWCAP_S390_VX flag
in hwcaps is set or <func>_c if not.
ChangeLog:
* sysdeps/s390/multiarch/ifunc-resolve.h (s390_vx_libc_ifunc,
s390_vx_libc_ifunc2): New macro function.
On s390 all ifunc resolvers were implemented in multiarch/ifunc-resolve.c.
The resulting single object files has undefined references to all ifunc-functions.
This patch introduces one multiarch/<func>.c file for each of memcpy, memcmp
and memset with the function specific ifunc resolver. The different function
implementations are now implemented in multiarch/<func>-s390x.S
(moved from multiarch/<func>.S).
The new multiarch/ifunc-resolve.h file contains the ifunc-resolver macro
and other helper-macros. They are merged and are now used in common for
32/64bit. Therefore the __<func>_g5/__<func>_z900 functions were renamed to
__<func>_default.
This patch also enables testing the ifunc implementations by implementing
the function __libc_ifunc_impl_list. It uses the helper-macros of ifunc-resolve.h.
ChangeLog:
* sysdeps/s390/s390-32/multiarch/Makefile (sysdep_routines):
Remove ifunc-resolve, add memset-s390, memcpy-s390, memcmp-s390.
* sysdeps/s390/s390-32/multiarch/ifunc-resolve.c: Delete File.
* sysdeps/s390/s390-32/multiarch/memcmp.S: Move to ...
* sysdeps/s390/s390-32/multiarch/memcmp-s390.S: ... here.
(memcmp, bcmp): Use __memcmp_default as alias source.
* sysdeps/s390/s390-32/multiarch/memcmp.c: New File.
* sysdeps/s390/s390-32/memcmp.S (__memcmp_g5):
Rename to __memcmp_default.
* sysdeps/s390/s390-32/multiarch/memcpy.S: Move to ...
* sysdeps/s390/s390-32/multiarch/memcpy-s390.S: ... here.
(memcpy): Use __memcpy_default as alias source.
* sysdeps/s390/s390-32/multiarch/memcpy.c: New File.
* sysdeps/s390/s390-32/memcpy.S (__memcpy_g5):
Rename to __memcpy_default.
* sysdeps/s390/s390-32/multiarch/memset.S: Move to ...
* sysdeps/s390/s390-32/multiarch/memset-s390.S: ... here.
(memset): Use __memset_default as alias source.
* sysdeps/s390/s390-32/multiarch/memset.c: New File.
* sysdeps/s390/s390-32/memset.S (__memset_g5):
Rename to __memset_default.
* sysdeps/s390/s390-64/multiarch/Makefile (sysdep_routines):
Remove ifunc-resolve, add memset-s390x, memcpy-s390x, memcmp-s390x.
* sysdeps/s390/s390-64/multiarch/ifunc-resolve.c: Delete File.
* sysdeps/s390/s390-64/multiarch/memcmp.S: Move to ...
* sysdeps/s390/s390-64/multiarch/memcmp-s390x.S: ... here.
(memcmp, bcmp): Use __memcmp_default as alias source.
* sysdeps/s390/s390-64/multiarch/memcmp.c: New File.
* sysdeps/s390/s390-64/memcmp.S (__memcmp_z900):
Rename to __memcmp_default.
* sysdeps/s390/s390-64/multiarch/memcpy.S: Move to ...
* sysdeps/s390/s390-64/multiarch/memcpy-s390x.S: ... here.
(memcpy): Use __memcpy_default as alias source.
* sysdeps/s390/s390-64/multiarch/memcpy.c: New File.
* sysdeps/s390/s390-64/memcpy.S (__memcpy_z900):
Rename to __memcpy_default.
* sysdeps/s390/s390-64/multiarch/memset.S: Move to ...
* sysdeps/s390/s390-64/multiarch/memset-s390x.S: ... here.
(memset): Use __memset_default as alias source.
* sysdeps/s390/s390-64/multiarch/memset.c: New File.
* sysdeps/s390/s390-64/memset.S (__memset_z900):
Rename to __memset_default.
* sysdeps/s390/multiarch/ifunc-resolve.h: New File.
* sysdeps/s390/multiarch/ifunc-impl-list.c: New File.
