glibc/sysdeps/unix/sysv/linux/powerpc/sysdep.h

259 lines
9.1 KiB
C
Raw Normal View History

Remove PREPARE_VERSION and PREPARE_VERSION_KNOW This patch removes the PREPARE_VERSION and PREPARE_VERSION_KNOW macro and uses a static inline function instead, get_vdso_symbol. Each architecture that supports vDSO must define the Linux version and its hash for symbol resolution (VDSO_NAME and VDSO_HASH macro respectively). It also organizes the HAVE_*_VSYSCALL for mips, powerpc, and s390 to define them on a common header. The idea is to require less code to configure and enable vDSO support for newer ports. No semantic changes are expected. Checked with a build against all affected architectures. * sysdeps/unix/make-syscalls.sh: Make vDSO call use get_vdso_symbol. * sysdeps/unix/sysv/linux/aarch64/gettimeofday.c (__gettimeofday): Use get_vdso_symbol instead of _dl_vdso_vsym. * sysdeps/unix/sysv/linux/powerpc/time.c (time): Likewise. * sysdeps/unix/sysv/linux/riscv/flush-icache.c (__lookup_riscv_flush_icache): Likewise. * sysdeps/unix/sysv/linux/x86/gettimeofday.c (__gettimeofday): Likewise. * sysdeps/unix/sysv/linux/x86/time.c (time): Likewise. * sysdeps/unix/sysv/linux/powerpc/gettimeofday.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/arm/init-first.c: Likewise. * sysdeps/unix/sysv/linux/i386/init-first.c: Likewise. * sysdeps/unix/sysv/linux/mips/init-first.c: Likewise. * sysdeps/unix/sysv/linux/powerpc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise. * sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/s390/init-first.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/sysdep.h (VDSO_NAME, VDSO_HASH): Define. * sysdeps/unix/sysv/linux/arm/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/i386/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/riscv/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/sparc/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/x86_64/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/mips/mips32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Remove definition. * sysdeps/unix/sysv/linux/mips/mips64/n32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/sysdep.h: New file. * sysdeps/unix/sysv/linux/powerpc/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/s390/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/dl-vdso.h (PREPARE_VERSION, PREPARE_VERSION_KNOWN, VDSO_NAME_LINUX_2_6, VDSO_HASH_LINUX_2_6, VDSO_NAME_LINUX_2_6_15, VDSO_HASH_LINUX_2_6_15, VDSO_NAME_LINUX_2_6_29, VDSO_HASH_LINUX_2_6_29, VDSO_NAME_LINUX_4_15, VDSO_HASH_LINUX_4_15): Remove defines. (get_vdso_symbol): New function.
2019-05-23 19:33:32 +00:00
/* Syscall definitions, Linux PowerPC generic version.
Copyright (C) 2019-2022 Free Software Foundation, Inc.
Remove PREPARE_VERSION and PREPARE_VERSION_KNOW This patch removes the PREPARE_VERSION and PREPARE_VERSION_KNOW macro and uses a static inline function instead, get_vdso_symbol. Each architecture that supports vDSO must define the Linux version and its hash for symbol resolution (VDSO_NAME and VDSO_HASH macro respectively). It also organizes the HAVE_*_VSYSCALL for mips, powerpc, and s390 to define them on a common header. The idea is to require less code to configure and enable vDSO support for newer ports. No semantic changes are expected. Checked with a build against all affected architectures. * sysdeps/unix/make-syscalls.sh: Make vDSO call use get_vdso_symbol. * sysdeps/unix/sysv/linux/aarch64/gettimeofday.c (__gettimeofday): Use get_vdso_symbol instead of _dl_vdso_vsym. * sysdeps/unix/sysv/linux/powerpc/time.c (time): Likewise. * sysdeps/unix/sysv/linux/riscv/flush-icache.c (__lookup_riscv_flush_icache): Likewise. * sysdeps/unix/sysv/linux/x86/gettimeofday.c (__gettimeofday): Likewise. * sysdeps/unix/sysv/linux/x86/time.c (time): Likewise. * sysdeps/unix/sysv/linux/powerpc/gettimeofday.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/arm/init-first.c: Likewise. * sysdeps/unix/sysv/linux/i386/init-first.c: Likewise. * sysdeps/unix/sysv/linux/mips/init-first.c: Likewise. * sysdeps/unix/sysv/linux/powerpc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise. * sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/s390/init-first.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/sysdep.h (VDSO_NAME, VDSO_HASH): Define. * sysdeps/unix/sysv/linux/arm/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/i386/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/riscv/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/sparc/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/x86_64/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/mips/mips32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Remove definition. * sysdeps/unix/sysv/linux/mips/mips64/n32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/sysdep.h: New file. * sysdeps/unix/sysv/linux/powerpc/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/s390/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/dl-vdso.h (PREPARE_VERSION, PREPARE_VERSION_KNOWN, VDSO_NAME_LINUX_2_6, VDSO_HASH_LINUX_2_6, VDSO_NAME_LINUX_2_6_15, VDSO_HASH_LINUX_2_6_15, VDSO_NAME_LINUX_2_6_29, VDSO_HASH_LINUX_2_6_29, VDSO_NAME_LINUX_4_15, VDSO_HASH_LINUX_4_15): Remove defines. (get_vdso_symbol): New function.
2019-05-23 19:33:32 +00:00
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#ifndef _LINUX_POWERPC_SYSDEP_H
#define _LINUX_POWERPC_SYSDEP_H 1
#include <sysdeps/unix/sysv/linux/sysdep.h>
#include <sysdeps/unix/powerpc/sysdep.h>
#include <tls.h>
/* Define __set_errno() for INLINE_SYSCALL macro below. */
#include <errno.h>
/* For Linux we can use the system call table in the header file
/usr/include/asm/unistd.h
of the kernel. But these symbols do not follow the SYS_* syntax
so we have to redefine the `SYS_ify' macro here. */
#undef SYS_ify
#define SYS_ify(syscall_name) __NR_##syscall_name
/* Define a macro which expands inline into the wrapper code for a system
call. This use is for internal calls that do not need to handle errors
normally. It will never touch errno. This returns just what the kernel
gave back in the non-error (CR0.SO cleared) case, otherwise (CR0.SO set)
the negation of the return value in the kernel gets reverted. */
#define INTERNAL_VSYSCALL_CALL_TYPE(funcptr, type, nr, args...) \
({ \
register void *r0 __asm__ ("r0"); \
register long int r3 __asm__ ("r3"); \
register long int r4 __asm__ ("r4"); \
register long int r5 __asm__ ("r5"); \
register long int r6 __asm__ ("r6"); \
register long int r7 __asm__ ("r7"); \
register long int r8 __asm__ ("r8"); \
register type rval __asm__ ("r3"); \
LOADARGS_##nr (funcptr, args); \
__asm__ __volatile__ \
("mtctr %0\n\t" \
"bctrl\n\t" \
"mfcr %0\n\t" \
"0:" \
: "+r" (r0), "+r" (r3), "+r" (r4), "+r" (r5), "+r" (r6), \
"+r" (r7), "+r" (r8) \
: : "r9", "r10", "r11", "r12", \
"cr0", "cr1", "cr5", "cr6", "cr7", \
"xer", "lr", "ctr", "memory"); \
__asm__ __volatile__ ("" : "=r" (rval) : "r" (r3)); \
(long int) r0 & (1 << 28) ? -rval : rval; \
})
#define INTERNAL_VSYSCALL_CALL(funcptr, nr, args...) \
INTERNAL_VSYSCALL_CALL_TYPE(funcptr, long int, nr, args)
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
#define DECLARE_REGS \
register long int r0 __asm__ ("r0"); \
register long int r3 __asm__ ("r3"); \
register long int r4 __asm__ ("r4"); \
register long int r5 __asm__ ("r5"); \
register long int r6 __asm__ ("r6"); \
register long int r7 __asm__ ("r7"); \
register long int r8 __asm__ ("r8");
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
#define SYSCALL_SCV(nr) \
({ \
__asm__ __volatile__ \
(".machine \"push\"\n\t" \
".machine \"power9\"\n\t" \
"scv 0\n\t" \
".machine \"pop\"\n\t" \
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
"0:" \
: "=&r" (r0), \
"=&r" (r3), "=&r" (r4), "=&r" (r5), \
"=&r" (r6), "=&r" (r7), "=&r" (r8) \
: ASM_INPUT_##nr \
: "r9", "r10", "r11", "r12", \
"cr0", "cr1", "cr5", "cr6", "cr7", \
"xer", "lr", "ctr", "memory"); \
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
r3; \
})
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
#define SYSCALL_SC(nr) \
({ \
__asm__ __volatile__ \
("sc\n\t" \
"mfcr %0\n\t" \
"0:" \
: "=&r" (r0), \
"=&r" (r3), "=&r" (r4), "=&r" (r5), \
"=&r" (r6), "=&r" (r7), "=&r" (r8) \
: ASM_INPUT_##nr \
: "r9", "r10", "r11", "r12", \
"xer", "cr0", "ctr", "memory"); \
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
r0 & (1 << 28) ? -r3 : r3; \
})
/* This will only be non-empty for 64-bit systems, see below. */
#define TRY_SYSCALL_SCV(nr)
#if defined(__PPC64__) || defined(__powerpc64__)
# define SYSCALL_ARG_SIZE 8
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
/* For the static case, unlike the dynamic loader, there is no compile-time way
to check if we are inside startup code. So we need to check if the thread
pointer has already been setup before trying to access the TLS. */
# ifndef SHARED
# define CHECK_THREAD_POINTER (__thread_register != 0)
# else
# define CHECK_THREAD_POINTER (1)
# endif
/* When inside the dynamic loader, the thread pointer may not have been
initialized yet, so don't check for scv support in that case. */
# if defined(USE_PPC_SCV) && !IS_IN(rtld)
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
# undef TRY_SYSCALL_SCV
# define TRY_SYSCALL_SCV(nr) \
CHECK_THREAD_POINTER && THREAD_GET_HWCAP() & PPC_FEATURE2_SCV ? \
SYSCALL_SCV(nr) :
# endif
#else
# define SYSCALL_ARG_SIZE 4
#endif
powerpc: Runtime selection between sc and scv for syscalls Linux kernel v5.9 added support for system calls using the scv instruction for POWER9 and later. The new codepath provides better performance (see below) if compared to using sc. For the foreseeable future, both sc and scv mechanisms will co-exist, so this patch enables glibc to do a runtime check and use scv when it is available. Before issuing the system call to the kernel, we check hwcap2 in the TCB for PPC_FEATURE2_SCV to see if scv is supported by the kernel. If not, we fallback to sc and keep the old behavior. The kernel implements a different error return convention for scv, so when returning from a system call we need to handle the return value differently depending on the instruction we used to enter the kernel. For syscalls implemented in ASM, entry and exit are implemented by different macros (PSEUDO and PSEUDO_RET, resp.), which may be used in sequence (e.g. for templated syscalls) or with other instructions in between (e.g. clone). To avoid accessing the TCB a second time on PSEUDO_RET to check which instruction we used, the value read from hwcap2 is cached on a non-volatile register. This is not needed when using INTERNAL_SYSCALL macro, since entry and exit are bundled into the same inline asm directive. The dynamic loader may issue syscalls before the TCB has been setup so it always uses sc with no extra checks. For the static case, there is no compile-time way to determine if we are inside startup code, so we also check the value of the thread pointer before effectively accessing the TCB. For such situations in which the availability of scv cannot be determined, sc is always used. Support for scv in syscalls implemented in their own ASM file (clone and vfork) will be added later. For now simply use sc as before. Average performance over 1M calls for each syscall "type": - stat: C wrapper calling INTERNAL_SYSCALL - getpid: templated ASM syscall - syscall: call to gettid using syscall function Standard: stat : 1.573445 us / ~3619 cycles getpid : 0.164986 us / ~379 cycles syscall : 0.162743 us / ~374 cycles With scv: stat : 1.537049 us / ~3535 cycles <~ -84 cycles / -2.32% getpid : 0.109923 us / ~253 cycles <~ -126 cycles / -33.25% syscall : 0.116410 us / ~268 cycles <~ -106 cycles / -28.34% Tested on powerpc, powerpc64, powerpc64le (with and without scv) Tested-by: Lucas A. M. Magalhães <lamm@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-12-03 17:15:27 +00:00
# define INTERNAL_SYSCALL_NCS(name, nr, args...) \
({ \
DECLARE_REGS; \
LOADARGS_##nr (name, ##args); \
TRY_SYSCALL_SCV(nr) \
SYSCALL_SC(nr); \
})
#undef INTERNAL_SYSCALL
#define INTERNAL_SYSCALL(name, nr, args...) \
INTERNAL_SYSCALL_NCS (__NR_##name, nr, args)
#define LOADARGS_0(name, dummy) \
r0 = name
#define LOADARGS_1(name, __arg1) \
long int _arg1 = (long int) (__arg1); \
LOADARGS_0(name, 0); \
extern void __illegally_sized_syscall_arg1 (void); \
if (__builtin_classify_type (__arg1) != 5 \
&& sizeof (__arg1) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg1 (); \
r3 = _arg1
#define LOADARGS_2(name, __arg1, __arg2) \
long int _arg2 = (long int) (__arg2); \
LOADARGS_1(name, __arg1); \
extern void __illegally_sized_syscall_arg2 (void); \
if (__builtin_classify_type (__arg2) != 5 \
&& sizeof (__arg2) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg2 (); \
r4 = _arg2
#define LOADARGS_3(name, __arg1, __arg2, __arg3) \
long int _arg3 = (long int) (__arg3); \
LOADARGS_2(name, __arg1, __arg2); \
extern void __illegally_sized_syscall_arg3 (void); \
if (__builtin_classify_type (__arg3) != 5 \
&& sizeof (__arg3) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg3 (); \
r5 = _arg3
#define LOADARGS_4(name, __arg1, __arg2, __arg3, __arg4) \
long int _arg4 = (long int) (__arg4); \
LOADARGS_3(name, __arg1, __arg2, __arg3); \
extern void __illegally_sized_syscall_arg4 (void); \
if (__builtin_classify_type (__arg4) != 5 \
&& sizeof (__arg4) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg4 (); \
r6 = _arg4
#define LOADARGS_5(name, __arg1, __arg2, __arg3, __arg4, __arg5) \
long int _arg5 = (long int) (__arg5); \
LOADARGS_4(name, __arg1, __arg2, __arg3, __arg4); \
extern void __illegally_sized_syscall_arg5 (void); \
if (__builtin_classify_type (__arg5) != 5 \
&& sizeof (__arg5) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg5 (); \
r7 = _arg5
#define LOADARGS_6(name, __arg1, __arg2, __arg3, __arg4, __arg5, __arg6) \
long int _arg6 = (long int) (__arg6); \
LOADARGS_5(name, __arg1, __arg2, __arg3, __arg4, __arg5); \
extern void __illegally_sized_syscall_arg6 (void); \
if (__builtin_classify_type (__arg6) != 5 \
&& sizeof (__arg6) > SYSCALL_ARG_SIZE) \
__illegally_sized_syscall_arg6 (); \
r8 = _arg6
#define ASM_INPUT_0 "0" (r0)
#define ASM_INPUT_1 ASM_INPUT_0, "1" (r3)
#define ASM_INPUT_2 ASM_INPUT_1, "2" (r4)
#define ASM_INPUT_3 ASM_INPUT_2, "3" (r5)
#define ASM_INPUT_4 ASM_INPUT_3, "4" (r6)
#define ASM_INPUT_5 ASM_INPUT_4, "5" (r7)
#define ASM_INPUT_6 ASM_INPUT_5, "6" (r8)
/* Pointer mangling support. */
#if IS_IN (rtld)
/* We cannot use the thread descriptor because in ld.so we use setjmp
earlier than the descriptor is initialized. */
#else
# ifdef __ASSEMBLER__
# if defined(__PPC64__) || defined(__powerpc64__)
# define LOAD ld
# define TPREG r13
# else
# define LOAD lwz
# define TPREG r2
# endif
# define PTR_MANGLE(reg, tmpreg) \
LOAD tmpreg,POINTER_GUARD(TPREG); \
xor reg,tmpreg,reg
# define PTR_MANGLE2(reg, tmpreg) \
xor reg,tmpreg,reg
# define PTR_MANGLE3(destreg, reg, tmpreg) \
LOAD tmpreg,POINTER_GUARD(TPREG); \
xor destreg,tmpreg,reg
# define PTR_DEMANGLE(reg, tmpreg) PTR_MANGLE (reg, tmpreg)
# define PTR_DEMANGLE2(reg, tmpreg) PTR_MANGLE2 (reg, tmpreg)
# define PTR_DEMANGLE3(destreg, reg, tmpreg) PTR_MANGLE3 (destreg, reg, tmpreg)
# else
# define PTR_MANGLE(var) \
(var) = (__typeof (var)) ((uintptr_t) (var) ^ THREAD_GET_POINTER_GUARD ())
# define PTR_DEMANGLE(var) PTR_MANGLE (var)
# endif
#endif
/* List of system calls which are supported as vsyscalls. */
Remove PREPARE_VERSION and PREPARE_VERSION_KNOW This patch removes the PREPARE_VERSION and PREPARE_VERSION_KNOW macro and uses a static inline function instead, get_vdso_symbol. Each architecture that supports vDSO must define the Linux version and its hash for symbol resolution (VDSO_NAME and VDSO_HASH macro respectively). It also organizes the HAVE_*_VSYSCALL for mips, powerpc, and s390 to define them on a common header. The idea is to require less code to configure and enable vDSO support for newer ports. No semantic changes are expected. Checked with a build against all affected architectures. * sysdeps/unix/make-syscalls.sh: Make vDSO call use get_vdso_symbol. * sysdeps/unix/sysv/linux/aarch64/gettimeofday.c (__gettimeofday): Use get_vdso_symbol instead of _dl_vdso_vsym. * sysdeps/unix/sysv/linux/powerpc/time.c (time): Likewise. * sysdeps/unix/sysv/linux/riscv/flush-icache.c (__lookup_riscv_flush_icache): Likewise. * sysdeps/unix/sysv/linux/x86/gettimeofday.c (__gettimeofday): Likewise. * sysdeps/unix/sysv/linux/x86/time.c (time): Likewise. * sysdeps/unix/sysv/linux/powerpc/gettimeofday.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/arm/init-first.c: Likewise. * sysdeps/unix/sysv/linux/i386/init-first.c: Likewise. * sysdeps/unix/sysv/linux/mips/init-first.c: Likewise. * sysdeps/unix/sysv/linux/powerpc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise. * sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/s390/init-first.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/sysdep.h (VDSO_NAME, VDSO_HASH): Define. * sysdeps/unix/sysv/linux/arm/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/i386/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/riscv/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/sparc/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/x86_64/sysdep.h (VDSO_NAME, VDSO_HASH): Likewise. * sysdeps/unix/sysv/linux/mips/mips32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Remove definition. * sysdeps/unix/sysv/linux/mips/mips64/n32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/s390-64/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/sysdep.h: New file. * sysdeps/unix/sysv/linux/powerpc/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/s390/sysdep.h: Likewise. * sysdeps/unix/sysv/linux/dl-vdso.h (PREPARE_VERSION, PREPARE_VERSION_KNOWN, VDSO_NAME_LINUX_2_6, VDSO_HASH_LINUX_2_6, VDSO_NAME_LINUX_2_6_15, VDSO_HASH_LINUX_2_6_15, VDSO_NAME_LINUX_2_6_29, VDSO_HASH_LINUX_2_6_29, VDSO_NAME_LINUX_4_15, VDSO_HASH_LINUX_4_15): Remove defines. (get_vdso_symbol): New function.
2019-05-23 19:33:32 +00:00
#define VDSO_NAME "LINUX_2.6.15"
#define VDSO_HASH 123718565
#if defined(__PPC64__) || defined(__powerpc64__)
#define HAVE_CLOCK_GETRES64_VSYSCALL "__kernel_clock_getres"
#define HAVE_CLOCK_GETTIME64_VSYSCALL "__kernel_clock_gettime"
#else
#define HAVE_CLOCK_GETRES_VSYSCALL "__kernel_clock_getres"
Refactor vDSO initialization code Linux vDSO initialization code the internal function pointers require a lot of duplicated boilerplate over different architectures. This patch aims to simplify not only the code but the required definition to enable a vDSO symbol. The changes are: 1. Consolidate all init-first.c on only one implementation and enable the symbol based on HAVE_*_VSYSCALL existence. 2. Set the HAVE_*_VSYSCALL to the architecture expected names string. 3. Add a new internal implementation, get_vdso_mangle_symbol, which returns a mangled function pointer. Currently the clock_gettime, clock_getres, gettimeofday, getcpu, and time are handled in an arch-independent way, powerpc still uses some arch-specific vDSO symbol handled in a specific init-first implementation. Checked on aarch64-linux-gnu, arm-linux-gnueabihf, i386-linux-gnu, mips64-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu, sparc64-linux-gnu, and x86_64-linux-gnu. * sysdeps/powerpc/powerpc32/backtrace.c (is_sigtramp_address, is_sigtramp_address_rt): Use HAVE_SIGTRAMP_{RT}32 instead of SHARED. * sysdeps/powerpc/powerpc64/backtrace.c (is_sigtramp_address): Likewise. * sysdeps/unix/sysv/linux/aarch64/init-first.c: Remove file. * sysdeps/unix/sysv/linux/aarch64/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/arm/init-first.c: Likewise. * sysdeps/unix/sysv/linux/arm/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/mips/init-first.c: Likewise. * sysdeps/unix/sysv/linux/mips/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/i386/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/s390/init-first.c: Likewise. * sysdeps/unix/sysv/linux/s390/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/sparc/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/x86/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Define value based on kernel exported name. * sysdeps/unix/sysv/linux/arm/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/i386/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETCPU_VSYSCALL, HAVE_TIME_VSYSCALL, HAVE_GET_TBFREQ, HAVE_SIGTRAMP_RT64, HAVE_SIGTRAMP_32, HAVE_SIGTRAMP_RT32i, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/riscv/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/sparc/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/x86_64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/dl-vdso.h (VDSO_NAME, VDSO_HASH): Define to invalid names if architecture does not define them. (get_vdso_mangle_symbol): New symbol. * sysdeps/unix/sysv/linux/init-first.c: New file. * sysdeps/unix/sysv/linux/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/powerpc/init-first.c (gettimeofday, clock_gettime, clock_getres, getcpu, time): Remove declaration. (__libc_vdso_platform_setup_arch): Likewise and use get_vdso_mangle_symbol to setup vDSO symbols. (sigtramp_rt64, sigtramp32, sigtramp_rt32, get_tbfreq): Add attribute_hidden. * sysdeps/unix/sysv/linux/powerpc/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/sysdep-vdso.h (VDSO_SYMBOL): Remove definition.
2019-06-03 13:22:13 +00:00
#define HAVE_CLOCK_GETTIME_VSYSCALL "__kernel_clock_gettime"
#endif
Refactor vDSO initialization code Linux vDSO initialization code the internal function pointers require a lot of duplicated boilerplate over different architectures. This patch aims to simplify not only the code but the required definition to enable a vDSO symbol. The changes are: 1. Consolidate all init-first.c on only one implementation and enable the symbol based on HAVE_*_VSYSCALL existence. 2. Set the HAVE_*_VSYSCALL to the architecture expected names string. 3. Add a new internal implementation, get_vdso_mangle_symbol, which returns a mangled function pointer. Currently the clock_gettime, clock_getres, gettimeofday, getcpu, and time are handled in an arch-independent way, powerpc still uses some arch-specific vDSO symbol handled in a specific init-first implementation. Checked on aarch64-linux-gnu, arm-linux-gnueabihf, i386-linux-gnu, mips64-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu, sparc64-linux-gnu, and x86_64-linux-gnu. * sysdeps/powerpc/powerpc32/backtrace.c (is_sigtramp_address, is_sigtramp_address_rt): Use HAVE_SIGTRAMP_{RT}32 instead of SHARED. * sysdeps/powerpc/powerpc64/backtrace.c (is_sigtramp_address): Likewise. * sysdeps/unix/sysv/linux/aarch64/init-first.c: Remove file. * sysdeps/unix/sysv/linux/aarch64/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/arm/init-first.c: Likewise. * sysdeps/unix/sysv/linux/arm/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/mips/init-first.c: Likewise. * sysdeps/unix/sysv/linux/mips/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/i386/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/init-first.c: Likewise. * sysdeps/unix/sysv/linux/riscv/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/s390/init-first.c: Likewise. * sysdeps/unix/sysv/linux/s390/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/sparc/init-first.c: Likewise. * sysdeps/unix/sysv/linux/sparc/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/x86/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/init-first.c: Likewise. * sysdeps/unix/sysv/linux/aarch64/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Define value based on kernel exported name. * sysdeps/unix/sysv/linux/arm/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/i386/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/mips/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/powerpc/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETCPU_VSYSCALL, HAVE_TIME_VSYSCALL, HAVE_GET_TBFREQ, HAVE_SIGTRAMP_RT64, HAVE_SIGTRAMP_32, HAVE_SIGTRAMP_RT32i, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/riscv/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/s390/sysdep.h (HAVE_CLOCK_GETRES_VSYSCALL, HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/sparc/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/x86_64/sysdep.h (HAVE_CLOCK_GETTIME_VSYSCALL, HAVE_GETTIMEOFDAY_VSYSCALL, HAVE_GETCPU_VSYSCALL): Likewise. * sysdeps/unix/sysv/linux/dl-vdso.h (VDSO_NAME, VDSO_HASH): Define to invalid names if architecture does not define them. (get_vdso_mangle_symbol): New symbol. * sysdeps/unix/sysv/linux/init-first.c: New file. * sysdeps/unix/sysv/linux/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/powerpc/init-first.c (gettimeofday, clock_gettime, clock_getres, getcpu, time): Remove declaration. (__libc_vdso_platform_setup_arch): Likewise and use get_vdso_mangle_symbol to setup vDSO symbols. (sigtramp_rt64, sigtramp32, sigtramp_rt32, get_tbfreq): Add attribute_hidden. * sysdeps/unix/sysv/linux/powerpc/libc-vdso.h: Likewise. * sysdeps/unix/sysv/linux/sysdep-vdso.h (VDSO_SYMBOL): Remove definition.
2019-06-03 13:22:13 +00:00
#define HAVE_GETCPU_VSYSCALL "__kernel_getcpu"
#define HAVE_TIME_VSYSCALL "__kernel_time"
#define HAVE_GETTIMEOFDAY_VSYSCALL "__kernel_gettimeofday"
#define HAVE_GET_TBFREQ "__kernel_get_tbfreq"
#endif /* _LINUX_POWERPC_SYSDEP_H */