Hide internal __strsep function to allow direct access within libc.so and
libc.a without using GOT nor PLT.
[BZ #18822]
* include/string.h (__strsep): Add libc_hidden_proto.
* string/strsep.c (__strsep): Add libc_hidden_def.
Hide internal __posix_spawn_file_actions_realloc and /__spawni functions
to allow direct access within libc.so and libc.a without using GOT nor
PLT.
[BZ #18822]
* posix/spawn_int.h (__posix_spawn_file_actions_realloc): Add
attribute_hidden.
(__spawni): Likewise.
Hide internal __mremap function to allow direct access within libc.so and
libc.a without using GOT nor PLT.
__GI___mremap is defined when sysdeps/unix/syscalls.list is used to
generate mremap. Otherwise libc_hidden_def is needed explicitly.
[BZ #18822]
* include/sys/mman.h (__mremap): Add libc_hidden_proto.
* sysdeps/unix/sysv/linux/m68k/mremap.S (__mremap): Add
libc_hidden_def.
Hide internal __malloc_check_init function to allow direct access within
libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* include/malloc.h (__malloc_check_init): Add attribute_hidden.
Hide internal __ioctl function to allow direct access within libc.so and
libc.a without using GOT nor PLT.
__GI___ioctl is defined when sysdeps/unix/syscalls.list is used to
generate ioctl. Otherwise libc_hidden_def is needed explicitly.
[BZ #18822]
* include/sys/ioctl.h (__ioctl): Add libc_hidden_proto.
* misc/ioctl.c (__ioctl): Add libc_hidden_def.
* sysdeps/mach/hurd/ioctl.c (__ioctl): Likewise.
* sysdeps/unix/sysv/linux/aarch64/ioctl.S (__ioctl): Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/ioctl.S (__ioctl):
Likewise.
* sysdeps/unix/sysv/linux/powerpc/ioctl.c (__ioctl): Likewise.
* sysdeps/unix/sysv/linux/tile/tilegx/ioctl.S (__ioctl): Likewise.
Mark internal intl functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* intl/gettextP.h (__dcngettext): Add attribute_hidden.
(__dcigettext): Likewise.
Mark internal netlink functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* sysdeps/unix/sysv/linux/netlinkaccess.h (__netlink_open): Add
attribute_hidden.
(__netlink_close): Likewise.
(__netlink_free_handle): Likewise.
(__netlink_request): Likewise.
Mark internal rpc functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* include/rpc/rpc.h (__rpc_thread_variables): Add
attribute_hidden.
(__rpc_thread_svc_cleanup): Likewise.
(__rpc_thread_clnt_cleanup): Likewise.
(__rpc_thread_key_cleanup): Likewise.
Mark internal dirent functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT. __readdir64
is hidden with libc_hidden_proto and libc_hidden_def since the exported
readdir64 is an alias of __readdir64.
[BZ #18822]
* include/dirent.h (__opendir): Always add attribute_hidden.
(__fdopendir): Likewise.
(__closedir): Likewise.
(__readdir): Likewise.
(__readdir64): Add libc_hidden_proto.
* sysdeps/mach/hurd/readdir64.c (__readdir64): Add libc_hidden_def.
* sysdeps/unix/sysv/linux/i386/readdir64.c (__readdir64): Likewise.
* sysdeps/unix/sysv/linux/readdir64.c (__readdir64): Likewise.
* sysdeps/unix/sysv/linux/wordsize-64/readdir.c (__GI___readdir64):
New alias.
Mark internal stdio functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* include/stdio.h (__fcloseall): Add attribute_hidden.
(__getline): Likewise.
(__path_search): Likewise.
(__gen_tempname): Likewise.
(__libc_message): Likewise.
(__flockfile): Likewise.
(__funlockfile): Likewise.
(__fxprintf): Likewise.
(__fxprintf_nocancel): Likewise.
Mark internal gshadow functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* include/gshadow.h (__fgetsgent_r): Add attribute_hidden.
(__sgetsgent_r): Likewise.
Mark internal time functions with attribute_hidden to allow direct
access within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* include/time.h (__tzstring): Add attribute_hidden.
(__tzfile_read): Likewise.
(__tzfile_compute): Likewise.
(__tzfile_default): Likewise.
(__tzset_parse_tz): Likewise.
(__offtime): Likewise.
(__asctime_r): Likewise.
(__tzset): Likewise.
(__tz_convert): Likewise.
(__getdate_r): Likewise.
(__getclktck): Likewise.
Mark internal argp functions with attribute_hidden to allow direct
access to them within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* argp/argp-fmtstream.c: Include <argp-fmtstream.h>.
* argp/argp-fs-xinl.c: Likewise.
* argp/argp-help.c: Include <argp.h> and <argp-fmtstream.h>.
* argp/argp-parse.c: Include <argp.h>.
* argp/argp-xinl.c: Likewise.
* include/argp-fmtstream.h: New file.
* include/argp.h (__argp_error): Add attribute_hidden.
(__argp_failure): Likewise.
(__argp_input): Likewise.
(__argp_state_help): Likewise.
Mark __internal_statvfs[64] with attribute_hidden to allow direct access
to them within libc.so and libc.a without using GOT nor PLT.
[BZ #18822]
* sysdeps/unix/sysv/linux/fstatvfs.c: Include "internal_statvfs.h"
instead of <sys/statvfs.h>.
(__internal_statvfs): Removed.
* sysdeps/unix/sysv/linux/fstatvfs64.c Include "internal_statvfs.h"
instead of <sys/statvfs.h>.
(__internal_statvfs64): Removed.
* sysdeps/unix/sysv/linux/internal_statvfs.c: Include
"internal_statvfs.h" instead of <sys/statvfs.h>.
* sysdeps/unix/sysv/linux/internal_statvfs.h: New file.
* sysdeps/unix/sysv/linux/statvfs.c Include "internal_statvfs.h"
instead of <sys/statvfs.h>.
(__internal_statvfs): Removed.
* sysdeps/unix/sysv/linux/statvfs64.c Include "internal_statvfs.h"
instead of <sys/statvfs.h>.
(__internal_statvfs64): Removed.
Mark __ptsname_internal, __mktime_internal and __fopen_internal with
attribute_hidden to allow direct access to them within libc.so and
libc.a without using GOT nor PLT.
[BZ #18822]
* include/stdlib.h (__ptsname_internal): Add attribute_hidden.
* include/time.h (__mktime_internal): Likewise.
* libio/iolibio.h (__fopen_internal): Likewise.
__setcontext on hppa.
* sysdeps/unix/sysv/linux/hppa/getcontext.S (__getcontext): Save return
pointer in frame.
* sysdeps/unix/sysv/linux/hppa/setcontext.S (__setcontext): Likewise.
Correct offset used to restore PIC register.
tst-tls1-static-non-pie is built with $(no-pie-ldflag) to make it a
non-PIE static executable, regardless if --enable-static-pie is used
to configure glibc.
* elf/Makefile (tests-static-internal): Add
tst-tls1-static-non-pie.
(LDFLAGS-tst-tls1-static-non-pie): New.
* elf/tst-tls1-static-non-pie.c: New file.
Continuing the move of libm aliases to common macros that can create
_FloatN / _FloatNx aliases in future, this patch converts some dbl-64
functions to using libm_alias_double, thereby eliminating the need for
some ldbl-opt wrappers.
This patch deliberately limits what functions are converted so that it
can be verified by comparison of stipped binaries. Specifically, atan
and tan are excluded because they first need converting to being weak
aliases; fma is omitted as it has additional complications with
versions in other directories (removing the ldbl-opt version can
e.g. cause the ldbl-128 version to be used instead of dbl-64); and
functions that have both dbl-64/wordsize-64 and ldbl-opt versions are
excluded because ldbl-opt currently always wraps dbl-64 function
versions, so changing those will result in platforms using both
ldbl-opt and dbl-64/wordsize-64 (i.e. alpha) starting to use the
dbl-64/wordsize-64 versions of those functions (which is good, as an
optimization, but still best separated from the present patch to get
better validation).
Tested for x86_64, and tested with build-many-glibcs.py that installed
stripped shared libraries are unchanged by the patch.
* sysdeps/ieee754/dbl-64/s_asinh.c: Include <libm-alias-double.h>.
(asinh): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_cbrt.c: Include <libm-alias-double.h>.
(cbrt): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_copysign.c: Include
<libm-alias-double.h>.
(copysign): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_erf.c: Include <libm-alias-double.h>.
(erf): Define using libm_alias_double.
(erfc): Likewise.
* sysdeps/ieee754/dbl-64/s_expm1.c: Include <libm-alias-double.h>.
(expm1): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_fabs.c: Include <libm-alias-double.h>.
(fabs): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_fromfp.c (fromfp): Define using
libm_alias_double.
* sysdeps/ieee754/dbl-64/s_fromfp_main.c: Include
<libm-alias-double.h>.
* sysdeps/ieee754/dbl-64/s_fromfpx.c (fromfpx): Define using
libm_alias_double.
* sysdeps/ieee754/dbl-64/s_getpayload.c: Include
<libm-alias-double.h>.
(getpayload): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_llrint.c: Include
<libm-alias-double.h>.
(llrint): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_lrint.c: Include <libm-alias-double.h>.
(lrint): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_nextup.c: Include
<libm-alias-double.h>.
(nextup): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_roundeven.c: Include
<libm-alias-double.h>.
(roundeven): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_setpayload.c (setpayload): Define using
libm_alias_double.
* sysdeps/ieee754/dbl-64/s_setpayload_main.c: Include
<libm-alias-double.h>.
* sysdeps/ieee754/dbl-64/s_setpayloadsig.c (setpayloadsig): Define
using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_sin.c: Include <libm-alias-double.h>.
(cos): Define using libm_alias_double.
(sin): Likewise.
* sysdeps/ieee754/dbl-64/s_sincos.c: Include
<libm-alias-double.h>.
(sincos): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_tanh.c: Include <libm-alias-double.h>.
(tanh): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_totalorder.c: Include
<libm-alias-double.h>.
(totalorder): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_totalordermag.c: Include
<libm-alias-double.h>.
(totalordermag): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/s_ufromfp.c (ufromfp): Define using
libm_alias_double.
* sysdeps/ieee754/dbl-64/s_ufromfpx.c (ufromfpx): Define using
libm_alias_double.
* sysdeps/ieee754/dbl-64/wordsize-64/s_getpayload.c: Include
<libm-alias-double.h>.
(getpayload): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/wordsize-64/s_roundeven.c: Include
<libm-alias-double.h>.
(roundeven): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/wordsize-64/s_setpayload_main.c: Include
<libm-alias-double.h>.
* sysdeps/ieee754/dbl-64/wordsize-64/s_totalorder.c: Include
<libm-alias-double.h>.
(totalorder): Define using libm_alias_double.
* sysdeps/ieee754/dbl-64/wordsize-64/s_totalordermag.c: Include
<libm-alias-double.h>.
(totalordermag): Define using libm_alias_double.
* sysdeps/ieee754/ldbl-opt/s_copysign.c (copysignl): Only define
libc compat symbol here.
* sysdeps/ieee754/ldbl-opt/s_asinh.c: Remove file.
* sysdeps/ieee754/ldbl-opt/s_cbrt.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_erf.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_expm1.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_fabs.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_llrint.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_lrint.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_sin.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_sincos.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_tanh.c: Likewise.
When --enable-static-pie is used to configure glibc, we need to use
_dl_relocate_static_pie to compute load address in static PIE.
* sysdeps/arm/dl-machine.h (elf_machine_load_address): Use
_dl_relocate_static_pie instead of _dl_start to compute load
address in static PIE. Return 0 if _DYNAMIC is undefined for
static executable.
mips uses a local label to compute load address, which works with static
PIE. We just need to return 0 if _DYNAMIC is undefined for static
executable.
* sysdeps/mips/dl-machine.h (elf_machine_dynamic): Return 0 if
_DYNAMIC is undefined for static executable.
A few math functions still use __fabs(f/l) rather than fabs, which
means they won't be inlined. Rename them so they are inlined.
Also add -fno-builtin-fabsl to nofpu powerpc makefile to work around
BZ #29253.
* sysdeps/ieee754/dbl-64/e_lgamma_r.c
(__ieee754_lgamma_r): Use fabs rather than __fabs.
* sysdeps/ieee754/dbl-64/e_log10.c (__ieee754_log10): Likewise.
* sysdeps/ieee754/dbl-64/e_log2.c (__ieee754_log2): Likewise.
* sysdeps/ieee754/flt-32/e_lgammaf_r.c
(__ieee754_lgammaf_r): Use fabsf rather than __fabsf.
* sysdeps/ieee754/flt-32/e_log10f.c (__ieee754_log10f): Likewise.
* sysdeps/ieee754/flt-32/e_log2f.c (__ieee754_log2f): Likewise.
* sysdeps/ieee754/ldbl-128/e_lgammal_r.c
(__ieee754_lgammal_r): Use fabsl rather than __fabsl.
* sysdeps/ieee754/ldbl-128/e_log10l.c (__ieee754_log10l): Likewise.
* sysdeps/ieee754/ldbl-128/e_log2l.c (__ieee754_log2l): Likewise.
* sysdeps/ieee754/ldbl-128ibm/e_lgammal_r.c
(__ieee754_lgammal_r): Use fabsl rather than __fabsl.
* sysdeps/ieee754/ldbl-128ibm/e_log10l.c (__ieee754_log10l): Likewise.
* sysdeps/ieee754/ldbl-128ibm/e_log2l.c (__ieee754_log2l): Likewise.
* sysdeps/powerpc/nofpu/Makefile: Add -fno-builtin-fabsl for BZ #29253.
Remove old cimag inlines that are only used in GCC 2.96 or older.
* math/bits/cmathcalls.h (cimag): Remove inline.
(creal): Remove inline.
(conj): Remove inline.
without wrapper on aarch64:
powf reciprocal-throughput: 4.2x faster
powf latency: 2.6x faster
old worst-case error: 1.11 ulp
new worst-case error: 0.82 ulp
aarch64 .text size: -780 bytes
aarch64 .rodata size: +144 bytes
powf(x,y) is implemented as exp2(y*log2(x)) with the same algorithms
that are used in exp2f and log2f, except that the log2f polynomial is
larger for extra precision and its output (and exp2f input) may be
scaled by a power of 2 (POWF_SCALE) to simplify the argument reduction
step of exp2 (possible when efficient round and convert toint operation
is available).
The special case handling tries to minimize the checks in the hot path.
When the input of exp2_inline is checked, int arithmetics is used as
that was faster on the tested aarch64 cores.
* math/Makefile (type-float-routines): Add e_powf_log2_data.
* sysdeps/ieee754/flt-32/e_powf.c: New implementation.
* sysdeps/ieee754/flt-32/e_powf_log2_data.c: New file.
* sysdeps/ieee754/flt-32/math_config.h (__powf_log2_data): Define.
(issignalingf_inline): Likewise.
(POWF_LOG2_TABLE_BITS): Likewise.
(POWF_LOG2_POLY_ORDER): Likewise.
(POWF_SCALE_BITS): Likewise.
(POWF_SCALE): Likewise.
* sysdeps/i386/fpu/e_powf_log2_data.c: New file.
* sysdeps/ia64/fpu/e_powf_log2_data.c: New file.
* sysdeps/m68k/m680x0/fpu/e_powf_log2_data.c: New file.
Similar to the new logf: double precision arithmetics and a small
lookup table is used. The argument reduction step is the same as in
the new logf.
without wrapper on aarch64:
log2f reciprocal-throughput: 2.3x faster
log2f latency: 2.1x faster
old worst case error: 1.72 ulp
new worst case error: 0.75 ulp
aarch64 .text size: -252 bytes
aarch64 .rodata size: +244 bytes
* math/Makefile (type-float-routines): Add e_log2f_data.
* sysdeps/ieee754/flt-32/e_log2f.c: New implementation.
* sysdeps/ieee754/flt-32/e_log2f_data.c: New file.
* sysdeps/ieee754/flt-32/math_config.h (__log2f_data): Define.
(LOG2F_TABLE_BITS, LOG2F_POLY_ORDER): Define.
* sysdeps/i386/fpu/e_log2f_data.c: New file.
* sysdeps/ia64/fpu/e_log2f_data.c: New file.
* sysdeps/m68k/m680x0/fpu/e_log2f_data.c: New file.
without wrapper on aarch64:
logf reciprocal-throughput: 2.2x faster
logf latency: 1.9x faster
old worst case error: 0.89 ulp
new worst case error: 0.82 ulp
aarch64 .text size: -356 bytes
aarch64 .rodata size: +240 bytes
Uses double precision arithmetics and a lookup table to allow smaller
polynomial and avoid the use of division.
Data is in a separate translation unit with fixed layout to prevent the
compiler generating suboptimal literal access.
Errors are handled inline according to POSIX rules, but this patch
keeps the wrapper with SVID compatible error handling.
Needs libm-test-ulps adjustment for clogf in non-nearest rounding mode.
* math/Makefile (type-float-routines): Add e_logf_data.
* sysdeps/ieee754/flt-32/e_logf.c: New implementation.
* sysdeps/ieee754/flt-32/e_logf_data.c: New file.
* sysdeps/ieee754/flt-32/math_config.h (__logf_data): Define.
(LOGF_TABLE_BITS, LOGF_POLY_ORDER): Define.
* sysdeps/i386/fpu/e_logf_data.c: New file.
* sysdeps/ia64/fpu/e_logf_data.c: New file.
* sysdeps/m68k/m680x0/fpu/e_logf_data.c: New file.
When --enable-static-pie is used to build static PIE, _DYNAMIC is used
to compute the load address of static PIE. But _DYNAMIC is undefined
when creating static executable. This patch makes _DYNAMIC weak in PIE
libc.a so that it can be undefined.
* sysdeps/i386/dl-machine.h (elf_machine_load_address): Allow
undefined _DYNAMIC in PIE libc.a.
* sysdeps/x86_64/dl-machine.h (elf_machine_load_address):
Likewse.
Simplify the C99 isgreater macros. Although some support was added
in GCC 2.97, not all targets added support until GCC 3.1. Therefore
only use the builtins in math.h from GCC 3.1 onwards, and defer to
generic macros otherwise. Improve the generic isunordered macro
to use compares rather than call fpclassify twice - this is not only
faster but also correct for signaling NaNs.
* math/math.h: Improve handling of C99 isgreater macros.
* sysdeps/alpha/fpu/bits/mathinline.h: Remove isgreater macros.
* sysdeps/m68k/m680x0/fpu/bits/mathinline.h: Likewise.
* sysdeps/powerpc/bits/mathinline.h: Likewise.
* sysdeps/sparc/fpu/bits/mathinline.h: Likewise.
* sysdeps/x86/fpu/bits/mathinline.h: Likewise.
In <https://sourceware.org/ml/libc-alpha/2013-05/msg00722.html> I
remarked on the possibility of arithmetic in various nearbyint
implementations being scheduled before feholdexcept calls, resulting
in spurious "inexact" exceptions.
I'm now actually observing this occurring in glibc built for ARM with
GCC 7 (in fact, both copies of the same addition/subtraction sequence
being combined and moved out before the conditionals and
feholdexcept/fesetenv pairs), resulting in test failures.
This patch makes the nearbyint implementations with this particular
feholdexcept / arithmetic / fesetenv pattern consistently use
math_opt_barrier on the function argument when first used in
arithmetic, and also consistently use math_force_eval before fesetenv
(the latter was generally already done, but the dbl-64/wordsize-64
implementation used math_opt_barrier instead, and as
math_opt_barrier's intended effect is through its output value being
used, such a use that doesn't use the return value is suspect).
Tested for x86_64 (--disable-multi-arch so more of these
implementations get used), and for ARM in a configuration where I saw
the problem scheduling.
[BZ #22225]
* sysdeps/ieee754/dbl-64/s_nearbyint.c (__nearbyint): Use
math_opt_barrier on argument when doing arithmetic on it.
* sysdeps/ieee754/dbl-64/wordsize-64/s_nearbyint.c (__nearbyint):
Likewise. Use math_force_eval not math_opt_barrier after
arithmetic.
* sysdeps/ieee754/flt-32/s_nearbyintf.c (__nearbyintf): Use
math_opt_barrier on argument when doing arithmetic on it.
* sysdeps/ieee754/ldbl-128/s_nearbyintl.c (__nearbyintl):
Likewise.
