Adds a fast path to e_exp.c when |x| < 1.03972053527832.
When values are tested in isolation, reduction in execution
time is: aarch 30%, sparc 18%, x86 37%.
When comparing benchtests/bench.out which includes values
outside that range, the gains are:
aarch 8%, sparc 5%, x86 9%.
make check is clean (no increase in ulp for any math test).
Testing 20M values for each rounding mode in that range shows
approximately one in 200 values is off by 1 ulp. No value tested
for exp(x) changed by 2 or more ulp.
No observed change in performance or accuracy for x outside
fast path range.
These changes will be active for all platforms that don't provide
their own exp() routines. They will also be active for ieee754
versions of ccos, ccosh, cosh, csin, csinh, sinh, exp10, gamma, and
erf.
Remove the slow paths from pow. Like several other double precision math
functions, pow is exactly rounded. This is not required from math functions
and causes major overheads as it requires multiple fallbacks using higher
precision arithmetic if a result is close to 0.5ULP. Ridiculous slowdowns
of up to 100000x have been reported when the highest precision path triggers.
All GLIBC math tests pass on AArch64 and x64 (with ULP of pow set to 1).
The worst case error is ~0.506ULP. A simple test over a few hundred million
values shows pow is 10% faster on average. This fixes BZ #13932.
[BZ #13932]
* sysdeps/ieee754/dbl-64/uexp.h (err_1): Remove.
* benchtests/pow-inputs: Update comment for slow path cases.
* manual/probes.texi (slowpow_p10): Delete removed probe.
(slowpow_p10): Likewise.
* math/Makefile: Remove halfulp.c and slowpow.c.
* sysdeps/aarch64/libm-test-ulps: Set ULP of pow to 1.
* sysdeps/generic/math_private.h (__exp1): Remove error argument.
(__halfulp): Remove.
(__slowpow): Remove.
* sysdeps/i386/fpu/halfulp.c: Delete file.
* sysdeps/i386/fpu/slowpow.c: Likewise.
* sysdeps/ia64/fpu/halfulp.c: Likewise.
* sysdeps/ia64/fpu/slowpow.c: Likewise.
* sysdeps/ieee754/dbl-64/e_exp.c (__exp1): Remove error argument,
improve comments and add error analysis.
* sysdeps/ieee754/dbl-64/e_pow.c (__ieee754_pow): Add error analysis.
(power1): Remove function:
(log1): Remove error argument, add error analysis.
(my_log2): Remove function.
* sysdeps/ieee754/dbl-64/halfulp.c: Delete file.
* sysdeps/ieee754/dbl-64/slowpow.c: Likewise.
* sysdeps/m68k/m680x0/fpu/halfulp.c: Likewise.
* sysdeps/m68k/m680x0/fpu/slowpow.c: Likewise.
* sysdeps/powerpc/power4/fpu/Makefile: Remove CPPFLAGS-slowpow.c.
* sysdeps/x86_64/fpu/libm-test-ulps: Set ULP of pow to 1.
* sysdeps/x86_64/fpu/multiarch/Makefile: Remove slowpow-fma.c,
slowpow-fma4.c, halfulp-fma.c, halfulp-fma4.c.
* sysdeps/x86_64/fpu/multiarch/e_pow-fma.c (__slowpow): Remove define.
* sysdeps/x86_64/fpu/multiarch/e_pow-fma4.c (__slowpow): Likewise.
* sysdeps/x86_64/fpu/multiarch/halfulp-fma.c: Delete file.
* sysdeps/x86_64/fpu/multiarch/halfulp-fma4.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowpow-fma.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowpow-fma4.c: Likewise.
These changes will be active for all platforms that don't provide
their own exp() routines. They will also be active for ieee754
versions of ccos, ccosh, cosh, csin, csinh, sinh, exp10, gamma, and
erf.
Typical performance gains is typically around 5x when measured on
Sparc s7 for common values between exp(1) and exp(40).
Using the glibc perf tests on sparc,
sparc (nsec) x86 (nsec)
old new old new
max 17629 395 5173 144
min 399 54 15 13
mean 5317 200 1349 23
The extreme max times for the old (ieee754) exp are due to the
multiprecision computation in the old algorithm when the true value is
very near 0.5 ulp away from an value representable in double
precision. The new algorithm does not take special measures for those
cases. The current glibc exp perf tests overrepresent those values.
Informal testing suggests approximately one in 200 cases might
invoke the high cost computation. The performance advantage of the new
algorithm for other values is still large but not as large as indicated
by the chart above.
Glibc correctness tests for exp() and expf() were run. Within the
test suite 3 input values were found to cause 1 bit differences (ulp)
when "FE_TONEAREST" rounding mode is set. No differences in exp() were
seen for the tested values for the other rounding modes.
Typical example:
exp(-0x1.760cd2p+0) (-1.46113312244415283203125)
new code: 2.31973271630014299393707e-01 0x1.db14cd799387ap-3
old code: 2.31973271630014271638132e-01 0x1.db14cd7993879p-3
exp = 2.31973271630014285508337 (high precision)
Old delta: off by 0.49 ulp
New delta: off by 0.51 ulp
In addition, because ieee754_exp() is used by other routines, cexp()
showed test results with very small imaginary input values where the
imaginary portion of the result was off by 3 ulp when in upward
rounding mode, but not in the other rounding modes. For x86, tgamma
showed a few values where the ulp increased to 6 (max ulp for tgamma
is 5). Sparc tgamma did not show these failures. I presume the tgamma
differences are due to compiler optimization differences within the
gamma function.The gamma function is known to be difficult to compute
accurately.
* sysdeps/ieee754/dbl-64/e_exp.c: Include <math-svid-compat.h> and
<errno.h>. Include "eexp.tbl".
(half): New constant.
(one): Likewise.
(__ieee754_exp): Rewrite.
(__slowexp): Remove prototype.
* sysdeps/ieee754/dbl-64/eexp.tbl: New file.
* sysdeps/ieee754/dbl-64/slowexp.c: Remove file.
* sysdeps/i386/fpu/slowexp.c: Likewise.
* sysdeps/ia64/fpu/slowexp.c: Likewise.
* sysdeps/m68k/m680x0/fpu/slowexp.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-avx.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-fma.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-fma4.c: Likewise.
* sysdeps/generic/math_private.h (__slowexp): Remove prototype.
* sysdeps/ieee754/dbl-64/e_pow.c: Remove mention of slowexp.c in
comment.
* sysdeps/powerpc/power4/fpu/Makefile [$(subdir) = math]
(CPPFLAGS-slowexp.c): Remove variable.
* sysdeps/x86_64/fpu/multiarch/Makefile (libm-sysdep_routines):
Remove slowexp-fma, slowexp-fma4 and slowexp-avx.
(CFLAGS-slowexp-fma.c): Remove variable.
(CFLAGS-slowexp-fma4.c): Likewise.
(CFLAGS-slowexp-avx.c): Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp-avx.c (__slowexp): Do not
define as macro.
* sysdeps/x86_64/fpu/multiarch/e_exp-fma.c (__slowexp): Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp-fma4.c (__slowexp): Likewise.
* math/Makefile (type-double-routines): Remove slowexp.
* manual/probes.texi (slowexp_p6): Remove.
(slowexp_p32): Likewise.
My recent exp patch introduced warnings about implicit __isinf
declarations in exp because e_exp.c didn't include <math.h>. This
patch fixes this. Because <math.h> can't be included after
<math_private.h> (because of macro definitions of __nan*), it was
necessary to put an include in sysdeps/x86_64/fpu/multiarch/e_exp.c as
well.
Tested x86_64.
* sysdeps/ieee754/dbl-64/e_exp.c: Include <math.h>.
* sysdeps/x86_64/fpu/multiarch/e_exp.c
[HAVE_FMA4_SUPPORT || HAVE_AVX_SUPPORT]: Likewise.
The dbl-64 version of exp needs round-to-nearest mode for its internal
computations, but that has the consequence of inappropriate
overflowing and underflowing results in other rounding modes. This
patch fixes this by recomputing the relevant results in cases where
the round-to-nearest result overflows to infinity or underflows to
zero (most of the diffs are actually just consequent reindentation).
Tests are enabled in all rounding modes for complex functions using
exp - but not for cexp because it turns out there are bugs causing
spurious underflows for cexp for some tests, which will need to be
fixed separately (I suspect ccos ccosh csin csinh ctan ctanh have
similar bugs, just not shown by the present set of test inputs).
Tested x86_64 and x86 and ulps updated accordingly.
[BZ #16284]
* sysdeps/ieee754/dbl-64/e_exp.c (__ieee754_exp): Use original
rounding mode to recompute results that overflow to infinity or
underflow to zero.
* math/auto-libm-test-in: Don't mark tests as expected to fail for
bug 16284.
* math/auto-libm-test-out: Regenerated.
* math/libm-test.inc (ccos_test): Use ALL_RM_TEST.
(ccosh_test): Likewise.
(csin_test_data): Use plus_oflow.
(csin_test): Use ALL_RM_TEST.
(csinh_test_data): Use plus_oflow.
(csinh_test): Use ALL_RM_TEST.
* sysdeps/i386/fpu/libm-test-ulps: Update.
* sysdeps/x86_64/fpu/libm-test-ulps: Likewise.
2001-03-11 Ulrich Drepper <drepper@redhat.com>
Last-bit accurate math library implementation by IBM Haifa.
Contributed by Abraham Ziv <ziv@il.ibm.com>, Moshe Olshansky
<olshansk@il.ibm.com>, Ealan Henis <ealan@il.ibm.com>, and
Anna Reitman <reitman@il.ibm.com>.
* math/Makefile (dbl-only-routines): New variable.
(libm-routines): Add $(dbl-only-routines).
* sysdeps/ieee754/dbl-64/e_acos.c: Empty, definition is in e_asin.c.
* sysdeps/ieee754/dbl-64/e_asin.c: Replaced with accurate asin
implementation.
* sysdeps/ieee754/dbl-64/e_atan2.c: Replaced with accurate atan2
implementation.
* sysdeps/ieee754/dbl-64/e_exp.c: Replaced with accurate exp
implementation.
* sysdeps/ieee754/dbl-64/e_lgamma_r.c: Don't use __kernel_sin and
__kernel_cos.
* sysdeps/ieee754/dbl-64/e_log.c: Replaced with accurate log
implementation.
* sysdeps/ieee754/dbl-64/e_remainder.c: Replaced with accurate
remainder implementation.
* sysdeps/ieee754/dbl-64/e_pow.c: Replaced with accurate pow
implementation.
* sysdeps/ieee754/dbl-64/e_sqrt.c: Replaced with accurate sqrt
implementation.
* sysdeps/ieee754/dbl-64/k_cos.c: Empty, definition is in s_sin.c.
* sysdeps/ieee754/dbl-64/k_sin.c: Empty, definition is in s_sin.c.
* sysdeps/ieee754/dbl-64/s_atan.c: Replaced with accurate atan
implementation.
* sysdeps/ieee754/dbl-64/s_cos.c: Empty, definition is in s_sin.c.
* sysdeps/ieee754/dbl-64/s_sin.c: Replaced with accurate sin/cos
implementation.
* sysdeps/ieee754/dbl-64/s_sincos.c: Rewritten to not use __kernel_sin
and __kernel_cos.
* sysdeps/ieee754/dbl-64/s_tan.c: Replaced with accurate tan
implementation.
* sysdeps/ieee754/dbl-64/Dist: Add new non-code files.
* sysdeps/ieee754/dbl-64/MathLib.h: New file.
* sysdeps/ieee754/dbl-64/asincos.tbl: New file.
* sysdeps/ieee754/dbl-64/atnat.h: New file.
* sysdeps/ieee754/dbl-64/atnat2.h: New file.
* sysdeps/ieee754/dbl-64/branred.c: New file.
* sysdeps/ieee754/dbl-64/branred.h: New file.
* sysdeps/ieee754/dbl-64/dla.h: New file.
* sysdeps/ieee754/dbl-64/doasin.c: New file.
* sysdeps/ieee754/dbl-64/doasin.h: New file.
* sysdeps/ieee754/dbl-64/dosincos.c: New file.
* sysdeps/ieee754/dbl-64/dosincos.h: New file.
* sysdeps/ieee754/dbl-64/endian.h: New file.
* sysdeps/ieee754/dbl-64/halfulp.c: New file.
* sysdeps/ieee754/dbl-64/mpa.c: New file.
* sysdeps/ieee754/dbl-64/mpa.h: New file.
* sysdeps/ieee754/dbl-64/mpa2.h: New file.
* sysdeps/ieee754/dbl-64/mpatan.c: New file.
* sysdeps/ieee754/dbl-64/mpatan.h: New file.
* sysdeps/ieee754/dbl-64/mpatan2.c: New file.
* sysdeps/ieee754/dbl-64/mpexp.c: New file.
* sysdeps/ieee754/dbl-64/mpexp.h: New file.
* sysdeps/ieee754/dbl-64/mplog.c: New file.
* sysdeps/ieee754/dbl-64/mplog.h: New file.
* sysdeps/ieee754/dbl-64/mpsqrt.c: New file.
* sysdeps/ieee754/dbl-64/mpsqrt.h: New file.
* sysdeps/ieee754/dbl-64/mptan.c: New file.
* sysdeps/ieee754/dbl-64/mydefs.h: New file.
* sysdeps/ieee754/dbl-64/powtwo.tbl: New file.
* sysdeps/ieee754/dbl-64/root.tbl: New file.
* sysdeps/ieee754/dbl-64/sincos.tbl: New file.
* sysdeps/ieee754/dbl-64/sincos32.c: New file.
* sysdeps/ieee754/dbl-64/sincos32.h: New file.
* sysdeps/ieee754/dbl-64/slowexp.c: New file.
* sysdeps/ieee754/dbl-64/slowpow.c: New file.
* sysdeps/ieee754/dbl-64/uasncs.h: New file.
* sysdeps/ieee754/dbl-64/uatan.tbl: New file.
* sysdeps/ieee754/dbl-64/uexp.h: New file.
* sysdeps/ieee754/dbl-64/uexp.tbl: New file.
* sysdeps/ieee754/dbl-64/ulog.h: New file.
* sysdeps/ieee754/dbl-64/ulog.tbl: New file.
* sysdeps/ieee754/dbl-64/upow.h: New file.
* sysdeps/ieee754/dbl-64/upow.tbl: New file.
* sysdeps/ieee754/dbl-64/urem.h: New file.
* sysdeps/ieee754/dbl-64/uroot.h: New file.
* sysdeps/ieee754/dbl-64/usncs.h: New file.
* sysdeps/ieee754/dbl-64/utan.h: New file.
* sysdeps/ieee754/dbl-64/utan.tbl: New file.
* sysdeps/i386/fpu/branred.c: New file.
* sysdeps/i386/fpu/doasin.c: New file.
* sysdeps/i386/fpu/dosincos.c: New file.
* sysdeps/i386/fpu/halfulp.c: New file.
* sysdeps/i386/fpu/mpa.c: New file.
* sysdeps/i386/fpu/mpatan.c: New file.
* sysdeps/i386/fpu/mpatan2.c: New file.
* sysdeps/i386/fpu/mpexp.c: New file.
* sysdeps/i386/fpu/mplog.c: New file.
* sysdeps/i386/fpu/mpsqrt.c: New file.
* sysdeps/i386/fpu/mptan.c: New file.
* sysdeps/i386/fpu/sincos32.c: New file.
* sysdeps/i386/fpu/slowexp.c: New file.
* sysdeps/i386/fpu/slowpow.c: New file.
* sysdeps/ia64/fpu/branred.c: New file.
* sysdeps/ia64/fpu/doasin.c: New file.
* sysdeps/ia64/fpu/dosincos.c: New file.
* sysdeps/ia64/fpu/halfulp.c: New file.
* sysdeps/ia64/fpu/mpa.c: New file.
* sysdeps/ia64/fpu/mpatan.c: New file.
* sysdeps/ia64/fpu/mpatan2.c: New file.
* sysdeps/ia64/fpu/mpexp.c: New file.
* sysdeps/ia64/fpu/mplog.c: New file.
* sysdeps/ia64/fpu/mpsqrt.c: New file.
* sysdeps/ia64/fpu/mptan.c: New file.
* sysdeps/ia64/fpu/sincos32.c: New file.
* sysdeps/ia64/fpu/slowexp.c: New file.
* sysdeps/ia64/fpu/slowpow.c: New file.
* sysdeps/m68k/fpu/branred.c: New file.
* sysdeps/m68k/fpu/doasin.c: New file.
* sysdeps/m68k/fpu/dosincos.c: New file.
* sysdeps/m68k/fpu/halfulp.c: New file.
* sysdeps/m68k/fpu/mpa.c: New file.
* sysdeps/m68k/fpu/mpatan.c: New file.
* sysdeps/m68k/fpu/mpatan2.c: New file.
* sysdeps/m68k/fpu/mpexp.c: New file.
* sysdeps/m68k/fpu/mplog.c: New file.
* sysdeps/m68k/fpu/mpsqrt.c: New file.
* sysdeps/m68k/fpu/mptan.c: New file.
* sysdeps/m68k/fpu/sincos32.c: New file.
* sysdeps/m68k/fpu/slowexp.c: New file.
* sysdeps/m68k/fpu/slowpow.c: New file.
* iconvdata/gconv-modules: Add a number of alias, mostly for IBM
codepages.
2000-06-28 Jes Sorensen <jes@linuxcare.com>
* sysdeps/unix/sysv/linux/ia64/bits/elfclass.h: Deleted. Hash
table entries are 32 bit on the ia64 (as specified by the ABI),
not 64 bit as on the Alpha.
* shlib-versions: Bump ia64 soname to ld-linux-ia64.so.2 to mark
change of objects to use 32 bit hash table indices (see above).
2000-06-28 Jes Sorensen <jes@linuxcare.com>
* sysdeps/ieee754/dbl-64/e_exp.c: Add include stdlib.h to get
prototype for abs().
