The nan, nanf and nanl functions handle payload strings by doing e.g.:
if (tagp[0] != '\0')
{
char buf[6 + strlen (tagp)];
sprintf (buf, "NAN(%s)", tagp);
return strtod (buf, NULL);
}
This is an unbounded stack allocation based on the length of the
argument. Furthermore, if the argument starts with an n-char-sequence
followed by ')', that n-char-sequence is wrongly treated as
significant for determining the payload of the resulting NaN, when ISO
C says the call should be equivalent to strtod ("NAN", NULL), without
being affected by that initial n-char-sequence. This patch fixes both
those problems by using the __strtod_nan etc. functions recently
factored out of strtod etc. for that purpose, with those functions
being exported from libc at version GLIBC_PRIVATE.
Tested for x86_64, x86, mips64 and powerpc.
[BZ #16961]
[BZ #16962]
* math/s_nan.c (__nan): Use __strtod_nan instead of constructing a
string on the stack for strtod.
* math/s_nanf.c (__nanf): Use __strtof_nan instead of constructing
a string on the stack for strtof.
* math/s_nanl.c (__nanl): Use __strtold_nan instead of
constructing a string on the stack for strtold.
* stdlib/Versions (libc): Add __strtof_nan, __strtod_nan and
__strtold_nan to GLIBC_PRIVATE.
* math/test-nan-overflow.c: New file.
* math/test-nan-payload.c: Likewise.
* math/Makefile (tests): Add test-nan-overflow and
test-nan-payload.
