The changes introduce a memory leak for gconv steps arrays whose
first element is an internal conversion, which has a fixed
reference count which is not decremented. As a result, after the
change in commit 50ce3eae5b, the steps
array is never freed, resulting in an unbounded memory leak.
This reverts commit 50ce3eae5b
("gconv: Check reference count in __gconv_release_cache
[BZ #24677]") and commit 7e740ab2e7
("libio: Fix gconv-related memory leak [BZ #24583]"). It
reintroduces bug 24583. (Bug 24677 was just a regression caused by
the second commit.)
Commit a601b74d31 aka glibc-2.23~693
("In preparation for fixing BZ#16734, fix failure in misc/tst-error1-mem
when _G_HAVE_MMAP is turned off.") introduced a regression:
_IO_unbuffer_all now invokes _IO_wsetb to free wide buffers of all
files, including legacy standard files which are small statically
allocated objects that do not have wide buffers and the _mode member,
causing memory corruption.
Another memory corruption in _IO_unbuffer_all happens when -1
is assigned to the _mode member of legacy standard files that
do not have it.
[BZ #24228]
* libio/genops.c (_IO_unbuffer_all)
[SHLIB_COMPAT (libc, GLIBC_2_0, GLIBC_2_1)]: Do not attempt to free wide
buffers and access _IO_FILE_complete members of legacy libio streams.
* libio/tst-bz24228.c: New file.
* libio/tst-bz24228.map: Likewise.
* libio/Makefile [build-shared] (tests): Add tst-bz24228.
[build-shared] (generated): Add tst-bz24228.mtrace and
tst-bz24228.check.
[run-built-tests && build-shared] (tests-special): Add
$(objpfx)tst-bz24228-mem.out.
(LDFLAGS-tst-bz24228, tst-bz24228-ENV): New variables.
($(objpfx)tst-bz24228-mem.out): New rule.
struct gconv_fcts for the C locale is statically allocated,
and __gconv_close_transform deallocates the steps object.
Therefore this commit introduces __wcsmbs_close_conv to avoid
freeing the statically allocated steps objects.
When computing the length of the converted part of the stdio buffer, use
the number of consumed wide characters, not the (negative) distance to the
end of the wide buffer.
GLIBC explicitly allows one to assign a new FILE pointer to stdout and
other standard streams. printf and wprintf were honouring assignment to
stdout and using the new value, but puts, putchar, and wide char variants
did not.
The stdout part is fixed here. The stdin part will be fixed in a followup.
According to ISO C99, passing the same buffer as source and destination
to sprintf, snprintf, vsprintf, or vsnprintf has undefined behavior.
Until the commit
commit 4e2f43f842
Author: Zack Weinberg <zackw@panix.com>
Date: Wed Mar 7 14:32:03 2018 -0500
Use PRINTF_FORTIFY instead of _IO_FLAGS2_FORTIFY (bug 11319)
a call to sprintf or vsprintf with overlapping buffers, for instance
vsprintf (buf, "%sTEXT", buf), would append `TEXT' into buf, while a
call to snprintf or vsnprintf would override the contents of buf.
After the aforementioned commit, the behavior of sprintf and vsprintf
changed (so that they also override the contents of buf).
This patch reverts this behavioral change, because it will likely break
applications that rely on the previous behavior, even though it is
undefined by ISO C. As noted by Szabolcs Nagy, this is used in SPEC2017
507.cactuBSSN_r/src/PUGH/PughUtils.c:
sprintf(mess," Size:");
for (i=0;i<dim+1;i++)
{
sprintf(mess,"%s %d",mess,pughGH->GFExtras[dim]->nsize[i]);
}
More important to notice is the fact that the overwriting of the
destination buffer is not the only behavior affected by the refactoring.
Before the refactoring, sprintf and vsprintf would use _IO_str_jumps,
whereas __sprintf_chk and __vsprintf_chk would use _IO_str_chk_jumps.
After the refactoring, all use _IO_str_chk_jumps, which would make
sprintf and vsprintf report buffer overflows and terminate the program.
This patch also reverts this behavior, by installing the appropriate
jump table for each *sprintf functions.
Apart from reverting the changes, this patch adds a test case that has
the old behavior hardcoded, so that regressions are noticed if something
else unintentionally changes the behavior.
Tested for powerpc64le.
As POSIX states [1] a freopen call should first flush the stream as if by a
call fflush. C99 (n1256) and C11 (n1570) only states the function should
first close any file associated with the specific stream. Although current
implementation only follow C specification, current BSD and other libc
implementation (musl) are in sync with POSIX and fflush the stream.
This patch change freopen{64} to fflush the stream before actually reopening
it (or returning if the stream does not support reopen). It also changes the
Linux implementation to avoid a dynamic allocation on 'fd_to_filename'.
Checked on x86_64-linux-gnu.
[BZ #21037]
* libio/Makefile (tests): Add tst-memstream4 and tst-wmemstream4.
* libio/freopen.c (freopen): Sync stream before reopen and adjust to
new fd_to_filename interface.
* libio/freopen64.c (freopen64): Likewise.
* libio/tst-memstream.h: New file.
* libio/tst-memstream4.c: Likewise.
* libio/tst-wmemstream4.c: Likewise.
* sysdeps/generic/fd_to_filename.h (fd_to_filename): Change signature.
* sysdeps/unix/sysv/linux/fd_to_filename.h (fd_to_filename): Likewise
and remove internal dynamic allocation.
[1] http://pubs.opengroup.org/onlinepubs/9699919799/
This is a variant of fgets which fails with ERANGE if the
buffer is too small, and the buffer length is given as an
argument of type size_t.
This function will be useful for implementing NSS file reading
operations. Compared to a direct implementation using the public API,
it avoids an lseek system call in case the line terminator can be
found in the internal read buffer.
C99 specifies that the EOF condition on a file is "sticky": once EOF
has been encountered, all subsequent reads should continue to return
EOF until the file is closed or something clears the "end-of-file
indicator" (e.g. fseek, clearerr). This is arguably a change from
C89, where the wording was ambiguous; the BSDs always had sticky EOF,
but the System V lineage would attempt to read from the underlying fd
again. GNU libc has followed System V for as long as we've been
using libio, but nowadays C99 conformance and BSD compatibility are
more important than System V compatibility.
You might wonder if changing the _underflow impls is sufficient to
apply the C99 semantics to all of the many stdio functions that
perform input. It should be enough to cover all paths to _IO_SYSREAD,
and the only other functions that call _IO_SYSREAD are the _seekoff
impls, which is OK because seeking clears EOF, and the _xsgetn impls,
which, as far as I can tell, are unused within glibc.
The test programs in this patch use a pseudoterminal to set up the
necessary conditions. To facilitate this I added a new test-support
function that sets up a pair of pty file descriptors for you; it's
almost the same as BSD openpty, the only differences are that it
allocates the optionally-returned tty pathname with malloc, and that
it crashes if anything goes wrong.
[BZ #1190]
[BZ #19476]
* libio/fileops.c (_IO_new_file_underflow): Return EOF immediately
if the _IO_EOF_SEEN bit is already set; update commentary.
* libio/oldfileops.c (_IO_old_file_underflow): Likewise.
* libio/wfileops.c (_IO_wfile_underflow): Likewise.
* support/support_openpty.c, support/tty.h: New files.
* support/Makefile (libsupport-routines): Add support_openpty.
* libio/tst-fgetc-after-eof.c, wcsmbs/test-fgetwc-after-eof.c:
New test cases.
* libio/Makefile (tests): Add tst-fgetc-after-eof.
* wcsmbs/Makefile (tests): Add tst-fgetwc-after-eof.
We shipped 2.27 with libio.h and _G_config.h still installed but
issuing warnings when used. Let's stop installing them early in 2.28
so that we have plenty of time to think of another plan if there are
problems.
The public stdio.h had a genuine dependency on libio.h for the
complete definitions of FILE and cookie_io_functions_t, and a genuine
dependency on _G_config.h for the complete definitions of fpos_t and
fpos64_t; these are moved to single-type headers.
bits/types/struct_FILE.h also provides a handful of accessor and
bitflags macros so that code is not duplicated between bits/stdio.h
and libio.h. All the other _IO_ and _G_ names used by the public
stdio.h can be replaced with either public names or __-names.
In order to minimize the risk of breaking our own compatibility code,
bits/types/struct_FILE.h preserves the _IO_USE_OLD_IO_FILE mechanism
exactly as it was in libio.h, but you have to define _LIBC to use it,
or it'll error out. Similarly, _IO_lock_t_defined is preserved
exactly, but will error out if used without defining _LIBC.
Internally, include/stdio.h continues to include libio.h, and libio.h
scrupulously provides every _IO_* and _G_* name that it always did,
perhaps now defined in terms of the public names. This is how this
patch avoids touching dozens of files throughout glibc and becoming
entangled with the _IO_MTSAFE_IO mess. The remaining patches in this
series eliminate most of the _G_ names.
Tested on x86_64-linux; in addition to the test suite, I installed the
library in a sysroot and verified that a simple program that uses
stdio.h could be compiled against the installed library, and I also
verified that installed stripped libraries are unchanged.
* libio/bits/types/__fpos_t.h, libio/bits/types/__fpos64_t.h:
New single-type headers split from _G_config.h.
* libio/bits/types/cookie_io_functions_t.h
* libio/bits/types/struct_FILE.h
New single-type headers split from libio.h.
* libio/Makefile: Install the above new headers. Don't install
libio.h, _G_config.h, bits/libio.h, bits/_G_config.h, or
bits/libio-ldbl.h.
* libio/_G_config.h, libio/libio.h: Delete file.
* libio/bits/libio.h: Remove improper-inclusion guard.
Include stdio.h and don't repeat anything that it does.
Define _IO_fpos_t as __fpos_t, _IO_fpos64_t as __fpos64_t,
_IO_BUFSIZ as BUFSIZ, _IO_va_list as __gnuc_va_list,
__io_read_fn as cookie_read_function_t,
__io_write_fn as cookie_write_function_t,
__io_seek_fn as cookie_seek_function_t,
__io_close_fn as cookie_close_function_t,
and _IO_cookie_io_functions_t as cookie_io_functions_t.
Define _STDIO_USES_IOSTREAM, __HAVE_COLUMN, and _IO_file_flags
here, in the "compatibility defines" section. Remove an #if 0
block. Use the "body" macros from bits/types/struct_FILE.h to
define _IO_getc_unlocked, _IO_putc_unlocked, _IO_feof_unlocked,
and _IO_ferror_unlocked.
Move prototypes of __uflow and __overflow...
* libio/stdio.h: ...here. Don't include bits/libio.h.
Don't define _STDIO_USES_IOSTREAM. Get __gnuc_va_list
directly from stdarg.h. Include bits/types/__fpos_t.h,
bits/types/__fpos64_t.h, bits/types/struct_FILE.h,
and, when __USE_GNU, bits/types/cookie_io_functions_t.h.
Use __gnuc_va_list, not _G_va_list; __fpos_t, not _G_fpos_t;
__fpos64_t, not _G_fpos64_t; FILE, not struct _IO_FILE;
cookie_io_functions_t, not _IO_cookie_io_functions_t;
__ssize_t, not _IO_ssize_t. Unconditionally define
BUFSIZ as 8192 and EOF as (-1).
* libio/bits/stdio.h: Add multiple-include guard. Use the "body"
macros from bits/types/struct_FILE.h instead of _IO_* macros
from libio.h; use __gnuc_va_list instead of va_list and __ssize_t
instead of _IO_ssize_t.
* libio/bits/stdio2.h: Similarly.
* libio/iolibio.h: Add multiple-include guard.
Include bits/libio.h after stdio.h.
* libio/libioP.h: Add multiple-include guard.
Include stdio.h and bits/libio.h before iolibio.h.
* include/bits/types/__fpos_t.h, include/bits/types/__fpos64_t.h
* include/bits/types/cookie_io_functions_t.h
* include/bits/types/struct_FILE.h: New wrappers.
* bits/_G_config.h, sysdeps/unix/sysv/linux/_G_config.h:
Get definitions of _G_fpos_t and _G_fpos64_t from
bits/types/__fpos_t.h and bits/types/__fpos64_t.h
respectively. Remove improper-inclusion guards.
* conform/data/stdio.h-data: Update expectations of va_list.
* scripts/check-installed-headers.sh: Remove special case for
libio.h and _G_config.h.
libio.h was originally the header for a set of supported GNU
extensions, but they have not been maintained as such in many years,
they are now standing in the way of improvements to stdio, and we
don't think there are any remaining external users. _G_config.h was
never intended for public use, but predates the bits convention.
Move both of these headers into the bits directory and provide stubs
at top level which issue deprecation warnings.
The contents of (bits/)libio.h and (bits/)_G_config.h are still
exposed to external software via stdio.h; changing that requires more
complex surgery than I have time to attempt right now.
* libio/libio.h, libio/_G_config.h: New stub headers which issue a
deprecation warning and then include <bits/libio.h>, <bits/_G_config.h>
respectively.
* libio/libio.h: Rename the original version of this file to
libio/bits/libio.h. Error out if not included by stdio.h or the
stub libio.h.
* include/libio.h: Move to include/bits. Forward to libio/bits/libio.h.
* sysdeps/generic/_G_config.h: Move to top-level bits/. Error out
if not included by bits/libio.h or the stub _G_config.h.
* sysdeps/unix/sysv/linux/_G_config.h: Move to
sysdeps/unix/sysv/linux/bits. Error out if not included by
bits/libio.h or the stub _G_config.h.
* libio/stdio.h: Include bits/libio.h, not libio.h.
* libio/Makefile: Install bits/libio.h and bits/_G_config.h as
well as libio.h and _G_config.h.
* csu/init.c, libio/fmemopen.c, libio/iolibio.h, libio/oldfmemopen.c
* libio/strfile.h, stdio-common/vfscanf.c
* sysdeps/pthread/flockfile.c, sysdeps/pthread/funlockfile.c
Include stdio.h, not _G_config.h nor libio.h.
* libio/iofgetpos.c: Also rename fgetpos64 out of the way.
* libio/iofsetpos.c: Also rename fsetpos64 out of the way.
* scripts/check-installed-headers.sh: Skip libio.h and _G_config.h.
Some libio operations fail to correctly free the backup area (created
by _IO_{w}default_pbackfail on unget{w}c) resulting in either invalid
buffer free operations or memory leaks.
For instance, on the example provided by BZ#22415 a following
fputc after a fseek to rewind the stream issues an invalid free on
the buffer. It is because although _IO_file_overflow correctly
(from fputc) correctly calls _IO_free_backup_area, the
_IO_new_file_seekoff (called by fseek) updates the FILE internal
pointers without first free the backup area (resulting in invalid
values in the internal pointers).
The wide version also shows an issue, but instead of accessing invalid
pointers it leaks the backup memory on fseek/fputwc operation.
Checked on x86_64-linux-gnu and i686-linux-gnu.
* libio/Makefile (tests): Add tst-bz22415.
(tst-bz22415-ENV): New rule.
(generated): Add tst-bz22415.mtrace and tst-bz22415.check.
(tests-special): Add tst-bz22415-mem.out.
($(objpfx)tst-bz22415-mem.out): New rule.
* libio/fileops.c (_IO_new_file_seekoff): Call _IO_free_backup_area
in case of a successful seek operation.
* libio/wfileops.c (_IO_wfile_seekoff): Likewise.
(_IO_wfile_overflow): Call _IO_free_wbackup_area in case a write
buffer is required.
* libio/tst-bz22415.c: New test.
wint_t is a little finicky because it might be defined by stddef.h, which
belongs to the compiler.
In addition to the _types_, a bunch of other declarations shared between
wctype.h and wchar.h are factored out to their own header.
* libio/bits/types/FILE.h, libio/bits/types/__FILE.h
* wcsmbs/bits/types/mbstate_t.h, wcsmbs/bits/types/__mbstate_t.h
* wcsmbs/bits/types/wint_t.h: New single-type definition files.
* wctype/bits/wctype-wchar.h: New file holding declarations shared
between wctype.h and wchar.h.
* libio/Makefile, wcsmbs/Makefile, wctype/Makefile:
Install them.
* include/bits/types/FILE.h, include/bits/types/__FILE.h
* include/bits/types/mbstate_t.h, include/bits/types/__mbstate_t.h
* include/bits/types/wint_t.h, include/bits/wcsmbs-wchar.h:
New wrappers.
* include/stdio.h, include/wchar.h, include/wctype.h:
No need to handle __need macros.
* grp/grp.h, gshadow/gshadow.h, hurd/hurd.h, iconv/gconv.h
* libio/stdio.h, mach/mach.h, misc/mntent.h, pwd/pwd.h
* shadow/shadow.h, stdio-common/printf.h, wcsmbs/uchar.h
* wcsmbs/wchar.h, wctype/wctype.h
* sysdeps/generic/_G_config.h, sysdeps/unix/sysv/linux/_G_config.h
Use the new files instead of __need macros.
This patches fixes multiples issues on open_{w}memstream reported on both
BZ#18241 and BZ#20181:
- failed fseek does not set errno.
- negative offset in fseek fails even when resulting position is
a valid one.
- a flush after write if the current write position is not at the
end of the stream currupt data.
The main fix is on seek operation for memstream (_IO_{w}str_seekoff), where
both _IO_read_ptr and _IO_read_end pointer are updated if a write operation
has occured (similar to default file operations). Also, to calculate the
offset on both read and write pointers, a temporary value is instead of
updating the argument supplied value. Negative offset are valid if resulting
internal pointer is within the range of _IO_{read,write}_base and
_IO_{read,write}_end.
Also POSIX states that a null or wide null shall be appended to the current
buffer iff a write moves the position to a value larger than the current
lenght. Current implementation appends a null or wide null regardless
of this condition. This patch fixes it by removing the 'else' condition
on _IO_{w}mem_sync.
Checked on x86_64.
[BZ #18241]
[BZ #20181]
* libio/Makefile (test): Add tst-memstream3 and tst-wmemstream3.
* libio/memstream.c (_IO_mem_sync): Only append a null byte if
write position is at the end the buffer.
* libio/wmemstream.c (_IO_wmem_sync): Likewise.
* libio/strops.c (_IO_str_switch_to_get_mode): New function.
(_IO_str_seekoff): Set correct offset from negative displacement and
set EINVAL for invalid ones.
* libio/wstrops.c (enlarge_userbuf): Use correct function to calculate
buffer length.
(_IO_wstr_switch_to_get_mode): New function.
(_IO_wstr_seekoff): Set correct offset from negative displacement and
set EINVAL for invalid ones.
* libio/tst-memstream3.c: New file.
* libio/tst-wmemstream3.c: Likewise.
* manual/examples/memstrm.c: Remove warning when priting size_t.
This commit puts all libio vtables in a dedicated, read-only ELF
section, so that they are consecutive in memory. Before any indirect
jump, the vtable pointer is checked against the section boundaries,
and the process is terminated if the vtable pointer does not fall into
the special ELF section.
To enable backwards compatibility, a special flag variable
(_IO_accept_foreign_vtables), protected by the pointer guard, avoids
process termination if libio stream object constructor functions have
been called earlier. Such constructor functions are called by the GCC
2.95 libstdc++ library, and this mechanism ensures compatibility with
old binaries. Existing callers inside glibc of these functions are
adjusted to call the original functions, not the wrappers which enable
vtable compatiblity.
The compatibility mechanism is used to enable passing FILE * objects
across a static dlopen boundary, too.
The bits/libc-lock.h and bits/stdio-lock.h headers are installed, but
not used by any other installed header.
Accordingly, this patch stops these headers from being installed. A
followup is intended to move these headers out of the bits/ namespace.
Tested for x86_64 (testsuite, and that installed stripped shared
libraries are unchanged by the patch).
* Makefile (headers): Remove bits/libc-lock.h.
* libio/Makefile (headers): Remove bits/stdio-lock.h.
This patch added a new fmemopen version, for glibc 2.22, that aims to be
POSIX complaint. It fixes some long-stading glibc fmemopen issues, such
as:
* it changes the way fseek with SEEK_END works on fmemopen to seek
relative to buffer size instead of first '\0'. This is default mode and
'b' opening mode does not change internal behavior (bz#6544).
* fix apending opening mode to use as start position either first null
byte of len specified in function call (bz#13152 and #13151).
* remove binary option 'b' and internal different handling (bz#12836)
* fix seek/SEE_END with negative values (bz#14292).
A compatibility symbol is provided to with old behavior for older symbols
version (2.2.5).
* include/stdio.h (fmemopen): Remove hidden prototype.
(__fmemopen): Add new hidden prototype.
* libio/Makefile: Add oldfmemopen object.
* libio/Versions [GLIBC_2.22]: Add new fmemopen symbol.
* libio/fmemopen.c (__fmemopen): Function rewrite to be POSIX
compliance.
* libio/oldfmemopen.c: New file: old fmemopen implementation for
symbol compatibility.
* stdio-common/Makefile [tests]: Add new tst-fmemopen3.
* stdio-common/psiginfo.c [psiginfo]: Call __fmemopen instead of
fmemopen.
* stdio-common/tst-fmemopen3.c: New file: more fmemopen tests, focus
on append and read mode.
* sysdeps/unix/sysv/linux/aarch64/libc.abilist [GLIBC_2.22]: Add
fmemopen.
* sysdeps/unix/sysv/linux/alpha/libc.abilist [GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/arm/libc.abilist [GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/i386/libc.abilist [GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/ia64/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/microblaze/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/sh/libc.abilist [GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/tile/tilegx/tilegx32/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/tile/tilegx/tilegx64/libc.abilist
[GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/tile/tilepro/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/64/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist [GLIBC_2.22]:
Likewise.
* sysdeps/unix/sysv/linux/hppa/libc.abilist [GLIBC_2.22]: Likewise.
* sysdeps/unix/sysv/linux/nios2/libc.abilist [GLIBC_2.22]: Likewise.
Various glibc build / install / test code has C locale settings that
are redundant with LC_ALL=C.
LC_ALL takes precedence over LANG, so anywhere that sets LC_ALL=C
(explicitly, or through it being in the default environment for
running tests) does not need to set LANG=C. LC_ALL=C also takes
precedence over LANGUAGE, since
2001-01-02 Ulrich Drepper <drepper@redhat.com>
* intl/dcigettext.c (guess_category_value): Rewrite so that LANGUAGE
value is ignored if the selected locale is the C locale.
* intl/tst-gettext.c: Set locale for above change.
* intl/tst-translit.c: Likewise.
and so settings of LANGUAGE=C are also redundant when LC_ALL=C is
set. One test also had LC_ALL=C in its -ENV setting, although it's
part of the default environment used for tests.
This patch removes the redundant settings. It removes a suggestion in
install.texi of setting LANGUAGE=C LC_ALL=C for "make install"; the
Makefile.in target "install" already sets LC_ALL_C so there's no need
for the user to set it (and nor should there be any need for the user
to set it).
If some build machine tool used by "make install" uses a version of
libintl predating that 2001 change, and the user has LANGUAGE set, the
removal of LANGUAGE=C from the Makefile.in "install" rule could in
principle affect the user's installation. However, I don't think we
need to be concerned about pre-2001 build tools.
Tested x86_64.
* Makefile (install): Don't set LANGUAGE.
* Makefile.in (install): Likewise.
* assert/Makefile (test-assert-ENV): Remove variable.
(test-assert-perr-ENV): Likewise.
* elf/Makefile (neededtest4-ENV): Likewise.
* iconvdata/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Don't set LANGUAGE.
* io/ftwtest-sh (LANG): Remove variable.
* libio/Makefile (tst-widetext-ENV): Likewise.
* manual/install.texi (Running make install): Don't refer to
environment settings for make install.
* INSTALL: Regenerated.
* nptl/tst-tls6.sh: Don't set LANG.
* posix/globtest.sh (LANG): Remove variable.
* string/Makefile (tester-ENV): Likewise.
(inl-tester-ENV): Likewise.
(noinl-tester-ENV): Likewise.
* sysdeps/s390/s390-64/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Don't set LANGUAGE.
* timezone/Makefile (build-testdata): Use $(built-program-cmd)
without explicit environment settings.
localedata/ChangeLog:
* tst-fmon.sh: Don't set LANGUAGE.
* tst-locale.sh: Likewise.
Tests run using the default $(make-test-out) automatically get
GCONV_PATH and LC_ALL set, whether or not those environment variables
are actually needed for the individual test. However, they do not get
LOCPATH set, meaning that a large number of tests have -ENV settings
just to set LOCPATH.
This patch moves LOCPATH into the default environment used for all
tests, on the principle that like GCONV_PATH any settings needed to
use files associated with the newly built library, rather than any old
installed files, are appropriate to use by default.
A further motivation is that various tests using .sh files also set
some combination of LC_ALL, GCONV_PATH and LOCPATH. Preferably .sh
files should also use the default environment with any additions
required for the individual test. Now, it was suggested in
<https://sourceware.org/ml/libc-alpha/2014-05/msg00715.html> that
various Makefile variables used in testing should be derived by
composing the -before-env and -after-env variables used when explicit
environment settings are required. With such a change, it's also
natural for those variables to include the default settings (via some
intermediate makefile variable also used in make-test-out).
Because some .sh files only set variables that correspond to the
default settings, or a subset thereof, and this applies to more of the
.sh files once LOCPATH is in the default settings, doing so reduces
the size of a revised version of
<https://sourceware.org/ml/libc-alpha/2014-05/msg00596.html>: scripts
only needing the (expanded) default settings will not need to receive
the separate -before-env and -after-env variables, only the single
variable they do at present. So moving LOCPATH into the default
settings can reduce churn caused by subsequent patches.
Tested x86_64 and x86.
* Rules (make-test-out): Include
LOCPATH=$(common-objpfx)localedata in default environment.
* debug/Makefile (tst-chk1-ENV): Remove variable.
(tst-chk2-ENV): Likewise.
(tst-chk3-ENV): Likewise.
(tst-chk4-ENV): Likewise.
(tst-chk5-ENV): Likewise.
(tst-chk6-ENV): Likewise.
(tst-lfschk1-ENV): Likewise.
(tst-lfschk2-ENV): Likewise.
(tst-lfschk3-ENV): Likewise.
(tst-lfschk4-ENV): Likewise.
(tst-lfschk5-ENV): Likewise.
(tst-lfschk6-ENV): Likewise.
* iconvdata/Makefile (bug-iconv6-ENV): Likewise.
(tst-iconv7-ENV): Likewise.
* intl/Makefile (LOCPATH-ENV): Likewise.
(tst-codeset-ENV): Likewise.
(tst-gettext3-ENV): Likewise.
(tst-gettext5-ENV): Likewise.
* libio/Makefile (tst-widetext-ENV): Don't set LOCPATH.
(tst-fopenloc-ENV): Likewise.
(tst-fgetws-ENV): Remove variable.
(tst-ungetwc1-ENV): Likewise.
(tst-ungetwc2-ENV): Likewise.
(bug-ungetwc2-ENV): Likewise.
(tst-swscanf-ENV): Likewise.
(bug-ftell-ENV): Likewise.
(tst-fgetwc-ENV): Likewise.
(tst-fseek-ENV): Likewise.
(tst-ftell-partial-wide-ENV): Likewise.
(tst-ftell-active-handler-ENV): Likewise.
(tst-ftell-append-ENV): Likewise.
* posix/Makefile (tst-fnmatch-ENV): Likewise.
(tst-regexloc-ENV): Likewise.
(bug-regex1-ENV): Likewise.
(tst-regex-ENV): Likewise.
(tst-regex2-ENV): Likewise.
(bug-regex5-ENV): Likewise.
(bug-regex6-ENV): Likewise.
(bug-regex17-ENV): Likewise.
(bug-regex18-ENV): Likewise.
(bug-regex19-ENV): Likewise.
(bug-regex20-ENV): Likewise.
(bug-regex22-ENV): Likewise.
(bug-regex23-ENV): Likewise.
(bug-regex25-ENV): Likewise.
(bug-regex26-ENV): Likewise.
(bug-regex30-ENV): Likewise.
(bug-regex32-ENV): Likewise.
(bug-regex33-ENV): Likewise.
(bug-regex34-ENV): Likewise.
(bug-regex35-ENV): Likewise.
(tst-rxspencer-ENV): Likewise.
(tst-rxspencer-no-utf8-ENV): Likewise.
* stdio-common/Makefile (tst-sprintf-ENV): Likewise.
(tst-sscanf-ENV): Likewise.
(tst-swprintf-ENV): Likewise.
(tst-swscanf-ENV): Likewise.
(test-vfprintf-ENV): Likewise.
(scanf13-ENV): Likewise.
(bug14-ENV): Likewise.
(tst-grouping-ENV): Likewise.
* stdlib/Makefile (tst-strtod-ENV): Likewise.
(tst-strtod3-ENV): Likewise.
(tst-strtod4-ENV): Likewise.
(tst-strtod5-ENV): Likewise.
(testmb2-ENV): Likewise./
* string/Makefile (tst-strxfrm-ENV): Likewise.
(tst-strxfrm2-ENV): Likewise.
(bug-strcoll1-ENV): Likewise.
(test-strcasecmp-ENV): Likewise.
(test-strncasecmp-ENV): Likewise.
* time/Makefile (tst-strptime-ENV): Likewise.
(tst-ftime_l-ENV): Likewise.
* wcsmbs/Makefile (tst-btowc-ENV): Likewise.
(tst-mbrtowc-ENV): Likewise.
(tst-wcrtomb-ENV): Likewise.
(tst-mbrtowc2-ENV): Likewise.
(tst-c16c32-1-ENV): Likewise.
(tst-mbsnrtowcs-ENV): Likewise.
localedata/ChangeLog:
* Makefile (TEST_MBWC_ENV): Remove variable.
(tst_iswalnum-ENV): Likewise.
(tst_iswalpha-ENV): Likewise.
(tst_iswcntrl-ENV): Likewise.
(tst_iswctype-ENV): Likewise.
(tst_iswdigit-ENV): Likewise.
(tst_iswgraph-ENV): Likewise.
(tst_iswlower-ENV): Likewise.
(tst_iswprint-ENV): Likewise.
(tst_iswpunct-ENV): Likewise.
(tst_iswspace-ENV): Likewise.
(tst_iswupper-ENV): Likewise.
(tst_iswxdigit-ENV): Likewise.
(tst_mblen-ENV): Likewise.
(tst_mbrlen-ENV): Likewise.
(tst_mbrtowc-ENV): Likewise.
(tst_mbsrtowcs-ENV): Likewise.
(tst_mbstowcs-ENV): Likewise.
(tst_mbtowc-ENV): Likewise.
(tst_strcoll-ENV): Likewise.
(tst_strfmon-ENV): Likewise.
(tst_strxfrm-ENV): Likewise.
(tst_swscanf-ENV): Likewise.
(tst_towctrans-ENV): Likewise.
(tst_towlower-ENV): Likewise.
(tst_towupper-ENV): Likewise.
(tst_wcrtomb-ENV): Likewise.
(tst_wcscat-ENV): Likewise.
(tst_wcschr-ENV): Likewise.
(tst_wcscmp-ENV): Likewise.
(tst_wcscoll-ENV): Likewise.
(tst_wcscpy-ENV): Likewise.
(tst_wcscspn-ENV): Likewise.
(tst_wcslen-ENV): Likewise.
(tst_wcsncat-ENV): Likewise.
(tst_wcsncmp-ENV): Likewise.
(tst_wcsncpy-ENV): Likewise.
(tst_wcspbrk-ENV): Likewise.
(tst_wcsrtombs-ENV): Likewise.
(tst_wcsspn-ENV): Likewise.
(tst_wcsstr-ENV): Likewise.
(tst_wcstod-ENV): Likewise.
(tst_wcstok-ENV): Likewise.
(tst_wcstombs-ENV): Likewise.
(tst_wcswidth-ENV): Likewise.
(tst_wcsxfrm-ENV): Likewise.
(tst_wctob-ENV): Likewise.
(tst_wctomb-ENV): Likewise.
(tst_wctrans-ENV): Likewise.
(tst_wctype-ENV): Likewise.
(tst_wcwidth-ENV): Likewise.
(tst-digits-ENV): Likewise.
(tst-mbswcs6-ENV): Likewise.
(tst-xlocale1-ENV): Likewise.
(tst-xlocale2-ENV): Likewise.
(tst-strfmon1-ENV): Likewise.
(tst-strptime-ENV): Likewise.
(tst-setlocale-ENV): Don't set LOCPATH.
(bug-iconv-trans-ENV): Remove variable.
(tst-sscanf-ENV): Likewise.
(tst-leaks-ENV): Don't set LOCPATH.
(bug-setlocale1-ENV): Remove variable.
(bug-setlocale1-static-ENV): Likewise.
(tst-setlocale2-ENV): Likewise.
The offset computation in write mode uses the fact that _IO_read_end
is kept in sync with the external file offset. This however is not
true when O_APPEND is in effect since switching to write mode ought to
send the external file offset to the end of file without making the
necessary adjustment to _IO_read_end.
Hence in append mode, offset computation when writing should only
consider the effect of unflushed writes, i.e. from _IO_write_base to
_IO_write_ptr.
The wiki has a detailed document that describes the rationale for
offsets returned by ftell in various conditions:
https://sourceware.org/glibc/wiki/File%20offsets%20in%20a%20stdio%20stream%20and%20ftell
This patch is a revised and updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00196.html>.
In order to generate overall summaries of the results of all tests in
the glibc testsuite, we need to identify and concatenate the files
with the results of individual tests.
Tomas Dohnalek's patch used $(common-objpfx)*/*.test-result for this.
However, the normal glibc approach is explicit enumeration of the
expected set of files with a given property, rather than all files
matching some pattern like that. Furthermore, we would like to be
able to mark tests as UNRESOLVED if the file with their results is for
some reason missing, and in future we would like to be able to mark
tests as UNSUPPORTED if they are disabled for a particular
configuration (rather than simply having them missing from the list of
tests as at present). Such handling of tests that were not run or did
not record results requires an explicit enumeration of tests.
For the tests following the default makefile rules, $(tests) (and
$(xtests)) provides such an enumeration. Others, however, are added
directly as dependencies of the "tests" and "xtests" makefile
targets. This patch changes the makefiles to put them in variables
tests-special and xtests-special, with appropriate dependencies on the
tests listed there then being added centrally.
Those variables are used in Rules and so need to be set before Rules
is included in a subdirectory makefile, which is often earlier in the
makefile than the dependencies were present before. We previously
discussed the question of where to include Rules; see the question at
<https://sourceware.org/ml/libc-alpha/2012-11/msg00798.html>, and a
discussion in
<https://sourceware.org/ml/libc-alpha/2013-01/msg00337.html> of why
Rules is included early rather than late in subdirectory makefiles.
It was necessary to avoid an indirection through the check-abi target
and get the check-abi-* targets for individual libraries into the
tests-special variable. The intl/ test $(objpfx)tst-gettext.out,
previously built only because of dependencies from other tests, was
also added to tests-special for the same reason.
The entries in tests-special are the full makefile targets, complete
with $(objpfx) and .out. If a future change causes tests to be named
consistently with a .out suffix, this can be changed to include just
the path relative to $(objpfx), without .out.
Tested x86_64, including that the same set of files is generated in
the build directory by a build and testsuite run both before and after
the patch (except for changes to the
elf/tst-null-argv.debug.out.<number> file name), and a build with
run-built-tests=no to verify there aren't any more obvious instances
of the issue Marcus Shawcroft reported with a previous version in
<https://sourceware.org/ml/libc-alpha/2014-01/msg00462.html>.
* Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(tests): Depend on $(tests-special).
* Makerules (check-abi-list): New variable.
(check-abi): Depend on $(check-abi-list).
[$(subdir) = elf] (tests-special): Add
$(objpfx)check-abi-libc.out.
[$(build-shared) = yes && subdir] (tests-special): Add
$(check-abi-list).
[$(build-shared) = yes && subdir] (tests): Do not depend on
check-abi.
* Rules (tests): Depend on $(tests-special).
(xtests): Depend on $(xtests-special).
* catgets/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* conform/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* elf/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* grp/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* iconv/Makefile (xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* iconvdata/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* intl/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable. Also add
$(objpfx)tst-gettext.out.
* io/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* libio/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* malloc/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* misc/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* nptl/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* nptl_db/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* posix/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* resolv/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* stdio-common/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(do-tst-unbputc): Remove target.
(do-tst-printf): Likewise.
* stdlib/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* string/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* sysdeps/x86/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
localedata:
* Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
ftell semantics are distinct from fseek(SEEK_CUR) especially when it
is called on a file handler that is not yet active. Due to this
caveat, much care needs to be taken while modifying the handler data
and hence, this first iteration on separating out ftell focusses on
maintaining handler data integrity at all times while it figures out
the current stream offset. The result is that it makes a syscall for
every offset request.
There is scope for optimizing this by caching offsets when we know
that the handler is active. A simple way to find out is when the
buffers have data. It is not so simple to find this out when the
buffer is empty without adding some kind of flag.
In <https://sourceware.org/ml/libc-alpha/2014-01/msg00196.html> I
noted it was necessary to add includes of Makeconfig early in various
subdirectory makefiles for the tests-special variable settings added
by that patch to be conditional on configuration information. No-one
commented on the general question there of whether Makeconfig should
always be included immediately after the definition of subdir.
This patch implements that early inclusion of Makeconfig in each
directory (which is a lot easier than consistent placement of includes
of Rules). Includes are added if needed, or moved up if already
present. Subdirectory "all:" targets are removed, since Makeconfig
provides one.
There is potential for further cleanups I haven't done. Rules and
Makerules have code such as
ifneq "$(findstring env,$(origin headers))" ""
headers :=
endif
to override to empty any value of various variables that came from the
environment. I think there is a case for Makeconfig setting all the
subdirectory variables (other than subdir) to empty to ensure no
outside value is going to take effect if a subdirectory fails to
define a variable. (A list of such variables, possibly out of date
and incomplete, is in manual/maint.texi.) Rules and Makerules would
give errors if Makeconfig hadn't already been included, instead of
including it themselves. The special code to override values coming
from the environment would then be obsolete and could be removed.
Tested x86_64, including that installed binaries are identical before
and after the patch.
* argp/Makefile: Include Makeconfig immediately after defining
subdir.
* assert/Makefile: Likewise.
* benchtests/Makefile: Likewise.
* catgets/Makefile: Likewise.
* conform/Makefile: Likewise.
* crypt/Makefile: Likewise.
* csu/Makefile: Likewise.
(all): Remove target.
* ctype/Makefile: Include Makeconfig immediately after defining
subdir.
* debug/Makefile: Likewise.
* dirent/Makefile: Likewise.
* dlfcn/Makefile: Likewise.
* gmon/Makefile: Likewise.
* gnulib/Makefile: Likewise.
* grp/Makefile: Likewise.
* gshadow/Makefile: Likewise.
* hesiod/Makefile: Likewise.
* hurd/Makefile: Likewise.
(all): Remove target.
* iconvdata/Makefile: Include Makeconfig immediately after
defining subdir.
* inet/Makefile: Likewise.
* intl/Makefile: Likewise.
* io/Makefile: Likewise.
* libio/Makefile: Likewise.
(all): Remove target.
* locale/Makefile: Include Makeconfig immediately after defining
subdir.
* login/Makefile: Likewise.
* mach/Makefile: Likewise.
(all): Remove target.
* malloc/Makefile: Include Makeconfig immediately after defining
subdir.
(all): Remove target.
* manual/Makefile: Include Makeconfig immediately after defining
subdir.
* math/Makefile: Likewise.
* misc/Makefile: Likewise.
* nis/Makefile: Likewise.
* nss/Makefile: Likewise.
* po/Makefile: Likewise.
(all): Remove target.
* posix/Makefile: Include Makeconfig immediately after defining
subdir.
* pwd/Makefile: Likewise.
* resolv/Makefile: Likewise.
* resource/Makefile: Likewise.
* rt/Makefile: Likewise.
* setjmp/Makefile: Likewise.
* shadow/Makefile: Likewise.
* signal/Makefile: Likewise.
* socket/Makefile: Likewise.
* soft-fp/Makefile: Likewise.
* stdio-common/Makefile: Likewise.
* stdlib/Makefile: Likewise.
* streams/Makefile: Likewise.
* string/Makefile: Likewise.
* sunrpc/Makefile: Likewise.
(all): Remove target.
* sysvipc/Makefile: Include Makeconfig immediately after defining
subdir.
* termios/Makefile: Likewise.
* time/Makefile: Likewise.
* timezone/Makefile: Likewise.
(all): Remove target.
* wcsmbs/Makefile: Include Makeconfig immediately after defining
subdir.
* wctype/Makefile: Likewise.
libidn/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
localedata/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
(all): Remove target.
nptl/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
nptl_db/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
Most glibc tests that use mtrace to verify that there were no memory
leaks from the glibc facilities used in a given test depend on the
.out file of the previous test so that the mtrace test runs mtrace and
nothing else.
Two, however, have a single target combining mtrace with something
else. In the case of libio/tst-fopenloc.check, the test both compares
the output with an expected baseline and runs mtrace. In the case of
posix/tst-rxspencer-mem, the test is run (with different command line
from the main run) and then mtrace is run, from the same makefile
target.
This patch splits both of these tests up to use separate makefile
targets for each thing tested; in the tst-rxspencer case, a file
tst-rxspencer-no-utf8.c is created that just includes tst-rxspencer.c,
as is usual for tests where the same code gets tested in different
compile-time or runtime configurations.
Adding $(evaluate-test) to test commands, as in
<https://sourceware.org/ml/libc-alpha/2014-01/msg00194.html>, will no
longer need to insert && between multiple commands, as all tests will
either have just a single command or already use &&.
Tested x86_64.
* libio/Makefile ($(objpfx)tst-fopenloc.check): Split into
separate $(objpfx)tst-fopenloc-cmp.out and
$(objpfx)tst-fopenloc-mem.out targets.
(tests): Update dependencies.
* posix/Makefile (tests variable): Add tst-rxspencer-no-utf8.
(generated): Change tst-rxspencer-mem and tst-rxspencer.mtrace to
tst-rxspencer-no-utf8-mem and tst-rxspencer-no-utf8.mtrace.
(tst-rxspencer-no-utf8-ARGS): New variable.
(tst-rxspencer-no-utf8-ENV): Likewise.
(tests target): Depend on $(objpfx)tst-rxspencer-no-utf8-mem
instead of $(objpfx)tst-rxspencer-mem.
($(objpfx)tst-rxspencer-mem): Change target to
$(objpfx)tst-rxspencer-no-utf8-mem. Depend on
$(objpfx)tst-rxspencer-no-utf8.out instead of running test program.
* posix/tst-rxspencer-no-utf8.c: New file.
ftell tries to avoid flushing the buffer when it is in write mode by
converting the wide char data and placing it into the binary buffer.
If the output buffer space is full and there is data to write, the
code reverts to flushing the buffer. This breaks when there is space
in the buffer but it is not enough to convert the next character in
the wide data buffer, due to which __codecvt_do_out returns a
__codecvt_partial status. In this case, ftell keeps running in an
infinite loop.
The fix here is to detect the __codecvt_partial status in addition to
checking if the buffer is full. I have also added a test case that
demonstrates the infinite loop.
In BZ #12724, partial support for POSIX 2008 fclose behavior was added.
Since it isn't entirely conforming to the spec, some applications are
known to be breaking in this intermediate state. So revert the partial
support until we can get things fully implemented.
This reverts commit fcabc0f8b1.
Signed-off-by: Mike Frysinger <vapier@gentoo.org>
