Some libio operations fail to correctly free the backup area (created
by _IO_{w}default_pbackfail on unget{w}c) resulting in either invalid
buffer free operations or memory leaks.
For instance, on the example provided by BZ#22415 a following
fputc after a fseek to rewind the stream issues an invalid free on
the buffer. It is because although _IO_file_overflow correctly
(from fputc) correctly calls _IO_free_backup_area, the
_IO_new_file_seekoff (called by fseek) updates the FILE internal
pointers without first free the backup area (resulting in invalid
values in the internal pointers).
The wide version also shows an issue, but instead of accessing invalid
pointers it leaks the backup memory on fseek/fputwc operation.
Checked on x86_64-linux-gnu and i686-linux-gnu.
* libio/Makefile (tests): Add tst-bz22415.
(tst-bz22415-ENV): New rule.
(generated): Add tst-bz22415.mtrace and tst-bz22415.check.
(tests-special): Add tst-bz22415-mem.out.
($(objpfx)tst-bz22415-mem.out): New rule.
* libio/fileops.c (_IO_new_file_seekoff): Call _IO_free_backup_area
in case of a successful seek operation.
* libio/wfileops.c (_IO_wfile_seekoff): Likewise.
(_IO_wfile_overflow): Call _IO_free_wbackup_area in case a write
buffer is required.
* libio/tst-bz22415.c: New test.
This commit puts all libio vtables in a dedicated, read-only ELF
section, so that they are consecutive in memory. Before any indirect
jump, the vtable pointer is checked against the section boundaries,
and the process is terminated if the vtable pointer does not fall into
the special ELF section.
To enable backwards compatibility, a special flag variable
(_IO_accept_foreign_vtables), protected by the pointer guard, avoids
process termination if libio stream object constructor functions have
been called earlier. Such constructor functions are called by the GCC
2.95 libstdc++ library, and this mechanism ensures compatibility with
old binaries. Existing callers inside glibc of these functions are
adjusted to call the original functions, not the wrappers which enable
vtable compatiblity.
The compatibility mechanism is used to enable passing FILE * objects
across a static dlopen boundary, too.
Both of "_IO_UNBUFFERED" and "_IO_LINE_BUF" are the bit flags, but I
find there are some codes looks like "_IO_LINE_BUF+_IO_UNBUFFERED",
while some codes are "_IO_LINE_BUF|_IO_UNBUFFERED".
I think the former is not good, even though the final result is same.
POSIX allows applications to switch file handles when a read results
in an end of file. Unset the cached offset at this point so that it
is queried again.
Currently we seek to end of file if there are unflushed writes or the
stream is in write mode, to get the current offset for writing in
append mode, which is the end of file. The latter case (i.e. stream
is in write mode, but no unflushed writes) is unnecessary since it
will only happen when the stream has just been flushed, in which case
the recorded offset ought to be reliable.
Removing that case lets ftell give the correct offset when it follows
an ftruncate. The latter truncates the file, but does not change the
file position, due to which it is permissible to call ftell without an
intervening fseek call.
Tested on x86_64 to verify that the added test case fails without the
patch and succeeds with it, and that there are no additional
regressions due to it.
[BZ #17647]
* libio/fileops.c (do_ftell): Seek only when there are
unflushed writes.
* libio/wfileops.c (do_ftell_wide): Likewise.
* libio/tst-ftell-active-handler.c (do_ftruncate_test): New
test case.
(do_one_test): Call it.
The offset computation in write mode uses the fact that _IO_read_end
is kept in sync with the external file offset. This however is not
true when O_APPEND is in effect since switching to write mode ought to
send the external file offset to the end of file without making the
necessary adjustment to _IO_read_end.
Hence in append mode, offset computation when writing should only
consider the effect of unflushed writes, i.e. from _IO_write_base to
_IO_write_ptr.
The wiki has a detailed document that describes the rationale for
offsets returned by ftell in various conditions:
https://sourceware.org/glibc/wiki/File%20offsets%20in%20a%20stdio%20stream%20and%20ftell
The ftell implementation was made conservative to ensure that
incorrectly cached offsets never affect it. However, this causes
problems for append mode when a file stream is rewound. Additionally,
the 'clever' trick of using stat to get position for append mode files
caused more problems than it solved and broke old behavior. I have
described the various problems that it caused and then finally the
solution.
For a and a+ mode files, rewinding the stream should result in ftell
returning 0 as the offset, but the stat() trick caused it to
(incorrectly) always return the end of file. Now I couldn't find
anything in POSIX that specifies the stream position after rewind()
for a file opened in 'a' mode, but for 'a+' mode it should be set to
0. For 'a' mode too, it probably makes sense to keep it set to 0 in
the interest of retaining old behavior.
The initial file position for append mode files is implementation
defined, so the implementation could either retain the current file
position or move the position to the end of file. The earlier ftell
implementation would move the offset to end of file for append-only
mode, but retain the old offset for a+ mode. It would also cache the
offset (this detail is important). My patch broke this and would set
the initial position to end of file for both append modes, thus
breaking old behavior. I was ignorant enough to write an incorrect
test case for it too.
The Change:
I have now brought back the behavior of seeking to end of file for
append-only streams, but with a slight difference. I don't cache the
offset though, since we would want ftell to query the current file
position through lseek while the stream is not active. Since the
offset is moved to the end of file, we can rely on the file position
reported by lseek and we don't need to resort to the stat() nonsense.
Finally, the cache is always reliable, except when there are unflished
writes in an append mode stream (i.e. both a and a+). In the latter
case, it is safe to just do an lseek to SEEK_END. The value can be
safely cached too, since the file handle is already active at this
point. Incidentally, this is the only state change we affect in the
file handle (apart from taking locks of course).
I have also updated the test case to correct my impression of the
initial file position for a+ streams to the initial behavior. I have
verified that this does not break any existing tests in the testsuite
and also passes with the new tests.
The cached offset is reliable to use in ftell when the stream handle
is active. We can consider a stream as being active when there is
unflushed data. However, even in this case, we can use the cached
offset only when the stream is not being written to in a+ mode,
because this case may have unflushed data and a stale offset; the
previous read could have sent it off somewhere other than the end of
the file.
There were a couple of adjustments necessary to get this to work.
Firstly, fdopen now ceases to use _IO_attach_fd because it sets the
offset cache to the current file position. This is not correct
because there could be changes to the file descriptor before the
stream handle is activated, which would not get reflected.
A similar offset caching action is done in _IO_fwide, claiming that
wide streams have 'problems' with the file offsets. There don't seem
to be any obvious problems with not having the offset cache available,
other than that it will have to be queried in a subsequent
read/write/seek. I have removed this as well.
The testsuite passes successfully with these changes on x86_64.
ftell semantics are distinct from fseek(SEEK_CUR) especially when it
is called on a file handler that is not yet active. Due to this
caveat, much care needs to be taken while modifying the handler data
and hence, this first iteration on separating out ftell focusses on
maintaining handler data integrity at all times while it figures out
the current stream offset. The result is that it makes a syscall for
every offset request.
There is scope for optimizing this by caching offsets when we know
that the handler is active. A simple way to find out is when the
buffers have data. It is not so simple to find this out when the
buffer is empty without adding some kind of flag.
ftell tries to avoid flushing the buffer when it is in write mode by
converting the wide char data and placing it into the binary buffer.
If the output buffer space is full and there is data to write, the
code reverts to flushing the buffer. This breaks when there is space
in the buffer but it is not enough to convert the next character in
the wide data buffer, due to which __codecvt_do_out returns a
__codecvt_partial status. In this case, ftell keeps running in an
infinite loop.
The fix here is to detect the __codecvt_partial status in addition to
checking if the buffer is full. I have also added a test case that
demonstrates the infinite loop.
[BZ #14543]
Set the internal buffer state correctly whenever the external buffer
state is modified by fseek by either computing the current
_IO_read_ptr/end for the internal buffer based on the new _IO_read_ptr
in the external buffer or converting the content read into the
external buffer, up to the extent of the requested fseek offset.
* libio/wfileops.c (_IO_wfile_underflow): Fix handling of
incomplete characters at end of input buffer.
* libio/Makefile (tests): Add tst-fgetwc.
* libio/tst-fgetwc.c: New file.
* libio/tst-fgetwc.input: New file.
* sysdeps/unix/sysv/linux/tcgetattr.c (__tcgetattr): Fill in c_ispeed
and c_ospeed fields.
* sysdeps/unix/sysv/linux/speed.c (cfsetospeed): Set c_ospeed field.
(cfsetispeed): Set c_ispeed field.
* sysdeps/unix/sysv/linux/tcsetattr.c (IBAUD0): Define unconditionally
to match corresponding speed.c code.
2003-09-06 Ulrich Drepper <drepper@redhat.com>
* libio/wfileops.c (_IO_wfile_underflow): Mark beginning of the
narrow character buffer.
* libio/Makefile: Add rules to build and run bug-ftell.
* libio/bug-ftell.c: New file.
* stdio-common/vfprintf.c: Don't use the first grouping number twice.
* stdio-common/vfscanf.c (vfscanf): Fix recognition of characters
matching the decimal point and possibly leading the thousands
separator. This caused the recognition of thousands separators to
always fail.
2003-09-05 Ulrich Drepper <drepper@redhat.com>
* libio/fileops.c (_IO_new_file_overflow): Handle switching to
write mode from read in backup buffer.
* libio/Makefile (tests): Add bug-ungetc2.
* libio/bug-ungetc2.c: New file.
2003-09-05 Roland McGrath <roland@redhat.com>
>>>>>>> 1.7905
* sysdeps/unix/sysv/linux/linux_fsinfo.h: Define VXFS_SUPER_MAGIC.
2002-08-26 Ulrich Drepper <drepper@redhat.com>
* libio/wfileops.c (_IO_wfile_seekoff): Set fp->_offset after
finding the read position [PR libc/4265].
* libio/Makefile (tests): Add bug-rewind2.
* libio/bug-rewind2.c: New file.
2002-08-04 Ulrich Drepper <drepper@redhat.com>
* stdio-common/psignal.c: Declare _sys_siglist_internal. Use USEINT
to access _sys_siglist.
* string/strsignal.c: Likewise.
* sysdeps/generic/siglist.c: Add _sys_siglist_internal alias.
* sysdeps/gnu/siglist.c: Likewise.
* sysdeps/unix/siglist.c: Likewise.
* sysdeps/unix/sysv/linux/arm/siglist.c: Likewise.
* libio/fileops.c: Add missing INTUSEs for _IO_file_jumps.
* libio/wfileops.c: Add missing INTUSE for _IO_file_close.
* intl/dcigettext.c: Define _nl_default_dirname_internal as hidden
alias and use it.
* intl/bindtextdom.c: Use _nl_default_dirname_internal.
* include/netinet/in.h: Add declaration of in6addr_loopback_internal.
* inet/in6_addr.c: Add INTVARDEF for in6addr_loopback.
* sysdeps/posix/getaddrinfo.c: Use INTUSE for in6addr_loopback access.
* include/time.h: Add libc_hidden_proto for __gmtime_r.
* time/gmtime.c (__gmtime_r): Add libc_hidden_def.
* iconv/Versions: Replace __gconv_alias_db, __gconv_modules_db,
and __gconv_cache with __gconv_get_alias_db, __gconv_get_modules_db,
and __gconv_get_cache respectively.
* iconv/gconv_cache.c (gconv_cache): Renamed for __gconv_cache and
defined static. Change all users.
(__gconv_get_cache): New function.
* iconv/gconv_db.c (__gconv_get_modules_db): New function.
(__gconv_get_alias_db): New function.
* iconv/gconv_int.h (__gconv_alias_db): Declare as hidden.
(__conv_modules_db): Likewise.
Add prototypes for __gconv_get_cache, __gconv_get_modules_db,
and __gconv_get_alias_db.
* iconv/iconv_prog.c: Use the new functions instead of accessing the
variables.
* include/stdlib.h: Add prototype and libc_hidden_proto for
__default_morecore.
* sysdeps/generic/morecore.c: Include <stdlib.h>.
* malloc/obstack.c: Remove fputs macro.
* malloc/mtrace.c: Remove fopen macro.
(_IO_file_jumps_mmap): Use it.
(_IO_file_underflow_mmap): Rewritten. If after EOF or fflush,
repeat the stat check and resize the mapped buffer as necessary.
2002-07-31 Roland McGrath <roland@frob.com>
* libio/fileops.c (decide_maybe_mmap): New static function.
Code taken from libio/iofopen.c:__fopen_maybe_mmap to try to
mmap the file contents. Then switch the jump tables to the mmap
tables if it worked, or the vanilla file tables if not.
(_IO_file_underflow_maybe_mmap): New function.
(_IO_file_seekoff_maybe_mmap): New function.
(_IO_file_xsgetn_maybe_mmap): New function.
(_IO_file_jumps_maybe_mmap): New variable, jump table using those.
* libio/libioP.h: Declare those.
* libio/wfileops.c (_IO_wfile_underflow_maybe_mmap): New function.
(_IO_wfile_jumps_maybe_mmap): New variable, jump table using that.
* libio/iofopen.c (__fopen_maybe_mmap): Don't try to mmap here.
If the stream is read-only, set its jump tables to those new ones.
* libio/iofdopen.c (_IO_new_fdopen) [_G_HAVE_MMAP]: Set the initial
jump tables to the maybe_mmap ones, and don't call __fopen_maybe_mmap.
We need the tables set before _IO_file_attach.
* libio/tst-mmap-eofsync.c: New file.
* libio/tst-mmap-fflushsync.c: New file.
* libio/bug-mmap-fflush.c: New file.
* libio/tst-mmap2-eofsync.c: New file.
* libio/Makefile (tests): Add them.
* libio/wfileops.c (_IO_wfile_underflow_mmap): Don't set EOF bit when
_IO_file_underflow_mmap fails, it already set the appropriate bit.
