The offset computation in write mode uses the fact that _IO_read_end
is kept in sync with the external file offset. This however is not
true when O_APPEND is in effect since switching to write mode ought to
send the external file offset to the end of file without making the
necessary adjustment to _IO_read_end.
Hence in append mode, offset computation when writing should only
consider the effect of unflushed writes, i.e. from _IO_write_base to
_IO_write_ptr.
The wiki has a detailed document that describes the rationale for
offsets returned by ftell in various conditions:
https://sourceware.org/glibc/wiki/File%20offsets%20in%20a%20stdio%20stream%20and%20ftell
ChangeLog:
2014-03-17 Will Newton <will.newton@linaro.org>
* libio/genops.c: Check whether
_IO_JUMPS_OFFSET is defined with #ifdef rather
than #if.
* libio/libioP.h: Likewise.
* stdio-common/vfprintf.c: Likewise.
fdopen should only be allowed to change the offset in the file it
attaches to if it is setting O_APPEND. If O_APPEND is already set, it
should not change the state of the handle.
The ftell implementation was made conservative to ensure that
incorrectly cached offsets never affect it. However, this causes
problems for append mode when a file stream is rewound. Additionally,
the 'clever' trick of using stat to get position for append mode files
caused more problems than it solved and broke old behavior. I have
described the various problems that it caused and then finally the
solution.
For a and a+ mode files, rewinding the stream should result in ftell
returning 0 as the offset, but the stat() trick caused it to
(incorrectly) always return the end of file. Now I couldn't find
anything in POSIX that specifies the stream position after rewind()
for a file opened in 'a' mode, but for 'a+' mode it should be set to
0. For 'a' mode too, it probably makes sense to keep it set to 0 in
the interest of retaining old behavior.
The initial file position for append mode files is implementation
defined, so the implementation could either retain the current file
position or move the position to the end of file. The earlier ftell
implementation would move the offset to end of file for append-only
mode, but retain the old offset for a+ mode. It would also cache the
offset (this detail is important). My patch broke this and would set
the initial position to end of file for both append modes, thus
breaking old behavior. I was ignorant enough to write an incorrect
test case for it too.
The Change:
I have now brought back the behavior of seeking to end of file for
append-only streams, but with a slight difference. I don't cache the
offset though, since we would want ftell to query the current file
position through lseek while the stream is not active. Since the
offset is moved to the end of file, we can rely on the file position
reported by lseek and we don't need to resort to the stat() nonsense.
Finally, the cache is always reliable, except when there are unflished
writes in an append mode stream (i.e. both a and a+). In the latter
case, it is safe to just do an lseek to SEEK_END. The value can be
safely cached too, since the file handle is already active at this
point. Incidentally, this is the only state change we affect in the
file handle (apart from taking locks of course).
I have also updated the test case to correct my impression of the
initial file position for a+ streams to the initial behavior. I have
verified that this does not break any existing tests in the testsuite
and also passes with the new tests.
The test functions used a variable ret to store failure codes for
individual tests, but the variable was incorrectly used to record
other failure codes too, resulting in overwriting of the tests status.
This is now fixed by making sure that the ret variable is used only
for recording test failures.
* libio/tst-ftell-active-handler.c (do_ftell_test): Don't mix
up test status with function return status.
(do_write_test): Likewise.
(do_append_test): Likewise.
This patch is a revised and updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00196.html>.
In order to generate overall summaries of the results of all tests in
the glibc testsuite, we need to identify and concatenate the files
with the results of individual tests.
Tomas Dohnalek's patch used $(common-objpfx)*/*.test-result for this.
However, the normal glibc approach is explicit enumeration of the
expected set of files with a given property, rather than all files
matching some pattern like that. Furthermore, we would like to be
able to mark tests as UNRESOLVED if the file with their results is for
some reason missing, and in future we would like to be able to mark
tests as UNSUPPORTED if they are disabled for a particular
configuration (rather than simply having them missing from the list of
tests as at present). Such handling of tests that were not run or did
not record results requires an explicit enumeration of tests.
For the tests following the default makefile rules, $(tests) (and
$(xtests)) provides such an enumeration. Others, however, are added
directly as dependencies of the "tests" and "xtests" makefile
targets. This patch changes the makefiles to put them in variables
tests-special and xtests-special, with appropriate dependencies on the
tests listed there then being added centrally.
Those variables are used in Rules and so need to be set before Rules
is included in a subdirectory makefile, which is often earlier in the
makefile than the dependencies were present before. We previously
discussed the question of where to include Rules; see the question at
<https://sourceware.org/ml/libc-alpha/2012-11/msg00798.html>, and a
discussion in
<https://sourceware.org/ml/libc-alpha/2013-01/msg00337.html> of why
Rules is included early rather than late in subdirectory makefiles.
It was necessary to avoid an indirection through the check-abi target
and get the check-abi-* targets for individual libraries into the
tests-special variable. The intl/ test $(objpfx)tst-gettext.out,
previously built only because of dependencies from other tests, was
also added to tests-special for the same reason.
The entries in tests-special are the full makefile targets, complete
with $(objpfx) and .out. If a future change causes tests to be named
consistently with a .out suffix, this can be changed to include just
the path relative to $(objpfx), without .out.
Tested x86_64, including that the same set of files is generated in
the build directory by a build and testsuite run both before and after
the patch (except for changes to the
elf/tst-null-argv.debug.out.<number> file name), and a build with
run-built-tests=no to verify there aren't any more obvious instances
of the issue Marcus Shawcroft reported with a previous version in
<https://sourceware.org/ml/libc-alpha/2014-01/msg00462.html>.
* Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(tests): Depend on $(tests-special).
* Makerules (check-abi-list): New variable.
(check-abi): Depend on $(check-abi-list).
[$(subdir) = elf] (tests-special): Add
$(objpfx)check-abi-libc.out.
[$(build-shared) = yes && subdir] (tests-special): Add
$(check-abi-list).
[$(build-shared) = yes && subdir] (tests): Do not depend on
check-abi.
* Rules (tests): Depend on $(tests-special).
(xtests): Depend on $(xtests-special).
* catgets/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* conform/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* elf/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* grp/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* iconv/Makefile (xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* iconvdata/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* intl/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable. Also add
$(objpfx)tst-gettext.out.
* io/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* libio/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* malloc/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* misc/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* nptl/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* nptl_db/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* posix/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* resolv/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(xtests): Change dependencies to ....
(xtests-special): ... additions to this variable.
* stdio-common/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
(do-tst-unbputc): Remove target.
(do-tst-printf): Likewise.
* stdlib/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* string/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
* sysdeps/x86/Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
localedata:
* Makefile (tests): Change dependencies to ....
(tests-special): ... additions to this variable.
The cached offset is reliable to use in ftell when the stream handle
is active. We can consider a stream as being active when there is
unflushed data. However, even in this case, we can use the cached
offset only when the stream is not being written to in a+ mode,
because this case may have unflushed data and a stale offset; the
previous read could have sent it off somewhere other than the end of
the file.
There were a couple of adjustments necessary to get this to work.
Firstly, fdopen now ceases to use _IO_attach_fd because it sets the
offset cache to the current file position. This is not correct
because there could be changes to the file descriptor before the
stream handle is activated, which would not get reflected.
A similar offset caching action is done in _IO_fwide, claiming that
wide streams have 'problems' with the file offsets. There don't seem
to be any obvious problems with not having the offset cache available,
other than that it will have to be queried in a subsequent
read/write/seek. I have removed this as well.
The testsuite passes successfully with these changes on x86_64.
ftell semantics are distinct from fseek(SEEK_CUR) especially when it
is called on a file handler that is not yet active. Due to this
caveat, much care needs to be taken while modifying the handler data
and hence, this first iteration on separating out ftell focusses on
maintaining handler data integrity at all times while it figures out
the current stream offset. The result is that it makes a syscall for
every offset request.
There is scope for optimizing this by caching offsets when we know
that the handler is active. A simple way to find out is when the
buffers have data. It is not so simple to find this out when the
buffer is empty without adding some kind of flag.
In <https://sourceware.org/ml/libc-alpha/2014-01/msg00196.html> I
noted it was necessary to add includes of Makeconfig early in various
subdirectory makefiles for the tests-special variable settings added
by that patch to be conditional on configuration information. No-one
commented on the general question there of whether Makeconfig should
always be included immediately after the definition of subdir.
This patch implements that early inclusion of Makeconfig in each
directory (which is a lot easier than consistent placement of includes
of Rules). Includes are added if needed, or moved up if already
present. Subdirectory "all:" targets are removed, since Makeconfig
provides one.
There is potential for further cleanups I haven't done. Rules and
Makerules have code such as
ifneq "$(findstring env,$(origin headers))" ""
headers :=
endif
to override to empty any value of various variables that came from the
environment. I think there is a case for Makeconfig setting all the
subdirectory variables (other than subdir) to empty to ensure no
outside value is going to take effect if a subdirectory fails to
define a variable. (A list of such variables, possibly out of date
and incomplete, is in manual/maint.texi.) Rules and Makerules would
give errors if Makeconfig hadn't already been included, instead of
including it themselves. The special code to override values coming
from the environment would then be obsolete and could be removed.
Tested x86_64, including that installed binaries are identical before
and after the patch.
* argp/Makefile: Include Makeconfig immediately after defining
subdir.
* assert/Makefile: Likewise.
* benchtests/Makefile: Likewise.
* catgets/Makefile: Likewise.
* conform/Makefile: Likewise.
* crypt/Makefile: Likewise.
* csu/Makefile: Likewise.
(all): Remove target.
* ctype/Makefile: Include Makeconfig immediately after defining
subdir.
* debug/Makefile: Likewise.
* dirent/Makefile: Likewise.
* dlfcn/Makefile: Likewise.
* gmon/Makefile: Likewise.
* gnulib/Makefile: Likewise.
* grp/Makefile: Likewise.
* gshadow/Makefile: Likewise.
* hesiod/Makefile: Likewise.
* hurd/Makefile: Likewise.
(all): Remove target.
* iconvdata/Makefile: Include Makeconfig immediately after
defining subdir.
* inet/Makefile: Likewise.
* intl/Makefile: Likewise.
* io/Makefile: Likewise.
* libio/Makefile: Likewise.
(all): Remove target.
* locale/Makefile: Include Makeconfig immediately after defining
subdir.
* login/Makefile: Likewise.
* mach/Makefile: Likewise.
(all): Remove target.
* malloc/Makefile: Include Makeconfig immediately after defining
subdir.
(all): Remove target.
* manual/Makefile: Include Makeconfig immediately after defining
subdir.
* math/Makefile: Likewise.
* misc/Makefile: Likewise.
* nis/Makefile: Likewise.
* nss/Makefile: Likewise.
* po/Makefile: Likewise.
(all): Remove target.
* posix/Makefile: Include Makeconfig immediately after defining
subdir.
* pwd/Makefile: Likewise.
* resolv/Makefile: Likewise.
* resource/Makefile: Likewise.
* rt/Makefile: Likewise.
* setjmp/Makefile: Likewise.
* shadow/Makefile: Likewise.
* signal/Makefile: Likewise.
* socket/Makefile: Likewise.
* soft-fp/Makefile: Likewise.
* stdio-common/Makefile: Likewise.
* stdlib/Makefile: Likewise.
* streams/Makefile: Likewise.
* string/Makefile: Likewise.
* sunrpc/Makefile: Likewise.
(all): Remove target.
* sysvipc/Makefile: Include Makeconfig immediately after defining
subdir.
* termios/Makefile: Likewise.
* time/Makefile: Likewise.
* timezone/Makefile: Likewise.
(all): Remove target.
* wcsmbs/Makefile: Include Makeconfig immediately after defining
subdir.
* wctype/Makefile: Likewise.
libidn/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
localedata/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
(all): Remove target.
nptl/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
nptl_db/ChangeLog:
* Makefile: Include Makeconfig immediately after defining subdir.
Most glibc tests that use mtrace to verify that there were no memory
leaks from the glibc facilities used in a given test depend on the
.out file of the previous test so that the mtrace test runs mtrace and
nothing else.
Two, however, have a single target combining mtrace with something
else. In the case of libio/tst-fopenloc.check, the test both compares
the output with an expected baseline and runs mtrace. In the case of
posix/tst-rxspencer-mem, the test is run (with different command line
from the main run) and then mtrace is run, from the same makefile
target.
This patch splits both of these tests up to use separate makefile
targets for each thing tested; in the tst-rxspencer case, a file
tst-rxspencer-no-utf8.c is created that just includes tst-rxspencer.c,
as is usual for tests where the same code gets tested in different
compile-time or runtime configurations.
Adding $(evaluate-test) to test commands, as in
<https://sourceware.org/ml/libc-alpha/2014-01/msg00194.html>, will no
longer need to insert && between multiple commands, as all tests will
either have just a single command or already use &&.
Tested x86_64.
* libio/Makefile ($(objpfx)tst-fopenloc.check): Split into
separate $(objpfx)tst-fopenloc-cmp.out and
$(objpfx)tst-fopenloc-mem.out targets.
(tests): Update dependencies.
* posix/Makefile (tests variable): Add tst-rxspencer-no-utf8.
(generated): Change tst-rxspencer-mem and tst-rxspencer.mtrace to
tst-rxspencer-no-utf8-mem and tst-rxspencer-no-utf8.mtrace.
(tst-rxspencer-no-utf8-ARGS): New variable.
(tst-rxspencer-no-utf8-ENV): Likewise.
(tests target): Depend on $(objpfx)tst-rxspencer-no-utf8-mem
instead of $(objpfx)tst-rxspencer-mem.
($(objpfx)tst-rxspencer-mem): Change target to
$(objpfx)tst-rxspencer-no-utf8-mem. Depend on
$(objpfx)tst-rxspencer-no-utf8.out instead of running test program.
* posix/tst-rxspencer-no-utf8.c: New file.
This patch cleans up cases of __USE_MISC that are trivially redundant
after the recent substitution of __USE_MISC for __USE_BSD and
__USE_SVID: either in constructs such as "defined __USE_MISC ||
defined __USE_MISC", or else (in the bits/mman.h case) a conditional
on __USE_MISC nested inside another __USE_MISC conditional. (The
cleanups remaining after this patch are still quite large, but it
seems a reasonable piece to separate out.)
Tested x86_64.
* bits/mman.h [__USE_MISC]: Remove redundant conditionals.
* ctype/ctype.h [__USE_MISC]: Likewise.
* dirent/dirent.h [__USE_MISC]: Likewise.
* grp/grp.h [__USE_MISC]: Likewise.
* io/fcntl.h [__USE_MISC]: Likewise.
* io/sys/stat.h [__USE_MISC]: Likewise.
* libio/stdio.h [__USE_MISC]: Likewise.
* posix/unistd.h [__USE_MISC]: Likewise.
* pwd/pwd.h [__USE_MISC]: Likewise.
* stdlib.h [__USE_MISC]: Likewise.
* string/bits/string2.h [__USE_MISC]: Likewise.
* string/string.h [__USE_MISC]: Likewise.
* time/time.h [__USE_MISC]: Likewise.
ftell tries to avoid flushing the buffer when it is in write mode by
converting the wide char data and placing it into the binary buffer.
If the output buffer space is full and there is data to write, the
code reverts to flushing the buffer. This breaks when there is space
in the buffer but it is not enough to convert the next character in
the wide data buffer, due to which __codecvt_do_out returns a
__codecvt_partial status. In this case, ftell keeps running in an
infinite loop.
The fix here is to detect the __codecvt_partial status in addition to
checking if the buffer is full. I have also added a test case that
demonstrates the infinite loop.
Partially revert commits 2b766585f9 and
de2fd463b1, which were intended to fix BZ#11741
but caused another, likely worse bug, namely that fwrite() and fputs() could,
in an error path, read data beyond the end of the specified buffer, and
potentially even write this data to the file.
Fix BZ#11741 properly by checking the return value from _IO_padn() in
stdio-common/vfprintf.c.
Fixes 15381.
Using wide character function is on byte oriented memstream is undefined
behaviour. This behaviour was masked by not initializing wide struct
info. We now initialize it to cause a predictable crash.
In BZ #12724, partial support for POSIX 2008 fclose behavior was added.
Since it isn't entirely conforming to the spec, some applications are
known to be breaking in this intermediate state. So revert the partial
support until we can get things fully implemented.
This reverts commit fcabc0f8b1.
Signed-off-by: Mike Frysinger <vapier@gentoo.org>
[BZ #14543]
Set the internal buffer state correctly whenever the external buffer
state is modified by fseek by either computing the current
_IO_read_ptr/end for the internal buffer based on the new _IO_read_ptr
in the external buffer or converting the content read into the
external buffer, up to the extent of the requested fseek offset.
