All the changes are in comments or '#error' messages.
Applying this commit results in bit-identical rebuild of iconvdata/*.so
Reviewed-by: Florian Weimer <fw@deneb.enyo.de>
When opening a temporary file without O_CLOEXEC we risk leaking the
file descriptor if another thread calls (fork and then) exec while we
have the fd open. Fix this by consistently passing O_CLOEXEC everywhere
where we open a file for internal use (and not to return it to the user,
in which case the API defines whether or not the close-on-exec flag
shall be set on the returned fd).
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
Message-Id: <20230419160207.65988-4-bugaevc@gmail.com>
They are both used by __libc_freeres to free all library malloc
allocated resources to help tooling like mtrace or valgrind with
memory leak tracking.
The current scheme uses assembly markers and linker script entries
to consolidate the free routine function pointers in the RELRO segment
and to be freed buffers in BSS.
This patch changes it to use specific free functions for
libc_freeres_ptrs buffers and call the function pointer array directly
with call_function_static_weak.
It allows the removal of both the internal macros and the linker
script sections.
Checked on x86_64-linux-gnu, i686-linux-gnu, and aarch64-linux-gnu.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Use put/get macros __builtin_bswap32 instead. It allows to remove
the unaligned routines, the compiler will generate unaligned access
if the ABI allows it.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
And use a packed structure instead. The compiler generates optimized
unaligned code if the architecture supports it.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Almost all uses of rawmemchr find the end of a string. Since most targets use
a generic implementation, replacing it with strchr is better since that is
optimized by compilers into strlen (s) + s. Also fix the generic rawmemchr
implementation to use a cast to unsigned char in the if statement.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
I've updated copyright dates in glibc for 2023. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files.
In the future, this will result in a compilation failure if the
macros are unexpectedly undefined (due to header inclusion ordering
or header inclusion missing altogether).
Assembler sources are more difficult to convert. In many cases,
they are hand-optimized for the mangling and no-mangling variants,
which is why they are not converted.
sysdeps/s390/s390-32/__longjmp.c and sysdeps/s390/s390-64/__longjmp.c
are special: These are C sources, but most of the implementation is
in assembler, so the PTR_DEMANGLE macro has to be undefined in some
cases, to match the assembler style.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This allows us to define a generic no-op version of PTR_MANGLE and
PTR_DEMANGLE. In the future, we can use PTR_MANGLE and PTR_DEMANGLE
unconditionally in C sources, avoiding an unintended loss of hardening
due to missing include files or unlucky header inclusion ordering.
In i386 and x86_64, we can avoid a <tls.h> dependency in the C
code by using the computed constant from <tcb-offsets.h>. <sysdep.h>
no longer includes these definitions, so there is no cyclic dependency
anymore when computing the <tcb-offsets.h> constants.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The Z modifier is a nonstandard synonymn for z (that predates z
itself) and compiler might issue an warning for in invalid
conversion specifier.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
It's possible that inode numbers are outside the 32-bit range.
The existing code only handles the in-libc case correctly, and
still uses the legacy interfaces when building iconv.
Suggested-by: Helge Deller <deller@gmx.de>
On s390x when compiling with GCC 12, I get this warning:
utf8-utf16-z9.c:
../iconv/loop.c: In function ‘__from_utf8_loop_etf3eh_single’:
../iconv/loop.c:445:22: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
445 | bytebuf[inlen++] = *inptr++;
| ~~~~~~~~~~~~~~~~~^~~~~~~~~~
../iconv/loop.c:381:17: note: at offset 4 into destination object ‘bytebuf’ of size 4
381 | unsigned char bytebuf[MAX_NEEDED_INPUT];
| ^~~~~~~
../iconv/loop.c:445:22: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
445 | bytebuf[inlen++] = *inptr++;
| ~~~~~~~~~~~~~~~~~^~~~~~~~~~
../iconv/loop.c:381:17: note: at offset 5 into destination object ‘bytebuf’ of size 4
381 | unsigned char bytebuf[MAX_NEEDED_INPUT];
| ^~~~~~~
This patch tells the compiler that inend is always behind inptr which
avoids the warning. Note that the SINGLE function is only used to
implement the mb*towc*() or wc*tomb*() functions. Those functions use
inptr and inend pointing to a variable on stack, compute the inend pointer
or explicitly check the arguments which always leads to inptr < inend.
Special notes for backporters (according to Siddhesh Poyarekar):
If someone wants to backport this patch to release branches, they should
also backport the following wcrtomb change. Otherwise the assumptions
assumed by this patch are not true.
commit 9bcd12d223
Author: Siddhesh Poyarekar <siddhesh@sourceware.org>
Date: Fri May 13 19:10:15 2022 +0530
wcrtomb: Make behavior POSIX compliant
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The issue is only when used within libc.so (iconvconfig already builds
with _TIME_SIZE=64).
This is a missing spot initially from 52a5fe70a2.
Checked on i686-linux-gnu.
On 32-bit machines this has no affect. On 64-bit machines
{u}int_fast{16|32} are set as {u}int64_t which is often not
ideal. Particularly x86_64 this change both saves code size and
may save instruction cost.
Full xcheck passes on x86_64.
This patch associates iconv_close as a deallocator for iconv_open. This
required moving the iconv_close declaration above iconv_open.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
1. Also generate .d dependency files for $(tests-container) and
$(tests-printers).
2. elf: Add tst-auditmod17.os to extra-test-objs.
3. iconv: Add tst-gconv-init-failure-mod.os to extra-test-objs.
4. malloc: Rename extra-tests-objs to extra-test-objs.
5. linux: Add tst-sysconf-iov_max-uapi.o to extra-test-objs.
6. x86_64: Add tst-x86_64mod-1.o, tst-platformmod-2.o, test-libmvec.o,
test-libmvec-avx.o, test-libmvec-avx2.o and test-libmvec-avx512f.o to
extra-test-objs.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
I've updated copyright dates in glibc for 2022. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files. As well as the usual annual
updates, mainly dates in --version output (minus csu/version.c which
previously had to be handled manually but is now successfully updated
by update-copyrights), there is a small change to the copyright notice
in NEWS which should let NEWS get updated automatically next year.
Please remember to include 2022 in the dates for any new files added
in future (which means updating any existing uncommitted patches you
have that add new files to use the new copyright dates in them).
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.
I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah. I don't
know why I run into these diagnostics whereas others evidently do not.
remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
Currently the timeout for each iconv test is hard coded to 3 seconds.
On my OpenRISC test platform this is too slow and the test fails with a
HANG error.
This change uses the available TIMEOUTFACTOR to compute the timeout.
The default value is still 3.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The consolidation of configuration parsing broke behaviour with
--prefix, where the prefix bled into the modules cache. Accept a
prefix which, when non-NULL, is prepended to the path when looking for
configuration files but only the original directory is added to the
modules cache.
This has no effect on the codegen of gconv_conf since it passes NULL.
Reported-by: Patrick McCarty <patrick.mccarty@intel.com>
Reported-by: Michael Hudson-Doyle <michael.hudson@canonical.com>
Reviewed-by: Andreas Schwab <schwab@linux-m68k.org>
We add a new C.UTF-8 locale. This locale is not builtin to glibc, but
is provided as a distinct locale. The locale provides full support for
UTF-8 and this includes full code point sorting via STRCMP-based
collation (strcmp or wcscmp).
The collation uses a new keyword 'codepoint_collation' which drops all
collation rules and generates an empty zero rules collation to enable
STRCMP usage in collation. This ensures that we get full code point
sorting for C.UTF-8 with a minimal 1406 bytes of overhead (LC_COLLATE
structure information and ASCII collating tables).
The new locale is added to SUPPORTED. Minimal test data for specific
code points (minus those not supported by collate-test) is provided in
C.UTF-8.in, and this verifies code point sorting is working reasonably
across the range. The locale was tested manually with the full set of
code points without failure.
The locale is harmonized with locales already shipping in various
downstream distributions. A new tst-iconv9 test is added which verifies
the C.UTF-8 locale is generally usable.
Testing for fnmatch, regexec, and recomp is provided by extending
bug-regex1, bugregex19, bug-regex4, bug-regex6, transbug, tst-fnmatch,
tst-regcomp-truncated, and tst-regex to use C.UTF-8.
Tested on x86_64 or i686 without regression.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Current GCC mainline produces -Wstringop-overflow errors building some
iconv converters, as discussed at
<https://gcc.gnu.org/pipermail/gcc/2021-July/236943.html>. Add an
__builtin_unreachable call as suggested so that GCC can see the case
that would involve a buffer overflow is unreachable; because the
unreachability depends on valid conversion state being passed into the
function from previous conversion steps, it's not something the
compiler can reasonably deduce on its own.
Tested with build-many-glibcs.py that, together with
<https://sourceware.org/pipermail/libc-alpha/2021-August/130244.html>,
it restores the glibc build for powerpc-linux-gnu.
The allocated `conf` would leak if we have to skip over the file due
to the underlying filesystem not supporting dt_type.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
Build of iconvconfig failed with CFLAGS=-Os since __feof_unlocked is
not a public symbol. Replace with feof_unlocked (defined to
__feof_unlocked when IS_IN (libc)) to fix this.
Reported-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
It was noticed on big-endian systems that msgfmt would fail with the
following error:
msgfmt: gconv_builtin.c:70: __gconv_get_builtin_trans: Assertion `cnt < sizeof (map) / sizeof (map[0])' failed.
Aborted (core dumped)
This is only seen on installed systems because it was due to a
corrupted gconv-modules.cache. iconvconfig had the following issues
(it was specifically freeing fulldir that caused this issue, but other
cleanups are also needed) that this patch fixes.
- Add prefix only if dir starts with '/'
- Use asprintf instead of mempcpy so that the directory string is NULL
terminated
- Make a copy of the directory reference in new_module so that fulldir
can be freed within the same scope in handle_dir.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
On filesystems that do not support dt_type, a regular file shows up as
DT_UNKNOWN. Fall back to using lstat64 to read file properties in
such cases.
Reviewed-by: DJ Delorie <dj@redhat.com>
Drop local copy of gconv file parsing and use the one in
gconv_parseconfdir.h instead. Now there is a single implementation of
configuration file parsing.
Reviewed-by: DJ Delorie <dj@redhat.com>
Split configuration file processing into a separate header file and
include it. Macroize all calls that need to go through internal
interfaces so that iconvconfig can also use them.
Reviewed-by: DJ Delorie <dj@redhat.com>
The modules and nmodules parameters passed to add_modules, add_alias,
etc. are not used and are hence unnecessary. Remove them so that
their signatures match the functions in iconvconfig.
Reviewed-by: DJ Delorie <dj@redhat.com>
Reviewed-by: Andreas Schwab <schwab@linux-m68k.org>
The alloca sizes ought to be constrained to PATH_MAX, but replace them
with dynamic allocation to be safe. A static PATH_MAX array would
have worked too but Hurd does not have PATH_MAX and the code path is
not hot enough to micro-optimise this allocation. Revisit if any of
those realities change.
Reviewed-by: DJ Delorie <dj@redhat.com>
For the legacy ABI with supports 32-bit time_t it calls the 64-bit
time directly, since the LFS symbols calls the 64-bit time_t ones
internally.
Checked on i686-linux-gnu and x86_64-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Read configuration files with names ending in .conf in
GCONV_PATH/gconv-modules.d to mirror configuration flexibility in
iconvconfig into the iconv program and function.
Reviewed-by: DJ Delorie <dj@redhat.com>
In addition to GCONV_PATH/gconv-modules, also read module
configuration from *.conf files in GCONV_PATH/gconv-modules.d. This
allows a single gconv directory to have multiple sets of gconv modules
but at the same time, a single modules cache.
With this feature, one could separate the glibc supported gconv
modules into a minimal essential set (ISO-8859-*, UTF, etc.) from the
remaining modules. In future, these could be further segregated into
langpack-associated sets with their own
gconv-modules.d/someconfig.conf.
Reviewed-by: DJ Delorie <dj@redhat.com>
Split out configuration file handling code from handle_dir into its
own function so that it can be reused for multiple configuration
files.
Reviewed-by: DJ Delorie <dj@redhat.com>
When glibc is configured with --enable-hardcoded-path-in-tests,
"make xcheck" failed with
...
env GCONV_PATH=/export/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/iconvdata LOCPATH=/export/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/localedata LC_ALL=C /export/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/iconv/iconvconfig --output=$tmp --nostdlib /usr/lib64/gconv;
...
/export/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/iconv/iconvconfig: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /export/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/iconv/iconvconfig)
...
FAIL: iconv/test-iconvconfig
Since $(objpfx)iconvconfig is an installed program, run it with
$(run-program-prefix).
I've updated copyright dates in glibc for 2021. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files. As well as the usual annual
updates, mainly dates in --version output (minus csu/version.c which
previously had to be handled manually but is now successfully updated
by update-copyrights), there is a small change to the copyright notice
in NEWS which should let NEWS get updated automatically next year.
Please remember to include 2021 in the dates for any new files added
in future (which means updating any existing uncommitted patches you
have that add new files to use the new copyright dates in them).
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
Previously, in UCS4 conversion routines we limit the number of
characters we examine to the minimum of the number of characters in the
input and the number of characters in the output. This is not the
correct behavior when __GCONV_IGNORE_ERRORS is set, as we do not consume
an output character when we skip a code unit. Instead, track the input
and output pointers and terminate the loop when either reaches its
limit.
This resolves assertion failures when resetting the input buffer in a step of
iconv, which assumes that the input will be fully consumed given sufficient
output space.
The IBM1364, IBM1371, IBM1388, IBM1390 and IBM1399 character sets
share converter logic (iconvdata/ibm1364.c) which would reject
redundant shift sequences when processing input in these character
sets. This led to a hang in the iconv program (CVE-2020-27618).
This commit adjusts the converter to ignore redundant shift sequences
and adds test cases for iconv_prog hangs that would be triggered upon
their rejection. This brings the implementation in line with other
converters that also ignore redundant shift sequences (e.g. IBM930
etc., fixed in commit 692de4b396).
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Commit 91927b7c76 (Rewrite iconv option parsing [BZ #19519]) did not
handle cases where the output codeset for translations (via the `gettext'
family of functions) might have a caller specified encoding suffix such as
TRANSLIT or IGNORE. This led to a regression where translations did not
work when the codeset had a suffix.
This commit fixes the above issue by parsing any suffixes passed to
__dcigettext and adds two new test-cases to intl/tst-codeset.c to
verify correct behaviour. The iconv-internal function __gconv_create_spec
and the static iconv-internal function gconv_destroy_spec are now visible
internally within glibc and used in intl/dcigettext.c.
It replaces the internal usage of __{f,l}xstat{at}{64} with the
__{f,l}stat{at}{64}. It should not change the generate code since
sys/stat.h explicit defines redirections to internal calls back to
xstat* symbols.
Checked with a build for all affected ABIs. I also check on
x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
This commit replaces string manipulation during `iconv_open' and iconv_prog
option parsing with a structured, flag based conversion specification. In
doing so, it alters the internal `__gconv_open' interface and accordingly
adjusts its uses.
This change fixes several hangs in the iconv program and therefore includes
a new test to exercise iconv_prog options that originally led to these hangs.
It also includes a new regression test for option handling in the iconv
function.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
