While there are now clean -Os build and test results on x86_64 (given
my patch <https://sourceware.org/ml/libc-alpha/2018-02/msg00602.html>,
pending review), testing with -Os with build-many-glibcs.py shows the
build is still failing with -Os everywhere except for x86_64, x86 and
s390x.
There are a variety of different build failures, but the most common
seem to be in strcoll / wcscoll, similar to existing such cases where
DIAG_* are used to disable -Wmaybe-uninitialized. There are various
different failures even within those functions. This patch fixes one
particular case that seems quite common, where the warning appears at
the declarations of seq1 and seq2.
Tested with build-many-glibcs.py that this fixes the -Os build for
aarch64-linux-gnu with GCC 7.
* string/strcoll_l.c: Include <libc-diag.h>.
(STRCOLL): Ignore -Wmaybe-uninitialized for -Os around
declarations of seq1 and seq2.
<locale.h> is specified to define locale_t in POSIX.1-2008, and so are
all of the headers that define functions that take locale_t arguments.
Under _GNU_SOURCE, the additional headers that define such functions
have also always defined locale_t. Therefore, there is no need to use
__locale_t in public function prototypes, nor in any internal code.
* ctype/ctype-c99_l.c, ctype/ctype.h, ctype/ctype_l.c
* include/monetary.h, include/stdlib.h, include/time.h
* include/wchar.h, locale/duplocale.c, locale/freelocale.c
* locale/global-locale.c, locale/langinfo.h, locale/locale.h
* locale/localeinfo.h, locale/newlocale.c
* locale/nl_langinfo_l.c, locale/uselocale.c
* localedata/bug-usesetlocale.c, localedata/tst-xlocale2.c
* stdio-common/vfscanf.c, stdlib/monetary.h, stdlib/stdlib.h
* stdlib/strfmon_l.c, stdlib/strtod_l.c, stdlib/strtof_l.c
* stdlib/strtol.c, stdlib/strtol_l.c, stdlib/strtold_l.c
* stdlib/strtoll_l.c, stdlib/strtoul_l.c, stdlib/strtoull_l.c
* string/strcasecmp.c, string/strcoll_l.c, string/string.h
* string/strings.h, string/strncase.c, string/strxfrm_l.c
* sysdeps/ieee754/float128/strtof128_l.c
* sysdeps/ieee754/float128/wcstof128.c
* sysdeps/ieee754/float128/wcstof128_l.c
* sysdeps/ieee754/ldbl-128ibm/strtold_l.c
* sysdeps/ieee754/ldbl-64-128/strtold_l.c
* sysdeps/ieee754/ldbl-opt/nldbl-compat.c
* sysdeps/ieee754/ldbl-opt/nldbl-strfmon_l.c
* sysdeps/ieee754/ldbl-opt/nldbl-strtold_l.c
* sysdeps/ieee754/ldbl-opt/nldbl-wcstold_l.c
* sysdeps/powerpc/powerpc32/power7/strcasecmp.S
* sysdeps/powerpc/powerpc64/power7/strcasecmp.S
* sysdeps/x86_64/strcasecmp_l-nonascii.c
* sysdeps/x86_64/strncase_l-nonascii.c, time/strftime_l.c
* time/strptime_l.c, time/time.h, wcsmbs/mbsrtowcs_l.c
* wcsmbs/wchar.h, wcsmbs/wcscasecmp.c, wcsmbs/wcsncase.c
* wcsmbs/wcstod.c, wcsmbs/wcstod_l.c, wcsmbs/wcstof.c
* wcsmbs/wcstof_l.c, wcsmbs/wcstol_l.c, wcsmbs/wcstold.c
* wcsmbs/wcstold_l.c, wcsmbs/wcstoll_l.c, wcsmbs/wcstoul_l.c
* wcsmbs/wcstoull_l.c, wctype/iswctype_l.c
* wctype/towctrans_l.c, wctype/wcfuncs_l.c
* wctype/wctrans_l.c, wctype/wctype.h, wctype/wctype_l.c:
Change all uses of __locale_t to locale_t.
posix/wordexp-test.c used libc-internal.h for PTR_ALIGN_DOWN; similar
to what was done with libc-diag.h, I have split the definitions of
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN
to a new header, libc-pointer-arith.h.
It then occurred to me that the remaining declarations in libc-internal.h
are mostly to do with early initialization, and probably most of the
files including it, even in the core code, don't need it anymore. Indeed,
only 19 files actually need what remains of libc-internal.h. 23 others
need libc-diag.h instead, and 12 need libc-pointer-arith.h instead.
No file needs more than one of them, and 16 don't need any of them!
So, with this patch, libc-internal.h stops including libc-diag.h as
well as losing the pointer arithmetic macros, and all including files
are adjusted.
* include/libc-pointer-arith.h: New file. Define
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and
PTR_ALIGN_DOWN here.
* include/libc-internal.h: Definitions of above macros
moved from here. Don't include libc-diag.h anymore either.
* posix/wordexp-test.c: Include stdint.h and libc-pointer-arith.h.
Don't include libc-internal.h.
* debug/pcprofile.c, elf/dl-tunables.c, elf/soinit.c, io/openat.c
* io/openat64.c, misc/ptrace.c, nptl/pthread_clock_gettime.c
* nptl/pthread_clock_settime.c, nptl/pthread_cond_common.c
* string/strcoll_l.c, sysdeps/nacl/brk.c
* sysdeps/unix/clock_settime.c
* sysdeps/unix/sysv/linux/i386/get_clockfreq.c
* sysdeps/unix/sysv/linux/ia64/get_clockfreq.c
* sysdeps/unix/sysv/linux/powerpc/get_clockfreq.c
* sysdeps/unix/sysv/linux/sparc/sparc64/get_clockfreq.c:
Don't include libc-internal.h.
* elf/get-dynamic-info.h, iconv/loop.c
* iconvdata/iso-2022-cn-ext.c, locale/weight.h, locale/weightwc.h
* misc/reboot.c, nis/nis_table.c, nptl_db/thread_dbP.h
* nscd/connections.c, resolv/res_send.c, soft-fp/fmadf4.c
* soft-fp/fmasf4.c, soft-fp/fmatf4.c, stdio-common/vfscanf.c
* sysdeps/ieee754/dbl-64/e_lgamma_r.c
* sysdeps/ieee754/dbl-64/k_rem_pio2.c
* sysdeps/ieee754/flt-32/e_lgammaf_r.c
* sysdeps/ieee754/flt-32/k_rem_pio2f.c
* sysdeps/ieee754/ldbl-128/k_tanl.c
* sysdeps/ieee754/ldbl-128ibm/k_tanl.c
* sysdeps/ieee754/ldbl-96/e_lgammal_r.c
* sysdeps/ieee754/ldbl-96/k_tanl.c, sysdeps/nptl/futex-internal.h:
Include libc-diag.h instead of libc-internal.h.
* elf/dl-load.c, elf/dl-reloc.c, locale/programs/locarchive.c
* nptl/nptl-init.c, string/strcspn.c, string/strspn.c
* malloc/malloc.c, sysdeps/i386/nptl/tls.h
* sysdeps/nacl/dl-map-segments.h, sysdeps/x86_64/atomic-machine.h
* sysdeps/unix/sysv/linux/spawni.c
* sysdeps/x86_64/nptl/tls.h:
Include libc-pointer-arith.h instead of libc-internal.h.
* elf/get-dynamic-info.h, sysdeps/nacl/dl-map-segments.h
* sysdeps/x86_64/atomic-machine.h:
Add multiple include guard.
This commit adds a new DIAG_IGNORE_Os_NEEDS_COMMENT which is only
enabled when compiling with -Os. This allows developers working on
-Os enabled builds to mark false-positive warnings without impacting the
warnings emitted at -O2.
Then using the new DIAG_IGNORE_Os_NEEDS_COMMENT we fix 6 warnings
generated with GCC 5 to get -Os builds working again.
The optimization introduced in commit
f13c2a8dff, causes regressions in
sorting for languages that have digraphs that change sort order, like
cs_CZ which sorts ch between h and i.
My analysis shows the fast-forwarding optimization in STRCOLL advances
through a digraph while possibly stopping in the middle which results
in a subsequent skipping of the digraph and incorrect sorting. The
optimization is incorrect as implemented and because of that I'm
removing it for 2.23, and I will also commit this fix for 2.22 where
it was originally introduced.
This patch reverts the optimization, introduces a new bug-strcoll2.c
regression test that tests both cs_CZ.UTF-8 and da_DK.ISO-8859-1 and
ensures they sort one digraph each correctly. The optimization can't be
applied without regressing this test.
Checked on x86_64, bug-strcoll2.c fails without this patch and passes
after. This will also get a fix on 2.22 which has the same bug.
this is a path that should solve bug 15884. It complains about the performance
of strcoll(). It was found out that the runtime of strcoll() is actually bound
to strlen which is needed for calculating the size of a cache that was
installed to improve the comparison performance.
The idea for this patch was that the cache is only useful in rare cases
(strings of same length and same first-level-chars) and that it would be
better to avoid memory allocation at all. To prove this I wrote a performance
test bench-strcoll.c with test data in benchtests-strcoll.tar.gz. Also
modifications in benchtests/Makefile and localedata/Makefile are necessary to
make it work.
After removing the cache the strcoll method showed the predicted behavior
(getting slightly faster) in all but the test case for hindi word sorting.
This was due the hindi text having much more equal words than the other ones.
For equal strings the performance was worse since all comparison levels were
run through and from the second level on the cache improved the comparison
performance of the original version.
Therefore I added a bytewise test via strcmp iff the first level comparison
found that both strings did match because in this case it is very likely that
equal strings are compared. This solved the problem with the hindi test case
and improved the performance of the others.
Performance comparison:
glibc files -33.77%
vi_VN.UTF-8 -34.12%
en_US.UTF-8 -42.42%
ar_SA.UTF-8 -27.49%
zh_CN.UTF-8 +07.90%
cs_CZ.UTF-8 -29.67%
en_GB.UTF-8 -28.50%
da_DK.UTF-8 -36.57%
pl_PL.UTF-8 -39.31%
fr_FR.UTF-8 -28.57%
pt_PT.UTF-8 -22.82%
el_GR.UTF-8 -26.77%
ru_RU.UTF-8 -35.81%
iw_IL.UTF-8 -35.34%
es_ES.UTF-8 -34.46%
hi_IN.UTF-8 -00.38%
sv_SE.UTF-8 -36.99%
hu_HU.UTF-8 -16.35%
tr_TR.UTF-8 -27.80%
is_IS.UTF-8 -33.24%
it_IT.UTF-8 -24.39%
sr_RS.UTF-8 -37.55%
ja_JP.UTF-8 +02.84%
strcoll is implemented using a cache for indices and weights of
collation sequences in the strings so that subsequent passes do not
have to search through collation data again. For very large string
inputs, the cache size computation could overflow. In such a case,
use the fallback function that does not cache indices and weights of
collation sequences.
Fixes CVE-2012-4412.
strcoll currently falls back to alloca if malloc fails, resulting in a
possible stack overflow. This patch implements sequence traversal and
comparison without caching indices and rules.
Fixes CVE-2012-4424.
2004-03-14 Ulrich Drepper <drepper@redhat.com>
Make the non-_l functions wrappers around the _l functions.
* include/monetary.h: Declare __vstrmon_l.
* include/string.h: Add libc_hidden_proto for __strcoll_l and
__strxfrm_l.
* include/time.h: Define ptime_locale_status. Declare
__strptime_internal.
* include/wchar.h: Add libc_hidden_proto for __wcscoll_l and
__wcsxfrm_l.
* stdlib/strfmon.c: Move the code to strfmon_l.c. Add little
wrapper around __vstrfmon_l.
* stdlib/strfmon_l.c: Add real implementation. Split into new
function __vstrfmon_l to allow calling it from strfmon.
* stdlib/strtod.c: Move real code to strtod_l.c and add wrapper.
* stdlib/strtod_l.c: Add real implementation.
* stdlib/strtof.c: Adjust to changed strtod.c.
* stdlib/strtof_l.c: Include strtod_l.c now.
* stdlib/strtold.c: New file.
* stdlib/strtold_l.c: Removed.
* string/strcoll.c: Move real code to strcoll_l.c: Add wrapper.
* string/strcoll_l.c: Add real implementation.
* string/strxfrm.c: Move real code to strxfrm_l.c: Add wrapper.
* string/strxfrm_l.c: Add real implementation.
* sysdeps/generic/strtol.c: Move real implementation to strtol_l.c.
Add wrappers.
* sysdeps/generic/strtol_l.c: Add real implementation.
* sysdeps/generic/strtold.c: Removed.
* sysdeps/generic/strtold_l.c: New file.
* sysdeps/generic/strtoll_l.c: Include strtol_l.c now. Adjust
#defines.
* sysdeps/generic/strtoul_l.c: Likewise.
* sysdeps/generic/strtoull_l.c: Likewise.
* sysdeps/generic/wcstol_l.c: Likewise.
* sysdeps/generic/wcstoll_l.c: Likewise.
* sysdeps/generic/wcstoul_l.c: Likewise.
* sysdeps/generic/wcstoull_l.c: Likewise.
* sysdeps/ieee754/ldbl-128/strtold.c: Removed.
* sysdeps/ieee754/ldbl-128/strtold_l.c: New file.
* sysdeps/ieee754/ldbl-96/strtold.c: Removed.
* sysdeps/ieee754/ldbl-96/strtold_l.c: New file.
* sysdeps/m68k/strtold.c: Removed.
* sysdeps/m68k/strtold_l.c: New file.
* time/strftime.c: Move real code to strftime_l.c. Add wrapper.
* time/strftime_l.c: Add real implementation.
* time/strptime.c: Move real code to strptime_l.c. Add wrapper.
* time/strptime_l.c: Add real implementation.
* time/wcsftime.c: Simplify since only wrappers are defined in
strftime.c.
* time/wcsftime_l.c: Include strftime_l.c.
* wcsmbs/wcscoll.c: Simplify since the file is not used by wcscoll_l.c
anymore.
* wcsmbs/wcscoll_l.c: Include strcoll_l.c.
* wcsmbs/wcsxfrm.c: Simplify since the file is not used by wcsxfrm_l.c
anymore.
* wcsmbs/wcsxfrm_l.c: Include strxfrm_l.c.
* wcsmbs/wcstod.c: Prepare to include new strtod.c.
* wcsmbs/wcstod_l.c: Include strtod_l.c.
* wcsmbs/wcstof.c: Prepare to include new strtof.c.
* wcsmbs/wcstof_l.c: Include strtof_l.c.
* wcsmbs/wcstold.c: Prepare to include new strtold.c.
* wcsmbs/wcstold_l.c: Include strtold_l.c.
* locale/uselocale.c: Use _NL_CURRENT_LOCALE instead of __libc_tsd_get.
* sysdeps/generic/strcasecmp.c: Optimize a bit. It's better to get
a reference to the current locale and then use the _l functions.
* sysdeps/generic/strncase.c: Likewise.
2001-07-06 Paul Eggert <eggert@twinsun.com>
* manual/argp.texi: Remove ignored LGPL copyright notice; it's
not appropriate for documentation anyway.
* manual/libc-texinfo.sh: "Library General Public License" ->
"Lesser General Public License".
2001-07-06 Andreas Jaeger <aj@suse.de>
* All files under GPL/LGPL version 2: Place under LGPL version
2.1.
