We add a new C.UTF-8 locale. This locale is not builtin to glibc, but
is provided as a distinct locale. The locale provides full support for
UTF-8 and this includes full code point sorting via STRCMP-based
collation (strcmp or wcscmp).
The collation uses a new keyword 'codepoint_collation' which drops all
collation rules and generates an empty zero rules collation to enable
STRCMP usage in collation. This ensures that we get full code point
sorting for C.UTF-8 with a minimal 1406 bytes of overhead (LC_COLLATE
structure information and ASCII collating tables).
The new locale is added to SUPPORTED. Minimal test data for specific
code points (minus those not supported by collate-test) is provided in
C.UTF-8.in, and this verifies code point sorting is working reasonably
across the range. The locale was tested manually with the full set of
code points without failure.
The locale is harmonized with locales already shipping in various
downstream distributions. A new tst-iconv9 test is added which verifies
the C.UTF-8 locale is generally usable.
Testing for fnmatch, regexec, and recomp is provided by extending
bug-regex1, bugregex19, bug-regex4, bug-regex6, transbug, tst-fnmatch,
tst-regcomp-truncated, and tst-regex to use C.UTF-8.
Tested on x86_64 or i686 without regression.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
With the increasing adoption of UTF-8, modern editors may (will?)
replace iso-8859-encoded characters in the range 0x80..0xff with
their UTF-8 equivalent, as will mailers and other tools. This breaks
our testsuite and corrupts patches.
So, this patch starts replacing these problematic characters with
\OCTal sequences instead (adding support for those in tst-fnmatch.c)
or with plain ASCII characters (PTESTS).
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Gnulib has added the proposed fix with aed23714d60 (done in 2005), but
recently with a glibc merge with 67306f6 (done in 2020 with sync back)
it has fallback to old semantic to return -1 on in case of failure.
From gnulib developer feedback it was an oversight. Although the full
fix for BZ #14185 would require to rewrite fnmatch implementation to use
mbrtowc instead of mbsrtowcs on the full input, this mitigate the issue
and it has been used by gnulib for a long time.
This patch also removes the alloca usage on the string convertion to
wide characters before calling the internal function.
Checked on x86_64-linux-gnu.
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
In commit 9479b6d5e0 we updated all of
the collation data to harmonize with the new version of ISO 14651
which is derived from Unicode 9.0.0. This collation update brought
with it some changes to locales which were not desirable by some
users, in particular it altered the meaning of the
locale-dependent-range regular expression, namely [a-z] and [A-Z], and
for en_US it caused uppercase letters to be matched by [a-z] for the
first time. The matching of uppercase letters by [a-z] is something
which is already known to users of other locales which have this
property, but this change could cause significant problems to en_US
and other similar locales that had never had this change before.
Whether this behaviour is desirable or not is contentious and GNU Awk
has this to say on the topic:
https://www.gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html
While the POSIX standard also has this further to say: "RE Bracket
Expression":
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap09.html
"The current standard leaves unspecified the behavior of a range
expression outside the POSIX locale. ... As noted above, efforts were
made to resolve the differences, but no solution has been found that
would be specific enough to allow for portable software while not
invalidating existing implementations."
In glibc we implement the requirement of ISO POSIX-2:1993 and use
collation element order (CEO) to construct the range expression, the
API internally is __collseq_table_lookup(). The fact that we use CEO
and also have 4-level weights on each collation rule means that we can
in practice reorder the collation rules in iso14651_t1_common (the new
data) to provide consistent range expression resolution *and* the
weights should maintain the expected total order. Therefore this
patch does three things:
* Reorder the collation rules for the LATIN script in
iso14651_t1_common to deinterlace uppercase and lowercase letters in
the collation element orders.
* Adds new test data en_US.UTF-8.in for sort-test.sh which exercises
strcoll* and strxfrm* and ensures the ISO 14651 collation remains.
* Add back tests to tst-fnmatch.input and tst-regexloc.c which
exercise that [a-z] does not match A or Z.
The reordering of the ISO 14651 data is done in an entirely mechanical
fashion using the following program attached to the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28
It is up for discussion if the iso14651_t1_common data should be
refined further to have 3 very tight collation element ranges that
include only a-z, A-Z, and 0-9, which would implement the solution
sought after in:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c12
and implemented here:
https://www.sourceware.org/ml/libc-alpha/2018-07/msg00854.html
No regressions on x86_64.
Verified that removal of the iso14651_t1_common change causes tst-fnmatch
to regress with:
422: fnmatch ("[a-z]", "A", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
...
425: fnmatch ("[A-Z]", "z", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
See:
http://pubs.opengroup.org/onlinepubs/7908799/xbd/re.html
> A range expression represents the set of collating elements that fall
> between two elements in the current collation sequence,
> inclusively. It is expressed as the starting point and the ending
> point separated by a hyphen (-).
>
> Range expressions must not be used in portable applications because
> their behaviour is dependent on the collating sequence. Ranges will be
> treated according to the current collating sequence, and include such
> characters that fall within the range based on that collating
> sequence, regardless of character values. This, however, means that
> the interpretation will differ depending on collating sequence. If,
> for instance, one collating sequence defines ä as a variant of a,
> while another defines it as a letter following z, then the expression
> [ä-z] is valid in the first language and invalid in the second.
Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
The new iso14651_t1_common lists upper case and lower case Latin characters
in a different order than the old one which causes surprising results
for example in the de_DE locale: [a-z] now includes A because A comes
after a in iso14651_t1_common but does not include Z because that comes
after z in iso14651_t1_common.
* posix/tst-fnmatch.input: Fix results for range expressions
for non C locales.
* posix/tst-regexloc.c: Do not use a range expression for
de_DE.ISO-8859-1 locale.
Update.
2004-09-01 Jakub Jelinek <jakub@redhat.com>
[BZ #361]
* posix/fnmatch_loop.c (FCT): For backslash between brackets, branch
to normal_bracket after fetching the next character.
* posix/tst-fnmatch.input: Add 25 new tests.
Reported by Markus Oberhumer <markus@oberhumer.com>.
2001-07-06 Paul Eggert <eggert@twinsun.com>
* manual/argp.texi: Remove ignored LGPL copyright notice; it's
not appropriate for documentation anyway.
* manual/libc-texinfo.sh: "Library General Public License" ->
"Lesser General Public License".
2001-07-06 Andreas Jaeger <aj@suse.de>
* All files under GPL/LGPL version 2: Place under LGPL version
2.1.
* posix/tst-fnmatch.input: Add test case for FNM_PERIOD handling with
FNM_EXTMATCH.
* posix/fnmatch_loop.c: Optimize handling of ?() and @().
* posix/fnmatch.c: Define STRLEN and STRCAT appropriately.
* posix/Versions [libc] (GLIBC_2.2.3): Add fnmatch.
* posix/fnmatch.c: Define with new default version GLIBC_2.2.3 to
avoid running binaries with libc versions without FNM_EXTMATCH support.
* include/wchar.h: Add prototype for __wcscat.
* wcsmbs/wcscat.c: Define __wcscat and make wcscat weak alias.
2001-03-15 Ulrich Drepper <drepper@redhat.com>
* posix/fnmatch.h (FNM_EXTMATCH): Define.
* posix/fnmatch.c (NO_LEADING_PERIOD): Define.
(posixly_correct): Move global variable here.
(INT, EXT, END): Name new functions defined in fnmatch_loop.c.
(fnmatch): Pretty printing.
* posix/fnmatch_loop.c: Add code to handle FNM_EXTMATCH.
* posix/tst-fnmatch.c: Recognize EXTMATCH flag.
* posix/tst-fnmatch.input: Add tests for extended matching.
* posix/testfnm.c: Add test for patterns with multiple ** before /.
* posix/fnmatch_loop.c: Fix problem with the test above.
2001-02-01 Ulrich Drepper <drepper@redhat.com>
* Makerules (build-shlib): Add $(extra-B-$(@F:lib%.so=%).so).
* configure.in: Test for -z initfirst linker option.
* config.make.in: Add have-z-initfirst.
* elf/dl-init.c (_dl_init): Split out actual initialization code in
new function call_init. If _dl_initfirst is non-NULL initialize first.
* elf/dl-load.c (_dl_map_from_fd): If DF_1_INITFIRST flag is set
remember object in _dl_initfirst.
* elf/soinit.c: Remove special support for calling
__pthread_initialize_minimal.
* conform/conformtest.pl: Add missing $prepend in type test.
* elf/elf.h (SHT_CHECKSUM): New definition.
* posix/tst-fnmatch.input: Add tests for [. .] in locales.
* posix/fnmatch_loop.c: Make FNM_LEADING_DIR behave as GNU tar
expects it. Patch by Colin Watson <riva.ucam.org>.
* posix/tst-fnmatch.input: Add test cases for FNM_LEADING_DIR.
2000-07-05 Ulrich Drepper <drepper@redhat.com>
* locale/loadlocale.c (_nl_unload_locale): Add cast to avoid warning.
* locale/programs/ld-collate.c (collate_output): Also write out the
collation sequence values and the wide character string for the
collation symbol table.
* posix/fnmatch.c: Include "../locale/elem-hash.h".
* posix/fnmatch_loop.c: Implement collating symbol handling.
* posix/tst-fnmatch.input: Add more tests, especially for collating
symbol handling.
* posix/regex.c: Fix comment.
