I made it to agree as much as possible with the rules from CLDR (see:
https://github.com/unicode-org/cldr/blob/main/common/collation/th.xml).
It seems to be impossible to follow the CLDR rules
&[before 1]๚<ฯ # should be "variable"
and
&๛<ๆ # should be "variable"
exactly though. These ask for a primary difference in punctuation
characters whose primary weight should be "IGNORE". But using a
secondary differnence instead still sorts the test data correctly and
the previously used collation in th_TH used tertiary differences for
these characters.
There was old localedata/th_TH.in test data in TIS-620 encoding which
was not used (it was not in the localedata/Makefile). I converted this
to UTF-8 and moved it to localedata/th_TH.UTF-8.in and added it to
localedata/Makefile.
Using the existing collation rules in the th_TH locale did not sort that
test file completely correct, I think my new collation rules based on
iso14651_t1 are better.
Unicode 15.1.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.1.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total removed characters in newly generated CHARMAP: 0
Total changed characters in newly generated CHARMAP: 0
Total added characters in newly generated CHARMAP: 627
Total removed characters in newly generated WIDTH: 0
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 627
alpha: Added 622 characters in new ctype which were not in old ctype
graph: Added 627 characters in new ctype which were not in old ctype
print: Added 627 characters in new ctype which were not in old ctype
punct: Added 5 characters in new ctype which were not in old ctype
The five characters added to punct are:
2FFC;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM RIGHT;So;0;ON;;;;;N;;;;;
2FFD;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM LOWER RIGHT;So;0;ON;;;;;N;;;;;
2FFE;IDEOGRAPHIC DESCRIPTION CHARACTER HORIZONTAL REFLECTION;So;0;ON;;;;;N;;;;;
2FFF;IDEOGRAPHIC DESCRIPTION CHARACTER ROTATION;So;0;ON;;;;;N;;;;;
31EF;IDEOGRAPHIC DESCRIPTION CHARACTER SUBTRACTION;So;0;ON;;;;;N;;;;;
The Unicode announcement blog entry says "[...] adds 627
characters, [...] additions include 622 CJK unified ideographs in
a new block, [...]", so that looks OK. The Unicode
blog mentions "six completely new emoji" but they don't appear here as
they are all sequences and not single code points.
Resolves: BZ #30854
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Add common emojis to the translit-able characters (mostly
faces and hearts), and translit them to old-fashioned
smileys.
Signed-off-by: Colin Leroy-Mira <colin@colino.net>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Unicode 15.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 4489
Total removed characters in newly generated WIDTH: 0
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 4257
alpha: Added 4389 characters in new ctype which were not in old ctype
combining: Added 42 characters in new ctype which were not in old ctype
combining_level3: Added 34 characters in new ctype which were not in old ctype
graph: Added 4489 characters in new ctype which were not in old ctype
lower: Added 73 characters in new ctype which were not in old ctype
print: Added 4489 characters in new ctype which were not in old ctype
punct: Missing 5 characters of old ctype in new ctype
punct: Missing: ఄ 0xc04 TELUGU SIGN COMBINING ANUSVARA ABOVE
punct: Missing: ྂ 0xf82 TIBETAN SIGN NYI ZLA NAA DA
punct: Missing: ྃ 0xf83 TIBETAN SIGN SNA LDAN
punct: Missing: 𑂀 0x11080 KAITHI SIGN CANDRABINDU
punct: Missing: 𑂁 0x11081 KAITHI SIGN ANUSVARA
That’s OK, because these are now Alphabetic in DerivedCoreProperties.txt
punct: Added 105 characters in new ctype which were not in old ctype
Resolves: BZ #29604
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The Z modifier is a nonstandard synonymn for z (that predates z
itself) and compiler might issue an warning for in invalid
conversion specifier.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
With LC_MONETARY parsing fixed we can now generate locales
without forcing output with '-c'.
Removing '-c' from localedef invocation is the equivalent of
using -Werror for localedef. The glibc locale sources should
always be clean and free from warnings.
We remove '-c' from both test locale generation and the targets
used for installing locales e.g. install-locale-archive, and
install-locale-files.
Tested on x86_64 and i686 without regressions.
Tested with install-locale-archive target.
Tested with install-locale-files target.
Reviewed-by: DJ Delorie <dj@redhat.com>
We have had one downstream report from Canonical [1] that
an rrdtool test was broken by the differences in LC_TIME
that we had in the non-builtin C locale (C.UTF-8). If one
application has an issue there are going to be others, and
so with this commit we review and fix all the issues that
cause the builtin C locale to be different from C.UTF-8,
which includes:
* mon_decimal_point should be empty e.g. ""
- Depends on mon_decimal_point_wc fix.
* negative_sign should be empty e.g. ""
* week should be aligned with the builtin C/POSIX locale
* d_fmt corrected with escaped slashes e.g. "%m//%d//%y"
* yesstr and nostr should be empty e.g. ""
* country_ab2 and country_ab3 should be empty e.g. ""
We bump LC_IDENTIFICATION version and adjust the date to
indicate the change in the locale.
A new tst-c-utf8-consistency test is added to ensure
consistency between C/POSIX and C.UTF-8.
Tested on x86_64 and i686 without regression.
[1] https://sourceware.org/pipermail/libc-alpha/2022-January/135703.html
Co-authored-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.
I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah. I don't
know why I run into these diagnostics whereas others evidently do not.
remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
Add the Abkhazian language in the Georgia territory
The ab_GE was just recently added to CLDR, it should be available
in CLDR v41, https://github.com/unicode-org/cldr/pull/1402
The Abkhazian language has been added to Gnome for localization
The locale has been tested on Ubuntu 20.04, Mint 20.2 and Fedora 35 Beta
Signed-off-by: Nart Tlisha <daniel.abzakh@gmail.com>
Reviewed-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
The collate-test.c triggers UB with an signed integer overflow,
which results in an error on some architectures (powerpc32).
Checked on x86_64, i686, and powerpc.
Unicode 14.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 14.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 838
Total removed characters in newly generated WIDTH: 1
(Characters not in WIDTH get width 1 by default, i.e. these have width 1 now.)
removed: <U1734> 0 : eaw=N category=Mc bidi=L name=HANUNOO SIGN PAMUDPOD
That seems intentional, the character had category Mn (Mark, nonspacing) before
and now has Mc (Mark, spacing combining)
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 175
We add a new C.UTF-8 locale. This locale is not builtin to glibc, but
is provided as a distinct locale. The locale provides full support for
UTF-8 and this includes full code point sorting via STRCMP-based
collation (strcmp or wcscmp).
The collation uses a new keyword 'codepoint_collation' which drops all
collation rules and generates an empty zero rules collation to enable
STRCMP usage in collation. This ensures that we get full code point
sorting for C.UTF-8 with a minimal 1406 bytes of overhead (LC_COLLATE
structure information and ASCII collating tables).
The new locale is added to SUPPORTED. Minimal test data for specific
code points (minus those not supported by collate-test) is provided in
C.UTF-8.in, and this verifies code point sorting is working reasonably
across the range. The locale was tested manually with the full set of
code points without failure.
The locale is harmonized with locales already shipping in various
downstream distributions. A new tst-iconv9 test is added which verifies
the C.UTF-8 locale is generally usable.
Testing for fnmatch, regexec, and recomp is provided by extending
bug-regex1, bugregex19, bug-regex4, bug-regex6, transbug, tst-fnmatch,
tst-regcomp-truncated, and tst-regex to use C.UTF-8.
Tested on x86_64 or i686 without regression.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Remove all malloc hook uses from core malloc functions and move it
into a new library libc_malloc_debug.so. With this, the hooks now no
longer have any effect on the core library.
libc_malloc_debug.so is a malloc interposer that needs to be preloaded
to get hooks functionality back so that the debugging features that
depend on the hooks, i.e. malloc-check, mcheck and mtrace work again.
Without the preloaded DSO these debugging features will be nops.
These features will be ported away from hooks in subsequent patches.
Similarly, legacy applications that need hooks functionality need to
preload libc_malloc_debug.so.
The symbols exported by libc_malloc_debug.so are maintained at exactly
the same version as libc.so.
Finally, static binaries will no longer be able to use malloc
debugging features since they cannot preload the debugging DSO.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reinstate gconv-modules as the main file so that the configuration
files in gconv-modules.d/ become add-on configuration. With this, the
effective user visible change is that GCONV_PATH can now have
supplementary configuration in GCONV_PATH/gconv-modules.d/ in addition
to the main GCONV_PATH/gconv-modules file.
Move all gconv-modules configuration files to gconv-modules.conf.
That is, the S390 extensions now become gconv-modules-s390.conf. Move
both configuration files into gconv-modules.d.
Now GCONV_PATH/gconv-modules is read only for backward compatibility
for third-party gconv modules directories.
Reviewed-by: DJ Delorie <dj@redhat.com>
This updates IBM256, IBM277, IBM278, IBM280, IBM284, IBM297, IBM424
in the same way that IBM273 was updated for bug 23290.
IBM256 and IBM424 still have holes after this change, so HAS_HOLES
is not updated.
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
The 13th edition of Svenska Akademiens ordlista lists 'W' as a
distinct letter that sorts after 'V'. We adjust the sv_SE locale
(and tests) to match this updated and "reformed" language change.
This harmonizes us with CLDR 1.5.0 (2007) for sv_SE sorting of
the letter 'W'.
No regressions on x86_64, and locale sorting tests all pass.
Co-authored-by: Carlos O'Donell <carlos@redhat.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
In 2000 when date_fmt was originally added as an extension the
en_US locale did not have a date_fmt specifier and so used the
default which resulted in the abbreviated month name coming
before the day of the month (as expected in the US and other
locales). In commit 7395f3a0ef the
date_fmt was added to en_US with a 12H time to better align with
US user expectations. Unfortunately the abbreviated month name
and day were inverted during that transition, and that was seen
as a regression and reported against Fedora 32:
https://bugzilla.redhat.com/show_bug.cgi?id=1830623
The progression of date_fmt looks like this:
"%a %b %e %H:%M:%S %Z %Y" <- Originally (2000)
"%a %d %b %Y %I:%M:%S %p %Z" <- glibc 2.29 (2019)
"%a %b %e %r %Z %Y" <- glibc 2.32 (2020) [this commit]
Note: "%r" is "%I:%M:%S %p" in en_US and so shorter to write.
Likewise the year is in the wrong place in commit
7395f3a0ef and this is corrected in
this patch.
For reference d_t_fmt:
"%a %d %b %Y %r %Z" <- d_t_fmt (1997)
Yes, d_t_fmt and date_fmt are *not* the same, this is just the
history of this locale. This commit does not change d_t_fmt to
better align with date_fmt. No users have requested we change
d_t_fmt or given any justification for such a change.
The only goals of this change are to place the abbreviated month
name before the day of the month as it has been printed since
2000, and place the year at the end. This minimizes the change
from commit 7395f3a0ef and makes
good on changing only from 24H clock to 12H clock.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
The new tst-localedef-hardlinks verifies that when compiling
two locales (with default output directory) one with
--no-hard-links and one without the option, results in the
expected behaviour. When --no-hard-links is used the link
counts on LC_CTYPE is 1, indicating that even thoug the two
locale are identical (though different named source files and
output direcotry) the localedef did not carry out the hard
link optimization. Then when --no-hard-links is omitted the
localedef hard link optimization is correctly carried out and
for 2 compiled locales the link count for LC_CTYPE is 2.
Reviewed-by: DJ Delorie <dj@redhat.com>
Unicode 13.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 13.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 5930
Total added characters in newly generated WIDTH: 5536
Confirmed by CLDR and a native speaker: "abril" is more often used even
if "abrial" is also correct. Both nominative (alt_mon) and genitive (mon)
cases are updated.
It is not specified what should be the content of d_t_fmt and date_fmt
but in the built-in C locale those fields have only one difference:
date_fmt contains "%Z" (the current time zone) while d_t_fmt does not.
For most of the locales this commit does the following operation:
copy d_t_fmt to date_fmt, and then remove "%Z" from d_t_fmt.
If "%Z" was originally missing from d_t_fmt add it to date_fmt.
It also corrects comments where necessary.
Exceptions:
* In bo_CN, dz_BT, and km_KH "%Z" has not been added to date_fmt because
it was too difficult. In these locales date_fmt has been set to the
copy of d_t_fmt.
* In en_DK "%Z" has not been removed from d_t_fmt in order to preserve
the conformance with the standard mentioned in the comment.
The command to identify and initially edit the locales that need the
update was:
for i in `grep -lw d_t_fmt *`
do
if ! grep -qw date_fmt $i ; then
awk '/d_t_fmt/ { print $0; gsub("d_t_fmt", "date_fmt"); } //{ print $0 }' < $i > $i.next
mv $i.next $i
fi
done
and then each file was further edited manually.
Currently d_t_fmt formats time as "plkst. %H un %M". A quick Google
search says that "plkst." means "o’clock" and "un" means "and".
Also this format does not display seconds.
CLDR does not mention anything like that. We have no reason to use
anything different than "%H:%M:%S".
