Unicode 16.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 16.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Changes in CHARMAP and WIDTH:
Total added characters in newly generated CHARMAP: 5185
Total removed characters in newly generated WIDTH: 1
Total added characters in newly generated WIDTH: 170
The removed character from WIDTH is U+1171E AHOM CONSONANT SIGN MEDIAL RA.
It changed like this:
UnicodeData.txt 15.1.0: 1171E;AHOM CONSONANT SIGN MEDIAL RA;Mn;0;NSM;;;;;N;;;;;
UnicodeData.txt 16.0.0: 1171E;AHOM CONSONANT SIGN MEDIAL RA;Mc;0;L;;;;;N;;;;;
EastAsianWidth.txt 15.1.0: 1171D..1171F ; N # Mn [3] AHOM CONSONANT SIGN MEDIAL LA..AHOM CONSONANT SIGN MEDIAL LIGATING RA
EastAsianWidth.txt 16.0.0: 1171E ; N # Mc AHOM CONSONANT SIGN MEDIAL RA
I.e it changed from Mn (Mark Nonspacing) to Mc (Mark Spacing
combining). So it should now have width 1 instead of 0, therefore it
is OK that it was removed from WIDTH, characters not in WIDTH get
width 1 by default.
Nothing suspicious when browsing the list of the 170 added characters.
Changes in ctype:
alpha: Added 4452 characters in new ctype which were not in old ctype
combining: Added 51 characters in new ctype which were not in old ctype
combining_level3: Added 43 characters in new ctype which were not in old ctype
graph: Added 5185 characters in new ctype which were not in old ctype
lower: Added 25 characters in new ctype which were not in old ctype
print: Added 5185 characters in new ctype which were not in old ctype
punct: Missing 33 characters of old ctype in new ctype
punct: Added 766 characters in new ctype which were not in old ctype
tolower: Added 27 characters in new ctype which were not in old ctype
totitle: Added 27 characters in new ctype which were not in old ctype
toupper: Added 27 characters in new ctype which were not in old ctype
upper: Added 27 characters in new ctype which were not in old ctype
Nothing suspicous in the additions.
About the 33 characters removed from `punct`:
U+0363 - U+036F are identical in UnicodeData.txt. Difference in DerivedCoreProperties.txt:
DerivedCoreProperties.txt 15.1.0: not there.
DerivedCoreProperties.txt 16.0.0: 0363..036F ; Alphabetic # Mn [13] COMBINING LATIN SMALL LETTER A..COMBINING LATIN SMALL LETTER X
So that’s the reason why they are added to `alpha` and removed from `punct`.
Same for U+1DD3 - U+1DE6, they are identical in UnicodeData.txt but there is a difference in DerivedCoreProperties.txt:
DerivedCoreProperties.txt 15.1.0: 1DE7..1DF4 ; Alphabetic # Mn [14] COMBINING LATIN SMALL LETTER ALPHA..COMBINING LATIN SMALL LETTER U WITH DIAERESIS
DerivedCoreProperties.txt 16.0.0: 1DD3..1DF4 ; Alphabetic # Mn [34] COMBINING LATIN SMALL LETTER FLATTENED OPEN A ABOVE..COMBINING LATIN SMALL LETTER U WITH DIAERESIS
So they became `Alphabetic` and were thus added to `alpha` and removed from `punct`.
Resolves: BZ #32168
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Unicode 15.1.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.1.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total removed characters in newly generated CHARMAP: 0
Total changed characters in newly generated CHARMAP: 0
Total added characters in newly generated CHARMAP: 627
Total removed characters in newly generated WIDTH: 0
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 627
alpha: Added 622 characters in new ctype which were not in old ctype
graph: Added 627 characters in new ctype which were not in old ctype
print: Added 627 characters in new ctype which were not in old ctype
punct: Added 5 characters in new ctype which were not in old ctype
The five characters added to punct are:
2FFC;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM RIGHT;So;0;ON;;;;;N;;;;;
2FFD;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM LOWER RIGHT;So;0;ON;;;;;N;;;;;
2FFE;IDEOGRAPHIC DESCRIPTION CHARACTER HORIZONTAL REFLECTION;So;0;ON;;;;;N;;;;;
2FFF;IDEOGRAPHIC DESCRIPTION CHARACTER ROTATION;So;0;ON;;;;;N;;;;;
31EF;IDEOGRAPHIC DESCRIPTION CHARACTER SUBTRACTION;So;0;ON;;;;;N;;;;;
The Unicode announcement blog entry says "[...] adds 627
characters, [...] additions include 622 CJK unified ideographs in
a new block, [...]", so that looks OK. The Unicode
blog mentions "six completely new emoji" but they don't appear here as
they are all sequences and not single code points.
Resolves: BZ #30854
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Unicode 15.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 4489
Total removed characters in newly generated WIDTH: 0
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 4257
alpha: Added 4389 characters in new ctype which were not in old ctype
combining: Added 42 characters in new ctype which were not in old ctype
combining_level3: Added 34 characters in new ctype which were not in old ctype
graph: Added 4489 characters in new ctype which were not in old ctype
lower: Added 73 characters in new ctype which were not in old ctype
print: Added 4489 characters in new ctype which were not in old ctype
punct: Missing 5 characters of old ctype in new ctype
punct: Missing: ఄ 0xc04 TELUGU SIGN COMBINING ANUSVARA ABOVE
punct: Missing: ྂ 0xf82 TIBETAN SIGN NYI ZLA NAA DA
punct: Missing: ྃ 0xf83 TIBETAN SIGN SNA LDAN
punct: Missing: 𑂀 0x11080 KAITHI SIGN CANDRABINDU
punct: Missing: 𑂁 0x11081 KAITHI SIGN ANUSVARA
That’s OK, because these are now Alphabetic in DerivedCoreProperties.txt
punct: Added 105 characters in new ctype which were not in old ctype
Resolves: BZ #29604
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Unicode 14.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 14.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 838
Total removed characters in newly generated WIDTH: 1
(Characters not in WIDTH get width 1 by default, i.e. these have width 1 now.)
removed: <U1734> 0 : eaw=N category=Mc bidi=L name=HANUNOO SIGN PAMUDPOD
That seems intentional, the character had category Mn (Mark, nonspacing) before
and now has Mc (Mark, spacing combining)
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 175
Unicode 13.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 13.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Total added characters in newly generated CHARMAP: 5930
Total added characters in newly generated WIDTH: 5536
Unicode 12.1.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 12.1.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Some info about the number of characters added or changed:
Total added characters in newly generated CHARMAP: 1
added: <U32FF> /xe3/x8b/xbf SQUARE ERA NAME REIWA
Total added characters in newly generated WIDTH: 1
added: <U32FF> 2 : eaw=W category=So bidi=L name=SQUARE ERA NAME REIWA
graph: Added 1 characters in new ctype which were not in old ctype
graph: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
print: Added 1 characters in new ctype which were not in old ctype
print: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
punct: Added 1 characters in new ctype which were not in old ctype
punct: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
Unicode 12.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 12.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Some info about the number of characters added or changed:
Total added characters in newly generated CHARMAP: 554
Total added characters in newly generated WIDTH: 106
alpha: Missing 8 characters of old ctype in new ctype
(These are combining marks, apparently they were removed from alpha
on purpose)
alpha: Added 295 characters in new ctype which were not in old ctype
combining: Missing 2 characters of old ctype in new ctype
(U+1CF2 VEDIC SIGN ARDHAVISARGA and U+1CF3 VEDIC SIGN ROTATED ARDHAVISARGA,
these are now "Alphabetic" in Unicode 12.0.0)
combining: Added 37 characters in new ctype which were not in old ctype
combining_level3: Missing 2 characters of old ctype in new ctype
(U+1CF2 VEDIC SIGN ARDHAVISARGA and U+1CF3 VEDIC SIGN ROTATED ARDHAVISARGA,
these are now "Alphabetic" in Unicode 12.0.0)
combining_level3: Added 26 characters in new ctype which were not in old ctype
graph: Added 554 characters in new ctype which were not in old ctype
lower: Added 6 characters in new ctype which were not in old ctype
print: Added 554 characters in new ctype which were not in old ctype
punct: Missing 29 characters of old ctype in new ctype
(These characters have all become "Alphabetic" in Unicode 12.0.0.
Therefore, they are not in "punct" anymore (see: is_punct() in unicode_utils.py))
punct: Added 296 characters in new ctype which were not in old ctype
tolower: Added 7 characters in new ctype which were not in old ctype
totitle: Added 7 characters in new ctype which were not in old ctype
toupper: Added 7 characters in new ctype which were not in old ctype
upper: Added 7 characters in new ctype which were not in old ctype
[BZ #24307]
* localedata/unicode-gen/Makefile (UNICODE_VERSION): Set to 12.0.0.
* localedata/unicode-gen/DerivedCoreProperties.txt: Update to Unicode 12.0.0.
* localedata/unicode-gen/EastAsianWidth.txt: Likewise.
* localedata/unicode-gen/PropList.txt: Likewise.
* localedata/unicode-gen/UnicodeData.txt: Likewise.
* localedata/unicode-gen/ctype_compatibility_test_cases.py: U+108D became
"Alphabetic" in Unicode 12.0.0. Adapt test case.
* localedata/charmaps/UTF-8: Regenerate.
* localedata/locales/i18n_ctype: Likewise.
* localedata/locales/tr_TR: Likewise.
* localedata/locales/translit_circle: Likewise.
* localedata/locales/translit_cjk_compat: Likewise.
* localedata/locales/translit_combining: Likewise.
* localedata/locales/translit_compat: Likewise.
* localedata/locales/translit_font: Likewise.
* localedata/locales/translit_fraction: Likewise.
Unicode 11.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 11.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).
Some info about the number of characters added:
Total added characters in newly generated CHARMAP: 684
Total added characters in newly generated WIDTH: 119
alpha: Added 380 characters in new ctype which were not in old ctype
combining: Added 56 characters in new ctype which were not in old ctype
combining_level3: Added 37 characters in new ctype which were not in old ctype
graph: Added 684 characters in new ctype which were not in old ctype
lower: Added 82 characters in new ctype which were not in old ctype
print: Added 684 characters in new ctype which were not in old ctype
punct: Added 304 characters in new ctype which were not in old ctype
tolower: Added 79 characters in new ctype which were not in old ctype
totitle: Added 33 characters in new ctype which were not in old ctype
toupper: Added 79 characters in new ctype which were not in old ctype
upper: Added 79 characters in new ctype which were not in old ctype
No characters were removed.
[BZ #23308]
* unicode-gen/Makefile (UNICODE_VERSION): Set to 11.0.0.
* localedata/unicode-gen/DerivedCoreProperties.txt: Update to Unicode 11.0.0.
* localedata/unicode-gen/EastAsianWidth.txt: likewise.
* localedata/unicode-gen/PropList.txt: likewise.
* localedata/unicode-gen/UnicodeData.txt: likewise.
* localedata/charmaps/UTF-8: Regenerate.
* localedata/locales/i18n_ctype: likewise.
* localedata/locales/tr_TR: likewise.
* localedata/locales/translit_circle: likewise.
* localedata/locales/translit_cjk_compat: likewise.
* localedata/locales/translit_combining: likewise.
* localedata/locales/translit_compat: likewise.
* localedata/locales/translit_font: likewise.
* localedata/locales/translit_fraction: likewise.
After the transition to generating a distinct file for Unicode ctype
information e.g. i18n_ctype, the check target was left with the wrong
target name. This patch fixes the check target and regenerates the
files with more information than previously used, filling in the the
LC_IDENTIFICATION data.
Tested on x86_64 by regenerating from Unicode source files, and
running checks. Tested by subsequently rebuilding all locales.
No regressions in testsuite.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Reported-by: Rafal Luzynski <digitalfreak@lingonborough.com>
* Unicode 10.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 10.0.0, using
generator scripts contributed by Mike FABIAN (Red Hat).
* Unicode 9.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 9.0.0, using
generator scripts contributed by Mike FABIAN (Red Hat).
Update __STDC_ISO_10646__ to 201505L for Unicode 8.0.0.
Update character encoding, ctype, and transliteration tables.
New scripts autogenerate transliteration tables.
* iconvdata/JISX0213.TXT: New file.
* iconvdata/jisx0213.h: New file.
* iconvdata/jisx0213.c: New file.
* iconvdata/euc-jisx0213.c: New file.
* iconvdata/shift_jisx0213.c: New file.
* iconvdata/gconv-modules (EUC-JISX0213, SHIFT_JISX0213): New modules.
* iconvdata/EUC-JISX0213.precomposed: New file.
* iconvdata/SHIFT_JISX0213.precomposed: New file.
* iconvdata/SHIFT_JISX0213.irreversible: New file.
* iconvdata/tst-table-to.c (main): Make it work for encodings for
which the "to" direction is stateful.
* iconvdata/tst-tables.sh: Add EUC-JISX0213, SHIFT_JISX0213.
* iconvdata/Makefile (modules): Add libJISX0213, EUC-JISX0213,
SHIFT_JISX0213.
(libJISX0213-routines): New variable.
(LDFLAGS-EUC-JISX0213.so, LDFLAGS-SHIFT_JISX0213.so): New variables.
(EUC-JISX0213.so, SHIFT_JISX0213.so): Depend on libJISX0213.so.
(LDFLAGS-libJISX0213.so): New variable.
(distribute): Add JISX0213.TXT, EUC-JISX0213.precomposed,
SHIFT_JISX0213.precomposed, SHIFT_JISX0213.irreversible,
jisx0213.c, jisx0213.h, euc-jisx0213.c, shift_jisx0213.c.
2002-04-15 Bruno Haible <bruno@clisp.org>
* iconvdata/big5hkscs.c (big5hkscs_to_ucs): Change element type to
uint32_t. Update to Unicode 3.2.
(from_ucs4, from_ucs4_idx): Update to Unicode 3.2.
(BODY for TO_LOOP): Handle the ASCII range specially.
* iconvdata/BIG5HKSCS.irreversible: Add one more entry.
* iconvdata/testdata/BIG5HKSCS: Remove a character.
* iconvdata/testdata/BIG5HKSCS..UTF-8: Regenerated.
2002-04-15 Bruno Haible <bruno@clisp.org>
* iconvdata/gb18030.c (__twobyte_to_ucs, __fourbyte_to_ucs,
__ucs_to_gb18030_tab1, __ucs_to_gb18030_tab2): Update to Unicode 3.2.
2002-04-15 Bruno Haible <bruno@clisp.org>
* iconvdata/cns11643.h (__cns11643_from_ucs4p2c_tab): New declaration.
(ucs4_to_cns11643): Update.
* iconvdata/cns11643.c: Update to Unicode 3.2.
(__cns11643l2*_to_ucs4_tab, __cns11643_from_ucs4p0_tab,
__cns11643_from_ucs4p2_tab): Regenerated.
(__cns11643_from_ucs4p2c_tab): New array.
* iconvdata/EUC-TW.irreversible: Remove two entries.
2002-04-15 Bruno Haible <bruno@clisp.org>
* locale/C-translit.h.in: Update to Unicode 3.2.
Add <U2047>, <U205F>..<U2063>, <U2145>..<U2149>, <U2A74>..<U2A76>,
<U30A0>, <U3251>..<U325F>, <U32B1>..<U32BF>, <UFE00>..<UFE0F>.
2002-04-15 Bruno Haible <bruno@clisp.org>
* posix/runtests.c (main): Don't use exit() to avoid warning with
broken compilers.
* string/bug-strncat1.c: Undefine __USE_STRING_INLINES before
defining it again to avoid warning.
* string/bug-strpbrk1.c: Likewise.
* string/bug-strspn1.c: Likewise.
2000-10-29 Bruno Haible <haible@clisp.cons.org>
* locale/C-translit.h.in: Include more rules from translit_neutral.
Remove a few German and Danish specific rules.
* locale/gen-translit.pl: Avoid an error when printing a string
containing a '%' character.
* locale/C-translit.h: Regenerated.
* intl/translit.po: Choose different translations.
* intl/tst-translit.c: Update.
2000-10-29 Bruno Haible <haible@clisp.cons.org>
* locale/programs/ld-ctype.c (translit_include_t): New type.
(locale_ctype_t): Remove translit_copy_locale, translit_copy_repertoire
fields, add translit_include field.
(ctype_read): After seeing "translit_start" in ignore_content mode,
skip the entire translit section, not only one line. Remove dead test
for tok_translit_end. Enqueue include statements into a list.
Signal error if tok_eof seen before tok_translit_end.
(translit_flatten): New function.
(allocate_arrays): Call translit_flatten.
2000-10-30 Ulrich Drepper <drepper@redhat.com>
