Commit Graph

44 Commits

Author SHA1 Message Date
Carlos O'Donell
cae9944a6c Fix whitespace related license issues.
Several copies of the licenses in files contained whitespace related
problems.  Two cases are addressed here, the first is two spaces
after a period which appears between "PURPOSE." and "See". The other
is a space after the last forward slash in the URL. Both issues are
corrected and the licenses now match the official textual description
of the license (and the other license in the sources).

Since these whitespaces changes do not alter the paragraph structure of
the license, nor create new sentences, they do not change the license.
2024-10-07 18:08:16 -04:00
Mike FABIAN
a7b5eb821d Update to Unicode 16.0.0 [BZ #32168]
Unicode 16.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 16.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Changes in CHARMAP and WIDTH:

    Total added characters in newly generated CHARMAP: 5185
    Total removed characters in newly generated WIDTH: 1
    Total added characters in newly generated WIDTH: 170

The removed character from WIDTH is U+1171E AHOM CONSONANT SIGN MEDIAL RA.
It changed like this:

UnicodeData.txt 15.1.0: 1171E;AHOM CONSONANT SIGN MEDIAL RA;Mn;0;NSM;;;;;N;;;;;
UnicodeData.txt 16.0.0: 1171E;AHOM CONSONANT SIGN MEDIAL RA;Mc;0;L;;;;;N;;;;;

EastAsianWidth.txt 15.1.0: 1171D..1171F   ; N  # Mn     [3] AHOM CONSONANT SIGN MEDIAL LA..AHOM CONSONANT SIGN MEDIAL LIGATING RA
EastAsianWidth.txt 16.0.0: 1171E          ; N  # Mc         AHOM CONSONANT SIGN MEDIAL RA

I.e it changed from Mn (Mark Nonspacing) to Mc (Mark Spacing
combining). So it should now have width 1 instead of 0, therefore it
is OK that it was removed from WIDTH, characters not in WIDTH get
width 1 by default.

Nothing suspicious when browsing the list of the 170 added characters.

Changes in ctype:

    alpha: Added 4452 characters in new ctype which were not in old ctype
    combining: Added 51 characters in new ctype which were not in old ctype
    combining_level3: Added 43 characters in new ctype which were not in old ctype
    graph: Added 5185 characters in new ctype which were not in old ctype
    lower: Added 25 characters in new ctype which were not in old ctype
    print: Added 5185 characters in new ctype which were not in old ctype
    punct: Missing 33 characters of old ctype in new ctype
    punct: Added 766 characters in new ctype which were not in old ctype
    tolower: Added 27 characters in new ctype which were not in old ctype
    totitle: Added 27 characters in new ctype which were not in old ctype
    toupper: Added 27 characters in new ctype which were not in old ctype
    upper: Added 27 characters in new ctype which were not in old ctype

Nothing suspicous in the additions.

About the 33 characters removed from `punct`:

U+0363 - U+036F are identical in UnicodeData.txt. Difference in DerivedCoreProperties.txt:

DerivedCoreProperties.txt 15.1.0: not there.
DerivedCoreProperties.txt 16.0.0: 0363..036F    ; Alphabetic # Mn  [13] COMBINING LATIN SMALL LETTER A..COMBINING LATIN SMALL LETTER X

So that’s the reason why they are added to `alpha` and removed from `punct`.

Same for U+1DD3 - U+1DE6, they are identical in UnicodeData.txt but there is a difference in DerivedCoreProperties.txt:

DerivedCoreProperties.txt 15.1.0: 1DE7..1DF4    ; Alphabetic # Mn  [14] COMBINING LATIN SMALL LETTER ALPHA..COMBINING LATIN SMALL LETTER U WITH DIAERESIS
DerivedCoreProperties.txt 16.0.0: 1DD3..1DF4    ; Alphabetic # Mn  [34] COMBINING LATIN SMALL LETTER FLATTENED OPEN A ABOVE..COMBINING LATIN SMALL LETTER U WITH DIAERESIS

So they became `Alphabetic` and were thus added to `alpha` and removed from `punct`.

Resolves: BZ #32168

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2024-09-27 14:43:38 +02:00
Jules Bertholet
25c9c3789e localedata: Fix several issues with the set of characters considered 0-width [BZ #31370]
= `Default_Ignorable_Code_Point`s should have width 0 =

Unicode specifies (https://www.unicode.org/faq/unsup_char.html#3) that characters
with the `Default_Ignorable_Code_Point` property

> should be rendered as completely invisible (and non advancing, i.e. “zero width”),
if not explicitly supported in rendering.

Hence, `wcwidth()` should give them all a width of 0, with two exceptions:

- the soft hyphen (U+00AD SOFT HYPHEN) is assigned width 1 by longstanding precedent
- U+115F HANGUL CHOSEONG FILLER needs a carveout
  due to the unique behavior of the conjoining Korean jamo characters.
  One composed Hangul "syllable block" like 퓛
  is made up of two to three individual component characters, or "jamo".
  These are all assigned an `East_Asian_Width` of `Wide`
  by Unicode, which would normally mean they would all be assigned
  width 2 by glibc; a combination of (leading choseong jamo) +
  (medial jungseong jamo) + (trailing jongseong jamo) would then have width 2 + 2 + 2 = 6.
  However, glibc (and other wcwidth implementations) special-cases jungseong and jongseong,
  assigning them all width 0,
  to ensure that the complete block has width 2 + 0 + 0 = 2 as it should.
  U+115F is meant for use in syllable blocks
  that are intentionally missing a leading jamo;
  it must be assigned a width of 2 even though it has no visible display
  to ensure that the complete block has width 2.

However, `wcwidth()` currently (before this patch)
incorrectly assigns non-zero width to
U+3164 HANGUL FILLER and U+FFA0 HALFWIDTH HANGUL FILLER;
this commit fixes that.

Unicode spec references:
- Hangul:  §3.12 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G24646 and
  §18.6 https://www.unicode.org/versions/Unicode15.0.0/ch18.pdf#G31028
- `Default_Ignorable_Code_Point`: §5.21 https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095.

= Non-`Default_Ignorable_Code_Point` format controls should be visible =

The Unicode Standard, §5.21 - Characters Ignored for Display
(https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095)
says the following:

> A small number of format characters (General_Category = Cf )
> are also not given the Default_Ignorable_Code_Point property.
> This may surprise implementers, who often assume
> that all format characters are generally ignored in fallback display.
> The exact list of these exceptional format characters
> can be found in the Unicode Character Database.
> There are, however, three important sets of such format characters to note:
>
> - prepended concatenation marks
> - interlinear annotation characters
> - Egyptian hieroglyph format controls
>
> The prepended concatenation marks always have a visible display.
> See “Prepended Concatenation Marks” in [*Section 23.2, Layout Controls*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.35858.HeadingBreak.132.Layout.Controls)
> for more discussion of the use and display of these signs.
>
> The other two notable sets of format characters that exceptionally are not ignored
> in fallback display consist of the interlinear annotation characters,
> U+FFF9 INTERLINEAR ANNOTATION ANCHOR through
> U+FFFB INTERLINEAR ANNOTATION TERMINATOR,
> and the Egyptian hieroglyph format controls,
> U+13430 EGYPTIAN HIEROGLYPH VERTICAL JOINER through
> U+1343F EGYPTIAN HIEROGLYPH END WALLED ENCLOSURE.
> These characters should have a visible glyph display for fallback rendering,
> because if they are not displayed,
> it is too easy to misread the resulting displayed text.
> See “Annotation Characters” in [*Section 23.8, Specials*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.21335.Heading.133.Specials),
> as well as [*Section 11.4, Egyptian Hieroglyphs*](https://www.unicode.org/versions/Unicode15.1.0/ch11.pdf#M9.73291.Heading.1418.Egyptian.Hieroglyphs)
> for more discussion of the use and display of these characters.

glibc currently correctly assigns non-zero width to the prepended concatenation marks,
but it incorrectly gives zero width to the interlinear annotation characters
(which a generic terminal cannot interpret)
and the Egyptian hieroglyph format controls
(which are not widely supported in rendering implementations at present).
This commit fixes both these issues as well.

= Derive Hangul syllable type from Unicode data =

Previosuly, the jungseong and jongseong jamo ranges
were hard-coded into the script. With this commit, they are instead parsed
from the HangulSyllableType.txt data file published by Unicode.
This does not affect the end result.

Signed-off-by: Jules Bertholet <julesbertholet@quoi.xyz>
2024-05-15 14:31:06 +02:00
Carlos O'Donell
db50990ddb Update SHARED-FILES and license for Unicode 15.1.0.
In 2018 the license changed to use Unicode-3.0 license.
The Unicode License is a permissive MIT type of license.
Automation is updated to fetch the correct license file to
keep it in sync with the data files.

The new license is OSI approved and has an SPDX identifer:
https://opensource.org/license/unicode-license-v3
https://spdx.org/licenses/Unicode-3.0.html

The FSF and the GNU Project have been contacted to update
the license list for this license:
https://www.gnu.org/licenses/license-list.html
2024-02-27 08:27:14 -05:00
Mike FABIAN
064c708c78 localedata/unicode-gen/utf8_gen.py: fix Hangul syllable name
Resolves: BZ # 29506
2024-01-14 11:42:28 +01:00
Mike FABIAN
d333a2e0fb localedata: unicode-gen: Remove redundant \s* from regexp, fix comments 2024-01-08 10:06:42 +01:00
Paul Eggert
dff8da6b3e Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
Mike FABIAN
bb5bbc2070 Update to Unicode 15.1.0 [BZ #30854]
Unicode 15.1.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.1.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

    Total removed characters in newly generated CHARMAP: 0
    Total changed characters in newly generated CHARMAP: 0
    Total added characters in newly generated CHARMAP: 627
    Total removed characters in newly generated WIDTH: 0
    Total changed characters in newly generated WIDTH: 0
    Total added characters in newly generated WIDTH: 627

    alpha: Added 622 characters in new ctype which were not in old ctype
    graph: Added 627 characters in new ctype which were not in old ctype
    print: Added 627 characters in new ctype which were not in old ctype
    punct: Added 5 characters in new ctype which were not in old ctype
        The five characters added to punct are:
        2FFC;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM RIGHT;So;0;ON;;;;;N;;;;;
        2FFD;IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM LOWER RIGHT;So;0;ON;;;;;N;;;;;
        2FFE;IDEOGRAPHIC DESCRIPTION CHARACTER HORIZONTAL REFLECTION;So;0;ON;;;;;N;;;;;
        2FFF;IDEOGRAPHIC DESCRIPTION CHARACTER ROTATION;So;0;ON;;;;;N;;;;;
        31EF;IDEOGRAPHIC DESCRIPTION CHARACTER SUBTRACTION;So;0;ON;;;;;N;;;;;

    The Unicode announcement blog entry says "[...] adds 627
    characters, [...] additions include 622 CJK unified ideographs in
    a new block, [...]", so that looks OK. The Unicode
    blog mentions "six completely new emoji" but they don't appear here as
    they are all sequences and not single code points.

Resolves: BZ #30854

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2023-09-16 08:37:03 +02:00
Mike FABIAN
71de3aead9 localedata/unicode-gen/utf8_gen.py: adapt regexp to get relevant lines from EastAsianWidth.txt
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2023-09-16 08:37:02 +02:00
Mike FABIAN
ba017b4f9d Fix regexp syntax warnings in localedata/unicode-gen/ctype_compatibility.py
Fix these:

$ python -m py_compile ./ctype_compatibility.py
./ctype_compatibility.py:146: SyntaxWarning: invalid escape sequence '\)'

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2023-09-16 08:37:02 +02:00
Joseph Myers
6d7e8eda9b Update copyright dates with scripts/update-copyrights 2023-01-06 21:14:39 +00:00
Mike FABIAN
7fe6734d28 Update to Unicode 15.0.0 [BZ #29604]
Unicode 15.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 15.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

    Total added characters in newly generated CHARMAP: 4489
    Total removed characters in newly generated WIDTH: 0
    Total changed characters in newly generated WIDTH: 0
    Total added characters in newly generated WIDTH: 4257

    alpha: Added 4389 characters in new ctype which were not in old ctype
    combining: Added 42 characters in new ctype which were not in old ctype
    combining_level3: Added 34 characters in new ctype which were not in old ctype
    graph: Added 4489 characters in new ctype which were not in old ctype
    lower: Added 73 characters in new ctype which were not in old ctype
    print: Added 4489 characters in new ctype which were not in old ctype
    punct: Missing 5 characters of old ctype in new ctype
        punct: Missing: ఄ 0xc04 TELUGU SIGN COMBINING ANUSVARA ABOVE
        punct: Missing: ྂ 0xf82 TIBETAN SIGN NYI ZLA NAA DA
        punct: Missing: ྃ 0xf83 TIBETAN SIGN SNA LDAN
        punct: Missing: 𑂀 0x11080 KAITHI SIGN CANDRABINDU
        punct: Missing: 𑂁 0x11081 KAITHI SIGN ANUSVARA
            That’s OK, because these are now Alphabetic in DerivedCoreProperties.txt
    punct: Added 105 characters in new ctype which were not in old ctype

Resolves: BZ #29604
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2022-10-06 08:58:33 +02:00
Paul Eggert
581c785bf3 Update copyright dates with scripts/update-copyrights
I used these shell commands:

../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")

and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.

I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah.  I don't
know why I run into these diagnostics whereas others evidently do not.

remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
2022-01-01 11:40:24 -08:00
Mike FABIAN
b517256015 Update to Unicode 14.0.0 [BZ #28390]
Unicode 14.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 14.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Total added characters in newly generated CHARMAP: 838
Total removed characters in newly generated WIDTH: 1
    (Characters not in WIDTH get width 1 by default, i.e. these have width 1 now.)
    removed: <U1734> 0 : eaw=N category=Mc bidi=L   name=HANUNOO SIGN PAMUDPOD
    That seems intentional, the character had category Mn (Mark, nonspacing) before
    and now has Mc (Mark, spacing combining)
Total changed characters in newly generated WIDTH: 0
Total added characters in newly generated WIDTH: 175
2021-10-04 08:54:27 +02:00
Siddhesh Poyarekar
30891f35fa Remove "Contributed by" lines
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date.  Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.

Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions.  These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.

The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively.  These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:

https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dc
https://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2021-09-03 22:06:44 +05:30
Paul Eggert
2b778ceb40 Update copyright dates with scripts/update-copyrights
I used these shell commands:

../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")

and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
2021-01-02 12:17:34 -08:00
Mike FABIAN
6e540caa21 Set width of JUNGSEONG/JONGSEONG characters from UD7B0 to UD7FB to 0 [BZ #26120]
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2020-06-26 09:54:43 +02:00
Mike FABIAN
8645f62469 Bug 25819: Update to Unicode 13.0.0
Unicode 13.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 13.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Total added characters in newly generated CHARMAP: 5930
Total added characters in newly generated WIDTH: 5536
2020-04-21 18:17:23 +02:00
Joseph Myers
d614a75396 Update copyright dates with scripts/update-copyrights. 2020-01-01 00:14:33 +00:00
Paul Eggert
5a82c74822 Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:

sed -ri '
  s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
  s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
  $(find $(git ls-files) -prune -type f \
      ! -name '*.po' \
      ! -name 'ChangeLog*' \
      ! -path COPYING ! -path COPYING.LIB \
      ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
      ! -path manual/texinfo.tex ! -path scripts/config.guess \
      ! -path scripts/config.sub ! -path scripts/install-sh \
      ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
      ! -path INSTALL ! -path  locale/programs/charmap-kw.h \
      ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
      ! '(' -name configure \
            -execdir test -f configure.ac -o -f configure.in ';' ')' \
      ! '(' -name preconfigure \
            -execdir test -f preconfigure.ac ';' ')' \
      -print)

and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:

  chmod a+x sysdeps/unix/sysv/linux/riscv/configure
  # Omit irrelevant whitespace and comment-only changes,
  # perhaps from a slightly-different Autoconf version.
  git checkout -f \
    sysdeps/csky/configure \
    sysdeps/hppa/configure \
    sysdeps/riscv/configure \
    sysdeps/unix/sysv/linux/csky/configure
  # Omit changes that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
  git checkout -f \
    sysdeps/powerpc/powerpc64/ppc-mcount.S \
    sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
  # Omit change that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
  git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 02:43:31 -07:00
Mike FABIAN
f6efec90c8 Bug 24535: Update to Unicode 12.1.0
Unicode 12.1.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 12.1.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Some info about the number of characters added or changed:

Total added characters in newly generated CHARMAP: 1
added: <U32FF>     /xe3/x8b/xbf SQUARE ERA NAME REIWA
Total added characters in newly generated WIDTH: 1
added: <U32FF> 2 : eaw=W category=So bidi=L   name=SQUARE ERA NAME REIWA
graph: Added 1 characters in new ctype which were not in old ctype
graph: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
print: Added 1 characters in new ctype which were not in old ctype
print: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
punct: Added 1 characters in new ctype which were not in old ctype
punct: Added: ㋿ U+32FF SQUARE ERA NAME REIWA
2019-05-13 17:25:03 +02:00
Mike FABIAN
86bdd49d93 Bug 24307: Update to Unicode 12.0.0
Unicode 12.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 12.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Some info about the number of characters added or changed:

Total added characters in newly generated CHARMAP: 554
Total added characters in newly generated WIDTH: 106
alpha: Missing 8 characters of old ctype in new ctype
       (These are combining marks, apparently they were removed from alpha
       on purpose)
alpha: Added 295 characters in new ctype which were not in old ctype
combining: Missing 2 characters of old ctype in new ctype
       (U+1CF2 VEDIC SIGN ARDHAVISARGA and U+1CF3 VEDIC SIGN ROTATED ARDHAVISARGA,
       these are now "Alphabetic" in Unicode 12.0.0)
combining: Added 37 characters in new ctype which were not in old ctype
combining_level3: Missing 2 characters of old ctype in new ctype
       (U+1CF2 VEDIC SIGN ARDHAVISARGA and U+1CF3 VEDIC SIGN ROTATED ARDHAVISARGA,
       these are now "Alphabetic" in Unicode 12.0.0)
combining_level3: Added 26 characters in new ctype which were not in old ctype
graph: Added 554 characters in new ctype which were not in old ctype
lower: Added 6 characters in new ctype which were not in old ctype
print: Added 554 characters in new ctype which were not in old ctype
punct: Missing 29 characters of old ctype in new ctype
       (These characters have all  become "Alphabetic" in Unicode 12.0.0.
       Therefore, they are not in "punct" anymore (see: is_punct() in unicode_utils.py))
punct: Added 296 characters in new ctype which were not in old ctype
tolower: Added 7 characters in new ctype which were not in old ctype
totitle: Added 7 characters in new ctype which were not in old ctype
toupper: Added 7 characters in new ctype which were not in old ctype
upper: Added 7 characters in new ctype which were not in old ctype

	[BZ #24307]
	* localedata/unicode-gen/Makefile (UNICODE_VERSION): Set to 12.0.0.
	* localedata/unicode-gen/DerivedCoreProperties.txt: Update to Unicode 12.0.0.
	* localedata/unicode-gen/EastAsianWidth.txt: Likewise.
	* localedata/unicode-gen/PropList.txt: Likewise.
	* localedata/unicode-gen/UnicodeData.txt: Likewise.
	* localedata/unicode-gen/ctype_compatibility_test_cases.py: U+108D became
        "Alphabetic" in Unicode 12.0.0. Adapt test case.
	* localedata/charmaps/UTF-8: Regenerate.
	* localedata/locales/i18n_ctype: Likewise.
	* localedata/locales/tr_TR: Likewise.
	* localedata/locales/translit_circle: Likewise.
	* localedata/locales/translit_cjk_compat: Likewise.
	* localedata/locales/translit_combining: Likewise.
	* localedata/locales/translit_compat: Likewise.
	* localedata/locales/translit_font: Likewise.
	* localedata/locales/translit_fraction: Likewise.
2019-03-08 12:20:35 +01:00
Joseph Myers
04277e02d7 Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2019-01-01 00:11:28 +00:00
Mike FABIAN
4beefeeb8e Put the correct Unicode version number 11.0.0 into the generated files
In some places there was still the old Unicode version 10.0.0 in the files.

	* localedata/charmaps/UTF-8: Use correct Unicode version 11.0.0 in comment.
	* localedata/locales/i18n_ctype: Use correct Unicode version in comments
	and headers.
	* localedata/unicode-gen/utf8_gen.py: Add option to specify Unicode version
	* localedata/unicode-gen/Makefile: Use option to specify Unicode version
	for utf8_gen.py
2018-07-10 17:30:31 +02:00
Mike FABIAN
b11643c21c Bug 23308: Update to Unicode 11.0.0
Unicode 11.0.0 Support: Character encoding, character type info, and
transliteration tables are all updated to Unicode 11.0.0, using
the generator scripts contributed by Mike FABIAN (Red Hat).

Some info about the number of characters added:

Total added characters in newly generated CHARMAP: 684
Total added characters in newly generated WIDTH: 119
alpha: Added 380 characters in new ctype which were not in old ctype
combining: Added 56 characters in new ctype which were not in old ctype
combining_level3: Added 37 characters in new ctype which were not in old ctype
graph: Added 684 characters in new ctype which were not in old ctype
lower: Added 82 characters in new ctype which were not in old ctype
print: Added 684 characters in new ctype which were not in old ctype
punct: Added 304 characters in new ctype which were not in old ctype
tolower: Added 79 characters in new ctype which were not in old ctype
totitle: Added 33 characters in new ctype which were not in old ctype
toupper: Added 79 characters in new ctype which were not in old ctype
upper: Added 79 characters in new ctype which were not in old ctype

No characters were removed.

	[BZ #23308]
	* unicode-gen/Makefile (UNICODE_VERSION): Set to 11.0.0.
	* localedata/unicode-gen/DerivedCoreProperties.txt: Update to Unicode 11.0.0.
	* localedata/unicode-gen/EastAsianWidth.txt: likewise.
	* localedata/unicode-gen/PropList.txt: likewise.
	* localedata/unicode-gen/UnicodeData.txt: likewise.
	* localedata/charmaps/UTF-8: Regenerate.
	* localedata/locales/i18n_ctype: likewise.
	* localedata/locales/tr_TR: likewise.
	* localedata/locales/translit_circle: likewise.
	* localedata/locales/translit_cjk_compat: likewise.
	* localedata/locales/translit_combining: likewise.
	* localedata/locales/translit_compat: likewise.
	* localedata/locales/translit_font: likewise.
	* localedata/locales/translit_fraction: likewise.
2018-07-04 12:03:33 +02:00
Joseph Myers
688903eb3e Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2018-01-01 00:32:25 +00:00
Rafal Luzynski
1bb3653925 localedata: Once again correct and regenerate i18n_ctype.
Following the previous work by Carlos O'Donell the category of LC_CTYPE
is correctly set to "i18n:2012" rather than "unicode:2014" and the
i18n_ctype file is once again regenerated from scratch to make sure it
does not contain any manual additions except the copyright message.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>

	* localedata/unicode-gen/gen_unicode_ctype.py (output_head):
	category of LC_CTYPE set to "i18n:2012".
	* localedata/locales/i18n_ctype: Regenerate.
2017-10-31 23:54:47 +01:00
Carlos O'Donell
337ff3c501 localedata: Fix unicode-gen check target.
After the transition to generating a distinct file for Unicode ctype
information e.g. i18n_ctype, the check target was left with the wrong
target name. This patch fixes the check target and regenerates the
files with more information than previously used, filling in the the
LC_IDENTIFICATION data.

Tested on x86_64 by regenerating from Unicode source files, and
running checks. Tested by subsequently rebuilding all locales.
No regressions in testsuite.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Reported-by: Rafal Luzynski <digitalfreak@lingonborough.com>
2017-10-25 09:17:46 -07:00
Carlos O'Donell
8dc8be75d2 localedata: Reorganize Unicode LC_CTYPE inclusion.
The commit does the following things:

* Move non-transliteration Unicode generated data to i18n_ctype.
* Copy the i18n_ctype data into i18n and add transliteration.

In the future, any locale which needs Unicode LC_CTYPE data can
also just use `copy i18n_ctype` and get the base character classes
and maps without transliteration.

Tested by compiling all the locales and my prototype C.UTF-8 which
uses it.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
2017-10-13 22:29:52 -07:00
Mike FABIAN
2ae5be041d Improve utf8_gen.py to set the width for characters with Prepended_Concatenation_Mark property to 1
[BZ #22070]
	* localedata/unicode-gen/utf8_gen.py: Set the width for
	characters with Prepended_Concatenation_Mark property to 1
	* localedata/charmaps/UTF-8: Updated using the improved script.
2017-09-06 12:39:49 +02:00
Mike FABIAN
af83ed5c46 Write all ranges of neighbouring characters with the same width using the range notation in charmaps/UTF-8
Writing ranges of neighbouring characters with the same with like this

    <U000E0100>...<U000E01EF>	0

in charmaps/UTF-8 is more efficient than writing many single character lines
like:

    <U000E0100>	0
    <U000E0101>	0
    ...

	[BZ #21750]
	* unicode-gen/utf8_gen.py: Write all ranges of neighbouring characters
	with the same width using the range notation in charmaps/UTF-8.
2017-09-06 12:37:49 +02:00
Thorsten Glaser
267ee5d7ab Resolve some historically special cases of ambiguous width
[BZ #21750]
* unicode-gen/utf8_gen.py (U+00AD): Set width to 1.
* unicode-gen/utf8_gen.py (U+1160..U+11FF): Set width to 0.
* unicode-gen/utf8_gen.py (U+3248..U+324F): Set width to 2.
* unicode-gen/utf8_gen.py (U+4DC0..U+4DFF): Likewise.
2017-08-17 11:06:08 +02:00
Thorsten Glaser
41b6f0ce85 Handle more cases of combining characters
[BZ #21750]
* unicode-gen/utf8_gen.py: Treat category Me and Mn as combining.
2017-08-17 11:06:08 +02:00
Thorsten Glaser
580be3035d UnicodeData has precedence over EastAsianWidth
[BZ #19852]
[BZ #21750]
* unicode-gen/utf8_gen.py: Process EastAsianWidth lines before
  UnicodeData lines so the latter have precedence; remove hack
  to group output by EastAsianWidth ranges.
2017-08-17 11:06:08 +02:00
Mike FABIAN
925fac7793 Bug 21533: Update to Unicode 10.0.0
* Unicode 10.0.0 Support: Character encoding, character type info, and
  transliteration tables are all updated to Unicode 10.0.0, using
  generator scripts contributed by Mike FABIAN (Red Hat).
2017-06-22 17:02:55 +02:00
Mike FABIAN
0b38d66a4e Bug 20313: Update to Unicode 9.0.0
* Unicode 9.0.0 Support: Character encoding, character type info, and
  transliteration tables are all updated to Unicode 9.0.0, using
  generator scripts contributed by Mike FABIAN (Red Hat).
2017-02-21 06:30:38 -05:00
Joseph Myers
bfff8b1bec Update copyright dates with scripts/update-copyrights. 2017-01-01 00:14:16 +00:00
Mike Frysinger
277da2ab88 unicode-gen: include standard comment file header
We deployed this header to all the locale files, so make sure
we include it in the generated ones too so we don't lose it.
2016-06-11 02:10:52 -04:00
Joseph Myers
f7a9f785e5 Update copyright dates with scripts/update-copyrights. 2016-01-04 16:05:18 +00:00
Joseph Myers
85bafe6f3d Automate LC_CTYPE generation for tr_TR, update to Unicode 8.0.0 (bug 18491).
This patch makes the automation of Unicode LC_CTYPE generation also
support generating the modified LC_CTYPE used for Turkish (where case
conversions of 'i' and 'I' differ from ASCII conventions), so allowing
that to be more readily kept in sync for future Unicode updates.  The
patch includes the locale update generated by the scripts.

Tested for x86_64.

	[BZ #18491]
	* unicode-gen/unicode_utils.py (to_upper_turkish): New function.
	(to_lower_turkish): Likewise.
	* unicode-gen/gen_unicode_ctype.py (output_tables): Support
	producing output with Turkish case conversions.
	(--turkish): New command-line option.
	* unicode-gen/Makefile (GENERATED): Add tr_TR.
	(tr_TR): New rule.
	* locales/tr_TR: Regenerate LC_CTYPE.
2015-12-11 12:45:19 +00:00
Mike FABIAN
23256f5ed8 Update to Unicode 8.0.0.
Update __STDC_ISO_10646__ to 201505L for Unicode 8.0.0.
Update character encoding, ctype, and transliteration tables.
New scripts autogenerate transliteration tables.
2015-12-10 00:33:48 -05:00
Carlos O'Donell
dd8e8e5476 Update transliteration support to Unicode 7.0.0.
The transliteration files are now autogenerated from upstream Unicode
data.
2015-12-09 22:52:13 -05:00
Alexandre Oliva
7b1ec6a05c Amendments to Unicode 7 update.
for  ChangeLog

	* include/stdc-predef.h (__STDC_ISO_10646__): Update to
	201304L, for Unicode 7.

for  localedata/ChangeLog

	* unicode-gen/ctype_compatibility.py: Use date ranges in
	copyright notice.
	* unicode-gen/ctype_compatibility_test_cases.py: Likewise.
	* unicode-gen/gen_unicode_ctype.py: Likewise.
	* unicode-gen/utf8_compatibility.py: Likewise.
	* unicode-gen/utf8_gen.py: Likewise.  Use upper case for
	global variables, use tuples for global constant arrays.  From
	Mike FABIAN.  Suggested by Mike Frysinger <vapier@gentoo.org>.
2015-02-23 11:35:24 -03:00
Alexandre Oliva
4a4839c94a Unicode 7.0.0 update; added generator scripts.
for  localedata/ChangeLog

	[BZ #17588]
	[BZ #13064]
	[BZ #14094]
	[BZ #17998]
	* unicode-gen/Makefile: New.
	* unicode-gen/unicode-license.txt: New, from Unicode.
	* unicode-gen/UnicodeData.txt: New, from Unicode.
	* unicode-gen/DerivedCoreProperties.txt: New, from Unicode.
	* unicode-gen/EastAsianWidth.txt: New, from Unicode.
	* unicode-gen/gen_unicode_ctype.py: New generator, from Mike
	FABIAN <mfabian@redhat.com>.
	* unicode-gen/ctype_compatibility.py: New verifier, from
	Pravin Satpute <psatpute@redhat.com> and Mike FABIAN.
	* unicode-gen/ctype_compatibility_test_cases.py: New verifier
	module, from Mike FABIAN.
	* unicode-gen/utf8_gen.py: New generator, from Pravin Satpute
	and Mike FABIAN.
	* unicode-gen/utf8_compatibility.py: New verifier, from Pravin
	Satpute and Mike FABIAN.
	* charmaps/UTF-8: Update.
	* locales/i18n: Update.
	* gen-unicode-ctype.c: Remove.
	* tst-ctype-de_DE.ISO-8859-1.in: Adjust, islower now returns
	true for ordinal indicators.
2015-02-20 20:14:59 -02:00