mirror of
https://sourceware.org/git/glibc.git
synced 2024-11-21 12:30:06 +00:00
25c9c3789e
= `Default_Ignorable_Code_Point`s should have width 0 = Unicode specifies (https://www.unicode.org/faq/unsup_char.html#3) that characters with the `Default_Ignorable_Code_Point` property > should be rendered as completely invisible (and non advancing, i.e. “zero width”), if not explicitly supported in rendering. Hence, `wcwidth()` should give them all a width of 0, with two exceptions: - the soft hyphen (U+00AD SOFT HYPHEN) is assigned width 1 by longstanding precedent - U+115F HANGUL CHOSEONG FILLER needs a carveout due to the unique behavior of the conjoining Korean jamo characters. One composed Hangul "syllable block" like 퓛 is made up of two to three individual component characters, or "jamo". These are all assigned an `East_Asian_Width` of `Wide` by Unicode, which would normally mean they would all be assigned width 2 by glibc; a combination of (leading choseong jamo) + (medial jungseong jamo) + (trailing jongseong jamo) would then have width 2 + 2 + 2 = 6. However, glibc (and other wcwidth implementations) special-cases jungseong and jongseong, assigning them all width 0, to ensure that the complete block has width 2 + 0 + 0 = 2 as it should. U+115F is meant for use in syllable blocks that are intentionally missing a leading jamo; it must be assigned a width of 2 even though it has no visible display to ensure that the complete block has width 2. However, `wcwidth()` currently (before this patch) incorrectly assigns non-zero width to U+3164 HANGUL FILLER and U+FFA0 HALFWIDTH HANGUL FILLER; this commit fixes that. Unicode spec references: - Hangul: §3.12 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G24646 and §18.6 https://www.unicode.org/versions/Unicode15.0.0/ch18.pdf#G31028 - `Default_Ignorable_Code_Point`: §5.21 https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095. = Non-`Default_Ignorable_Code_Point` format controls should be visible = The Unicode Standard, §5.21 - Characters Ignored for Display (https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095) says the following: > A small number of format characters (General_Category = Cf ) > are also not given the Default_Ignorable_Code_Point property. > This may surprise implementers, who often assume > that all format characters are generally ignored in fallback display. > The exact list of these exceptional format characters > can be found in the Unicode Character Database. > There are, however, three important sets of such format characters to note: > > - prepended concatenation marks > - interlinear annotation characters > - Egyptian hieroglyph format controls > > The prepended concatenation marks always have a visible display. > See “Prepended Concatenation Marks” in [*Section 23.2, Layout Controls*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.35858.HeadingBreak.132.Layout.Controls) > for more discussion of the use and display of these signs. > > The other two notable sets of format characters that exceptionally are not ignored > in fallback display consist of the interlinear annotation characters, > U+FFF9 INTERLINEAR ANNOTATION ANCHOR through > U+FFFB INTERLINEAR ANNOTATION TERMINATOR, > and the Egyptian hieroglyph format controls, > U+13430 EGYPTIAN HIEROGLYPH VERTICAL JOINER through > U+1343F EGYPTIAN HIEROGLYPH END WALLED ENCLOSURE. > These characters should have a visible glyph display for fallback rendering, > because if they are not displayed, > it is too easy to misread the resulting displayed text. > See “Annotation Characters” in [*Section 23.8, Specials*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.21335.Heading.133.Specials), > as well as [*Section 11.4, Egyptian Hieroglyphs*](https://www.unicode.org/versions/Unicode15.1.0/ch11.pdf#M9.73291.Heading.1418.Egyptian.Hieroglyphs) > for more discussion of the use and display of these characters. glibc currently correctly assigns non-zero width to the prepended concatenation marks, but it incorrectly gives zero width to the interlinear annotation characters (which a generic terminal cannot interpret) and the Egyptian hieroglyph format controls (which are not widely supported in rendering implementations at present). This commit fixes both these issues as well. = Derive Hangul syllable type from Unicode data = Previosuly, the jungseong and jongseong jamo ranges were hard-coded into the script. With this commit, they are instead parsed from the HangulSyllableType.txt data file published by Unicode. This does not affect the end result. Signed-off-by: Jules Bertholet <julesbertholet@quoi.xyz> |
||
---|---|---|
.. | ||
charmaps | ||
locales | ||
tests | ||
tests-mbwc | ||
tst-fmon-locales | ||
tst-localedef-hardlinks.root | ||
unicode-gen | ||
am_ET.UTF-8.in | ||
az_AZ.UTF-8.in | ||
be_BY.UTF-8.in | ||
ber_DZ.UTF-8.in | ||
ber_MA.UTF-8.in | ||
bg_BG.UTF-8.in | ||
br_FR.UTF-8.in | ||
bs_BA.UTF-8.in | ||
bug-iconv-trans.c | ||
bug-setlocale1-static.c | ||
bug-setlocale1.c | ||
bug-usesetlocale.c | ||
C.UTF-8.in | ||
ckb_IQ.UTF-8.in | ||
cmn_TW.UTF-8.in | ||
collate-test.c | ||
crh_UA.UTF-8.in | ||
cs_CZ.in | ||
cs_CZ.UTF-8.in | ||
csb_PL.UTF-8.in | ||
cv_RU.UTF-8.in | ||
cy_GB.UTF-8.in | ||
da_DK.ISO-8859-1.in | ||
de_DE.ISO-8859-1.in | ||
Depend | ||
dsb_DE.UTF-8.in | ||
dump-ctype.c | ||
dz_BT.UTF-8.in | ||
en_US.ISO-8859-1.in | ||
en_US.UTF-8.in | ||
eo.UTF-8.in | ||
es_ES.UTF-8.in | ||
et_EE.UTF-8.in | ||
fa_IR.UTF-8.in | ||
fi_FI.UTF-8.in | ||
fil_PH.UTF-8.in | ||
fr_CA.UTF-8.in | ||
fr_FR.UTF-8.in | ||
fur_IT.UTF-8.in | ||
gen-locale.sh | ||
gez_ER.UTF-8@abegede.in | ||
ha_NG.UTF-8.in | ||
hr_HR.ISO-8859-2.in | ||
hr_HR.UTF-8.in | ||
hsb_DE.UTF-8.in | ||
hu_HU.UTF-8.in | ||
ig_NG.UTF-8.in | ||
ik_CA.UTF-8.in | ||
is_IS.UTF-8.in | ||
kk_KZ.UTF-8.in | ||
ku_TR.UTF-8.in | ||
ky_KG.UTF-8.in | ||
ln_CD.UTF-8.in | ||
lt_LT.UTF-8.in | ||
lv_LV.UTF-8.in | ||
Makefile | ||
mdf_RU.UTF-8.in | ||
mi_NZ.UTF-8.in | ||
ml_IN.UTF-8.in | ||
mn_MN.UTF-8.in | ||
mr_IN.UTF-8.in | ||
mt_MT.UTF-8.in | ||
nan_TW.UTF-8@latin.in | ||
nb_NO.UTF-8.in | ||
om_KE.UTF-8.in | ||
or_IN.UTF-8.in | ||
os_RU.UTF-8.in | ||
pl_PL.UTF-8.in | ||
ps_AF.UTF-8.in | ||
README | ||
rif_MA.UTF-8.in | ||
ro_RO.UTF-8.in | ||
ru_RU.UTF-8.in | ||
sah_RU.UTF-8.in | ||
sc_IT.UTF-8.in | ||
se_NO.UTF-8.in | ||
show-ucs-data.c | ||
si_LK.UTF-8.in | ||
sort-test.sh | ||
sq_AL.UTF-8.in | ||
sr_RS.UTF-8.in | ||
SUPPORTED | ||
sv_SE.ISO-8859-1.in | ||
sv_SE.UTF-8.in | ||
syr.UTF-8.in | ||
szl_PL.UTF-8.in | ||
tg_TJ.UTF-8.in | ||
th_TH.UTF-8.in | ||
tk_TM.UTF-8.in | ||
tr_TR.UTF-8.in | ||
tst-c-utf8-consistency.c | ||
tst-ctype-de_DE.ISO-8859-1.in | ||
tst-ctype.c | ||
tst-ctype.sh | ||
tst-digits.c | ||
tst-fmon.c | ||
tst-fmon.data | ||
tst-fmon.sh | ||
tst-iconv-emojis-trans.c | ||
tst-iconv-math-trans.c | ||
tst-langinfo-newlocale-static.c | ||
tst-langinfo-newlocale.c | ||
tst-langinfo-setlocale-static.c | ||
tst-langinfo-setlocale.c | ||
tst-langinfo-static.c | ||
tst-langinfo.c | ||
tst-langinfo.sh | ||
tst-leaks.c | ||
tst-locale-loadlocale.c | ||
tst-locale.sh | ||
tst-localedef-hardlinks.c | ||
tst-mbswcs1.c | ||
tst-mbswcs2.c | ||
tst-mbswcs3.c | ||
tst-mbswcs4.c | ||
tst-mbswcs5.c | ||
tst-mbswcs6.c | ||
tst-numeric.c | ||
tst-numeric.data | ||
tst-numeric.sh | ||
tst-rpmatch.c | ||
tst-rpmatch.sh | ||
tst-setlocale2.c | ||
tst-setlocale3.c | ||
tst-setlocale.c | ||
tst-sscanf.c | ||
tst-strfmon1.c | ||
tst-trans.c | ||
tst-trans.sh | ||
tst-wctype.c | ||
tst-wctype.input | ||
tst-xlocale1.c | ||
tst-xlocale2.c | ||
tt_RU.UTF-8.in | ||
tt_RU.UTF-8@iqtelif.in | ||
ug_CN.UTF-8.in | ||
uk_UA.UTF-8.in | ||
uz_UZ.UTF-8.in | ||
vi_VN.UTF-8.in | ||
xfrm-test.c | ||
yi_US.UTF-8.in | ||
yo_NG.UTF-8.in | ||
zh_CN.UTF-8.in |
POSIX locale descriptions and POSIX character set descriptions Ulrich Drepper Time-stamp: <2004/11/27 13:06:54 drepper> drepper@redhat.com This directory contains the data needed to build the locale data files to use the internationalization features of the GNU libc. POSIX.2 describes the `localedef' utility which is part of the GNU libc. You need this program to "compile" the locale description in a form suitable for fast access by the GNU libc functions. Any compilation is based on a given character set. Once you run `make install' for the GNU libc the data files are automatically installed in the right place, ready for use by the `localedef' program. To compile the locale data files you simply have to decide which locale (based on the location and the language) and which character set you use. E.g., French speaking Canadians would use the locale `fr_CA' and the character set `ISO_8859-1,1987'. Calling `localedef' to get the desired data should happen like this: localedef -i fr_CA -f ISO-8859-1 fr_CA This will place the 6 output files in the appropriate directory where the GNU libc functions can find them. Please note that you need permission to write to this directory ($(prefix)/share/locale, where $(prefix) is the value you specified while configuring GNU libc). If you do not have the necessary permissions, you can write the files into an arbitrary directory by giving a path including a '/' character instead of `fr_CA'. E.g., to put the new files in a subdirectory of the current directory simply use localedef -i fr_CA -f ISO-8859-1 ./fr_CA How to use these data files is described in the GNU libc manual, especially in the section describing the `setlocale' function. All problems should be reported using https://sourceware.org/bugzilla/ One more note: the `POSIX' locale definition is not meant to be used as an input file for `localedef'. It is rather there to show the values with are built in the libc binaries as default values when no legal locale is found or the "C" or "POSIX" locale is selected. The collation test suite ######################## This package also contains a (beginning of a) test suite for the collation functions in the GNU libc. The files are provided sorted. The test program shuffles the lines and sort them afterwards. Some of the files are provided in 8bit form, i.e., not only ASCII characters. So the tools you use to process the files should be 8bit clean. To run the test program the appropriate locale information must be installed. Therefore the localedef program is used to generate this data used the locale and charmap description files contained here. Since we cannot run the localedef program in case of cross-compilation no tests at all are performed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Local Variables: mode:text eval:(load-library "time-stamp") eval:(make-local-variable 'write-file-hooks) eval:(add-hook 'write-file-hooks 'time-stamp) eval:(setq time-stamp-format '(time-stamp-yyyy/mm/dd time-stamp-hh:mm:ss user-login-name)) End: