glibc/localedata
Carlos O'Donell 466f2be6c0 Add generic C.UTF-8 locale (Bug 17318)
We add a new C.UTF-8 locale. This locale is not builtin to glibc, but
is provided as a distinct locale. The locale provides full support for
UTF-8 and this includes full code point sorting via STRCMP-based
collation (strcmp or wcscmp).

The collation uses a new keyword 'codepoint_collation' which drops all
collation rules and generates an empty zero rules collation to enable
STRCMP usage in collation. This ensures that we get full code point
sorting for C.UTF-8 with a minimal 1406 bytes of overhead (LC_COLLATE
structure information and ASCII collating tables).

The new locale is added to SUPPORTED. Minimal test data for specific
code points (minus those not supported by collate-test) is provided in
C.UTF-8.in, and this verifies code point sorting is working reasonably
across the range. The locale was tested manually with the full set of
code points without failure.

The locale is harmonized with locales already shipping in various
downstream distributions. A new tst-iconv9 test is added which verifies
the C.UTF-8 locale is generally usable.

Testing for fnmatch, regexec, and recomp is provided by extending
bug-regex1, bugregex19, bug-regex4, bug-regex6, transbug, tst-fnmatch,
tst-regcomp-truncated, and tst-regex to use C.UTF-8.

Tested on x86_64 or i686 without regression.

Reviewed-by: Florian Weimer <fweimer@redhat.com>
2021-09-06 11:30:28 -04:00
..
charmaps localedata: Use U+00AF MACRON in more EBCDIC charsets [BZ #27882] 2021-05-18 07:21:45 +02:00
locales Add generic C.UTF-8 locale (Bug 17318) 2021-09-06 11:30:28 -04:00
tests Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tests-mbwc Break some lines before not after operators. 2019-02-22 01:32:36 +00:00
tst-fmon-locales Update. 2003-12-05 09:51:08 +00:00
tst-localedef-hardlinks.root localedef: Add tests-container test for --no-hard-links. 2020-04-30 16:28:07 -04:00
unicode-gen Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
am_ET.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
az_AZ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
be_BY.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ber_DZ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ber_MA.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
bg_BG.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
br_FR.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
bs_BA.UTF-8.in sr_RS and bs_BA locales: make collation rules the same as for hr_HR [BZ #22534] 2017-11-30 16:03:22 +01:00
bug-iconv-trans.c Update. 2001-08-14 22:29:43 +00:00
bug-setlocale1-static.c Support run bug-setlocale1 directly 2013-01-10 14:15:49 -08:00
bug-setlocale1.c Avoid re-exec-self in bug-setlocale1. 2015-03-05 12:58:49 -08:00
bug-usesetlocale.c Use locale_t, not __locale_t, throughout glibc 2017-06-20 20:30:06 -04:00
C.UTF-8.in Add generic C.UTF-8 locale (Bug 17318) 2021-09-06 11:30:28 -04:00
ckb_IQ.UTF-8.in Fix ckb_IQ [BZ #9809] 2020-02-03 10:19:20 +01:00
cmn_TW.UTF-8.in Remove the lines from cmn_TW.UTF-8.in which cannot work at the moment. 2018-02-27 17:47:51 +01:00
collate-test.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
crh_UA.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
cs_CZ.in cs_CZ test data for locale tests. 1999-02-27 18:14:22 +00:00
cs_CZ.UTF-8.in Collation order of ȥ has changed in new iso14651_t1_common file, adapt test files 2018-02-27 16:54:31 +01:00
csb_PL.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
cv_RU.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
cy_GB.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
da_DK.ISO-8859-1.in Collation order of @-. and space has changed in new iso14651_t1_common file, adapt test files 2018-02-27 16:56:25 +01:00
de_DE.ISO-8859-1.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
Depend Update. 2000-07-01 05:19:18 +00:00
dsb_DE.UTF-8.in dsb_DE locale: Fix syntax error and add tests (bug 23208). 2018-07-13 23:06:32 +02:00
dump-ctype.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
dz_BT.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
en_US.ISO-8859-1.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
en_US.UTF-8.in Add missing localedata/en_US.UTF-8.in (Bug 23393). 2018-07-25 21:58:10 -04:00
eo.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
es_ES.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
et_EE.UTF-8.in et_EE locale: Base collation on iso14651_t1 [BZ #22517] 2017-12-05 17:10:08 +01:00
fa_IR.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
fi_FI.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
fil_PH.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
fr_CA.UTF-8.in Collation order of @-. and space has changed in new iso14651_t1_common file, adapt test files 2018-02-27 16:56:25 +01:00
fr_FR.UTF-8.in Collation order of @-. and space has changed in new iso14651_t1_common file, adapt test files 2018-02-27 16:56:25 +01:00
fur_IT.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
gen-locale.sh Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
gez_ER.UTF-8@abegede.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ha_NG.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
hr_HR.ISO-8859-2.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
hr_HR.UTF-8.in hr_HR locale: fix collation and expand collation test file 2017-11-30 14:57:46 +01:00
hsb_DE.UTF-8.in hsb_DE locale: Base collation on copy "iso14651_t1" [BZ #22515] 2017-12-06 12:04:29 +01:00
hu_HU.UTF-8.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
ig_NG.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ik_CA.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
is_IS.UTF-8.in is_IS locale: Base collation on iso14651_t1 [BZ #22519] 2017-12-01 14:08:58 +01:00
kk_KZ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ku_TR.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ky_KG.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ln_CD.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
lt_LT.UTF-8.in lt_LT locale: Base collation on copy "iso14651_t1" [BZ #22524] 2017-12-07 15:38:11 +01:00
lv_LV.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
Makefile Add generic C.UTF-8 locale (Bug 17318) 2021-09-06 11:30:28 -04:00
mi_NZ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ml_IN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
mn_MN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
mr_IN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
mt_MT.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
nan_TW.UTF-8@latin.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
nb_NO.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
om_KE.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
or_IN.UTF-8.in Update or_IN collation [BZ #22525] 2020-02-03 10:19:20 +01:00
os_RU.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
pl_PL.UTF-8.in Collation order of ȥ has changed in new iso14651_t1_common file, adapt test files 2018-02-27 16:54:31 +01:00
ps_AF.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
README Prefer https for Sourceware links 2017-11-16 11:49:26 +05:30
ro_RO.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ru_RU.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
sah_RU.UTF-8.in New locale: Yakut (Sakha) for Russia (sah_RU) [BZ #22241] 2018-07-18 11:45:44 +02:00
sc_IT.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
se_NO.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
show-ucs-data.c [BZ #4098] 2007-10-14 20:51:58 +00:00
si_LK.UTF-8.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
sort-test.sh Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
sq_AL.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
sr_RS.UTF-8.in sr_RS and bs_BA locales: make collation rules the same as for hr_HR [BZ #22534] 2017-11-30 16:03:22 +01:00
SUPPORTED Add generic C.UTF-8 locale (Bug 17318) 2021-09-06 11:30:28 -04:00
sv_SE.ISO-8859-1.in Update sv_SE to treate 'W' as a distinct character (Bug 25036) 2021-04-06 12:34:02 -04:00
sv_SE.UTF-8.in Update sv_SE to treate 'W' as a distinct character (Bug 25036) 2021-04-06 12:34:02 -04:00
szl_PL.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
tg_TJ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
th_TH.in Update. 1999-04-16 16:08:19 +00:00
tk_TM.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
tr_TR.UTF-8.in localedata: Locale and test name are the same. 2017-10-25 12:24:31 -07:00
tst-ctype-de_DE.ISO-8859-1.in Unicode 7.0.0 update; added generator scripts. 2015-02-20 20:14:59 -02:00
tst-ctype.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-ctype.sh Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-digits.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-fmon.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-fmon.data Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-fmon.sh Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-iconv-math-trans.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-langinfo-newlocale-static.c locale: XFAIL newlocale usage in static binary (Bug 23164) 2018-07-04 15:23:50 -04:00
tst-langinfo-newlocale.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-langinfo-setlocale-static.c locale: XFAIL newlocale usage in static binary (Bug 23164) 2018-07-04 15:23:50 -04:00
tst-langinfo-setlocale.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-langinfo-static.c fix nl_langinfo with static linking (BZ #16915) 2014-05-20 18:43:14 +02:00
tst-langinfo.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-langinfo.sh Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-leaks.c Don't reduce test timeout to less than default 2018-10-17 09:34:13 +02:00
tst-locale.sh Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-localedef-hardlinks.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-mbswcs1.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-mbswcs2.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-mbswcs3.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-mbswcs4.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-mbswcs5.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-mbswcs6.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-numeric.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-numeric.data Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-numeric.sh Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-rpmatch.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-rpmatch.sh Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-setlocale2.c Optimize access to isXYZ and toXYZ tables 2011-10-15 16:27:08 -04:00
tst-setlocale3.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-setlocale.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-sscanf.c Use fa_IR.UTF-8 which exists internally. 2007-03-14 22:32:31 +00:00
tst-strfmon1.c nl_NL locale: Correct the negative monetary format (bug 24614). 2019-06-17 23:42:06 +02:00
tst-trans.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-trans.sh Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
tst-wctype.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
tst-wctype.input Update. 2000-07-25 04:37:22 +00:00
tst-xlocale1.c Modify several tests to use test-skeleton.c 2014-11-05 15:24:08 +05:30
tst-xlocale2.c Use locale_t, not __locale_t, throughout glibc 2017-06-20 20:30:06 -04:00
tt_RU.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
tt_RU.UTF-8@iqtelif.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
ug_CN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
uk_UA.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
uz_UZ.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
vi_VN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
xfrm-test.c Remove "Contributed by" lines 2021-09-03 22:06:44 +05:30
yi_US.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
yo_NG.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00
zh_CN.UTF-8.in Adapt collation in several locales to the new iso14651_t1_common file 2018-02-27 17:47:50 +01:00

		       POSIX locale descriptions
				  and
		    POSIX character set descriptions

Ulrich Drepper			Time-stamp: <2004/11/27 13:06:54 drepper>
drepper@redhat.com


This directory contains the data needed to build the locale data files
to use the internationalization features of the GNU libc.

POSIX.2 describes the `localedef' utility which is part of the GNU libc.
You need this program to "compile" the locale description in a form
suitable for fast access by the GNU libc functions.  Any compilation is
based on a given character set.

Once you run `make install' for the GNU libc the data files are
automatically installed in the right place, ready for use by the
`localedef' program.

To compile the locale data files you simply have to decide which locale
(based on the location and the language) and which character set you
use.  E.g., French speaking Canadians would use the locale `fr_CA' and
the character set `ISO_8859-1,1987'.  Calling `localedef' to get the
desired data should happen like this:

	localedef -i fr_CA -f ISO-8859-1 fr_CA

This will place the 6 output files in the appropriate directory where
the GNU libc functions can find them.  Please note that you need
permission to write to this directory ($(prefix)/share/locale, where
$(prefix) is the value you specified while configuring GNU libc).  If
you do not have the necessary permissions, you can write the files into an
arbitrary directory by giving a path including a '/' character instead
of `fr_CA'.  E.g., to put the new files in a subdirectory of the
current directory simply use

	localedef -i fr_CA -f ISO-8859-1 ./fr_CA

How to use these data files is described in the GNU libc manual,
especially in the section describing the `setlocale' function.

All problems should be reported using

  https://sourceware.org/bugzilla/


One more note: the `POSIX' locale definition is not meant to be used
as an input file for `localedef'.  It is rather there to show the
values with are built in the libc binaries as default values when no
legal locale is found or the "C" or "POSIX" locale is selected.


		       The collation test suite
		       ########################

This package also contains a (beginning of a) test suite for the
collation functions in the GNU libc.  The files are provided sorted.
The test program shuffles the lines and sort them afterwards.

Some of the files are provided in 8bit form, i.e., not only ASCII
characters.  So the tools you use to process the files should be 8bit
clean.

To run the test program the appropriate locale information must be
installed.  Therefore the localedef program is used to generate this
data used the locale and charmap description files contained here.
Since we cannot run the localedef program in case of cross-compilation
no tests at all are performed.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local Variables:
 mode:text
 eval:(load-library "time-stamp")
 eval:(make-local-variable 'write-file-hooks)
 eval:(add-hook 'write-file-hooks 'time-stamp)
 eval:(setq time-stamp-format '(time-stamp-yyyy/mm/dd time-stamp-hh:mm:ss user-login-name))
End: