Commit Graph

182 Commits

Author SHA1 Message Date
Carlos O'Donell
df6c63ebbc localedef: Add tests-container test for --no-hard-links.
The new tst-localedef-hardlinks verifies that when compiling
two locales (with default output directory) one with
--no-hard-links and one without the option, results in the
expected behaviour.  When --no-hard-links is used the link
counts on LC_CTYPE is 1, indicating that even thoug the two
locale are identical (though different named source files and
output direcotry) the localedef did not carry out the hard
link optimization.  Then when --no-hard-links is omitted the
localedef hard link optimization is correctly carried out and
for 2 compiled locales the link count for LC_CTYPE is 2.

Reviewed-by: DJ Delorie <dj@redhat.com>
2020-04-30 16:28:07 -04:00
Mike FABIAN
cdeae33d71 Update or_IN collation [BZ #22525]
- Add a test file or_IN.UTF-8.in.
- Make the collation agree with CLDR.
2020-02-03 10:19:20 +01:00
Mike FABIAN
ae199e7d64 Fix ckb_IQ [BZ #9809]
Add ckb_IQ to SUPPORTED file.
Add ckb_IQ.UTF-8.in collation test file.
Mention new ckb_IQ locale in NEWS.
2020-02-03 10:19:20 +01:00
Joseph Myers
d614a75396 Update copyright dates with scripts/update-copyrights. 2020-01-01 00:14:33 +00:00
Arjun Shankar
513aaa0d78 Add Transliterations for Unicode Misc. Mathematical Symbols-A/B [BZ #23132]
This commit adds previously missing transliterations for several code points
in the Unicode blocks "Miscellaneous Mathematical Symbols-A/B" -
transliterated to their approximate ASCII representations.  It also adds a
corresponding iconv transliteration test.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2019-10-25 19:45:55 +02:00
DJ Delorie
97476447ed Install charmaps uncompressed in testroot
The testroot does not have a gunzip command, so the charmap files
should not be installed gzipped else they cannot be used (and thus
tested).  With this patch, installing with INSTALL_UNCOMPRESSED=yes
installs uncompressed charmaps instead.

Note that we must purge the $(symbolic_link_list) as it contains
references to $(DESTDIR), which we change during the testroot
installation.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2019-10-24 17:01:04 -04:00
Paul Eggert
5a82c74822 Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:

sed -ri '
  s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
  s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
  $(find $(git ls-files) -prune -type f \
      ! -name '*.po' \
      ! -name 'ChangeLog*' \
      ! -path COPYING ! -path COPYING.LIB \
      ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
      ! -path manual/texinfo.tex ! -path scripts/config.guess \
      ! -path scripts/config.sub ! -path scripts/install-sh \
      ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
      ! -path INSTALL ! -path  locale/programs/charmap-kw.h \
      ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
      ! '(' -name configure \
            -execdir test -f configure.ac -o -f configure.in ';' ')' \
      ! '(' -name preconfigure \
            -execdir test -f preconfigure.ac ';' ')' \
      -print)

and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:

  chmod a+x sysdeps/unix/sysv/linux/riscv/configure
  # Omit irrelevant whitespace and comment-only changes,
  # perhaps from a slightly-different Autoconf version.
  git checkout -f \
    sysdeps/csky/configure \
    sysdeps/hppa/configure \
    sysdeps/riscv/configure \
    sysdeps/unix/sysv/linux/csky/configure
  # Omit changes that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
  git checkout -f \
    sysdeps/powerpc/powerpc64/ppc-mcount.S \
    sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
  # Omit change that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
  git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 02:43:31 -07:00
Rafal Luzynski
02d8b5ab1c nl_NL locale: Correct the negative monetary format (bug 24614).
According to CLDR 35.1 and the bug report the correct monetary format
for negative amounts should be "EUR -1 234,56" while previously it was
"EUR 1 234,56-".

This patch does not change the thousands (grouping) separator.

	[BZ #24614]
	* localedata/Makefile (LOCALES): Add nl_NL.UTF-8.
	* localedata/locales/nl_NL (n_sep_by_space): Set to 2 (a space
	between the currency symbol and the minus sign).
	(n_sign_posn): Set to 4 (the minus sign after the currency symbol).
	* localedata/tst-strfmon1.c (tests): Add test data for nl_NL.UTF-8.
2019-06-17 23:42:06 +02:00
Joseph Myers
04277e02d7 Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2019-01-01 00:11:28 +00:00
Carlos O'Donell
8cebd4ffe6 Add --no-hard-links option to localedef (bug 23923)
Downstream distributions need consistent sets of hardlinks in
order for rpm to operate effectively. This means that even if
locales are built with a high level of parallelism that the
resulting files need to have consistent hardlink counts. The only
way to achieve this is with a post-install hardlink pass using a
program like 'hardlink' (shipped in Fedora).

If the downstream distro wants to post-process the hardlinks then
the time spent in localedef looking up sibling directories and
processing hardlinks is wasted effort.

To optimize the build and install pass we add a --no-hard-links
option to localedef to avoid doing the hardlink optimziation for
size.

Tested on x86_64 with 'make localedata/install-locale-files'
before and after. Without the patch we have files with 100+
hardlink counts. After the patch and running with --no-hard-links
all link counts are 1. This patch also alters the convenience
target 'make localedata/install-locale-files' to use the new
option.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
2018-12-03 10:15:39 -05:00
Carlos O'Donell
08a5ee14c6 Add convenience target 'install-locale-files'.
The convenience install target 'install-locale-files' is created
to allow distributions to install all of the SUPPORTED locales as
files instead of into the locale-archive.

You invoke the new convenience target like this:
make localedata/install-locale-files DESTDIR=<prefix>
2018-08-02 15:31:12 -04:00
Carlos O'Donell
7cd7d36f1f Keep expected behaviour for [a-z] and [A-z] (Bug 23393).
In commit 9479b6d5e0 we updated all of
the collation data to harmonize with the new version of ISO 14651
which is derived from Unicode 9.0.0.  This collation update brought
with it some changes to locales which were not desirable by some
users, in particular it altered the meaning of the
locale-dependent-range regular expression, namely [a-z] and [A-Z], and
for en_US it caused uppercase letters to be matched by [a-z] for the
first time.  The matching of uppercase letters by [a-z] is something
which is already known to users of other locales which have this
property, but this change could cause significant problems to en_US
and other similar locales that had never had this change before.
Whether this behaviour is desirable or not is contentious and GNU Awk
has this to say on the topic:
https://www.gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html
While the POSIX standard also has this further to say: "RE Bracket
Expression":
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap09.html
"The current standard leaves unspecified the behavior of a range
expression outside the POSIX locale. ... As noted above, efforts were
made to resolve the differences, but no solution has been found that
would be specific enough to allow for portable software while not
invalidating existing implementations."
In glibc we implement the requirement of ISO POSIX-2:1993 and use
collation element order (CEO) to construct the range expression, the
API internally is __collseq_table_lookup().  The fact that we use CEO
and also have 4-level weights on each collation rule means that we can
in practice reorder the collation rules in iso14651_t1_common (the new
data) to provide consistent range expression resolution *and* the
weights should maintain the expected total order.  Therefore this
patch does three things:

* Reorder the collation rules for the LATIN script in
  iso14651_t1_common to deinterlace uppercase and lowercase letters in
  the collation element orders.

* Adds new test data en_US.UTF-8.in for sort-test.sh which exercises
  strcoll* and strxfrm* and ensures the ISO 14651 collation remains.

* Add back tests to tst-fnmatch.input and tst-regexloc.c which
  exercise that [a-z] does not match A or Z.

The reordering of the ISO 14651 data is done in an entirely mechanical
fashion using the following program attached to the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28

It is up for discussion if the iso14651_t1_common data should be
refined further to have 3 very tight collation element ranges that
include only a-z, A-Z, and 0-9, which would implement the solution
sought after in:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c12
and implemented here:
https://www.sourceware.org/ml/libc-alpha/2018-07/msg00854.html

No regressions on x86_64.
Verified that removal of the iso14651_t1_common change causes tst-fnmatch
to regress with:
422: fnmatch ("[a-z]", "A", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
...
425: fnmatch ("[A-Z]", "z", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
2018-07-25 17:00:45 -04:00
Valery Timiriliyev
61c4aad705 New locale: Yakut (Sakha) for Russia (sah_RU) [BZ #22241]
* localedata/Makefile (test-input): Add sah_RU.UTF-8.
	(LOCALES): Likewise.
	* localedata/SUPPORTED (sah_RU/UTF-8): New entry.
	* localedata/locales/sah_RU: New file.
	* localedata/sah_RU.UTF-8.in: New file.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2018-07-18 11:45:44 +02:00
Rafal Luzynski
0a83bad2aa dsb_DE locale: Fix syntax error and add tests (bug 23208).
Fixed syntax error in the collation rules of Lower Sorbian language.
Collation test added in order to test the bugs like this early.

Reported-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>

	[BZ #23208]
	* localedata/Makefile (test-input): Add dsb_DE.UTF-8.
	(LOCALES): Likewise.
	* localedata/dsb_DE.UTF-8.in: New file.
	* localedata/locales/dsb_DE (LC_COLLATE): Fix syntax error.
2018-07-13 23:06:32 +02:00
Carlos O'Donell
34fdb893e0 locale: XFAIL newlocale usage in static binary (Bug 23164)
There is a glibc optimization which allows for locale categories
to be removed during static compilation. There have been various
bugs for this support over the years, with bug 16915 being the
most recent. The solution there was to emit a reference to all the
categories to avoid any being removed. This fix, although it's in
the generic __nl_langinfo_l function, doesn't appear to be enough
to fix the case for a statically linked program that uses newlocale
and nl_langinfo_l. This commit doesn't fix the problem, but it does
add a XFAIL'd test case such that a fix can be applied against this
and the XFAIL removed. It's not entirely clear that the problem is
the same as that which was seen in bug 16915.
2018-07-04 15:23:50 -04:00
Mike FABIAN
1597385481 Adapt collation in several locales to the new iso14651_t1_common file
[BZ #22550] - es_ES locale (and other es_* locales): collation should
treat ñ as a primary different character, sync the collation
for Spanish with CLDR
[BZ #21547] - Tibetan script collation broken (Dzongkha and Tibetan)

	* localedata/Makefile: Add new test files.
	* localedata/lv_LV.UTF-8.in: Adapt test file to new collation order.
	* localedata/sv_SE.ISO-8859-1.in: Adapt test file to new collation order.
	* localedata/uk_UA.UTF-8.in: Adapt test file to new collation order.
	* localedata/am_ET.UTF-8.in: New test file.
	* localedata/az_AZ.UTF-8.in: Likewise.
	* localedata/be_BY.UTF-8.in: Likewise.
	* localedata/ber_DZ.UTF-8.in: Likewise.
	* localedata/ber_MA.UTF-8.in: Likewise.
	* localedata/bg_BG.UTF-8.in: Likewise.
	* localedata/br_FR.UTF-8.in: Likewise.
	* localedata/cmn_TW.UTF-8.in: Likewise.
	* localedata/crh_UA.UTF-8.in: Likewise.
	* localedata/csb_PL.UTF-8.in: Likewise.
	* localedata/cv_RU.UTF-8.in: Likewise.
	* localedata/cy_GB.UTF-8.in: Likewise.
	* localedata/dz_BT.UTF-8.in: Likewise.
	* localedata/eo.UTF-8.in: Likewise.
	* localedata/es_ES.UTF-8.in: Likewise.
	* localedata/fa_IR.UTF-8.in: Likewise.
	* localedata/fi_FI.UTF-8.in: Likewise.
	* localedata/fil_PH.UTF-8.in: Likewise.
	* localedata/fur_IT.UTF-8.in: Likewise.
	* localedata/gez_ER.UTF-8@abegede.in: Likewise.
	* localedata/ha_NG.UTF-8.in: Likewise.
	* localedata/ig_NG.UTF-8.in: Likewise.
	* localedata/ik_CA.UTF-8.in: Likewise.
	* localedata/kk_KZ.UTF-8.in: Likewise.
	* localedata/ku_TR.UTF-8.in: Likewise.
	* localedata/ky_KG.UTF-8.in: Likewise.
	* localedata/ln_CD.UTF-8.in: Likewise.
	* localedata/mi_NZ.UTF-8.in: Likewise.
	* localedata/ml_IN.UTF-8.in: Likewise.
	* localedata/mn_MN.UTF-8.in: Likewise.
	* localedata/mr_IN.UTF-8.in: Likewise.
	* localedata/mt_MT.UTF-8.in: Likewise.
	* localedata/nb_NO.UTF-8.in: Likewise.
	* localedata/om_KE.UTF-8.in: Likewise.
	* localedata/os_RU.UTF-8.in: Likewise.
	* localedata/ps_AF.UTF-8.in: Likewise.
	* localedata/ro_RO.UTF-8.in: Likewise.
	* localedata/ru_RU.UTF-8.in: Likewise.
	* localedata/sc_IT.UTF-8.in: Likewise.
	* localedata/se_NO.UTF-8.in: Likewise.
	* localedata/sq_AL.UTF-8.in: Likewise.
	* localedata/sv_SE.UTF-8.in: Likewise.
	* localedata/szl_PL.UTF-8.in: Likewise.
	* localedata/tg_TJ.UTF-8.in: Likewise.
	* localedata/tk_TM.UTF-8.in: Likewise.
	* localedata/tt_RU.UTF-8.in: Likewise.
	* localedata/tt_RU.UTF-8@iqtelif.in: Likewise.
	* localedata/ug_CN.UTF-8.in: Likewise.
	* localedata/uz_UZ.UTF-8.in: Likewise.
	* localedata/vi_VN.UTF-8.in: Likewise.
	* localedata/yi_US.UTF-8.in: Likewise.
	* localedata/yo_NG.UTF-8.in: Likewise.
	* localedata/zh_CN.UTF-8.in: Likewise.
	* localedata/locales/am_ET: Adapt collation rules to new iso14651_t1_common
        file and fix bugs in the collation.
	* localedata/locales/az_AZ: Likewise.
	* localedata/locales/be_BY: Likewise.
	* localedata/locales/ber_DZ: Likewise.
	* localedata/locales/ber_MA: Likewise.
	* localedata/locales/bg_BG: Likewise.
	* localedata/locales/br_FR: Likewise.
	* localedata/locales/br_FR@euro: Likewise.
	* localedata/locales/ca_ES: Likewise.
	* localedata/locales/cns11643_stroke: Likewise.
	* localedata/locales/crh_UA: Likewise.
	* localedata/locales/cs_CZ: Likewise.
	* localedata/locales/csb_PL: Likewise.
	* localedata/locales/cv_RU: Likewise.
	* localedata/locales/cy_GB: Likewise.
	* localedata/locales/da_DK: Likewise.
	* localedata/locales/dz_BT: Likewise.
	* localedata/locales/en_CA: Likewise.
	* localedata/locales/eo: Likewise.
	* localedata/locales/es_CU: Likewise.
	* localedata/locales/es_EC: Likewise.
	* localedata/locales/es_ES: Likewise.
	* localedata/locales/es_US: Likewise.
	* localedata/locales/et_EE: Likewise.
	* localedata/locales/fa_IR: Likewise.
	* localedata/locales/fi_FI: Likewise.
	* localedata/locales/fil_PH: Likewise.
	* localedata/locales/fur_IT: Likewise.
	* localedata/locales/gez_ER@abegede: Likewise.
	* localedata/locales/ha_NG: Likewise.
	* localedata/locales/hr_HR: Likewise.
	* localedata/locales/hsb_DE: Likewise.
	* localedata/locales/hu_HU: Likewise.
	* localedata/locales/ig_NG: Likewise.
	* localedata/locales/ik_CA: Likewise.
	* localedata/locales/is_IS: Likewise.
	* localedata/locales/iso14651_t1_pinyin: Likewise.
	* localedata/locales/kk_KZ: Likewise.
	* localedata/locales/ku_TR: Likewise.
	* localedata/locales/ky_KG: Likewise.
	* localedata/locales/ln_CD: Likewise.
	* localedata/locales/lt_LT: Likewise.
	* localedata/locales/lv_LV: Likewise.
	* localedata/locales/mi_NZ: Likewise.
	* localedata/locales/ml_IN: Likewise.
	* localedata/locales/mn_MN: Likewise.
	* localedata/locales/mr_IN: Likewise.
	* localedata/locales/mt_MT: Likewise.
	* localedata/locales/nb_NO: Likewise.
	* localedata/locales/om_KE: Likewise.
	* localedata/locales/os_RU: Likewise.
	* localedata/locales/pl_PL: Likewise.
	* localedata/locales/ps_AF: Likewise.
	* localedata/locales/ro_RO: Likewise.
	* localedata/locales/ru_RU: Likewise.
	* localedata/locales/ru_UA: Likewise.
	* localedata/locales/sc_IT: Likewise.
	* localedata/locales/se_NO: Likewise.
	* localedata/locales/si_LK: Likewise.
	* localedata/locales/sq_AL: Likewise.
	* localedata/locales/sv_FI: Likewise.
	* localedata/locales/sv_FI@euro: Likewise.
	* localedata/locales/sv_SE: Likewise.
	* localedata/locales/szl_PL: Likewise.
	* localedata/locales/tg_TJ: Likewise.
	* localedata/locales/ti_ER: Likewise.
	* localedata/locales/tk_TM: Likewise.
	* localedata/locales/tl_PH: Likewise.
	* localedata/locales/tr_TR: Likewise.
	* localedata/locales/tt_RU: Likewise.
	* localedata/locales/tt_RU@iqtelif: Likewise.
	* localedata/locales/ug_CN: Likewise.
	* localedata/locales/uk_UA: Likewise.
	* localedata/locales/uz_UZ: Likewise.
	* localedata/locales/uz_UZ@cyrillic: Likewise.
	* localedata/locales/vi_VN: Likewise.
	* localedata/locales/yi_US: Likewise.
	* localedata/locales/yo_NG: Likewise.
2018-02-27 17:47:50 +01:00
Mike FABIAN
81be4b5e38 Remove --quiet argument when installing locales
Using this argument hides problems. I would like to see when something fails.

	* localedata/Makefile: Remove --quiet argument when
	installing locales
2018-02-23 09:51:56 +01:00
Joseph Myers
688903eb3e Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2018-01-01 00:32:25 +00:00
Mike FABIAN
d89756ebe1 lt_LT locale: Base collation on copy "iso14651_t1" [BZ #22524]
[BZ #22524]
	* localedata/Makefile: Add lt_LT.UTF-8 to test-input
	and to the list of locales to be built for testing.
	* localedata/lt_LT.UTF-8.in: New file for testing the collation.
	* localedata/locales/lt_LT (LC_COLLATE): Use “copy "iso14651_t1"”
	and build the collation rules upon that.
2017-12-07 15:38:11 +01:00
Mike FABIAN
62ea2193ee hsb_DE locale: Base collation on copy "iso14651_t1" [BZ #22515]
[BZ #22515]
	* localedata/Makefile: Add hsb_DE.UTF-8 to test-input
	and to the list of locales to be built for testing.
	* localedata/hsb_DE.UTF-8.in: New file for testing the collation.
	* localedata/locales/hsb_DE (LC_COLLATE): Use “copy "iso14651_t1"”
	and build the collation rules upon that.
2017-12-06 12:04:29 +01:00
Mike FABIAN
de9661d6be et_EE locale: Base collation on iso14651_t1 [BZ #22517]
[BZ #22517]
	* localedata/Makefile: Add et_EE.UTF-8 to test-input
	and to the list of locales to be built for testing.
	* localedata/et_EE.UTF-8.in: New file for testing the collation.
	* localedata/locales/et_EE (LC_COLLATE): Use “copy "iso14651_t1"”
        and build the collation rules upon that.
2017-12-05 17:10:08 +01:00
Mike FABIAN
d985adae22 is_IS locale: Base collation on iso14651_t1 [BZ #22519] 2017-12-01 14:08:58 +01:00
Mike FABIAN
fbb5fd03d3 sr_RS and bs_BA locales: make collation rules the same as for hr_HR [BZ #22534]
According to CLDR, collation rules for Serbian and Bosnian
	should be the same as for Croatian.

	[BZ #22534]
	* localedata/Makefile: Add sr_RS.UTF-8 and bs_BA.UTF-8 to test-input
	and to the list of locales to be built for testing.
	* localedata/bs_BA.UTF-8.in: New file (same as hr_HR.UTF-8.in).
	* localedata/sr_RS.UTF-8.in: New file (same as hr_HR.UTF-8.in).
	* localedata/locales/bs_BA (LC_COLLATE): Use “copy "hr_HR"”.
	* localedata/locales/sr_RS (LC_COLLATE): Use “copy "hr_HR"”.
2017-11-30 16:03:22 +01:00
Dragan Stanojević - Nevidljivi
9ca6b34378 Add test case for collation in hr_HR locale
* localedata/Makefile: Add hr_HR.UTF-8 to test-input and to
	the list of locales to built for testing.
	* localedata/hr_HR.UTF-8.in: New file.
2017-11-30 14:55:38 +01:00
Alexandre Oliva
8da25eec0a Collation fix: make forward accent sorting the default [BZ #17750]
[BZ #17750]
	* Makefile: add fr_CA.UTF-8 to test-input and LOCALES.
	* localedata/fr_CA.UTF-8.in: New file with test data for backward
	accents sorting.
	* localedata/fr_FR.UTF-8.in: Fix test data for forward accents
	sorting.
	* localedata/locales/cs_CZ (LC_COLLATE): Remove “define DIACRIT_FORWARD”
	* localedata/locales/de_DE (LC_COLLATE): Likewise.
	* localedata/locales/hu_HU (LC_COLLATE): Likewise.
	* localedata/locales/lb_LU (LC_COLLATE): Likewise.
	* localedata/locales/yuw_PG (LC_COLLATE): Likewise.
	* localedata/locales/fr_CA (LC_COLLATE): Add “define DIACRIT_BACKWARD”
	* localedata/locales/iso14651_t1_common: Use “ifdef DIACRIT_FORWARD”
	instead of “ifdef DIACRIT_BACKWARD”.

The only locale which currently needs backward accents sorting is fr_CA.
Therefore, forward accents sorting should be the default.

Before this patch, backwards accent sorting was the default and all
locales except fr_CA had to use

    define DIACRIT_FORWARD

before

    copy "iso14651_t1"

Most locales didn’t do that and thus got the inappropriate backwards accents sorting
by accident. Now only the fr_CA locale needs to use

    define DIACRIT_BACKWARD

before

    copy "iso14651_t1"

Original patch slightly modified by: Mike FABIAN <mfabian@redhat.com>
2017-11-29 11:56:46 +01:00
Siddhesh Poyarekar
8d7d3ba8c5 localedata: Remove duplicate cs_CZ from LOCALES
The LOCALES variable in the localedata had two instances of cs_CZ
which generated the following warning:

../gen-locales.mk:11: target '/opt/build/localedata/cs_CZ.UTF-8/LC_CTYPE' given more than once in the same rule

Dropped the duplicate entry.
2017-11-28 20:04:31 +05:30
Mike FABIAN
f9bb1ef233 cs_CZ locale: Base collation on iso14651_t1 [BZ #22336]
[BZ #22336]
	* localedata/locales/cs_CZ (LC_COLLATE): Use “copy "iso14651_t1"”
	and implement the collation rules for cs from CLDR on top of that.
	* Makefile: Add cs_CZ.UTF-8 to test-input and to the list
	of locales to be built for testing.
	* cs_CZ.UTF-8.in: New file with test data to test the Czech sorting.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2017-11-28 09:58:36 +01:00
Mike FABIAN
3ffc4cc1ad pl_PL locale: Base collation on iso14651_t1
[BZ #22469]
	* localedata/locales/pl_PL (LC_COLLATE): Use “copy "iso14651_t1"”
	and implement the collation rules for pl from CLDR on top of that.
	* Makefile: Add pl_PL.UTF-8 to test-input and to the list
	of locales to be built for testing.
	* pl_PL.UTF-8.in: New file with test data to test the Polish sorting.
2017-11-24 10:07:05 +05:30
Mike FABIAN
4b7af5fca7 lv_LV locale: fix collation [BZ #15537]
[BZ #15537]
	* localedata/locales/lv_LV (LC_COLLATE): Fix collation by
	using “copy "iso14651_t1"” and then implementing the
	collation rules for lv from CLDR on top of that.
	* Makefile: Add lv_LV.UTF-8 to test-input and to the list
	of locales to be built for testing.
	* lv_LV.UTF-8.in: New file with test data to test the Latvian
	sorting.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2017-11-22 10:31:47 +05:30
Carlos O'Donell
02eec68167 localedef: Add --no-warnings/--warnings option
From localedef --help:

Output control:
...
      --no-warnings=<warnings>   Comma-separated list of warnings to disable;
                             supported warnings are: ascii, intcurrsym
...
      --warnings=<warnings>  Comma-separated list of warnings to enable;
                             supported warnings are: ascii, intcurrsym

Locales using SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII
compatible. In order to build locales using these character maps, and
have localedef exit with a status of 0, we add new option to localedef
to disable or enable specific warnings. The options are --no-warnings
and --warnings, to disable and enable specific warnings respectively.
The options take a comma-separated list of warning names. The warning
names are taken directly from the generated warning.  When a warning
that can be disabled is issued it will print something like this: foo is
not defined [--no-warnings=foo]

For the initial implementation we add two controllable warnings; first
'ascii' which is used by the localedata installation makefile target to
install SHIFT_JIS and SHIFT_JISX0213-using locales without error; second
'intcurrsym' which allows a program to use a non-standard international
currency symbol without triggering a warning.  The 'intcurrsym' is
useful in the future if country codes are added that are not in our
current ISO 4217 list, and the user wants to avoid the warning. Having
at least two warnings to control gives an example for how the changes
can be extended to more warnings if required in the future.

These changes allow ja_JP.SHIFT_JIS and ja_JP.SHIFT_JISX0213 to be
compiled without warnings using --no-warnings=ascii. The
localedata/Makefile $(INSTALL-SUPPORTED-LOCALES) target is adjusted to
automatically add `--no-warnings=ascii` for such charmaps, and likewise
localedata/gen-locale.sh is adjusted with similar logic.

v2: Bring verbose, be_quiet, and all warning control booleans into
record-status.c, and compile this object file to be used by locale,
iconv, and localedef. Any users include record-status.h.
v3: Fix an instance of boolean coercion in set_warning().

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
2017-10-25 13:36:54 -07:00
Carlos O'Donell
56fa555a83 localedata: Locale and test name are the same.
The localedata collation test data is encoded in a particular
character set. We rename the test data to match the full locale
name with encoding, and adjust the Makefile and sort-test.sh
script. This allows us to have a future C.UTF-8 test that is
disambiguated from the built-in C locale.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
2017-10-25 12:24:31 -07:00
Andreas Schwab
d5afb38503 Fix missing redirects in testsuite targets 2017-08-07 18:13:36 +02:00
Steve Ellcey
b7f95f493b Remove CFLAG settings that turn off warning (no longer needed).
2017-08-02  Steve Ellcey  <sellcey@cavium.com>

	* localedata/Makefile (CFLAGS-tst_iswalnum.c, CFLAGS-tst_iswalpha.c
	CFLAGS-tst_iswcntrl.c, CFLAGS-tst_iswdigit.c, CFLAGS-tst_iswgraph.c,
	CFLAGS-tst_iswlower.c, CFLAGS-tst_iswprint.c, CFLAGS-tst_iswpunct.c,
	CFLAGS-tst_iswspace.c, CFLAGS-tst_iswupper.c, CFLAGS-tst_iswxdigit.c,
	CFLAGS-tst_towlower.c, CFLAGS-tst_towupper.c): Remove.
2017-08-02 12:57:58 -07:00
Steve Ellcey
6259e62c02 Fix localedata test builds with latest GCC
* localedata/Makefile (CFLAGS-tst_iswalnum.c, CFLAGS-tst_iswalpha.c
	CFLAGS-tst_iswcntrl.c, CFLAGS-tst_iswdigit.c, CFLAGS-tst_iswgraph.c,
	CFLAGS-tst_iswlower.c, CFLAGS-tst_iswprint.c, CFLAGS-tst_iswpunct.c,
	CFLAGS-tst_iswspace.c, CFLAGS-tst_iswupper.c, CFLAGS-tst_iswxdigit.c,
	CFLAGS-tst_towlower.c, CFLAGS-tst_towupper.c): New macros.
2017-07-21 10:33:30 -07:00
Egmont Koblinger
ea1898dded localedata: hu_HU: fix multiple sorting bugs (bug 18934)
Fix the incorrect sorting order of a digraph and its geminated variant,
regression introduced by a faulty fix to bug 13547 in commit
b008d4c856.

Fix two inconsistencies in sorting unusual capitalization of digraphs
(bug #18587).

Enable DIACRIT_FORWARD to work around bug #17750.

Sort foreign accents after the Hungarian ones.

Add extensive unittests containing all the examples from The Rules of
Hungarian Orthography and many more, including explanatory comments.
2017-03-28 10:35:58 -04:00
Joseph Myers
bfff8b1bec Update copyright dates with scripts/update-copyrights. 2017-01-01 00:14:16 +00:00
Joseph Myers
f7a9f785e5 Update copyright dates with scripts/update-copyrights. 2016-01-04 16:05:18 +00:00
Carlos O'Donell
90fe682d30 Rename localedir to complocaledir (bug 14259).
In preparation to fix the --localedir configure argument we must
move the existing conflicting definition of localedir to a more
appropriate name. Given that all current internal uses of localedir
relate to the compiled locales we rename to complocaledir.
2015-11-27 10:22:38 -05:00
Andriy Rysin
6afb9c0175 Fix sorting order for Ukrainian locale (BZ 17293)
In the introduction for the official orthography rules for Ukrainian
language (http://spelling.ulif.org.ua/peredmova.htm) there's a note
that only apostrophe does not affect order of the words when sorting.
As could be seen from the official alphabet the soft sign
(U+044C/U+042C) has its hard position and thus affects the order and
also letters "е" and "є" (CYR-IE: U+0435/U+0415 and UKR-IE:
U+0454/U+0404) have their own positions and should have separate place
when sorting.
This also corresponds to official Unicode collation chart for these
letters: http://unicode.org/charts/collation/chart_Cyrillic.html
2015-05-26 23:51:18 +05:30
Carlos O'Donell
6c307927ac Fail locale installation if localedef fails.
If any locale fails to compile then the installation
of locales via `make localedata/install-locales`
also fails.
2015-05-16 02:14:49 -04:00
Leonhard Holz
9f53d7ad57 Split locale generation snippet into a separate file
This patch prepares for the strcoll benchmark by moving the makefile
code for generating the locale files into a standalone snippet that
can be used elsewhere.
2015-05-13 13:05:28 +05:30
Roland McGrath
9162c01d09 Avoid re-exec-self in bug-setlocale1. 2015-03-05 12:58:49 -08:00
Joseph Myers
b168057aaa Update copyright dates with scripts/update-copyrights. 2015-01-02 16:29:47 +00:00
Joseph Myers
47d51f4f07 Clean up localedata tests printf formats, don't use -Wno-format. 2014-12-11 13:15:08 -08:00
Will Newton
85bb81c91b Allow cross-building of tests
Allow building tests in a cross configuration without a test wrapper
defined. This is helpful for doing simple build testing of tests.

ChangeLog:

2014-09-30  Will Newton  <will.newton@linaro.org>

	* localedata/Makefile: Move assignment to tests-special
	into an ifdef testing run-built-tests.
	* timezone/Makefile: Likewise.
2014-09-30 15:02:41 +01:00
Florian Weimer
4e8f95a0df _nl_find_locale: Improve handling of crafted locale names [BZ #17137]
Prevent directory traversal in locale-related environment variables
(CVE-2014-0475).
2014-07-10 16:29:55 +02:00
Stefan Liebler
d8c2869d44 Generate en_GB.UTF-8 during testing
This patch generates en_GB.UTF-8, which is used in
tst-strcoll-overflow.  This solves the issue of failing xtests on
s390/s390x.
2014-06-20 07:44:42 +05:30
Joseph Myers
85d8927856 Use $(rtld-prefix) more consistently.
The glibc makefiles have a standard variable, $(rtld-prefix), to run
the dynamic linker with a default --library-path option; this is used
as the basis of lots of other variables for running programs compiled
with the newly built library.

A few places however use $(elf-objpfx)ld.so or
$(elf-objpfx)${rtld-installed-name} directly, with such a
--library-path option.  This patch makes such places use
$(rtld-prefix) instead.  I'm not aware of any significance in these
cases to the choice of ld.so or ${rtld-installed-name} when running
the dynamic linker, or to whether $(patsubst
%,:%,$(sysdep-library-path)) is included in the library-path as it is
in $(rtld-prefix) and just one of the places being changed.

Tested x86_64.

	* elf/Makefile ($(objpfx)tst-unused-dep.out): Use $(rtld-prefix).
	* iconvdata/Makefile ($(inst_gconvdir)/gconv-modules)
	[$(cross-compiling) = no]: Likewise.
	* sysdeps/s390/s390-64/Makefile ($(inst_gconvdir)/gconv-modules)
	[$(cross-compiling) = no]: Likewise.

localedata/ChangeLog:
	* Makefile (LOCALEDEF): Use $(rtld-prefix).
2014-06-18 10:35:29 +00:00
Joseph Myers
2976feab1f Remove special makefile rules / .sh files for some localedata tests.
The localedata tests tst-mbswcs and tst-wctype use custom .sh scripts
and makefile rules, but have no need to do so.  tst-mbswcs.sh runs a
series of test programs in succession (and nothing special is done
with the output of the programs); this patch makes the separate tests
into ordinary tests run directly by the usual makefile rules.
tst-wctype.sh runs one test with an environment variable and input
redirection; generic makefile rules also cover that, so again this
patch converts it into an ordinary test.  (The makefile dependency of
tst-wctype.out on sort-test.out that this patch removes appears to be
a cut-and-paste error; the test does not appear to use that file.
There is already a generic dependency of ordinary tests in this
directory on $(addprefix $(objpfx),$(CTYPE_FILES)).)

Tested x86_64.

localedata/ChangeLog:
	* Makefile (test-srcs): Remove tst-mbswcs1, tst-mbswcs2,
	tst-mbswcs3, tst-mbswcs4, tst-mbswcs5 and tst-wctype.
	(generated): Remove tst-mbswcs.out.
	(tests): Add tst-mbswcs1, tst-mbswcs2, tst-mbswcs3, tst-mbswcs4,
	tst-mbswcs5 and tst-wctype.
	(tests-special): Remove $(objpfx)tst-mbswcs.out and
	$(objpfx)tst-wctype.out.
	($(objpfx)tst-mbswcs.out): Remove rule.
	($(objpfx)tst-wctype.out): Likewise.
	(tst-wctype-ENV): New variable.
	* tst-mbswcs.sh: Remove file.
	* tst-wctype.sh: Likewise.
2014-06-09 18:19:44 +00:00
Joseph Myers
8540f6d2a7 Don't require test wrappers to preserve environment variables, use more consistent environment.
One wart in the original support for test wrappers for cross testing,
as noted in
<https://sourceware.org/ml/libc-alpha/2012-10/msg00722.html>, is the
requirement for test wrappers to pass a poorly-defined set of
environment variables from the build system to the system running the
glibc under test.  Although some variables are passed explicitly via
$(test-wrapper-env), including LD_* variables that simply can't be
passed implicitly because of the side effects they'd have on the build
system's dynamic linker, others are passed implicitly, including
variables such as GCONV_PATH and LOCPATH that could potentially affect
the build system's libc (so effectively relying on any such effects
not breaking the wrappers).  In addition, the code in
cross-test-ssh.sh for preserving environment variables is fragile (it
depends on how bash formats a list of exported variables, and could
well break for multi-line variable definitions where the contents
contain things looking like other variable definitions).

This patch moves to explicitly passing environment variables via
$(test-wrapper-env).  Makefile variables that previously used
$(test-wrapper) are split up into -before-env and -after-env parts
that can be passed separately to the various .sh files used in
testing, so those files can then insert environment settings between
the two parts.

The common default environment settings in make-test-out are made into
a separate makefile variable that can also be passed to scripts,
rather than many scripts duplicating those settings (for testing an
installed glibc, it is desirable to have the GCONV_PATH setting on
just one place, so just that one place needs to support it pointing to
an installed sysroot instead of the build tree).  The default settings
are included in the variables such as $(test-program-prefix), so that
if tests do not need any non-default settings they can continue to use
single variables rather than the split-up variables.

Although this patch cleans up LC_ALL=C settings (that being part of
the common defaults), various LANG=C and LANGUAGE=C settings remain.
Those are generally unnecessary and I propose a subsequent cleanup to
remove them.  LC_ALL takes precedence over LANG, and while LANGUAGE
takes precedence over LC_ALL, it only does so for settings other than
LC_ALL=C.  So LC_ALL=C on its own is sufficient to ensure the C
locale, and anything that gets LC_ALL=C does not need the other
settings.

While preparing this patch I noticed some tests with .sh files that
appeared to do nothing beyond what the generic makefile support for
tests can do (localedata/tst-wctype.sh - the makefiles support -ENV
variables and .input files - and localedata/tst-mbswcs.sh - just runs
five tests that could be run individually from the makefile).  So I
propose another subsequent cleanup to move those to using the generic
support instead of special .sh files.

Tested x86_64 (native) and powerpc32 (cross).

	* Makeconfig (run-program-env): New variable.
	(run-program-prefix-before-env): Likewise.
	(run-program-prefix-after-env): Likewise.
	(run-program-prefix): Define in terms of new variables.
	(built-program-cmd-before-env): New variable.
	(built-program-cmd-after-env): Likewise.
	(built-program-cmd): Define in terms of new variables.
	(test-program-prefix-before-env): New variable.
	(test-program-prefix-after-env): Likewise.
	(test-program-prefix): Define in terms of new variables.
	(test-program-cmd-before-env): New variable.
	(test-program-cmd-after-env): Likewise.
	(test-program-cmd): Define in terms of new variables.
	* Rules (make-test-out): Use $(run-program-env).
	* scripts/cross-test-ssh.sh (env_blacklist): Remove variable.
	(help): Do not mention environment variables.  Mention
	--timeoutfactor option.
	(timeoutfactor): New variable.
	(blacklist_exports): Remove function.
	(exports): Remove variable.
	(command): Do not include ${exports}.
	* manual/install.texi (Configuring and compiling): Do not mention
	test wrappers preserving environment variables.  Mention that last
	assignment to a variable must take precedence.
	* INSTALL: Regenerated.
	* benchtests/Makefile (run-bench): Use $(run-program-env).
	* catgets/Makefile ($(objpfx)test1.cat): Use
	$(built-program-cmd-before-env), $(run-program-env) and
	$(built-program-cmd-after-env).
	($(objpfx)test2.cat): Do not specify environment variables
	explicitly.
	($(objpfx)de/libc.cat): Use $(built-program-cmd-before-env),
	$(run-program-env) and $(built-program-cmd-after-env).
	($(objpfx)test-gencat.out): Use $(test-program-cmd-before-env),
	$(run-program-env) and $(test-program-cmd-after-env).
	($(objpfx)sample.SJIS.cat): Do not specify environment variables
	explicitly.
	* catgets/test-gencat.sh: Use test_program_cmd_before_env,
	run_program_env and test_program_cmd_after_env arguments.
	* elf/Makefile ($(objpfx)tst-pathopt.out): Use $(run-program-env).
	* elf/tst-pathopt.sh: Use run_program_env argument.
	* iconvdata/Makefile ($(objpfx)iconv-test.out): Use
	$(test-wrapper-env) and $(run-program-env).
	* iconvdata/run-iconv-test.sh: Use test_wrapper_env and
	run_program_env arguments.
	* iconvdata/tst-table.sh: Do not set GCONV_PATH explicitly.
	* intl/Makefile ($(objpfx)tst-gettext.out): Use
	$(test-program-prefix-before-env), $(run-program-env) and
	$(test-program-prefix-after-env).
	($(objpfx)tst-gettext2.out): Likewise.
	* intl/tst-gettext.sh: Use test_program_prefix_before_env,
	run_program_env and test_program_prefix_after_env arguments.
	* intl/tst-gettext2.sh: Likewise.
	* intl/tst-gettext4.sh: Do not set environment variables
	explicitly.
	* intl/tst-gettext6.sh: Likewise.
	* intl/tst-translit.sh: Likewise.
	* malloc/Makefile ($(objpfx)tst-mtrace.out): Use
	$(test-program-prefix-before-env), $(run-program-env) and
	$(test-program-prefix-after-env).
	* malloc/tst-mtrace.sh: Use test_program_prefix_before_env,
	run_program_env and test_program_prefix_after_env arguments.
	* math/Makefile (run-regen-ulps): Use $(run-program-env).
	* nptl/Makefile ($(objpfx)tst-tls6.out): Use $(run-program-env).
	* nptl/tst-tls6.sh: Use run_program_env argument.  Set LANG=C
	explicitly with each use of ${test_wrapper_env}.
	* posix/Makefile ($(objpfx)wordexp-tst.out): Use
	$(test-program-prefix-before-env), $(run-program-env) and
	$(test-program-prefix-after-env).
	* posix/tst-getconf.sh: Do not set environment variables
	explicitly.
	* posix/wordexp-tst.sh: Use test_program_prefix_before_env,
	run_program_env and test_program_prefix_after_env arguments.
	* stdio-common/tst-printf.sh: Do not set environment variables
	explicitly.
	* stdlib/Makefile ($(objpfx)tst-fmtmsg.out): Use
	$(test-program-prefix-before-env), $(run-program-env) and
	$(test-program-prefix-after-env).
	* stdlib/tst-fmtmsg.sh: Use test_program_prefix_before_env,
	run_program_env and test_program_prefix_after_env arguments.
	Split $test calls into $test_pre and $test.
	* timezone/Makefile (build-testdata): Use
	$(built-program-cmd-before-env), $(run-program-env) and
	$(built-program-cmd-after-env).

localedata/ChangeLog:
	* Makefile ($(addprefix $(objpfx),$(CTYPE_FILES))): Use
	$(built-program-cmd-before-env), $(run-program-env) and
	$(built-program-cmd-after-env).
	($(objpfx)sort-test.out): Use $(test-program-prefix-before-env),
	$(run-program-env) and $(test-program-prefix-after-env).
	($(objpfx)tst-fmon.out): Use $(run-program-prefix-before-env),
	$(run-program-env) and $(run-program-prefix-after-env).
	($(objpfx)tst-locale.out): Use $(built-program-cmd-before-env),
	$(run-program-env) and $(built-program-cmd-after-env).
	($(objpfx)tst-trans.out): Use $(run-program-prefix-before-env),
	$(run-program-env), $(run-program-prefix-after-env),
	$(test-program-prefix-before-env) and
	$(test-program-prefix-after-env).
	($(objpfx)tst-ctype.out): Use $(test-program-cmd-before-env),
	$(run-program-env) and $(test-program-cmd-after-env).
	($(objpfx)tst-wctype.out): Likewise.
	($(objpfx)tst-langinfo.out): Likewise.
	($(objpfx)tst-langinfo-static.out): Likewise.
	* gen-locale.sh: Use localedef_before_env, run_program_env and
	localedef_after_env arguments.
	* sort-test.sh: Use test_program_prefix_before_env,
	run_program_env and test_program_prefix_after_env arguments.
	* tst-ctype.sh: Use tst_ctype_before_env, run_program_env and
	tst_ctype_after_env arguments.
	* tst-fmon.sh: Use run_program_prefix_before_env, run_program_env
	and run_program_prefix_after_env arguments.
	* tst-langinfo.sh: Use tst_langinfo_before_env, run_program_env
	and tst_langinfo_after_env arguments.
	* tst-locale.sh: Use localedef_before_env, run_program_env and
	localedef_after_env arguments.
	* tst-mbswcs.sh: Do not set environment variables explicitly.
	* tst-numeric.sh: Likewise.
	* tst-rpmatch.sh: Likewise.
	* tst-trans.sh: Use run_program_prefix_before_env,
	run_program_env, run_program_prefix_after_env,
	test_program_prefix_before_env and test_program_prefix_after_env
	arguments.
	* tst-wctype.sh: Use tst_wctype_before_env, run_program_env and
	tst_wctype_after_env arguments.
2014-06-06 22:19:27 +00:00