The new tst-localedef-hardlinks verifies that when compiling
two locales (with default output directory) one with
--no-hard-links and one without the option, results in the
expected behaviour. When --no-hard-links is used the link
counts on LC_CTYPE is 1, indicating that even thoug the two
locale are identical (though different named source files and
output direcotry) the localedef did not carry out the hard
link optimization. Then when --no-hard-links is omitted the
localedef hard link optimization is correctly carried out and
for 2 compiled locales the link count for LC_CTYPE is 2.
Reviewed-by: DJ Delorie <dj@redhat.com>
This commit adds previously missing transliterations for several code points
in the Unicode blocks "Miscellaneous Mathematical Symbols-A/B" -
transliterated to their approximate ASCII representations. It also adds a
corresponding iconv transliteration test.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The testroot does not have a gunzip command, so the charmap files
should not be installed gzipped else they cannot be used (and thus
tested). With this patch, installing with INSTALL_UNCOMPRESSED=yes
installs uncompressed charmaps instead.
Note that we must purge the $(symbolic_link_list) as it contains
references to $(DESTDIR), which we change during the testroot
installation.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
According to CLDR 35.1 and the bug report the correct monetary format
for negative amounts should be "EUR -1 234,56" while previously it was
"EUR 1 234,56-".
This patch does not change the thousands (grouping) separator.
[BZ #24614]
* localedata/Makefile (LOCALES): Add nl_NL.UTF-8.
* localedata/locales/nl_NL (n_sep_by_space): Set to 2 (a space
between the currency symbol and the minus sign).
(n_sign_posn): Set to 4 (the minus sign after the currency symbol).
* localedata/tst-strfmon1.c (tests): Add test data for nl_NL.UTF-8.
Downstream distributions need consistent sets of hardlinks in
order for rpm to operate effectively. This means that even if
locales are built with a high level of parallelism that the
resulting files need to have consistent hardlink counts. The only
way to achieve this is with a post-install hardlink pass using a
program like 'hardlink' (shipped in Fedora).
If the downstream distro wants to post-process the hardlinks then
the time spent in localedef looking up sibling directories and
processing hardlinks is wasted effort.
To optimize the build and install pass we add a --no-hard-links
option to localedef to avoid doing the hardlink optimziation for
size.
Tested on x86_64 with 'make localedata/install-locale-files'
before and after. Without the patch we have files with 100+
hardlink counts. After the patch and running with --no-hard-links
all link counts are 1. This patch also alters the convenience
target 'make localedata/install-locale-files' to use the new
option.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
The convenience install target 'install-locale-files' is created
to allow distributions to install all of the SUPPORTED locales as
files instead of into the locale-archive.
You invoke the new convenience target like this:
make localedata/install-locale-files DESTDIR=<prefix>
In commit 9479b6d5e0 we updated all of
the collation data to harmonize with the new version of ISO 14651
which is derived from Unicode 9.0.0. This collation update brought
with it some changes to locales which were not desirable by some
users, in particular it altered the meaning of the
locale-dependent-range regular expression, namely [a-z] and [A-Z], and
for en_US it caused uppercase letters to be matched by [a-z] for the
first time. The matching of uppercase letters by [a-z] is something
which is already known to users of other locales which have this
property, but this change could cause significant problems to en_US
and other similar locales that had never had this change before.
Whether this behaviour is desirable or not is contentious and GNU Awk
has this to say on the topic:
https://www.gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html
While the POSIX standard also has this further to say: "RE Bracket
Expression":
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap09.html
"The current standard leaves unspecified the behavior of a range
expression outside the POSIX locale. ... As noted above, efforts were
made to resolve the differences, but no solution has been found that
would be specific enough to allow for portable software while not
invalidating existing implementations."
In glibc we implement the requirement of ISO POSIX-2:1993 and use
collation element order (CEO) to construct the range expression, the
API internally is __collseq_table_lookup(). The fact that we use CEO
and also have 4-level weights on each collation rule means that we can
in practice reorder the collation rules in iso14651_t1_common (the new
data) to provide consistent range expression resolution *and* the
weights should maintain the expected total order. Therefore this
patch does three things:
* Reorder the collation rules for the LATIN script in
iso14651_t1_common to deinterlace uppercase and lowercase letters in
the collation element orders.
* Adds new test data en_US.UTF-8.in for sort-test.sh which exercises
strcoll* and strxfrm* and ensures the ISO 14651 collation remains.
* Add back tests to tst-fnmatch.input and tst-regexloc.c which
exercise that [a-z] does not match A or Z.
The reordering of the ISO 14651 data is done in an entirely mechanical
fashion using the following program attached to the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28
It is up for discussion if the iso14651_t1_common data should be
refined further to have 3 very tight collation element ranges that
include only a-z, A-Z, and 0-9, which would implement the solution
sought after in:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c12
and implemented here:
https://www.sourceware.org/ml/libc-alpha/2018-07/msg00854.html
No regressions on x86_64.
Verified that removal of the iso14651_t1_common change causes tst-fnmatch
to regress with:
422: fnmatch ("[a-z]", "A", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
...
425: fnmatch ("[A-Z]", "z", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
Fixed syntax error in the collation rules of Lower Sorbian language.
Collation test added in order to test the bugs like this early.
Reported-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
[BZ #23208]
* localedata/Makefile (test-input): Add dsb_DE.UTF-8.
(LOCALES): Likewise.
* localedata/dsb_DE.UTF-8.in: New file.
* localedata/locales/dsb_DE (LC_COLLATE): Fix syntax error.
There is a glibc optimization which allows for locale categories
to be removed during static compilation. There have been various
bugs for this support over the years, with bug 16915 being the
most recent. The solution there was to emit a reference to all the
categories to avoid any being removed. This fix, although it's in
the generic __nl_langinfo_l function, doesn't appear to be enough
to fix the case for a statically linked program that uses newlocale
and nl_langinfo_l. This commit doesn't fix the problem, but it does
add a XFAIL'd test case such that a fix can be applied against this
and the XFAIL removed. It's not entirely clear that the problem is
the same as that which was seen in bug 16915.
[BZ #22524]
* localedata/Makefile: Add lt_LT.UTF-8 to test-input
and to the list of locales to be built for testing.
* localedata/lt_LT.UTF-8.in: New file for testing the collation.
* localedata/locales/lt_LT (LC_COLLATE): Use “copy "iso14651_t1"”
and build the collation rules upon that.
[BZ #22515]
* localedata/Makefile: Add hsb_DE.UTF-8 to test-input
and to the list of locales to be built for testing.
* localedata/hsb_DE.UTF-8.in: New file for testing the collation.
* localedata/locales/hsb_DE (LC_COLLATE): Use “copy "iso14651_t1"”
and build the collation rules upon that.
[BZ #22517]
* localedata/Makefile: Add et_EE.UTF-8 to test-input
and to the list of locales to be built for testing.
* localedata/et_EE.UTF-8.in: New file for testing the collation.
* localedata/locales/et_EE (LC_COLLATE): Use “copy "iso14651_t1"”
and build the collation rules upon that.
According to CLDR, collation rules for Serbian and Bosnian
should be the same as for Croatian.
[BZ #22534]
* localedata/Makefile: Add sr_RS.UTF-8 and bs_BA.UTF-8 to test-input
and to the list of locales to be built for testing.
* localedata/bs_BA.UTF-8.in: New file (same as hr_HR.UTF-8.in).
* localedata/sr_RS.UTF-8.in: New file (same as hr_HR.UTF-8.in).
* localedata/locales/bs_BA (LC_COLLATE): Use “copy "hr_HR"”.
* localedata/locales/sr_RS (LC_COLLATE): Use “copy "hr_HR"”.
[BZ #17750]
* Makefile: add fr_CA.UTF-8 to test-input and LOCALES.
* localedata/fr_CA.UTF-8.in: New file with test data for backward
accents sorting.
* localedata/fr_FR.UTF-8.in: Fix test data for forward accents
sorting.
* localedata/locales/cs_CZ (LC_COLLATE): Remove “define DIACRIT_FORWARD”
* localedata/locales/de_DE (LC_COLLATE): Likewise.
* localedata/locales/hu_HU (LC_COLLATE): Likewise.
* localedata/locales/lb_LU (LC_COLLATE): Likewise.
* localedata/locales/yuw_PG (LC_COLLATE): Likewise.
* localedata/locales/fr_CA (LC_COLLATE): Add “define DIACRIT_BACKWARD”
* localedata/locales/iso14651_t1_common: Use “ifdef DIACRIT_FORWARD”
instead of “ifdef DIACRIT_BACKWARD”.
The only locale which currently needs backward accents sorting is fr_CA.
Therefore, forward accents sorting should be the default.
Before this patch, backwards accent sorting was the default and all
locales except fr_CA had to use
define DIACRIT_FORWARD
before
copy "iso14651_t1"
Most locales didn’t do that and thus got the inappropriate backwards accents sorting
by accident. Now only the fr_CA locale needs to use
define DIACRIT_BACKWARD
before
copy "iso14651_t1"
Original patch slightly modified by: Mike FABIAN <mfabian@redhat.com>
The LOCALES variable in the localedata had two instances of cs_CZ
which generated the following warning:
../gen-locales.mk:11: target '/opt/build/localedata/cs_CZ.UTF-8/LC_CTYPE' given more than once in the same rule
Dropped the duplicate entry.
[BZ #22336]
* localedata/locales/cs_CZ (LC_COLLATE): Use “copy "iso14651_t1"”
and implement the collation rules for cs from CLDR on top of that.
* Makefile: Add cs_CZ.UTF-8 to test-input and to the list
of locales to be built for testing.
* cs_CZ.UTF-8.in: New file with test data to test the Czech sorting.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
[BZ #22469]
* localedata/locales/pl_PL (LC_COLLATE): Use “copy "iso14651_t1"”
and implement the collation rules for pl from CLDR on top of that.
* Makefile: Add pl_PL.UTF-8 to test-input and to the list
of locales to be built for testing.
* pl_PL.UTF-8.in: New file with test data to test the Polish sorting.
[BZ #15537]
* localedata/locales/lv_LV (LC_COLLATE): Fix collation by
using “copy "iso14651_t1"” and then implementing the
collation rules for lv from CLDR on top of that.
* Makefile: Add lv_LV.UTF-8 to test-input and to the list
of locales to be built for testing.
* lv_LV.UTF-8.in: New file with test data to test the Latvian
sorting.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
From localedef --help:
Output control:
...
--no-warnings=<warnings> Comma-separated list of warnings to disable;
supported warnings are: ascii, intcurrsym
...
--warnings=<warnings> Comma-separated list of warnings to enable;
supported warnings are: ascii, intcurrsym
Locales using SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII
compatible. In order to build locales using these character maps, and
have localedef exit with a status of 0, we add new option to localedef
to disable or enable specific warnings. The options are --no-warnings
and --warnings, to disable and enable specific warnings respectively.
The options take a comma-separated list of warning names. The warning
names are taken directly from the generated warning. When a warning
that can be disabled is issued it will print something like this: foo is
not defined [--no-warnings=foo]
For the initial implementation we add two controllable warnings; first
'ascii' which is used by the localedata installation makefile target to
install SHIFT_JIS and SHIFT_JISX0213-using locales without error; second
'intcurrsym' which allows a program to use a non-standard international
currency symbol without triggering a warning. The 'intcurrsym' is
useful in the future if country codes are added that are not in our
current ISO 4217 list, and the user wants to avoid the warning. Having
at least two warnings to control gives an example for how the changes
can be extended to more warnings if required in the future.
These changes allow ja_JP.SHIFT_JIS and ja_JP.SHIFT_JISX0213 to be
compiled without warnings using --no-warnings=ascii. The
localedata/Makefile $(INSTALL-SUPPORTED-LOCALES) target is adjusted to
automatically add `--no-warnings=ascii` for such charmaps, and likewise
localedata/gen-locale.sh is adjusted with similar logic.
v2: Bring verbose, be_quiet, and all warning control booleans into
record-status.c, and compile this object file to be used by locale,
iconv, and localedef. Any users include record-status.h.
v3: Fix an instance of boolean coercion in set_warning().
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
The localedata collation test data is encoded in a particular
character set. We rename the test data to match the full locale
name with encoding, and adjust the Makefile and sort-test.sh
script. This allows us to have a future C.UTF-8 test that is
disambiguated from the built-in C locale.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Fix the incorrect sorting order of a digraph and its geminated variant,
regression introduced by a faulty fix to bug 13547 in commit
b008d4c856.
Fix two inconsistencies in sorting unusual capitalization of digraphs
(bug #18587).
Enable DIACRIT_FORWARD to work around bug #17750.
Sort foreign accents after the Hungarian ones.
Add extensive unittests containing all the examples from The Rules of
Hungarian Orthography and many more, including explanatory comments.
In preparation to fix the --localedir configure argument we must
move the existing conflicting definition of localedir to a more
appropriate name. Given that all current internal uses of localedir
relate to the compiled locales we rename to complocaledir.
In the introduction for the official orthography rules for Ukrainian
language (http://spelling.ulif.org.ua/peredmova.htm) there's a note
that only apostrophe does not affect order of the words when sorting.
As could be seen from the official alphabet the soft sign
(U+044C/U+042C) has its hard position and thus affects the order and
also letters "е" and "є" (CYR-IE: U+0435/U+0415 and UKR-IE:
U+0454/U+0404) have their own positions and should have separate place
when sorting.
This also corresponds to official Unicode collation chart for these
letters: http://unicode.org/charts/collation/chart_Cyrillic.html
This patch prepares for the strcoll benchmark by moving the makefile
code for generating the locale files into a standalone snippet that
can be used elsewhere.
Allow building tests in a cross configuration without a test wrapper
defined. This is helpful for doing simple build testing of tests.
ChangeLog:
2014-09-30 Will Newton <will.newton@linaro.org>
* localedata/Makefile: Move assignment to tests-special
into an ifdef testing run-built-tests.
* timezone/Makefile: Likewise.
The glibc makefiles have a standard variable, $(rtld-prefix), to run
the dynamic linker with a default --library-path option; this is used
as the basis of lots of other variables for running programs compiled
with the newly built library.
A few places however use $(elf-objpfx)ld.so or
$(elf-objpfx)${rtld-installed-name} directly, with such a
--library-path option. This patch makes such places use
$(rtld-prefix) instead. I'm not aware of any significance in these
cases to the choice of ld.so or ${rtld-installed-name} when running
the dynamic linker, or to whether $(patsubst
%,:%,$(sysdep-library-path)) is included in the library-path as it is
in $(rtld-prefix) and just one of the places being changed.
Tested x86_64.
* elf/Makefile ($(objpfx)tst-unused-dep.out): Use $(rtld-prefix).
* iconvdata/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Likewise.
* sysdeps/s390/s390-64/Makefile ($(inst_gconvdir)/gconv-modules)
[$(cross-compiling) = no]: Likewise.
localedata/ChangeLog:
* Makefile (LOCALEDEF): Use $(rtld-prefix).
The localedata tests tst-mbswcs and tst-wctype use custom .sh scripts
and makefile rules, but have no need to do so. tst-mbswcs.sh runs a
series of test programs in succession (and nothing special is done
with the output of the programs); this patch makes the separate tests
into ordinary tests run directly by the usual makefile rules.
tst-wctype.sh runs one test with an environment variable and input
redirection; generic makefile rules also cover that, so again this
patch converts it into an ordinary test. (The makefile dependency of
tst-wctype.out on sort-test.out that this patch removes appears to be
a cut-and-paste error; the test does not appear to use that file.
There is already a generic dependency of ordinary tests in this
directory on $(addprefix $(objpfx),$(CTYPE_FILES)).)
Tested x86_64.
localedata/ChangeLog:
* Makefile (test-srcs): Remove tst-mbswcs1, tst-mbswcs2,
tst-mbswcs3, tst-mbswcs4, tst-mbswcs5 and tst-wctype.
(generated): Remove tst-mbswcs.out.
(tests): Add tst-mbswcs1, tst-mbswcs2, tst-mbswcs3, tst-mbswcs4,
tst-mbswcs5 and tst-wctype.
(tests-special): Remove $(objpfx)tst-mbswcs.out and
$(objpfx)tst-wctype.out.
($(objpfx)tst-mbswcs.out): Remove rule.
($(objpfx)tst-wctype.out): Likewise.
(tst-wctype-ENV): New variable.
* tst-mbswcs.sh: Remove file.
* tst-wctype.sh: Likewise.
