ISO-8859-8-based locales will need this transliteration if the locale
files contain these Unicode characters.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
We automatically create $(complocaledir) in the testroot.root
now and so we don't need to create it in the test.
Tested on x86_64.
Reviewed-by: DJ Delorie <dj@redhat.com>
On the current AArch64 buildbot the default 20s timeout is not
enough to run this test, it seems
make test t=locale/tst-localedef-path-norm
takes about 25s, so i increased the timeout to 30s.
During testing of localedef running in a minimal container
there were several error cases which were hard to diagnose
since they appeared as strerror (errno) values printed by the
higher level functions. This change adds three new verbose
messages for potential failure paths. The new messages give
the user the opportunity to use -v and display additional
information about why localedef might be failing. I found
these messages useful myself while writing a localedef
container test for --no-hard-links.
Since the changes cleanup the code that handle codeset
normalization we add tst-localedef-path-norm which contains
many sub-tests to verify the correct expected normalization of
codeset strings both when installing to default paths (the
only time normalization is enabled) and installing to absolute
paths. During the refactoring I created at least one
buffer-overflow which valgrind caught, but these tests did not
catch because the exec in the container had a very clean heap
with zero-initialized memory. However, between valgrind and
the tests the results are clean.
The new tst-localedef-path-norm passes without regression on
x86_64.
Change-Id: I28b9f680711ff00252a2cb15625b774cc58ecb9d
The locale/tst-locale-locpath test unsets LANG, then runs a test with
test_wrapper_env and expects LANG to remain unset for that test. This
does not work for cross-testing with cross-test-ssh.sh when sshd (on
the system specified as an argument to cross-test-ssh.sh) is
configured to have a default LANG setting.
The general design used in cross testing, after commit
8540f6d2a7 ("Don't require test wrappers
to preserve environment variables, use more consistent environment.",
6 June 2014), is that environment settings required by tests should be
passed explicitly to $(test-wrapper-env). This patch changes
tst-locale-locpath.sh to pass an explicit LANG= rather than expecting
"unset LANG" to be in effect for the program run under
test_wrapper_env. Note that this does slightly change the environment
in which the test is run natively (empty LANG instead of unset LANG)
but that difference does not appear relevant to what it is trying to
test.
Tested for Arm that this fixes the failure seen for that test in
cross-testing.
I've updated copyright dates in glibc for 2020. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files. As well as the usual annual
updates, mainly dates in --version output (minus libc.texinfo which
previously had to be handled manually but is now successfully updated
by update-copyrights), there is a fix to
sysdeps/unix/sysv/linux/powerpc/bits/termios-c_lflag.h where a typo in
the copyright notice meant it failed to be updated automatically.
Please remember to include 2020 in the dates for any new files added
in future (which means updating any existing uncommitted patches you
have that add new files to use the new copyright dates in them).
_nl_load_locale_from_archive() checks for a zero size, but
divides by both (size) and (size-2). Extend the check to
guard against a size of two or less.
Tested by manually corrupting locale-archive and running a program
that calls setlocale() with LOCPATH unset (size is typically very
large).
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This propagates the recent http->https URL changes.
Since I used gperf 3.1 to regenerate, this is also a minor
internal-to-localedef API change.
URL problem reported by Joseph Myers in:
https://www.sourceware.org/ml/libc-alpha/2019-09/msg00143.html
* locale/programs/charmap-kw.h, locale/programs/locfile-kw.h:
Regenerate with gperf 3.1.
* locale/programs/linereader.h (kw_hash_fct_t):
* locale/programs/repertoire.c (repertoiremap_hash):
2nd arg is now size_t not unsigned, for compatibility with gperf 3.1.
struct charseq used a zero-length array instead of a flexible array
member. This required a strange construct to initialize struct
charseq objects, and GCC 10 warns about that:
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
In file included from programs/repertoire.h:24,
from programs/localedef.h:32,
from programs/ld-ctype.c:35:
programs/charmap.h:63:17: note: destination object declared here
63 | unsigned char bytes[0];
| ^~~~~
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
The change makes the object physically const, but it is not expected
to be modified.
This patch adds Cyrillic to plain ASCII transliteration table according
to GOST 7.79-2000 System B standard to the C locale.
[BZ #2872]
* locale/C-translit.h.in: Add Cyrillic transliteration.
The arguments passed by Makefile are missing match with the arguments
taken by locale/tst-locale-locpath.sh. Without this patch
cross-test-ssh.sh will be called twice in a single command while
doing the make check test wish ssh test wrapper.
* locale/Makefile: Fix tst-locale-locpath arguments passing.
This patch fixes the gcc warnings seen with gcc 9 -march>=z13 on s390x:
programs/ld-ctype.c: In function ‘ctype_read’:
programs/ld-ctype.c:1392:13: error: ‘wch’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1392 | uint32_t wch;
| ^~~
programs/ld-ctype.c:1401:7: error: ‘seq’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1401 | if (seq != NULL && seq->nbytes == 1)
| ^
programs/ld-ctype.c:1391:20: note: ‘seq’ was declared here
1391 | struct charseq *seq;
| ^~~
Both seq and wch are uninitialized if get_character fails.
Thus we are now returning with an error.
ChangeLog:
* locale/programs/ld-ctype.c (charclass_symbolic_ellipsis):
Return error if get_character fails.
When -Werror=parentheses is in use, iconvconfig.c builds fail with:
iconvconfig.c: In function ‘write_output’:
iconvconfig.c:1084:34: error: suggest parentheses around ‘+’ inside ‘>>’ [-Werror=parentheses]
hash_size = next_prime (nnames + nnames >> 1);
~~~~~~~^~~~~~~~
This patch adds parentheses to the expression. Not where suggested by
the compiler warning, but where it produces the expected result, i.e.:
where it has the effect of multiplying nnames by 1.5.
Likewise for elem_size in ld-collate.c.
Tested for powerpc64le.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Two cases of "int * 1.4" may result in imprecise results, which
in at least one case resulted in i686 and x86-64 producing
different locale files. This replaced that floating point multiply
with integer operations. While the hash table margin is increased
from 40% to 50%, testing shows only 2% increase in overall size
of the locale archive.
https://bugzilla.redhat.com/show_bug.cgi?id=1311954
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This patch adds fall-through comments in some cases where -Wextra
produces implicit-fallthrough warnings.
The patch is non-exhaustive. Apart from architecture-specific code
for non-x86_64 architectures, it does not change sunrpc/xdr.c (legacy
code, probably should have such changes, but left to be dealt with
separately), or places that already had comments about the
fall-through but not matching the form expected by
-Wimplicit-fallthrough=3 (the default level with -Wextra; my
inclination is to adjust those comments to match rather than
downgrading to -Wimplicit-fallthrough=1 to allow any comment), or one
place where I thought the implicit fallthrough was not correct and so
should be handled separately as a bug fix. I think the key thing to
consider in review of this patch is whether the fall-through is indeed
intended and correct in each place where such a comment is added.
Tested for x86_64.
* elf/dl-exception.c (_dl_exception_create_format): Add
fall-through comments.
* elf/ldconfig.c (parse_conf_include): Likewise.
* elf/rtld.c (print_statistics): Likewise.
* locale/programs/charmap.c (parse_charmap): Likewise.
* misc/mntent_r.c (__getmntent_r): Likewise.
* posix/wordexp.c (parse_arith): Likewise.
(parse_backtick): Likewise.
* resolv/ns_ttl.c (ns_parse_ttl): Likewise.
* sysdeps/x86/cpu-features.c (init_cpu_features): Likewise.
* sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise.
Downstream distributions need consistent sets of hardlinks in
order for rpm to operate effectively. This means that even if
locales are built with a high level of parallelism that the
resulting files need to have consistent hardlink counts. The only
way to achieve this is with a post-install hardlink pass using a
program like 'hardlink' (shipped in Fedora).
If the downstream distro wants to post-process the hardlinks then
the time spent in localedef looking up sibling directories and
processing hardlinks is wasted effort.
To optimize the build and install pass we add a --no-hard-links
option to localedef to avoid doing the hardlink optimziation for
size.
Tested on x86_64 with 'make localedata/install-locale-files'
before and after. Without the patch we have files with 100+
hardlink counts. After the patch and running with --no-hard-links
all link counts are 1. This patch also alters the convenience
target 'make localedata/install-locale-files' to use the new
option.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Continuing fixes for -Os build issues shown with build-many-glibcs.py,
this patch adds uses of DIAG_* to disable -Wmaybe-uninitialized in two
more places where code inlined from strcoll / wcscoll is wrongly
diagnosed as possibly using uninitialized structure fields. (All
these warnings in different places for these functions are I think
essentially the same bug.)
Tested with build-many-glibcs.py for alpha-linux-gnu and
mips-linux-gnu that this fixes the -Os build failures for those
configurations with GCC 7.
* locale/weightwc.h (findidx): Ignore -Wmaybe-uninitialized for
-Os in two more places.
The findidx functions used in implementing strcoll / wcscoll already
use DIAG_IGNORE_Os_NEEDS_COMMENT for spurious -Wmaybe-uninitialized
warnings that appear with -Os. In building with GCC 7 for x86_64 with
-Os, I find there are additional such warnings, for the same structure
elements, which are spurious for the same reasons given in the
existing comments (and this was also reported for MIPS with GCC 5 in
bug 21313). This patch adds corresponding uses of DIAG_* in the
places that get the additional warnings.
Tested for x86_64 with -Os that this eliminates those warnings and so
allows the build to progress further.
[BZ #21313]
* locale/weight.h (findidx): Disable -Wmaybe-uninitialized for -Os
in another place.
* locale/weightwc.h (findidx): Likewise.
All the previous changes also repeated to support abbreviated
alternative month names. In most languages which have declension and
need nominative/genitive month names the abbreviated forms for both
cases are the same. An example where they do differ is May in Russian:
this name is too short to be abbreviated so even the abbreviated form
features the declension suffixes.
[BZ #10871]
* locale/C-time.c (_nl_C_LC_TIME): Add abbreviated alternative month
names, define them as the same as abbreviated month names explicitly.
* locale/categories.def (LC_TIME): Add ab_alt_mon and wide-ab_alt_mon.
* locale/langinfo.h: (_NL_ABALTMON_1, _NL_ABALTMON_2, _NL_ABALTMON_3,
_NL_ABALTMON_4, _NL_ABALTMON_5, _NL_ABALTMON_6, _NL_ABALTMON_7,
_NL_ABALTMON_8, _NL_ABALTMON_9, _NL_ABALTMON_10, _NL_ABALTMON_11,
_NL_ABALTMON_12, _NL_WABALTMON_1, _NL_WABALTMON_2, _NL_WABALTMON_3,
_NL_WABALTMON_4, _NL_WABALTMON_5, _NL_WABALTMON_6, _NL_WABALTMON_7,
_NL_WABALTMON_8, _NL_WABALTMON_9, _NL_WABALTMON_10, _NL_WABALTMON_11,
_NL_WABALTMON_12): New enum constants.
* locale/programs/ld-time.c (struct locale_time_t): Add ab_alt_mon,
wab_alt_mon, and ab_alt_mon_defined members.
(time_output): Output ab_alt_mon and wab_alt_mon members.
(time_read): Read them, initialize them as copies of abmon and wabmon
respectively if they are missing, initialize ab_alt_mon_defined.
* locale/programs/locfile-kw.gperf (ab_alt_mon): Define.
* locale/programs/locfile-kw.h: Regenerate.
* locale/programs/locfile-token.h (tok_ab_alt_mon): New enum constant.
* time/Makefile [$(run-built-tests) = yes] (LOCALES): Add es_ES.UTF-8
and ru_RU.UTF-8.
* time/strftime_l.c (a_altmonth, aam_len): New macros.
[!COMPILE_WIDE] (ABALTMON_1): New macro.
(__strftime_internal): Handle %Ob and %Oh formats.
* time/strptime_l.c [_LIBC] (ab_alt_month_name): New macro.
(__strptime_internal): Handle %Ob and %Oh formats.
* time/tst-strptime.c (day_tests): Add more tests to parse different
forms of month names including the new %Ob format specifier.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Some languages (Slavic, Baltic, etc.) require a genitive case of the
month name when formatting a full date (with the day number) while
they require a nominative case when referring to the month standalone.
This requirement cannot be fulfilled without providing two forms for
each month name. From now it is specified that nl_langinfo(MON_1)
series (up to MON_12) and strftime("%B") generate the month names in
the grammatical form used when the month is a part of a complete date.
If the grammatical form used when the month is named by itself is needed,
the new values nl_langinfo(ALTMON_1) (up to ALTMON_12) and
strftime("%OB") are supported. This new feature is optional so the
languages which do not need it or do not yet provide the updated
locales simply do not use it and their behaviour is unchanged.
[BZ #10871]
* locale/C-time.c (_nl_C_LC_TIME): Add alternative month names,
define them as the same as primary full month names explicitly.
* locale/categories.def (LC_TIME): Add alt_mon and wide-alt_mon.
* locale/langinfo.h (__ALTMON_1, __ALTMON_2, __ALTMON_3, __ALTMON_4,
__ALTMON_5, __ALTMON_6, __ALTMON_7, __ALTMON_8, __ALTMON_9, __ALTMON_10,
__ALTMON_11, __ALTMON_12, _NL_WALTMON_1, _NL_WALTMON_2, _NL_WALTMON_3,
_NL_WALTMON_4, _NL_WALTMON_5, _NL_WALTMON_6, _NL_WALTMON_7,
_NL_WALTMON_8, _NL_WALTMON_9, _NL_WALTMON_10, _NL_WALTMON_11,
_NL_WALTMON_12): New enum constants.
[__USE_GNU] (ALTMON_1, ALTMON_2, ALTMON_3, ALTMON_4, ALTMON_5, ALTMON_6,
ALTMON_7, ALTMON_8, ALTMON_9, ALTMON_10, ALTMON_11, ALTMON_12): New
macros.
* locale/programs/ld-time.c (struct locale_time_t): Add alt_mon,
walt_mon, and alt_mon_defined members.
(time_output): Output alt_mon and walt_mon members.
(time_read): Read them, initialize them as copies of mon and wmon
respectively if they are missing, initialize alt_mon_defined.
* locale/programs/locfile-kw.gperf (alt_mon): Define.
* locale/programs/locfile-kw.h: Regenerate.
* locale/programs/locfile-token.h (tok_alt_mon): New enum constant.
* localedata/tst-langinfo.c (map): Add tests for the new constants
ALTMON_1 .. ALTMON_12.
* time/Makefile [$(run-built-tests) = yes] (LOCALES): Add fr_FR.UTF-8
and pl_PL.UTF-8.
* time/strftime_l.c (f_altmonth): New macro.
(__strftime_internal): Handle %OB format.
* time/strptime_l.c [_LIBC] (alt_month_name): New macro.
(__strptime_internal): Handle %OB format.
* time/tst-strptime.c (day_tests): Add tests to parse different forms
of month names including the new %OB format specifier.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
There were several problems with checking the array size in the past,
for example BZ#356, caused by incorrectly assuming that every locale
token represents one element. In fact, if a token represented
a subarray, for example an array of month names or characters category
and it appeared at the end of the array the compiler assumed that
the array ends just after the first element of the subarray.
A workaround used in the past was to skip some categories while testing,
for example LC_CTYPE. Now when we are about to add alternative month
names to LC_TIME (BZ#10871) this will fail again.
* locale/loadlocale.c: Correct size of
_nl_value_type_LC_<category> arrays.
Reviewed-by: Zack Weinberg <zackw@panix.com>
From localedef --help:
Output control:
...
--no-warnings=<warnings> Comma-separated list of warnings to disable;
supported warnings are: ascii, intcurrsym
...
--warnings=<warnings> Comma-separated list of warnings to enable;
supported warnings are: ascii, intcurrsym
Locales using SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII
compatible. In order to build locales using these character maps, and
have localedef exit with a status of 0, we add new option to localedef
to disable or enable specific warnings. The options are --no-warnings
and --warnings, to disable and enable specific warnings respectively.
The options take a comma-separated list of warning names. The warning
names are taken directly from the generated warning. When a warning
that can be disabled is issued it will print something like this: foo is
not defined [--no-warnings=foo]
For the initial implementation we add two controllable warnings; first
'ascii' which is used by the localedata installation makefile target to
install SHIFT_JIS and SHIFT_JISX0213-using locales without error; second
'intcurrsym' which allows a program to use a non-standard international
currency symbol without triggering a warning. The 'intcurrsym' is
useful in the future if country codes are added that are not in our
current ISO 4217 list, and the user wants to avoid the warning. Having
at least two warnings to control gives an example for how the changes
can be extended to more warnings if required in the future.
These changes allow ja_JP.SHIFT_JIS and ja_JP.SHIFT_JISX0213 to be
compiled without warnings using --no-warnings=ascii. The
localedata/Makefile $(INSTALL-SUPPORTED-LOCALES) target is adjusted to
automatically add `--no-warnings=ascii` for such charmaps, and likewise
localedata/gen-locale.sh is adjusted with similar logic.
v2: Bring verbose, be_quiet, and all warning control booleans into
record-status.c, and compile this object file to be used by locale,
iconv, and localedef. Any users include record-status.h.
v3: Fix an instance of boolean coercion in set_warning().
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Recorded verbose messages no longer need to pass \n in their
message string since the record_verbose function adds \n to
the messages (like error and warnings do also). The avoids
seeing a double \n for verbose messages.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
In "Is it OK to write ASCII strings directly into locale source files?"
https://sourceware.org/ml/libc-alpha/2017-07/msg00807.html there is
universal consensus that we do not have to keep writing <Uxxxx> symbolic
characters in locale files.
Ulrich Drepper's historical comment was that symbolic characters were
used for the eventuality of converting the source files to any encoding
system. Fast forward to today and UTF-8 is the standard. So the
requirement of <Uxxxx> is hard to justify.
Zack Weinberg's excellent scripts are coming along we can use these to
find instances of human errors in the scripts:
https://sourceware.org/ml/libc-alpha/2017-07/msg00860.htmlhttps://sourceware.org/ml/libc-alpha/2017-08/msg00136.html
It still won't be easy to distinguish from i for í, but that's still the
case for <Uxxxx> characters which humans can't read either.
Since we all agreed that we should be able to use non-symbolic (<Uxxxx>)
characters in locale files, the following change removes the verbose
warning that is raised if you use non-symbolic characters in the locale
file.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
The builtin POSIX locale has "" as the international currency symbol,
but a non-builtin locale may not have such a blank int_curr_symbol.
Therefore to support non-builtin locales with similar "" int_curr_symbol
we adjust the LC_MONETARY parser to allow the normal 4-character
int_curr_symbol *and* the empty "" no symbol. Anything else remains
invalid.
Tested by building all the locales. Tested also with a custom C.UTF-8
locale with "" for int_curr_symbol.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
The error and warning handling in localedef, locale, and iconv
is a bit of a mess.
We use ugly constructs like this:
WITH_CUR_LOCALE (error (1, errno, gettext ("\
cannot read character map directory `%s'"), directory));
to issue errors, and read error_message_count directly from the
error API to detect errors. The problem with that is that the
code also uses error to print warnings, and informative messages.
All of this leads to problems where just having warnings will
produce an exit status as-if errors had been seen.
To fix this situation I have adopted the following high-level
changes:
* All errors are counted distinctly.
* All warnings are counted distinctly.
* All informative messages are not counted.
* Increasing verbosity cannot generate *more* errors, and
it previously did for errors conditional on verbose,
this is now fixed.
* Increasing verbosity *can* generate *more* warnings.
* Making the output quiet cannot generate *fewer* errors,
and it previously did for errors conditional on be_quiet,
this is now fixed.
* Each of error, warning, and informative message has it's
own function to call defined in record-status.h, and they
are: record_error, record_warning, and record_verbose.
* The record_error function always records an error, but
conditional on be_quiet may not print it.
* The record_warning function always records a warning,
but conditional on be_quiet may not print it.
* The record_verbose function only prints the verbose
message if verbose is true and be_quiet is false.
This has allowed the following fix:
* Previously any warnings were being treated as errors
because they incremented error_message_count, but now
we properly return an exit status of 1 if there are
warnings but output was generated.
All of this allows localedef to correctly decide if errors,
or warnings were present, and produce the correct exit code.
The locale and iconv programs now also use record-status.h
and we have removed the WITH_CUR_LOCALE hack, and instead
have internal push_locale/pop_locale functions centralized
in the record routines.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
“Bengali” still remained in some comments in the bn_BD locale file,
in iso-639.def and in a test input file. Change it there as well.
“Bangla” is now used as the English name for this language in CLDR.
[BZ #14925]
* libio/tst-widetext.input: Change “Bengali” to “Bangla”.
* locale/iso-639.def: Change “Bengali” to “Bangla”.
* localedata/locales/bn_BD: “Bengali” was still used in some
comments. Change it to “Bangla”.
