Commit Graph

23 Commits

Author SHA1 Message Date
Carlos O'Donell
466f2be6c0 Add generic C.UTF-8 locale (Bug 17318)
We add a new C.UTF-8 locale. This locale is not builtin to glibc, but
is provided as a distinct locale. The locale provides full support for
UTF-8 and this includes full code point sorting via STRCMP-based
collation (strcmp or wcscmp).

The collation uses a new keyword 'codepoint_collation' which drops all
collation rules and generates an empty zero rules collation to enable
STRCMP usage in collation. This ensures that we get full code point
sorting for C.UTF-8 with a minimal 1406 bytes of overhead (LC_COLLATE
structure information and ASCII collating tables).

The new locale is added to SUPPORTED. Minimal test data for specific
code points (minus those not supported by collate-test) is provided in
C.UTF-8.in, and this verifies code point sorting is working reasonably
across the range. The locale was tested manually with the full set of
code points without failure.

The locale is harmonized with locales already shipping in various
downstream distributions. A new tst-iconv9 test is added which verifies
the C.UTF-8 locale is generally usable.

Testing for fnmatch, regexec, and recomp is provided by extending
bug-regex1, bugregex19, bug-regex4, bug-regex6, transbug, tst-fnmatch,
tst-regcomp-truncated, and tst-regex to use C.UTF-8.

Tested on x86_64 or i686 without regression.

Reviewed-by: Florian Weimer <fweimer@redhat.com>
2021-09-06 11:30:28 -04:00
Siddhesh Poyarekar
30891f35fa Remove "Contributed by" lines
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date.  Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.

Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions.  These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.

The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively.  These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:

https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dc
https://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2021-09-03 22:06:44 +05:30
Stefan Liebler
2457175e8b Fix stringop-overflow warning in bug-regex19.c.
Starting with commit
26492c0a14
"Annotate additional APIs with GCC attribute access.",
gcc emits this warning on s390x:
In function 'do_one_test',
    inlined from 'do_mb_tests' at bug-regex19.c:385:11:
bug-regex19.c:271:9: error: 're_search' specified size 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Werror=stringop-overflow=]
  271 |   res = re_search (&regbuf, test->string, strlen (test->string),
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  272 |      test->start, strlen (test->string) - test->start, NULL);
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../include/regex.h:2,
                 from bug-regex19.c:22:
bug-regex19.c: In function 'do_mb_tests':
../posix/regex.h:554:17: note: in a call to function 're_search' declared with attribute 'read_only (2, 3)'
  554 | extern regoff_t re_search (struct re_pattern_buffer *__buffer,
      |                 ^~~~~~~~~
...

The function do_one_test is inlined into do_mb_tests on s390x (at least with
gcc 10).  If do_one_test is marked with __attribute__ ((noinline)), there are
no warnings on s390x. If do_one_test is marked with
__attribute__ ((always_inline)), there are the same warnings on x86_64.

test->string points to a variable length array on stack of do_mb_tests
and the content is generated based on the passed test struct.

According to Martin Sebor, this is a false positive caused by the same bug as
the one in nss/makedb.c.  It's fixed in GCC 11 and will also be available in
the next GCC 10.4 release.
2021-05-18 10:07:30 +02:00
Paul Eggert
2b778ceb40 Update copyright dates with scripts/update-copyrights
I used these shell commands:

../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")

and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
2021-01-02 12:17:34 -08:00
Joseph Myers
d614a75396 Update copyright dates with scripts/update-copyrights. 2020-01-01 00:14:33 +00:00
Paul Eggert
5a82c74822 Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:

sed -ri '
  s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
  s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
  $(find $(git ls-files) -prune -type f \
      ! -name '*.po' \
      ! -name 'ChangeLog*' \
      ! -path COPYING ! -path COPYING.LIB \
      ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
      ! -path manual/texinfo.tex ! -path scripts/config.guess \
      ! -path scripts/config.sub ! -path scripts/install-sh \
      ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
      ! -path INSTALL ! -path  locale/programs/charmap-kw.h \
      ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
      ! '(' -name configure \
            -execdir test -f configure.ac -o -f configure.in ';' ')' \
      ! '(' -name preconfigure \
            -execdir test -f preconfigure.ac ';' ')' \
      -print)

and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:

  chmod a+x sysdeps/unix/sysv/linux/riscv/configure
  # Omit irrelevant whitespace and comment-only changes,
  # perhaps from a slightly-different Autoconf version.
  git checkout -f \
    sysdeps/csky/configure \
    sysdeps/hppa/configure \
    sysdeps/riscv/configure \
    sysdeps/unix/sysv/linux/csky/configure
  # Omit changes that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
  git checkout -f \
    sysdeps/powerpc/powerpc64/ppc-mcount.S \
    sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
  # Omit change that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
  git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 02:43:31 -07:00
Joseph Myers
04277e02d7 Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2019-01-01 00:11:28 +00:00
Joseph Myers
688903eb3e Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2018-01-01 00:32:25 +00:00
Joseph Myers
bfff8b1bec Update copyright dates with scripts/update-copyrights. 2017-01-01 00:14:16 +00:00
Joseph Myers
f7a9f785e5 Update copyright dates with scripts/update-copyrights. 2016-01-04 16:05:18 +00:00
Joseph Myers
b168057aaa Update copyright dates with scripts/update-copyrights. 2015-01-02 16:29:47 +00:00
Allan McRae
d4697bc93d Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
Joseph Myers
568035b787 Update copyright notices with scripts/update-copyrights. 2013-01-02 19:05:09 +00:00
Paul Eggert
59ba27a63a Replace FSF snail mail address with URLs. 2012-02-09 23:18:22 +00:00
Ulrich Drepper
24992143d8 [BZ #693]
Update.
2005-01-26  Jakub Jelinek  <jakub@redhat.com>

	* sysdeps/unix/sysv/linux/i386/sysdep.h
	(SYSCALL_ERROR_HANDLER_TLS_STORE): Remove unnecessary 0 imm.

	[BZ #693]
	* posix/regex_internal.h (DUMMY_CONSTRAINT): Rename to...
	(WORD_DELIM_CONSTRAINT): ...this.
	(NOT_WORD_DELIM_CONSTRAINT): Define.
	(re_context_type): Add INSIDE_NOTWORD and NOT_WORD_DELIM,
	change WORD_DELIM to use WORD_DELIM_CONSTRAINT.
	* posix/regcomp.c (peek_token): For \B create NOT_WORD_DELIM
	anchor instead of INSIDE_WORD.
	(parse_expression): Handle NOT_WORD_DELIM constraint.
	* posix/bug-regex19.c (tests): Adjust tests that relied on \B
	being inside word instead of not word delim.
	* posix/tst-rxspencer.c (mb_frob_pattern): Don't frob escaped
	characters.
	* posix/rxspencer/tests: Add some new tests.
2005-01-26 19:56:03 +00:00
Ulrich Drepper
a334319f65 (CFLAGS-tst-align.c): Add -mpreferred-stack-boundary=4. 2004-12-22 20:10:10 +00:00
Jakub Jelinek
0ecb606cb6 2.5-18.1 2007-07-12 18:26:36 +00:00
Ulrich Drepper
1cef7b3cae Update.
2003-12-21  Jakub Jelinek  <jakub@redhat.com>

	* posix/regcomp.c (duplicate_node, duplicate_node_closure): Revert
	2003-11-24 changes.
	* posix/regexec.c (group_nodes_into_DFAstates): For CHARACTER with
	NEXT_{,NOT}WORD_CONSTRAINT check word_char bit.
	* posix/bug-regex19.c (tests): Add new tests.

	* posix/regexec.c (check_dst_limits_calc_pos): Fix formatting.

	* posix/regcomp.c (parse_dup_op): Return NULL if dup_elem is NULL,
	after checking syntax.  Optimize.
	(calc_first): Fix comment.
	* posix/bug-regex13.c (tests): Add new tests.
2003-12-22 06:52:35 +00:00
Ulrich Drepper
65e6becf5b Update.
2003-11-24  Jakub Jelinek  <jakub@redhat.com>

	* posix/regex_internal.h (re_token_t): Add word_char bit.  Add
	comment.
	(re_dfa_t): Add sb_char field.
	(bitset_mask): New function.
	* posix/regcomp.c (free_dfa_content): Free sb_char.
	(init_dfa): Don't initialize word_char unnecessarily.
	Initialize sb_char.
	(duplicate_node): Don't duplicate !word_char CHARACTERs with
	NEXT_WORD_CONSTRAINT constraint or word_char CHARACTERs with
	NEXT_NOTWORD_CONSTRAINT.  Return -1 in *new_idx instead.
	(duplicate_node_closure): Handle clone_dest == -1 from
	duplicate_node.
	(peek_token): Initialize word_char bit.
	(parse_expression, parse_dup_op): Add comments.
	(parse_bracket_exp): Don't set bitmask bits for multi-byte char
	starting bytes here at the beginning.  Mask off the bits right
	before creating SIMPLE_BRACKET.
	(build_charclass_op): Likewise.
	* posix/regexec.c (group_nodes_into_DFAstates) <case OP_PERIOD>: Only
	set accept bits for single-byte characters.
	(group_nodes_into_DFAstates): Don't rely on characters 0 .. 127
	being single byte encoded and the rest multi-byte.
	* posix/bug-regex19.c (tests): Add new tests.
	(do_mb_tests): Initialize t to *test.
	(main): Fail even on do_mb_tests errors.
2003-11-24 19:30:51 +00:00
Ulrich Drepper
73163a0973 Update.
* posix/bug-regex19.c (tests): Add more tests.
2003-11-21 22:43:47 +00:00
Ulrich Drepper
53f9084e3f Update.
2003-11-21  Ulrich Drepper  <drepper@redhat.com>

	* Makefile: Add rules to build and run tst-cond12.
	* tst-cond12.c: New file.
2003-11-21 09:20:45 +00:00
Ulrich Drepper
c13c99fa92 Update.
2003-11-20  Ulrich Drepper  <drepper@redhat.com>

	* posix/PTESTS: Fix first test in GA143.

2003-11-20  Jakub Jelinek  <jakub@redhat.com>

	* posix/regex_internal.h (re_dfastate_t): Remove trtable_search.
	Add word_trtable.
	* posix/regex_internal.c (create_newstate_common, free_state):
	Don't free trtable_search.
	* posix/regexec.c (check_matching): Remove fl_search argument.
	(transit_state_sb): Likewise.  #ifdef out as unused.
	(build_trtable): Remove fl_search argument.  Set state->word_trtable
	and state->trtable.  Build separate word and non-word tables if
	multi-byte and they differ for some character.
	(transit_state): Remove fl_search argument.  Don't update
	state->trtable here.  Handle state->word_trtable.
	#ifdef out unused call to transit_state_sb.
	(re_search_internal): Update check_matching caller.
	(group_nodes_into_DFAstates): Don't clear non-ascii chars in accepts
	bitmask for multi-byte locales.
	* posix/bug-regex19.c (tests): Enable some commented out tests, add
	2 new tests.
	* posix/tst-rxspencer.c (mb_tests): Don't test [[=b=]] for now as
	multi-byte.  Don't run identical multi-byte tests multiple times
	unnecessarily.
	(main): Check setlocale return value.
	* posix/Makefile (tst-rxspencer-ARGS): Add --utf8 argument.
	(tst-rxspencer-ENV): Remove MALLOC_TRACE, add LOCPATH.
	($(objpfx)tst-rxspencer-mem): Run another tst-rxspencer test
	here, without --utf8 argument but with MALLOC_TRACE.
2003-11-20 23:36:40 +00:00
Ulrich Drepper
3c0fb5745f Update.
2003-11-12  Jakub Jelinek  <jakub@redhat.com>

	* io/ftw.c (NFTW_NEW_NAME, NFTW_OLD_NAME): Add prototypes.

2003-11-12  Jakub Jelinek  <jakub@redhat.com>

	* posix/tst-regex.c (umemlen): New variable.
	(test_expr): Add expectedicase argument.  Test case insensitive
	searches as well as backwards searches (case sensitive and
	insensitive) too.
	(run_test): Add icase argument.  Use it to compute regcomp flags.
	(run_test_backwards): New function.
	(main): Cast read to size_t to avoid warning.  Set umemlen.
	Add expectedicase arguments to test_expr.
	* posix/regex_internal.c (re_string_reconstruct): If is_utf8,
	find previous character by walking back instead of converting
	all chars from beginning.

2003-11-12  Jakub Jelinek  <jakub@redhat.com>

	* posix/regex_internal.h (struct re_string_t): Add is_utf8
	and mb_cur_max fields.
	(struct re_dfa_t): Likewise.  Reorder fields to make structure
	smaller on 64-bit arches.
	(re_string_allocate, re_string_construct): Add mb_cur_max and
	is_utf8 arguments.
	(re_string_char_size_at, re_string_wchar_at): Use pstr->mb_cur_max
	instead of MB_CUR_MAX.
	* posix/regcomp.c (re_compile_fastmap_iter): Use dfa->mb_cur_max
	instead of MB_CUR_MAX.
	(re_compile_internal): Pass new arguments to re_string_construct.
	(init_dfa): Initialize mb_cur_max and is_utf8 fields.
	(peek_token, peek_token_bracket): Use input->mb_cur_max instead
	of MB_CUR_MAX.
	(parse_expression, parse_bracket_exp, parse_charclass_op): Use
	dfa->mb_cur_max instead of MB_CUR_MAX.
	* posix/regex_internal.c (re_string_construct_common): Add
	mb_cur_max and is_utf8 arguments.  Initialize fields with them.
	(re_string_allocate, re_string_construct): Add mb_cur_max and
	is_utf8 arguments, pass them to re_string_construct_common.
	Use mb_cur_max instead of MB_CUR_MAX.
	(re_string_realloc_buffers): Use pstr->mb_cur_max instead of
	MB_CUR_MAX.
	(re_string_reconstruct): Likewise.
	(re_string_context_at): Use input->mb_cur_max instead of
	MB_CUR_MAX.
	(create_ci_newstate, create_cd_newstate): Use dfa->mb_cur_max
	instead of MB_CUR_MAX.
	* posix/regexec.c (re_search_internal): Likewise.
	Pass new arguments to re_string_allocate.
	(check_matching, transit_state_sb): Use dfa->mb_cur_max instead of
	MB_CUR_MAX.
	(extend_buffers): Use pstr->mb_cur_max instead of MB_CUR_MAX.

2003-11-12  Jakub Jelinek  <jakub@redhat.com>

	* posix/Makefile (tests): Add bug-regex19.
	(bug-regex19-ENV): Add LOCPATH.
	* posix/bug-regex19.c: New test.
2003-11-12 17:47:46 +00:00