In several converters, a __GCONV_ILLEGAL_INPUT result gets overwritten
with __GCONV_FULL_OUTPUT. As a result, iconv (the function) returns
E2BIG instead of EILSEQ. The iconv program does not see the original
EILSEQ failure, does not recognize the invalid input, and may
incorrectly exit successfully.
To address this, a new __flags bit is used to indicate a sticky input
error state. All __GCONV_ILLEGAL_INPUT results are replaced with a
function call that sets this new __GCONV_ENCOUNTERED_ILLEGAL_INPUT and
returns __GCONV_ILLEGAL_INPUT. The iconv program checks for
__GCONV_ENCOUNTERED_ILLEGAL_INPUT and overrides the exit status.
The converter changes introducing __gconv_mark_illegal_input are
mostly mechanical, except for the res variable initialization in
iconvdata/iso-2022-jp.c: this error gets overwritten with __GCONV_OK
and other results in the following code. If res ==
__GCONV_ILLEGAL_INPUT afterwards, STANDARD_TO_LOOP_ERR_HANDLER below
will handle it.
The __gconv_mark_illegal_input changes do not alter the errno value
set by the iconv function. This is simpler to implement than
reviewing each __GCONV_FULL_OUTPUT result and adjust it not to
override a previous __GCONV_ILLEGAL_INPUT result. Doing it that way
would also change some E2BIG errors in to EILSEQ errors, so it had to
be done conditionally (under a flag set by the iconv program only), to
avoid confusing buffer management in other applications.
Reviewed-by: DJ Delorie <dj@redhat.com>
All the changes are in comments or '#error' messages.
Applying this commit results in bit-identical rebuild of iconvdata/*.so
Reviewed-by: Florian Weimer <fw@deneb.enyo.de>
And use a packed structure instead. The compiler generates optimized
unaligned code if the architecture supports it.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.
I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah. I don't
know why I run into these diagnostics whereas others evidently do not.
remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
Commit 91927b7c76 (Rewrite iconv option parsing [BZ #19519]) did not
handle cases where the output codeset for translations (via the `gettext'
family of functions) might have a caller specified encoding suffix such as
TRANSLIT or IGNORE. This led to a regression where translations did not
work when the codeset had a suffix.
This commit fixes the above issue by parsing any suffixes passed to
__dcigettext and adds two new test-cases to intl/tst-codeset.c to
verify correct behaviour. The iconv-internal function __gconv_create_spec
and the static iconv-internal function gconv_destroy_spec are now visible
internally within glibc and used in intl/dcigettext.c.
This commit replaces string manipulation during `iconv_open' and iconv_prog
option parsing with a structured, flag based conversion specification. In
doing so, it alters the internal `__gconv_open' interface and accordingly
adjusts its uses.
This change fixes several hangs in the iconv program and therefore includes
a new test to exercise iconv_prog options that originally led to these hangs.
It also includes a new regression test for option handling in the iconv
function.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Commit ba7b4d294b ("Complete the
removal of __gconv_translit_find") added a declaration of the
GLIBC_PRIVATE function, __gconv_transliterate, to the installed
header <gconv.h>. It should have been added to the internal
<gconv_int.h> header.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
In iconv/gconv_conf.c, __gconv_get_path unnecessarily obtains a lock when
populating the array pointed to by __gconv_path_elem. The locking is not
necessary because all calls to __gconv_read_conf (which in turn calls
__gconv_get_path) are serialized using __libc_once.
This patch:
- removes all locking in __gconv_get_path;
- replaces all explicitly serialized __gconv_read_conf calls with calls to
__gconv_load_conf, a new wrapper that is serialized internally;
- adds a new test, iconv/tst-iconv_mt.c, to exercise iconv initialization,
usage, and cleanup in a multi-threaded program;
- indents __gconv_get_path correctly, removing tab characters (which makes
the patch look a little bigger than it really is).
After removing the unnecessary locking, it was confirmed that the test case
fails if the relevant __libc_once is removed. Additionally, four localedata
and iconvdata tests also fail. This gives confidence that the testsuite
sufficiently guards against some regressions relating to multi-threading
with iconv.
Tested on x86_64 and i686.
The variables __gconv_path_elem, __gconv_max_path_elem_len and function
__gconv_get_path declared in, as well as the type path_elem and macro
GCONV_NCHAR_GOAL defined in gconv_int.h are all used in only one iconv
compilation unit each. In addition, the extern declaration of the variable
__gconv_nmodules refers to a variable that does not exist any more.
Considering this, these symbols do not need to be exposed via a header file.
This patch removes the extern declarations from the header file and moves
the definitions to the compilation units where they are used.
Prior to the 2.20 release, the function was just changed to fail
unconditionally, in commit a1a6a401ab.
This commit removes the function completely, including gconv bits
which depend on it.
This changes the gconv ABI, which is not a public interface.
* iconv/gconv_cache.c (__gconv_lookup_cache): Return __GCONV_NULCONV
if from and to charsets are the same.
* iconv/gconv_db.c (__gconv_find_transform): Likewise.
* intl/dcigettext.c (_nl_find_msg): Return NULL even if __gconv_open
returns __GCONV_NOCONV, but not for __GCONV_NULCONV.
2007-07-17 Jakub Jelinek <jakub@redhat.com>
* wcsmbs/wchar.h: Only define wint_t if __need_wint_t.
Don't define wint_t when __need_mbstate_t unless it
is necessary.
(__mbstate_t): Use __WINT_TYPE__ rather than wint_t
in the typedef if possible.
* wctype/wctype.h (wint_t): Define by including
wchar.h with __need_wint_t instead of including stddef.h
with __need_wint_t and as fallback definining it ourselves.
* iconv/gconv.h (__need_wint_t): Define before including
wchar.h.
* sysdeps/gnu/_G_config.h: Don't include gconv.h if not _LIBC
or _GLIBCPP_USE_WCHAR_T.
(__need_wchar_t): Don't define
if not _LIBC or _GLIBCPP_USE_WCHAR_T.
(__need_wint_t): Don't define before including stddef.h,
define before including wchar.h only if _LIBC or
_GLIBCPP_USE_WCHAR_T.
(_G_iconv_t): Don't define if not _LIBC or _GLIBCPP_USE_WCHAR_T.
* sysdeps/mach/hurd/_G_config.h: Likewise.
* sysdeps/generic/_G_config.h: Likewise.
* libio/libio.h (__wunderflow, __wuflow, __woverflow): Only
prototype if _LIBC or _GLIBCPP_USE_WCHAR_T.
(_IO_getwc_unlocked, _IO_putwc_unlocked): Only define
if _LIBC or _GLIBCPP_USE_WCHAR_T.
* nscd/nscd_initgroups.c: Mark __gr_map_handle as hidden.
* nscd/nscd_getgr_r.c: Likewise.
* include/rpc/pmap_rmt.h: Mark all functions as hidden.
* nscd/nscd_getai.c (__hst_map_handle): Mark as hidden.
* nscd/nscd_gethst_r.c (__hst_map_handle): Likewise.
* stdlib/longlong.h (__clz_tab): Mark as hidden.
* iconv/gconv_int.h: Mark __gconv_lock as hidden.
* iconv/gconv_db.c: Don't define lock as static. Rename to
__gconv_lock and export from the file.
* iconv/gconv_int.h: Declare __gconv_lock.
* libio/iofclose.c [_LIBC] (_IO_new_fclose): Lock gconv lock
before __gconv_release_step calls.
Patch by Shunichi Sagawa <s-sagawa@jp.fujitsu.com>.
* iconv/gconv_simple.c (internal_ucs4_loop): Fix typo in last change.
* iconv/gconv_db.c
2002-11-30 Bruno Haible <bruno@clisp.org>
* iconv/gconv.h (__gconv_btowc_fct): New typedef.
(struct __gconv_step): New field __btowc_fct.
* wcsmbs/btowc.c (__btowc): Use the __btowc_fct shortcut if possible.
* iconv/gconv_int.h (__BUILTIN_TRANSFORM): Renamed from
__BUILTIN_TRANS.
(__gconv_btwoc_ascii): New declaration.
* iconv/gconv_simple.c (BUILTIN_TRANSFORMATION): Add BtowcFct argument.
(__gconv_btwoc_ascii): New function.
* iconv/gconv_builtin.h: Add BtowcFct argument to all
BUILTIN_TRANSFORMATION invocations.
* iconv/gconv_conf.c (BUILTIN_TRANSFORMATION): Add BtowcFct argument.
* iconv/iconvconfig.c (BUILTIN_TRANSFORMATION): Likewise.
* iconv/gconv_builtin.c (map): New field btowc_fct.
(BUILTIN_TRANSFORMATION): Add BtowcFct argument. Use it to initialize
btowc_fct field.
(__gconv_get_builtin_trans): Initialize __btowc_fct field.
* iconv/gconv_cache.c (find_module): Initialize __btowc_fct field.
* iconv/gconv_db.c (gen_steps, increment_counter): Likewise.
* wcsmbs/wcsmbsload.c (to_wc, to_mb): Likewise.
* iconv/skeleton.c: Document STORE_REST and FROM_ONEBYTE.
(gconv_init): Initialize __btowc_fct field.
Undefine EXTRA_LOOP_ARGS and FROM_ONEBYTE at the end.
* iconv/loop.c: Document ONEBYTE_BODY.
(gconv_btowc, FROM_ONEBYTE): Define if ONEBYTE_BODY is defined.
Undefine ONEBYTE_BODY at the end.
* iconvdata/8bit-generic.c (ONEBYTE_BODY): New macro.
* iconvdata/8bit-gap.c (NONNUL): New macro.
(BODY for FROM_LOOP): Use it.
(ONEBYTE_BODY): New macro.
* iconvdata/isiri-3342.c (HAS_HOLES): Set to 1.
(NONNUL): New macro.
* iconvdata/ansi_x3.110.c (ONEBYTE_BODY): New macro.
* iconvdata/armscii-8.c (ONEBYTE_BODY): New macro.
* iconvdata/cp1255.c (ONEBYTE_BODY): New macro.
* iconvdata/cp1258.c (ONEBYTE_BODY): New macro.
* iconvdata/tcvn5712-1.c (ONEBYTE_BODY): New macro.
* iconvdata/big5.c (ONEBYTE_BODY): New macro.
* iconvdata/big5hkscs.c (ONEBYTE_BODY): New macro.
* iconvdata/euc-cn.c (ONEBYTE_BODY): New macro.
* iconvdata/euc-jp.c (ONEBYTE_BODY): New macro.
* iconvdata/euc-jisx0213.c (ONEBYTE_BODY): New macro.
* iconvdata/euc-kr.c (ONEBYTE_BODY): New macro.
* iconvdata/euc-tw.c (ONEBYTE_BODY): New macro.
* iconvdata/gbk.c (ONEBYTE_BODY): New macro.
* iconvdata/gb18030.c (ONEBYTE_BODY): New macro.
* iconvdata/ibm932.c: Include <stdbool.h>.
(TRUE, FALSE): Remove macros.
(BODY for FROM_LOOP): Remove unused variable rp1.
(ONEBYTE_BODY): New macro.
(BODY for TO_LOOP): Use bool.
* iconvdata/ibm932.h (__ibm932sb_to_ucs4_idx): Remove array.
* iconvdata/ibm943.c: Include <stdbool.h>.
(TRUE, FALSE): Remove macros.
(BODY for FROM_LOOP): Remove unused variable rp1.
(ONEBYTE_BODY): New macro.
(BODY for TO_LOOP): Use bool.
* iconvdata/ibm943.h (__ibm943sb_to_ucs4_idx): Remove array.
* iconvdata/iso8859-1.c (ONEBYTE_BODY): New macro.
* iconvdata/iso_6937-2.c (ONEBYTE_BODY): New macro.
* iconvdata/iso_6937.c (ONEBYTE_BODY): New macro.
* iconvdata/johab.c (ONEBYTE_BODY): New macro.
* iconvdata/sjis.c (ONEBYTE_BODY): New macro.
* iconvdata/shift_jisx0213.c (ONEBYTE_BODY): New macro.
* iconvdata/t.61.c (ONEBYTE_BODY): New macro.
* iconvdata/uhc.c (ONEBYTE_BODY): New macro.
* iconvdata/gbbig5.c: Tweak comment.
* iconv/gconv_int.h: Include <stdlib.h> for alloca decl.
(norm_add_slashes): alloca -> __alloca
* sysdeps/generic/readelflib.c (process_elf_file): Make LOADADDR
p_vaddr/p_offset when computing LOADADDR.
2002-08-04 Ulrich Drepper <drepper@redhat.com>
* stdio-common/psignal.c: Declare _sys_siglist_internal. Use USEINT
to access _sys_siglist.
* string/strsignal.c: Likewise.
* sysdeps/generic/siglist.c: Add _sys_siglist_internal alias.
* sysdeps/gnu/siglist.c: Likewise.
* sysdeps/unix/siglist.c: Likewise.
* sysdeps/unix/sysv/linux/arm/siglist.c: Likewise.
* libio/fileops.c: Add missing INTUSEs for _IO_file_jumps.
* libio/wfileops.c: Add missing INTUSE for _IO_file_close.
* intl/dcigettext.c: Define _nl_default_dirname_internal as hidden
alias and use it.
* intl/bindtextdom.c: Use _nl_default_dirname_internal.
* include/netinet/in.h: Add declaration of in6addr_loopback_internal.
* inet/in6_addr.c: Add INTVARDEF for in6addr_loopback.
* sysdeps/posix/getaddrinfo.c: Use INTUSE for in6addr_loopback access.
* include/time.h: Add libc_hidden_proto for __gmtime_r.
* time/gmtime.c (__gmtime_r): Add libc_hidden_def.
* iconv/Versions: Replace __gconv_alias_db, __gconv_modules_db,
and __gconv_cache with __gconv_get_alias_db, __gconv_get_modules_db,
and __gconv_get_cache respectively.
* iconv/gconv_cache.c (gconv_cache): Renamed for __gconv_cache and
defined static. Change all users.
(__gconv_get_cache): New function.
* iconv/gconv_db.c (__gconv_get_modules_db): New function.
(__gconv_get_alias_db): New function.
* iconv/gconv_int.h (__gconv_alias_db): Declare as hidden.
(__conv_modules_db): Likewise.
Add prototypes for __gconv_get_cache, __gconv_get_modules_db,
and __gconv_get_alias_db.
* iconv/iconv_prog.c: Use the new functions instead of accessing the
variables.
* include/stdlib.h: Add prototype and libc_hidden_proto for
__default_morecore.
* sysdeps/generic/morecore.c: Include <stdlib.h>.
* malloc/obstack.c: Remove fputs macro.
* malloc/mtrace.c: Remove fopen macro.
* iconv/gconv_cache.c (find_module): Don't allocate room for the
filename. Use alloca, we don't need it beyond this function.
(__gconv_release_cache): New function.
* iconv/gconv_db.c (__gconv_close_transform): Call
__gconv_release_cache after the steps are handled.
* iconv/gconv_dl.c (__gconv_find_shlib): Allocate file name in the
record as well.
* iconv/gconv_int.h: Add prototype fpr __gconv_release_cache.
2001-07-22 Ulrich Drepper <drepper@redhat.com>
* iconv/gconv_builtin.c (struct builtin_map): Remove init and end
elements.
(BUILTIN_TRANSFORMATION): Remove Init and End parameters.
(__gconv_get_builtin_trans): Initialize __init_fct and __end_fct to
NULL.
* iconv/gconv_builtin.h: Remove NULL parameters for Init and End in
all BUILTIN_TRANSFORMATION calls.
* iconv/gconv_conf.c (BUILTIN_TRANSFORMATION): Remove Init and End
parameters.
* iconv/gconv_simple.c: Likewise.
* iconv/gconv_db.c (gen_steps): Internal converters don't have
initializers, move the code accordingly.
* iconv/gconv_conf.c (__gconv_read_conf): Don't read configuration
file if STATIC_GCONV is defined.
* iconv/gconv_conf.c (__gconv_path_envvar): New global variable.
(__gconv_get_path): Use it instead of call getenv.
(__gconv_read_conf): First see whether cache can be used. If yes,
don't do any work here.
* iconv/gconv_db.c (__gconv_release_step): Renamed from release_step
and exported. Change callers.
(__gconv_find_transform): First call __gconv_lookup_cache and only
continue if it signals no cache available. Remove some unnecessary
tests.
* iconv/gconv_int.h: Declare __gconv_path_envvar, __gconv_lookup_cache,
__gconv_release_step, and __gconv_loaded_cache.
* iconv/gconv_cache.c: New file.
* iconv/iconvconfig.c: New file.
* iconv/iconvconfig.h: New file.
* iconv/strtab.c: New file.
* iconv/Makefile: Add rules to build new files and programs.
