2024-01-01 18:12:26 +00:00
|
|
|
/* Copyright (C) 2005-2024 Free Software Foundation, Inc.
|
wcrtomb: Make behavior POSIX compliant
The GNU implementation of wcrtomb assumes that there are at least
MB_CUR_MAX bytes available in the destination buffer passed to wcrtomb
as the first argument. This is not compatible with the POSIX
definition, which only requires enough space for the input wide
character.
This does not break much in practice because when users supply buffers
smaller than MB_CUR_MAX (e.g. in ncurses), they compute and dynamically
allocate the buffer, which results in enough spare space (thanks to
usable_size in malloc and padding in alloca) that no actual buffer
overflow occurs. However when the code is built with _FORTIFY_SOURCE,
it runs into the hard check against MB_CUR_MAX in __wcrtomb_chk and
hence fails. It wasn't evident until now since dynamic allocations
would result in wcrtomb not being fortified but since _FORTIFY_SOURCE=3,
that limitation is gone, resulting in such code failing.
To fix this problem, introduce an internal buffer that is MB_LEN_MAX
long and use that to perform the conversion and then copy the resultant
bytes into the destination buffer. Also move the fortification check
into the main implementation, which checks the result after conversion
and aborts if the resultant byte count is greater than the destination
buffer size.
One complication is that applications that assume the MB_CUR_MAX
limitation to be gone may not be able to run safely on older glibcs if
they use static destination buffers smaller than MB_CUR_MAX; dynamic
allocations will always have enough spare space that no actual overruns
will occur. One alternative to fixing this is to bump symbol version to
prevent them from running on older glibcs but that seems too strict a
constraint. Instead, since these users will only have made this
decision on reading the manual, I have put a note in the manual warning
them about the pitfalls of having static buffers smaller than
MB_CUR_MAX and running them on older glibc.
Benchmarking:
The wcrtomb microbenchmark shows significant increases in maximum
execution time for all locales, ranging from 10x for ar_SA.UTF-8 to
1.5x-2x for nearly everything else. The mean execution time however saw
practically no impact, with some results even being quicker, indicating
that cache locality has a much bigger role in the overhead.
Given that the additional copy uses a temporary buffer inside wcrtomb,
it's likely that a hot path will end up putting that buffer (which is
responsible for the additional overhead) in a similar place on stack,
giving the necessary cache locality to negate the overhead. However in
situations where wcrtomb ends up getting called at wildly different
spots on the call stack (or is on different call stacks, e.g. with
threads or different execution contexts) and is still a hotspot, the
performance lag will be visible.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
2022-05-13 13:40:15 +00:00
|
|
|
Copyright The GNU Toolchain Authors.
|
* wcsmbs/bits/wchar2.h: Add definitions for wcrtomb, mbsrtowcs,
wcsrtombs, mbsnrtowcs, and wcsnrtombs.
* debug/Makefile (routines): Add wcrtomb_chk, mbsrtowcs_chk,
wcsrtombs_chk, mbsnrtowcs_chk, and wcsnrtombs_chk.
* debug/Versions: Add __wcrtomb_chk, __mbsrtowcs_chk,
__wcsrtombs_chk, __mbsnrtowcs_chk, and __wcsnrtombs_chk.
* debug/tst-chk1.c: Add tests for new functions.
* debug/mbsnrtowcs_chk.c: New file.
* debug/mbsrtowcs_chk.c: New file.
* debug/wcrtomb_chk.c: New file.
* debug/wcsnrtombs_chk.c: New file.
* debug/wcsrtombs_chk.c: New file.
* include/stdio.h: Add declaration for __fxprintf.
2005-07-20 07:43:27 +00:00
|
|
|
This file is part of the GNU C Library.
|
|
|
|
|
|
|
|
|
|
The GNU C Library is free software; you can redistribute it and/or
|
|
|
|
|
modify it under the terms of the GNU Lesser General Public
|
|
|
|
|
License as published by the Free Software Foundation; either
|
|
|
|
|
version 2.1 of the License, or (at your option) any later version.
|
|
|
|
|
|
|
|
|
|
The GNU C Library is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
|
|
|
Lesser General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU Lesser General Public
|
2012-02-09 23:18:22 +00:00
|
|
|
License along with the GNU C Library; if not, see
|
Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:
sed -ri '
s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
$(find $(git ls-files) -prune -type f \
! -name '*.po' \
! -name 'ChangeLog*' \
! -path COPYING ! -path COPYING.LIB \
! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
! -path manual/texinfo.tex ! -path scripts/config.guess \
! -path scripts/config.sub ! -path scripts/install-sh \
! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
! -path INSTALL ! -path locale/programs/charmap-kw.h \
! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
! '(' -name configure \
-execdir test -f configure.ac -o -f configure.in ';' ')' \
! '(' -name preconfigure \
-execdir test -f preconfigure.ac ';' ')' \
-print)
and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:
chmod a+x sysdeps/unix/sysv/linux/riscv/configure
# Omit irrelevant whitespace and comment-only changes,
# perhaps from a slightly-different Autoconf version.
git checkout -f \
sysdeps/csky/configure \
sysdeps/hppa/configure \
sysdeps/riscv/configure \
sysdeps/unix/sysv/linux/csky/configure
# Omit changes that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
git checkout -f \
sysdeps/powerpc/powerpc64/ppc-mcount.S \
sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
# Omit change that caused a pre-commit check to fail like this:
# remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
|
|
|
<https://www.gnu.org/licenses/>. */
|
* wcsmbs/bits/wchar2.h: Add definitions for wcrtomb, mbsrtowcs,
wcsrtombs, mbsnrtowcs, and wcsnrtombs.
* debug/Makefile (routines): Add wcrtomb_chk, mbsrtowcs_chk,
wcsrtombs_chk, mbsnrtowcs_chk, and wcsnrtombs_chk.
* debug/Versions: Add __wcrtomb_chk, __mbsrtowcs_chk,
__wcsrtombs_chk, __mbsnrtowcs_chk, and __wcsnrtombs_chk.
* debug/tst-chk1.c: Add tests for new functions.
* debug/mbsnrtowcs_chk.c: New file.
* debug/mbsrtowcs_chk.c: New file.
* debug/wcrtomb_chk.c: New file.
* debug/wcsnrtombs_chk.c: New file.
* debug/wcsrtombs_chk.c: New file.
* include/stdio.h: Add declaration for __fxprintf.
2005-07-20 07:43:27 +00:00
|
|
|
|
|
|
|
|
#include <langinfo.h>
|
|
|
|
|
#include <locale.h>
|
|
|
|
|
#include <stdlib.h>
|
|
|
|
|
#include <wchar.h>
|
|
|
|
|
#include <locale/localeinfo.h>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
size_t
|
|
|
|
|
__wcrtomb_chk (char *s, wchar_t wchar, mbstate_t *ps, size_t buflen)
|
|
|
|
|
{
|
wcrtomb: Make behavior POSIX compliant
The GNU implementation of wcrtomb assumes that there are at least
MB_CUR_MAX bytes available in the destination buffer passed to wcrtomb
as the first argument. This is not compatible with the POSIX
definition, which only requires enough space for the input wide
character.
This does not break much in practice because when users supply buffers
smaller than MB_CUR_MAX (e.g. in ncurses), they compute and dynamically
allocate the buffer, which results in enough spare space (thanks to
usable_size in malloc and padding in alloca) that no actual buffer
overflow occurs. However when the code is built with _FORTIFY_SOURCE,
it runs into the hard check against MB_CUR_MAX in __wcrtomb_chk and
hence fails. It wasn't evident until now since dynamic allocations
would result in wcrtomb not being fortified but since _FORTIFY_SOURCE=3,
that limitation is gone, resulting in such code failing.
To fix this problem, introduce an internal buffer that is MB_LEN_MAX
long and use that to perform the conversion and then copy the resultant
bytes into the destination buffer. Also move the fortification check
into the main implementation, which checks the result after conversion
and aborts if the resultant byte count is greater than the destination
buffer size.
One complication is that applications that assume the MB_CUR_MAX
limitation to be gone may not be able to run safely on older glibcs if
they use static destination buffers smaller than MB_CUR_MAX; dynamic
allocations will always have enough spare space that no actual overruns
will occur. One alternative to fixing this is to bump symbol version to
prevent them from running on older glibcs but that seems too strict a
constraint. Instead, since these users will only have made this
decision on reading the manual, I have put a note in the manual warning
them about the pitfalls of having static buffers smaller than
MB_CUR_MAX and running them on older glibc.
Benchmarking:
The wcrtomb microbenchmark shows significant increases in maximum
execution time for all locales, ranging from 10x for ar_SA.UTF-8 to
1.5x-2x for nearly everything else. The mean execution time however saw
practically no impact, with some results even being quicker, indicating
that cache locality has a much bigger role in the overhead.
Given that the additional copy uses a temporary buffer inside wcrtomb,
it's likely that a hot path will end up putting that buffer (which is
responsible for the additional overhead) in a similar place on stack,
giving the necessary cache locality to negate the overhead. However in
situations where wcrtomb ends up getting called at wildly different
spots on the call stack (or is on different call stacks, e.g. with
threads or different execution contexts) and is still a hotspot, the
performance lag will be visible.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
2022-05-13 13:40:15 +00:00
|
|
|
return __wcrtomb_internal (s, wchar, ps, buflen);
|
* wcsmbs/bits/wchar2.h: Add definitions for wcrtomb, mbsrtowcs,
wcsrtombs, mbsnrtowcs, and wcsnrtombs.
* debug/Makefile (routines): Add wcrtomb_chk, mbsrtowcs_chk,
wcsrtombs_chk, mbsnrtowcs_chk, and wcsnrtombs_chk.
* debug/Versions: Add __wcrtomb_chk, __mbsrtowcs_chk,
__wcsrtombs_chk, __mbsnrtowcs_chk, and __wcsnrtombs_chk.
* debug/tst-chk1.c: Add tests for new functions.
* debug/mbsnrtowcs_chk.c: New file.
* debug/mbsrtowcs_chk.c: New file.
* debug/wcrtomb_chk.c: New file.
* debug/wcsnrtombs_chk.c: New file.
* debug/wcsrtombs_chk.c: New file.
* include/stdio.h: Add declaration for __fxprintf.
2005-07-20 07:43:27 +00:00
|
|
|
}
|
2023-06-07 09:06:50 +00:00
|
|
|
libc_hidden_def (__wcrtomb_chk)
|
