Continuing the consolidation on Python for various miscellaneous build
and test scripts, this patch moves conformtest from Perl to Python.
The substance of the tests run is intended to be the same as before,
except that the previous test for tags did not actually achieve the
intended purpose of verifying whether a tag was already declared, so
is changed to one that would actually fail for a tag that wasn't
declared, and a typo in the old test for variables being available
($xyzzy instead of xyzzy) would have made that test not use the
correct type (but it would have passed anyway with warnings). No
attempt is made to keep the details of what the test output looks
like; instead, tests are given names which are made to follow PASS: /
FAIL: / XFAIL: / SKIP: / MISSING: as appropriate.
In the new version, there is more consistent parsing of test lines
(into a series of words, either surrounded by {} or separated by
spaces) that applies for all kinds of test lines, rather than the old
approach of different regular expressions for every kind of test. A
few of the conform/data/ files are adjusted so their syntax works with
the new script (which now requires spaces in certain cases where the
old script tolerated them being missing, and does not allow stray
semicolons at the end of "function" lines). Similarly, common logic
is used around what happens with a second subtest if a first one fails
(e.g., a test for a symbol's type if the test for availability fails),
rather than such logic being replicated separately for each kind of
test. Common parsing also applies for test lines both when they are
lines for the header under test and when they are lines for another
header specified with allow-header, again unlike the old script.
Tested for x86_64, and with build-many-glibcs.py.
* conform/conformtest.py: New file.
* conform/conformtest.pl: Remove.
* conform/GlibcConform.pm: Likewise.
* conform/glibcconform.py (KEYWORDS_C90): New constant.
(KEYWORDS_C99): Likewise.
(KEYWORDS): Likewise.
* conform/Makefile ($(conformtest-header-tests)): Use
conformtest.py instead of conformtest.pl. Do not pass --tmpdir
option. Use --header instead of --headers.
* conform/data/arpa/inet.h-data: Remove trailing semicolons on
function entries.
* conform/data/spawn.h-data: Likewise.
* conform/data/fcntl.h-data (openat): Add space after function
name.
* conform/data/wchar.h-data (wcscasecmp): Likewise.
(wcscasecmp_l): Likewise.
* conform/data/termios.h-data (c_cc): Add space after element
name.
The commit
commit 1df872fd74
Author: Florian Weimer <fweimer@redhat.com>
Date: Wed Nov 7 12:42:44 2018 +0100
support: Implement TEST_COMPARE_STRING
added the new macro TEST_COMPARE_STRING, which compares the output of
functions under test against expected strings, and, when there's a
mismatch, automatically reports an error and prints the differences.
This patch adapts recently added test cases to use this new macro.
Tested for powerpc64le (as is, and locally patched to intentionally fail
and produce error output).
* argp/tst-ldbl-argp.c (do_one_test): Use TEST_COMPARE_STRING,
instead of manually comparing and reporting mismatching strings.
* misc/tst-ldbl-error.c (do_one_test): Likewise.
* misc/tst-ldbl-warn.c (do_one_test): Likewise.
The __ASSUME_SOCKETCALL macro in kernel-features.h is no longer used
for anything. (It used to be used in defining other macros related to
accept4 / recvmmsg / sendmmsg availability, but the code in that area
was simplified once we could assume a kernel with those features,
whether through a syscall or through socketcall, so allowing those
functions to be handled much like other socket operations, without
requring __ASSUME_SOCKETCALL.) This patch removes that unused macro.
(Note: once we can assume a Linux 4.4 or later kernel, much of the
support for using socketcall at all can be removed from glibc,
although a few functions may need that support in glibc for longer.)
Tested with build-many-glibcs.py.
* sysdeps/unix/sysv/linux/kernel-features.h: Remove comment about
__ASSUME_SOCKETCALL.
* sysdeps/unix/sysv/linux/i386/kernel-features.h
(__ASSUME_SOCKETCALL): Remove.
* sysdeps/unix/sysv/linux/m68k/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
* sysdeps/unix/sysv/linux/microblaze/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
* sysdeps/unix/sysv/linux/powerpc/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
* sysdeps/unix/sysv/linux/s390/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
* sysdeps/unix/sysv/linux/sh/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/kernel-features.h
(__ASSUME_SOCKETCALL): Likewise.
Linkers group input note sections with the same name into one output
note section with the same name. One output note section is placed in
one PT_NOTE segment. Since new linkers merge input .note.gnu.property
sections into one output .note.gnu.property section, there is only
one NT_GNU_PROPERTY_TYPE_0 note in one PT_NOTE segment with new linkers.
Since older linkers treat input .note.gnu.property section as a generic
note section and just concatenate all input .note.gnu.property sections
into one output .note.gnu.property section without merging them, we may
see multiple NT_GNU_PROPERTY_TYPE_0 notes in one PT_NOTE segment with
older linkers.
When an older linker is used to created the program on CET-enabled OS,
the linker output has a single .note.gnu.property section with multiple
NT_GNU_PROPERTY_TYPE_0 notes, some of which have IBT and SHSTK enable
bits set even if the program isn't CET enabled. Such programs will
crash on CET-enabled machines. This patch updates the note parser:
1. Skip note parsing if a NT_GNU_PROPERTY_TYPE_0 note has been processed.
2. Check multiple NT_GNU_PROPERTY_TYPE_0 notes.
[BZ #23509]
* sysdeps/x86/dl-prop.h (_dl_process_cet_property_note): Skip
note parsing if a NT_GNU_PROPERTY_TYPE_0 note has been processed.
Update the l_cet field when processing NT_GNU_PROPERTY_TYPE_0 note.
Check multiple NT_GNU_PROPERTY_TYPE_0 notes.
* sysdeps/x86/link_map.h (l_cet): Expand to 3 bits, Add
lc_unknown.
The generic kernel-features.h defines __ASSUME_MLOCK2 for 4.4 and
later kernels. However, for 32-bit ARM binaries running on 64-bit ARM
kernels, and for MicroBlaze, the syscall was only wired up in the 4.7
kernel. (32-bit ARM kernels did have the syscall from 4.4 onwards.)
This patch duly arranges for the macro to be undefined for those
architectures for kernels before 4.7.
Tested with build-many-glibcs.py for its ARM and MicroBlaze
configurations.
[BZ #23867]
* sysdeps/unix/sysv/linux/arm/kernel-features.h
[__LINUX_KERNEL_VERSION < 0x040700] (__ASSUME_MLOCK2): Undefine.
* sysdeps/unix/sysv/linux/microblaze/kernel-features.h
[__LINUX_KERNEL_VERSION < 0x040700] (__ASSUME_MLOCK2): Undefine.
Fix the following on 32 bits targets:
support_test_compare_string.c: In function ‘support_test_compare_string’:
support_test_compare_string.c:80:37: error: format ‘%lu’ expects argument of
type ‘long unsigned int’, but argument 2 has type ‘size_t’ {aka ‘unsigned int’}
[-Werror=format=]
printf (" string length: %lu bytes\n", left_length);
~~^ ~~~~~~~~~~~
%u
Checked on arm-linux-gnueabihf.
* support/support_test_compare_string.c
(support_test_compare_string): Fix printf format.
The SH kernel-features.h undefines __ASSUME_RENAMEAT2 for kernel
versions before 4.8, but fails to undefine __ASSUME_EXECVEAT,
__ASSUME_MLOCK2 and __ASSUME_COPY_FILE_RANGE, although all those
syscalls (and several others) were added for SH in the same Linux
kernel commit (first released in 4.8). This patch adds the proper
undefines of those macros.
Tested with build-many-glibcs.py for its SH configurations.
[BZ #23862]
* sysdeps/unix/sysv/linux/sh/kernel-features.h
[__LINUX_KERNEL_VERSION < 0x040800] (__ASSUME_EXECVEAT): Undefine.
[__LINUX_KERNEL_VERSION < 0x040800] (__ASSUME_MLOCK2): Likewise.
[__LINUX_KERNEL_VERSION < 0x040800] (__ASSUME_COPY_FILE_RANGE):
Likewise.
Similarly to what has been done for argp_error, and argp_failure, as
well as for warn, warnx, vwarn, and vwarnx, this patch adds new tests
for the following functions: err, errx, verr, verrx, error, and
error_at_line. The new tests check that the conversion of long double
variables into string works correctly on the default format of the type.
Future patches will reuse these tests for other formats that long double
can take.
Tested for powerpc64le.
* misc/Makefile (tests): Add tst-ldbl-error.
* misc/tst-ldbl-error.c: New file.
Similarly to what has been done for argp_error and argp_failure, this
patch patch adds new tests for the warn, warnx, vwarn, and vwarnx
functions. The new tests use the format string to request the
conversion of long double parameters into string. Currently, these
tests only check that the default format of the long double type works.
Future patches will extend the test for platforms that can have an
optional format for long double.
Tested for powerpc64le.
* misc/Makefile (tests): Add tst-ldbl-warn.
* misc/tst-ldbl-warn.c: New file.
The functions argp_error and argp_failure, from argp.h, have a format
string as parameter, which can possibly request the printing of
floating-point values. These values could be of long double type, which
can have different formats, depending on the architecture and on
compilation parameters (for instance, on powerpc, long double values can
have double format (-mlong-double-64) or IBM Extended Precision format
(-mlong-double-128)).
This patch adds tests for argp_error and argp_failure that contain a
format string with double and long double conversion specifiers ('%f'
and '%Lf'). These tests automatically check that the default format of
the long double type works. A future patch will extend the test for
platforms that can have an optional format for long double.
Tested for powerpc64le.
* argp/Makefile (tests): Add tst-ldbl-argp.
* argp/tst-ldbl-argp.c: New file.
__gconv_read_conf is only ever called once during the program's lifetime.
This means that __gconv_path_elem is always uninitialized when the function
begins executing. __gconv_get_path has an assert to ensure that this
expected runtime behaviour is always exhibited. Given this, checking for a
NULL value before calling __gconv_get_path is unnecessary. This commit
drops the condition and calls __gconv_get_path unconditionally.
In the description of gcvt, add missing closing parenthesis in the
comparison with sprintf.
* manual/arith.texi (gcvt): Add missing closing parenthesis.
Here only add the implementation when building the RV32 port.
These macros are used when the following situations occur at the same
time: soft-fp fma, ldbl-128 and 32-bit _FP_W_TYPE_SIZE. The RISC-V
32-bit port is the first port which use all three together.
This is the building flow about the situation:
When building soft-fp/s_fmal.c, there uses the FP_FMA_Q in __fmal.
The _FP_W_TYPE_SIZE is defined to 32-bit in sysdeps/riscv/sfp-machine.h,
so the FP_FMA_Q was defined to _FP_FMA (Q, 4, 8, R, X, Y, Z) in
soft-fp/quad.h.
Something in the soft-fp/quad.h:
#if _FP_W_TYPE_SIZE < 64
# define FP_FMA_Q(R, X, Y, Z) _FP_FMA (Q, 4, 8, R, X, Y, Z)
#else
# define FP_FMA_Q(R, X, Y, Z) _FP_FMA (Q, 2, 4, R, X, Y, Z)
#endif
Finally, in _FP_FMA (fs, wc, dwc, R, X, Y, Z), it will use the
_FP_FRAC_HIGHBIT_DW_##dwc macro, and it will be expanded to
_FP_FRAC_HIGHBIT_DW_8, but the _FP_FRAC_HIGHBIT_DW_8 is not be
implemented in soft-fp/op-8.h. there is only _FP_FRAC_HIGHBIT_DW_1,
_FP_FRAC_HIGHBIT_DW_2 and _FP_FRAC_HIGHBIT_DW_4 in the
soft-fp/op-*.h.
After this modification, we can pass the soft floating testing of glibc
testsuites on RV32.
* soft-fp/op-8.h (_FP_FRAC_SET_8, _FP_FRAC_ADD_8, _FP_FRAC_SUB_8)
(_FP_FRAC_CLZ_8, _FP_MINFRAC_8, _FP_FRAC_NEGP_8, _FP_FRAC_ZEROP_8)
(_FP_FRAC_HIGHBIT_DW_8, _FP_FRAC_COPY_4_8, _FP_FRAC_COPY_8_4)
(__FP_FRAC_SET_8): Add implementation for RV32 use.
In FRAC_SUB_3(R, X, Y) and FRAC_SUB_4(R,, X, Y), it reference both
the X[N] and X[N] after R[N] have been set. If one of the X and Y is
the same address with R, the result of the calculation is wrong,
because the value of the original X and Y are overwritten.
In glibc, there are two places use FRAC_SUB and occurs the overlap.
The first is _FP_DIV_MEAT_N_loop in op-common.h, it uses the source
_FP_DIV_MEAT_N_loop_u as the destination. This macro only be used
when N is one(_FP_DIV_MEAT_1_loop) and then the _FP_FRAC_SUB_##wc
extend to _FP_FRAC_SUB_1 in this macro. so it also work because
_FP_FRAC_SUB_1 has no overlap problem in its implementation.
The second places is _FP_DIV_MEAT_4_udiv, the original value of X##_f[0]
is overwritten before the calculatation.
In FRAC_SUB_1 and FRAC_SUB_2, there don't refer the source after
destination have been set, so they have no problem.
After this modification, we can pass the soft floating testing of glibc
testsuites on RV32.
* soft-fp/op-4.h (_FP_FRAC_SUB_3, _FP_FRAC_SUB_4): Use temporary
variable to avoid overlap arguments.
Building posix/bug-regex22.c fails with GCC mainline because of
-Wformat-overflow= warnings for NULL arguments to %s formats.
This is *not* testing how glibc handles such format arguments; in the
context of the messages in question it makes no sense to pass NULL to
such a %s format (the code passes s, inside "if (s == NULL)"). So
this patch changes the code not to pass such a format argument at all
(which means the string passed is constant, so no need to use printf
at all - however, there are two separate tests here with different
length arguments passed to re_compile_pattern, so it *does* make sense
to make the strings used different so that in the event of failure
it's clear which one of the tests failed).
Tested with build-many-glibcs.py with GCC mainline for
aarch64-linux-gnu.
* posix/bug-regex22.c (main): Use puts with distinct error
messages for unexpected success of re_compile_pattern, not printf
with NULL argument to %s.
Recent GCC -Wformat-overflow= changes result in some printf tests
failing to build, because those tests are deliberately testing the
handling of formats writing more than INT_MAX characters and the
handling of NULL arguments to the %s format, which GCC now warns
about. This patch duly disables -Wformat-overflow= for the relevant
calls to printf functions.
Tested with build-many-glibcs.py with GCC mainline for
aarch64-linux-gnu.
* stdio-common/bug22.c: Include <libc-diag.h>.
(do_test): Disable -Wformat-overflow= warnings around fprintf
calls outputting more than INT_MAX characters.
* stdio-common/tst-printf.c: Disable -Wformat-overflow= warnings
around printf call with NULL %s argument.
Looking at kernel-features.h files, I saw that SPARC was missing full
information on when it gained separate socket syscalls.
This patch adds such information to the SPARC kernel-features.h. It
also corrects what appear to be bugs in the existing code (that would
cause syscalls to be assumed to be present when not actually present).
Various __ASSUME_* macros, defined by default, were not undefined for
32-bit despite those syscalls only being added for 32-bit in Linux
4.4. Some syscalls were used in the SPARC64 syscalls.list but only
added in 4.4; this was harmless before the __NR_* macros were defined
at all, but once the macros were defined it means a build with
post-4.4 headers would assume the syscalls to be present regardless of
--enable-kernel version. Then, various __ASSUME_* macros were
previously not defined in cases where they could be defined (this part
of the patch is just an optimization, not a bug fix).
Note the observation in a comment in the patch that even the latest
Linux kernel for SPARC does not have getpeername and getsockname
syscalls in the compat syscall table for 32-bit binaries on 64-bit
kernels (so glibc can't assume those syscalls to be present for 32-bit
at all, although the 32-bit syscall table gained them in 4.4).
Tested (compilation only) for SPARC with build-many-glibcs.py.
[BZ #23848]
* sysdeps/unix/sysv/linux/sparc/kernel-features.h [!__arch64__ &&
__LINUX_KERNEL_VERSION < 0x040400] (__ASSUME_SENDMSG_SYSCALL):
Undefine.
[!__arch64__ && __LINUX_KERNEL_VERSION < 0x040400]
(__ASSUME_RECVMSG_SYSCALL): Likewise.
[!__arch64__ && __LINUX_KERNEL_VERSION < 0x040400]
(__ASSUME_SENDTO_SYSCALL): Likewise.
[!__arch64__ && __LINUX_KERNEL_VERSION < 0x040400]
(__ASSUME_ACCEPT_SYSCALL): Undefine under this condition, not just
[!__arch64__].
[!__arch64__ && __LINUX_KERNEL_VERSION < 0x040400]
(__ASSUME_CONNECT_SYSCALL): Likewise.
[!__arch64__ && __LINUX_KERNEL_VERSION < 0x040400]
(__ASSUME_RECVFROM_SYSCALL): Likewise.
[__LINUX_KERNEL_VERSION >= 0x040400] (__ASSUME_BIND_SYSCALL):
Define.
[__LINUX_KERNEL_VERSION >= 0x040400] (__ASSUME_LISTEN_SYSCALL):
Likewise.
[__LINUX_KERNEL_VERSION >= 0x040400]
(__ASSUME_SETSOCKOPT_SYSCALL): Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/syscalls.list (bind):
Remove.
(listen): Likewise.
(setsockopt): Likewise.
GAS treats the R5900 as MIPS III, with some modifications. The MIPS III
designation means that the GNU C Library will try to assemble the LL and
SC instructions, even though they are not implemented in the R5900. GAS
will therefore produce the following errors:
Error: opcode not supported on this processor: r5900 (mips3) `ll $2,0($4)'
Error: opcode not supported on this processor: r5900 (mips3) `sc $6,0($4)'
The MIPS II ISA override as used here enables the kernel to trap and
emulate the LL and SC instructions, as required.
This change has been tested by compiling the GNU C Library 2.27 with a
GCC 8.2.0 cross-compiler for mipsr5900el-unknown-linux-gnu under Gentoo.
* sysdeps/mips/sys/tas.h (_test_and_set): Handle the R5900 CPU
with the ISA override.
The #else of two nested #if clauses were identical.
* sysdeps/unix/sysv/linux/sysdep-vdso.h: Simplify an #if #else
#endif.
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
This patch replaces conform/linknamespace.pl with a new
conform/linknamespace.py, so continuing the consolidation on Python
instead of Perl for miscellaneous scripts used in building and testing
glibc. The new script follows the same logic as the old one; as a
recently-added script, there were no major cleanups to be made in the
course of the language conversion.
Tested for x86_64, and with build-many-glibcs.py. For x86_64 I also
tested that if the Perl and Python scripts were made to print all the
symbols in seen_where and the paths of symbol references by which
those symbols were linked in, even when those symbols were OK,
identical symbol lists appeared in the output with both versions of
the script (the differences in linknamespace.out files were only in
paths to temporary files in diagnostics for e.g. deprecated functions,
and error output for the expected compilation failures when testing
ndbm.h and varargs.h).
* conform/linknamespace.py: New file.
* conform/linknamespace.pl: Remove file.
* conform/Makefile ($(linknamespace-header-tests)): Use
linknamespace.py instead of linknamespace.pl. Do not use --tmpdir
option.
On systems without enough random-access memory, stdlib/test-bz22786
will go deeply into swap and time out, even with a substantial
TIMEOUTFACTOR. This commit adds a facility to construct repeating
strings with alias mappings, so that the requirement for physical
memory, and uses it in stdlib/test-bz22786.
Mark the ra register as undefined in _start, so that unwinding through
main works correctly. Also, don't use a tail call so that ra points after
the call to __libc_start_main, not after the previous call.
Now that we require Python 3.4 or later, Python code creating
temporary directories can use tempfile.TemporaryDirectory in "with" to
have the directory deleted automatically instead of needing to use
try/finally to handle removing a directory created with
tempfile.mkdtemp. This patch does so in conform/glibcconform.py.
Tested for x86_64.
* conform/glibcconform.py: Do not import shutil.
(list_exported_functions): Use tempfile.TemporaryDirectory instead
of mkdtemp.
This patch makes Python 3.4 or later a required tool for building
glibc, so allowing changes of awk, perl etc. code used in the build
and test to Python code without any such changes needing makefile
conditionals or to handle older Python versions.
This patch makes the configure test for Python check the version and
give an error if Python is missing or too old, and removes makefile
conditionals that are no longer needed. It does not itself convert
any code from another language to Python, and does not remove any
compatibility with older Python versions from existing scripts.
Tested for x86_64.
* configure.ac (PYTHON_PROG): Use AC_CHECK_PROG_VER. Set
critic_missing for versions before 3.4.
* configure: Regenerated.
* manual/install.texi (Tools for Compilation): Document
requirement for Python to build glibc.
* INSTALL: Regenerated.
* Rules [PYTHON]: Make code unconditional.
* benchtests/Makefile [PYTHON]: Likewise.
* conform/Makefile [PYTHON]: Likewise.
* manual/Makefile [PYTHON]: Likewise.
* math/Makefile [PYTHON]: Likewise.
* sysdeps/mach/hurd/i386/intr-msg.h (INTR_MSG_TRAP): Make
_hurd_intr_rpc_msg_about_to global point to start of controlled
assembly snippet. Make it check canceled flag.
* hurd/hurdsig.c (_hurdsig_abort_rpcs): Only mutate thread if it passed
the _hurd_intr_rpc_msg_about_to point.
* hurd/intr-msg.c (_hurd_intr_rpc_mach_msg): Remove comment on mutation
issue, remove cancel flag check.
* sysdeps/mach/hurd/i386/intr-msg.h (INTR_MSG_TRAP): Set
intr_port to 0 when canceled.
* hurd/intr-msg.c (_hurd_intr_rpc_mach_msg): Pass intr_port
address.
* sysdeps/mach/hurd/i386/intr-msg.h (INTR_MSG_TRAP): Make
_hurd_intr_rpc_msg_about_to global point to start of controlled
assembly snippet. Make it check canceled flag again.
* hurd/hurdsig.c (_hurdsig_abort_rpcs): Only mutate thread if it passed the
_hurd_intr_rpc_msg_about_to point.
* hurd/intr-msg.c (_hurd_intr_rpc_mach_msg): Remove comment on mutation
issue.
since we do not actually know whether the RPC was completed or not,
which makes a huge difference for e.g. write(), so better really error
out than letting caller think that the RPC did not happen.
* hurd/intr-msg.c (_hurd_intr_rpc_mach_msg): When the server does not
answer to interrupt_operation, return EIO instead of EINTR.
Seeing a server not able to get interrupted for 3s is not so surprising when
e.g. a lot of writes are happening. 1 minute allows to actually notice the
issue and be able to debug it.
* hurd/hurdsig.c (_hurd_interrupted_rpc_timeout): Set to 60000.
Since we have consensus on requiring Python 3.4 or later to build
glibc, it follows that compatibility with older Python versions is
also no longer relevant to auxiliary Python scripts for use in glibc
development. This patch removes such compatibility code from
build-many-glibcs.py (compatibility code needed for 3.4, which lacks
the newer subprocess interface, is kept). Because
build-many-glibcs.py is not itself called from the glibc build system,
this patch is independent of the configure checks for having a
new-enough Python version, which are only relevant to uses of Python
from the main build and test process.
Tested with build-many-glibcs.py building glibc for aarch64-linux-gnu
(with Python 3.4 to make sure that still works).
* scripts/build-many-glibcs.py: Remove compatibility for missing
os.cpu_count and re.fullmatch.
