glibc/scripts/gen-as-const.py

121 lines
4.5 KiB
Python
Raw Normal View History

Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
#!/usr/bin/python3
# Produce headers of assembly constants from C expressions.
# Copyright (C) 2018-2020 Free Software Foundation, Inc.
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
# This file is part of the GNU C Library.
#
# The GNU C Library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# The GNU C Library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with the GNU C Library; if not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
# <https://www.gnu.org/licenses/>.
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
# The input to this script looks like:
# #cpp-directive ...
# NAME1
# NAME2 expression ...
# A line giving just a name implies an expression consisting of just that name.
import argparse
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
import glibcextract
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
def gen_test(sym_data):
"""Generate a test for the values of some C constants.
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
The first argument is as for glibcextract.compute_c_consts.
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
"""
out_lines = []
for arg in sym_data:
if isinstance(arg, str):
Make gen-as-const.py handle '--' consistently with awk script. It was reported in <https://sourceware.org/ml/libc-alpha/2018-12/msg00045.html> that gen-as-const.py fails to generate test code in the case where a .sym file has no symbols in it, so resulting in a test failing to link for Hurd. The relevant difference from the old awk script is that the old script treated '--' lines as indicating that the text to do at the start of the test (or file used to compute constants) should be output at that point if not already output, as well as treating lines with actual entries for constants like that. This patch changes gen-as-const.py accordingly, making it the sole responsibility of the code parsing .sym files to determine when such text should be output and ensuring it's always output at some point even if there are no symbols and no '--' lines, since not outputting it means the test fails to link. Handling '--' like that also avoids any problems that would arise if the first entry for a symbol were inside #ifdef (since the text in question must not be output inside #ifdef). Tested for x86_64, and with build-many-glibcs.py for i686-gnu. Note that there are still compilation test failures for i686-gnu (linknamespace tests, possibly arising from recent posix_spawn-related changes). * scripts/gen-as-const.py (compute_c_consts): Take an argument 'START' to indicate that start text should be output. (gen_test): Likewise. (main): Generate 'START' for first symbol or '--' line, or at end of input if not previously generated.
2018-12-03 22:08:50 +00:00
if arg == 'START':
out_lines.append('#include <stdint.h>\n'
'#include <stdio.h>\n'
'#include <bits/wordsize.h>\n'
'#if __WORDSIZE == 64\n'
'typedef uint64_t c_t;\n'
'# define U(n) UINT64_C (n)\n'
'#else\n'
'typedef uint32_t c_t;\n'
'# define U(n) UINT32_C (n)\n'
'#endif\n'
'static int\n'
'do_test (void)\n'
'{\n'
# Compilation test only, using static
# assertions.
' return 0;\n'
'}\n'
'#include <support/test-driver.c>')
else:
out_lines.append(arg)
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
continue
name = arg[0]
value = arg[1]
out_lines.append('_Static_assert (U (asconst_%s) == (c_t) (%s), '
'"value of %s");'
% (name, value, name))
return '\n'.join(out_lines)
def main():
"""The main entry point."""
parser = argparse.ArgumentParser(
description='Produce headers of assembly constants.')
parser.add_argument('--cc', metavar='CC',
help='C compiler (including options) to use')
parser.add_argument('--test', action='store_true',
help='Generate test case instead of header')
Use gen-as-const.py to process .pysym files. This patch eliminates the gen-py-const.awk variant of gen-as-const, switching to use of gnu-as-const.py (with a new --python option) to process .pysym files (i.e., to generate nptl_lock_constants.py), as the syntax of those files is identical to that of .sym files. Note that the generated nptl_lock_constants.py is *not* identical to the version generated by the awk script. Apart from the trivial changes (comment referencing the new script, and output being sorted), the constant FUTEX_WAITERS, PTHREAD_MUTEXATTR_FLAG_BITS, PTHREAD_MUTEXATTR_FLAG_PSHARED and PTHREAD_MUTEX_PRIO_CEILING_MASK are now output as positive rather than negative constants (on x86_64 anyway; maybe not necessarily on 32-bit systems): < FUTEX_WAITERS = -2147483648 --- > FUTEX_WAITERS = 2147483648 < PTHREAD_MUTEXATTR_FLAG_BITS = -251662336 < PTHREAD_MUTEXATTR_FLAG_PSHARED = -2147483648 --- > PTHREAD_MUTEXATTR_FLAG_BITS = 4043304960 > PTHREAD_MUTEXATTR_FLAG_PSHARED = 2147483648 < PTHREAD_MUTEX_PRIO_CEILING_MASK = -524288 --- > PTHREAD_MUTEX_PRIO_CEILING_MASK = 4294443008 This is because gen-as-const has a cast of the constant value to long int, which gen-py-const lacks. I think the positive values are more logically correct, since the constants in question are in fact unsigned in C. But to reliably produce gen-as-const.py output for constants that always (in C and Python) reflects the signedness of values with the high bit of "long int" set would mean more complicated logic needs to be used in computing values. The more correct positive values by themselves produce a failure of nptl/test-mutexattr-printers, because masking with ~PTHREAD_MUTEXATTR_FLAG_BITS & ~PTHREAD_MUTEX_NO_ELISION_NP now leaves a bit -1 << 32 in the Python value, resulting in a KeyError exception. To avoid that, places masking with ~ of one of the constants in question are changed to mask with 0xffffffff as well (this reflects how ~ in Python applies to an infinite-precision integer whereas ~ in C does not do any promotions beyond the width of int). Tested for x86_64. * scripts/gen-as-const.py (main): Handle --python option. * scripts/gen-py-const.awk: Remove. * Makerules (py-const-script): Use gen-as-const.py. ($(py-const)): Likewise. * nptl/nptl-printers.py (MutexPrinter.read_status_no_robust): Mask with 0xffffffff together with ~(PTHREAD_MUTEX_PRIO_CEILING_MASK). (MutexAttributesPrinter.read_values): Mask with 0xffffffff together with ~PTHREAD_MUTEXATTR_FLAG_BITS and ~PTHREAD_MUTEX_NO_ELISION_NP. * manual/README.pretty-printers: Update reference to gen-py-const.awk.
2018-12-10 22:56:59 +00:00
parser.add_argument('--python', action='store_true',
help='Generate Python file instead of header')
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
parser.add_argument('sym_file',
help='.sym file to process')
args = parser.parse_args()
sym_data = []
with open(args.sym_file, 'r') as sym_file:
Make gen-as-const.py handle '--' consistently with awk script. It was reported in <https://sourceware.org/ml/libc-alpha/2018-12/msg00045.html> that gen-as-const.py fails to generate test code in the case where a .sym file has no symbols in it, so resulting in a test failing to link for Hurd. The relevant difference from the old awk script is that the old script treated '--' lines as indicating that the text to do at the start of the test (or file used to compute constants) should be output at that point if not already output, as well as treating lines with actual entries for constants like that. This patch changes gen-as-const.py accordingly, making it the sole responsibility of the code parsing .sym files to determine when such text should be output and ensuring it's always output at some point even if there are no symbols and no '--' lines, since not outputting it means the test fails to link. Handling '--' like that also avoids any problems that would arise if the first entry for a symbol were inside #ifdef (since the text in question must not be output inside #ifdef). Tested for x86_64, and with build-many-glibcs.py for i686-gnu. Note that there are still compilation test failures for i686-gnu (linknamespace tests, possibly arising from recent posix_spawn-related changes). * scripts/gen-as-const.py (compute_c_consts): Take an argument 'START' to indicate that start text should be output. (gen_test): Likewise. (main): Generate 'START' for first symbol or '--' line, or at end of input if not previously generated.
2018-12-03 22:08:50 +00:00
started = False
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
for line in sym_file:
line = line.strip()
if line == '':
continue
# Pass preprocessor directives through.
if line.startswith('#'):
sym_data.append(line)
continue
words = line.split(maxsplit=1)
Make gen-as-const.py handle '--' consistently with awk script. It was reported in <https://sourceware.org/ml/libc-alpha/2018-12/msg00045.html> that gen-as-const.py fails to generate test code in the case where a .sym file has no symbols in it, so resulting in a test failing to link for Hurd. The relevant difference from the old awk script is that the old script treated '--' lines as indicating that the text to do at the start of the test (or file used to compute constants) should be output at that point if not already output, as well as treating lines with actual entries for constants like that. This patch changes gen-as-const.py accordingly, making it the sole responsibility of the code parsing .sym files to determine when such text should be output and ensuring it's always output at some point even if there are no symbols and no '--' lines, since not outputting it means the test fails to link. Handling '--' like that also avoids any problems that would arise if the first entry for a symbol were inside #ifdef (since the text in question must not be output inside #ifdef). Tested for x86_64, and with build-many-glibcs.py for i686-gnu. Note that there are still compilation test failures for i686-gnu (linknamespace tests, possibly arising from recent posix_spawn-related changes). * scripts/gen-as-const.py (compute_c_consts): Take an argument 'START' to indicate that start text should be output. (gen_test): Likewise. (main): Generate 'START' for first symbol or '--' line, or at end of input if not previously generated.
2018-12-03 22:08:50 +00:00
if not started:
sym_data.append('START')
started = True
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
# Separator.
if words[0] == '--':
continue
name = words[0]
value = words[1] if len(words) > 1 else words[0]
sym_data.append((name, value))
Make gen-as-const.py handle '--' consistently with awk script. It was reported in <https://sourceware.org/ml/libc-alpha/2018-12/msg00045.html> that gen-as-const.py fails to generate test code in the case where a .sym file has no symbols in it, so resulting in a test failing to link for Hurd. The relevant difference from the old awk script is that the old script treated '--' lines as indicating that the text to do at the start of the test (or file used to compute constants) should be output at that point if not already output, as well as treating lines with actual entries for constants like that. This patch changes gen-as-const.py accordingly, making it the sole responsibility of the code parsing .sym files to determine when such text should be output and ensuring it's always output at some point even if there are no symbols and no '--' lines, since not outputting it means the test fails to link. Handling '--' like that also avoids any problems that would arise if the first entry for a symbol were inside #ifdef (since the text in question must not be output inside #ifdef). Tested for x86_64, and with build-many-glibcs.py for i686-gnu. Note that there are still compilation test failures for i686-gnu (linknamespace tests, possibly arising from recent posix_spawn-related changes). * scripts/gen-as-const.py (compute_c_consts): Take an argument 'START' to indicate that start text should be output. (gen_test): Likewise. (main): Generate 'START' for first symbol or '--' line, or at end of input if not previously generated.
2018-12-03 22:08:50 +00:00
if not started:
sym_data.append('START')
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
if args.test:
print(gen_test(sym_data))
Use gen-as-const.py to process .pysym files. This patch eliminates the gen-py-const.awk variant of gen-as-const, switching to use of gnu-as-const.py (with a new --python option) to process .pysym files (i.e., to generate nptl_lock_constants.py), as the syntax of those files is identical to that of .sym files. Note that the generated nptl_lock_constants.py is *not* identical to the version generated by the awk script. Apart from the trivial changes (comment referencing the new script, and output being sorted), the constant FUTEX_WAITERS, PTHREAD_MUTEXATTR_FLAG_BITS, PTHREAD_MUTEXATTR_FLAG_PSHARED and PTHREAD_MUTEX_PRIO_CEILING_MASK are now output as positive rather than negative constants (on x86_64 anyway; maybe not necessarily on 32-bit systems): < FUTEX_WAITERS = -2147483648 --- > FUTEX_WAITERS = 2147483648 < PTHREAD_MUTEXATTR_FLAG_BITS = -251662336 < PTHREAD_MUTEXATTR_FLAG_PSHARED = -2147483648 --- > PTHREAD_MUTEXATTR_FLAG_BITS = 4043304960 > PTHREAD_MUTEXATTR_FLAG_PSHARED = 2147483648 < PTHREAD_MUTEX_PRIO_CEILING_MASK = -524288 --- > PTHREAD_MUTEX_PRIO_CEILING_MASK = 4294443008 This is because gen-as-const has a cast of the constant value to long int, which gen-py-const lacks. I think the positive values are more logically correct, since the constants in question are in fact unsigned in C. But to reliably produce gen-as-const.py output for constants that always (in C and Python) reflects the signedness of values with the high bit of "long int" set would mean more complicated logic needs to be used in computing values. The more correct positive values by themselves produce a failure of nptl/test-mutexattr-printers, because masking with ~PTHREAD_MUTEXATTR_FLAG_BITS & ~PTHREAD_MUTEX_NO_ELISION_NP now leaves a bit -1 << 32 in the Python value, resulting in a KeyError exception. To avoid that, places masking with ~ of one of the constants in question are changed to mask with 0xffffffff as well (this reflects how ~ in Python applies to an infinite-precision integer whereas ~ in C does not do any promotions beyond the width of int). Tested for x86_64. * scripts/gen-as-const.py (main): Handle --python option. * scripts/gen-py-const.awk: Remove. * Makerules (py-const-script): Use gen-as-const.py. ($(py-const)): Likewise. * nptl/nptl-printers.py (MutexPrinter.read_status_no_robust): Mask with 0xffffffff together with ~(PTHREAD_MUTEX_PRIO_CEILING_MASK). (MutexAttributesPrinter.read_values): Mask with 0xffffffff together with ~PTHREAD_MUTEXATTR_FLAG_BITS and ~PTHREAD_MUTEX_NO_ELISION_NP. * manual/README.pretty-printers: Update reference to gen-py-const.awk.
2018-12-10 22:56:59 +00:00
elif args.python:
consts = glibcextract.compute_c_consts(sym_data, args.cc)
print('# GENERATED FILE\n'
'\n'
'# Constant definitions.\n'
'# See gen-as-const.py for details.\n')
print(''.join('%s = %s\n' % c for c in sorted(consts.items())), end='')
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
else:
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
consts = glibcextract.compute_c_consts(sym_data, args.cc)
print(''.join('#define %s %s\n' % c for c in sorted(consts.items())), end='')
Replace gen-as-const.awk by gen-as-const.py. This patch replaces gen-as-const.awk, and some fragments of the Makefile code that used it, by a Python script. The point is not such much that awk is problematic for this particular script, as that I'd like to build up a general Python infrastructure for extracting information from C headers, for use in writing tests of such headers. Thus, although this patch does not set up such infrastructure, the compute_c_consts function in gen-as-const.py might be moved to a separate Python module in a subsequent patch as a starting point for such infrastructure. The general idea of the code is the same as in the awk version, but no attempt is made to make the output files textually identical. When generating a header, a dict of constant names and values is generated internally then defines are printed in sorted order (rather than the order in the .sym file, which would have been used before). When generating a test that the values computed match those from a normal header inclusion, the test code is made into a compilation test using _Static_assert, where previously the comparisons were done only when the test was executed. One fragment of test generation (converting the previously generated header to use asconst_* prefixes on its macro names) is still in awk code in the makefiles; only the .sym processing and subsequent execution of the compiler to extract constants have moved to the Python script. Tested for x86_64, and with build-many-glibcs.py. * scripts/gen-as-const.py: New file. * scripts/gen-as-const.awk: Remove. * Makerules ($(common-objpfx)%.h $(common-objpfx)%.h.d): Use gen-as-const.py. ($(objpfx)test-as-const-%.c): Likewise.
2018-11-30 15:20:41 +00:00
if __name__ == '__main__':
main()