glibc/scripts/glibcextract.py

195 lines
8.1 KiB
Python
Raw Normal View History

Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
#!/usr/bin/python3
# Extract information from C headers.
# Copyright (C) 2018-2023 Free Software Foundation, Inc.
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
# This file is part of the GNU C Library.
#
# The GNU C Library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# The GNU C Library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with the GNU C Library; if not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
# <https://www.gnu.org/licenses/>.
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
import collections
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
import os.path
import re
import subprocess
import tempfile
def compute_c_consts(sym_data, cc):
"""Compute the values of some C constants.
The first argument is a list whose elements are either strings
(preprocessor directives, or the special string 'START' to
indicate this function should insert its initial boilerplate text
in the output there) or pairs of strings (a name and a C
expression for the corresponding value). Preprocessor directives
in the middle of the list may be used to select which constants
end up being evaluated using which expressions.
"""
out_lines = []
for arg in sym_data:
if isinstance(arg, str):
if arg == 'START':
out_lines.append('void\ndummy (void)\n{')
else:
out_lines.append(arg)
continue
name = arg[0]
value = arg[1]
out_lines.append('asm ("/* @@@name@@@%s@@@value@@@%%0@@@end@@@ */" '
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
': : \"i\" ((long int) (%s)));'
% (name, value))
out_lines.append('}')
out_lines.append('')
out_text = '\n'.join(out_lines)
with tempfile.TemporaryDirectory() as temp_dir:
c_file_name = os.path.join(temp_dir, 'test.c')
s_file_name = os.path.join(temp_dir, 'test.s')
with open(c_file_name, 'w') as c_file:
c_file.write(out_text)
# Compilation has to be from stdin to avoid the temporary file
# name being written into the generated dependencies.
cmd = ('%s -S -o %s -x c - < %s' % (cc, s_file_name, c_file_name))
subprocess.check_call(cmd, shell=True)
consts = {}
with open(s_file_name, 'r') as s_file:
for line in s_file:
match = re.search('@@@name@@@([^@]*)'
'@@@value@@@[^0-9Xxa-fA-F-]*'
'([0-9Xxa-fA-F-]+).*@@@end@@@', line)
if match:
if (match.group(1) in consts
and match.group(2) != consts[match.group(1)]):
raise ValueError('duplicate constant %s'
% match.group(1))
consts[match.group(1)] = match.group(2)
return consts
def list_macros(source_text, cc):
"""List the preprocessor macros defined by the given source code.
The return value is a pair of dicts, the first one mapping macro
names to their expansions and the second one mapping macro names
to lists of their arguments, or to None for object-like macros.
"""
with tempfile.TemporaryDirectory() as temp_dir:
c_file_name = os.path.join(temp_dir, 'test.c')
i_file_name = os.path.join(temp_dir, 'test.i')
with open(c_file_name, 'w') as c_file:
c_file.write(source_text)
cmd = ('%s -E -dM -o %s %s' % (cc, i_file_name, c_file_name))
subprocess.check_call(cmd, shell=True)
macros_exp = {}
macros_args = {}
with open(i_file_name, 'r') as i_file:
for line in i_file:
match = re.fullmatch('#define ([0-9A-Za-z_]+)(.*)\n', line)
if not match:
raise ValueError('bad -dM output line: %s' % line)
name = match.group(1)
value = match.group(2)
if value.startswith(' '):
value = value[1:]
args = None
elif value.startswith('('):
match = re.fullmatch(r'\((.*?)\) (.*)', value)
if not match:
raise ValueError('bad -dM output line: %s' % line)
args = match.group(1).split(',')
value = match.group(2)
else:
raise ValueError('bad -dM output line: %s' % line)
if name in macros_exp:
raise ValueError('duplicate macro: %s' % line)
macros_exp[name] = value
macros_args[name] = args
return macros_exp, macros_args
def compute_macro_consts(source_text, cc, macro_re, exclude_re=None):
"""Compute the integer constant values of macros defined by source_text.
Macros must match the regular expression macro_re, and if
exclude_re is defined they must not match exclude_re. Values are
computed with compute_c_consts.
"""
macros_exp, macros_args = list_macros(source_text, cc)
macros_set = {m for m in macros_exp
if (macros_args[m] is None
and re.fullmatch(macro_re, m)
and (exclude_re is None
or not re.fullmatch(exclude_re, m)))}
sym_data = [source_text, 'START']
sym_data.extend(sorted((m, m) for m in macros_set))
return compute_c_consts(sym_data, cc)
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
def compare_macro_consts(source_1, source_2, cc, macro_re, exclude_re=None,
allow_extra_1=False, allow_extra_2=False):
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
"""Compare the values of macros defined by two different sources.
The sources would typically be includes of a glibc header and a
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
kernel header. If allow_extra_1, the first source may define
extra macros (typically if the kernel headers are older than the
version glibc has taken definitions from); if allow_extra_2, the
second source may define extra macros (typically if the kernel
headers are newer than the version glibc has taken definitions
from). Return 1 if there were any differences other than those
allowed, 0 if the macro values were the same apart from any
allowed differences.
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
"""
macros_1 = compute_macro_consts(source_1, cc, macro_re, exclude_re)
macros_2 = compute_macro_consts(source_2, cc, macro_re, exclude_re)
if macros_1 == macros_2:
return 0
print('First source:\n%s\n' % source_1)
print('Second source:\n%s\n' % source_2)
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
ret = 0
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
for name, value in sorted(macros_1.items()):
if name not in macros_2:
print('Only in first source: %s' % name)
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
if not allow_extra_1:
ret = 1
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
elif macros_1[name] != macros_2[name]:
print('Different values for %s: %s != %s'
% (name, macros_1[name], macros_2[name]))
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
ret = 1
Move tst-signal-numbers to Python. This patch converts the tst-signal-numbers test from shell + awk to Python. As with gen-as-const, the point is not so much that shell and awk are problematic for this code, as that it's useful to build up general infrastructure in Python for use of a range of code involving extracting values from C headers. This patch moves some code from gen-as-const.py to a new glibcextract.py, which also gains functions relating to listing macros, and comparing the values of a set of macros from compiling two different pieces of code. It's not just signal numbers that should have such tests; pretty much any case where glibc copies constants from Linux kernel headers should have such tests that the values and sets of constants agree except where differences are known to be OK. Much the same also applies to structure layouts (although testing those without hardcoding lists of fields to test will be more complicated). Given this patch, another test for a set of macros would essentially be just a call to glibcextract.compare_macro_consts (plus boilerplate code - and we could move to having separate text files defining such tests, like the .sym inputs to gen-as-const, so that only a single Python script is needed for most such tests). Some such tests would of course need new features, e.g. where the set of macros changes in new kernel versions (so you need to allow new macro names on the kernel side if the kernel headers are newer than the version known to glibc, and extra macros on the glibc side if the kernel headers are older). tst-syscall-list.sh could become a Python script that uses common code to generate lists of macros but does other things with its own custom logic. There are a few differences from the existing shell + awk test. Because the new test evaluates constants using the compiler, no special handling is needed any more for one signal name being defined to another. Because asm/signal.h now needs to pass through the compiler, not just the preprocessor, stddef.h is included as well (given the asm/signal.h issue that it requires an externally provided definition of size_t). The previous code defined __ASSEMBLER__ with asm/signal.h; this is removed (__ASSEMBLY__, a different macro, eliminates the requirement for stddef.h on some but not all architectures). Tested for x86_64, and with build-many-glibcs.py. * scripts/glibcextract.py: New file. * scripts/gen-as-const.py: Do not import os.path, re, subprocess or tempfile. Import glibcexctract. (compute_c_consts): Remove. Moved to glibcextract.py. (gen_test): Update reference to compute_c_consts. (main): Likewise. * sysdeps/unix/sysv/linux/tst-signal-numbers.py: New file. * sysdeps/unix/sysv/linux/tst-signal-numbers.sh: Remove. * sysdeps/unix/sysv/linux/Makefile ($(objpfx)tst-signal-numbers.out): Use tst-signal-numbers.py. Redirect stderr as well as stdout.
2018-12-10 22:27:13 +00:00
for name in sorted(macros_2.keys()):
if name not in macros_1:
print('Only in second source: %s' % name)
Add test that MAP_* constants agree with kernel. Continuing the process of building up and using Python infrastructure for extracting and using values in headers, this patch adds a test that MAP_* constants from sys/mman.h agree with those in the Linux kernel headers. (Other sys/mman.h constants could be added to the test separately.) This set of constants has grown over time, so the generic code is enhanced to allow saying extra constants are OK on either side of the comparison (where the caller sets those parameters based on the Linux kernel headers version, compared with the version the headers were last updated from). Although the test is a custom Python file, my intention is to move in future to a single Python script for such tests and text files it takes as inputs, once there are enough examples to provide a guide to the common cases in such tests (I'd like to end up with most or all such sets of constants copied from kernel headers having such tests, and likewise for structure layouts from the kernel). The Makefile code is essentially the same as for tst-signal-numbers, but I didn't try to find an object file to depend on to represent the dependency on the headers used by the test (the conform/ tests don't try to represent such header dependencies at all, for example). Tested with build-many-glibcs.py, and also for x86_64 with older kernel headers. * scripts/glibcextract.py (compare_macro_consts): Take parameters to allow extra macros from first or second sources. * sysdeps/unix/sysv/linux/tst-mman-consts.py: New file. * sysdeps/unix/sysv/linux/Makefile [$(subdir) = misc] (tests-special): Add $(objpfx)tst-mman-consts.out. ($(objpfx)tst-mman-consts.out): New makefile target.
2018-12-17 18:29:36 +00:00
if not allow_extra_2:
ret = 1
return ret
CompileResult = collections.namedtuple("CompileResult", "returncode output")
def compile_c_snippet(snippet, cc, extra_cc_args=''):
"""Compile and return whether the SNIPPET can be build with CC along
EXTRA_CC_ARGS compiler flags. Return a CompileResult with RETURNCODE
being 0 for success, or the failure value and the compiler output.
"""
with tempfile.TemporaryDirectory() as temp_dir:
c_file_name = os.path.join(temp_dir, 'test.c')
obj_file_name = os.path.join(temp_dir, 'test.o')
with open(c_file_name, 'w') as c_file:
c_file.write(snippet + '\n')
cmd = cc.split() + extra_cc_args.split() + ['-c', '-o', obj_file_name,
c_file_name]
r = subprocess.run(cmd, check=False, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return CompileResult(r.returncode, r.stdout)