In <https://sourceware.org/ml/libc-alpha/2018-11/msg00225.html>,
Florian reported that the change from conformtest.pl to conformtest.py
had increased conform/ test time, possibly because of increased
startup overhead for Python scripts.
This patch improves conformtest.py performance by arranging for as
many tests of a (header, standard) pair as possible to use a single
execution of the compiler, so it does not need to initialize and parse
the whole header under test separately for every test assertion.
Specifically, compilation tests that are not marked as "optional" or
"xfail" are combined into a single source file, and are only then run
separately if compilation of that combined file fails. For me, this
reduces the wall clock time for the conformtest.py tests (not the
whole of the conform/ directory) from two minutes to 15 seconds.
Tested for x86_64, and with build-many-glibcs.py.
* conform/conformtest.py (CompileSubTest.__init__): Set
self.run_early to False.
(ExecuteSubTest.__init__): Likewise.
(HeaderTests.run): Try running all non-optional, non-XFAILed
compilation tests in a single execution of the compiler.
This patch continues moving conformtest towards running more tests in
a single compiler execution by separating the generation and execution
of the subtests of each test.
Instead of test classes having a run method that both generates the
text of the programs to be compiled or executed, and compiles or
executes them, they are changed to having a gen_subtests method that
just generates CompileSubTest and ExecuteSubTest objects to store the
subtest names and text, and then a separate loop in HeaderTests.run
deals with actually executing those subtests.
This will allow for future changes to extract the text for all
non-optional, non-xfail compilation subtests to try compiling those
all at once, with separate compilations only if that fails, so
massively reducing the number of separate compiler executions (each of
which needs to parse the entire contents of the header under test, in
addition to the startup cost that applies even for compiling an empty
file).
Tested for x86_64, and with build-many-glibcs.py.
* conform/conformtest.py (CompileSubTest): New class.
(ExecuteSubTest): Likewise.
(ElementTest.run): Rename to gen_subtests. Append tests to
self.subtests instead of running them.
(ConstantTest.run): Likewise.
(SymbolTest.run): Likewise.
(TypeTest.run): Likewise.
(TagTest.run): Likewise.
(FunctionTest.run): Likewise.
(VariableTest.run): Likewise.
(MacroFunctionTest.run): Likewise.
(MacroStrTest.run): Likewise.
(HeaderTests.handle_test_line): Generate subtests for tests.
(HeaderTests.run): Run subtests for tests.
This patch makes tests in conformtest use unique identifiers, in
preparation for trying to cover more tests in a single compilation to
speed up these tests as suggested in
<https://sourceware.org/ml/libc-alpha/2018-11/msg00229.html>.
Tests are assigned a number, used in generating identifiers; where a
single call to a run method does multiple compilations (sharing that
number), identifiers are changed as needed to avoid duplication
between those compilations, so they can be combined in future.
Large numbers of positional arguments to format strings make the code
harder to follow, and using the test numbers serves to increase the
number of arguments to such format strings, so the code is generally
changed to use %(name)s where all the arguments come from attributes
of the test object and so vars(self) is sufficient to provide all
those names for the format string. Cases where some arguments aren't
attributes of self still use positional format arguments.
Tested for x86_64, and with build-many-glibcs.py.
* conform/conformtest.py (ElementTest.run): Use unique identifiers
in tests. Use names for format arguments.
(ConstantTest.run): Likewise.
(SymbolTest.run): Likewise.
(TypeTest.run): Likewise.
(TagTest.run): Likewise.
(FunctionTest.run): Likewise.
(VariableTest.run): Likewise.
(MacroFunctionTest.run): Likewise.
(MacroStrTest.run): Likewise.
(HeaderTests.__init__): Set self.num_tests.
(HeaderTests.handle_test_line): Set test.num. Increment
self.num_tests.
Continuing the consolidation on Python for various miscellaneous build
and test scripts, this patch moves conformtest from Perl to Python.
The substance of the tests run is intended to be the same as before,
except that the previous test for tags did not actually achieve the
intended purpose of verifying whether a tag was already declared, so
is changed to one that would actually fail for a tag that wasn't
declared, and a typo in the old test for variables being available
($xyzzy instead of xyzzy) would have made that test not use the
correct type (but it would have passed anyway with warnings). No
attempt is made to keep the details of what the test output looks
like; instead, tests are given names which are made to follow PASS: /
FAIL: / XFAIL: / SKIP: / MISSING: as appropriate.
In the new version, there is more consistent parsing of test lines
(into a series of words, either surrounded by {} or separated by
spaces) that applies for all kinds of test lines, rather than the old
approach of different regular expressions for every kind of test. A
few of the conform/data/ files are adjusted so their syntax works with
the new script (which now requires spaces in certain cases where the
old script tolerated them being missing, and does not allow stray
semicolons at the end of "function" lines). Similarly, common logic
is used around what happens with a second subtest if a first one fails
(e.g., a test for a symbol's type if the test for availability fails),
rather than such logic being replicated separately for each kind of
test. Common parsing also applies for test lines both when they are
lines for the header under test and when they are lines for another
header specified with allow-header, again unlike the old script.
Tested for x86_64, and with build-many-glibcs.py.
* conform/conformtest.py: New file.
* conform/conformtest.pl: Remove.
* conform/GlibcConform.pm: Likewise.
* conform/glibcconform.py (KEYWORDS_C90): New constant.
(KEYWORDS_C99): Likewise.
(KEYWORDS): Likewise.
* conform/Makefile ($(conformtest-header-tests)): Use
conformtest.py instead of conformtest.pl. Do not pass --tmpdir
option. Use --header instead of --headers.
* conform/data/arpa/inet.h-data: Remove trailing semicolons on
function entries.
* conform/data/spawn.h-data: Likewise.
* conform/data/fcntl.h-data (openat): Add space after function
name.
* conform/data/wchar.h-data (wcscasecmp): Likewise.
(wcscasecmp_l): Likewise.
* conform/data/termios.h-data (c_cc): Add space after element
name.
