qt5base-lts/util/testrunner
Alexandru Croitor 146e22fff6 qt-testrunner: Save test output into file
Aside from the generation of the xml and junit xml files
for each test, also save the test output into a
tst_{name}-{timestamp}.txt file.

This will allow developers to inspect the output of passed tests,
by downloading the test results archive, even if we use ctest's
--output-on-failure.

Change-Id: I0a6c0ee04b4525d3ad9b207b28117d2182d29c28
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-12-08 17:56:48 +01:00
..
tests Use SPDX license identifiers 2022-05-16 16:37:38 +02:00
qt-testrunner.py qt-testrunner: Save test output into file 2022-12-08 17:56:48 +01:00
README qt-testrunner: Commit forgotten README file 2022-06-16 21:04:48 +02:00
sanitizer-testrunner.py sanitizer-testrunner: enforce line buffering 2022-06-16 21:04:48 +02:00

qt-testrunner.py is a script that can wrap the execution of every test in Qt
in order to iron out common issues with testing. It can be turned on by
setting the environment variable TESTRUNNER=/path/to/qt-testrunner.py before
invoking ctest.

It is commonly in use in Coin, the Continuous Integration system of Qt.

It offers the following functionality
+ Reads the XML test log and understands exactly which function of the test failed.
+ Checks the exit code and if needed repeats only the failed function of the test.
+ Appends output argument to it: "-o file.xml"
+ In case the test is executed multiple times
  - the previous output files are saved
  - the verbosity level is increased

The script itself has a testsuite that is simply run by invoking

qtbase/util/testrunner/tests/tst_testrunner.py

Please *run this manually* before submitting a change to qt-testrunner and
make sure it's passing. The reason it does not run automatically during the
usual qtbase test run, is because
+ the test run should not depend on Python
+ we don't want to wrap the testrunner tests with testrunner.