Aside from the generation of the xml and junit xml files
for each test, also save the test output into a
tst_{name}-{timestamp}.txt file.
This will allow developers to inspect the output of passed tests,
by downloading the test results archive, even if we use ctest's
--output-on-failure.
Change-Id: I0a6c0ee04b4525d3ad9b207b28117d2182d29c28
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The JUnit format is easier to feed to tools that don't support
the native Qt Test XML format.
Change-Id: Ie9803cc0fb0577b3b7258b05faa78d8fb1aad1d1
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
Under coin-agent, the stdout/err are not a tty, so flushing needs to be
forced.
Change-Id: I06de43328a4f4d1c17df7188f31b5f7bc63e3335
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The test run is wrapped with a special TESTRUNNER script that ignores
failing tests (there are several tests failing when built with ASAN) and
also ignores LSAN errors (memory leaks - but still visible in the
output).
The test run only fails if a test reports ASAN errors or if it
crashes (or times out, which is like a crash caused by qtestlib's
watchdog timer).
Fixes: QTQAINFRA-5025
Change-Id: I861756ab49388ac4a52409d3a780684244e469b1
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
QDoc recently added some tests that use Catch2 instead of testlib.
Similarly to selftests, the normal behavior of `qt-testrunner.py` will
fail, trying to pass an argument that is not supported by Catch2 when
running the tests in CI.
To avoid the issue, an exception for the test executables that QDoc uses
was added to `qt-testrunner.py`, using the already available mechanism
that is used for selftests.
Change-Id: I5eb4b75f239eda0c493fb06ece164545d159372b
Reviewed-by: Topi Reiniö <topi.reinio@qt.io>
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
It seems there are some tests in the various Qt modules that skip
writing the XML testlog file. If there are good reasons for that (for
example test is a custom shellscript) then it should be added to the
DUMB_TESTS list.
Change-Id: Iffb00e1fda42bb8e7338a94c9320e45900db1bc1
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
Do not print full stack traces when a controlled exception occurs,
instead print just the exception name and the message.
Change-Id: I5669862d2e98c550d88d1d11fb2bf49f197cfc17
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
If a test returns 0 but writes an XML logfile that contains FAIL or a
corrupted XML file, then qt-testrunner considers it a CRASH and exits
with 3.
Previously any test execution returning 0 (success) was considered
a PASS. Changing this behavior with this patch finds a lot of test
crashes on Android (QTBUG-100470), because the tests are run indirectly
on the emulator and the test wrapper process could not detect the crash,
thus returning 0 to qt-testrunner. But the corrupt XML file is caught
now.
Likewise, if a test returns != 0 but the XML logfile contains only PASS,
qt-testrunner considers it a FAIL. This used to be the case but now
tests are added.
Finally changed logging for such cases from INFO to WARNING.
Task-number: QTBUG-100470
Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest
and run the tests. Simplify it a lot and fix the way it passes arguments
so that they retain spaces and quotation.
Also wrap it with the generic TESTRUNNER script for CI, which is
qt-testrunner.py, so that tests can re-run in case of flakiness.
Delete prefix.sh as the environment variables are now properly set in
the Coin platform config files.
Avoid executing extra chmod commands to make scripts executable,
but use the right Coin instruction for that. For reference, we
use 493 for file mode which equals to 755 in octal, as Coin expects
the file mode in decimal.
Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
Previously the qt-testrunner script was avoiding to write XML log files
for the flaky test re-runs, since parsing them always provides limited
information about what went wrong (for example unparseable files) and
might miss issues completely (for example a crashed test might not even
write a log file), or even mislead if multiple different data points
about a single testcase end up into the database..
But currently Coin is the system that uploads test information to the
testresults database and expects to find such log files in an attempt to
send as much information as possible regarding failures.
Re-enabling these log files will allow Coin to deduce as much
information as possible about the test runs.
This is a temporary step at least, until the test-uploading logic is
separated from Coin. Then it will be easier to find the full information
about what happened with flaky or crashed tests by special logs from
qt-testrunner.py, which is the test re-running orchestrator and
therefore has more knowledge about what went wrong.
Fixes: QTQAINFRA-4754
Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Toni Saario <toni.saario@qt.io>
Reviewed-by: Simo Fält <simo.falt@qt.io>
Some function names are special for qtestlib, in the sense that they can
not be specified as a command line argument to run individually.
In such cases qt-testrunner treats the failure specially and tries once
to re-run the full test executable.
Fixes: QTBUG-89011
Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
It causes crashes because of interleaved execution of Quick tests.
Task-number: QTBUG-98350
Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Script that wraps Qt test execution in order to iron-out flakiness.
Task-number: QTBUG-96353
Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
