Commit Graph

11 Commits

Author SHA1 Message Date
Lucie Gérard
05fc3aef53 Use SPDX license identifiers
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.

Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-05-16 16:37:38 +02:00
Dimitrios Apostolou
72793ef31d Generalize exclusion list
It seems there are some tests in the various Qt modules that skip
writing the XML testlog file. If there are good reasons for that (for
example test is a custom shellscript) then it should be added to the
DUMB_TESTS list.

Change-Id: Iffb00e1fda42bb8e7338a94c9320e45900db1bc1
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-04-22 15:39:03 +02:00
Dimitrios Apostolou
399dbba775 qt-testrunner: improve error logging
Do not print full stack traces when a controlled exception occurs,
instead print just the exception name and the message.

Change-Id: I5669862d2e98c550d88d1d11fb2bf49f197cfc17
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-04-22 15:38:59 +02:00
Dimitrios Apostolou
0b4446e16f qt-testrunner: be more strict if bad XML log files are written
If a test returns 0 but writes an XML logfile that contains FAIL or a
corrupted XML file, then qt-testrunner considers it a CRASH and exits
with 3.

Previously any test execution returning 0 (success) was considered
a PASS. Changing this behavior with this patch finds a lot of test
crashes on Android (QTBUG-100470), because the tests are run indirectly
on the emulator and the test wrapper process could not detect the crash,
thus returning 0 to qt-testrunner. But the corrupt XML file is caught
now.

Likewise, if a test returns != 0 but the XML logfile contains only PASS,
qt-testrunner considers it a FAIL. This used to be the case but now
tests are added.

Finally changed logging for such cases from INFO to WARNING.

Task-number: QTBUG-100470
Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
2022-04-05 12:06:23 +03:00
Dimitrios Apostolou
c37d0e84bc Clean up QNX platform configs and improve testrunner scripts
QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest
and run the tests. Simplify it a lot and fix the way it passes arguments
so that they retain spaces and quotation.

Also wrap it with the generic TESTRUNNER script for CI, which is
qt-testrunner.py, so that tests can re-run in case of flakiness.

Delete prefix.sh as the environment variables are now properly set in
the Coin platform config files.

Avoid executing extra chmod commands to make scripts executable,
but use the right Coin instruction for that. For reference, we
use 493 for file mode which equals to 755 in octal, as Coin expects
the file mode in decimal.

Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
2022-02-14 12:56:06 +01:00
Dimitrios Apostolou
b013f94086 Write XML test log files even for the flaky test re-runs
Previously the qt-testrunner script was avoiding to write XML log files
for the flaky test re-runs, since parsing them always provides limited
information about what went wrong (for example unparseable files) and
might miss issues completely (for example a crashed test might not even
write a log file), or even mislead if multiple different data points
about a single testcase end up into the database..

But currently Coin is the system that uploads test information to the
testresults database and expects to find such log files in an attempt to
send as much information as possible regarding failures.

Re-enabling these log files will allow Coin to deduce as much
information as possible about the test runs.

This is a temporary step at least, until the test-uploading logic is
separated from Coin. Then it will be easier to find the full information
about what happened with flaky or crashed tests by special logs from
qt-testrunner.py, which is the test re-running orchestrator and
therefore has more knowledge about what went wrong.

Fixes: QTQAINFRA-4754
Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Toni Saario <toni.saario@qt.io>
Reviewed-by: Simo Fält <simo.falt@qt.io>
2022-01-30 01:00:35 +01:00
Dimitrios Apostolou
b4d9d5c89c qt-testrunner: do not try to re-run initTestCase and others
Some function names are special for qtestlib, in the sense that they can
not be specified as a command line argument to run individually.

In such cases qt-testrunner treats the failure specially and tries once
to re-run the full test executable.

Fixes: QTBUG-89011
Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-30 01:00:35 +01:00
Dimitrios Apostolou
5180e70dbe qt-testrunner: be more clear that it was the test that crashed
Task-number: QTBUG-99970
Change-Id: Id2d01b6ab7d428356d9dfc953107014791393d35
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
2022-01-18 19:43:54 +01:00
Dimitrios Apostolou
3acfd39653 qt-testrunner: disable JUnit XML test logs
It causes crashes because of interleaved execution of Quick tests.

Task-number: QTBUG-98350
Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-12 16:14:18 +01:00
Dimitrios Apostolou
24572fccae Enable verbose logging in the last re-run of a failing test
Fixes: COIN-728
Change-Id: I08802e377e26e5dd7f7d1c44b5efe4280b09f957
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-03 17:40:35 +01:00
Dimitrios Apostolou
63ffae3fa3 Introduce qt-testrunner.py
Script that wraps Qt test execution in order to iron-out flakiness.

Task-number: QTBUG-96353
Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-03 16:32:56 +01:00