Repeated test execution can be useful, under a debugger, to catch an
intermittent failure or, under memory instrumentation, to make memory
leaks easier to recognize.
The new -repeat flag allows running the entire test suite multiple times
within the same process. It works by executing all tests sequentially
before repeating the execution again.
This switch is a developer tool, and is not intended for CI. It can only
be used with the plain text logger.
Change-Id: I2439462c5c44d1c8aa3d3b5656de3eef44898c68
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Ensure that this environment variable is respected, as ASAN errors
(e.g. memory leaks) can cause the selftests to fail since they expect
no errors in their output.
Fix an incorrect function name in a comment, while we're at it.
Fixes: QTBUG-118041
Pick-to: 6.5 6.6
Change-Id: I07502101aabd8743df898ae8fe4a693c4733c4af
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This allows us to control whether to run this particular test in ASan
mode or with specific loggers or not. Adding the expected log output for
tst_silent for other loggers is left as an exercise to the reader.
Change-Id: Ifa1111900d6945ea8e05fffd177f1548c8c8e714
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This functionality was lost when we switched to Catch2.
Change-Id: I2b24e1d3cad44897906efffd177fb4cff641546f
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Amends 118f2210c6. That commit added
#ifdef Q_OS_UNIX
if (test == "assert"
|| test == "crashes"
|| test == "failfetchtype"
|| test == "faildatatype")
return; // Outputs "Received signal 6 (SIGABRT)"
#endif
Which duplicated 4 out of the 5 tests in the block:
#ifdef Q_OS_LINUX
// QEMU outputs to stderr about uncaught signals
if (QTestPrivate::isRunningArmOnX86() &&
(test == "assert"
|| test == "crashes"
|| test == "faildatatype"
|| test == "failfetchtype"
|| test == "silent"
))
return;
#endif
But as Linux is Unix, we never got to that second block for those 4
tests.
Pick-to: 6.6
Change-Id: I2b24e1d3cad44897906efffd177fb4b5507d190a
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The loop was iterating over a temporary container, so it couldn't have
changed it. Use a const auto variable to hold the container and port to
ranged-for
Task-number: QTBUG-115839
Change-Id: I66e7cbdb811666ca352cdf064b1228caa346d876
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
The density of Q_FOREACH uses in this and some other modules is still
extremely high, too high for anyone to tackle in a short amount of
time. Even if they're not concentrated in just a few TUs, we need to
make progress on a global QT_NO_FOREACH default, so grab the nettle
and stick to our strategy:
Mark the whole of Qt with QT_NO_FOREACH, to prevent new uses from
creeping in, and whitelist the affected TUs by #undef'ing
QT_NO_FOREACH locally, at the top of each file. For TUs that are part
of a larger executable, this requires these files to be compiled
separately, so add them to NO_PCH_SOURCES (which implies
NO_UNITY_BUILD_SOURCES, too).
In tst_qglobal.cpp and tst_qcollections.cpp change the comment on the
#undef QT_NO_FOREACH to indicate that these actually test the macro.
Task-number: QTBUG-115839
Change-Id: Iecc444eb7d43d7e4d037f6e155abe0e14a00a5d6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The blacklisted test does not crash, as its expected output indeed shows.
Pick-to: 6.5 6.6
Change-Id: I07522a7d065b5f39620975a3546bcd156024c41d
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
In porting the selftest machinery to Catch2 in 24e83de8d1 we
accidentally added an unconditional early return when determining
whether to check for unexpected stderr output, resulting in not
checking error output on any platform.
The return statement has now been moved into the Q_CC_MINGW
condition, but as we now seem to have similar issues on macOS
and Linux with some of the tests outputting "Received signal 6
(SIGABRT)" as we do for QEMU, we need to add a few more explicit
early return conditions to the function.
Pick-to: 6.5 6.6
Change-Id: I7a25f000843b5f1003a5db883f08285185046b46
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This is a semantic patch using ClangTidyTransformator as in
qtbase/df9d882d41b741fef7c5beeddb0abe9d904443d8, but extended to
handle typedefs and accesses through pointers, too:
const std::string o = "object";
auto hasTypeIgnoringPointer = [](auto type) { return anyOf(hasType(type), hasType(pointsTo(type))); };
auto derivedFromAnyOfClasses = [&](ArrayRef<StringRef> classes) {
auto exprOfDeclaredType = [&](auto decl) {
return expr(hasTypeIgnoringPointer(hasUnqualifiedDesugaredType(recordType(hasDeclaration(decl))))).bind(o);
};
return exprOfDeclaredType(cxxRecordDecl(isSameOrDerivedFrom(hasAnyName(classes))));
};
auto renameMethod = [&] (ArrayRef<StringRef> classes,
StringRef from, StringRef to) {
return makeRule(cxxMemberCallExpr(on(derivedFromAnyOfClasses(classes)),
callee(cxxMethodDecl(hasName(from), parameterCountIs(0)))),
changeTo(cat(access(o, cat(to)), "()")),
cat("use '", to, "' instead of '", from, "'"));
};
renameMethod(<classes>, "count", "size");
renameMethod(<classes>, "length", "size");
except that the on() matcher has been replaced by one that doesn't
ignoreParens().
a.k.a qt-port-to-std-compatible-api V5 with config Scope: 'Container'.
Added two NOLINTNEXTLINEs in tst_qbitarray and tst_qcontiguouscache,
to avoid porting calls that explicitly test count().
Change-Id: Icfb8808c2ff4a30187e9935a51cad26987451c22
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
QWindow::requestActivate() is not supported.
We have one tst_selftests binary, and will test it with both xcb and
wayland qpa plugin. A runtime check and skip will have different
restult files, which is not implemented in testlib yet.
Task-number: QTBUG-107578
Pick-to: 6.4 6.2
Change-Id: Idc8cb24c6f42a9f0f4dc9493e3fd1a5803ba7ce0
Reviewed-by: Liang Qi <liang.qi@qt.io>
Previously trying to execute a test function with an unknown data tag
would print an error message but exit with 0.
This patch stores a test failure, and continues trying to execute the
rest of the command line arguments, if any. In the end the process exits
with the usual exit code (number of failed tests) which is now !=0.
Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-24240
Change-Id: Id4d422035f173e01e77ca88028dfd94dc0f9085c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
We don't support it any more. I don't think it has ever properly
compiled Qt 6 (and it's no longer working for me against GCC 12's
libstdc++ headers). If you report a bug against it, Intel support's
first question is if you can try instead the new Clang/LLVM-based oneAPI
C++ compiler.
So we support only that one, which identifies itself as Q_CC_CLANG.
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb57a092c8439e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
When running a test under increased verbosity levels, QTestLib would
print "failure location" for every QTest function (QVERIFY, QCOMPARE,
...), even if there's no failure at all.
Keep the code centralized, but split the formatting of failures and
non-failures (other messages).
Task-number: QTBUG-96844
Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c
Pick-to: 5.15 6.2 6.3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
- It should not be built for systems without process feature.
Task-number: QTBUG-99123
Pick-to: 6.2 6.3
Change-Id: I71caa59c2168435894c7d1afcc8226e44178439f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.
Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The actual files present for expected data, the test function that
selected which tests to skip and the python function to select which
to not generate data for weren't in sync with one another.
The test-code's reason for omitting three tests was that we lacked
data files for them. So generate those and skip that exception. The
generator script's code to decide which to generate didn't exclude
anything like as many, so update it to match the test-code. In the
process, save repeating a startswith test that was used both
positively and negatively, unifying two conditions.
Extend the generator script's handling of its --skip-callgrind option
by auto-setting that option if valgrind isn't available, to match the
driver program's similar skipping.
The generated data included many files for tests we skip and, as
mentioned already, lacked files for some tests we only skipped because
we lacked them. Remove the unused files, add the lacking ones.
Change-Id: If91696cdd95b7b0d5f3d686bff839f1bf15e121b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When deciding whether to run benchlibcallbrind, the code only tested
_i386; it now tests __x86_64 as well, to match a recent change in the
test itself. As there, reverse the test to reduce negations, flipping
the stanzas it selects between. Also tidy up the code that tests for
valgrind being present - and actually return true, to skip the test,
when it claims to be skipping the test.
Updated test results, now that the test can actually be run and
produce sensible output. Added an _2.txt that matches the results
presently seen in Coin on RHEL 8.4 (despite the fact that a local
build on such a VM produces output matching the _1.txt results).
Change-Id: Ibce09dca06a1eeb73e90fb1345834998683df9d8
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
It unconditionally added -callgrind to its own command-line options,
but the way testlib handles this argument is, in QTest::qRun(), to
re-run the program under the control of valgrind --tool=callgrind,
removing the -callgrind command-line option from the test and adding
-callgrindchild to its command-line options. So we shouldn't re-add
the -callgrind option in the resulting recursive call.
The test now runs quickly, producing sensible output, where previously
it took a very long time. Revised the drivers to reflect this
speed-up, but continue skipping the non-.txt formats to save the need
for variant-output files for many formats. To match that, removed the
unused non-.txt results files.
Change-Id: Iaa99c1b5964d50bccfc6076a21896791b6bbf289
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:
https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java
Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:
https://plugins.jenkins.io/junit/
Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:
https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsdhttps://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd
The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.
The following changes have been made to improve conformance:
- The 'timestamp' attribute on <testsuite> is is now in ISO
8601 local time, without any time zone specified
- The 'hostname' attribute on <testsuite> is now included
- The 'classname' attribute on <testcase> is now included
- The non-standard 'result' attribute on <testcase> has
been removed
- The non-standard 'result' attribute on <failure> has
been renamed to 'type'
- The <system-out> element on <testsuite> is always included,
even when empty
- The non-standard 'tag' attribute on <failure> has been
removed. Data-driven tests are now represented as individual
<testcase> elements, e.g.:
<testcase name="someTest(someData X)" ...>
<testcase name="someTest(someData Y)" ...>
<testcase name="someTest(someData Z)" ...>
The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.
Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
For historical reasons a few of the subtests are skipped when
running with anything but the plain text logger to stdout.
To ensure we have as broad test coverage as possible for the
expected output of the various loggers we run these tests in
stdout-mode.
Pick-to: 6.2
Change-Id: I856905d1543afe89710533657a55bd599c0305fd
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit reporter was initially named xunit, but the naming was inaccurate
and the reporter was renamed in 27db9e458c.
The corresponding test has now been renamed as well, and as an added bonus
we only run it for that reporter.
Pick-to: 6.2
Change-Id: I59cb7d949514cdf46a0199a53a7a3e39f833207c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
The emulation detection has been usable only on qtbase tests, move it to
QTest so that it can be used in other modules as well.
Pick-to: 6.1
Change-Id: I4b2321b7856414d7b1cfd5e6b1405a633c6bb878
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Complete search and replace of QtTest and QtTest/QtTest with QTest, as
QtTest includes the whole module. Replace all such instances with
correct header includes. See Jira task for more discussion.
Fixes: QTBUG-88831
Change-Id: I981cfae18a1cabcabcabee376016b086d9d01f44
Pick-to: 6.0
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Otherwise XOpenDisplay in a newly spawned process may fail with
a 'No protocol specified' message (as it recently happened in
CentOS 8.1 vm).
Fixes: QTBUG-87621
Pick-to: 5.15
Change-Id: Ib6c08c7f154fb2a126d32a4aa52b535e5daa1589
Reviewed-by: Liang Qi <liang.qi@qt.io>
The old test harness used to spit out stderr only, but
to be on the safe side we spit out both.
Change-Id: Ib8e57fd1b0e4d8542ac552a6fe58c07016df7f5f
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
As defined by https://llg.cubic.org/docs/junit/
Change-Id: Ic7683f3d49c529674f8467d591528d4a65d3add8
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
The attributes are, like the elements, maintained in reverse
order in the underlying QTestCoreList, so we need to iterate
them backwards when printing out the resulting XML to reflect
the order they were added.
This results in e.g.:
<testcase name="passingBenchmark" result="pass">
Instead of:
<testcase result="pass" name="passingBenchmark">
Change-Id: Ic2eeab8de05ffedd0c41977358d5b40ff77878b1
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
We don't want to print the QString as represented by the debug
operator, but instead want to expand line feeds and other character
codes as normal.
Change-Id: I7261d8f94e7b4382733f06eb22f9a740a5c0488f
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
[ChangeLog][QtCore][QByteArray] Remove method overloads taking
QString as argument, all of which were equivalent to passing the
toUtf8() of the string instead.
Change-Id: I9251733a9b3711153b2faddbbc907672a7cba190
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
We use the Catch2 testing framework to test Qt Testlib, which also opens
up the possibility of using it for other internal testing once it's made
available through the build system.
The test now has a --rebase mode which will write out the actual results
as new expected files. Once we add the required post-processing to the
results to remove timestamps and other testrun-specific data we can
remove the standalone python script generate_expected_output.py that
today has to be kept in sync with the test itself.
No attempt has been made to clean up the comparison-functions, but
these could all benefit from moving their logic from the comparison
to the sanitization step. This will both make the expected files
more generic, and will reduce the diff once a failure occurs, since
we're not seeing all the hunks that the comparison-functions ignored.
Change-Id: I1769d42e7958d56d1ad5da958db0e8fe3a2a3c23
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
As discussed in the mailing list and in the Qt Contributor Summit 2019.
Tested on Linux, macOS and FreeBSD, showing these fallbacks:
OS environment setlocale() call
FreeBSD empty "C.UTF-8"
FreeBSD LC_ALL=C "C.UTF-8"
Linux empty "C.UTF-8"
Linux LC_ALL=C "C.UTF-8"
Linux LANG=en_US "en_US.UTF-8"
Linux LANG=de_DE@euro "de_DE.UTF-8"
Linux LANG=en_GB.iso885915 "en_GB.UTF-8"
Linux LANG=hy_AM.armscii8 "hy_AM.UTF-8"
Linux LANG=ja_JP.sjis "ja_JP.UTF-8"
Linux LANG=ru_RU.koi8r "ru_RU.UTF-8"
macOS empty "UTF-8"
macOS LC_ALL=C "UTF-8"
Versions tested: FreeBSD 12.1, Linux w/ glibc 2.30, macOS 10.14.2.
See
* https://wiki.qt.io/Qt_Contributor_Summit_2019_-_QtCore
* https://lists.qt-project.org/pipermail/development/2019-October/037791.html
Change-Id: Ia2aa807ffa8a4c798425fffd15d97ddb4f35b0ae
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
We hope this shall avoid some flaky failures noticed in quick tests,
e.g. tst_QQuickMenu::Material::subMenuPosition(cascading,flip) was
recently seen failing with 3.88e-11 != 0. This required some revision
to test data in the testlib selftest for floats; the resulting
expected output differs in details but not in which tests pass or
fail. QEMU, naturally, made life difficult, requiring special-case
code in the test-driver.
[ChangeLog][QtTestLib][QCOMPARE] QCOMPARE() now treats its values as
equal when qFuzzyIsNull() is true for both of them.
Change-Id: Icc6ad5164b609937eddbe39cc69120f0abf0f3b4
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
The test orchestrator shouldn't have to deal with the individual options
needed for each test.
Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Change-Id: I2fd7a39684bde44d82c4d877086f606413d68520
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The reporter was probably named 'xunit' based on the historical use of
xUnit to refer to testing frameworks derived from Smalltalk's SUnit.
These frameworks typically added their own prefix, e.g. JUnit for Java,
RUnit for R, etc.
The most popular of these was the JUnit framework, and the corresponding
XML output produced by the Ant built tool became somewhat of a de facto
standard, which is probably why we chose to model our reporter after it.
Nowadays however, naming it 'xunit' is problematic as there is actually
a testing famework named xUnit.net, typically shortened to, you guessed
it: xunit.
Test report consumers will typically have a junit mode, and an xunit
mode, and the latter could easily be mistaken for what testlib outputs,
unless we clarify this.
The clarification also allows us to safely extend our support for the
JUnit XML format to incorporate some elements that are nowadays common,
but where we are lagging behind the standard.
[ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has
been renamed to what it actually is: a JUnit test reporter, and is now
triggered by passing -o junitxml to the test binary.
Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The test is not failing anymore on QEMU targets.
This partially reverts commit
71bd06d516.
Fixes: QTBUG-71915
Change-Id: I68593edf0ec245e14879833c8aa90661a3c2e227
Reviewed-by: Liang Qi <liang.qi@qt.io>
Tidied up the existing float tests in the process.
(In particular, s/SUCCESS/PASS/ since that matches real test output.)
These verify that QCOMPARE() handles floats and doubles as intended.
Extended the existing qFuzzyCompare tests to probe the boundaries of
the ranges of values of both types, in the process.
Revised the toString<double> that qCompare() uses to give enough
precision to actually show some of the differences being tested there
(12 digits, to match what qFuzzyCompare tests, so as to show different
values rather than, e.g. 1e12 for both expected and actual) and to
give consistent results for infinities and NaN (MinGW had eccentric
versions for these, leading to different output from tests, which thus
failed); did the latter also for toString<float> and fixed stray zeros
in MinGW's exponents (which made a kludge in tst_selftest.cpp
redundant, so I removed that, too).
That's further complicated handling of floating-point types, so let's
just keep an eye on how expensive that's getting by adding a benchmark
test for QTest::toString(). Unfortunately, default settings only get
runs that take modest numbers of milliseconds (some as low as 40)
while increasing this with -minumumvalue 100 or more gets the process
killed - and I'm unable to find out who's doing the killing (it's not
QProcess::kill, ::kill or the QtTest WatchDog, as far as I can tell).
So results are rather noisy; the integral tests exhibit speed-ups by
factors up to 5, and slow-downs by factors up to 100, between runs
with and without this change, which does not affec the integral tests.
The relatively modest slow-downs and speed-ups in the floating point
tests thus seem likely to be happenstance rather than signal.
Change-Id: I4a6bbbab6a43bf14a4089e96238a7c8da2c3127e
Reviewed-by: Ulf Hermann <ulf.hermann@qt.io>
