[ChangeLog][QTestLib] Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT
macros that repeatedly execute QCOMPARE_{EQ,NE,LT,LE,GT,GE} until either
the comparison returns true or the timeout expires. Also add
QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE} macros that simply invoke the
*_WITH_TIMEOUT versions with the usual timeout of five seconds.
Task-number: QTBUG-98873
Change-Id: Ib0d7d1c8c997f442b46acd85da738a8f512cc875
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
[ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
macros. These new macros behave similarly to QVERIFY(a op b),
where 'op' is ==, !=, <, <=, >, >= respectively, but print
a formatted error message with argument values in case of failure.
The formatting is done lazily, which means that the strings will
be generated only when the comparison fails.
Also add a new test for tst_selftest and generate expected output
for it.
Fixes: QTBUG-98873
Task-number: QTBUG-98874
Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Add the Q_OS_WIN32 guard for the include of QWinEventNotifier.
Change-Id: I7824b2ee236a370c83fd85a2f594a39cf36b36e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
We don't support it any more. I don't think it has ever properly
compiled Qt 6 (and it's no longer working for me against GCC 12's
libstdc++ headers). If you report a bug against it, Intel support's
first question is if you can try instead the new Clang/LLVM-based oneAPI
C++ compiler.
So we support only that one, which identifies itself as Q_CC_CLANG.
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb57a092c8439e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.
So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.
This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).
Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.
Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.
Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When producing a diagnostics block, include the file and line
information, if we have it, to describe it. This presently only adds
this information for skip, but could in principle do the same for a
B?XPass.
Task-number: QTBUG-96844
Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.
However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.
In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)
Add methods to QTestCharBuffer, and a helper function, to support this.
Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
It relied on the set(QT_REPO_MODULE_VERSION ...) line being the first
thing in the file. It no longer is, and the variable is referenced in
an earlier condition.
Change-Id: I2bd2abd2a5680782ddd61f4bf4a4a3265b444d52
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When a B?Fail's description doesn't match the QVERIFY/QCOMPARE
regexes, it got output as a comment in the YAML block after the test
line had already reported it as a TODO comment. An empty description
would also have lead to an empty comment in the YAML block. Condition
this fallback output case on there being a description that hasn't yet
been reported in the test line.
Task-number: QTBUG-96844
Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In all places but one they were treated the same; fix that last one to
match the rest. This removes one line from the YAML block for each
blacklisted XFail test; in each case, this message duplicates the one
on its "not ok ... # TODO..." test line.
Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Multiple tests use qt_internal_add_resource that copies the
functionality that is already implemented inside the
qt_internal_add_test function. Simplify these test by replacing
the qt_internal_add_resource call with the new BUILTIN_TESTDATA
option.
Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When running a test under increased verbosity levels, QTestLib would
print "failure location" for every QTest function (QVERIFY, QCOMPARE,
...), even if there's no failure at all.
Keep the code centralized, but split the formatting of failures and
non-failures (other messages).
Task-number: QTBUG-96844
Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c
Pick-to: 5.15 6.2 6.3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Update Catch2 header to latest released version. Needed
to build correctly with glibc 2.34.
Two patches applied on top of the upstream release. Previously
used Qt specific change extracted to a separate patch file.
Fixes: QTBUG-95842
Pick-to: 6.3 6.2
Change-Id: I67e442bbe756ff00c96a45eabf593f7aecddd628
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
- It should not be built for systems without process feature.
Task-number: QTBUG-99123
Pick-to: 6.2 6.3
Change-Id: I71caa59c2168435894c7d1afcc8226e44178439f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
It was missing #if-ery on feature regularexpression for one test that
depends on it. One of its comments had a long line. Added some
annotations to make clear what's going on in messier tests.
Change-Id: I06d8748a134591f93b36029713e52ffd826a24dc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.
Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Without the fix, some tests from tst_selftests were failing with:
Invalid MIT-MAGIC-COOKIE-1 key
Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
If an incident has an empty description we produced output with a
dangling space. Avoid triggering the inanity 'bot with those.
Reorganise the code to save some duplication.
Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The rationale for skipping pass after XFail is "to emit a single test
point for" the test; emitting several XFails violates that aim.
Task-number: QTBUG-96844
Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Skip ends the test (albeit inconclusively). Rearrange the enums in
the abstract logger, move code to handle skip between relevant
function and tidy up various things that became simpler as a result.
Also reorder the message enum, and its switches, to separate testlib's
internals from the usual Qt messages, and put each group in ascending
order of severity.
Task-number: QTBUG-96844
Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The actual files present for expected data, the test function that
selected which tests to skip and the python function to select which
to not generate data for weren't in sync with one another.
The test-code's reason for omitting three tests was that we lacked
data files for them. So generate those and skip that exception. The
generator script's code to decide which to generate didn't exclude
anything like as many, so update it to match the test-code. In the
process, save repeating a startswith test that was used both
positively and negatively, unifying two conditions.
Extend the generator script's handling of its --skip-callgrind option
by auto-setting that option if valgrind isn't available, to match the
driver program's similar skipping.
The generated data included many files for tests we skip and, as
mentioned already, lacked files for some tests we only skipped because
we lacked them. Remove the unused files, add the lacking ones.
Change-Id: If91696cdd95b7b0d5f3d686bff839f1bf15e121b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The various variants duplicated some rather complex code around
varying setup in the middle. Rework in terms of a macro that defines
main() and takes the setup code as a parameter. That setup code also
had some common structure, so package that in a setup macro that takes
the class to be used.
Reworked various testlib selftests that were using QTEST_MAIN_IMPL();
change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP().
These might be better dealt with by supporting a second form of the
initMain() test-setup function in the test classes, that takes
references for argc and argv, to let a test massage its command-line
options.
Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When deciding whether to run benchlibcallbrind, the code only tested
_i386; it now tests __x86_64 as well, to match a recent change in the
test itself. As there, reverse the test to reduce negations, flipping
the stanzas it selects between. Also tidy up the code that tests for
valgrind being present - and actually return true, to skip the test,
when it claims to be skipping the test.
Updated test results, now that the test can actually be run and
produce sensible output. Added an _2.txt that matches the results
presently seen in Coin on RHEL 8.4 (despite the fact that a local
build on such a VM produces output matching the _1.txt results).
Change-Id: Ibce09dca06a1eeb73e90fb1345834998683df9d8
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
It unconditionally added -callgrind to its own command-line options,
but the way testlib handles this argument is, in QTest::qRun(), to
re-run the program under the control of valgrind --tool=callgrind,
removing the -callgrind command-line option from the test and adding
-callgrindchild to its command-line options. So we shouldn't re-add
the -callgrind option in the resulting recursive call.
The test now runs quickly, producing sensible output, where previously
it took a very long time. Revised the drivers to reflect this
speed-up, but continue skipping the non-.txt formats to save the need
for variant-output files for many formats. To match that, removed the
unused non-.txt results files.
Change-Id: Iaa99c1b5964d50bccfc6076a21896791b6bbf289
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The given reason for needing to define it was that there was no way to
test whether <valgrind/valgrind.h> is available to #include; but we
now require C++17 hence __has_include(). However, moc doesn't seem to
grok that, so move the test's #if-ery inside the test, since otherwise
it gets omitted because the test's metatype doesn't know it's there.
Change-Id: I75a100787b98a52fad4cfb0b047318a115c998e2
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Its twoHundredMillionInstructions() test has #if-ery to limit it to
gcc and x86; however, it was testing only __i386 for the x86 part,
where gcc defines __x86_64 instead on modern 64-bit systems. In the
process, invert the condition and the branches it controls - positive
tests are easier to comprehend.
Change-Id: I8e906c606c48aa5034e02e3ed5d042fbb1f2ecbc
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Extract Method QTest::qCaught() to take the string handling out of the
header. This should help a bit in speeding up compilation of large
unit test files (provided they use QVERIFY_THROWS_EXCEPTION), although
I have no data to support that.
Since we changed the error message, update the selftest accordingly.
Change-Id: Id4a3c8c34d5df8d0c7a861106d269097f4a6de5c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This solves the long-standing problem of not being able to easily
fail a test when a certain warning is output.
[ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test
function, any warning that matches the given pattern will cause a test
failure. The test will continue execution when a failure is added.
All patterns are cleared at the end of each test function.
Fixes: QTBUG-70029
Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Use variable args macros to swallow any extra commas in the
expression. To use this, the type of the exception has to be first.
Use Eddy's suggestion for a new name to avoid breaking the old macro.
[ChangeLog][QtTest] Added QVERIFY_THROWS_EXCEPTION, replacing
QVERIFY_EXCEPTION_THROWN, which has therefore been deprecated.
Change-Id: I16825c35bae0631c5fad5a9a3ace4d6edc067f83
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Swallowing unknown exceptions is dangerous business, as the exception
might be a pthread cancellation token, the swallowing of which would
terminate the program.
Instead of returning from the catch-all-clause, therefore, re-throw
the unknown exception.
Fix tst_verifyexceptionthrown failure cases that use
non-std::exception-derived true negative exceptions to not let the
exception escape from the test function.
As a drive-by, pretty up the macro's docs.
[ChangeLog][QtTest][QVERIFY_EXCEPTION_THROWN] Now re-throws unknown
exceptions (= not derived from std::exception) (was: swallowed them
and returned from the test function), in order to play nice with
pthread cancellation.
Pick-to: 6.2 5.15
Change-Id: Ic036d4a9ed4b7683fa67e27af8bcbae0eefdd0da
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
We have some special handling in qt_windows.h,
use it instead of the original windows.h
Change-Id: I12fa45b09d3f2aad355573dce45861d7d28e1d77
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The --skip-benchlib option was misnamed: there are several benchlib
tests, of which it only skipped the callgrind one. As there is no
other test involving callgrind, rename to --skip-callgrind.
Change-Id: I0179fd35dd79c525f79e4a28a626e964323409bf
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
This ensures that a command such as
$ ninja tst_qlocale && ninja tst_qlocale_check
will automagically build the syslocaleapp program that the test runs
from a subtest. Similar for testlib's selftests and tst_QProcess.
As a drive-by, pruned some legacy comments from when CMakeLists.txt
files were generated from .pro files.
Change-Id: I67691a8175aaef124d4104cf1898193993408bdf
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
- Exclude Green Hills compiler from Catch2's POSIX signal handling.
GHS, at least on INTEGRITY, doesn't support full POSIX signals.
Task-number: QTBUG-96176
Pick-to: 6.2 6.2.0
Change-Id: Ifec06dca43ed766cb7335e40fc357d0d7bc463a6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Minor clean-up.
Split two long lines and added braces, called for by Qt coding style.
Turned some QDebug("this line should never be reached...") into QFAIL();
that should make the effor a bit more prominent if it ever happens.
Likewise for a qDebug() << "ERROR: ..." for code not meant to run.
Change-Id: I1d34fe202f269eb3ad8f58feaabed809e4fec99c
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
This mimics the effects of test-code delegating checks to other
functions, which can lead to repeated failures. The failing test
should only contribute one to the total of failed tsts. Drive-by:
correct an existing test's reporting of its own name.
Task-number: QTBUG-95661
Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Include counting of test types, to catch two more cases where totals
don't add up.
Task-number: QTBUG-95661
Change-Id: I9fe5424bc6652c61a065bf2889333e2ed9437c81
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Count how many we expect to pass, fail and be skipped, so we can
report that for comparison with the actual totals line: this reveals
some double-counting, marked with a FIXME.
Task-number: QTBUG-95661
Change-Id: I8b0f13ded5202ed476d8abfee70ed60f9e639bf9
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
This reverts commit 904617dfb8 and makes
clear to future readers why that wouldn't be a sensible change.
Update the test's data, eliminating a case of duplicate counting that
was caused by the reverted commit.
Task-number: QTBUG-95661
Pick-to: 5.12 5.15 6.1 6.2
Change-Id: Ice6d3ab06ca171e0d6eb0fac757f1ab774e229f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Test both Abort and Continue cases. Test more with successive marked
as expected failure. Test cases with a QSKIP after the marked check.
Unify data functions where practical.
Change-Id: I2eade5e4dd0907d23e37137ce3d93cd5ca79f802
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The separation between <system-out> and <system-err> is sufficient, and
we can't expect consumers to interpret our custom comment format.
The type of the text node has been renamed to more accurately identify
its purpose.
Pick-to: 6.2
Change-Id: I63c8ff17529fc087e1b695698350a6711eb5e68d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
It's used to handle qCritical() messages, and corresponds to QtMsgType's
QtCriticalMsg, so change its name to reflect what it really is.
In the process, reclassify the -maxwarnings overflow message as a warning,
rather than a critical message.
Pick-to: 6.2
Change-Id: I87626117a547ae4498d5dc352b93bd6db8bfb332
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The original Ant JUnit reporter only writes <system-err> and <system-out>
to the <testsuite>, but more modern reporters such as Maven Surefire
scopes output to each individual <testcase>.
This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.:
https://github.com/jenkinsci/junit-plugin/commit/145eb5c98
Pick-to: 6.2
Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The -silent option to tests is only supported with the plain text
logger, so we don't need to maintain expectation files for the
others.
Pick-to: 6.2
Change-Id: I0f42bfe90d82b7ce04f550c747d4a80e99621e74
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Test errors represents unanticipated problems, e.g., an unhandled
exception, or a problem with the implementation of the test.
Pick-to: 6.2
Change-Id: I87219e7ffdea56862278f005de44526ad97545f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We don't produce <failure> elements for them, so we shouldn't include
them in the total count of failures.
In the future we might produce <skipped> elements instead, but for now
remove the incorrect increment.
Pick-to: 6.2
Change-Id: Ia89a5cdaf79f6aa2f92be53180112c40ad6da2c6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The data itself is not indented, as consumers may read it as verbatim
data.
Pick-to: 6.2
Change-Id: Ia934616cea273feadc3a45d7c74726d4f804f0dc
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
33d7f76f0e attempted to exercise more
of the blacklisting code, but in doing so didn't account for the
fact that the features it added didn't actually work.
Adding a global blacklisting of * will result in blacklisting
all tests, regardless of any inverse conditions "!*" later on,
such as for the 'pass' test. This would have been evident by
looking at the expectation files, which still showed the test
as a BPASS.
The broken feature should be fixed, but in the meantime let's
clean up the test so it doesn't misrepresent reality.
At the same time we also remove the messages() tests, which was
copied straight out of tst_silent, but doesn't serve any purpose
for the blacklist selftest.
Pick-to: 6.2
Change-Id: I20e4693300aad5e9ff5f17ad22e3dfcb5c49fd0e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The Apache Ant and Surefire Maven specs document a <skipped> element that
can be used to signify skipped test, with a corresponding total skipped
test attribute on the <testsuite>.
The element includes an optional message attribute, documented in the
Surefire spec, and in the Ant source code, but not yet documented in
the reverse-engineered Ant spec:
https://github.com/windyroad/JUnit-Schema/pull/11
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The errors attribute on the <testsuite> element represents the number of
<error> elements, but we do not produce any at the moment.
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: I7196d622a9a6bbb7e79ed2c2886984d539abb1da
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:
https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java
Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:
https://plugins.jenkins.io/junit/
Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:
https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsdhttps://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd
The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.
The following changes have been made to improve conformance:
- The 'timestamp' attribute on <testsuite> is is now in ISO
8601 local time, without any time zone specified
- The 'hostname' attribute on <testsuite> is now included
- The 'classname' attribute on <testcase> is now included
- The non-standard 'result' attribute on <testcase> has
been removed
- The non-standard 'result' attribute on <failure> has
been renamed to 'type'
- The <system-out> element on <testsuite> is always included,
even when empty
- The non-standard 'tag' attribute on <failure> has been
removed. Data-driven tests are now represented as individual
<testcase> elements, e.g.:
<testcase name="someTest(someData X)" ...>
<testcase name="someTest(someData Y)" ...>
<testcase name="someTest(someData Z)" ...>
The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.
Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
- Use the right name for the attribute (AI_Message),
rather than fixing it up in QTestJUnitStreamer.
- Don't pretend that we're adding line and file information,
only to discard it in QTestJUnitStreamer.
- Don't pretend to add benchmark information,
only to discard it in QTestJUnitStreamer.
Pick-to: 6.2
Change-Id: Ib6eadc12300157216fe9c6e8bcfebd7eb8a3ea68
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The QtTest best practices documentations recommends using output
mechanisms such as qDebug() and qWarning() for diagnostic messages,
and this is also what most of our own tests do.
The QWARN() macro and corresponding internal QTest::qWarn() function
was added when QtTest was first implemented, but was likely meant as
an internal implementation detail, like its cousin QTestLog::info(),
which does not have any corresponding macro.
This theory is backed by our own QtTest self-test (tst_silent)
describing the output from QWARN() as "an internal testlib warning".
The only difference between QWARN() and qWarning(), besides the much
richer feature set of the latter, is that qWarning() will not pass
on file and line number information in release mode, but QWARN() will.
This is an acceptable loss of functionality, considering that the user
can override this behavior by defining QT_MESSAGELOGCONTEXT.
[ChangeLog][QtTest] QWARN() has been deprecated in favor of qWarning()
Pick-to: 6.2
Change-Id: I5a2431ce48c47392244560dd520953b9fc735c85
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
For historical reasons a few of the subtests are skipped when
running with anything but the plain text logger to stdout.
To ensure we have as broad test coverage as possible for the
expected output of the various loggers we run these tests in
stdout-mode.
Pick-to: 6.2
Change-Id: I856905d1543afe89710533657a55bd599c0305fd
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit reporter was initially named xunit, but the naming was inaccurate
and the reporter was renamed in 27db9e458c.
The corresponding test has now been renamed as well, and as an added bonus
we only run it for that reporter.
Pick-to: 6.2
Change-Id: I59cb7d949514cdf46a0199a53a7a3e39f833207c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
In commit 9bad096c09 I gave
qFuzzyIsNull(qfloat16) a more forgiving threshold, consistent with
qFloatCompare(qfloat16, qfloat16)'s fractional threshold. The selftest
failed to catch two of the tests failing, so fix one of them to use
different values, and remove one.
Updated test expection for txt and deleted for other formats, as
they're skipped (in tst_selftests.cpp) for this test. Refined the
generator script to know about this test only being tested for txt.
Pick-to: 6.2
Change-Id: I109547cf92178bb9f5ff0b06e0b3bb40c881b41b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The MinGW resource compiler fails to handle compile definitions
with multiple values.
When the resource file is compiled as part of the main target rather
than a separate object library, the resource generation rule will
inherit all the compile definitions from the main target.
For the case of tst_selftests this causes errors like
gcc: error: badxml\: No such file or directory
gcc: error: benchlibcallgrind\: No such file or directory
gcc: error: benchlibcounting\: No such file or directory
gcc: error: benchlibeventcounter\: No such file or directory
gcc: error: benchliboptions\: No such file or directory
Limit the compile definition to the C++ language only, so the multiple
values are not passed to the resource compiler.
Pick-to: 6.2
Change-Id: Ie53666839272556323b50d79c090f0dc71745d11
Reviewed-by: Cristian Adam <cristian.adam@qt.io>
Says GCC:
In function ‘char* QTest::toString(QPair<T1, T2>&) [with T1 = QWidget*; T2 = QEvent::Type]’,
warning: ‘%s’ directive argument is null [-Wformat-overflow=]
Fix by re-using formatString(), once introduced for std::tuple.
As a side-effect, this gets rid of the funny double-quotes around the
output.
Change-Id: I2dd5f10fa2b3a392370bf487c1b7e98f3d190978
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Previously, it went direct to QTestResults::addFailure() without going
via the checking for expected failure. Add QTestResults::fail() to
take care of this checking, as for verify() and compare().
Tidied up the code implementing expected failure and QFAIL(), while I
was about it. Adjusted an existing test to verify that expecting a
QFAIL() works, by using QFAIL() instead of QVERIFY(false).
Remove the QVERIFY(false) whose comment brought this to my attention.
[ChangeLog][QtTestLib][QFAIL] QEXPECT_FAIL() now correctly anticipates
a subsequent QFAIL(). Previously QFAIL() counted as a fail regardless.
Change-Id: Icc28cf70e5ff3006363791ea03aa01f2f591eb71
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Output object name and class in QCOMPARE(). This should help
to debug flaky QWidget tests that for example check on focusWidget().
[ChangeLog][QtTestLib] QCOMPARE() now reports QObject * values by class and objectName().
Task-number: QTBUG-64446
Change-Id: Ife04e89bba04fc78d077c8f0f07af17a17c9cf8c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Cast size_t return from std::vector explicitly to int to silence compiler
warnings.
Change-Id: I0c425b3cec7feec0712e1173ab7e60b28695d6d7
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When inserting rows to a branch with no columns
the tester should not complain about indexes being invalid
Pick-to: 6.1 6.0 5.15
Change-Id: I466f4e5140b10f6dcf65a71f109c2d3be7336507
Reviewed-by: David Faure <david.faure@kdab.com>
When rows are removed from a model with no columns,
the test should not report a problem if indexes are invalid
Fixes: QTBUG-92886
Pick-to: 6.1 6.0 5.15
Change-Id: I7a042dfdb2575f87208a00cbed13db3869807f84
Reviewed-by: David Faure <david.faure@kdab.com>
We are still using Catch2 2.11.3 which is not adapted to Apple
silicon yet. This patch backports the required change
from Catch v3.0.0-preview.3.
Change-Id: Ifa14a1fdd6cd1f661c94a0a78648cb01bd9699c1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Wayland does not support WindowActivation capability, which
will trigger a warning in qWaitForWindowActive(). Since the
selftests depend on matching the exact output of the tests,
it gets confused by this, even though the tests are passing.
So we suppress the warning when the capability is not
supported.
Change-Id: I1e1f530a7b237183ee4ef123618e868b746ba863
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The emulation detection has been usable only on qtbase tests, move it to
QTest so that it can be used in other modules as well.
Pick-to: 6.1
Change-Id: I4b2321b7856414d7b1cfd5e6b1405a633c6bb878
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Add an enumeration for system-out and alog element
for it. Redirect the messages types that are not warnings/errors
to this element. For compatibility, write it out only
if it is not empty. Rename enumerations and members accordingly.
[ChangeLog][QtTestLib] In JUnit XML, output that is
not a warning/error is now logged under <system-out>
instead of <system-err>.
Fixes: QTBUG-86540
Change-Id: I55598eafa7dafa486ac5a8221029c332ff47413b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
- Generate the expected files in the source tree,
removing the need to copy them over
- Add proper option parsing, add options for formats
and to skip the callgrind test, which locks up
- Determine the script location by __file__
- Determine the Qt version by reading the .cmake.conf file
- Introduce f-strings
- Print the usage when invoked in the wrong directory
Task-number: QTBUG-86540
Change-Id: Idabb50a14db60127374b7a1271951dbbbc85d131
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Those serve no purpose anymore, now that the .pro files are gone.
Task-number: QTBUG-88742
Change-Id: I39943327b8c9871785b58e9973e4e7602371793e
Reviewed-by: Cristian Adam <cristian.adam@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
Remove the qmake project files for most of Qt.
Leave the qmake project files for examples, because we still test those
in the CI to ensure qmake does not regress.
Also leave the qmake project files for utils and other minor parts that
lack CMake project files.
Task-number: QTBUG-88742
Change-Id: I6cdf059e6204816f617f9624f3ea9822703f73cc
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
This is standard for all test cases. It matters, because cmake targets
are generated, and e.g.
$ ninja tst_qabstractitemmodeltester
should do what the user expects.
Change-Id: Iac8160c53d5005382e61c03b7daceaba0a4c2596
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: David Skoland <david.skoland@qt.io>
Complete search and replace of QtTest and QtTest/QtTest with QTest, as
QtTest includes the whole module. Replace all such instances with
correct header includes. See Jira task for more discussion.
Fixes: QTBUG-88831
Change-Id: I981cfae18a1cabcabcabee376016b086d9d01f44
Pick-to: 6.0
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
This test was failing but was returning a pass because of a bug
in androidtestrunner.
Task-number: QTBUG-88507
Task-number: QTBUG-88508
Pick-to: 6.0
Change-Id: I4fda7c2ddf8cd1e6858617e7bb447078c5768a75
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The RegisterMethodArgumentMetaType had been changed to take a QMetaType
instead of a type id in 0161f00e50.
Unfortunately, the usage of it in QSignalSpy was missed. This patch
adjusts the metacall to correctly use a QMetaType.
Moreover, use parameterMetaType instead of parameterType to benefit from
metatypes which are already resolved at compile time.
Task-number: QTBUG-88260
Fixes: QTBUG-88356
Change-Id: Id8fa46581a005d62818971ea24d8aa2e39dcd6d0
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Otherwise XOpenDisplay in a newly spawned process may fail with
a 'No protocol specified' message (as it recently happened in
CentOS 8.1 vm).
Fixes: QTBUG-87621
Pick-to: 5.15
Change-Id: Ib6c08c7f154fb2a126d32a4aa52b535e5daa1589
Reviewed-by: Liang Qi <liang.qi@qt.io>
The old test harness used to spit out stderr only, but
to be on the safe side we spit out both.
Change-Id: Ib8e57fd1b0e4d8542ac552a6fe58c07016df7f5f
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
It's been obsolete for a long time already. Make sure
the compiler now warns about it and remove all remaining
uses in qtbase.
Change-Id: I0ff80311184dba52d2ba5f4e2fabe0d47fdc59d7
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
This time based on grepping to also include documentation, tests and
examples previously missed by the automatic tool.
Change-Id: Ied1703f4bcc470fbc275f759ed5b7c588a5c4e9f
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
addBFail() asserts on the file being non-null. The convention seems to
be "Unknown File" for cases where we cannot determine the file.
Change-Id: I3a4d0130352d77d75f264fad6f3bd47c6700ef4c
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Modify special case locations to use the new API as well.
Clean up some stale .prev files that are not needed anymore.
Clean up some project files that are not used anymore.
Task-number: QTBUG-86815
Change-Id: I9947da921f98686023c6bb053dfcc101851276b5
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Resolve remaining Qt6 TODO
[ChangeLog][QtCore][QAbstractEventDispatcher] The signature of the abstract virtual registerTime function now takes a qint64 value for the interval parameter.
Change-Id: I10166ad5cfb455edc404d465a3731ff094a8977e
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
As per ### Qt6 comment. Also rename the LibraryLocation enum
to LibraryPath.
Change-Id: I556025a19c5bcdf2ff52598eaba32269522d4128
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Remove around 1000 compiler warnings about missing overrides
in our auto tests.
This significantly reduce the compiler warning noise in our auto
tests, so that one can actually better see the real problems
inbetween.
Change-Id: Id0c04dba43fcaf55d8cd2b5c6697358857c31bf9
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
It was marked internal anyway. Use the constructor taking a
QMetaType instead.
Change-Id: I15b9cd0911aac063a0f0fe0352fa2c84b7f7c691
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
As defined by https://llg.cubic.org/docs/junit/
Change-Id: Ic7683f3d49c529674f8467d591528d4a65d3add8
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
