The timestamp will no longer be incremented by 500ms after a mouse
release if the delay has been explicitly specified.
The default delay is 1 ms since f5010c49a3
but the running timestamp was unconditionally post-incremented by 500ms
after every mouse release, to prevent double-clicks, which were always
deemed as unintended (because we have a mouseDClick function for that).
Now, we do that 500ms increment only if the user has not provided a
delay value in the function argument at all. We have often found it
useful in our own tests to generate double-clicks "the hard way", by
sending indivdual events, so as to be able to check state in some target
object at each step, as shown in the new snippet.
[ChangeLog][QtTest] QTest::mouseRelease() and mouseClick() can now be
used to test double-clicks, by specifying a realistic timestamp delay.
Fixes: QTBUG-102441
Change-Id: I8e8d242061f79efb4c6e02638645e03661a9cd92
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
Unix systems have got crash loggers in the past 15-20 years, notably
macOS and Linux (abrtd, systemd-coredumpd, etc.). By setting the core
dump limit to zero, those tools should be mostly inhibited from running
and thus not interfere with the parent process' timeouts. Even for
systems without core dump loggers, disabling the writing of a core dump
to the filesystem should also help.
Pick-to: 6.4
Change-Id: I12a088d1ae424825abd3fffd171d112d0671effe
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
QWindow::requestActivate() is not supported.
We have one tst_selftests binary, and will test it with both xcb and
wayland qpa plugin. A runtime check and skip will have different
restult files, which is not implemented in testlib yet.
Task-number: QTBUG-107578
Pick-to: 6.4 6.2
Change-Id: Idc8cb24c6f42a9f0f4dc9493e3fd1a5803ba7ce0
Reviewed-by: Liang Qi <liang.qi@qt.io>
The QTRY_* macros and QTestEventLoop exit early if the test has
resolved; however, in the cleanup phase of a test, even if the test
has failed, these loops should continue as normal.
[ChangeLog][QtTest] During the cleanup() phase of a test, the QTRY_*
macros and QTestEventLoop now ignore the test resolution, in contrast
to when they are used from the test itself, which (since 6.3.0) exits
the loops early if the test has failed.
Pick-to: 6.4 6.3
Fixes: QTBUG-104441
Change-Id: I2673161967cbbc57815155af698a9338ab98a686
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The regex used a greedy .+ to match an actual dot and a space,
preceding an open parenthesis, with the result that if the message
contained any parentheses that .+ swallowed everything up to the last
of them. Correct the regex so that recently-added tests' error
messages show up correctly.
Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The recently added test for QTestEventLoop produced a message which
the TAP test-logger mis-parses. Since that message shall soon go away
(when I fix the bug the new test exists to verify), modify one of
cmptest's QVERIFY2() messages to trip up the same bug, so we can spot
any regressions on this in future (after the imminent fix to this TAP
issue).
Change-Id: Ibbe9931c01c75df529c9571e2bbdbd34010dd9ec
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Use the standard mechanism instead of a hand-rolled way of identifying
function/data-tag combinations. Adds missing data relevant to
identifying test cases, some of which previously had apparently the
same name, with no hint at why the test-case was seemingly run
repeatedly.
Change-Id: I6225c6d1990069c94a1f1c8dbb179993b96076e7
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
As discovered in the expanded testing of QTRY_COMPARE() using the same
class, the timer needs a context object and a slot to call.
This amends commit 35ad157d88
Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I41fc23de84ce8c7d6608db0005276a2071974494
Reviewed-by: Jason McDonald <macadder1@gmail.com>
In the process, simplify the latter while adding some actual
time-variation for the QTRY_* loop to navigate round - based on the
extendedcompare test's ClassWithDeferredSetter. Testing remains
primitive, but is at least a bit more thorough.
Pick-to: 6.4
Change-Id: I40be8fb485f3f18f0a4f4bc62ad36cccac691979
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
These contain output only ever sent to stderr, which tst_selftests no
longer looks at; it merely verifies stderr is empty for all tests
except those expected to produce stderr output, and ignores the stderr
output for these last.
Change-Id: I50fee445d84c2d125e6db5303fc389e87746455e
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Fairly minimal for now, just enough to verify a bug and serve as the
sign of success when it's fixed. Tests fail in ways they shouldn't,
for now; see expected_eventloop.* for details, notably "Earlier test
failed to clean up" messages.
Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I59be4aa5f21fed23b19a0593a8c2f6c9956507df
Reviewed-by: Jason McDonald <macadder1@gmail.com>
CMakeLists.txt and .cmake files of significant size
(more than 2 lines according to our check in tst_license.pl)
now have the copyright and license header.
Existing copyright statements remain intact
Task-number: QTBUG-88621
Change-Id: I3b98cdc55ead806ec81ce09af9271f9b95af97fa
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
The skipcleanup and failcleanup tests were actually testing skip and
fail in cleanupTestCase(), not in cleanup(). Add almost-duplicate
tests and clean up so that we now have {fail,skip}cleanup(,testcase}
tests to cover all four cases. Generated expected output. The new
tests (with old names) get their fail or skip - during cleanup() -
reported against the test instead of the cleanupTestCase function.
(Results for {init,cleanup}TestCase() are always reported, even when
these slots are not defined, as no-op passes.)
Pick-to: 6.4
Change-Id: I0988d1696b50c0e2f30c45ddc25e1bd0bfd2151a
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
For built-in types, this is a compile-time assert - we should not have
any types in Qt for which we have neither debug streaming nor a
QTest::toString specialization implemented. A build of most of Qt
submodules passes with this change, after minor modifications to some
tests. We cannot declare QSizeHint::Policy as a metatype after the
QMetaType has already been instantiated for it, and the QDebug stream
operator for QElaspedTimer needs to be correctly declared within the
namespace.
Add a self-test function for a custom type, and update reference files
of the self-test.
Task-number: QTBUG-104867
Pick-to: 6.4
Change-Id: I2936db5933f4589fce45f47cf2f3224ed614d8c9
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Before 0681a2dd5a, QCOMPARE'ing types
for which no QTest::toString specialization exists did not output
Actual and Expected lines on failure, as that would only print <null>
for both values (which then look like the same value, confusingly).
Commit 0681a2dd5a changed that behavior,
and started printing the confusing <null> values.
Take care of the logic in the formatFailMessage function: if both values
are nullptr, then print only the variable names, but not the confusing
<null> text representation of the values.
Remove dead and duplicated code related to the formatting logic, add a
self-test function, and update the expected_cmptest files.
Fixes: QTBUG-104867
Pick-to: 6.4
Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Previously trying to execute a test function with an unknown data tag
would print an error message but exit with 0.
This patch stores a test failure, and continues trying to execute the
rest of the command line arguments, if any. In the end the process exits
with the usual exit code (number of failed tests) which is now !=0.
Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-24240
Change-Id: Id4d422035f173e01e77ca88028dfd94dc0f9085c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The siginfo_t parameter allows us to show what process sent a signal or
the crashing address. Additionally, it allows us to determine if the
crashing signal was indeed sent due to a crash.
The selftest tst_crashes produces now:
$ QTEST_DISABLE_STACK_DUMP=1 ./crashes
********* Start testing of tst_Crashes *********
Config: Using QtTest library 6.4.0, Qt 6.4.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by GCC 11.2.1 20220420 [revision 691af15031e00227ba6d5935c1d737026cda4129]), opensuse-tumbleweed 20220428
PASS : tst_Crashes::initTestCase()
Received signal 11 (SIGSEGV), code 1, for address 0x0000000000000004
Function time: 0ms, total time: 0ms
[1] 201995 segmentation fault (core dumped) QTEST_DISABLE_STACK_DUMP=1 ./crashes
The last line comes from the shell. The code isn't decoded, but on Linux
it's a SEGV_MAPERR. macOS prints exactly the same thing.
I've updated one of the expected_crashes_*.txt output that doesn't seem
possible (the "Received a fatal error" message does not appear in Qt
anywhere).
Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc8391234f0e2
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
It's easier to remember what "SIGSEGV" means instead of "11".
GNU libc has offered sigabbrev_np() (non-portable) since 2.32; for older
libcs, we'll be happy with a hardcoded list.
Selftest updated to match... though it didn't seem to be necessary.
Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc66ecf6e9465
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Two patches applied to upstream release.
Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-103732
Change-Id: Id64b65c4567433806047a2a34fa85ab5f260e6cc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
[ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its
arguments lazily, speeding up the general case where the comparison
doesn't fail. This is true for the QCOMPARE functionality provided
by Qt. If you specialized qCompare() for your own types, then you
need to change its implementation in line with Qt's own qCompare()
specializations in order to enable this feature.
[ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will
now print the actual and expected values upon failure.
Previously it was not like that because of the compareHelper()
overload in qtestresult.cpp that treated the presence of
nullptr-arguments as a reason to ignore formatFailMessage() call.
New implementation does not have this check, and correctly
executes formatFailMessage() for all arguments.
Note that the qCompare() overloads that call QTestResult::compare()
internally were not affected by this patch, because they already
defer toString() invocation until the comparison fails.
Some numbers, collected against shared release developer build.
I checked how this change affects the test execution. The idea was
to pick some tests for types that do not have a specific
QTestResult::compare overload, so I picked a couple of QByteArray
tests.
The comparison is done by running a test 10 times and taking the
average execution duration, as reported in the log.
tst_qbytearrayapisymmetry:
Before: 15.6 ms
After: 14.2 ms
tst_qbytearray:
Before: 41 ms
After: 36 ms
The benefit is around 9% and 12% respectively.
Fixes: QTBUG-98874
Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
[ChangeLog][QTestLib] Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT
macros that repeatedly execute QCOMPARE_{EQ,NE,LT,LE,GT,GE} until either
the comparison returns true or the timeout expires. Also add
QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE} macros that simply invoke the
*_WITH_TIMEOUT versions with the usual timeout of five seconds.
Task-number: QTBUG-98873
Change-Id: Ib0d7d1c8c997f442b46acd85da738a8f512cc875
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
[ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
macros. These new macros behave similarly to QVERIFY(a op b),
where 'op' is ==, !=, <, <=, >, >= respectively, but print
a formatted error message with argument values in case of failure.
The formatting is done lazily, which means that the strings will
be generated only when the comparison fails.
Also add a new test for tst_selftest and generate expected output
for it.
Fixes: QTBUG-98873
Task-number: QTBUG-98874
Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Add the Q_OS_WIN32 guard for the include of QWinEventNotifier.
Change-Id: I7824b2ee236a370c83fd85a2f594a39cf36b36e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
We don't support it any more. I don't think it has ever properly
compiled Qt 6 (and it's no longer working for me against GCC 12's
libstdc++ headers). If you report a bug against it, Intel support's
first question is if you can try instead the new Clang/LLVM-based oneAPI
C++ compiler.
So we support only that one, which identifies itself as Q_CC_CLANG.
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb57a092c8439e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.
So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.
This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).
Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.
Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.
Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When producing a diagnostics block, include the file and line
information, if we have it, to describe it. This presently only adds
this information for skip, but could in principle do the same for a
B?XPass.
Task-number: QTBUG-96844
Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.
However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.
In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)
Add methods to QTestCharBuffer, and a helper function, to support this.
Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
It relied on the set(QT_REPO_MODULE_VERSION ...) line being the first
thing in the file. It no longer is, and the variable is referenced in
an earlier condition.
Change-Id: I2bd2abd2a5680782ddd61f4bf4a4a3265b444d52
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When a B?Fail's description doesn't match the QVERIFY/QCOMPARE
regexes, it got output as a comment in the YAML block after the test
line had already reported it as a TODO comment. An empty description
would also have lead to an empty comment in the YAML block. Condition
this fallback output case on there being a description that hasn't yet
been reported in the test line.
Task-number: QTBUG-96844
Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In all places but one they were treated the same; fix that last one to
match the rest. This removes one line from the YAML block for each
blacklisted XFail test; in each case, this message duplicates the one
on its "not ok ... # TODO..." test line.
Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Multiple tests use qt_internal_add_resource that copies the
functionality that is already implemented inside the
qt_internal_add_test function. Simplify these test by replacing
the qt_internal_add_resource call with the new BUILTIN_TESTDATA
option.
Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When running a test under increased verbosity levels, QTestLib would
print "failure location" for every QTest function (QVERIFY, QCOMPARE,
...), even if there's no failure at all.
Keep the code centralized, but split the formatting of failures and
non-failures (other messages).
Task-number: QTBUG-96844
Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c
Pick-to: 5.15 6.2 6.3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Update Catch2 header to latest released version. Needed
to build correctly with glibc 2.34.
Two patches applied on top of the upstream release. Previously
used Qt specific change extracted to a separate patch file.
Fixes: QTBUG-95842
Pick-to: 6.3 6.2
Change-Id: I67e442bbe756ff00c96a45eabf593f7aecddd628
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
- It should not be built for systems without process feature.
Task-number: QTBUG-99123
Pick-to: 6.2 6.3
Change-Id: I71caa59c2168435894c7d1afcc8226e44178439f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
It was missing #if-ery on feature regularexpression for one test that
depends on it. One of its comments had a long line. Added some
annotations to make clear what's going on in messier tests.
Change-Id: I06d8748a134591f93b36029713e52ffd826a24dc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.
Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Without the fix, some tests from tst_selftests were failing with:
Invalid MIT-MAGIC-COOKIE-1 key
Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
If an incident has an empty description we produced output with a
dangling space. Avoid triggering the inanity 'bot with those.
Reorganise the code to save some duplication.
Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The rationale for skipping pass after XFail is "to emit a single test
point for" the test; emitting several XFails violates that aim.
Task-number: QTBUG-96844
Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Skip ends the test (albeit inconclusively). Rearrange the enums in
the abstract logger, move code to handle skip between relevant
function and tidy up various things that became simpler as a result.
Also reorder the message enum, and its switches, to separate testlib's
internals from the usual Qt messages, and put each group in ascending
order of severity.
Task-number: QTBUG-96844
Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
