Commit Graph

469 Commits

Author SHA1 Message Date
Ahmad Samir
9d9c25e214 Use QString/QByteArray range erase()
Change-Id: I0fb81306ebe8fc7acd63bb62dc6720c734461da0
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-12-07 15:03:03 +02:00
Marc Mutz
1c6bf3e09e Port from container::count() and length() to size() - V5
This is a semantic patch using ClangTidyTransformator as in
qtbase/df9d882d41b741fef7c5beeddb0abe9d904443d8, but extended to
handle typedefs and accesses through pointers, too:

    const std::string o = "object";

    auto hasTypeIgnoringPointer = [](auto type) { return anyOf(hasType(type), hasType(pointsTo(type))); };

    auto derivedFromAnyOfClasses = [&](ArrayRef<StringRef> classes) {
        auto exprOfDeclaredType = [&](auto decl) {
            return expr(hasTypeIgnoringPointer(hasUnqualifiedDesugaredType(recordType(hasDeclaration(decl))))).bind(o);
        };
        return exprOfDeclaredType(cxxRecordDecl(isSameOrDerivedFrom(hasAnyName(classes))));
    };

    auto renameMethod = [&] (ArrayRef<StringRef> classes,
                            StringRef from, StringRef to) {
        return makeRule(cxxMemberCallExpr(on(derivedFromAnyOfClasses(classes)),
                            callee(cxxMethodDecl(hasName(from), parameterCountIs(0)))),
                        changeTo(cat(access(o, cat(to)), "()")),
                        cat("use '", to, "' instead of '", from, "'"));
    };

    renameMethod(<classes>, "count", "size");
    renameMethod(<classes>, "length", "size");

except that the on() matcher has been replaced by one that doesn't
ignoreParens().

a.k.a qt-port-to-std-compatible-api V5 with config Scope: 'Container'.

Added two NOLINTNEXTLINEs in tst_qbitarray and tst_qcontiguouscache,
to avoid porting calls that explicitly test count().

Change-Id: Icfb8808c2ff4a30187e9935a51cad26987451c22
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-11-03 14:59:24 +01:00
Marc Mutz
3d6f5e95f0 Update Catch2 to v2.13.10
Two patches applied to upstream release.

Pick-to: 6.4 6.2
Fixes: QTBUG-103321
Task-number: QTBUG-99122
Change-Id: Ief451a21e5f61851cb03a9ca94ffe0f864e4b1f8
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2022-10-17 14:07:43 +02:00
Shawn Rutledge
0f94430a0f testlib: make it possible to test double-clicks with discrete events
The timestamp will no longer be incremented by 500ms after a mouse
release if the delay has been explicitly specified.

The default delay is 1 ms since f5010c49a3
but the running timestamp was unconditionally post-incremented by 500ms
after every mouse release, to prevent double-clicks, which were always
deemed as unintended (because we have a mouseDClick function for that).
Now, we do that 500ms increment only if the user has not provided a
delay value in the function argument at all. We have often found it
useful in our own tests to generate double-clicks "the hard way", by
sending indivdual events, so as to be able to check state in some target
object at each step, as shown in the new snippet.

[ChangeLog][QtTest] QTest::mouseRelease() and mouseClick() can now be
used to test double-clicks, by specifying a realistic timestamp delay.

Fixes: QTBUG-102441
Change-Id: I8e8d242061f79efb4c6e02638645e03661a9cd92
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
2022-10-16 08:36:03 +02:00
Thiago Macieira
c849c48d19 Autotest/Unix: request zero-sized core dumps for crashing code
Unix systems have got crash loggers in the past 15-20 years, notably
macOS and Linux (abrtd, systemd-coredumpd, etc.). By setting the core
dump limit to zero, those tools should be mostly inhibited from running
and thus not interfere with the parent process' timeouts. Even for
systems without core dump loggers, disabling the writing of a core dump
to the filesystem should also help.

Pick-to: 6.4
Change-Id: I12a088d1ae424825abd3fffd171d112d0671effe
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2022-10-14 08:18:53 -07:00
Liang Qi
bdd2b68696 tests: skip tst_selftests on Wayland and XWayland
QWindow::requestActivate() is not supported.

We have one tst_selftests binary, and will test it with both xcb and
wayland qpa plugin. A runtime check and skip will have different
restult files, which is not implemented in testlib yet.

Task-number: QTBUG-107578
Pick-to: 6.4 6.2
Change-Id: Idc8cb24c6f42a9f0f4dc9493e3fd1a5803ba7ce0
Reviewed-by: Liang Qi <liang.qi@qt.io>
2022-10-14 17:18:53 +02:00
Marc Mutz
df9d882d41 Port from container.count()/length() to size()
This is semantic patch using ClangTidyTransformator:

  auto QtContainerClass = expr(hasType(namedDecl(hasAnyName(<classes>)))).bind(o)
  makeRule(cxxMemberCallExpr(on(QtContainerClass),
                             callee(cxxMethodDecl(hasAnyName({"count", "length"),
                                                  parameterCountIs(0))))),
           changeTo(cat(access(o, cat("size"), "()"))),
           cat("use 'size()' instead of 'count()/length()'"))

a.k.a qt-port-to-std-compatible-api with config Scope: 'Container'.

<classes> are:

    // sequential:
    "QByteArray",
    "QList",
    "QQueue",
    "QStack",
    "QString",
    "QVarLengthArray",
    "QVector",
    // associative:
    "QHash",
    "QMultiHash",
    "QMap",
    "QMultiMap",
    "QSet",
    // Qt has no QMultiSet

Change-Id: Ibe8837be96e8d30d1846881ecd65180c1bc459af
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-10-04 07:40:08 +02:00
Edward Welbourne
0462dba766 Skip early return from test loops during cleanup()
The QTRY_* macros and QTestEventLoop exit early if the test has
resolved; however, in the cleanup phase of a test, even if the test
has failed, these loops should continue as normal.

[ChangeLog][QtTest] During the cleanup() phase of a test, the QTRY_*
macros and QTestEventLoop now ignore the test resolution, in contrast
to when they are used from the test itself, which (since 6.3.0) exits
the loops early if the test has failed.

Pick-to: 6.4 6.3
Fixes: QTBUG-104441
Change-Id: I2673161967cbbc57815155af698a9338ab98a686
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-09-22 17:34:51 +02:00
Edward Welbourne
abab3c5dad Fix error in TAP test logger's regex to recognize QVERIFY() output
The regex used a greedy .+ to match an actual dot and a space,
preceding an open parenthesis, with the result that if the message
contained any parentheses that .+ swallowed everything up to the last
of them. Correct the regex so that recently-added tests' error
messages show up correctly.

Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-09-22 17:34:51 +02:00
Edward Welbourne
16f5132b21 Add test-case for TAP logging glitch
The recently added test for QTestEventLoop produced a message which
the TAP test-logger mis-parses. Since that message shall soon go away
(when I fix the bug the new test exists to verify), modify one of
cmptest's QVERIFY2() messages to trip up the same bug, so we can spot
any regressions on this in future (after the imminent fix to this TAP
issue).

Change-Id: Ibbe9931c01c75df529c9571e2bbdbd34010dd9ec
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-09-22 17:34:51 +02:00
Edward Welbourne
5832087859 Include global data tags in teamcity's test function names
Use the standard mechanism instead of a hand-rolled way of identifying
function/data-tag combinations. Adds missing data relevant to
identifying test cases, some of which previously had apparently the
same name, with no hint at why the test-case was seemingly run
repeatedly.

Change-Id: I6225c6d1990069c94a1f1c8dbb179993b96076e7
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-09-13 14:51:00 +02:00
Volker Hilsheimer
bf3500a631 Fix cmake warnings from using deprecated PUBLIC_LIBRARIES
Pick-to: 6.4
Change-Id: I8d4f0d9e98edc4e06dda9f58e62197572b4cbdbc
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-09-05 18:29:10 +02:00
Lucie Gérard
32df595275 Change the license of all CMakeLists.txt and *.cmake files to BSD
Task-number: QTBUG-105718
Change-Id: I5d3ef70a31235868b9be6cb479b7621bf2a8ba39
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-08-23 23:58:42 +02:00
Edward Welbourne
45cbb1e31e Fix DeferredFlag implementation for QTestEventLoop
As discovered in the expanded testing of QTRY_COMPARE() using the same
class, the timer needs a context object and a slot to call.
This amends commit 35ad157d88

Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I41fc23de84ce8c7d6608db0005276a2071974494
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-08-12 20:32:30 +02:00
Edward Welbourne
54bcefb25c Test QTRY_COMPARE() and expand testing of QTRY_VERIFY*()
In the process, simplify the latter while adding some actual
time-variation for the QTRY_* loop to navigate round - based on the
extendedcompare test's ClassWithDeferredSetter. Testing remains
primitive, but is at least a bit more thorough.

Pick-to: 6.4
Change-Id: I40be8fb485f3f18f0a4f4bc62ad36cccac691979
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-08-12 20:32:30 +02:00
Edward Welbourne
f5840692b0 Remove the stale expected_crashes_[345].txt
These contain output only ever sent to stderr, which tst_selftests no
longer looks at; it merely verifies stderr is empty for all tests
except those expected to produce stderr output, and ignores the stderr
output for these last.

Change-Id: I50fee445d84c2d125e6db5303fc389e87746455e
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-08-12 20:32:30 +02:00
Edward Welbourne
35ad157d88 Add some testing of QTestEventLoop
Fairly minimal for now, just enough to verify a bug and serve as the
sign of success when it's fixed. Tests fail in ways they shouldn't,
for now; see expected_eventloop.* for details, notably "Earlier test
failed to clean up" messages.

Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I59be4aa5f21fed23b19a0593a8c2f6c9956507df
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-08-08 20:49:03 +02:00
Lucie Gérard
fb1b20eab3 Add license headers to cmake files
CMakeLists.txt and .cmake files of significant size
(more than 2 lines according to our check in tst_license.pl)
now have the copyright and license header.

Existing copyright statements remain intact

Task-number: QTBUG-88621
Change-Id: I3b98cdc55ead806ec81ce09af9271f9b95af97fa
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-08-03 17:14:55 +02:00
Alexandru Croitor
4d22405e48 CMake: Don't use PUBLIC_LIBRARIES for tests and test helpers
Change-Id: I9b7404e1d3a78fe0726ec0f5ce1461f6c209e90d
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2022-07-28 14:46:53 +02:00
Edward Welbourne
21e9c223b7 Test skip and fail in cleanup() as well as in cleanupTestCase()
The skipcleanup and failcleanup tests were actually testing skip and
fail in cleanupTestCase(), not in cleanup(). Add almost-duplicate
tests and clean up so that we now have {fail,skip}cleanup(,testcase}
tests to cover all four cases. Generated expected output. The new
tests (with old names) get their fail or skip - during cleanup() -
reported against the test instead of the cleanupTestCase function.
(Results for {init,cleanup}TestCase() are always reported, even when
these slots are not defined, as no-op passes.)

Pick-to: 6.4
Change-Id: I0988d1696b50c0e2f30c45ddc25e1bd0bfd2151a
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
2022-07-25 19:29:07 +02:00
Volker Hilsheimer
92e696b4ba Use debug stream in QTest::toString's default fallback if possible
For built-in types, this is a compile-time assert - we should not have
any types in Qt for which we have neither debug streaming nor a
QTest::toString specialization implemented. A build of most of Qt
submodules passes with this change, after minor modifications to some
tests. We cannot declare QSizeHint::Policy as a metatype after the
QMetaType has already been instantiated for it, and the QDebug stream
operator for QElaspedTimer needs to be correctly declared within the
namespace.

Add a self-test function for a custom type, and update reference files
of the self-test.

Task-number: QTBUG-104867
Pick-to: 6.4
Change-Id: I2936db5933f4589fce45f47cf2f3224ed614d8c9
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
2022-07-13 15:14:35 +02:00
Volker Hilsheimer
54b276be0b testlib: Don't print QCOMPARE values if they lack string representation
Before 0681a2dd5a, QCOMPARE'ing types
for which no QTest::toString specialization exists did not output
Actual and Expected lines on failure, as that would only print <null>
for both values (which then look like the same value, confusingly).

Commit 0681a2dd5a changed that behavior,
and started printing the confusing <null> values.

Take care of the logic in the formatFailMessage function: if both values
are nullptr, then print only the variable names, but not the confusing
<null> text representation of the values.

Remove dead and duplicated code related to the formatting logic, add a
self-test function, and update the expected_cmptest files.

Fixes: QTBUG-104867
Pick-to: 6.4
Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-07-10 03:08:32 +00:00
Dimitrios Apostolou
d23dcc0542 qtestlib: exit with error if a test is invoked with unknown data tag
Previously trying to execute a test function with an unknown data tag
would print an error message but exit with 0.

This patch stores a test failure, and continues trying to execute the
rest of the command line arguments, if any. In the end the process exits
with the usual exit code (number of failed tests) which is now !=0.

Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-24240
Change-Id: Id4d422035f173e01e77ca88028dfd94dc0f9085c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-28 18:06:33 +02:00
Thiago Macieira
5e67e7efaa FatalSignalHandler: print some more information from siginfo_t
The siginfo_t parameter allows us to show what process sent a signal or
the crashing address. Additionally, it allows us to determine if the
crashing signal was indeed sent due to a crash.

The selftest tst_crashes produces now:

$ QTEST_DISABLE_STACK_DUMP=1 ./crashes
********* Start testing of tst_Crashes *********
Config: Using QtTest library 6.4.0, Qt 6.4.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by GCC 11.2.1 20220420 [revision 691af15031e00227ba6d5935c1d737026cda4129]), opensuse-tumbleweed 20220428
PASS   : tst_Crashes::initTestCase()
Received signal 11 (SIGSEGV), code 1, for address 0x0000000000000004
         Function time: 0ms, total time: 0ms
[1]    201995 segmentation fault (core dumped)  QTEST_DISABLE_STACK_DUMP=1 ./crashes

The last line comes from the shell. The code isn't decoded, but on Linux
it's a SEGV_MAPERR. macOS prints exactly the same thing.

I've updated one of the expected_crashes_*.txt output that doesn't seem
possible (the "Received a fatal error" message does not appear in Qt
anywhere).

Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc8391234f0e2
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-06-28 00:39:35 +00:00
Thiago Macieira
17454bf9c6 FatalSignalHandler: print the signal name on crash
It's easier to remember what "SIGSEGV" means instead of "11".

GNU libc has offered sigabbrev_np() (non-portable) since 2.32; for older
libcs, we'll be happy with a hardcoded list.

Selftest updated to match... though it didn't seem to be necessary.

Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc66ecf6e9465
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-08 20:39:18 +00:00
Marc Mutz
a7dcdf063d Update Catch2 to v2.13.9
Two patches applied to upstream release.

Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-103732
Change-Id: Id64b65c4567433806047a2a34fa85ab5f260e6cc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-06-08 19:43:34 +02:00
Ivan Solovev
0681a2dd5a QTestLib: rework QTest::compare_helper()
[ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its
arguments lazily, speeding up the general case where the comparison
doesn't fail. This is true for the QCOMPARE functionality provided
by Qt. If you specialized qCompare() for your own types, then you
need to change its implementation in line with Qt's own qCompare()
specializations in order to enable this feature.

[ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will
now print the actual and expected values upon failure.

Previously it was not like that because of the compareHelper()
overload in qtestresult.cpp that treated the presence of
nullptr-arguments as a reason to ignore formatFailMessage() call.
New implementation does not have this check, and correctly
executes formatFailMessage() for all arguments.

Note that the qCompare() overloads that call QTestResult::compare()
internally were not affected by this patch, because they already
defer toString() invocation until the comparison fails.

Some numbers, collected against shared release developer build.
I checked how this change affects the test execution. The idea was
to pick some tests for types that do not have a specific
QTestResult::compare overload, so I picked a couple of QByteArray
tests.
The comparison is done by running a test 10 times and taking the
average execution duration, as reported in the log.

tst_qbytearrayapisymmetry:
 Before: 15.6 ms
 After:  14.2 ms

tst_qbytearray:
 Before: 41 ms
 After:  36 ms

The benefit is around 9% and 12% respectively.

Fixes: QTBUG-98874
Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-06-03 20:37:51 +00:00
Ivan Solovev
cc6d984390 Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT()
[ChangeLog][QTestLib] Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT
macros that repeatedly execute QCOMPARE_{EQ,NE,LT,LE,GT,GE} until either
the comparison returns true or the timeout expires. Also add
QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE} macros that simply invoke the
*_WITH_TIMEOUT versions with the usual timeout of five seconds.

Task-number: QTBUG-98873
Change-Id: Ib0d7d1c8c997f442b46acd85da738a8f512cc875
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-06-03 22:37:50 +02:00
Ivan Solovev
343e0ff485 Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
[ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
macros. These new macros behave similarly to QVERIFY(a op b),
where 'op' is ==, !=, <, <=, >, >= respectively, but print
a formatted error message with argument values in case of failure.
The formatting is done lazily, which means that the strings will
be generated only when the comparison fails.

Also add a new test for tst_selftest and generate expected output
for it.

Fixes: QTBUG-98873
Task-number: QTBUG-98874
Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-03 22:37:50 +02:00
Alexey Edelev
b576d7f6d1 Guard the use of QWinEventNotifier in tests
Add the Q_OS_WIN32 guard for the include of QWinEventNotifier.

Change-Id: I7824b2ee236a370c83fd85a2f594a39cf36b36e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-05-17 02:51:22 +02:00
Lucie Gérard
05fc3aef53 Use SPDX license identifiers
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.

Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-05-16 16:37:38 +02:00
Thiago Macieira
20104bb237 Remove remnants of the old Intel C++ compiler
We don't support it any more. I don't think it has ever properly
compiled Qt 6 (and it's no longer working for me against GCC 12's
libstdc++ headers). If you report a bug against it, Intel support's
first question is if you can try instead the new Clang/LLVM-based oneAPI
C++ compiler.

So we support only that one, which identifies itself as Q_CC_CLANG.

Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb57a092c8439e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-05-11 12:50:08 -07:00
Sona Kurazyan
a885f28933 Replace uses of _qs with _s in tests
Task-number: QTBUG-101408
Change-Id: If092a68828a1e8056259cf90d035d9a87989244b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-04-07 19:30:17 +02:00
Edward Welbourne
784f5d76b9 TAP test logger: report B?XFAIL (mostly) as a message
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.

So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.

This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).

Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.

Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-29 22:21:58 +02:00
Edward Welbourne
ac38401d31 Move TAP's messages block to after the primary report details
In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.

Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-29 22:21:58 +02:00
Edward Welbourne
7d60fde9a7 Include file and line information in TAP diagnostics blocks
When producing a diagnostics block, include the file and line
information, if we have it, to describe it. This presently only adds
this information for skip, but could in principle do the same for a
B?XPass.

Task-number: QTBUG-96844
Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-29 00:05:12 +01:00
Edward Welbourne
ae37fa0464 TAP test logger: move messages into the diagnostics block
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.

However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.

In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)

Add methods to QTestCharBuffer, and a helper function, to support this.

Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-29 00:05:12 +01:00
Edward Welbourne
a8a7f6c4e6 Fix parsing of .cmake.conf in generate_expected_output.py
It relied on the set(QT_REPO_MODULE_VERSION ...) line being the first
thing in the file. It no longer is, and the variable is referenced in
an earlier condition.

Change-Id: I2bd2abd2a5680782ddd61f4bf4a4a3265b444d52
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-03-25 21:55:51 +01:00
Edward Welbourne
02d130db7e Avoid duplicate descriptions in TAP's incident YAML
When a B?Fail's description doesn't match the QVERIFY/QCOMPARE
regexes, it got output as a comment in the YAML block after the test
line had already reported it as a TODO comment. An empty description
would also have lead to an empty comment in the YAML block. Condition
this fallback output case on there being a description that hasn't yet
been reported in the test line.

Task-number: QTBUG-96844
Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-22 17:43:13 +01:00
Edward Welbourne
58b0d72f8d TAP test logger: treat blacklisted XFail the same as XFail
In all places but one they were treated the same; fix that last one to
match the rest. This removes one line from the YAML block for each
blacklisted XFail test; in each case, this message duplicates the one
on its "not ok ... # TODO..." test line.

Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2022-03-22 17:43:13 +01:00
Alexey Edelev
fdf5d11ff6 CMake: Add BUILTIN_TESTDATA option to qt_internal_add_test
Multiple tests use qt_internal_add_resource that copies the
functionality that is already implemented inside the
qt_internal_add_test function. Simplify these test by replacing
the qt_internal_add_resource call with the new BUILTIN_TESTDATA
option.

Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-02-11 21:54:44 +01:00
Giuseppe D'Angelo
a605b9297d QPlainTestLogger: do not print "failure location" if there's no failure
When running a test under increased verbosity levels, QTestLib would
print "failure location" for every QTest function (QVERIFY, QCOMPARE,
...), even if there's no failure at all.

Keep the code centralized, but split the formatting of failures and
non-failures (other messages).

Task-number: QTBUG-96844
Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c
Pick-to: 5.15 6.2 6.3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-02-02 23:08:37 +01:00
Samuli Piippo
42221aa84c Catch2: upgrade to 2.13.8
Update Catch2 header to latest released version. Needed
to build correctly with glibc 2.34.

Two patches applied on top of the upstream release. Previously
used Qt specific change extracted to a separate patch file.

Fixes: QTBUG-95842
Pick-to: 6.3 6.2
Change-Id: I67e442bbe756ff00c96a45eabf593f7aecddd628
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-31 09:26:00 +02:00
Tatiana Borisova
e76d27d3ed Exclude tst_selftests when feature process is disabled
- It should not be built for systems without process feature.

Task-number: QTBUG-99123
Pick-to: 6.2 6.3
Change-Id: I71caa59c2168435894c7d1afcc8226e44178439f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-27 21:35:58 +02:00
Edward Welbourne
5d80d24d3c Tidy up tst_Warnings: comments and #if-ery
It was missing #if-ery on feature regularexpression for one test that
depends on it. One of its comments had a long line. Added some
annotations to make clear what's going on in messier tests.

Change-Id: I06d8748a134591f93b36029713e52ffd826a24dc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-21 16:17:48 +01:00
Joerg Bornemann
238e3beb6f Remove unused .qrc files
Task-number: QTBUG-94446
Change-Id: I136d8b4ab070a832866aa50b5701fc6bd863df8a
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-01-17 23:17:01 +01:00
Edward Welbourne
13d2e13290 Make counts of various types of test result add up correctly
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test.  The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.

Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.

Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.

Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-13 19:47:22 +01:00
Dimitrios Apostolou
725fc4da44 Properly indent qtestlib's XML output
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.

Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-12-17 07:03:26 +01:00
Dimitrios Apostolou
df0351836e Fix helper script to run successfully on X11
Without the fix, some tests from tst_selftests were failing with:

Invalid MIT-MAGIC-COOKIE-1 key

Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-12-17 07:03:25 +01:00
Edward Welbourne
6729a4b29c TAP test logger: avoid dangling space on TODO and SKIP lines
If an incident has an empty description we produced output with a
dangling space. Avoid triggering the inanity 'bot with those.
Reorganise the code to save some duplication.

Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2021-12-09 20:54:26 +01:00