Repeated test execution can be useful, under a debugger, to catch an
intermittent failure or, under memory instrumentation, to make memory
leaks easier to recognize.
The new -repeat flag allows running the entire test suite multiple times
within the same process. It works by executing all tests sequentially
before repeating the execution again.
This switch is a developer tool, and is not intended for CI. It can only
be used with the plain text logger.
Change-Id: I2439462c5c44d1c8aa3d3b5656de3eef44898c68
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
QPalette specifically has quite a large amount of output (1363
characters) when toString
is called on it. We should make sure that we can fit that in our
failure messages. This patch does that by increasing the limit from
1024 characters to 4096.
Fixes: QTBUG-5903
Fixes: QTBUG-87039
Pick-to: 6.5 6.6
Change-Id: I1dc5078ad05858bb6542c3a06c6b84711af79e4f
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
It deoesn't like single quotes, which must be replaced with doubled-up
double quotes.
Pick-to: 6.2 6.5 6.6
Change-Id: I8e5f8047e72e4433926a9f4f2044407a4d823682
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Ensure that this environment variable is respected, as ASAN errors
(e.g. memory leaks) can cause the selftests to fail since they expect
no errors in their output.
Fix an incorrect function name in a comment, while we're at it.
Fixes: QTBUG-118041
Pick-to: 6.5 6.6
Change-Id: I07502101aabd8743df898ae8fe4a693c4733c4af
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
rowsAboutToBeMoved() and rowsMoved() signals aren't emitted for
QSortFilterProxyModel, which meant the two connections in the
ObservingObject's constructor didn't trigger the slots, which let the
test pass (the store/checkPersistentFailureCount stayed at 0).
- Instead connect to layoutAboutToBeChanged() and layoutChanged()
respectively, these two are emitted for QSFPM
- Use PMF syntax
- Verify m_persistent{Proxy,Source}Indexes aren't empty
Change-Id: I8b83989de02c2bfb22bde9b230cb5b68814f74b6
Reviewed-by: David Faure <david.faure@kdab.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
This allows us to control whether to run this particular test in ASan
mode or with specific loggers or not. Adding the expected log output for
tst_silent for other loggers is left as an exercise to the reader.
Change-Id: Ifa1111900d6945ea8e05fffd177f1548c8c8e714
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This reverts commit f20adcde30. The
implementation had the right idea, but this is not expected to work
reliably in C++. It's jumping out of several frames without cleaning
them out properly and our ASan-based memory leak-checker has started
complaining (the next commit will move this test elsewhere).
==19313==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 258 byte(s) in 1 object(s) allocated from:
#0 0x7ffa505c8e48 in __interceptor_malloc (/usr/lib64/libasan.so.5+0x109e48)
#1 0x7ffa4f2d7ff9 (/home/qt/work/install/lib/libQt6Core.so.6+0x896ff9)
#2 0x7ffa4f2d834d in QArrayData::allocate(QArrayData**, long long, long long, long long, QArrayData::AllocationOption) (/home/qt/work/install/lib/libQt6Core.so.6+0x89734d)
#3 0x7ffa4f23b700 (/home/qt/work/install/lib/libQt6Core.so.6+0x7fa700)
#4 0x7ffa4f1f6cc8 in QString::reallocData(long long, QArrayData::AllocationOption) (/home/qt/work/install/lib/libQt6Core.so.6+0x7b5cc8)
#5 0x7ffa4f1f68a7 in QString::resize(long long) (/home/qt/work/install/lib/libQt6Core.so.6+0x7b58a7)
#6 0x7ffa4f2092ff (/home/qt/work/install/lib/libQt6Core.so.6+0x7c82ff)
#7 0x7ffa4f209e09 in QString::vasprintf(char const*, __va_list_tag*) (/home/qt/work/install/lib/libQt6Core.so.6+0x7c8e09)
#8 0x7ffa4ed0d83d (/home/qt/work/install/lib/libQt6Core.so.6+0x2cc83d)
#9 0x7ffa4ed114a9 in QMessageLogger::fatal(char const*, ...) const (/home/qt/work/install/lib/libQt6Core.so.6+0x2d04a9)
#10 0x5641d2604c40 in tst_Silent::messages() /home/qt/work/qt/qtbase/tests/auto/testlib/selftests/silent/tst_silent.cpp:77
#11 0x5641d26050fb in tst_Silent::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) tests/auto/testlib/selftests/silent/silent_autogen/include/tst_silent.moc:118
The restoration of the signal handler (which QtTest now has) is also
wrong: this needed to use sigaction() instead.
Change-Id: Ifa1111900d6945ea8e05fffd177f14fbc09a1d7d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This functionality was lost when we switched to Catch2.
Change-Id: I2b24e1d3cad44897906efffd177fb4cff641546f
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Amends 118f2210c6. That commit added
#ifdef Q_OS_UNIX
if (test == "assert"
|| test == "crashes"
|| test == "failfetchtype"
|| test == "faildatatype")
return; // Outputs "Received signal 6 (SIGABRT)"
#endif
Which duplicated 4 out of the 5 tests in the block:
#ifdef Q_OS_LINUX
// QEMU outputs to stderr about uncaught signals
if (QTestPrivate::isRunningArmOnX86() &&
(test == "assert"
|| test == "crashes"
|| test == "faildatatype"
|| test == "failfetchtype"
|| test == "silent"
))
return;
#endif
But as Linux is Unix, we never got to that second block for those 4
tests.
Pick-to: 6.6
Change-Id: I2b24e1d3cad44897906efffd177fb4b5507d190a
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The loop was iterating over a temporary container, so it couldn't have
changed it. Use a const auto variable to hold the container and port to
ranged-for
Task-number: QTBUG-115839
Change-Id: I66e7cbdb811666ca352cdf064b1228caa346d876
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
badString(): const'ify the static QList, this is both faster as the
compiler doesn't need to check if it has been already initialized, and
it means we can use it directly in ranged-for as the method returns
const QList&.
Drive-by change: don't go the long way around to get a const char*:
qPrintable(QString("fail %1").arg(ba))
instead use:
QTest::addRow("fail %s", ba.constData())
(thanks to dfaure for pointing it out in review).
Task-number: QTBUG-115839
Change-Id: I6e8efa6df47ee94f1d71a63e22ab121647e6bf20
Reviewed-by: David Faure <david.faure@kdab.com>
The density of Q_FOREACH uses in this and some other modules is still
extremely high, too high for anyone to tackle in a short amount of
time. Even if they're not concentrated in just a few TUs, we need to
make progress on a global QT_NO_FOREACH default, so grab the nettle
and stick to our strategy:
Mark the whole of Qt with QT_NO_FOREACH, to prevent new uses from
creeping in, and whitelist the affected TUs by #undef'ing
QT_NO_FOREACH locally, at the top of each file. For TUs that are part
of a larger executable, this requires these files to be compiled
separately, so add them to NO_PCH_SOURCES (which implies
NO_UNITY_BUILD_SOURCES, too).
In tst_qglobal.cpp and tst_qcollections.cpp change the comment on the
#undef QT_NO_FOREACH to indicate that these actually test the macro.
Task-number: QTBUG-115839
Change-Id: Iecc444eb7d43d7e4d037f6e155abe0e14a00a5d6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The blacklisted test does not crash, as its expected output indeed shows.
Pick-to: 6.5 6.6
Change-Id: I07522a7d065b5f39620975a3546bcd156024c41d
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
In porting the selftest machinery to Catch2 in 24e83de8d1 we
accidentally added an unconditional early return when determining
whether to check for unexpected stderr output, resulting in not
checking error output on any platform.
The return statement has now been moved into the Q_CC_MINGW
condition, but as we now seem to have similar issues on macOS
and Linux with some of the tests outputting "Received signal 6
(SIGABRT)" as we do for QEMU, we need to add a few more explicit
early return conditions to the function.
Pick-to: 6.5 6.6
Change-Id: I7a25f000843b5f1003a5db883f08285185046b46
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
QThreeWidgetItems that have been removed from their parents (or the
widget) must be deleted manually. The treeWidgetModel() test forgot
that, driving asan nuts.
Code predates the beginning of the public history, so picking to all
active branches.
Pick-to: 6.6 6.5 6.2 5.15
Change-Id: I139549b0bd8baf4abfb90f926f6290119471046f
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
a611a9f537 (in Qt 5) added support for
mixed-type comparisons through QCOMPARE. That commit added a new
overload for qCompare taking two types, T1 and T2; but it also left the
same-type qCompare(T, T) overload around, guarded by a Qt 6 version
check.
The mixed-type version is however not a generalization of the same-type
one, because it won't work if one of the arguments doesn't participate
in FTAD. Case in point: braced-init-lists. In Qt 5 this worked:
QCOMPARE(some_container, {42});
but in Qt 6 it does not work any more. The mixed-type overload fails
deduction (can't deduce T2); in Qt 5 the same-type overload deduced
T=SomeContainer, and {42} was used to select a constructor for
SomeContainer.
--
There's a partial, straightforward workaround for this: default T2 to
T1 in the mized-type overload. In that case T2 has a "fallback" if it
cannot be deduced. This is partial because of course doesn't address
the case in which T1 cannot be deduced, but I don't think that is
common at all.
QList is special here, because it has qCompare overloads that makes it
comparable with arrays, initializer lists and more. I don't like that
very much -- we should probably have a qCompare(input_range,
input_range) overload, but that's an exercise for C++20.
Change-Id: I344ba33167829984978cd8d649a1904349a9edab
Pick-to: 6.5 6.6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Milian Wolff <milian.wolff@kdab.com>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Add the boilerplate standalone test prelude to each test, so that they
can be opened with an IDE without the qt-cmake-standalone-test script,
but directly with qt-cmake or cmake.
Boilerplate was added using the following scripts:
https://git.qt.io/alcroito/cmake_refactor
Manual adjustments were made where the code was inserted in the wrong
location.
Task-number: QTBUG-93020
Change-Id: I77299f990692b4fe4721a9bc35071608d0d23982
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Amir Masoud Abdol <amir.abdol@qt.io>
Using chrono means one can write 10s instead of 10'000.
[ChangeLog][QTest] Added qSleep(std::chrono::milliseconds) overload.
Change-Id: Iac1b12a3fc3f692b557e2d459e6f3bc565f20e93
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
As requested in code review. Big improvement, code-wise.
tst_Sleep::wait() was failing on the CI, so be more accurate by using
QElapsedTimer::durationElapsed(), which returns nanoseconds.
Pick-to: 6.6
Change-Id: I5bed6d6bd768adfdecab2475e6cbe245c20aabd7
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
[ChangeLog][QtCore][QDebug] Added pretty formatting of C++ <chrono>
durations.
[ChangeLog][QtTest] Added pretty formatting of C++ <chrono> durations
for QCOMPARE expressions.
Change-Id: I3b169860d8bd41e9be6bfffd1757cc087ba957fa
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
The test is accessing an out of bounds index into an array on purpose.
Change-Id: I23cce4093fdfabd177eb34639f26fcd6752c8b34
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
It's unlikely we will ever use pro2cmake at this project stage,
so it doesn't make any sense to keep the 'special case' markers
in the CMake scripts. Remove them and replace with TODO where
needed.
Change-Id: I84290c20679dabbfdec3c5937ce0428fecb3e5a7
Reviewed-by: Amir Masoud Abdol <amir.abdol@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Change-Id: I3087bff60e77c0eaa5818a5405c97628ce7f6b79
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The parameter names were previously "lhs" and "rhs", which is incredibly
abstract for a testing framework. One of the parameters will tend to be
a baseline value to compare against while the other is the result of
some action we want to test. Thus I suggest they be renamed "computed"
and "baseline".
This way we can, hopefully, retain the semantic that the 'left'/first
argument is the computed ('actual' in QCOMPARE) value while the
'right'/second argument is the baseline ('expected' in QCOMPARE.)
Change-Id: I3e0fdce2a3f1faca06fdf7184ef6e0eb9724d990
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
In the process actually handle all time-spec cases in various places
that only handled UTC or LocalTime, or at least note that they don't
where that's not practical. Also tidy up header ordering and ensure
QDateTime's header is included wherever it's used, while adding the
include for QTimeZone where needed.
Task-number: QTBUG-108199
Change-Id: Ic1a5cdf0aaf737bf1396aa8ac58ce2004cef7e19
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
This is a semantic patch using ClangTidyTransformator as in
qtbase/df9d882d41b741fef7c5beeddb0abe9d904443d8, but extended to
handle typedefs and accesses through pointers, too:
const std::string o = "object";
auto hasTypeIgnoringPointer = [](auto type) { return anyOf(hasType(type), hasType(pointsTo(type))); };
auto derivedFromAnyOfClasses = [&](ArrayRef<StringRef> classes) {
auto exprOfDeclaredType = [&](auto decl) {
return expr(hasTypeIgnoringPointer(hasUnqualifiedDesugaredType(recordType(hasDeclaration(decl))))).bind(o);
};
return exprOfDeclaredType(cxxRecordDecl(isSameOrDerivedFrom(hasAnyName(classes))));
};
auto renameMethod = [&] (ArrayRef<StringRef> classes,
StringRef from, StringRef to) {
return makeRule(cxxMemberCallExpr(on(derivedFromAnyOfClasses(classes)),
callee(cxxMethodDecl(hasName(from), parameterCountIs(0)))),
changeTo(cat(access(o, cat(to)), "()")),
cat("use '", to, "' instead of '", from, "'"));
};
renameMethod(<classes>, "count", "size");
renameMethod(<classes>, "length", "size");
except that the on() matcher has been replaced by one that doesn't
ignoreParens().
a.k.a qt-port-to-std-compatible-api V5 with config Scope: 'Container'.
Added two NOLINTNEXTLINEs in tst_qbitarray and tst_qcontiguouscache,
to avoid porting calls that explicitly test count().
Change-Id: Icfb8808c2ff4a30187e9935a51cad26987451c22
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
The timestamp will no longer be incremented by 500ms after a mouse
release if the delay has been explicitly specified.
The default delay is 1 ms since f5010c49a3
but the running timestamp was unconditionally post-incremented by 500ms
after every mouse release, to prevent double-clicks, which were always
deemed as unintended (because we have a mouseDClick function for that).
Now, we do that 500ms increment only if the user has not provided a
delay value in the function argument at all. We have often found it
useful in our own tests to generate double-clicks "the hard way", by
sending indivdual events, so as to be able to check state in some target
object at each step, as shown in the new snippet.
[ChangeLog][QtTest] QTest::mouseRelease() and mouseClick() can now be
used to test double-clicks, by specifying a realistic timestamp delay.
Fixes: QTBUG-102441
Change-Id: I8e8d242061f79efb4c6e02638645e03661a9cd92
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
Unix systems have got crash loggers in the past 15-20 years, notably
macOS and Linux (abrtd, systemd-coredumpd, etc.). By setting the core
dump limit to zero, those tools should be mostly inhibited from running
and thus not interfere with the parent process' timeouts. Even for
systems without core dump loggers, disabling the writing of a core dump
to the filesystem should also help.
Pick-to: 6.4
Change-Id: I12a088d1ae424825abd3fffd171d112d0671effe
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
QWindow::requestActivate() is not supported.
We have one tst_selftests binary, and will test it with both xcb and
wayland qpa plugin. A runtime check and skip will have different
restult files, which is not implemented in testlib yet.
Task-number: QTBUG-107578
Pick-to: 6.4 6.2
Change-Id: Idc8cb24c6f42a9f0f4dc9493e3fd1a5803ba7ce0
Reviewed-by: Liang Qi <liang.qi@qt.io>
The QTRY_* macros and QTestEventLoop exit early if the test has
resolved; however, in the cleanup phase of a test, even if the test
has failed, these loops should continue as normal.
[ChangeLog][QtTest] During the cleanup() phase of a test, the QTRY_*
macros and QTestEventLoop now ignore the test resolution, in contrast
to when they are used from the test itself, which (since 6.3.0) exits
the loops early if the test has failed.
Pick-to: 6.4 6.3
Fixes: QTBUG-104441
Change-Id: I2673161967cbbc57815155af698a9338ab98a686
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The regex used a greedy .+ to match an actual dot and a space,
preceding an open parenthesis, with the result that if the message
contained any parentheses that .+ swallowed everything up to the last
of them. Correct the regex so that recently-added tests' error
messages show up correctly.
Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The recently added test for QTestEventLoop produced a message which
the TAP test-logger mis-parses. Since that message shall soon go away
(when I fix the bug the new test exists to verify), modify one of
cmptest's QVERIFY2() messages to trip up the same bug, so we can spot
any regressions on this in future (after the imminent fix to this TAP
issue).
Change-Id: Ibbe9931c01c75df529c9571e2bbdbd34010dd9ec
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Use the standard mechanism instead of a hand-rolled way of identifying
function/data-tag combinations. Adds missing data relevant to
identifying test cases, some of which previously had apparently the
same name, with no hint at why the test-case was seemingly run
repeatedly.
Change-Id: I6225c6d1990069c94a1f1c8dbb179993b96076e7
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
As discovered in the expanded testing of QTRY_COMPARE() using the same
class, the timer needs a context object and a slot to call.
This amends commit 35ad157d88
Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I41fc23de84ce8c7d6608db0005276a2071974494
Reviewed-by: Jason McDonald <macadder1@gmail.com>
In the process, simplify the latter while adding some actual
time-variation for the QTRY_* loop to navigate round - based on the
extendedcompare test's ClassWithDeferredSetter. Testing remains
primitive, but is at least a bit more thorough.
Pick-to: 6.4
Change-Id: I40be8fb485f3f18f0a4f4bc62ad36cccac691979
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
These contain output only ever sent to stderr, which tst_selftests no
longer looks at; it merely verifies stderr is empty for all tests
except those expected to produce stderr output, and ignores the stderr
output for these last.
Change-Id: I50fee445d84c2d125e6db5303fc389e87746455e
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Fairly minimal for now, just enough to verify a bug and serve as the
sign of success when it's fixed. Tests fail in ways they shouldn't,
for now; see expected_eventloop.* for details, notably "Earlier test
failed to clean up" messages.
Pick-to: 6.4 6.3
Task-number: QTBUG-104441
Change-Id: I59be4aa5f21fed23b19a0593a8c2f6c9956507df
Reviewed-by: Jason McDonald <macadder1@gmail.com>
CMakeLists.txt and .cmake files of significant size
(more than 2 lines according to our check in tst_license.pl)
now have the copyright and license header.
Existing copyright statements remain intact
Task-number: QTBUG-88621
Change-Id: I3b98cdc55ead806ec81ce09af9271f9b95af97fa
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
