TestMethods::invokeTest() has an outer loop on global data (albeit
with a comment that said otherwise). On its first cycle, we run the
test function's *_data() method, if it has one; there is an inner loop
on the rows this created. If the *_data() QSKIP()s, we need to skip
the whole test; otherwise, a QSKIP() in one sub-test should not lead
to skipping the remaining sub-tests.
Moved the check for *_data() QSKIP()ping to right after *_data()
returns, inside the "first global cycle" block that runs it.
Previously, this check was done before entering the loop on local data
rows, but outside that "first global cycle" block: consequently, later
global cycles would fall foul of this check (even though the *_data()
hasn't been run in this cycle, much less QSKIP()ped in it) if the last
sub-test of the previous global cycle had QSKIP()ped.
When running a single test for one specific data row, if the test's
*_data() QSKIP()ped, this misplaced check would also have lead to a
misleading "Unknown testdata" warning.
Changed testlib/selftests' tst_globaldata::skipSingle() to trigger the
bug (by having its last local row of first global row skip, which
caused the second global row to be omitted) to verify this is also
fixed; and amended one of its comments to reflect what's now to be
expected. Updated the test's expected output files.
Task-number: QTBUG-61774
Change-Id: I99596b595c6d1184038f23383844c6ff51a0cd91
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The test was added as commented in 2012, before it was not even
mentioned in the pro file. I guess it was never run.
Change-Id: Ib787c7811ab4f7f32a99869167cde2e12dbe3156
In particular, document the mis-behavior it takes for granted in one
test. Renamed some variables so that they're visually distinct from
one another and actually mean something intelligible. Renamed the
sub-tests to say what's actually happening. Reading the test output
now at least makes it possible to see what's wrong. I'll fix that in
a later commit.
Task-number: QTBUG-61774
Change-Id: I3e5f83b8baa3c6afbca9231b5bbc89d17f3e57e2
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
It has been completely untested for a while and saw some breakage.
So let's add a selftest for it. ('-vs' when running tests)
Change-Id: Ibfb5ac0a2d741de7c3f519d91202d4977996045e
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The QTest::qCompare() implementations were almost duplicates; pull the
common code out into a templated version. Tweaked the
QTest::toString() specialization for float and double (a macro) and
fixed a bous modifier in double's format.
The doubleComparisons and floatComparisons tests in the tst_float.cpp
selftest shared a large block of tests in common, aside from the
difference of type. Break this out into a templated static function
to save duplication.
This prepares the way for using the same templated code for qfloat16.
Change-Id: I2823fd006910c5ff88335d625d1fa05cb7753513
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Conflicts:
tests/auto/widgets/itemviews/qheaderview/tst_qheaderview.cpp
Added tests/auto/testlib/selftests/expected_crashes_5.txt to work
round the output of the crashes test (which exercises UB, see
QTBUG-73903) being truncated on one test platform.
Change-Id: I9cd3f2639b4e50c3c4513e14629a40bdca8f8273
The silent and blacklisted selftests of testlib end in a qFatal(), to
test its messaging is handled correctly. However, this prevents hooks
in main() from saving coverage data when we're gathering that. So use
a transient signal handler that longjmp()s back to a setjmp() just
before the qFatal() to let the test complete normally (but, since
qFatal() does something different on MS-Win, don't apply this to it).
Note that testlib's internal FatalSignalHandler handles all fatal
signals *except* SIGABRT, so this isn't over-riding it. (In any case,
this restores the prior signal handler in setjmp()'s catch branch.)
Added missing expected_silent.tap test output while checking that this
change doesn't affect (the rest of) the test output.
Change-Id: I7e460581ad93e26639c066b3229438a66fd299de
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Tidied up the existing float tests in the process.
(In particular, s/SUCCESS/PASS/ since that matches real test output.)
These verify that QCOMPARE() handles floats and doubles as intended.
Extended the existing qFuzzyCompare tests to probe the boundaries of
the ranges of values of both types, in the process.
Revised the toString<double> that qCompare() uses to give enough
precision to actually show some of the differences being tested there
(12 digits, to match what qFuzzyCompare tests, so as to show different
values rather than, e.g. 1e12 for both expected and actual) and to
give consistent results for infinities and NaN (MinGW had eccentric
versions for these, leading to different output from tests, which thus
failed); did the latter also for toString<float> and fixed stray zeros
in MinGW's exponents (which made a kludge in tst_selftest.cpp
redundant, so I removed that, too).
That's further complicated handling of floating-point types, so let's
just keep an eye on how expensive that's getting by adding a benchmark
test for QTest::toString(). Unfortunately, default settings only get
runs that take modest numbers of milliseconds (some as low as 40)
while increasing this with -minumumvalue 100 or more gets the process
killed - and I'm unable to find out who's doing the killing (it's not
QProcess::kill, ::kill or the QtTest WatchDog, as far as I can tell).
So results are rather noisy; the integral tests exhibit speed-ups by
factors up to 5, and slow-downs by factors up to 100, between runs
with and without this change, which does not affec the integral tests.
The relatively modest slow-downs and speed-ups in the floating point
tests thus seem likely to be happenstance rather than signal.
Change-Id: I4a6bbbab6a43bf14a4089e96238a7c8da2c3127e
Reviewed-by: Ulf Hermann <ulf.hermann@qt.io>
Only for tests that have existing expected_*.* files for other
formats, though.
Change-Id: I34ca1900d88454f300e04d849a608c378009489b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Regexes have long specified that a [ as the first character inside a
[...] is just a literal [, but apparently we need to escape it now, to
avoid a "nested set" FutureWarning.
Change-Id: I76a48c9aafb0684a1d6b0d5284fe9852c9ea0e43
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Samuel Gaist <samuel.gaist@idiap.ch>
Prioritize blacklisting over QEXPECT_FAIL so that a test that is
blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To
reflect this state properly, the two values of BXPASS and BXFAIL were
added to testlib's output.
[ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests
will be taken into account for XPASS and XFAIL. A blacklisted test that
causes an XPASS will no longer be a fail.
Task-number: QTBUG-72928
Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Non-local ASM labels break for this test when compiled with clang.
Change-Id: I15bd250a991c3b03bbc88459a6358090bd157444
Reviewed-by: Jesus Fernandez <Jesus.Fernandez@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Based on Asmo Saarela's advice (QTPM-686), adapted on advice from
FrogLogic support and converted to a feature so that the selftest and
testlib qmake config can be co-ordinated.
Task-number: QTPM-1385
Change-Id: Icd706f086009e1e08b3f8c5cd553f792402e28c0
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
The output files for the tuplediagnostics selftest of testlib had a
stray non-canoical path fragment in them; so replaced with its
canonical form.
Change-Id: Ib421380036c3fb1b91447eb8c87be4ad0dfe5c96
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Verify that it does cut in after the specified time has elapsed.
Task-number: QTPM-1385
Change-Id: Ib18e8d6af28339f79cca4d62b869287ce07b8cc1
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Match the environment tst_selftests.cpp uses for subtests more
faithfully. Extends b22e50acda. In the process, tweak how crashers
are handling, in preparation for the watchdog test.
Change-Id: I09a046460f6f3bff0b12069fad6c1437d89572ce
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
One test for bad data for the column, another for a bad QFETCH.
Incidentally extend blacklist testing by blacklisting them.
Reorganise a QEMU condition that needed extended as part of this.
Task-number: QTPM-1385
Change-Id: Iac72ada19760321c5c9264ddfff7740d1fdd0700
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Some tests were fixed and others were skipped/blacklisted.
Task-number: QTBUG-63152
Change-Id: Ica7df555f8d152ee589865911130525101d4b941
Reviewed-by: Liang Qi <liang.qi@qt.io>
In the process, corrected an inaccurate XFAIL message (an XPASS is
normally an error, unless blacklisting ignores it so turns it into a
BPASS). Added the missing .tap file to its expected output.
Documented the similarity to the silent/ selftest.
Task-number: QTPM-1385
Change-Id: Id74a1353d54af2f3bfe2c764e33c1f051958ab21
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The testlib selftest sets various things in the environment for
crashing tests; the generator for its expected output should set the
same things, as they affect what output is produced.
Change-Id: Iec2ed59982ea1043582573530c33619d8e8ed08e
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
If the two lines have identical texts, the comparison returns true.
So don't complicate various other conditions on the way there with
filtering out that case; deal with it first so they don't need to.
Change-Id: Iebd230704ce5f53d12d5afa64aab30f83bb9d407
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This only enables compilation, it doesn't fix any test.
Qt on Android supports process, but not TEST_HELPER_INSTALLS. See also
acdd57cb for winrt.
android-ndk-r10e is used to compile, see
http://doc-snapshots.qt.io/qt5-5.11/androidgs.html .
corelib/io/{qdir,qresourceengine} need to be fixed later.
Done-with: Frederik Gladhorn <frederik.gladhorn@qt.io>
Done-with: Mårten Nordheim <marten.nordheim@qt.io>
Change-Id: I34b924c8ae5d46d6835b8f0a6606450920f4423b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Before the fix is applied this test fails because QFINDTESTDATA will
return "/usr/" instead of the folder with the same name in the current
directory.
The 'usr' folder can't be located 'next to' the application since this
does not trigger the issue (QFINDTESTDATA looks for the folder next
to the executable early on). So we put it in a subdirectory and change
the current working directory to its parent directory.
Change-Id: I627679dcb6f2f6954264e23bfc1a71de3bff7203
Reviewed-by: Jesus Fernandez <Jesus.Fernandez@qt.io>
Currently when doing comparison with std::tuple the fallback toString
method is called which returns a Q_NULLPTR thus not allowing proper
diagnostic of the values that triggered an error. This patch
adds support for std::tuple to improve the tests output readability.
[ChangeLog][QtTest][QCOMPARE] Now outputs contents of std::tuple on
failure.
Change-Id: I046a55e2ce44c3f7728d51e4745120d38aa5e007
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This supplements b1945604a7, which
removed the qrc file in favor of test/test.pro coding for it.
Change-Id: I15507c89ca14fa6e6b8223de671ffff7092272d0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Put it in alphabetical order like the rest of the list.
Change-Id: I3da3bb68d1847f53419bb79490b946c935ebb518
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
They didn't exist up until now, and future patches rely on them, so
add them.
Change-Id: I8afdb9417263b45d43355c688a813bdf99ea5fc8
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
When generate_expected_output.py is run for an in-source build, the
raw output contains no paths to the sources for the script to whittle
down, as it does for shadow builds, to just the path from qtbase down.
So kludge together some extra regexes that can fix that up and tweak
some relevant code to provide them with the data they need.
Change-Id: I656d7126087bd9ad20b2af6835fba314d90a171d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
This is what the other reporters also do, in various forms.
Task-number: QTBUG-67351
Change-Id: I16f2c4e0991176145ee0fbcbbfeeda071603a3c2
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Using QFileInfo to check if the file exists based on the filename
alone ignores the fact that all the expected files are embedded
as QRC resources.
The expectedResult() function already does a similar check, so we
can use that directly instead of checking twice if the file exists.
Task-number: QTBUG-66981
Change-Id: I0beb8d3503ed49682ae7d7e2a5172922fab5420d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Otherwise random debug messages from Qt might mess up the expected vs
actual results.
The setting of QT_LOGGING_RULES in initTestcase has been removed, as
the selftest overrides that for each invocation of a subtests, via
the processEnvironment() function.
Task-number: QTQAINFRA-1631
Change-Id: I855d31274f8261f8b125df23409353f7101be0e4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Instead of keeping a static list in a qrc file.
Change-Id: I7783db5cad5929cc3f449889e1cbcbb87f7f2637
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Silence debug output by setting QT_LOGGING_RULES
to turn off all debug output.
Task-number: QTQAINFRA-1631
Change-Id: I5c2366b4fe4bac341dcfd92f68b6da8071c5b089
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
After a37785ec76 went in, it become
apparent that multi mouse button state handling in qtestlib is
non-existent, for details see QTBUG-64030 and QTBUG-63786. What
happened behind the scenes often was not what one would expect based
on the provided QTest::mouse* input sequence - events went missing,
incorrect events were generated, each subsequent test function
started with a state set from the function that run earlier. It is
easy to see how a minor change in one test could easily affect outcome
of other tests.
With a37785ec76, Qt platform plugins
are now responsible for sending explicit mouse button type and state
information; qtestlib should take full responsibility now as well.
But using the new API from a37785ec7 alone in qtestlib is not sufficient.
We need to reset mouse state between each new test function run (we do
this at function scope as that fits with the current qtestlib API user
expectations). This patch implements the necessary reseting logic.
Updated tst_qwindow.cpp::generatedMouseMove() to use QTest::mouse* APIs.
That test requires pressing multiple buttons, it was not possible with
QTest::mouse* APIs before this patch.
Added an auto test for multiple mouse button pressing/release in
tests/auto/testlib/selftests/mouse/. And few other tests which are
currently QSKIP-ed, but should be considered when re-designing qtestlib
APIs.
Task-number: QTBUG-64030
Change-Id: I39fdcbc73a467a7463ce2aed622bf22484095635
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Add formatting for registered enumerations based on QMetaEnum
and unregistered enumerations as hex values.
[ChangeLog][QtTest] QtTest now prints values of QFlags that failed to
compare with QCOMPARE.
Task-number: QTBUG-65845
Change-Id: I3eae6d20d3c0d72441ca6c4037d9a8dafa4b6357
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
