The test orchestrator shouldn't have to deal with the individual options
needed for each test.
Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Change-Id: I2fd7a39684bde44d82c4d877086f606413d68520
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
These tests are apparently not run at the moment, otherwise they would
have failed to even start, but they should definitely not have the tst
prefix.
Change-Id: Iafcec2764ebb3570e6bc6ebfba27d92a94639893
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
If we had one test function that just did
tst_Mouse::f1()
{
QTest::mouseMove(w, QPoint(0,0));
}
and another test function that did
tst_Mouse::f2()
{
QTest::mouseMove(w, QPoint(500,500));
}
their corresponding event timestamps were only 1 apart from each other.
This meant that any code that tried to estimate the velocity of a mouse
cursor would get a really high velocity estimate inside f2(). This would
come as a surprise to most people. So to avoid this, we add a 500 ms
timestamp delay between each test function call.
In theory this could also prevent generating a mouseDoubleClickEvent
when a pair of test functions containing a press-release sequence was
run, but there is a separate pre-existing mechanism to handle that case.
Change-Id: Icd4fc35853c09f080466d22411208c7b5c4174b5
Reviewed-by: Shawn Rutledge <shawn.rutledge@qt.io>
The reporter was probably named 'xunit' based on the historical use of
xUnit to refer to testing frameworks derived from Smalltalk's SUnit.
These frameworks typically added their own prefix, e.g. JUnit for Java,
RUnit for R, etc.
The most popular of these was the JUnit framework, and the corresponding
XML output produced by the Ant built tool became somewhat of a de facto
standard, which is probably why we chose to model our reporter after it.
Nowadays however, naming it 'xunit' is problematic as there is actually
a testing famework named xUnit.net, typically shortened to, you guessed
it: xunit.
Test report consumers will typically have a junit mode, and an xunit
mode, and the latter could easily be mistaken for what testlib outputs,
unless we clarify this.
The clarification also allows us to safely extend our support for the
JUnit XML format to incorporate some elements that are nowadays common,
but where we are lagging behind the standard.
[ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has
been renamed to what it actually is: a JUnit test reporter, and is now
triggered by passing -o junitxml to the test binary.
Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
This was used to support QFlags f = 0 initialization, but with 0 used
as a pointer literal now considered bad form, it had been changed many
places to QFlags f = nullptr, which is meaningless and confusing.
Change-Id: I4bc592151c255dc5cab1a232615caecc520f02e8
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Testlib's signaldumper functionality would crash inside
testlib as it dereferenced the sender after it was deleted.
Change-Id: I6013b75b0a121e2768429d8a3cf0339a940314f2
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
The test is not failing anymore on QEMU targets.
This partially reverts commit
71bd06d516.
Fixes: QTBUG-71915
Change-Id: I68593edf0ec245e14879833c8aa90661a3c2e227
Reviewed-by: Liang Qi <liang.qi@qt.io>
When accidentally running a test doing screen-grabbing
with High DPI scaling active, sizes of the obtained pixmaps
can differ due to the device pixel ratio. Add a check to make that
clearer.
[ChangeLog][QtTestLib] Comparison of QImage, QPixmap now checks for the
device pixel ratio.
Change-Id: Id8d5187e99c565c44a7bfb8b9cfb09737815fb15
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Conflicts:
src/corelib/tools/qlocale_data_p.h
(Regenerated by running the scripts in util/local_database/)
src/gui/opengl/qopengltextureuploader.cpp
Done-With: Edward Welbourne <edward.welbourne@qt.io>
Done-With: Allan Sandfeld Jensen <allan.jensen@qt.io>
Change-Id: I12df7f066ed0a25eb109f61c4b8d8dea63b683e2
If the deadline is far in the future, the conversions to nanoseconds
or internal arithmetic may overflow and give an invalid object, thus
the deadline may end up in the past. Added a test to the testlib
selftest for sleep.
[ChangeLog][QtCore][QDeadlineTimer] Fixed integer overflows
leading to immediate timeouts.
Task-number: QTBUG-69750
Change-Id: I9814eccdf9f9b3add9ca66ec3e27e10cd5ad54a8
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Tidied up the existing float tests in the process.
(In particular, s/SUCCESS/PASS/ since that matches real test output.)
These verify that QCOMPARE() handles floats and doubles as intended.
Extended the existing qFuzzyCompare tests to probe the boundaries of
the ranges of values of both types, in the process.
Revised the toString<double> that qCompare() uses to give enough
precision to actually show some of the differences being tested there
(12 digits, to match what qFuzzyCompare tests, so as to show different
values rather than, e.g. 1e12 for both expected and actual) and to
give consistent results for infinities and NaN (MinGW had eccentric
versions for these, leading to different output from tests, which thus
failed); did the latter also for toString<float> and fixed stray zeros
in MinGW's exponents (which made a kludge in tst_selftest.cpp
redundant, so I removed that, too).
That's further complicated handling of floating-point types, so let's
just keep an eye on how expensive that's getting by adding a benchmark
test for QTest::toString(). Unfortunately, default settings only get
runs that take modest numbers of milliseconds (some as low as 40)
while increasing this with -minumumvalue 100 or more gets the process
killed - and I'm unable to find out who's doing the killing (it's not
QProcess::kill, ::kill or the QtTest WatchDog, as far as I can tell).
So results are rather noisy; the integral tests exhibit speed-ups by
factors up to 5, and slow-downs by factors up to 100, between runs
with and without this change, which does not affec the integral tests.
The relatively modest slow-downs and speed-ups in the floating point
tests thus seem likely to be happenstance rather than signal.
Change-Id: I4a6bbbab6a43bf14a4089e96238a7c8da2c3127e
Reviewed-by: Ulf Hermann <ulf.hermann@qt.io>
Only for tests that have existing expected_*.* files for other
formats, though.
Change-Id: I34ca1900d88454f300e04d849a608c378009489b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Regexes have long specified that a [ as the first character inside a
[...] is just a literal [, but apparently we need to escape it now, to
avoid a "nested set" FutureWarning.
Change-Id: I76a48c9aafb0684a1d6b0d5284fe9852c9ea0e43
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Samuel Gaist <samuel.gaist@idiap.ch>
Prioritize blacklisting over QEXPECT_FAIL so that a test that is
blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To
reflect this state properly, the two values of BXPASS and BXFAIL were
added to testlib's output.
[ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests
will be taken into account for XPASS and XFAIL. A blacklisted test that
causes an XPASS will no longer be a fail.
Task-number: QTBUG-72928
Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Non-local ASM labels break for this test when compiled with clang.
Change-Id: I15bd250a991c3b03bbc88459a6358090bd157444
Reviewed-by: Jesus Fernandez <Jesus.Fernandez@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Based on Asmo Saarela's advice (QTPM-686), adapted on advice from
FrogLogic support and converted to a feature so that the selftest and
testlib qmake config can be co-ordinated.
Task-number: QTPM-1385
Change-Id: Icd706f086009e1e08b3f8c5cd553f792402e28c0
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
The output files for the tuplediagnostics selftest of testlib had a
stray non-canoical path fragment in them; so replaced with its
canonical form.
Change-Id: Ib421380036c3fb1b91447eb8c87be4ad0dfe5c96
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Verify that it does cut in after the specified time has elapsed.
Task-number: QTPM-1385
Change-Id: Ib18e8d6af28339f79cca4d62b869287ce07b8cc1
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Match the environment tst_selftests.cpp uses for subtests more
faithfully. Extends b22e50acda. In the process, tweak how crashers
are handling, in preparation for the watchdog test.
Change-Id: I09a046460f6f3bff0b12069fad6c1437d89572ce
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
One test for bad data for the column, another for a bad QFETCH.
Incidentally extend blacklist testing by blacklisting them.
Reorganise a QEMU condition that needed extended as part of this.
Task-number: QTPM-1385
Change-Id: Iac72ada19760321c5c9264ddfff7740d1fdd0700
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Some tests were fixed and others were skipped/blacklisted.
Task-number: QTBUG-63152
Change-Id: Ica7df555f8d152ee589865911130525101d4b941
Reviewed-by: Liang Qi <liang.qi@qt.io>
In the process, corrected an inaccurate XFAIL message (an XPASS is
normally an error, unless blacklisting ignores it so turns it into a
BPASS). Added the missing .tap file to its expected output.
Documented the similarity to the silent/ selftest.
Task-number: QTPM-1385
Change-Id: Id74a1353d54af2f3bfe2c764e33c1f051958ab21
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The testlib selftest sets various things in the environment for
crashing tests; the generator for its expected output should set the
same things, as they affect what output is produced.
Change-Id: Iec2ed59982ea1043582573530c33619d8e8ed08e
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
If the two lines have identical texts, the comparison returns true.
So don't complicate various other conditions on the way there with
filtering out that case; deal with it first so they don't need to.
Change-Id: Iebd230704ce5f53d12d5afa64aab30f83bb9d407
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This only enables compilation, it doesn't fix any test.
Qt on Android supports process, but not TEST_HELPER_INSTALLS. See also
acdd57cb for winrt.
android-ndk-r10e is used to compile, see
http://doc-snapshots.qt.io/qt5-5.11/androidgs.html .
corelib/io/{qdir,qresourceengine} need to be fixed later.
Done-with: Frederik Gladhorn <frederik.gladhorn@qt.io>
Done-with: Mårten Nordheim <marten.nordheim@qt.io>
Change-Id: I34b924c8ae5d46d6835b8f0a6606450920f4423b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Before the fix is applied this test fails because QFINDTESTDATA will
return "/usr/" instead of the folder with the same name in the current
directory.
The 'usr' folder can't be located 'next to' the application since this
does not trigger the issue (QFINDTESTDATA looks for the folder next
to the executable early on). So we put it in a subdirectory and change
the current working directory to its parent directory.
Change-Id: I627679dcb6f2f6954264e23bfc1a71de3bff7203
Reviewed-by: Jesus Fernandez <Jesus.Fernandez@qt.io>
Currently when doing comparison with std::tuple the fallback toString
method is called which returns a Q_NULLPTR thus not allowing proper
diagnostic of the values that triggered an error. This patch
adds support for std::tuple to improve the tests output readability.
[ChangeLog][QtTest][QCOMPARE] Now outputs contents of std::tuple on
failure.
Change-Id: I046a55e2ce44c3f7728d51e4745120d38aa5e007
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This supplements b1945604a7, which
removed the qrc file in favor of test/test.pro coding for it.
Change-Id: I15507c89ca14fa6e6b8223de671ffff7092272d0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Put it in alphabetical order like the rest of the list.
Change-Id: I3da3bb68d1847f53419bb79490b946c935ebb518
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
They didn't exist up until now, and future patches rely on them, so
add them.
Change-Id: I8afdb9417263b45d43355c688a813bdf99ea5fc8
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
When generate_expected_output.py is run for an in-source build, the
raw output contains no paths to the sources for the script to whittle
down, as it does for shadow builds, to just the path from qtbase down.
So kludge together some extra regexes that can fix that up and tweak
some relevant code to provide them with the data they need.
Change-Id: I656d7126087bd9ad20b2af6835fba314d90a171d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
This is what the other reporters also do, in various forms.
Task-number: QTBUG-67351
Change-Id: I16f2c4e0991176145ee0fbcbbfeeda071603a3c2
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Using QFileInfo to check if the file exists based on the filename
alone ignores the fact that all the expected files are embedded
as QRC resources.
The expectedResult() function already does a similar check, so we
can use that directly instead of checking twice if the file exists.
Task-number: QTBUG-66981
Change-Id: I0beb8d3503ed49682ae7d7e2a5172922fab5420d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
