Match the environment tst_selftests.cpp uses for subtests more
faithfully. Extends b22e50acda. In the process, tweak how crashers
are handling, in preparation for the watchdog test.
Change-Id: I09a046460f6f3bff0b12069fad6c1437d89572ce
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
One test for bad data for the column, another for a bad QFETCH.
Incidentally extend blacklist testing by blacklisting them.
Reorganise a QEMU condition that needed extended as part of this.
Task-number: QTPM-1385
Change-Id: Iac72ada19760321c5c9264ddfff7740d1fdd0700
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Some tests were fixed and others were skipped/blacklisted.
Task-number: QTBUG-63152
Change-Id: Ica7df555f8d152ee589865911130525101d4b941
Reviewed-by: Liang Qi <liang.qi@qt.io>
QTreeWidget::(is|set)Item(Selected|Hidden|Expanded)() are deprecated
for a long time but not marked as such. Therefore explicitly mark them
as deprecated so they can get removed with Qt6.
Change-Id: Ie4971350de61326811e0788df0d359ed3c442869
Reviewed-by: Konstantin Shegunov <kshegunov@gmail.com>
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
It's not actually exercised by anything in our source tree, but is
potentially useful and has been part of the documented public API for
some time. So mention that the caller is responsible for delete[]ing
its return and add a test that exercises it.
Task-number: QTPM-1385
Change-Id: Ifc5284b9eb1b678cf3c9708c681311e874838fc6
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
QTreeView/TableView::sortByColumn(int) was deprecated a long time ago
but never got removed. Therefore mark it with QT_DEPRECATED_SINCE(5, 13)
so we can remove it with Qt6.
Also sync the handling of the sort order changes in QTableView with the
one from QTreeView.
Change-Id: I0371d9a9c21116edaa9125835827f1a200075d36
Reviewed-by: Luca Beldi <v.ronin@yahoo.it>
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
In the process, corrected an inaccurate XFAIL message (an XPASS is
normally an error, unless blacklisting ignores it so turns it into a
BPASS). Added the missing .tap file to its expected output.
Documented the similarity to the silent/ selftest.
Task-number: QTPM-1385
Change-Id: Id74a1353d54af2f3bfe2c764e33c1f051958ab21
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The testlib selftest sets various things in the environment for
crashing tests; the generator for its expected output should set the
same things, as they affect what output is produced.
Change-Id: Iec2ed59982ea1043582573530c33619d8e8ed08e
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
If the two lines have identical texts, the comparison returns true.
So don't complicate various other conditions on the way there with
filtering out that case; deal with it first so they don't need to.
Change-Id: Iebd230704ce5f53d12d5afa64aab30f83bb9d407
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This only enables compilation, it doesn't fix any test.
Qt on Android supports process, but not TEST_HELPER_INSTALLS. See also
acdd57cb for winrt.
android-ndk-r10e is used to compile, see
http://doc-snapshots.qt.io/qt5-5.11/androidgs.html .
corelib/io/{qdir,qresourceengine} need to be fixed later.
Done-with: Frederik Gladhorn <frederik.gladhorn@qt.io>
Done-with: Mårten Nordheim <marten.nordheim@qt.io>
Change-Id: I34b924c8ae5d46d6835b8f0a6606450920f4423b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Before the fix is applied this test fails because QFINDTESTDATA will
return "/usr/" instead of the folder with the same name in the current
directory.
The 'usr' folder can't be located 'next to' the application since this
does not trigger the issue (QFINDTESTDATA looks for the folder next
to the executable early on). So we put it in a subdirectory and change
the current working directory to its parent directory.
Change-Id: I627679dcb6f2f6954264e23bfc1a71de3bff7203
Reviewed-by: Jesus Fernandez <Jesus.Fernandez@qt.io>
Currently when doing comparison with std::tuple the fallback toString
method is called which returns a Q_NULLPTR thus not allowing proper
diagnostic of the values that triggered an error. This patch
adds support for std::tuple to improve the tests output readability.
[ChangeLog][QtTest][QCOMPARE] Now outputs contents of std::tuple on
failure.
Change-Id: I046a55e2ce44c3f7728d51e4745120d38aa5e007
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This supplements b1945604a7, which
removed the qrc file in favor of test/test.pro coding for it.
Change-Id: I15507c89ca14fa6e6b8223de671ffff7092272d0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Put it in alphabetical order like the rest of the list.
Change-Id: I3da3bb68d1847f53419bb79490b946c935ebb518
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
They didn't exist up until now, and future patches rely on them, so
add them.
Change-Id: I8afdb9417263b45d43355c688a813bdf99ea5fc8
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
When generate_expected_output.py is run for an in-source build, the
raw output contains no paths to the sources for the script to whittle
down, as it does for shadow builds, to just the path from qtbase down.
So kludge together some extra regexes that can fix that up and tweak
some relevant code to provide them with the data they need.
Change-Id: I656d7126087bd9ad20b2af6835fba314d90a171d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
This is what the other reporters also do, in various forms.
Task-number: QTBUG-67351
Change-Id: I16f2c4e0991176145ee0fbcbbfeeda071603a3c2
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Using QFileInfo to check if the file exists based on the filename
alone ignores the fact that all the expected files are embedded
as QRC resources.
The expectedResult() function already does a similar check, so we
can use that directly instead of checking twice if the file exists.
Task-number: QTBUG-66981
Change-Id: I0beb8d3503ed49682ae7d7e2a5172922fab5420d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Otherwise random debug messages from Qt might mess up the expected vs
actual results.
The setting of QT_LOGGING_RULES in initTestcase has been removed, as
the selftest overrides that for each invocation of a subtests, via
the processEnvironment() function.
Task-number: QTQAINFRA-1631
Change-Id: I855d31274f8261f8b125df23409353f7101be0e4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Instead of keeping a static list in a qrc file.
Change-Id: I7783db5cad5929cc3f449889e1cbcbb87f7f2637
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Silence debug output by setting QT_LOGGING_RULES
to turn off all debug output.
Task-number: QTQAINFRA-1631
Change-Id: I5c2366b4fe4bac341dcfd92f68b6da8071c5b089
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
After a37785ec76 went in, it become
apparent that multi mouse button state handling in qtestlib is
non-existent, for details see QTBUG-64030 and QTBUG-63786. What
happened behind the scenes often was not what one would expect based
on the provided QTest::mouse* input sequence - events went missing,
incorrect events were generated, each subsequent test function
started with a state set from the function that run earlier. It is
easy to see how a minor change in one test could easily affect outcome
of other tests.
With a37785ec76, Qt platform plugins
are now responsible for sending explicit mouse button type and state
information; qtestlib should take full responsibility now as well.
But using the new API from a37785ec7 alone in qtestlib is not sufficient.
We need to reset mouse state between each new test function run (we do
this at function scope as that fits with the current qtestlib API user
expectations). This patch implements the necessary reseting logic.
Updated tst_qwindow.cpp::generatedMouseMove() to use QTest::mouse* APIs.
That test requires pressing multiple buttons, it was not possible with
QTest::mouse* APIs before this patch.
Added an auto test for multiple mouse button pressing/release in
tests/auto/testlib/selftests/mouse/. And few other tests which are
currently QSKIP-ed, but should be considered when re-designing qtestlib
APIs.
Task-number: QTBUG-64030
Change-Id: I39fdcbc73a467a7463ce2aed622bf22484095635
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Add formatting for registered enumerations based on QMetaEnum
and unregistered enumerations as hex values.
[ChangeLog][QtTest] QtTest now prints values of QFlags that failed to
compare with QCOMPARE.
Task-number: QTBUG-65845
Change-Id: I3eae6d20d3c0d72441ca6c4037d9a8dafa4b6357
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The "crashes" subtest has several "expected" reference files. The matching
one was previously determined by checking the line count of the output.
This however does not work when there are several reference files with
identical line count as is now the case with boot2qt.
Refactor the comparison code from the QTest/preliminary void return to
the bool f(QString* errorMessage) convention so that all files can be tried.
While doing so, streamline the code and remove numerous unneeded
allocations of strings and regular expressions per compared line.
Task-number: QTBUG-65845
Change-Id: I722159d1753f2a36f0e497e315ffd81cb58cac0b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
It has been observed that sometimes the output is not captured by QEMU.
Task-number: QTBUG-65845
Change-Id: Iddf888fb3aa019209dbe71f424d87884471784e0
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Use qInfo() instead of qDebug() in case QDebug is turned off.
If the 'diff' tool is available, write the output to temporary files and
run diff on it. Otherwise, print the lines as was before, but onto one
stream to avoid indentation by the testlib handler.
Change-Id: Ib5a5dfb66ce481b493b85b915aa8c785ecb6b387
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Move the modeltest autotest in the right place, and fix the
other autotests that were using it to use the version now in QtTestLib.
Change-Id: Ic6838945f616d580f357c872ce0956c341be3b16
Reviewed-by: David Faure <david.faure@kdab.com>
Currently, when two colors are equal except for their alpha values,
QCOMPARE produces the following failure message:
FAIL! : tst_Test::test() Compared values are not the same
Actual (colorA): #ff0000
Expected (colorB): #ff0000
By using the HexArgb format instead of the default HexRgb, we can
see the full hex string, with alpha values included:
FAIL! : tst_Test::test() Compared values are not the same
Actual (colorA): #88ff0000
Expected (colorB): #ffff0000
Task-number: QTBUG-55574
Change-Id: Id82c60a1b473ac6025a6f6ac560fce95a910d782
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
On some systems (e.g. Debian/buster) a standard Qt package may install
a qtlogging.ini file, e.g. in /etc/xdg/QtProject/, which suppresses
QDEBUG output. A ~/.config/QtProject/qtlogging.ini could interfere
similarly. This can break the selftest, when run with expected_*
files generated without such interference. Likewise, if those
expected_* files are generated on a system such similar, they won't
work on CI. Given that this caused confusion and delay with
integrating the latest set of updates to the expected_* files, it
seemed best to save others from similar bafflement.
So set a standard value for QT_LOGGING_RULES in both the selftest and
the generator script, so we get consistency.
Change-Id: I649e2f1f6ead21edf8af051aaee286e369fed064
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
In selftests.qrc, imposed alphabetic order (on stem of name, then on
suffix, effectively treating . as sorting before any letter) while
removing old tests and adding new tests and data. Updated all non-csv
files and added many missing files. (Not clear on csv support status;
the script seems to have dropped it after 5.6, but the test still uses
it.)
Left expected_crashes* alone (no new files added, no update to old) as
I don't get results resembling those anticipated.
Omitted printdatatagswithglobaltags, printdatatags due to dangling
hspace on output lines, which upset sanity-bot. A change to the test
cpp is needed to make it viable to skip that dangling hspace.
Change-Id: Iab3fb626c44a91c249b2fb626c12c75ea0317098
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
This makes qtestlib behave more like real Qt applications with an
event loop, where exiting the application does indeed flush deleteLaters.
And since every test method is supposed to be independent from other
test methods, we should even cleanup between tests.
For "app less" tests, no flushing happens.
Real life use cases:
* A unittest for some code (e.g. KIO job) which uses a socket, runs an
event loop until the socket communication is done, and ends. However
slotDisconnected() does, as recommended, socket->deleteLater(). So the
test finishes before the socket has a chance to actually get deleted,
and memory leak tools flag a memory leak, which doesn't actually happen
outside the unittest.
* Deleting a QWebEngineView with deleteLater is even worse than a memleak,
it leads to an assert (from a global object destructor) in QtWebEngine
if the view deletion hasn't actually been processed.
Change-Id: I18fc822fd26988a0fa4e75313c1947fcaa7d9e56
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Conflicts:
src/widgets/kernel/qwidget.cpp
This merge also extends the expected output of the pairdiagnostics
teamcity output (added in dev in commit
c608ffc56a) after the recent addition of
the flowId attribute to the teamcity output (commit
8f03656211 in 5.9).
Change-Id: I3868166e5efc45538544fffd14d8aba438f9173c
Added flowId='name' to each message when using TeamCity logging format.
This is necessary to distinguish separate processes running in parallel.
[ChangeLog][QtTest] Added flowId to messages when logging in TeamCity
format. FlowId is used to distinguish logging from multiple processes
running in parallel.
Change-Id: I7f5046c1058ff02770404caa2c9b3a5398f97f6b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Two more ways line numbers were making it through.
Corrected a doc-string to tell nearer to the truth.
Change-Id: I946aaeb936d47fffe50d7ec15e2524992cc9e428
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
