Currently when doing comparison with std::tuple the fallback toString
method is called which returns a Q_NULLPTR thus not allowing proper
diagnostic of the values that triggered an error. This patch
adds support for std::tuple to improve the tests output readability.
[ChangeLog][QtTest][QCOMPARE] Now outputs contents of std::tuple on
failure.
Change-Id: I046a55e2ce44c3f7728d51e4745120d38aa5e007
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
They didn't exist up until now, and future patches rely on them, so
add them.
Change-Id: I8afdb9417263b45d43355c688a813bdf99ea5fc8
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
This is what the other reporters also do, in various forms.
Task-number: QTBUG-67351
Change-Id: I16f2c4e0991176145ee0fbcbbfeeda071603a3c2
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Using QFileInfo to check if the file exists based on the filename
alone ignores the fact that all the expected files are embedded
as QRC resources.
The expectedResult() function already does a similar check, so we
can use that directly instead of checking twice if the file exists.
Task-number: QTBUG-66981
Change-Id: I0beb8d3503ed49682ae7d7e2a5172922fab5420d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Otherwise random debug messages from Qt might mess up the expected vs
actual results.
The setting of QT_LOGGING_RULES in initTestcase has been removed, as
the selftest overrides that for each invocation of a subtests, via
the processEnvironment() function.
Task-number: QTQAINFRA-1631
Change-Id: I855d31274f8261f8b125df23409353f7101be0e4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Silence debug output by setting QT_LOGGING_RULES
to turn off all debug output.
Task-number: QTQAINFRA-1631
Change-Id: I5c2366b4fe4bac341dcfd92f68b6da8071c5b089
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
The "crashes" subtest has several "expected" reference files. The matching
one was previously determined by checking the line count of the output.
This however does not work when there are several reference files with
identical line count as is now the case with boot2qt.
Refactor the comparison code from the QTest/preliminary void return to
the bool f(QString* errorMessage) convention so that all files can be tried.
While doing so, streamline the code and remove numerous unneeded
allocations of strings and regular expressions per compared line.
Task-number: QTBUG-65845
Change-Id: I722159d1753f2a36f0e497e315ffd81cb58cac0b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Use qInfo() instead of qDebug() in case QDebug is turned off.
If the 'diff' tool is available, write the output to temporary files and
run diff on it. Otherwise, print the lines as was before, but onto one
stream to avoid indentation by the testlib handler.
Change-Id: Ib5a5dfb66ce481b493b85b915aa8c785ecb6b387
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
On some systems (e.g. Debian/buster) a standard Qt package may install
a qtlogging.ini file, e.g. in /etc/xdg/QtProject/, which suppresses
QDEBUG output. A ~/.config/QtProject/qtlogging.ini could interfere
similarly. This can break the selftest, when run with expected_*
files generated without such interference. Likewise, if those
expected_* files are generated on a system such similar, they won't
work on CI. Given that this caused confusion and delay with
integrating the latest set of updates to the expected_* files, it
seemed best to save others from similar bafflement.
So set a standard value for QT_LOGGING_RULES in both the selftest and
the generator script, so we get consistency.
Change-Id: I649e2f1f6ead21edf8af051aaee286e369fed064
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
The test was added as commented in 2012, before it was not even
mentioned in the pro file. I guess it was never run.
Change-Id: Ib787c7811ab4f7f32a99869167cde2e12dbe3156
It has been completely untested for a while and saw some breakage.
So let's add a selftest for it. ('-vs' when running tests)
Change-Id: Ibfb5ac0a2d741de7c3f519d91202d4977996045e
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Currently when doing comparison with pair-like classes the fallback
toString method is called which returns a Q_NULLPTR thus not allowing
proper diagnostic of the values that triggered an error. This patch
adds support for QPair and std::pair to improve the tests output
readability.
[ChangeLog][QtTest][QCOMPARE] Now outputs contents of QPair and
std::pair on failure.
Change-Id: Ib0b4aad7640590d56275b6f1306fb9fb39fa81bc
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Two changes are needed to pass tst_Selftest on QEMU
1. Pass QEMU specific env variables to the subtests
2. Ignore output on stderr on some tests when running on QEMU
Change-Id: Ie1f722fd183aac5973e87d408005e06cbafcde17
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Replace all QT_NO_PROCESS with QT_CONFIG(process), define it in
qconfig-bootstrapped.h, add QT_REQUIRE_CONFIG(process) to the qprocess
headers, exclude the sources from compilation when switched off, guard
header inclusions in places where compilation without QProcess seems
supported, drop some unused includes, and fix some tests that were
apparently designed to work with QT_NO_PROCESS but failed to.
Change-Id: Ieceea2504dea6fdf43b81c7c6b65c547b01b9714
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@qt.io>
If QT_PRESERVE_TESTLIB_PATH is set, then LD_LIBRARY_PATH and
QT_PLUGIN_PATH won't be filtered out for the sub-tests started by
tst_selftest.
Change-Id: Ic43ba9b4d882ee36b2f7495b1c880f26aefd2629
Reviewed-by: aavit <eirik.aavitsland@theqtcompany.com>
This allows TeamCity to parse realtime test-results instead of using
post-build XML Report Processing. This does not support logging of
benchmarks.
[ChangeLog][QTest] Added a new logging mode that allow test-results to
be parsed on-the-fly when using Jetbrains TeamCity as CI-server. This
mode is enabled by using the -teamcity option on the command-line.
Change-Id: Ie730beb643043eb0f448f99abe6c0b5ac48aaf03
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
I wrote a script to help find the files, but I reviewed the
contributions manually to be sure I wasn't claiming copyright for search
& replace, adding Q_DECL_NOTHROW or adding "We mean it" headers.
Change-Id: I7a9e11d7b64a4cc78e24ffff142b506368fc8842
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some
exceptions, see
http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/
Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one
(in those files which will be under GPL 3 with exceptions)
Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
Previously, only QXmlTestLogger had timers to take elapsed times
and log them. Move those into class QTestLog for access by
the loggers and output the times in the crash dump to make it
easier to spot hangs/recursion crashes.
Produces:
QFATAL : foo() Received signal 11
Function time: 22ms Total time: 23ms
A crash occurred in ...
Function time: 24ms Total time: 26ms
Task-number: QTBUG-47370
Change-Id: Ia530a63104087daffc9a15f68c15d93378b9407e
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Use QVERIFY2() with QTemporaryDir/File::errorString() consistently.
Attempt to catch issues like the below warning and follow-up issues.
QSYSTEM: tst_QFiledialog::clearLineEdit() QFileSystemWatcher: FindNextChangeNotification failed for "C:\Users\qt\_____aaaaaaaaaaaaaaaaaaaaaa" (Access is denied.)
Task-number: QTBUG-47370
Change-Id: I58a6e87c502627e976efa62ad73c912f3b2d49fa
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@theqtcompany.com>
Make it easier to capture information about crashing tests in log files,
since it is not possible to do something like
foo > log.txt 2> errlog.txt
on Windows.
Adapt selftest.
[ChangeLog][QtTest][Important Behavior Changes] Crash/exception
information is now logged to standard output on Windows.
Task-number: QTBUG-47370
Change-Id: I3e6a2d25855ed0f20516418a031990b562f5f757
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
From MSVC 2015 onwards, MSVC formats floats using printf("%g") like
g++, so the fiddling of the expected output needs to restricted
accordingly.
Change-Id: I6e7f78e5e90f70886a8b2ef37c0fb9bf82b5e1a3
Reviewed-by: Kai Koehne <kai.koehne@theqtcompany.com>
This is for example useful when looking for a possible BLACKLIST
file while doing a shadow build.
Add autotests for blacklist.
Change-Id: I41d3939d31d21d10187fefcb82604736d911b6ad
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@theqtcompany.com>
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Dump a stack trace on Linux if the process crashes. This will give us
much better log output in the new CI system. In addition, create a
watch dog thread, that kills the test if the test function times
out (ie. hangs)
Implementations of the stack trace dumping for Mac and Windows
are still pending.
Change-Id: I65426c5afe290a0a2019b881436a0c278f1cafaf
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Similar to what we've done to QString, only we print each byte that is
not ASCII as \OOO (octal representation).
[ChangeLog][QtTest] QtTest now prints an escaped version of
QByteArrays that failed to compare with QCOMPARE, instead of the hex
dump.
Change-Id: I6a8c43f138c66c998280998a242b43cd579666a0
Reviewed-by: Richard J. Moore <rich@kde.org>
Qt copyrights are now in The Qt Company, so we could update the source
code headers accordingly. In the same go we should also fix the links to
point to qt.io.
Outdated header.LGPL removed (use header.LGPL21 instead)
Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing
combination. New header.LGPL-COMM taken in the use file which were
using old header.LGPL3 (src/plugins/platforms/android/extract.cpp)
Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license
combination
Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe
Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
GRAPHICS_ROOT and TZ environment variables are needed in child
processes in order to successfully run the auto test selftests.
Change-Id: I7befabd535b4c47b1e75acbe3d6158d0d9b811b3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Output the file name and fix line number to be 1..n.
Change-Id: I3ba7f84b99ce3664bc3e403b230307511b15fe1f
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
This is only useful for logging benchmarks, since it won't print test
passes, failures, etc. It's useful for importing to spreadsheets to do
number-crunching.
[ChangeLog][QtTest]Added a CSV logging mode that is suitable for
importing benchmark results into spreadsheets. This can be enabled by
the -csv option on the command-line. The CSV logging mode will not print
test failures, debug messages, warnings, etc.
Change-Id: I245d6f86bb380645c9bc0d748cf474b3ed42cab8
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Task-number: QTBUG-35743
[ChangeLog][QtTest][Windows] Use correct UTF-8 encoding for XML test results on platforms with different console encoding.
Change-Id: Ice9d03192098f931e5dac358928e0c4421ab715e
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
[ChangeLog][QtTest] Added test duration to xml output.
When running tests with xml output a new tag of the form
<duration msecs="123"/> is added to each test function and the test as a
whole.
Change-Id: Ibc4db066b6acf5fac6c578f5e5ca5ce4b5d8ea8e
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The units that were supposed to have variance were completely off in the
test leading to all benchmarks with slight variance being completely
flaky.
Change-Id: Ib76593813974adee462b3f03be9370d2248d770c
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
For the conflicts in msvc_nmake.cpp the ifdefs are extended since we
need to support windows phone in the target branch while it is not there
in the current stable branch (as of Qt 5.2).
Conflicts:
configure
qmake/generators/win32/msvc_nmake.cpp
src/3rdparty/angle/src/libEGL/Surface.cpp
src/angle/src/common/common.pri
src/corelib/global/qglobal.h
src/corelib/io/qstandardpaths.cpp
src/plugins/platforms/qnx/qqnxintegration.cpp
src/plugins/platforms/qnx/qqnxscreeneventhandler.h
src/plugins/platforms/xcb/qglxintegration.h
src/widgets/kernel/win.pri
tests/auto/corelib/thread/qreadwritelock/tst_qreadwritelock.cpp
tests/auto/corelib/tools/qdatetime/tst_qdatetime.cpp
tests/auto/gui/text/qtextdocument/tst_qtextdocument.cpp
tools/configure/configureapp.cpp
Change-Id: I00b579eefebaf61d26ab9b00046d2b5bd5958812
New macro QVERIFY_EXCEPTION_THROWN may be used to check that some code
really throws an exception of specified type.
Change-Id: I7cf499c7c37a35407862bc604c6eb862c5f329d3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Give processes a more complete environment (openSUSE
requires more variables to start an X11-process),
add test column indicating whether a test is
supposed to crash. If it is not, report crash reason.
Change-Id: I1e0ad59824963f69ee425f331d845741be2b3928
Reviewed-by: Tony Sarajärvi <tony.sarajarvi@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
A handful of tests lack QT_NO_PROCESS guards, making them non-compilable
on Qt builds with no QProcess support. This commit does not change
QProcess-specific tests, which should be left out of the build using the
.pro file mechanism.
Change-Id: Iac8562428abc1f59ccbb23bf5c3a919785e41f12
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Fix float format and exclude crashing sub-binary.
Task-number: QTBUG-29014
Change-Id: I404f971edeb128263122a194f23e2806d6fd3bd0
Reviewed-by: Kai Koehne <kai.koehne@digia.com>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
Use a QTemporaryDir for temporary files, instead of the current working
directory.
Change-Id: Ifeb2944238f785a1f7beb0dc2a7c1e092d121db5
Reviewed-by: Kalle Lehtonen <kalle.ju.lehtonen@nokia.com>
Reviewed-by: Toby Tomkins <toby.tomkins@nokia.com>
If the expected output file was missing (e.g. not included in
selftests.qrc), tst_selftests would trigger an assert inside QList by
calling QList::at() on an empty list. Make tst_selftests detect this
error instead and give a meaningful error message.
When loading expected output for the crashes selftest, where there are
several alternative versions of the expected output, the code reused the
"exp" variable when loading the alternative test output files. This
caused the last file loaded to be used unintentionally if none of the
alternative files had the correct number of lines. Use a different
variable so that exp remains empty if none of the alternatives are
valid and a failure can be reported.
Change-Id: I35b2a3d905d069d3ee8dcb1447836eb68d5c8612
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as
"COMPARE()", making it impossible to see what was compared and difficult
to match the output to the source of a test containing many calls to
QCOMPARE.
This commit changes testlib's internal compare_helper API so that string
representations of the compared expressions are always passed to
QTestResult::compare() when available, and can thus be shown in the
verbose and XPASS output. The XPASS output has also been changed to
state explicitly that the comparison succeeded unexpectedly, bringing it
in line with the XPASS output resulting from a call to QVERIFY.
This commit also changes all calls to compare_helper() to call the
eight-argument version of the function, which simplifies much of the
calling code. The now obsolete four-argument version of
compare_helper() has been changed to output a warning that it is
obsolete. It will be removed once other modules have had some time to
catch up.
The improved XPASS and verbose output is demonstrated by the expectfail
and verbose2 selftests.
Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
These tests don't have their own source code but rather reuse the
counting selftest with additional command-line options.
Note that currently the -v1 switch only changes the plain text output,
and the expected xml output is identical to that of the counting test.
This may change in the future however.
This commit also restores a couple of lists to alphabetical order, where
the findtestdata selftest was not sorted into the list correctly.
Change-Id: Ie38e255f8029157b34162b3864b5fa66e137d74a
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously the silent logging mode suppressed passes, skips and internal
testlib info messages, but did not suppress debugging output, making it
hard to see the fails in a noisy test. This commit changes silent mode
so that it suppresses all output except test failures and fatal errors,
making silent mode truly useful for seeing just the important test
output.
This commit also adds a selftest to verify the behaviour of silent mode.
Change-Id: I75420aead03682306210746a87e2a3b608b58fc6
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, a benchmark test could report 0..n passes and 0..m
fails or skips, where n is the number of accumulation iterations used to
collect benchmark data and m is the number of times the test function
was invoked. Depending on the type of benchmark measurer being used,
this could result in a very large volume of test output and inconsistent
pass, fail and skip counts between test runs.
This commit changes the behaviour so that each benchmark test reports
one pass, fail or skip, regardless of the number of iterations used to
collect benchmark data.
This commit also prevents benchmark data being reported in the test
output if the benchmark test failed or skipped, as any benchmark data is
of dubious value in such cases.
The latter change in behaviour requires a minor modification to the
badxml selftest, which now tests quoting of literal strings in xml test
output for both passing and failing benchmarks.
Finally, this commit also adds a new selftest specifically for verifying
correct behaviour for benchmarks that fail or skip.
Task-number: QTBUG-24313
Change-Id: I3426dc659a7511b62fd183a031c7235bc753f497
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Set QT_TESTCOCOON_ACTIVE environment variable when the coverage is installed
for a test and unset it when the coverage data is saved. Tests that run when
QT_TESTCOCOON_ACTIVE is set are subtests and will not be considered as
stand-alone tests for the coverage.
When a test is run as a subtest its coverage data will not be saved for
itself but for the main test it is merged with. Also its status will not be
reported since only the status of the main test is expected in the test report,
e.g. the test tests/auto/testlib/selftests.
Change-Id: Icfdf99300aae18040e1a3441a8af21f68df4c0db
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The test simply generates each possible pair of Pass, Fail and Skip
results. At present the test simply serves to demonstrate the current
shortcomings of testlib's plain text logging, namely:
* If a test function passes for all data rows, that is counted as one
pass, but each skipped or failed row counts as one skip or fail.
* Only skipped and failed rows are reported individually in the test
output. Passed rows are not reported, so it is impossible to see how
many rows were executed.
* A skip followed by a pass will be reported as an overall pass for the
test function, but the same rows in reverse order will not report any
overall result for the test function.
Future commits will attempt to correct these problems.
Task-number: QTBUG-22124
Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
- Changed to detect valgrind at runtime and skip test instead of failing.
- subprograms inherit QT_QPA_PLATFORM value from parent if set.
Change-Id: I280acee389df1ee74ee6758a0dd1601226e103c7
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Changed selftests unittest to use specific classes of QtCore instead of
pulling in all of QtCore headers by using include <QtCore>
- Decreasing build time.
Change-Id: Ic53a70423ecafeb342cc05ae6d7dee795533d205
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Changed these tests to use QFINDTESTDATA macro to detect location of testdata.
Checking for a specific file contained in the testdata so as not to be confused by empty directories
created during configure.
Change-Id: Iac2ac6304b6b9ac79e00886025b93ec0af5a8507
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Introduce a specialization for qCompare(QImage,QImage) that
checks isNull, size and format and outputs verbose messages.
Check isNull, size similarly for QPixmap.
Add an autotest:
- Add test to cmptest and make it a GUI application since
QImage requires QGuiApplication.
- Make testlib/selftests capable of running X11-GUI applications
by passing DISPLAY.
- Ignore stderr output for cmptest
- Add test data
Change-Id: I2b29c7822fbeedf2b22c90889739ed7ff859ce92
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Required for launching the sub-processes. The test worked only
when launched locally, not from 'make check'.
Change-Id: I42c9202a7726c3135f94445fb336b2b8241535a4
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@nokia.com>
The overload is used in Qt Creator
(see src/libs/extensionsystem/pluginmanager.cpp).
The use case here is an application whose internal
QObjects can be tested by passing a command line parameter.
For this use case, it is inconvenient to have to allocate
memory and create a char argv[]- array.
This reverts commit ad80d42f8e.
Change-Id: I2a2f91e2840100fd62743f6d03b33005d67b18f8
Reviewed-by: Daniel Teske <daniel.teske@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- Do not run with empty environment. At least PATH
is required at least (Qt + MSVC/MinGW runtime)
- Account for MSVC's different formatting of double numbers.
Change-Id: Ic7b1cf4a16a88c5384347b2651b011ac13c92d70
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The overload of QTest::qExec() that takes a QStringList is not used
anywhere in Qt's autotests, despite having been in the qtestlib API
since Qt 4.4.
This lack of use most likely derives from the fact that none of the
QTEST_MAIN macros use the overload, and more than 99% of Qt's tests
use those macros to avoid explicitly calling QTest::qExec().
Change-Id: I264b21d7fe1a9f2d565f748cf8bbe32414a73bb0
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Automated tests often need to load some data from external files.
Currently, a wide variety of approaches for this have been used in Qt
autotests, including:
- embed the source directory into the test binary at compile time, and
find the testdata relative to that; this fails when the source tree
is no longer available (e.g. when the tests are deployed to a device).
- use a path relative to the current working directory, and trust that
the caller always sets the current working directory such that the
testdata can be found; this fails when the caller uses a different
working directory than expected.
- use a path relative to QCoreApplication::applicationDirPath();
this fails when source tree != build tree (since testdata is not
automatically copied into the build tree).
- compile the files into the binary using the Qt resource system; this
should work, but does not allow for testing of code which genuinely
needs external files.
It seems that there is not a simple method for determining the testdata
path which can be reliably used in all circumstances, so various tests
have reinvented the testdata location method in different ways.
Therefore, this is a good candidate for an addition to the testlib API.
The current implementation of QFINDTESTDATA is able to find testdata
in all three of (build tree, install tree, source tree), in that order.
Change-Id: Ib2fed860723ccf437240da3b00db22dfe1a6b56c
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
(Note: This feature is ported from Qt 4.8.
See the following commits:
01575deafb7d26ca2431374e92c6d71de96547c7
4866d1ba8afbab61e102942d1ea93b81fea053d6
)
Passing the -datatags option to a QTestLib program prints the
available data tags to standard output.
For completeness, the test case name is also printed
at the start of each output line. (Although the file name
is supposed to match the lower-case version of the test case
name, this is currently not true in all cases (particularly not
under tests/benchmarks/). Even if there was a script to enforce this
convention, the -datatags option provides this information in a
reliable way.)
Data tags for each test function (f() in this case) are printed in
four different ways depending on the presence of local and global
data tags:
Case 1: No tags:
tst_MyTestCasetst_MyTestCase f
Case 2: Local tags only:
tst_MyTestCase f local tag 1
tst_MyTestCase f local tag 2
...
Case 3: Global tags only:
tst_MyTestCase f __global__ global tag 1
tst_MyTestCase f __global__ global tag 2
...
Case 4: Local and global tags:
tst_MyTestCase f local tag 1 __global__ global tag 1
tst_MyTestCase f local tag 2 __global__ global tag 1
...
tst_MyTestCase f local tag 1 __global__ global tag 2
tst_MyTestCase f local tag 2 __global__ global tag 2
...
...
Note that the string __global__ is assumed to be highly unlikely to occur
in a data tag (if it does, an ambiguity results).
Change-Id: Ib51aa0c3c32ad52e52ce519729292cf8f0ec5d50
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@nokia.com>
This test duplicates the skipinitdata selftest and has slightly less
informative output.
Change-Id: Ifd40e3ef8030059ec8fa0089ce5b2a994624abeb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The addition of testing for multiple loggers greatly increased the total
runtime of this test, in fact making it the slowest testcase in Qt.
Fortunately this is only due to a couple of slow subtests whose behavior
is unlikely to be affected by the loggers. Change it to run these slow
subtests just for a couple of loggers, instead of all the combinations.
Change-Id: Ie90f6c0ca29470ed6a7c4e2e185f852602a2d162
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The selftests already used the @INSERT_QT_VERSION_HERE@ placeholder
in place of the real Qt and QTestlib versions in the expected test
output of the subtests. For unknown reasons, the same was not true for
the expected plain text output. In the past, this has caused Release
Managers to waste time incrementing the version numbers in these files.
Change-Id: I52f7870486fce128c04d53ff06978afa947474fd
Reviewed-on: http://codereview.qt-project.org/5375
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
There is no test with a datatag "float", so this code is never executed.
Change-Id: I9dd234e6575eb1c33075705edf77eb1d977061c9
Reviewed-on: http://codereview.qt-project.org/5362
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Q_ASSERT uses __FILE__, which has undefined contents, and thus the
selftest cannot always match Q_ASSERT messages in actual output with
those in expected output. The test was calling QEXPECT_FAIL() in this
case, but doing so implies that (a) it's a failure and (b) we intend to
fix it one day. Neither of these is true, so instead we should simply
tolerate this difference in actual and expected outputs, in the same way
that we do in a couple of other similar cases.
Change-Id: I8b8609a400e9b798936875f6eb1ca18a83660efc
Reviewed-on: http://codereview.qt-project.org/5358
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Each destination and the format of output to write there is specified by
adding "-o filename,format" to the command-line. The special filename
"-" indicates that the log output is written to the standard output
stream, though standard output can be used as a destination at most
once.
The old-style testlib output options are still supported, but can only
be used to specify one logging destination, as before.
If no logging options are given on the command-line, a plain text log
will go to the console, as before.
To log to the console in plain text and to the file "test_output" in
xunit format, one would invoke a test in the following way:
tst_foo -o test_output,xunitxml -o -,txt
This commit also enhances the selftests to test with multiple loggers,
but negative tests (e.g. bad combinations of command-line options) are
left for future task QTBUG-21567.
Task-number: QTBUG-20615
Change-Id: If91e752bc7001657e15e427aba9d25ab0a29a0b0
Reviewed-on: http://codereview.qt-project.org/4125
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Modify the selftest to be able to run each subtest with a list of one or
more test loggers. The addition of tests that use this capability will
be part of a subsequent commit.
Task-number: QTBUG-20615
Change-Id: Iac3efe8220e8245aa7e5589348d2c86b8034dd28
Reviewed-on: http://codereview.qt-project.org/5292
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The multiexec test runs the same test object five times. If the -o
option is given, the output file is overwritten by each run of the test
object, meaning that tst_selftest only sees 1/5 of the test output in a
file compared with what it sees on the console. This makes it
impossible to use the expected output file to verify the test for both
console and file output. This issue is noted in QTBUG-21561.
Change-Id: I00031a2ea43a7ef78e8317473a089306ec062d8e
Reviewed-on: http://codereview.qt-project.org/5270
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test was attempting to verify two completely unrelated things, so
this commit splits it into two tests.
Also, printf calls are replaced by qDebug so that the test does not
bypass the testlib loggers.
Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee
Reviewed-on: http://codereview.qt-project.org/5178
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test is not useful for finding bugs in qtestlib's logging code,
because it bypasses the qtestlib loggers and doesn't play nice with
tst_selftest. Neither is this test very useful for finding bugs in
QTest::qWait(), as the test only proves the qWait() terminates, not that
it waits accurately, or even that it waits at all.
Change-Id: Ia5dd7cbaf3a6fbb4e94e54ed155263580e495694
Reviewed-on: http://codereview.qt-project.org/5173
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Don't store separate strings for the logger name and the associated file
suffix -- just use the same string everywhere.
Change-Id: Ie7d1af6bf906b5ac09fbd5fcc5754b68036fb370
Reviewed-on: http://codereview.qt-project.org/5060
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The selftest was only attempting to delete the output file at the end of
the entire test run. This file should actually be deleted after each
subtest to ensure that it does not cause confusing errors if the next
subtest fails to overwrite the output file.
Change-Id: Ia3827926f71a697c6108de3f2b08c969f761d978
Reviewed-on: http://codereview.qt-project.org/5051
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The removed calls related to tests that were removed when redundant
logging functionality was removed.
Change-Id: Iaebbc109eaaddb440ae78e1f4a1b4e874a8b2960
Reviewed-on: http://codereview.qt-project.org/5050
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Use C++ comment delimiters consistently
* Use a space after control flow keywords
* Don't nest "else if" blocks unnecessarily
* Follow indenting rules
* Added comments describing some internal functions
Change-Id: I72fd1eaf5c5f3130945fbd2b64fa59e19ad2913d
Reviewed-on: http://codereview.qt-project.org/5049
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
