Previously, only QXmlTestLogger had timers to take elapsed times
and log them. Move those into class QTestLog for access by
the loggers and output the times in the crash dump to make it
easier to spot hangs/recursion crashes.
Produces:
QFATAL : foo() Received signal 11
Function time: 22ms Total time: 23ms
A crash occurred in ...
Function time: 24ms Total time: 26ms
Task-number: QTBUG-47370
Change-Id: Ia530a63104087daffc9a15f68c15d93378b9407e
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Use QVERIFY2() with QTemporaryDir/File::errorString() consistently.
Attempt to catch issues like the below warning and follow-up issues.
QSYSTEM: tst_QFiledialog::clearLineEdit() QFileSystemWatcher: FindNextChangeNotification failed for "C:\Users\qt\_____aaaaaaaaaaaaaaaaaaaaaa" (Access is denied.)
Task-number: QTBUG-47370
Change-Id: I58a6e87c502627e976efa62ad73c912f3b2d49fa
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@theqtcompany.com>
The keyword no longer has a meaning for the new CI.
Change-Id: Ibcea4c7a82fb7f982cf4569fdff19f82066543d1
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Make it easier to capture information about crashing tests in log files,
since it is not possible to do something like
foo > log.txt 2> errlog.txt
on Windows.
Adapt selftest.
[ChangeLog][QtTest][Important Behavior Changes] Crash/exception
information is now logged to standard output on Windows.
Task-number: QTBUG-47370
Change-Id: I3e6a2d25855ed0f20516418a031990b562f5f757
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
[ChangeLog][Important Behavior Changes] QDebug output for QStrings
changed compared to Qt 5.5.0 to more closely match the output of
previous Qt versions. Like Qt 5.5.0, QDebug will escape non-printable
characters, the backslash and quote characters, but will no longer
escape the printable characters.
Task-number: QTBUG-47316
Change-Id: I52dd43c12685407bb9a6ffff13f62ef68cbc80c5
Reviewed-by: Kai Koehne <kai.koehne@theqtcompany.com>
Add QTRY_VERIFY2_WITH_TIMEOUT() similar to QTRY_VERIFY_WITH_TIMEOUT()
except that QTRY_VERIFY2() is used below the loop and QTRY_VERIFY2()
based on it.
Add tests to cmptest.
[ChangeLog][QtTest] Added macros QTRY_VERIFY2_WITH_TIMEOUT(), QTRY_VERIFY2()
making it possible to output a message after the timeout has expired.
Change-Id: I587e24f3aeb73542dbf3ccb936a16f2e0806555f
Reviewed-by: Mitch Curtis <mitch.curtis@theqtcompany.com>
From MSVC 2015 onwards, MSVC formats floats using printf("%g") like
g++, so the fiddling of the expected output needs to restricted
accordingly.
Change-Id: I6e7f78e5e90f70886a8b2ef37c0fb9bf82b5e1a3
Reviewed-by: Kai Koehne <kai.koehne@theqtcompany.com>
This is for example useful when looking for a possible BLACKLIST
file while doing a shadow build.
Add autotests for blacklist.
Change-Id: I41d3939d31d21d10187fefcb82604736d911b6ad
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@theqtcompany.com>
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Dump a stack trace on Linux if the process crashes. This will give us
much better log output in the new CI system. In addition, create a
watch dog thread, that kills the test if the test function times
out (ie. hangs)
Implementations of the stack trace dumping for Mac and Windows
are still pending.
Change-Id: I65426c5afe290a0a2019b881436a0c278f1cafaf
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
Similar to what we've done to QString, only we print each byte that is
not ASCII as \OOO (octal representation).
[ChangeLog][QtTest] QtTest now prints an escaped version of
QByteArrays that failed to compare with QCOMPARE, instead of the hex
dump.
Change-Id: I6a8c43f138c66c998280998a242b43cd579666a0
Reviewed-by: Richard J. Moore <rich@kde.org>
Qt copyrights are now in The Qt Company, so we could update the source
code headers accordingly. In the same go we should also fix the links to
point to qt.io.
Outdated header.LGPL removed (use header.LGPL21 instead)
Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing
combination. New header.LGPL-COMM taken in the use file which were
using old header.LGPL3 (src/plugins/platforms/android/extract.cpp)
Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license
combination
Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe
Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
Selftests for testlib fail when executed in UTC timezone because local and
UTC are the same, but expected to be different. A custom timezone is used
instead.
Debug output of qCompare does only handle local and non-local timezones, using
new Qt5 features allows to show the correct timezone in format string.
Change-Id: I753884a12370952b7b62a90d62896db4f2d3d1b4
Reviewed-by: Jason McDonald <macadder1@gmail.com>
[ChangeLog][QtCore][QDebug] Printing of QStrings and QStringRefs
whenever "noquote" is not active now prints the strings in a format that
can be copied back to C++ code. All characters that aren't printable in
US-ASCII are escaped (this includes printable Unicode characters outside
of US-ASCII). Pretty-printing will not respect QTextFormat padding or
field widths.
Change-Id: I169a8a0508e24693f5652f0129defe7f709e5d08
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@theqtcompany.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Add an 'info' message type that can be used for messages that are neither
warnings (QtWarningMsg), nor for debugging only (QtDebugMsg). This is
useful mainly for applications that do not have to adhere to the
'do not print anything by default' paradigm that we have for
the Qt libraries itself.
[ChangeLog][QtCore][Logging] QtInfoMsg got added as a new QtMsgType.
Use the new qInfo(), qCInfo() macros to log to it.
Change-Id: I810995d63de46c41a9a99a34d37c0d417fa87a05
Reviewed-by: Jason McDonald <macadder1@gmail.com>
We need to have a finer grained control over the tests
we skip in our CI system. This adds a blacklisting
mechanism that allows blacklisting individual test
functions (or even test data) using a set of predefined
matching keys for the operating system and some other
relevant variables.
QTestlib will search for a file called BLACKLIST in the test
directory and parse it if found. The file contains a simple
ini style list of functions to blacklist. For details see
qtestblacklist.cpp.
Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
GRAPHICS_ROOT and TZ environment variables are needed in child
processes in order to successfully run the auto test selftests.
Change-Id: I7befabd535b4c47b1e75acbe3d6158d0d9b811b3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Added the function pointer constructor to QSignalSpy to take advantage
of the new connect syntax.
Change-Id: I94218a096c677cdba73e1b2cfa8b9c09bc28145f
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Output the file name and fix line number to be 1..n.
Change-Id: I3ba7f84b99ce3664bc3e403b230307511b15fe1f
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
This also has the advantage of not requiring the use of the locale
codec. Quite an advantage if you're debugging the locale codec. But it's
mostly so that we don't get question marks that hide the difference we
were trying to locate.
[ChangeLog][QtTest] QtTest now prints an escaped version of QStrings
that failed to compare with QCOMPARE. That is, instead of converting
non-printable characters to question marks, QtTest will print the
Unicode representation of the character in question.
Change-Id: I44c1ef3246b188c913dacd3ca4df02581356ea41
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
This is only useful for logging benchmarks, since it won't print test
passes, failures, etc. It's useful for importing to spreadsheets to do
number-crunching.
[ChangeLog][QtTest]Added a CSV logging mode that is suitable for
importing benchmark results into spreadsheets. This can be enabled by
the -csv option on the command-line. The CSV logging mode will not print
test failures, debug messages, warnings, etc.
Change-Id: I245d6f86bb380645c9bc0d748cf474b3ed42cab8
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Task-number: QTBUG-35743
[ChangeLog][QtTest][Windows] Use correct UTF-8 encoding for XML test results on platforms with different console encoding.
Change-Id: Ice9d03192098f931e5dac358928e0c4421ab715e
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Do not apply additional encoding when reading process output.
Fixes errors when encountering UTF-8:
Traceback (most recent call last):
File "./generate_expected_output.py", line 117, in <module>
generateTestData(path)
File "./generate_expected_output.py", line 106, in generateTestData
out.write(data.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 5485: ordinal not in range(128)
Change-Id: Ib827787a59a18b4d3d0601645856517f43c01fc3
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Evaluate command line arguments and use directories only when empty.
Change-Id: I818ec13c686018a3f607e91174e57d8f8bbf15f7
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
The actual code in testlib now indents the duration tag when it is for a
function.
Change-Id: Iee62db9c81f11dc54e57f166bf9fb2b7012b7e03
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
[ChangeLog][QtTest] Added test duration to xml output.
When running tests with xml output a new tag of the form
<duration msecs="123"/> is added to each test function and the test as a
whole.
Change-Id: Ibc4db066b6acf5fac6c578f5e5ca5ce4b5d8ea8e
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The units that were supposed to have variance were completely off in the
test leading to all benchmarks with slight variance being completely
flaky.
Change-Id: Ib76593813974adee462b3f03be9370d2248d770c
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
This makes testlib a lot more hackable since when the output is changed
in a defined way we can regenerate the expected output (eg adding the
test duration which will be a follow-up patch).
Note that the script does not work properly on Windows and not all
benchmarks work, but at least it reduces the work of adapting files to a few lines.
Change-Id: I910387cd92ff82aa629747a3a3033dae17fbd711
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
The newly added generate_expected_output.py was used to get the expected
output into a more reproducible state.
Change-Id: I1ca75c8e0c5778d25c1df531bd298007aac0ff4a
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
It's unexpected that all messages generated by the stream version
of qDebug and friends have a trailing space. It also makes switching
to categorized logging (which only supports the stream version) difficult,
since all autotests checking for debug output would have to be adapted.
Task-number: QTBUG-15256
Change-Id: I8d627a8379dc273d9689f5611184f03607b73823
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
When comparing expected with actual debug messages, allow the expected
ones to contain a trailing space. This is needed to not break all
autotests in a follow up patch, which will prevent ~QDebug() from
generating a trailing space by default.
Task-number: QTBUG-15256
Change-Id: I7f67393ddfbfe37fde1ca5ef45e4ad7f4b5324b4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
For the conflicts in msvc_nmake.cpp the ifdefs are extended since we
need to support windows phone in the target branch while it is not there
in the current stable branch (as of Qt 5.2).
Conflicts:
configure
qmake/generators/win32/msvc_nmake.cpp
src/3rdparty/angle/src/libEGL/Surface.cpp
src/angle/src/common/common.pri
src/corelib/global/qglobal.h
src/corelib/io/qstandardpaths.cpp
src/plugins/platforms/qnx/qqnxintegration.cpp
src/plugins/platforms/qnx/qqnxscreeneventhandler.h
src/plugins/platforms/xcb/qglxintegration.h
src/widgets/kernel/win.pri
tests/auto/corelib/thread/qreadwritelock/tst_qreadwritelock.cpp
tests/auto/corelib/tools/qdatetime/tst_qdatetime.cpp
tests/auto/gui/text/qtextdocument/tst_qtextdocument.cpp
tools/configure/configureapp.cpp
Change-Id: I00b579eefebaf61d26ab9b00046d2b5bd5958812
New macro QVERIFY_EXCEPTION_THROWN may be used to check that some code
really throws an exception of specified type.
Change-Id: I7cf499c7c37a35407862bc604c6eb862c5f329d3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Give processes a more complete environment (openSUSE
requires more variables to start an X11-process),
add test column indicating whether a test is
supposed to crash. If it is not, report crash reason.
Change-Id: I1e0ad59824963f69ee425f331d845741be2b3928
Reviewed-by: Tony Sarajärvi <tony.sarajarvi@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Make it possible to match messages by a pattern.
Change-Id: I713312e86db5471755459f1ecc43e8f1ac7a95fb
Reviewed-by: Jason McDonald <macadder1@gmail.com>
- Remove irrelevant test subdirs via .pro files
- Follow WinCE codepaths where applicable
- Replace unsupported Win32 APIs with WinRT equivalents
This does not aim to fix any failures in the tests themselves; it only
makes them compile.
Change-Id: Ia82bc0cc402891f8f6238d4c261ee9152b51be80
Reviewed-by: Maurice Kalinowski <maurice.kalinowski@digia.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
A handful of tests lack QT_NO_PROCESS guards, making them non-compilable
on Qt builds with no QProcess support. This commit does not change
QProcess-specific tests, which should be left out of the build using the
.pro file mechanism.
Change-Id: Iac8562428abc1f59ccbb23bf5c3a919785e41f12
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Before:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual): F0O
Expected (expected): FOO
Now:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual) : F0O
Expected (expected): FOO
Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef
Reviewed-by: Jason McDonald <macadder1@gmail.com>
"QTest" is the C++ namespace; "QtTest" is the library name
- Edited the logger output in qplaintestlogger.cpp
- Updated documentation
- Updated expected outputs for self-tests
Change-Id: I43c525c43221a8d4e843a00d6d55b0f06ef55fd7
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Remove all trailing whitespace from the following list of files:
*.cpp *.h *.conf *.qdoc *.pro *.pri *.mm *.rc *.pl *.qps *.xpm *.txt *README
excluding 3rdparty, test-data and auto generated code.
Note A): the only non 3rdparty c++-files that still
have trailing whitespace after this change are:
* src/corelib/codecs/cp949codetbl_p.h
* src/corelib/codecs/qjpunicode.cpp
* src/corelib/codecs/qbig5codec.cpp
* src/corelib/xml/qxmlstream_p.h
* src/tools/qdoc/qmlparser/qqmljsgrammar.cpp
* src/tools/uic/ui4.cpp
* tests/auto/other/qtokenautomaton/tokenizers/*
* tests/benchmarks/corelib/tools/qstring/data.cpp
* util/lexgen/tokenizer.cpp
Note B): in about 30 files some overlapping 'leading tab' and
'TAB character in non-leading whitespace' issues have been fixed
to make the sanity bot happy. Plus some general ws-fixes here
and there as asked for during review.
Change-Id: Ia713113c34d82442d6ce4d93d8b1cf545075d11d
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@digia.com>
Fix float format and exclude crashing sub-binary.
Task-number: QTBUG-29014
Change-Id: I404f971edeb128263122a194f23e2806d6fd3bd0
Reviewed-by: Kai Koehne <kai.koehne@digia.com>
Only QStringList was handled before, now any QList is handled.
A specialization for QStringList is still needed though, due to the way
template matching works.
Example with QList<int>. Before:
FAIL! : tst_QTextCodec::threadSafety() Compared values are not the same
Loc: [../tst_qtextcodec.cpp(2057)]
After:
FAIL! : tst_QTextCodec::threadSafety() Compared lists differ at index 0.
Actual (res2.toList()): '0'
Expected (mibList): '3'
Loc: [../tst_qtextcodec.cpp(2057)]
Change-Id: If0fdec3236ddb78a679ee549aba569ef5571c395
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
These types are either built-in or 'automatically declared' and so
don't need to be explicitly declared as metatypes.
Change-Id: Ifd116dee32a450ff89a9a1011e26b434765d6e95
Reviewed-by: David Faure <david.faure@kdab.com>
As they are built-in, they are effectively registered at compile-time
already.
Change-Id: I7ae6ba16088eab5d19213fa7b07c2a7760988a86
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
that way we don't have to auto-generate code for that in the configures.
note that we now load qt_build_config.prf instead of just qmodule.pri,
which means that exceptions_off is set everywhere. we forcibly re-enable
them for testcases to minimize the deviation from default 3rd party usage.
testlib selftests are not qt testcases, so the one that needs exceptions
needs to enable them explicitly.
Change-Id: I1b9360bb11f2e80c92a2b63a7c45991ad17fda1b
Reviewed-by: Joerg Bornemann <joerg.bornemann@digia.com>
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@digia.com>
If an URL is invalid, let's indicate that in the test output. To be
helpful, let's make QUrl::errorString() include the component form of
the URL.
Change-Id: Iaafe16973ded79c7ea688fbb23808d91253e8c14
Reviewed-by: David Faure <faure@kde.org>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
Previously, a QVariant parameter would be wrapped inside a new
QVariant, and you would have to cast the QSignalSpy's QVariant to
a QVariant to get the actual value. This behavior was unintuitive
and undocumented.
Check if the parameter type is QVariant, and copy it directly if it
is. This makes the QSignalSpy's QVariant directly usable (no need to
"unwrap" the value in user code).
Existing tests that use QSignalSpy together with QVariant parameters
(such as tst_QPropertyAnimation::valueChanged()) and do cast the
QVariant parameter to a QVariant, continue to work after this change;
this is because qvariant_cast<QVariant>() returns its input value
(unchanged) when the type is not QMetaType::QVariant.
Task-number: QTBUG-21645
Change-Id: Ibfb171edd60c0d3f7ca1d5419e5c5f3d0380d5b3
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Qt 5.0 beta requires changing the default to the 5.0 API, disabling
the deprecated code. However, tests should test (and often do) the
compatibility API too, so turn it back on.
Task-number: QTBUG-25053
Change-Id: I8129c3ef3cb58541c95a32d083850d9e7f768927
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
The previous API was hard to use (global function, no type safety,
manual chaining), and confusing (app vs dispatcher split only made
sense on Windows). Installing and removing out of order would have
the risk of setting back a dangling pointer (crash). Meanwhile QPA
added type safety, and this new API models the QObject::installEventFilter
API for ease of use. The virtual method is in a new interface,
QAbstractNativeEventFilter.
QPA was even calling the dispatcher event filter with QPA-private event
classes, which made no sense (refactoring leftover from when the code
was in the dispatcher). Now the QPA plugins trigger the qcoreapp event
filters with the actual native events directly.
Change-Id: Ie35e47c59c862383bcaf857b28d54f7c72547882
Reviewed-by: Marius Storm-Olsen <marius.storm-olsen@nokia.com>
Make test projects declare TEST_HELPER_INSTALLS rather than calling a
function exported by testcase.prf. load(testcase) may be unsafe, as
testcase.prf should be processed after default_post.prf.
Fixes silent disabling of various autotests.
Change-Id: I56b35ffd653a637ad5ab18d64dd1a1edadfac59f
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Future-proofing. Since Qt source code is now mandated to be in UTF-8,
it is entirely possible that someone will use non-ASCII in data tags.
Though it would be interesting to see how to access them from the
Windows command-line.
Change-Id: I880fc312432b62143888ff1e1d9abbd54f704601
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Use a QTemporaryDir for temporary files, instead of the current working
directory.
Change-Id: Ifeb2944238f785a1f7beb0dc2a7c1e092d121db5
Reviewed-by: Kalle Lehtonen <kalle.ju.lehtonen@nokia.com>
Reviewed-by: Toby Tomkins <toby.tomkins@nokia.com>
This way, QWinEventNotifier will work on all Windows systems, not just
with the default event dispatcher. Other dispatchers (other than
QWin32EventDispatcher) are permitted, so the class should not abort just
because of that.
If a dispatcher really doesn't want to implement this, they need to
implement the virtuals to do nothing, possibly print a warning.
Change-Id: I2c132bcde95b9d5941c8906a0fcd2ad964087772
Reviewed-by: Bradley T. Hughes <bradley.hughes@nokia.com>
It is an extension coming from the use case when you, for instance, need to
implement a countdown timer in client codes, and manually maintain a dedicated
variable for counting down with the help of yet another Timer. There might be
other use cases as well. The returned value is meant to be in milliseconds, as
the method documentation says, since it is reasonable, and consistent with the
rest (ie. the interval accessor).
The elapsed time is already being tracked inside the event dispatcher, thus the
effort is only exposing that for all platforms supported according to the
desired timer identifier, and propagating up to the QTimer public API. It is
done by using the QTimerInfoList class in the glib and unix dispatchers, and the
WinTimeInfo struct for the windows dispatcher.
It might be a good idea to to establish a QWinTimerInfo
(qtimerinfo_win{_p.h,cpp}) in the future for resembling the interface for
windows with the glib/unix management so that it would be consistent. That would
mean abstracting out a base class (~interface) for the timer info classes.
Something like that QAbstractTimerInfo.
Test: Build test only on (Arch)Linux, Windows and Mac. I have also run the unit
tests and they passed as well.
Change-Id: Ie37b3aff909313ebc92e511e27d029abb070f110
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Bradley T. Hughes <bradley.hughes@nokia.com>
Callers should just call the standard allocation functions directly.
Adding an extra function call onto all basic memory management for the sake of
making it instrumentable in rare cases isn't really fair to everyone else.
What's more, this wasn't completely reliable, as not everything was using them
in a number of places. Memory management can still be overridden using tricks
like LD_PRELOAD if needed.
Their aligned equivilents cannot be deprecated, as no standard equivilents
exist, although investigation into posix_memalign(3) is a possibility
for the future.
Change-Id: Ic5f74b14be33f8bc188fe7236c55e15c36a23fc7
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Make the various versions of the failure message align consistently so
that it's a little easier to compare the actual and expected values. Of
course, the value won't align nicely unless the "actual" and "expected"
strings are the same length, but at least this commit makes that
consistent across all versions of the message.
Change-Id: If9ce231df3b5d279a06f6458fdb5da0aa4586068
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
If the expected output file was missing (e.g. not included in
selftests.qrc), tst_selftests would trigger an assert inside QList by
calling QList::at() on an empty list. Make tst_selftests detect this
error instead and give a meaningful error message.
When loading expected output for the crashes selftest, where there are
several alternative versions of the expected output, the code reused the
"exp" variable when loading the alternative test output files. This
caused the last file loaded to be used unintentionally if none of the
alternative files had the correct number of lines. Use a different
variable so that exp remains empty if none of the alternatives are
valid and a failure can be reported.
Change-Id: I35b2a3d905d069d3ee8dcb1447836eb68d5c8612
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as
"COMPARE()", making it impossible to see what was compared and difficult
to match the output to the source of a test containing many calls to
QCOMPARE.
This commit changes testlib's internal compare_helper API so that string
representations of the compared expressions are always passed to
QTestResult::compare() when available, and can thus be shown in the
verbose and XPASS output. The XPASS output has also been changed to
state explicitly that the comparison succeeded unexpectedly, bringing it
in line with the XPASS output resulting from a call to QVERIFY.
This commit also changes all calls to compare_helper() to call the
eight-argument version of the function, which simplifies much of the
calling code. The now obsolete four-argument version of
compare_helper() has been changed to output a warning that it is
obsolete. It will be removed once other modules have had some time to
catch up.
The improved XPASS and verbose output is demonstrated by the expectfail
and verbose2 selftests.
Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The verbose output for QCOMPARE is not very helpful. Make the verbose2
selftest (which reuses the counting selftest) demonstrate this, so that
a future commit can demonstrate improvement.
Change-Id: I6b3bc8f5199e984aa11d0a67b76a8c916be86380
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
These tests don't have their own source code but rather reuse the
counting selftest with additional command-line options.
Note that currently the -v1 switch only changes the plain text output,
and the expected xml output is identical to that of the counting test.
This may change in the future however.
This commit also restores a couple of lists to alphabetical order, where
the findtestdata selftest was not sorted into the list correctly.
Change-Id: Ie38e255f8029157b34162b3864b5fa66e137d74a
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Do this regardless of whether the event subclass
is public API or only used in examples. Examples
are examples, used by others as templates or even
copied verbatim, so they should also follow sound
engineering rules.
Anyway, there's only one in examples/...
Change-Id: I586ff16407a956c9e89288fdd4377eed73f45c0f
Reviewed-by: Samuel Rødal <samuel.rodal@nokia.com>
The private git history shows that the test program associated with this
data file was missing from the original commit and was never
subsequently added.
Change-Id: I3401724ac04168158a48eb06436db83d3557711f
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously the silent logging mode suppressed passes, skips and internal
testlib info messages, but did not suppress debugging output, making it
hard to see the fails in a noisy test. This commit changes silent mode
so that it suppresses all output except test failures and fatal errors,
making silent mode truly useful for seeing just the important test
output.
This commit also adds a selftest to verify the behaviour of silent mode.
Change-Id: I75420aead03682306210746a87e2a3b608b58fc6
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The meta-object format is going to change for Qt5. Use QMOB to
insulate the badxml test from such changes.
(It just so happens that the QFAIL("a failure") statement is still on
line 109 after the refactoring, so the expected_badxml.* files'
location tags did not have to be changed.)
Change-Id: I04421d13c4df71c8004fa71cafc4823a59079a41
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Prior to this commit, a benchmark test could report 0..n passes and 0..m
fails or skips, where n is the number of accumulation iterations used to
collect benchmark data and m is the number of times the test function
was invoked. Depending on the type of benchmark measurer being used,
this could result in a very large volume of test output and inconsistent
pass, fail and skip counts between test runs.
This commit changes the behaviour so that each benchmark test reports
one pass, fail or skip, regardless of the number of iterations used to
collect benchmark data.
This commit also prevents benchmark data being reported in the test
output if the benchmark test failed or skipped, as any benchmark data is
of dubious value in such cases.
The latter change in behaviour requires a minor modification to the
badxml selftest, which now tests quoting of literal strings in xml test
output for both passing and failing benchmarks.
Finally, this commit also adds a new selftest specifically for verifying
correct behaviour for benchmarks that fail or skip.
Task-number: QTBUG-24313
Change-Id: I3426dc659a7511b62fd183a031c7235bc753f497
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, when an unexpected pass occurred the test output
showed the message "XPASS : tst_foo::function() 'expr' returned FALSE",
where the problem was actually that the expression evaluated to true
when it was expected to be false.
This commit changes the output to make it clear that the expression
evaluated to true unexpectedly.
Task-number: QTBUG-22118
Change-Id: Id22c178073d3b75789675ca37a8ef019029b1f91
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, the following statement in the qtestlib
documentation was untrue: "If init() fails, the following testfunction
will not be executed, the test will proceed to the next testfunction."
If init() called QSKIP, the test function would be skipped, but if
init() reported a failure, the test function would still be executed
(even though doing so could be unsafe).
This commit makes testlib skip a test function if init() reports a
failure and enhances the selftests to cover skips and fails in both
init() and cleanup().
Task-number: QTBUG-20371
Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
For data-driven tests, testlib previously counted one fail or skip for
each data row that failed or skipped, while it counted only one pass
for a test function where all rows passed and counted no passes for a
test function where some rows passed and some rows failed. A similar
problem also existed for benchmark tests, which could run multiple
iterations of the same test, with each fail and skip being counted but
only a single pass being counted for the entire series of iterations.
This commit makes testlib count one pass, fail or skip for each data
row. Test functions that are not data-driven count one result for the
test function, as before. Benchmark tests count one pass, fail or skip
per iteration.
A side-effect of this change is that the test output in plain text, xml
and light xml formats now shows a result for every data row and
benchmark iteration executed, allowing post-processors to correctly
calculate the total number of tests executed. Previously, individual
rows were not shown in the test output if they passed, making such
calculations impossible.
The only change to the xunitxml output format is to correct a bug where
no test result was recorded for a test function if the last data row
was skipped and all other rows passed -- in which case the overall
result should be a pass. Note that there is also a pre-existing bug
in the xunit logger, where no result is reported if all rows are
skipped; that bug is unaffected by this commit.
Task-number: QTBUG-21848
Task-number: QTBUG-22124
Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b
Reviewed-by: Ed Baak <ed.baak@nokia.com>
Set QT_TESTCOCOON_ACTIVE environment variable when the coverage is installed
for a test and unset it when the coverage data is saved. Tests that run when
QT_TESTCOCOON_ACTIVE is set are subtests and will not be considered as
stand-alone tests for the coverage.
When a test is run as a subtest its coverage data will not be saved for
itself but for the main test it is merged with. Also its status will not be
reported since only the status of the main test is expected in the test report,
e.g. the test tests/auto/testlib/selftests.
Change-Id: Icfdf99300aae18040e1a3441a8af21f68df4c0db
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Correct a bug in the test and use unique names for data rows. The bug
was that the test assumed that QCOMPARE for float values 100001 and
100002 would fail, but it actually succeeds. QCOMPARE for floats uses
qFuzzyCompare(), which succeeds if the numbers differ by no more than
1/100,000th of the smaller value. Thus QCOMPARE(100001, 100002) passes,
while QCOMPARE(99998, 99999) fails.
Change-Id: Ia35d3126c2e3ebe91d64daa309048514a365d9fb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Some of the subtests are only run with plain-text output format. For
those subtests, the other output formats were unused and gradually
becoming out-of-date.
Change-Id: I4c10f7f5bab2d2cc7d2d2ad641fbf5d4df02b798
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Our OSX 10.7 test machines send a SIGILL rather than a SIGSEGV to a
process which attempts to dereference a null pointer.
Change the "crashes" test to dereference an invalid pointer with a
value slightly greater than 0 so that we get the same crash behavior on
all (unix) platforms.
Change-Id: I700a2c7d654a9468af5e5996010a258695ed2ae5
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Previously, ignored messages were only cleared at the end of each test
function, i.e. after all data rows were finished. This meant that if a
data row in a data-driven test function didn't cause all of the expected
messages to be generated, the remaining messages would be carried over
to the next data row. This would result in errors about missing
messages being associated with the last data row rather than with the
correct data row.
This commit makes testlib check for missing ignored messages after
running each data row rather than only doing so after the last data
row.
This commit also adds a regression test to demonstrate that ignored
messages can no longer be carried over from one data row to another.
Change-Id: Ibee51aa6e96866fbcbcb4acee1a8340a86a6a4ba
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, expected failures were only cleared at the end of each test
function, i.e. after all data rows were finished. This meant that if a
data-driven test function called QEXPECT_FAIL and then didn't perform
any further verification steps to trigger the expected failure, the
expected failure would be carried over to the next data row, probably
causing the first verification step in the test function to XPASS (with
a seemingly irrelevant error message) for the next data row.
This commit adds the new function QTestResult::finishedCurrentTestData()
to cleanup after each data row is executed. This function treats calls
to QEXPECT_FAIL without subsequent verification steps as a test failure.
This commit also adds a regression test to demonstrate that expected
failures can no longer be carried over from one data row to another. If
run against the previous version of testlib, the new test would report a
pass instead of an error.
Change-Id: Ida5c7f080815b0dca9531131fed582b0918334cb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
- Needed to add toLower() for compare as c:\ and C:\ was causing
a failure.
Change-Id: Idd55774d118b7249e88362688166b4f9d9712c7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Rename some of the test functions to improve consistency, and change the
order so that all the xfail tests are together (some new xpass tests
will be added later).
Change-Id: Ice1ac5bf61bc26109bb0d08f72eb74e1a0424101
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The test simply generates each possible pair of Pass, Fail and Skip
results. At present the test simply serves to demonstrate the current
shortcomings of testlib's plain text logging, namely:
* If a test function passes for all data rows, that is counted as one
pass, but each skipped or failed row counts as one skip or fail.
* Only skipped and failed rows are reported individually in the test
output. Passed rows are not reported, so it is impossible to see how
many rows were executed.
* A skip followed by a pass will be reported as an overall pass for the
test function, but the same rows in reverse order will not report any
overall result for the test function.
Future commits will attempt to correct these problems.
Task-number: QTBUG-22124
Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test case for calling QEXPECT_FAIL in a data-driven test without
passing a specific data tag for the expected failure, in which the
expected failure should apply to the current data row.
Change-Id: I70a6592de77b1718c3a749fd2216e66f8f312805
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where a call to QEXPECT_FAIL nominates a
different data row from the current row, in which case the call to
QEXPECT_FAIL should be ignored.
Change-Id: Iff774b89f9374fa26105acdd8769177272a95229
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where QEXPECT_FAIL is called twice in a row
without an intervening verification step, which is supposed to result in
a failure.
Change-Id: Ie004652a76a2c35c2679945670ff5561e8f5b62e
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Make the test fail if calling QEXPECT_FAIL with the mode set to Abort
does not cause the current test function to abort. Previously, such an
error would not cause the selftest to fail and someone would have to see
the warning message in the test output to know that there is a problem.
Change-Id: I2bd58fafe6b51ea0ab86fde7c5afb781e4534ee4
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The existing expectfail selftest did not test QEXPECT_FAIL with a
data-driven test function. This commit adds such a test.
Change-Id: I39fa9aa227b58779ce5268dd37bf55468e7269c5
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test relies on being able to set a custom QLibraryInfo::PrefixPath
for the duration of the test. However, the test code neglected to
account for the following behavior on mac (from "Using qt.conf"
documentation):
On Mac OS X, the Prefix is relative to the Contents in the application
bundle.
Change-Id: Ie9b6d5ebfe8af7d7f332e827069b60a830d6c6f2
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- Changed to detect valgrind at runtime and skip test instead of failing.
- subprograms inherit QT_QPA_PLATFORM value from parent if set.
Change-Id: I280acee389df1ee74ee6758a0dd1601226e103c7
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- All subprograms installed as part of test instead of separate installs
- Added installTestHelperApp() to testcase.prf
Change-Id: I02fbbb299f095054c9216ad0e5f91f574fb0fe3d
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The previous change missed some headers from years prior to 2011, and a
few new files were merged after the previous change.
Change-Id: Ib7d1a2b7062228c2a5373da64242b2ee1f0981e1
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Changed selftests unittest to use specific classes of QtCore instead of
pulling in all of QtCore headers by using include <QtCore>
- Decreasing build time.
Change-Id: Ic53a70423ecafeb342cc05ae6d7dee795533d205
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Changed these tests to use QFINDTESTDATA macro to detect location of testdata.
Checking for a specific file contained in the testdata so as not to be confused by empty directories
created during configure.
Change-Id: Iac2ac6304b6b9ac79e00886025b93ec0af5a8507
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
... and deprecate the old registerTimer() functions. The new pure-
virtual registerTimer() breaks source-compatibility. Subclasses cannot
be instantiated anymore, since the pure virtual function signature has
changed.
QAbstractEventDispatcher::TimerInfo is no longer a QPair. It is now a
struct with timerId, interval, and timerType members. This is a source
incompatibility that should only affect subclasses of
QAbstractEventDispatcher, which will need to pass 3 arguments to the
TimerInfo constructor instead of 2. If the subclass used QPair<int,int>
instead of the TimerInfo typedef, the QPair<int,int> declarations will
need to be replaced with TimerInfo.
Call the new registerTimer() function with the type from
QObject::startTimer(). Change all subclasses of QAbstractEventDispatcher
to reimplement the new virtual function. The type argument is unused at
the momemnt, except to ensure that registeredTimers() returns the type
each timer was registered with. Implementations for the various
dispatchers will be done in separate commits.
Author: Thiago Macieira <thiago.macieira@nokia.com>
Change-Id: Ia22697e0ab0847810c5d162ef473e0e5a17a904b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Introduce a specialization for qCompare(QImage,QImage) that
checks isNull, size and format and outputs verbose messages.
Check isNull, size similarly for QPixmap.
Add an autotest:
- Add test to cmptest and make it a GUI application since
QImage requires QGuiApplication.
- Make testlib/selftests capable of running X11-GUI applications
by passing DISPLAY.
- Ignore stderr output for cmptest
- Add test data
Change-Id: I2b29c7822fbeedf2b22c90889739ed7ff859ce92
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Required for launching the sub-processes. The test worked only
when launched locally, not from 'make check'.
Change-Id: I42c9202a7726c3135f94445fb336b2b8241535a4
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@nokia.com>
Changed to use QT_INSTALL_TESTS to install subtests.
Change-Id: I4a2a7bd2d3e7d6da34dbb922bf377bee98cdedb0
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The old test only verified the behaviour of QCOMPARE when comparing
lists that were different. Add data rows for comparing empty lists and
non-empty lists that are equal.
Note that testlib currently does not report passing data rows (only
failing rows and completely passing test functions), so the new data
rows do not cause any change in the expected test output.
Change-Id: I137650ce0ca6250cee36bd9cb74b01f8abd4e89c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The (disabled) alive selftest contained a test for the QStringList
specialization of the QTest::qCompare template. This test is unrelated
to the rest of the alive selftest, so move it to the cmptest selftest,
where QCOMPARE is tested.
Change-Id: Ic6f0e491dd3b3ce8b4ca1d49666a099815575eaa
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The overload is used in Qt Creator
(see src/libs/extensionsystem/pluginmanager.cpp).
The use case here is an application whose internal
QObjects can be tested by passing a command line parameter.
For this use case, it is inconvenient to have to allocate
memory and create a char argv[]- array.
This reverts commit ad80d42f8e.
Change-Id: I2a2f91e2840100fd62743f6d03b33005d67b18f8
Reviewed-by: Daniel Teske <daniel.teske@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- Do not run with empty environment. At least PATH
is required at least (Qt + MSVC/MinGW runtime)
- Account for MSVC's different formatting of double numbers.
Change-Id: Ic7b1cf4a16a88c5384347b2651b011ac13c92d70
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The overload of QTest::qExec() that takes a QStringList is not used
anywhere in Qt's autotests, despite having been in the qtestlib API
since Qt 4.4.
This lack of use most likely derives from the fact that none of the
QTEST_MAIN macros use the overload, and more than 99% of Qt's tests
use those macros to avoid explicitly calling QTest::qExec().
Change-Id: I264b21d7fe1a9f2d565f748cf8bbe32414a73bb0
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
The subtests are intended to be executed by tst_selftests and therefore
don't need to have "CONFIG += testcase", which causes 'make check" to
treat them as stand-alone tests.
Change-Id: I1b47e0186be99c591bb1e03bbec6386518f8a449
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Make the test object names correspond to the test program names, so that
it is easier to see which test produced each block of test output.
Change-Id: Ib19a955938b549f27cdcaac83bfa9150285b396d
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test uses qt.conf to enforce a predictable tests install location.
This was not working in the shadow build case, as qt.conf was only
present in the source tree; embed it as a resource at :/qt/etc/qt.conf
so it works in the shadow build case too.
Change-Id: Iec708d9686c29a1fc5ff0020dc11a93a593d20c5
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Automated tests often need to load some data from external files.
Currently, a wide variety of approaches for this have been used in Qt
autotests, including:
- embed the source directory into the test binary at compile time, and
find the testdata relative to that; this fails when the source tree
is no longer available (e.g. when the tests are deployed to a device).
- use a path relative to the current working directory, and trust that
the caller always sets the current working directory such that the
testdata can be found; this fails when the caller uses a different
working directory than expected.
- use a path relative to QCoreApplication::applicationDirPath();
this fails when source tree != build tree (since testdata is not
automatically copied into the build tree).
- compile the files into the binary using the Qt resource system; this
should work, but does not allow for testing of code which genuinely
needs external files.
It seems that there is not a simple method for determining the testdata
path which can be reliably used in all circumstances, so various tests
have reinvented the testdata location method in different ways.
Therefore, this is a good candidate for an addition to the testlib API.
The current implementation of QFINDTESTDATA is able to find testdata
in all three of (build tree, install tree, source tree), in that order.
Change-Id: Ib2fed860723ccf437240da3b00db22dfe1a6b56c
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Each QVariant instance has internal storage which may be
used for well-know basic types. This patch changes the
behavior by delegating type dependent operation to QMetaType
class which knows more types than QVariant itself.
The patch significantly reduce amount of code in QVariant
implementation.
There are few side effects of this patch:
- better performance:
* for Core types when using Gui (QGuiVariant is able to
construct Core types)
* for small custom types (QVariant::Private::Data is used for all
types that has size small enough)
- comparing two QVariants can give different result for small custom
types (binary comparison instead of pointer comparison)
Change-Id: Ic17fa500d6a882110bfba896fd456c8e6c7a63a9
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
(Note: This feature is ported from Qt 4.8.
See the following commits:
01575deafb7d26ca2431374e92c6d71de96547c7
4866d1ba8afbab61e102942d1ea93b81fea053d6
)
Passing the -datatags option to a QTestLib program prints the
available data tags to standard output.
For completeness, the test case name is also printed
at the start of each output line. (Although the file name
is supposed to match the lower-case version of the test case
name, this is currently not true in all cases (particularly not
under tests/benchmarks/). Even if there was a script to enforce this
convention, the -datatags option provides this information in a
reliable way.)
Data tags for each test function (f() in this case) are printed in
four different ways depending on the presence of local and global
data tags:
Case 1: No tags:
tst_MyTestCasetst_MyTestCase f
Case 2: Local tags only:
tst_MyTestCase f local tag 1
tst_MyTestCase f local tag 2
...
Case 3: Global tags only:
tst_MyTestCase f __global__ global tag 1
tst_MyTestCase f __global__ global tag 2
...
Case 4: Local and global tags:
tst_MyTestCase f local tag 1 __global__ global tag 1
tst_MyTestCase f local tag 2 __global__ global tag 1
...
tst_MyTestCase f local tag 1 __global__ global tag 2
tst_MyTestCase f local tag 2 __global__ global tag 2
...
...
Note that the string __global__ is assumed to be highly unlikely to occur
in a data tag (if it does, an ambiguity results).
Change-Id: Ib51aa0c3c32ad52e52ce519729292cf8f0ec5d50
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@nokia.com>
qttest_p4.prf was added as a convenience for Qt's own autotests in Qt4.
It enables various crufty undocumented magic, of dubious value.
Stop using it, and explicitly enable the things from it which we want.
Change-Id: I02fe27b2c1800f929250fa8694ca2976c9661a12
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
When the SkipAll mode is used, tests only report a SKIP for the first
line of test data and subsequent lines are not reported at all. This
behaviour makes it impossible for anything post-processing test results
to accurately report test pass- and run- rates because they cannot see
how many lines of test data were skipped.
This commit removes SkipMode. QSKIPs in regular test functions and data
functions are treated the same as SkipSingle, so that every skipped line
of local or global test data is reported in the test log. QSKIPs
elsewhere are treated the same as SkipAll -- skipping in init() causes
the next test function to be skipped entirely, and skipping in
initTestCase() or initTestCase_data() causes all test functions to be
skipped.
This commit only changes qtestlib and the selftests. A further commit
will change the autotests to remove the SkipMode parameter from QSKIP
calls.
Note that the change in expected output for the globaldata selftest is
deliberate, as the QSKIP in the skipLocal test function has effectively
changed from SkipAll to SkipSingle.
Task-number: QTBUG-21851, QTBUG-21652
Change-Id: I7b1c53fe7ca9dde032810b789d967e2a402bbe5d
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Alex <alex.blasche@nokia.com>
Use a standard path for filenames that appear in the selftest's expected
test data. This will make future patches smaller.
Change-Id: I04b2e739d261f80d20b834e5b33c5b6e88d26379
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test duplicates the skipinitdata selftest and has slightly less
informative output.
Change-Id: Ifd40e3ef8030059ec8fa0089ce5b2a994624abeb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The addition of testing for multiple loggers greatly increased the total
runtime of this test, in fact making it the slowest testcase in Qt.
Fortunately this is only due to a couple of slow subtests whose behavior
is unlikely to be affected by the loggers. Change it to run these slow
subtests just for a couple of loggers, instead of all the combinations.
Change-Id: Ie90f6c0ca29470ed6a7c4e2e185f852602a2d162
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
This change makes QTest::qWarn() consistent with similar testlib
functions, which take __FILE__ and __LINE__.
Change-Id: I12977cb30672899ab38877b4a656f0cc0c7ea02c
Reviewed-on: http://codereview.qt-project.org/3559
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The selftests already used the @INSERT_QT_VERSION_HERE@ placeholder
in place of the real Qt and QTestlib versions in the expected test
output of the subtests. For unknown reasons, the same was not true for
the expected plain text output. In the past, this has caused Release
Managers to waste time incrementing the version numbers in these files.
Change-Id: I52f7870486fce128c04d53ff06978afa947474fd
Reviewed-on: http://codereview.qt-project.org/5375
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
There is no test with a datatag "float", so this code is never executed.
Change-Id: I9dd234e6575eb1c33075705edf77eb1d977061c9
Reviewed-on: http://codereview.qt-project.org/5362
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Q_ASSERT uses __FILE__, which has undefined contents, and thus the
selftest cannot always match Q_ASSERT messages in actual output with
those in expected output. The test was calling QEXPECT_FAIL() in this
case, but doing so implies that (a) it's a failure and (b) we intend to
fix it one day. Neither of these is true, so instead we should simply
tolerate this difference in actual and expected outputs, in the same way
that we do in a couple of other similar cases.
Change-Id: I8b8609a400e9b798936875f6eb1ca18a83660efc
Reviewed-on: http://codereview.qt-project.org/5358
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Each destination and the format of output to write there is specified by
adding "-o filename,format" to the command-line. The special filename
"-" indicates that the log output is written to the standard output
stream, though standard output can be used as a destination at most
once.
The old-style testlib output options are still supported, but can only
be used to specify one logging destination, as before.
If no logging options are given on the command-line, a plain text log
will go to the console, as before.
To log to the console in plain text and to the file "test_output" in
xunit format, one would invoke a test in the following way:
tst_foo -o test_output,xunitxml -o -,txt
This commit also enhances the selftests to test with multiple loggers,
but negative tests (e.g. bad combinations of command-line options) are
left for future task QTBUG-21567.
Task-number: QTBUG-20615
Change-Id: If91e752bc7001657e15e427aba9d25ab0a29a0b0
Reviewed-on: http://codereview.qt-project.org/4125
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Modify the selftest to be able to run each subtest with a list of one or
more test loggers. The addition of tests that use this capability will
be part of a subsequent commit.
Task-number: QTBUG-20615
Change-Id: Iac3efe8220e8245aa7e5589348d2c86b8034dd28
Reviewed-on: http://codereview.qt-project.org/5292
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The multiexec test runs the same test object five times. If the -o
option is given, the output file is overwritten by each run of the test
object, meaning that tst_selftest only sees 1/5 of the test output in a
file compared with what it sees on the console. This makes it
impossible to use the expected output file to verify the test for both
console and file output. This issue is noted in QTBUG-21561.
Change-Id: I00031a2ea43a7ef78e8317473a089306ec062d8e
Reviewed-on: http://codereview.qt-project.org/5270
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The warnings selftest verifies that qtestlib correctly handles output
sent via qDebug() and friends. The test had a number of printf calls
that were used to tell the user what output they should expect to see
next, but by bypassing testlib's logging, these made the testlog differ
depending on whether the test log was directed to the console or a file.
The printf calls are no longer needed, as any regressions in qDebug will
be detected by tst_selftests, which compares the output of the warnings
test with a reference copy of the expected output.
Change-Id: I8c3a3237463141fa494c50ca02062760cb583090
Reviewed-on: http://codereview.qt-project.org/5200
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Make the test use qDebug instead of printf, so that the reference test
output is the same regardless of whether the log goes to the console or
a file.
Also rename the test class, as it was evidently cut-n-pasted from the
subtest selftest without being renamed.
Change-Id: I8df1f0989f2e8c33c1f829ba8110fd740ff70f63
Reviewed-on: http://codereview.qt-project.org/5184
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test was attempting to verify two completely unrelated things, so
this commit splits it into two tests.
Also, printf calls are replaced by qDebug so that the test does not
bypass the testlib loggers.
Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee
Reviewed-on: http://codereview.qt-project.org/5178
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test is not useful for finding bugs in qtestlib's logging code,
because it bypasses the qtestlib loggers and doesn't play nice with
tst_selftest. Neither is this test very useful for finding bugs in
QTest::qWait(), as the test only proves the qWait() terminates, not that
it waits accurately, or even that it waits at all.
Change-Id: Ia5dd7cbaf3a6fbb4e94e54ed155263580e495694
Reviewed-on: http://codereview.qt-project.org/5173
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This is in preparation of removing testlib's
dependency on QtGui and QtWidgets.
Autotests that need QtWidgets api must
explicitly include it (since the types are no
longer provided by the QtGui master header).
Autotests that don't need QtGui or QtWidgets
api shouldn't link against those libraries.
Change-Id: I2808289068514fcac582808828ad4634e2631733
Reviewed-on: http://codereview.qt-project.org/5093
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Don't store separate strings for the logger name and the associated file
suffix -- just use the same string everywhere.
Change-Id: Ie7d1af6bf906b5ac09fbd5fcc5754b68036fb370
Reviewed-on: http://codereview.qt-project.org/5060
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The selftest was only attempting to delete the output file at the end of
the entire test run. This file should actually be deleted after each
subtest to ensure that it does not cause confusing errors if the next
subtest fails to overwrite the output file.
Change-Id: Ia3827926f71a697c6108de3f2b08c969f761d978
Reviewed-on: http://codereview.qt-project.org/5051
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The removed calls related to tests that were removed when redundant
logging functionality was removed.
Change-Id: Iaebbc109eaaddb440ae78e1f4a1b4e874a8b2960
Reviewed-on: http://codereview.qt-project.org/5050
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Use C++ comment delimiters consistently
* Use a space after control flow keywords
* Don't nest "else if" blocks unnecessarily
* Follow indenting rules
* Added comments describing some internal functions
Change-Id: I72fd1eaf5c5f3130945fbd2b64fa59e19ad2913d
Reviewed-on: http://codereview.qt-project.org/5049
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Commit 949be8cbdd modified the assert test
case but failed to update the line number of the assert in the expected
test output files.
Change-Id: I0d4f55c0b8bc1de8487dd8c6267aa459000cda26
Reviewed-on: http://codereview.qt-project.org/5048
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This change disables failing tests to increase the
likelihood that 'make check' terminates with a zero
exit status. This is done to prevent the CI system
from blocking further commits once qtbase#refactor
is merged into qtbase#master.
Failing tests are marked as insignificant (by putting
'CONFIG += insignificant_test' in the .pro file).
Note: This is a temporary measure that needs to be cleaned up
once the refactor->master integration is complete.
Ideally all disabled tests need to be enabled and passing.
All changes will be marked by the string QTBUG-21402 in a comment
for easy location.
Task-number: QTBUG-21402
Change-Id: Ic4400671671f6d9b75b106a6aa4755b20e74378d
Reviewed-on: http://codereview.qt-project.org/4678
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Gunnar Sletta <gunnar.sletta@nokia.com>
