We need to have a finer grained control over the tests
we skip in our CI system. This adds a blacklisting
mechanism that allows blacklisting individual test
functions (or even test data) using a set of predefined
matching keys for the operating system and some other
relevant variables.
QTestlib will search for a file called BLACKLIST in the test
directory and parse it if found. The file contains a simple
ini style list of functions to blacklist. For details see
qtestblacklist.cpp.
Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
GRAPHICS_ROOT and TZ environment variables are needed in child
processes in order to successfully run the auto test selftests.
Change-Id: I7befabd535b4c47b1e75acbe3d6158d0d9b811b3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Added the function pointer constructor to QSignalSpy to take advantage
of the new connect syntax.
Change-Id: I94218a096c677cdba73e1b2cfa8b9c09bc28145f
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Output the file name and fix line number to be 1..n.
Change-Id: I3ba7f84b99ce3664bc3e403b230307511b15fe1f
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
This also has the advantage of not requiring the use of the locale
codec. Quite an advantage if you're debugging the locale codec. But it's
mostly so that we don't get question marks that hide the difference we
were trying to locate.
[ChangeLog][QtTest] QtTest now prints an escaped version of QStrings
that failed to compare with QCOMPARE. That is, instead of converting
non-printable characters to question marks, QtTest will print the
Unicode representation of the character in question.
Change-Id: I44c1ef3246b188c913dacd3ca4df02581356ea41
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
This is only useful for logging benchmarks, since it won't print test
passes, failures, etc. It's useful for importing to spreadsheets to do
number-crunching.
[ChangeLog][QtTest]Added a CSV logging mode that is suitable for
importing benchmark results into spreadsheets. This can be enabled by
the -csv option on the command-line. The CSV logging mode will not print
test failures, debug messages, warnings, etc.
Change-Id: I245d6f86bb380645c9bc0d748cf474b3ed42cab8
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Task-number: QTBUG-35743
[ChangeLog][QtTest][Windows] Use correct UTF-8 encoding for XML test results on platforms with different console encoding.
Change-Id: Ice9d03192098f931e5dac358928e0c4421ab715e
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Do not apply additional encoding when reading process output.
Fixes errors when encountering UTF-8:
Traceback (most recent call last):
File "./generate_expected_output.py", line 117, in <module>
generateTestData(path)
File "./generate_expected_output.py", line 106, in generateTestData
out.write(data.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 5485: ordinal not in range(128)
Change-Id: Ib827787a59a18b4d3d0601645856517f43c01fc3
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Evaluate command line arguments and use directories only when empty.
Change-Id: I818ec13c686018a3f607e91174e57d8f8bbf15f7
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
The actual code in testlib now indents the duration tag when it is for a
function.
Change-Id: Iee62db9c81f11dc54e57f166bf9fb2b7012b7e03
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
[ChangeLog][QtTest] Added test duration to xml output.
When running tests with xml output a new tag of the form
<duration msecs="123"/> is added to each test function and the test as a
whole.
Change-Id: Ibc4db066b6acf5fac6c578f5e5ca5ce4b5d8ea8e
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The units that were supposed to have variance were completely off in the
test leading to all benchmarks with slight variance being completely
flaky.
Change-Id: Ib76593813974adee462b3f03be9370d2248d770c
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
This makes testlib a lot more hackable since when the output is changed
in a defined way we can regenerate the expected output (eg adding the
test duration which will be a follow-up patch).
Note that the script does not work properly on Windows and not all
benchmarks work, but at least it reduces the work of adapting files to a few lines.
Change-Id: I910387cd92ff82aa629747a3a3033dae17fbd711
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
The newly added generate_expected_output.py was used to get the expected
output into a more reproducible state.
Change-Id: I1ca75c8e0c5778d25c1df531bd298007aac0ff4a
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
It's unexpected that all messages generated by the stream version
of qDebug and friends have a trailing space. It also makes switching
to categorized logging (which only supports the stream version) difficult,
since all autotests checking for debug output would have to be adapted.
Task-number: QTBUG-15256
Change-Id: I8d627a8379dc273d9689f5611184f03607b73823
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
When comparing expected with actual debug messages, allow the expected
ones to contain a trailing space. This is needed to not break all
autotests in a follow up patch, which will prevent ~QDebug() from
generating a trailing space by default.
Task-number: QTBUG-15256
Change-Id: I7f67393ddfbfe37fde1ca5ef45e4ad7f4b5324b4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
For the conflicts in msvc_nmake.cpp the ifdefs are extended since we
need to support windows phone in the target branch while it is not there
in the current stable branch (as of Qt 5.2).
Conflicts:
configure
qmake/generators/win32/msvc_nmake.cpp
src/3rdparty/angle/src/libEGL/Surface.cpp
src/angle/src/common/common.pri
src/corelib/global/qglobal.h
src/corelib/io/qstandardpaths.cpp
src/plugins/platforms/qnx/qqnxintegration.cpp
src/plugins/platforms/qnx/qqnxscreeneventhandler.h
src/plugins/platforms/xcb/qglxintegration.h
src/widgets/kernel/win.pri
tests/auto/corelib/thread/qreadwritelock/tst_qreadwritelock.cpp
tests/auto/corelib/tools/qdatetime/tst_qdatetime.cpp
tests/auto/gui/text/qtextdocument/tst_qtextdocument.cpp
tools/configure/configureapp.cpp
Change-Id: I00b579eefebaf61d26ab9b00046d2b5bd5958812
New macro QVERIFY_EXCEPTION_THROWN may be used to check that some code
really throws an exception of specified type.
Change-Id: I7cf499c7c37a35407862bc604c6eb862c5f329d3
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Give processes a more complete environment (openSUSE
requires more variables to start an X11-process),
add test column indicating whether a test is
supposed to crash. If it is not, report crash reason.
Change-Id: I1e0ad59824963f69ee425f331d845741be2b3928
Reviewed-by: Tony Sarajärvi <tony.sarajarvi@digia.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Make it possible to match messages by a pattern.
Change-Id: I713312e86db5471755459f1ecc43e8f1ac7a95fb
Reviewed-by: Jason McDonald <macadder1@gmail.com>
- Remove irrelevant test subdirs via .pro files
- Follow WinCE codepaths where applicable
- Replace unsupported Win32 APIs with WinRT equivalents
This does not aim to fix any failures in the tests themselves; it only
makes them compile.
Change-Id: Ia82bc0cc402891f8f6238d4c261ee9152b51be80
Reviewed-by: Maurice Kalinowski <maurice.kalinowski@digia.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
A handful of tests lack QT_NO_PROCESS guards, making them non-compilable
on Qt builds with no QProcess support. This commit does not change
QProcess-specific tests, which should be left out of the build using the
.pro file mechanism.
Change-Id: Iac8562428abc1f59ccbb23bf5c3a919785e41f12
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
Before:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual): F0O
Expected (expected): FOO
Now:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual) : F0O
Expected (expected): FOO
Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef
Reviewed-by: Jason McDonald <macadder1@gmail.com>
"QTest" is the C++ namespace; "QtTest" is the library name
- Edited the logger output in qplaintestlogger.cpp
- Updated documentation
- Updated expected outputs for self-tests
Change-Id: I43c525c43221a8d4e843a00d6d55b0f06ef55fd7
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Remove all trailing whitespace from the following list of files:
*.cpp *.h *.conf *.qdoc *.pro *.pri *.mm *.rc *.pl *.qps *.xpm *.txt *README
excluding 3rdparty, test-data and auto generated code.
Note A): the only non 3rdparty c++-files that still
have trailing whitespace after this change are:
* src/corelib/codecs/cp949codetbl_p.h
* src/corelib/codecs/qjpunicode.cpp
* src/corelib/codecs/qbig5codec.cpp
* src/corelib/xml/qxmlstream_p.h
* src/tools/qdoc/qmlparser/qqmljsgrammar.cpp
* src/tools/uic/ui4.cpp
* tests/auto/other/qtokenautomaton/tokenizers/*
* tests/benchmarks/corelib/tools/qstring/data.cpp
* util/lexgen/tokenizer.cpp
Note B): in about 30 files some overlapping 'leading tab' and
'TAB character in non-leading whitespace' issues have been fixed
to make the sanity bot happy. Plus some general ws-fixes here
and there as asked for during review.
Change-Id: Ia713113c34d82442d6ce4d93d8b1cf545075d11d
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@digia.com>
Fix float format and exclude crashing sub-binary.
Task-number: QTBUG-29014
Change-Id: I404f971edeb128263122a194f23e2806d6fd3bd0
Reviewed-by: Kai Koehne <kai.koehne@digia.com>
Only QStringList was handled before, now any QList is handled.
A specialization for QStringList is still needed though, due to the way
template matching works.
Example with QList<int>. Before:
FAIL! : tst_QTextCodec::threadSafety() Compared values are not the same
Loc: [../tst_qtextcodec.cpp(2057)]
After:
FAIL! : tst_QTextCodec::threadSafety() Compared lists differ at index 0.
Actual (res2.toList()): '0'
Expected (mibList): '3'
Loc: [../tst_qtextcodec.cpp(2057)]
Change-Id: If0fdec3236ddb78a679ee549aba569ef5571c395
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
These types are either built-in or 'automatically declared' and so
don't need to be explicitly declared as metatypes.
Change-Id: Ifd116dee32a450ff89a9a1011e26b434765d6e95
Reviewed-by: David Faure <david.faure@kdab.com>
As they are built-in, they are effectively registered at compile-time
already.
Change-Id: I7ae6ba16088eab5d19213fa7b07c2a7760988a86
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@digia.com>
that way we don't have to auto-generate code for that in the configures.
note that we now load qt_build_config.prf instead of just qmodule.pri,
which means that exceptions_off is set everywhere. we forcibly re-enable
them for testcases to minimize the deviation from default 3rd party usage.
testlib selftests are not qt testcases, so the one that needs exceptions
needs to enable them explicitly.
Change-Id: I1b9360bb11f2e80c92a2b63a7c45991ad17fda1b
Reviewed-by: Joerg Bornemann <joerg.bornemann@digia.com>
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@digia.com>
If an URL is invalid, let's indicate that in the test output. To be
helpful, let's make QUrl::errorString() include the component form of
the URL.
Change-Id: Iaafe16973ded79c7ea688fbb23808d91253e8c14
Reviewed-by: David Faure <faure@kde.org>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
Previously, a QVariant parameter would be wrapped inside a new
QVariant, and you would have to cast the QSignalSpy's QVariant to
a QVariant to get the actual value. This behavior was unintuitive
and undocumented.
Check if the parameter type is QVariant, and copy it directly if it
is. This makes the QSignalSpy's QVariant directly usable (no need to
"unwrap" the value in user code).
Existing tests that use QSignalSpy together with QVariant parameters
(such as tst_QPropertyAnimation::valueChanged()) and do cast the
QVariant parameter to a QVariant, continue to work after this change;
this is because qvariant_cast<QVariant>() returns its input value
(unchanged) when the type is not QMetaType::QVariant.
Task-number: QTBUG-21645
Change-Id: Ibfb171edd60c0d3f7ca1d5419e5c5f3d0380d5b3
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Qt 5.0 beta requires changing the default to the 5.0 API, disabling
the deprecated code. However, tests should test (and often do) the
compatibility API too, so turn it back on.
Task-number: QTBUG-25053
Change-Id: I8129c3ef3cb58541c95a32d083850d9e7f768927
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
The previous API was hard to use (global function, no type safety,
manual chaining), and confusing (app vs dispatcher split only made
sense on Windows). Installing and removing out of order would have
the risk of setting back a dangling pointer (crash). Meanwhile QPA
added type safety, and this new API models the QObject::installEventFilter
API for ease of use. The virtual method is in a new interface,
QAbstractNativeEventFilter.
QPA was even calling the dispatcher event filter with QPA-private event
classes, which made no sense (refactoring leftover from when the code
was in the dispatcher). Now the QPA plugins trigger the qcoreapp event
filters with the actual native events directly.
Change-Id: Ie35e47c59c862383bcaf857b28d54f7c72547882
Reviewed-by: Marius Storm-Olsen <marius.storm-olsen@nokia.com>
Make test projects declare TEST_HELPER_INSTALLS rather than calling a
function exported by testcase.prf. load(testcase) may be unsafe, as
testcase.prf should be processed after default_post.prf.
Fixes silent disabling of various autotests.
Change-Id: I56b35ffd653a637ad5ab18d64dd1a1edadfac59f
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Future-proofing. Since Qt source code is now mandated to be in UTF-8,
it is entirely possible that someone will use non-ASCII in data tags.
Though it would be interesting to see how to access them from the
Windows command-line.
Change-Id: I880fc312432b62143888ff1e1d9abbd54f704601
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
