Automated tests often need to load some data from external files.
Currently, a wide variety of approaches for this have been used in Qt
autotests, including:
- embed the source directory into the test binary at compile time, and
find the testdata relative to that; this fails when the source tree
is no longer available (e.g. when the tests are deployed to a device).
- use a path relative to the current working directory, and trust that
the caller always sets the current working directory such that the
testdata can be found; this fails when the caller uses a different
working directory than expected.
- use a path relative to QCoreApplication::applicationDirPath();
this fails when source tree != build tree (since testdata is not
automatically copied into the build tree).
- compile the files into the binary using the Qt resource system; this
should work, but does not allow for testing of code which genuinely
needs external files.
It seems that there is not a simple method for determining the testdata
path which can be reliably used in all circumstances, so various tests
have reinvented the testdata location method in different ways.
Therefore, this is a good candidate for an addition to the testlib API.
The current implementation of QFINDTESTDATA is able to find testdata
in all three of (build tree, install tree, source tree), in that order.
Change-Id: Ib2fed860723ccf437240da3b00db22dfe1a6b56c
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
(Note: This feature is ported from Qt 4.8.
See the following commits:
01575deafb7d26ca2431374e92c6d71de96547c7
4866d1ba8afbab61e102942d1ea93b81fea053d6
)
Passing the -datatags option to a QTestLib program prints the
available data tags to standard output.
For completeness, the test case name is also printed
at the start of each output line. (Although the file name
is supposed to match the lower-case version of the test case
name, this is currently not true in all cases (particularly not
under tests/benchmarks/). Even if there was a script to enforce this
convention, the -datatags option provides this information in a
reliable way.)
Data tags for each test function (f() in this case) are printed in
four different ways depending on the presence of local and global
data tags:
Case 1: No tags:
tst_MyTestCasetst_MyTestCase f
Case 2: Local tags only:
tst_MyTestCase f local tag 1
tst_MyTestCase f local tag 2
...
Case 3: Global tags only:
tst_MyTestCase f __global__ global tag 1
tst_MyTestCase f __global__ global tag 2
...
Case 4: Local and global tags:
tst_MyTestCase f local tag 1 __global__ global tag 1
tst_MyTestCase f local tag 2 __global__ global tag 1
...
tst_MyTestCase f local tag 1 __global__ global tag 2
tst_MyTestCase f local tag 2 __global__ global tag 2
...
...
Note that the string __global__ is assumed to be highly unlikely to occur
in a data tag (if it does, an ambiguity results).
Change-Id: Ib51aa0c3c32ad52e52ce519729292cf8f0ec5d50
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@nokia.com>
This test duplicates the skipinitdata selftest and has slightly less
informative output.
Change-Id: Ifd40e3ef8030059ec8fa0089ce5b2a994624abeb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The addition of testing for multiple loggers greatly increased the total
runtime of this test, in fact making it the slowest testcase in Qt.
Fortunately this is only due to a couple of slow subtests whose behavior
is unlikely to be affected by the loggers. Change it to run these slow
subtests just for a couple of loggers, instead of all the combinations.
Change-Id: Ie90f6c0ca29470ed6a7c4e2e185f852602a2d162
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
The selftests already used the @INSERT_QT_VERSION_HERE@ placeholder
in place of the real Qt and QTestlib versions in the expected test
output of the subtests. For unknown reasons, the same was not true for
the expected plain text output. In the past, this has caused Release
Managers to waste time incrementing the version numbers in these files.
Change-Id: I52f7870486fce128c04d53ff06978afa947474fd
Reviewed-on: http://codereview.qt-project.org/5375
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
There is no test with a datatag "float", so this code is never executed.
Change-Id: I9dd234e6575eb1c33075705edf77eb1d977061c9
Reviewed-on: http://codereview.qt-project.org/5362
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Q_ASSERT uses __FILE__, which has undefined contents, and thus the
selftest cannot always match Q_ASSERT messages in actual output with
those in expected output. The test was calling QEXPECT_FAIL() in this
case, but doing so implies that (a) it's a failure and (b) we intend to
fix it one day. Neither of these is true, so instead we should simply
tolerate this difference in actual and expected outputs, in the same way
that we do in a couple of other similar cases.
Change-Id: I8b8609a400e9b798936875f6eb1ca18a83660efc
Reviewed-on: http://codereview.qt-project.org/5358
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Each destination and the format of output to write there is specified by
adding "-o filename,format" to the command-line. The special filename
"-" indicates that the log output is written to the standard output
stream, though standard output can be used as a destination at most
once.
The old-style testlib output options are still supported, but can only
be used to specify one logging destination, as before.
If no logging options are given on the command-line, a plain text log
will go to the console, as before.
To log to the console in plain text and to the file "test_output" in
xunit format, one would invoke a test in the following way:
tst_foo -o test_output,xunitxml -o -,txt
This commit also enhances the selftests to test with multiple loggers,
but negative tests (e.g. bad combinations of command-line options) are
left for future task QTBUG-21567.
Task-number: QTBUG-20615
Change-Id: If91e752bc7001657e15e427aba9d25ab0a29a0b0
Reviewed-on: http://codereview.qt-project.org/4125
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Modify the selftest to be able to run each subtest with a list of one or
more test loggers. The addition of tests that use this capability will
be part of a subsequent commit.
Task-number: QTBUG-20615
Change-Id: Iac3efe8220e8245aa7e5589348d2c86b8034dd28
Reviewed-on: http://codereview.qt-project.org/5292
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The multiexec test runs the same test object five times. If the -o
option is given, the output file is overwritten by each run of the test
object, meaning that tst_selftest only sees 1/5 of the test output in a
file compared with what it sees on the console. This makes it
impossible to use the expected output file to verify the test for both
console and file output. This issue is noted in QTBUG-21561.
Change-Id: I00031a2ea43a7ef78e8317473a089306ec062d8e
Reviewed-on: http://codereview.qt-project.org/5270
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test was attempting to verify two completely unrelated things, so
this commit splits it into two tests.
Also, printf calls are replaced by qDebug so that the test does not
bypass the testlib loggers.
Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee
Reviewed-on: http://codereview.qt-project.org/5178
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test is not useful for finding bugs in qtestlib's logging code,
because it bypasses the qtestlib loggers and doesn't play nice with
tst_selftest. Neither is this test very useful for finding bugs in
QTest::qWait(), as the test only proves the qWait() terminates, not that
it waits accurately, or even that it waits at all.
Change-Id: Ia5dd7cbaf3a6fbb4e94e54ed155263580e495694
Reviewed-on: http://codereview.qt-project.org/5173
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Don't store separate strings for the logger name and the associated file
suffix -- just use the same string everywhere.
Change-Id: Ie7d1af6bf906b5ac09fbd5fcc5754b68036fb370
Reviewed-on: http://codereview.qt-project.org/5060
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The selftest was only attempting to delete the output file at the end of
the entire test run. This file should actually be deleted after each
subtest to ensure that it does not cause confusing errors if the next
subtest fails to overwrite the output file.
Change-Id: Ia3827926f71a697c6108de3f2b08c969f761d978
Reviewed-on: http://codereview.qt-project.org/5051
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The removed calls related to tests that were removed when redundant
logging functionality was removed.
Change-Id: Iaebbc109eaaddb440ae78e1f4a1b4e874a8b2960
Reviewed-on: http://codereview.qt-project.org/5050
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Use C++ comment delimiters consistently
* Use a space after control flow keywords
* Don't nest "else if" blocks unnecessarily
* Follow indenting rules
* Added comments describing some internal functions
Change-Id: I72fd1eaf5c5f3130945fbd2b64fa59e19ad2913d
Reviewed-on: http://codereview.qt-project.org/5049
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
