Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as
"COMPARE()", making it impossible to see what was compared and difficult
to match the output to the source of a test containing many calls to
QCOMPARE.
This commit changes testlib's internal compare_helper API so that string
representations of the compared expressions are always passed to
QTestResult::compare() when available, and can thus be shown in the
verbose and XPASS output. The XPASS output has also been changed to
state explicitly that the comparison succeeded unexpectedly, bringing it
in line with the XPASS output resulting from a call to QVERIFY.
This commit also changes all calls to compare_helper() to call the
eight-argument version of the function, which simplifies much of the
calling code. The now obsolete four-argument version of
compare_helper() has been changed to output a warning that it is
obsolete. It will be removed once other modules have had some time to
catch up.
The improved XPASS and verbose output is demonstrated by the expectfail
and verbose2 selftests.
Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, when an unexpected pass occurred the test output
showed the message "XPASS : tst_foo::function() 'expr' returned FALSE",
where the problem was actually that the expression evaluated to true
when it was expected to be false.
This commit changes the output to make it clear that the expression
evaluated to true unexpectedly.
Task-number: QTBUG-22118
Change-Id: Id22c178073d3b75789675ca37a8ef019029b1f91
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
For data-driven tests, testlib previously counted one fail or skip for
each data row that failed or skipped, while it counted only one pass
for a test function where all rows passed and counted no passes for a
test function where some rows passed and some rows failed. A similar
problem also existed for benchmark tests, which could run multiple
iterations of the same test, with each fail and skip being counted but
only a single pass being counted for the entire series of iterations.
This commit makes testlib count one pass, fail or skip for each data
row. Test functions that are not data-driven count one result for the
test function, as before. Benchmark tests count one pass, fail or skip
per iteration.
A side-effect of this change is that the test output in plain text, xml
and light xml formats now shows a result for every data row and
benchmark iteration executed, allowing post-processors to correctly
calculate the total number of tests executed. Previously, individual
rows were not shown in the test output if they passed, making such
calculations impossible.
The only change to the xunitxml output format is to correct a bug where
no test result was recorded for a test function if the last data row
was skipped and all other rows passed -- in which case the overall
result should be a pass. Note that there is also a pre-existing bug
in the xunit logger, where no result is reported if all rows are
skipped; that bug is unaffected by this commit.
Task-number: QTBUG-21848
Task-number: QTBUG-22124
Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b
Reviewed-by: Ed Baak <ed.baak@nokia.com>
Previously, expected failures were only cleared at the end of each test
function, i.e. after all data rows were finished. This meant that if a
data-driven test function called QEXPECT_FAIL and then didn't perform
any further verification steps to trigger the expected failure, the
expected failure would be carried over to the next data row, probably
causing the first verification step in the test function to XPASS (with
a seemingly irrelevant error message) for the next data row.
This commit adds the new function QTestResult::finishedCurrentTestData()
to cleanup after each data row is executed. This function treats calls
to QEXPECT_FAIL without subsequent verification steps as a test failure.
This commit also adds a regression test to demonstrate that expected
failures can no longer be carried over from one data row to another. If
run against the previous version of testlib, the new test would report a
pass instead of an error.
Change-Id: Ida5c7f080815b0dca9531131fed582b0918334cb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Rename some of the test functions to improve consistency, and change the
order so that all the xfail tests are together (some new xpass tests
will be added later).
Change-Id: Ice1ac5bf61bc26109bb0d08f72eb74e1a0424101
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test case for calling QEXPECT_FAIL in a data-driven test without
passing a specific data tag for the expected failure, in which the
expected failure should apply to the current data row.
Change-Id: I70a6592de77b1718c3a749fd2216e66f8f312805
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where a call to QEXPECT_FAIL nominates a
different data row from the current row, in which case the call to
QEXPECT_FAIL should be ignored.
Change-Id: Iff774b89f9374fa26105acdd8769177272a95229
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where QEXPECT_FAIL is called twice in a row
without an intervening verification step, which is supposed to result in
a failure.
Change-Id: Ie004652a76a2c35c2679945670ff5561e8f5b62e
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The existing expectfail selftest did not test QEXPECT_FAIL with a
data-driven test function. This commit adds such a test.
Change-Id: I39fa9aa227b58779ce5268dd37bf55468e7269c5
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Use a standard path for filenames that appear in the selftest's expected
test data. This will make future patches smaller.
Change-Id: I04b2e739d261f80d20b834e5b33c5b6e88d26379
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The selftests already used the @INSERT_QT_VERSION_HERE@ placeholder
in place of the real Qt and QTestlib versions in the expected test
output of the subtests. For unknown reasons, the same was not true for
the expected plain text output. In the past, this has caused Release
Managers to waste time incrementing the version numbers in these files.
Change-Id: I52f7870486fce128c04d53ff06978afa947474fd
Reviewed-on: http://codereview.qt-project.org/5375
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
