Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.
Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
This mimics the effects of test-code delegating checks to other
functions, which can lead to repeated failures. The failing test
should only contribute one to the total of failed tsts. Drive-by:
correct an existing test's reporting of its own name.
Task-number: QTBUG-95661
Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
In selftests.qrc, imposed alphabetic order (on stem of name, then on
suffix, effectively treating . as sorting before any letter) while
removing old tests and adding new tests and data. Updated all non-csv
files and added many missing files. (Not clear on csv support status;
the script seems to have dropped it after 5.6, but the test still uses
it.)
Left expected_crashes* alone (no new files added, no update to old) as
I don't get results resembling those anticipated.
Omitted printdatatagswithglobaltags, printdatatags due to dangling
hspace on output lines, which upset sanity-bot. A change to the test
cpp is needed to make it viable to skip that dangling hspace.
Change-Id: Iab3fb626c44a91c249b2fb626c12c75ea0317098
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
This also has the advantage of not requiring the use of the locale
codec. Quite an advantage if you're debugging the locale codec. But it's
mostly so that we don't get question marks that hide the difference we
were trying to locate.
[ChangeLog][QtTest] QtTest now prints an escaped version of QStrings
that failed to compare with QCOMPARE. That is, instead of converting
non-printable characters to question marks, QtTest will print the
Unicode representation of the character in question.
Change-Id: I44c1ef3246b188c913dacd3ca4df02581356ea41
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
[ChangeLog][QtTest] Added test duration to xml output.
When running tests with xml output a new tag of the form
<duration msecs="123"/> is added to each test function and the test as a
whole.
Change-Id: Ibc4db066b6acf5fac6c578f5e5ca5ce4b5d8ea8e
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The newly added generate_expected_output.py was used to get the expected
output into a more reproducible state.
Change-Id: I1ca75c8e0c5778d25c1df531bd298007aac0ff4a
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
It's unexpected that all messages generated by the stream version
of qDebug and friends have a trailing space. It also makes switching
to categorized logging (which only supports the stream version) difficult,
since all autotests checking for debug output would have to be adapted.
Task-number: QTBUG-15256
Change-Id: I8d627a8379dc273d9689f5611184f03607b73823
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Before:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual): F0O
Expected (expected): FOO
Now:
FAIL! tst_testcase::testcase: Compared values are not the same
Actual (actual) : F0O
Expected (expected): FOO
Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Make the various versions of the failure message align consistently so
that it's a little easier to compare the actual and expected values. Of
course, the value won't align nicely unless the "actual" and "expected"
strings are the same length, but at least this commit makes that
consistent across all versions of the message.
Change-Id: If9ce231df3b5d279a06f6458fdb5da0aa4586068
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
For data-driven tests, testlib previously counted one fail or skip for
each data row that failed or skipped, while it counted only one pass
for a test function where all rows passed and counted no passes for a
test function where some rows passed and some rows failed. A similar
problem also existed for benchmark tests, which could run multiple
iterations of the same test, with each fail and skip being counted but
only a single pass being counted for the entire series of iterations.
This commit makes testlib count one pass, fail or skip for each data
row. Test functions that are not data-driven count one result for the
test function, as before. Benchmark tests count one pass, fail or skip
per iteration.
A side-effect of this change is that the test output in plain text, xml
and light xml formats now shows a result for every data row and
benchmark iteration executed, allowing post-processors to correctly
calculate the total number of tests executed. Previously, individual
rows were not shown in the test output if they passed, making such
calculations impossible.
The only change to the xunitxml output format is to correct a bug where
no test result was recorded for a test function if the last data row
was skipped and all other rows passed -- in which case the overall
result should be a pass. Note that there is also a pre-existing bug
in the xunit logger, where no result is reported if all rows are
skipped; that bug is unaffected by this commit.
Task-number: QTBUG-21848
Task-number: QTBUG-22124
Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b
Reviewed-by: Ed Baak <ed.baak@nokia.com>
This test was attempting to verify two completely unrelated things, so
this commit splits it into two tests.
Also, printf calls are replaced by qDebug so that the test does not
bypass the testlib loggers.
Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee
Reviewed-on: http://codereview.qt-project.org/5178
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
