In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.
Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.
However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.
In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)
Add methods to QTestCharBuffer, and a helper function, to support this.
Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
This solves the long-standing problem of not being able to easily
fail a test when a certain warning is output.
[ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test
function, any warning that matches the given pattern will cause a test
failure. The test will continue execution when a failure is added.
All patterns are cleared at the end of each test function.
Fixes: QTBUG-70029
Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
addBFail() asserts on the file being non-null. The convention seems to
be "Unknown File" for cases where we cannot determine the file.
Change-Id: I3a4d0130352d77d75f264fad6f3bd47c6700ef4c
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
