AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
32df595275
qt5base-lts/tests/auto/testlib/selftests/expected_expectfail.tap
Edward Welbourne 784f5d76b9 TAP test logger: report B?XFAIL (mostly) as a message 
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.

So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.

This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).

Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.

Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-29 22:21:58 +02:00

269 lines
10 KiB
Plaintext
Raw Blame History

TAP version 13
# tst_ExpectFail
ok 1 - initTestCase()
not ok 2 - xfailAndContinue() # TODO This should xfail
---
extensions:
messages:
- severity: debug
message: begin
- severity: xfail
message: This should xfail
at: tst_ExpectFail::xfailAndContinue() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
- severity: debug
message: after
...
not ok 3 - xfailAndAbort() # TODO This should xfail
---
extensions:
messages:
- severity: debug
message: begin
- severity: xfail
message: This should xfail
at: tst_ExpectFail::xfailAndAbort() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 4 - xfailContinueSkip() # SKIP This skip should be reported and counted
---
at: tst_ExpectFail::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
extensions:
messages:
- severity: xfail
message: This should xfail then skip
at: tst_ExpectFail::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 5 - xfailAbortSkip() # TODO This should xfail
---
extensions:
messages:
- severity: xfail
message: This should xfail
at: tst_ExpectFail::xfailAbortSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 6 - xfailTwice()
---
# Already expecting a fail
at: tst_ExpectFail::xfailTwice() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 7 - xfailDataDrivenTwice(Pass Abort)
---
# Already expecting a fail
at: tst_ExpectFail::xfailDataDrivenTwice() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 8 - xfailDataDrivenTwice(Pass Continue)
---
# Already expecting a fail
at: tst_ExpectFail::xfailDataDrivenTwice() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 9 - xfailDataDrivenTwice(Fail Abort)
---
# Already expecting a fail
at: tst_ExpectFail::xfailDataDrivenTwice() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 10 - xfailDataDrivenTwice(Fail Continue)
---
# Already expecting a fail
at: tst_ExpectFail::xfailDataDrivenTwice() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 11 - xfailWithQString() # TODO A string
---
extensions:
messages:
- severity: xfail
message: A string
at: tst_ExpectFail::xfailWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
- severity: xfail
message: Bug 5 (The message)
at: tst_ExpectFail::xfailWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
ok 12 - xfailDataDrivenWithQString(Pass Abort) # SKIP Each Continue or Pass reports this and increments skip-count
ok 13 - xfailDataDrivenWithQString(Pass Continue) # SKIP Each Continue or Pass reports this and increments skip-count
not ok 14 - xfailDataDrivenWithQString(Fail Abort) # TODO A string
---
extensions:
messages:
- severity: xfail
message: A string
at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 15 - xfailDataDrivenWithQString(Fail Continue) # SKIP Each Continue or Pass reports this and increments skip-count
---
at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
extensions:
messages:
- severity: xfail
message: A string
at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
- severity: xfail
message: Bug 5 (The message)
at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
ok 16 - xfailDataDrivenWithQVerify(Pass Abort)
ok 17 - xfailDataDrivenWithQVerify(Pass Continue)
not ok 18 - xfailDataDrivenWithQVerify(Fail Abort) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 19 - xfailDataDrivenWithQVerify(Fail Continue) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
ok 20 - xfailDataDrivenWithQCompare(Pass Abort)
ok 21 - xfailDataDrivenWithQCompare(Pass Continue)
not ok 22 - xfailDataDrivenWithQCompare(Fail Abort) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 23 - xfailDataDrivenWithQCompare(Fail Continue) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
ok 24 - xfailOnWrongRow(Fail Abort)
---
extensions:
messages:
- severity: debug
message: Should pass (*not* xpass), despite test-case name
...
ok 25 - xfailOnWrongRow(Fail Continue)
---
extensions:
messages:
- severity: debug
message: Should pass (*not* xpass), despite test-case name
...
not ok 26 - xfailOnAnyRow(Fail Abort) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 27 - xfailOnAnyRow(Fail Continue) # TODO This test should xfail
---
extensions:
messages:
- severity: xfail
message: This test should xfail
at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
line: 0
...
not ok 28 - xfailWithoutCheck(Fail Abort)
---
# QEXPECT_FAIL was called without any subsequent verification statements
extensions:
messages:
- severity: debug
message: Should fail (*not* xfail), despite test-case name
...
not ok 29 - xfailWithoutCheck(Fail Continue)
---
# QEXPECT_FAIL was called without any subsequent verification statements
extensions:
messages:
- severity: debug
message: Should fail (*not* xfail), despite test-case name
...
ok 30 - xpassAbort() # TODO 'true' returned TRUE unexpectedly. ()
ok 31 - xpassAbortSkip() # TODO 'true' returned TRUE unexpectedly. ()
ok 32 - xpassAbortXfailContinue() # TODO 'true' returned TRUE unexpectedly. ()
ok 33 - xpassContinue() # TODO 'true' returned TRUE unexpectedly. ()
# debug: This should be reached
ok 34 - xpassContinueSkip() # TODO 'true' returned TRUE unexpectedly. ()
ok 34 - xpassContinueSkip() # SKIP This should be reached but not increment skip-count
ok 35 - xpassContinueXfailAbort() # TODO 'true' returned TRUE unexpectedly. ()
# xfail: This test should xfail but not add to totals
ok 36 - xpassAbortDataDrivenWithQVerify(XPass) # TODO 'true' returned TRUE unexpectedly. ()
ok 37 - xpassAbortDataDrivenWithQVerify(Pass)
ok 38 - xpassContinueDataDrivenWithQVerify(XPass) # TODO 'true' returned TRUE unexpectedly. ()
# debug: Test should Continue past XPASS
ok 39 - xpassContinueDataDrivenWithQVerify(Pass)
---
extensions:
messages:
- severity: debug
message: Test should simply PASS
...
ok 40 - xpassAbortDataDrivenWithQCompare(XPass) # TODO QCOMPARE(1, 1) returned TRUE unexpectedly.
ok 41 - xpassAbortDataDrivenWithQCompare(Pass)
ok 42 - xpassContinueDataDrivenWithQCompare(XPass) # TODO QCOMPARE(1, 1) returned TRUE unexpectedly.
# debug: Test should Continue past XPASS
ok 43 - xpassContinueDataDrivenWithQCompare(Pass)
---
extensions:
messages:
- severity: debug
message: Test should simply PASS
...
ok 44 - cleanupTestCase()
---
extensions:
messages:
- severity: debug
message: Totals should add up to 44: 23 passed, 17 failed, 4 skipped
...
1..44
# tests 44
# pass 23
# fail 17
Reference in New Issue View Git Blame Copy Permalink