AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
95e4996688
qt5base-lts/tests/auto/testlib/selftests/expected_expectfail.txt
Edward Welbourne be72cb9cfd Include check for the Totals line in testlib's expectedfail selftest 
Count how many we expect to pass, fail and be skipped, so we can
report that for comparison with the actual totals line: this reveals
some double-counting, marked with a FIXME.

Task-number: QTBUG-95661
Change-Id: I8b0f13ded5202ed476d8abfee70ed60f9e639bf9
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-08-27 20:16:18 +02:00

115 lines
8.6 KiB
Plaintext
Raw Blame History

********* Start testing of tst_ExpectFail *********
Config: Using QtTest library
PASS : tst_ExpectFail::initTestCase()
QDEBUG : tst_ExpectFail::xfailAndContinue() begin
XFAIL : tst_ExpectFail::xfailAndContinue() This should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
QDEBUG : tst_ExpectFail::xfailAndContinue() after
PASS : tst_ExpectFail::xfailAndContinue()
QDEBUG : tst_ExpectFail::xfailAndAbort() begin
XFAIL : tst_ExpectFail::xfailAndAbort() This should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailAndAbort()
XFAIL : tst_ExpectFail::xfailContinueSkip() This should xfail then skip
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
SKIP : tst_ExpectFail::xfailContinueSkip() This skip should be reported and counted
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xfailAbortSkip() This should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailAbortSkip()
FAIL! : tst_ExpectFail::xfailTwice() Already expecting a fail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
FAIL! : tst_ExpectFail::xfailDataDrivenTwice(Pass Abort) Already expecting a fail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
FAIL! : tst_ExpectFail::xfailDataDrivenTwice(Pass Continue) Already expecting a fail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
FAIL! : tst_ExpectFail::xfailDataDrivenTwice(Fail Abort) Already expecting a fail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
FAIL! : tst_ExpectFail::xfailDataDrivenTwice(Fail Continue) Already expecting a fail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xfailWithQString() A string
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xfailWithQString() Bug 5 (The message)
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailWithQString()
SKIP : tst_ExpectFail::xfailDataDrivenWithQString(Pass Abort) Each Continue or Pass reports this and increments skip-count
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
SKIP : tst_ExpectFail::xfailDataDrivenWithQString(Pass Continue) Each Continue or Pass reports this and increments skip-count
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xfailDataDrivenWithQString(Fail Abort) A string
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQString(Fail Abort)
XFAIL : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) A string
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) Bug 5 (The message)
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
SKIP : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) Each Continue or Pass reports this and increments skip-count
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass Abort)
PASS : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass Continue)
XFAIL : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Abort) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Abort)
XFAIL : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Continue) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Continue)
PASS : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass Abort)
PASS : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass Continue)
XFAIL : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Abort) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Abort)
XFAIL : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Continue) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Continue)
QDEBUG : tst_ExpectFail::xfailOnWrongRow(Fail Abort) Should pass (*not* xpass), despite test-case name
PASS : tst_ExpectFail::xfailOnWrongRow(Fail Abort)
QDEBUG : tst_ExpectFail::xfailOnWrongRow(Fail Continue) Should pass (*not* xpass), despite test-case name
PASS : tst_ExpectFail::xfailOnWrongRow(Fail Continue)
XFAIL : tst_ExpectFail::xfailOnAnyRow(Fail Abort) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailOnAnyRow(Fail Abort)
XFAIL : tst_ExpectFail::xfailOnAnyRow(Fail Continue) This test should xfail
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xfailOnAnyRow(Fail Continue)
QDEBUG : tst_ExpectFail::xfailWithoutCheck(Fail Abort) Should fail (*not* xfail), despite test-case name
FAIL! : tst_ExpectFail::xfailWithoutCheck(Fail Abort) QEXPECT_FAIL was called without any subsequent verification statements
QDEBUG : tst_ExpectFail::xfailWithoutCheck(Fail Continue) Should fail (*not* xfail), despite test-case name
FAIL! : tst_ExpectFail::xfailWithoutCheck(Fail Continue) QEXPECT_FAIL was called without any subsequent verification statements
XPASS : tst_ExpectFail::xpassAbort() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XPASS : tst_ExpectFail::xpassAbortSkip() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XPASS : tst_ExpectFail::xpassAbortXfailContinue() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XPASS : tst_ExpectFail::xpassContinue() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
QDEBUG : tst_ExpectFail::xpassContinue() This should be reached
XPASS : tst_ExpectFail::xpassContinueSkip() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
SKIP : tst_ExpectFail::xpassContinueSkip() This should be reached but not increment skip-count
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XPASS : tst_ExpectFail::xpassContinueXfailAbort() 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XFAIL : tst_ExpectFail::xpassContinueXfailAbort() This test should xfail but not add to totals
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
XPASS : tst_ExpectFail::xpassAbortDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xpassAbortDataDrivenWithQVerify(Pass)
XPASS : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(XPass) Test should Continue past XPASS
QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(Pass) Test should simply PASS
PASS : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(Pass)
XPASS : tst_ExpectFail::xpassAbortDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
PASS : tst_ExpectFail::xpassAbortDataDrivenWithQCompare(Pass)
XPASS : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(XPass) Test should Continue past XPASS
QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(Pass) Test should simply PASS
PASS : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(Pass)
QDEBUG : tst_ExpectFail::cleanupTestCase() Totals should add up to 44: 23 passed, 17 failed, 4 skipped
PASS : tst_ExpectFail::cleanupTestCase()
Totals: 23 passed, 17 failed, 5 skipped, 0 blacklisted, 0ms
********* Finished testing of tst_ExpectFail *********
Reference in New Issue View Git Blame Copy Permalink