AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

testlib: Clean up tst_blacklist

Browse Source 
33d7f76f0e attempted to exercise more
of the blacklisting code, but in doing so didn't account for the
fact that the features it added didn't actually work.

Adding a global blacklisting of * will result in blacklisting
all tests, regardless of any inverse conditions "!*" later on,
such as for the 'pass' test. This would have been evident by
looking at the expectation files, which still showed the test
as a BPASS.

The broken feature should be fixed, but in the meantime let's
clean up the test so it doesn't misrepresent reality.

At the same time we also remove the messages() tests, which was
copied straight out of tst_silent, but doesn't serve any purpose
for the blacklist selftest.

Pick-to: 6.2
Change-Id: I20e4693300aad5e9ff5f17ad22e3dfcb5c49fd0e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
This commit is contained in:
Tor Arne Vestbø 2021-08-01 15:17:17 +02:00
parent c95d7741b7
commit a71b9c9377
8 changed files with 16 additions and 141 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
tests/auto/testlib/selftests/blacklisted/BLACKLIST
View File

@ -1,8 +1,7 @@
obscure # no such platform; is ignored obscure # no such platform; is ignored
*
[pass] [pass]
!* *
[skip] [skip]
* *
@ -15,6 +14,3 @@ obscure # no such platform; is ignored
[xpass] [xpass]
* *
[messages]
*

36
tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
View File

@ -40,13 +40,9 @@ private slots:
void fail(); void fail();
void xfail(); void xfail();
void xpass(); void xpass();
// This test function must be last, as it calls qFatal().
void messages();
}; };
// All the tests below except pass() have been blacklisted in blacklisted/BLACKLIST // All the tests below have been blacklisted in blacklisted/BLACKLIST
// Contrast with ../silent/, for the same tests without blacklisting but with -silent
void tst_Blacklisted::pass() void tst_Blacklisted::pass()
{ {
@ -75,35 +71,5 @@ void tst_Blacklisted::xpass()
QVERIFY2(true, "This test should BXPASS"); QVERIFY2(true, "This test should BXPASS");
} }
#ifndef Q_OS_WIN
#include <signal.h>
#include <setjmp.h>
static jmp_buf state;
static void abort_handler(int)
{
longjmp(state, 1);
}
#endif
void tst_Blacklisted::messages()
{
qWarning("This is a warning that should not appear in silent test output");
QTestLog::warn("This is an internal testlib warning that should not appear in silent test output", __FILE__, __LINE__);
qDebug("This is a debug message that should not appear in silent test output");
qCritical("This is a critical message that should not appear in silent test output");
qInfo("This is an info message that should not appear in silent test output");
QTestLog::info("This is an internal testlib info message that should not appear in silent test output", __FILE__, __LINE__);
#ifndef Q_OS_WIN
// We're testing qFatal, but we don't want to actually std::abort() !
auto prior = signal(SIGABRT, abort_handler);
if (setjmp(state))
signal(SIGABRT, prior);
else
#endif
qFatal("This is a fatal error message that should still appear in silent test output");
}
QTEST_MAIN(tst_Blacklisted) QTEST_MAIN(tst_Blacklisted)
#include "tst_blacklisted.moc" #include "tst_blacklisted.moc"

25
tests/auto/testlib/selftests/expected_blacklisted.junitxml
View File

@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8" ?> <?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="tst_Blacklisted" timestamp="@TEST_START_TIME@" hostname="@HOSTNAME@" tests="7" failures="3" errors="0" skipped="1" time="@TEST_DURATION@"> <testsuite name="tst_Blacklisted" timestamp="@TEST_START_TIME@" hostname="@HOSTNAME@" tests="7" failures="2" errors="0" skipped="1" time="@TEST_DURATION@">
<properties> <properties>
<property name="QTestVersion" value="@INSERT_QT_VERSION_HERE@"/> <property name="QTestVersion" value="@INSERT_QT_VERSION_HERE@"/>
<property name="QtVersion" value="@INSERT_QT_VERSION_HERE@"/> <property name="QtVersion" value="@INSERT_QT_VERSION_HERE@"/>
@ -13,24 +13,7 @@
<testcase name="fail" classname="tst_Blacklisted" time="@TEST_DURATION@"/> <testcase name="fail" classname="tst_Blacklisted" time="@TEST_DURATION@"/>
<testcase name="xfail" classname="tst_Blacklisted" time="@TEST_DURATION@"/> <testcase name="xfail" classname="tst_Blacklisted" time="@TEST_DURATION@"/>
<testcase name="xpass" classname="tst_Blacklisted" time="@TEST_DURATION@"/> <testcase name="xpass" classname="tst_Blacklisted" time="@TEST_DURATION@"/>
<testcase name="messages" classname="tst_Blacklisted" time="@TEST_DURATION@"> <testcase name="cleanupTestCase" classname="tst_Blacklisted" time="@TEST_DURATION@"/>
<!-- type="qwarn" message="This is a warning that should not appear in silent test output" --> <system-out/>
<!-- type="warn" message="This is an internal testlib warning that should not appear in silent test output" --> <system-err/>
<!-- type="qdebug" message="This is a debug message that should not appear in silent test output" -->
<!-- type="system" message="This is a critical message that should not appear in silent test output" -->
<!-- type="qinfo" message="This is an info message that should not appear in silent test output" -->
<!-- type="info" message="This is an internal testlib info message that should not appear in silent test output" -->
<!-- type="qfatal" message="This is a fatal error message that should still appear in silent test output" -->
</testcase>
<system-out>
<![CDATA[This is a debug message that should not appear in silent test output]]>
<![CDATA[This is a critical message that should not appear in silent test output]]>
<![CDATA[This is an info message that should not appear in silent test output]]>
<![CDATA[This is an internal testlib info message that should not appear in silent test output]]>
</system-out>
<system-err>
<![CDATA[This is a warning that should not appear in silent test output]]>
<![CDATA[This is an internal testlib warning that should not appear in silent test output]]>
<![CDATA[This is a fatal error message that should still appear in silent test output]]>
</system-err>
</testsuite> </testsuite>

27
tests/auto/testlib/selftests/expected_blacklisted.lightxml
View File

@ -36,31 +36,8 @@
</Incident> </Incident>
<Duration msecs="0"/> <Duration msecs="0"/>
</TestFunction> </TestFunction>
<TestFunction name="messages"> <TestFunction name="cleanupTestCase">
<Message type="qwarn" file="" line="0"> <Incident type="pass" file="" line="0" />
<Description><![CDATA[This is a warning that should not appear in silent test output]]></Description>
</Message>
<Message type="warn" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This is an internal testlib warning that should not appear in silent test output]]></Description>
</Message>
<Message type="qdebug" file="" line="0">
<Description><![CDATA[This is a debug message that should not appear in silent test output]]></Description>
</Message>
<Message type="system" file="" line="0">
<Description><![CDATA[This is a critical message that should not appear in silent test output]]></Description>
</Message>
<Message type="qinfo" file="" line="0">
<Description><![CDATA[This is an info message that should not appear in silent test output]]></Description>
</Message>
<Message type="info" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This is an internal testlib info message that should not appear in silent test output]]></Description>
</Message>
<Message type="qfatal" file="" line="0">
<Description><![CDATA[This is a fatal error message that should still appear in silent test output]]></Description>
</Message>
<Incident type="bfail" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[Received a fatal error.]]></Description>
</Incident>
<Duration msecs="0"/> <Duration msecs="0"/>
</TestFunction> </TestFunction>
<Duration msecs="0"/> <Duration msecs="0"/>

17
tests/auto/testlib/selftests/expected_blacklisted.tap
View File

@ -23,21 +23,8 @@ not ok 6 - xfail() # TODO This test should BXFAIL then BPASS
line: 0 line: 0
... ...
ok 7 - xpass() # TODO 'true' returned TRUE unexpectedly. (This test should BXPASS) ok 7 - xpass() # TODO 'true' returned TRUE unexpectedly. (This test should BXPASS)
# This is a warning that should not appear in silent test output ok 8 - cleanupTestCase()
# This is an internal testlib warning that should not appear in silent test output
# This is a debug message that should not appear in silent test output
# This is a critical message that should not appear in silent test output
# This is an info message that should not appear in silent test output
# This is an internal testlib info message that should not appear in silent test output
# This is a fatal error message that should still appear in silent test output
not ok 8 - messages() # TODO Received a fatal error.
---
# Received a fatal error.
at: tst_Blacklisted::messages() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
line: 0
...
1..8 1..8
# tests 8 # tests 8
# pass 1 # pass 2
# fail 0 # fail 0

5
tests/auto/testlib/selftests/expected_blacklisted.teamcity
View File

@ -11,7 +11,6 @@
##teamcity[testFinished name='xfail()' flowId='tst_Blacklisted'] ##teamcity[testFinished name='xfail()' flowId='tst_Blacklisted']
##teamcity[testStarted name='xpass()' flowId='tst_Blacklisted'] ##teamcity[testStarted name='xpass()' flowId='tst_Blacklisted']
##teamcity[testFinished name='xpass()' flowId='tst_Blacklisted'] ##teamcity[testFinished name='xpass()' flowId='tst_Blacklisted']
##teamcity[testStarted name='messages()' flowId='tst_Blacklisted'] ##teamcity[testStarted name='cleanupTestCase()' flowId='tst_Blacklisted']
##teamcity[testStdOut name='messages()' out='QWARN: This is a warning that should not appear in silent test output|nWARNING |[Loc: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)|]: This is an internal testlib warning that should not appear in silent test output|nQDEBUG: This is a debug message that should not appear in silent test output|nQSYSTEM: This is a critical message that should not appear in silent test output|nQINFO: This is an info message that should not appear in silent test output|nINFO |[Loc: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)|]: This is an internal testlib info message that should not appear in silent test output|nQFATAL: This is a fatal error message that should still appear in silent test output' flowId='tst_Blacklisted'] ##teamcity[testFinished name='cleanupTestCase()' flowId='tst_Blacklisted']
##teamcity[testFinished name='messages()' flowId='tst_Blacklisted']
##teamcity[testSuiteFinished name='tst_Blacklisted' flowId='tst_Blacklisted'] ##teamcity[testSuiteFinished name='tst_Blacklisted' flowId='tst_Blacklisted']

14
tests/auto/testlib/selftests/expected_blacklisted.txt
View File

@ -11,16 +11,6 @@ BXFAIL : tst_Blacklisted::xfail() This test should BXFAIL then BPASS
BPASS : tst_Blacklisted::xfail() BPASS : tst_Blacklisted::xfail()
BXPASS : tst_Blacklisted::xpass() 'true' returned TRUE unexpectedly. (This test should BXPASS) BXPASS : tst_Blacklisted::xpass() 'true' returned TRUE unexpectedly. (This test should BXPASS)
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)] Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
QWARN : tst_Blacklisted::messages() This is a warning that should not appear in silent test output PASS : tst_Blacklisted::cleanupTestCase()
WARNING: tst_Blacklisted::messages() This is an internal testlib warning that should not appear in silent test output Totals: 2 passed, 0 failed, 1 skipped, 5 blacklisted, 0ms
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
QDEBUG : tst_Blacklisted::messages() This is a debug message that should not appear in silent test output
QSYSTEM: tst_Blacklisted::messages() This is a critical message that should not appear in silent test output
QINFO : tst_Blacklisted::messages() This is an info message that should not appear in silent test output
INFO : tst_Blacklisted::messages() This is an internal testlib info message that should not appear in silent test output
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
QFATAL : tst_Blacklisted::messages() This is a fatal error message that should still appear in silent test output
BFAIL : tst_Blacklisted::messages() Received a fatal error.
Loc: [qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp(0)]
Totals: 1 passed, 0 failed, 1 skipped, 6 blacklisted, 0ms
********* Finished testing of tst_Blacklisted ********* ********* Finished testing of tst_Blacklisted *********

27
tests/auto/testlib/selftests/expected_blacklisted.xml
View File

@ -38,31 +38,8 @@
</Incident> </Incident>
<Duration msecs="0"/> <Duration msecs="0"/>
</TestFunction> </TestFunction>
<TestFunction name="messages"> <TestFunction name="cleanupTestCase">
<Message type="qwarn" file="" line="0"> <Incident type="pass" file="" line="0" />
<Description><![CDATA[This is a warning that should not appear in silent test output]]></Description>
</Message>
<Message type="warn" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This is an internal testlib warning that should not appear in silent test output]]></Description>
</Message>
<Message type="qdebug" file="" line="0">
<Description><![CDATA[This is a debug message that should not appear in silent test output]]></Description>
</Message>
<Message type="system" file="" line="0">
<Description><![CDATA[This is a critical message that should not appear in silent test output]]></Description>
</Message>
<Message type="qinfo" file="" line="0">
<Description><![CDATA[This is an info message that should not appear in silent test output]]></Description>
</Message>
<Message type="info" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[This is an internal testlib info message that should not appear in silent test output]]></Description>
</Message>
<Message type="qfatal" file="" line="0">
<Description><![CDATA[This is a fatal error message that should still appear in silent test output]]></Description>
</Message>
<Incident type="bfail" file="qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp" line="0">
<Description><![CDATA[Received a fatal error.]]></Description>
</Incident>
<Duration msecs="0"/> <Duration msecs="0"/>
</TestFunction> </TestFunction>
<Duration msecs="0"/> <Duration msecs="0"/>