AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
db33628452
qt5base-lts/tests/auto/testlib/selftests/counting/tst_counting.cpp

283 lines
6.1 KiB
C++
Raw Normal View History

Use SPDX license identifiers Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-05-10 10:06:48 +00:00
// Copyright (C) 2021 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GPL-3.0-only WITH Qt-GPL-exception-1.0
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
#include <QtCore/QCoreApplication>
Replace QtTest headers with QTest Complete search and replace of QtTest and QtTest/QtTest with QTest, as QtTest includes the whole module. Replace all such instances with correct header includes. See Jira task for more discussion. Fixes: QTBUG-88831 Change-Id: I981cfae18a1cabcabcabee376016b086d9d01f44 Pick-to: 6.0 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2020-11-26 16:31:50 +00:00
#include <QTest>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
class tst_Counting : public QObject
{
Q_OBJECT
private slots:
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
// The following test functions exercise each possible combination of test
// results for two data rows.
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
void testPassPass_data();
void testPassPass();
void testPassSkip_data();
void testPassSkip();
void testPassFail_data();
void testPassFail();
void testSkipPass_data();
void testSkipPass();
void testSkipSkip_data();
void testSkipSkip();
void testSkipFail_data();
void testSkipFail();
void testFailPass_data();
void testFailPass();
void testFailSkip_data();
void testFailSkip();
void testFailFail_data();
void testFailFail();
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
// The following test functions test skips and fails in the special
// init() and cleanup() slots.
void init();
void cleanup();
void testFailInInit_data();
void testFailInInit();
void testFailInCleanup_data();
void testFailInCleanup();
void testSkipInInit_data();
void testSkipInInit();
void testSkipInCleanup_data();
void testSkipInCleanup();
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
private:
void helper();
};
enum TestResult
{
Pass,
Fail,
Skip
};
Q_DECLARE_METATYPE(TestResult);
void tst_Counting::helper()
{
QFETCH(TestResult, result);
switch (result) {
case Pass:
QVERIFY(true);
testlib: Make verbose2 selftest use QCOMPARE as well as QVERIFY. The verbose output for QCOMPARE is not very helpful. Make the verbose2 selftest (which reuses the counting selftest) demonstrate this, so that a future commit can demonstrate improvement. Change-Id: I6b3bc8f5199e984aa11d0a67b76a8c916be86380 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-03-01 05:14:18 +00:00
QCOMPARE(2 + 1, 3);
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
break;
case Fail:
QVERIFY(false);
break;
case Skip:
QSKIP("Skipping");
break;
}
}
void tst_Counting::testPassPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testPassPass()
{
helper();
}
void tst_Counting::testPassSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testPassSkip()
{
helper();
}
void tst_Counting::testPassFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testPassFail()
{
helper();
}
void tst_Counting::testSkipPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testSkipPass()
{
helper();
}
void tst_Counting::testSkipSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testSkipSkip()
{
helper();
}
void tst_Counting::testSkipFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testSkipFail()
{
helper();
}
void tst_Counting::testFailPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testFailPass()
{
helper();
}
void tst_Counting::testFailSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testFailSkip()
{
helper();
}
void tst_Counting::testFailFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testFailFail()
{
helper();
}
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
void tst_Counting::init()
{
Tidy up in counting, skip and skipinit selftests Minor clean-up. Split two long lines and added braces, called for by Qt coding style. Turned some QDebug("this line should never be reached...") into QFAIL(); that should make the effor a bit more prominent if it ever happens. Likewise for a qDebug() << "ERROR: ..." for code not meant to run. Change-Id: I1d34fe202f269eb3ad8f58feaabed809e4fec99c Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2021-09-07 11:44:29 +00:00
if (strcmp(QTest::currentTestFunction(), "testFailInInit") == 0
&& strcmp(QTest::currentDataTag(), "fail") == 0) {
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
QFAIL("Fail in init()");
Tidy up in counting, skip and skipinit selftests Minor clean-up. Split two long lines and added braces, called for by Qt coding style. Turned some QDebug("this line should never be reached...") into QFAIL(); that should make the effor a bit more prominent if it ever happens. Likewise for a qDebug() << "ERROR: ..." for code not meant to run. Change-Id: I1d34fe202f269eb3ad8f58feaabed809e4fec99c Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2021-09-07 11:44:29 +00:00
} else if (strcmp(QTest::currentTestFunction(), "testSkipInInit") == 0
&& strcmp(QTest::currentDataTag(), "skip") == 0) {
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
QSKIP("Skip in init()");
Tidy up in counting, skip and skipinit selftests Minor clean-up. Split two long lines and added braces, called for by Qt coding style. Turned some QDebug("this line should never be reached...") into QFAIL(); that should make the effor a bit more prominent if it ever happens. Likewise for a qDebug() << "ERROR: ..." for code not meant to run. Change-Id: I1d34fe202f269eb3ad8f58feaabed809e4fec99c Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2021-09-07 11:44:29 +00:00
}
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
}
void tst_Counting::cleanup()
{
if (strcmp(QTest::currentTestFunction(), "testFailInCleanup") == 0 && strcmp(QTest::currentDataTag(), "fail") == 0)
QFAIL("Fail in cleanup()");
else if (strcmp(QTest::currentTestFunction(), "testSkipInCleanup") == 0 && strcmp(QTest::currentDataTag(), "skip") == 0)
QSKIP("Skip in cleanup()");
}
void tst_Counting::testFailInInit_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("fail") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testFailInInit()
{
if (strcmp(QTest::currentDataTag(), "fail") == 0)
QFAIL("This test function should have been skipped due to QFAIL in init()");
}
void tst_Counting::testFailInCleanup_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("fail") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testFailInCleanup()
{
if (strcmp(QTest::currentDataTag(), "fail") == 0)
qDebug() << "This test function should execute and then QFAIL in cleanup()";
}
void tst_Counting::testSkipInInit_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("skip") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testSkipInInit()
{
if (strcmp(QTest::currentDataTag(), "skip") == 0)
QFAIL("This test function should have been skipped due to QSKIP in init()");
}
void tst_Counting::testSkipInCleanup_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("skip") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testSkipInCleanup()
{
if (strcmp(QTest::currentDataTag(), "skip") == 0)
qDebug() << "This test function should execute and then QSKIP in cleanup()";
}
testlib: Move subtest argument handling to individual tests The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2020-02-06 11:07:04 +00:00
#ifdef TESTLIB_VERBOSITY_ARG
Refactor QTEST*_MAIN() implementations The various variants duplicated some rather complex code around varying setup in the middle. Rework in terms of a macro that defines main() and takes the setup code as a parameter. That setup code also had some common structure, so package that in a setup macro that takes the class to be used. Reworked various testlib selftests that were using QTEST_MAIN_IMPL(); change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP(). These might be better dealt with by supporting a second form of the initMain() test-setup function in the test classes, that takes references for argc and argv, to let a test massage its command-line options. Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-09-28 13:12:48 +00:00
#define SETUP() \
std::vector<const char*> args(argv, argv + argc); \
args.push_back(QT_STRINGIFY(TESTLIB_VERBOSITY_ARG)); \
argc = int(args.size()); \
testlib: Move subtest argument handling to individual tests The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2020-02-06 11:07:04 +00:00
argv = const_cast<char**>(&args[0]);
Refactor QTEST*_MAIN() implementations The various variants duplicated some rather complex code around varying setup in the middle. Rework in terms of a macro that defines main() and takes the setup code as a parameter. That setup code also had some common structure, so package that in a setup macro that takes the class to be used. Reworked various testlib selftests that were using QTEST_MAIN_IMPL(); change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP(). These might be better dealt with by supporting a second form of the initMain() test-setup function in the test classes, that takes references for argc and argv, to let a test massage its command-line options. Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-09-28 13:12:48 +00:00
#else
#define SETUP()
testlib: Move subtest argument handling to individual tests The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2020-02-06 11:07:04 +00:00
#endif
Refactor QTEST*_MAIN() implementations The various variants duplicated some rather complex code around varying setup in the middle. Rework in terms of a macro that defines main() and takes the setup code as a parameter. That setup code also had some common structure, so package that in a setup macro that takes the class to be used. Reworked various testlib selftests that were using QTEST_MAIN_IMPL(); change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP(). These might be better dealt with by supporting a second form of the initMain() test-setup function in the test classes, that takes references for argc and argv, to let a test massage its command-line options. Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-09-28 13:12:48 +00:00
QTEST_MAIN_WRAPPER(tst_Counting,
SETUP()
QTEST_MAIN_SETUP())
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
#include "tst_counting.moc"
Reference in New Issue Copy Permalink