AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
16b11bb81e
qt5base-lts/tests/auto/testlib/selftests/counting/tst_counting.cpp

302 lines
7.1 KiB
C++
Raw Normal View History

Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
/****************************************************************************
**
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** Copyright (C) 2016 The Qt Company Ltd.
** Contact: https://www.qt.io/licensing/
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
**
** This file is part of the test suite of the Qt Toolkit.
**
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** $QT_BEGIN_LICENSE:GPL-EXCEPT$
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
Update copyright headers Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
2015-01-28 08:44:43 +00:00
** a written agreement between you and The Qt Company. For licensing terms
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
**
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** GNU General Public License Usage
** Alternatively, this file may be used under the terms of the GNU
** General Public License version 3 as published by the Free Software
** Foundation with exceptions as appearing in the file LICENSE.GPL3-EXCEPT
** included in the packaging of this file. Please review the following
** information to ensure the GNU General Public License requirements will
** be met: https://www.gnu.org/licenses/gpl-3.0.html.
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
**
** $QT_END_LICENSE$
**
****************************************************************************/
#include <QtCore/QCoreApplication>
Replace QtTest headers with QTest Complete search and replace of QtTest and QtTest/QtTest with QTest, as QtTest includes the whole module. Replace all such instances with correct header includes. See Jira task for more discussion. Fixes: QTBUG-88831 Change-Id: I981cfae18a1cabcabcabee376016b086d9d01f44 Pick-to: 6.0 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2020-11-26 16:31:50 +00:00
#include <QTest>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
class tst_Counting : public QObject
{
Q_OBJECT
private slots:
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
// The following test functions exercise each possible combination of test
// results for two data rows.
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
void testPassPass_data();
void testPassPass();
void testPassSkip_data();
void testPassSkip();
void testPassFail_data();
void testPassFail();
void testSkipPass_data();
void testSkipPass();
void testSkipSkip_data();
void testSkipSkip();
void testSkipFail_data();
void testSkipFail();
void testFailPass_data();
void testFailPass();
void testFailSkip_data();
void testFailSkip();
void testFailFail_data();
void testFailFail();
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
// The following test functions test skips and fails in the special
// init() and cleanup() slots.
void init();
void cleanup();
void testFailInInit_data();
void testFailInInit();
void testFailInCleanup_data();
void testFailInCleanup();
void testSkipInInit_data();
void testSkipInInit();
void testSkipInCleanup_data();
void testSkipInCleanup();
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
private:
void helper();
};
enum TestResult
{
Pass,
Fail,
Skip
};
Q_DECLARE_METATYPE(TestResult);
void tst_Counting::helper()
{
QFETCH(TestResult, result);
switch (result) {
case Pass:
QVERIFY(true);
testlib: Make verbose2 selftest use QCOMPARE as well as QVERIFY. The verbose output for QCOMPARE is not very helpful. Make the verbose2 selftest (which reuses the counting selftest) demonstrate this, so that a future commit can demonstrate improvement. Change-Id: I6b3bc8f5199e984aa11d0a67b76a8c916be86380 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-03-01 05:14:18 +00:00
QCOMPARE(2 + 1, 3);
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
break;
case Fail:
QVERIFY(false);
break;
case Skip:
QSKIP("Skipping");
break;
}
}
void tst_Counting::testPassPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testPassPass()
{
helper();
}
void tst_Counting::testPassSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testPassSkip()
{
helper();
}
void tst_Counting::testPassFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Pass;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testPassFail()
{
helper();
}
void tst_Counting::testSkipPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testSkipPass()
{
helper();
}
void tst_Counting::testSkipSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testSkipSkip()
{
helper();
}
void tst_Counting::testSkipFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Skip;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testSkipFail()
{
helper();
}
void tst_Counting::testFailPass_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Pass;
}
void tst_Counting::testFailPass()
{
helper();
}
void tst_Counting::testFailSkip_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Skip;
}
void tst_Counting::testFailSkip()
{
helper();
}
void tst_Counting::testFailFail_data()
{
QTest::addColumn<TestResult>("result");
QTest::newRow("row 1") << Fail;
QTest::newRow("row 2") << Fail;
}
void tst_Counting::testFailFail()
{
helper();
}
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
void tst_Counting::init()
{
if (strcmp(QTest::currentTestFunction(), "testFailInInit") == 0 && strcmp(QTest::currentDataTag(), "fail") == 0)
QFAIL("Fail in init()");
else if (strcmp(QTest::currentTestFunction(), "testSkipInInit") == 0 && strcmp(QTest::currentDataTag(), "skip") == 0)
QSKIP("Skip in init()");
}
void tst_Counting::cleanup()
{
if (strcmp(QTest::currentTestFunction(), "testFailInCleanup") == 0 && strcmp(QTest::currentDataTag(), "fail") == 0)
QFAIL("Fail in cleanup()");
else if (strcmp(QTest::currentTestFunction(), "testSkipInCleanup") == 0 && strcmp(QTest::currentDataTag(), "skip") == 0)
QSKIP("Skip in cleanup()");
}
void tst_Counting::testFailInInit_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("fail") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testFailInInit()
{
if (strcmp(QTest::currentDataTag(), "fail") == 0)
QFAIL("This test function should have been skipped due to QFAIL in init()");
}
void tst_Counting::testFailInCleanup_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("fail") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testFailInCleanup()
{
if (strcmp(QTest::currentDataTag(), "fail") == 0)
qDebug() << "This test function should execute and then QFAIL in cleanup()";
}
void tst_Counting::testSkipInInit_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("skip") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testSkipInInit()
{
if (strcmp(QTest::currentDataTag(), "skip") == 0)
QFAIL("This test function should have been skipped due to QSKIP in init()");
}
void tst_Counting::testSkipInCleanup_data()
{
QTest::addColumn<bool>("dummy");
QTest::newRow("before") << true;
QTest::newRow("skip") << true;
QTest::newRow("after") << true;
}
void tst_Counting::testSkipInCleanup()
{
if (strcmp(QTest::currentDataTag(), "skip") == 0)
qDebug() << "This test function should execute and then QSKIP in cleanup()";
}
testlib: Move subtest argument handling to individual tests The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2020-02-06 11:07:04 +00:00
int main(int argc, char *argv[])
{
#ifdef TESTLIB_VERBOSITY_ARG
std::vector<const char*> args(argv, argv + argc);
args.push_back(QT_STRINGIFY(TESTLIB_VERBOSITY_ARG));
Auto tests: Fix some MSVC warnings about integer conversions Change-Id: Ibab8028ccdb9a4b02cadc6d2e85e8a0472f0d96f Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2020-10-20 12:26:43 +00:00
argc = int(args.size());
testlib: Move subtest argument handling to individual tests The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2020-02-06 11:07:04 +00:00
argv = const_cast<char**>(&args[0]);
#endif
QTEST_MAIN_IMPL(tst_Counting)
}
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
#include "tst_counting.moc"
Reference in New Issue Copy Permalink