AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9b2971cbb4
qt5base-lts/tests/auto/testlib/selftests
History
Jason McDonald 9b2971cbb4 testlib: Count passes, fails and skips consistently. 
For data-driven tests, testlib previously counted one fail or skip for
each data row that failed or skipped, while it counted only one pass
for a test function where all rows passed and counted no passes for a
test function where some rows passed and some rows failed. A similar
problem also existed for benchmark tests, which could run multiple
iterations of the same test, with each fail and skip being counted but
only a single pass being counted for the entire series of iterations.

This commit makes testlib count one pass, fail or skip for each data
row.  Test functions that are not data-driven count one result for the
test function, as before.  Benchmark tests count one pass, fail or skip
per iteration.

A side-effect of this change is that the test output in plain text, xml
and light xml formats now shows a result for every data row and
benchmark iteration executed, allowing post-processors to correctly
calculate the total number of tests executed.  Previously, individual
rows were not shown in the test output if they passed, making such
calculations impossible.

The only change to the xunitxml output format is to correct a bug where
no test result was recorded for a test function if the last data row
was skipped and all other rows passed -- in which case the overall
result should be a pass.  Note that there is also a pre-existing bug
in the xunit logger, where no result is reported if all rows are
skipped; that bug is unaffected by this commit.

Task-number: QTBUG-21848
Task-number: QTBUG-22124
Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b
Reviewed-by: Ed Baak <ed.baak@nokia.com>
2012-02-13 01:46:09 +01:00
..
alive Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
assert Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
badxml Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibcallgrind Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibeventcounter Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchliboptions Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibtickcounter Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibwalltime Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
cmptest Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
commandlinedata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
counting Add testlib selftest for counting of test results. 2012-02-02 05:42:07 +01:00
crashes Fixed tst_selftests on OSX 10.7 2012-02-09 03:31:08 +01:00
datatable Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
datetime Add data tag names to datetime selftest. 2012-02-09 04:07:45 +01:00
differentexec Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
exceptionthrow Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
expectfail testlib: Clear expected failures after every data row. 2012-02-08 03:13:21 +01:00
failcleanup testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
failinit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
failinitdata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
fetchbogus Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
findtestdata Fixed selftests unittest on windows 2012-02-02 05:42:07 +01:00
float Improve float selftest 2012-02-09 04:07:50 +01:00
globaldata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
longstring Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
maxwarnings Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
multiexec Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
printdatatags Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
printdatatagswithglobaltags Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
qexecstringlist Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
singleskip Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skip Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skipcleanup testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
skipinit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skipinitdata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
sleep Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
strcmp Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
subtest Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
test Changed selftests unittest to work from installation directory. 2012-01-11 04:09:26 +01:00
warnings testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
xunit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
.gitignore Revert "Remove unused overload of QTest::qExec." 2011-12-12 10:43:44 +01:00
expected_alive.txt Add testlib selftest for QStringList comparison. 2011-12-14 02:40:17 +01:00
expected_assert.lightxml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_assert.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_assert.xml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_assert.xunitxml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_badxml.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_badxml.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_badxml.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_badxml.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibcallgrind.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchliboptions.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibtickcounter.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibtickcounter.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_benchlibtickcounter.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibtickcounter.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_benchlibwalltime.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_cmptest.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.xunitxml QTestlib: Make QImage comparison more verbose. 2011-12-21 22:59:28 +01:00
expected_commandlinedata.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_counting.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_counting.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_counting.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_counting.xunitxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_crashes_1.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_crashes_2.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_crashes_3.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_datatable.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_datetime.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.xunitxml Add data tag names to datetime selftest. 2012-02-09 04:07:45 +01:00
expected_differentexec.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_exceptionthrow.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_expectfail.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_expectfail.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_expectfail.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_expectfail.xunitxml testlib: Clear expected failures after every data row. 2012-02-08 03:13:21 +01:00
expected_failcleanup.lightxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.txt testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.xml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.xunitxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failinit.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinit.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_failinit.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinit.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_failinitdata.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fatal.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_fetchbogus.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fetchbogus.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_fetchbogus.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fetchbogus.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_findtestdata.lightxml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.txt testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.xml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.xunitxml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_float.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.xunitxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_longstring.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_maxwarnings.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_multiexec.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_printdatatags.txt Fixing naming of selftests. 2011-12-05 09:02:15 +01:00
expected_printdatatagswithglobaltags.txt Fixing naming of selftests. 2011-12-05 09:02:15 +01:00
expected_qexecstringlist.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_singleskip.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_singleskip.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_singleskip.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_singleskip.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skip.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skipcleanup.lightxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.txt testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.xml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.xunitxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipinit.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_skipinit.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinit.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_skipinit.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skipinitdata.lightxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.xml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.xunitxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_sleep.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_strcmp.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_strcmp.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_strcmp.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_strcmp.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_subtest.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.xunitxml Split subtest selftest 2011-09-20 08:46:18 +02:00
expected_warnings.lightxml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.txt testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.xml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.xunitxml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_xunit.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_xunit.txt Make QTest::qWarn() use __FILE__ and __LINE__. 2011-09-28 12:17:45 +02:00
expected_xunit.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_xunit.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
selftests.pri Make selftest.pri more readable/maintainable. 2012-02-09 04:07:53 +01:00
selftests.pro Changed selftests unittest to work from installation directory. 2012-01-11 04:09:26 +01:00
selftests.qrc Remove unused test results from testlib selftests. 2012-02-09 04:07:48 +01:00
tst_selftests.cpp CodeCoverage: Handle QTest based subtests. 2012-02-12 23:25:19 +01:00