AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
eb52d78e90
qt5base-lts/tests/auto/testlib/selftests
History
Jason McDonald eb52d78e90 testlib: Report one test result per benchmark test. 
Prior to this commit, a benchmark test could report 0..n passes and 0..m
fails or skips, where n is the number of accumulation iterations used to
collect benchmark data and m is the number of times the test function
was invoked.  Depending on the type of benchmark measurer being used,
this could result in a very large volume of test output and inconsistent
pass, fail and skip counts between test runs.

This commit changes the behaviour so that each benchmark test reports
one pass, fail or skip, regardless of the number of iterations used to
collect benchmark data.

This commit also prevents benchmark data being reported in the test
output if the benchmark test failed or skipped, as any benchmark data is
of dubious value in such cases.

The latter change in behaviour requires a minor modification to the
badxml selftest, which now tests quoting of literal strings in xml test
output for both passing and failing benchmarks.

Finally, this commit also adds a new selftest specifically for verifying
correct behaviour for benchmarks that fail or skip.

Task-number: QTBUG-24313
Change-Id: I3426dc659a7511b62fd183a031c7235bc753f497
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-20 08:05:58 +01:00
..
alive Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
assert Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
badxml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
benchlibcallgrind Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibcounting testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
benchlibeventcounter Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchliboptions Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibtickcounter Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
benchlibwalltime Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
cmptest Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
commandlinedata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
counting testlib: Skip test function if init() fails. 2012-02-17 05:24:46 +01:00
crashes Fixed tst_selftests on OSX 10.7 2012-02-09 03:31:08 +01:00
datatable Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
datetime Add data tag names to datetime selftest. 2012-02-09 04:07:45 +01:00
differentexec Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
exceptionthrow Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
expectfail testlib: Add selftests for XPASS and XFAIL involving QCOMPARE. 2012-02-20 03:27:02 +01:00
failcleanup testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
failinit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
failinitdata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
fetchbogus Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
findtestdata Fixed selftests unittest on windows 2012-02-02 05:42:07 +01:00
float Improve float selftest 2012-02-09 04:07:50 +01:00
globaldata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
longstring Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
maxwarnings Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
multiexec Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
printdatatags Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
printdatatagswithglobaltags Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
qexecstringlist Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
singleskip Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skip Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skipcleanup testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
skipinit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
skipinitdata Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
sleep Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
strcmp Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
subtest Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
test Changed selftests unittest to work from installation directory. 2012-01-11 04:09:26 +01:00
warnings testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
xunit Remove "All rights reserved" line from license headers. 2012-01-30 03:54:59 +01:00
.gitignore Revert "Remove unused overload of QTest::qExec." 2011-12-12 10:43:44 +01:00
expected_alive.txt Add testlib selftest for QStringList comparison. 2011-12-14 02:40:17 +01:00
expected_assert.lightxml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_assert.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_assert.xml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_assert.xunitxml Fix XFAIL in testlib selftest 2011-09-17 03:21:44 +02:00
expected_badxml.lightxml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_badxml.txt testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_badxml.xml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_badxml.xunitxml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibcallgrind.txt testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibcounting.lightxml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibcounting.txt testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibcounting.xml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibcounting.xunitxml testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibeventcounter.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_benchlibeventcounter.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchliboptions.txt testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
expected_benchlibtickcounter.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibtickcounter.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_benchlibtickcounter.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibtickcounter.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_benchlibwalltime.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_benchlibwalltime.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_cmptest.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_cmptest.xunitxml QTestlib: Make QImage comparison more verbose. 2011-12-21 22:59:28 +01:00
expected_commandlinedata.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_commandlinedata.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_counting.lightxml testlib: Skip test function if init() fails. 2012-02-17 05:24:46 +01:00
expected_counting.txt testlib: Skip test function if init() fails. 2012-02-17 05:24:46 +01:00
expected_counting.xml testlib: Skip test function if init() fails. 2012-02-17 05:24:46 +01:00
expected_counting.xunitxml testlib: Skip test function if init() fails. 2012-02-17 05:24:46 +01:00
expected_crashes_1.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_crashes_2.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_crashes_3.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_datatable.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datatable.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_datetime.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_datetime.xunitxml Add data tag names to datetime selftest. 2012-02-09 04:07:45 +01:00
expected_differentexec.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_exceptionthrow.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_exceptionthrow.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_expectfail.lightxml testlib: Add selftests for XPASS and XFAIL involving QCOMPARE. 2012-02-20 03:27:02 +01:00
expected_expectfail.txt testlib: Add selftests for XPASS and XFAIL involving QCOMPARE. 2012-02-20 03:27:02 +01:00
expected_expectfail.xml testlib: Add selftests for XPASS and XFAIL involving QCOMPARE. 2012-02-20 03:27:02 +01:00
expected_expectfail.xunitxml testlib: Add selftests for XPASS and XFAIL involving QCOMPARE. 2012-02-20 03:27:02 +01:00
expected_failcleanup.lightxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.txt testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.xml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failcleanup.xunitxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_failinit.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinit.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_failinit.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinit.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_failinitdata.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_failinitdata.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fatal.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_fetchbogus.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fetchbogus.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_fetchbogus.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_fetchbogus.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_findtestdata.lightxml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.txt testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.xml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_findtestdata.xunitxml testlib: add QFINDTESTDATA macro for finding testdata files 2011-12-01 09:12:57 +01:00
expected_float.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_globaldata.xunitxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_longstring.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.txt Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_longstring.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.lightxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_maxwarnings.xml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_maxwarnings.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_multiexec.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_printdatatags.txt Fixing naming of selftests. 2011-12-05 09:02:15 +01:00
expected_printdatatagswithglobaltags.txt Fixing naming of selftests. 2011-12-05 09:02:15 +01:00
expected_qexecstringlist.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_singleskip.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_singleskip.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_singleskip.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_singleskip.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skip.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_skip.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skipcleanup.lightxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.txt testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.xml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipcleanup.xunitxml testlib: Add selftests for skip and fail inside cleanupTestCase(). 2012-02-09 04:07:33 +01:00
expected_skipinit.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_skipinit.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinit.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_skipinit.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_skipinitdata.lightxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.txt Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.xml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_skipinitdata.xunitxml Remove QTest::SkipMode from qtestlib API. 2011-10-21 01:20:29 +02:00
expected_sleep.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_strcmp.lightxml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_strcmp.txt Use placeholder for Qt version in expected output 2011-09-26 07:50:20 +02:00
expected_strcmp.xml Sanitize paths in selftest data. 2011-10-19 07:24:40 +02:00
expected_strcmp.xunitxml Moved autotests in category 'testlib' into new directory structure 2011-09-12 08:36:15 +02:00
expected_subtest.lightxml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.txt testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.xml testlib: Count passes, fails and skips consistently. 2012-02-13 01:46:09 +01:00
expected_subtest.xunitxml Split subtest selftest 2011-09-20 08:46:18 +02:00
expected_warnings.lightxml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.txt testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.xml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_warnings.xunitxml testlib: Clear ignored messages after every data row 2012-02-08 03:13:23 +01:00
expected_xunit.lightxml testlib: make XPASS output slightly less confusing. 2012-02-20 03:27:02 +01:00
expected_xunit.txt testlib: make XPASS output slightly less confusing. 2012-02-20 03:27:02 +01:00
expected_xunit.xml testlib: make XPASS output slightly less confusing. 2012-02-20 03:27:02 +01:00
expected_xunit.xunitxml testlib: make XPASS output slightly less confusing. 2012-02-20 03:27:02 +01:00
selftests.pri testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
selftests.pro Changed selftests unittest to work from installation directory. 2012-01-11 04:09:26 +01:00
selftests.qrc testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00
tst_selftests.cpp testlib: Report one test result per benchmark test. 2012-02-20 08:05:58 +01:00