AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f3d89adf6d
qt5base-lts/tests/auto/testlib/selftests/expected_subtest.junitxml

108 lines
4.2 KiB
Plaintext
Raw Normal View History

Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
<?xml version="1.0" encoding="UTF-8" ?>
Make counts of various types of test result add up correctly Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-09-07 13:30:34 +00:00
<testsuite name="tst_Subtest" timestamp="@TEST_START_TIME@" hostname="@HOSTNAME@" tests="11" failures="3" errors="0" skipped="1" time="@TEST_DURATION@">
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
<properties>
testlib: Output JUnitXML attributes in right order The attributes are, like the elements, maintained in reverse order in the underlying QTestCoreList, so we need to iterate them backwards when printing out the resulting XML to reflect the order they were added. This results in e.g.: <testcase name="passingBenchmark" result="pass"> Instead of: <testcase result="pass" name="passingBenchmark"> Change-Id: Ic2eeab8de05ffedd0c41977358d5b40ff77878b1 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-21 23:38:42 +00:00
<property name="QTestVersion" value="@INSERT_QT_VERSION_HERE@"/>
<property name="QtVersion" value="@INSERT_QT_VERSION_HERE@"/>
<property name="QtBuild" value=""/>
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
</properties>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="initTestCase" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[initTestCase initTestCase (null)]]>
</system-out>
Split subtest selftest This test was attempting to verify two completely unrelated things, so this commit splits it into two tests. Also, printf calls are replaced by qDebug so that the test does not bypass the testlib loggers. Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee Reviewed-on: http://codereview.qt-project.org/5178 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2011-09-20 01:56:31 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="test1" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[init test1 (null)]]>
<![CDATA[test1 test1 (null)]]>
<![CDATA[cleanup test1 (null)]]>
</system-out>
Split subtest selftest This test was attempting to verify two completely unrelated things, so this commit splits it into two tests. Also, printf calls are replaced by qDebug so that the test does not bypass the testlib loggers. Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee Reviewed-on: http://codereview.qt-project.org/5178 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2011-09-20 01:56:31 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="test2(data0)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[test2_data test2 (null)]]>
<![CDATA[test2_data end]]>
<![CDATA[init test2 data0]]>
<![CDATA[test2 test2 data0]]>
<![CDATA[test2 end]]>
<![CDATA[cleanup test2 data0]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="test2(data1)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[init test2 data1]]>
<![CDATA[test2 test2 data1]]>
<![CDATA[test2 end]]>
<![CDATA[cleanup test2 data1]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="test2(data2)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[init test2 data2]]>
<![CDATA[test2 test2 data2]]>
<![CDATA[test2 end]]>
<![CDATA[cleanup test2 data2]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="test3(data0)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[test3_data test3 (null)]]>
<![CDATA[test3_data end]]>
<![CDATA[init test3 data0]]>
Add a repeated-failure selftest for testlib This mimics the effects of test-code delegating checks to other functions, which can lead to repeated failures. The failing test should only contribute one to the total of failed tsts. Drive-by: correct an existing test's reporting of its own name. Task-number: QTBUG-95661 Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-16 15:43:00 +00:00
<![CDATA[test3 test3 data0]]>
<![CDATA[test3 end]]>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<![CDATA[cleanup test3 data0]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="test3(data1)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Write failure details as <failure> content in JUnit reporter Pick-to: 6.2 Change-Id: Ica48769e7dfcabdc4bc8f0ed058bc22e29a0b632 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-02 12:29:28 +00:00
<failure type="fail" message="Compared values are not the same">
<![CDATA[ Actual (str) : "hello1"
Expected (QString("hello0")): "hello0"]]>
</failure>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[init test3 data1]]>
Add a repeated-failure selftest for testlib This mimics the effects of test-code delegating checks to other functions, which can lead to repeated failures. The failing test should only contribute one to the total of failed tsts. Drive-by: correct an existing test's reporting of its own name. Task-number: QTBUG-95661 Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-16 15:43:00 +00:00
<![CDATA[test3 test3 data1]]>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<![CDATA[cleanup test3 data1]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="test3(data2)" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Write failure details as <failure> content in JUnit reporter Pick-to: 6.2 Change-Id: Ica48769e7dfcabdc4bc8f0ed058bc22e29a0b632 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-02 12:29:28 +00:00
<failure type="fail" message="Compared values are not the same">
<![CDATA[ Actual (str) : "hello2"
Expected (QString("hello0")): "hello0"]]>
</failure>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[init test3 data2]]>
Add a repeated-failure selftest for testlib This mimics the effects of test-code delegating checks to other functions, which can lead to repeated failures. The failing test should only contribute one to the total of failed tsts. Drive-by: correct an existing test's reporting of its own name. Task-number: QTBUG-95661 Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-16 15:43:00 +00:00
<![CDATA[test3 test3 data2]]>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<![CDATA[cleanup test3 data2]]>
</system-out>
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
</testcase>
Add a repeated-failure selftest for testlib This mimics the effects of test-code delegating checks to other functions, which can lead to repeated failures. The failing test should only contribute one to the total of failed tsts. Drive-by: correct an existing test's reporting of its own name. Task-number: QTBUG-95661 Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-16 15:43:00 +00:00
<testcase name="multiFail" classname="tst_Subtest" time="@TEST_DURATION@">
<failure type="fail" message="This failure message should be repeated ten times"/>
<system-out>
<![CDATA[init multiFail (null)]]>
<![CDATA[cleanup multiFail (null)]]>
</system-out>
</testcase>
Make counts of various types of test result add up correctly Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2021-09-07 13:30:34 +00:00
<testcase name="multiSkip" classname="tst_Subtest" time="@TEST_DURATION@">
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="This skip should be repeated ten times"/>
<skipped message="But this test should only contribute one to the skip count"/>
<system-out>
<![CDATA[init multiSkip (null)]]>
<![CDATA[cleanup multiSkip (null)]]>
</system-out>
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="cleanupTestCase" classname="tst_Subtest" time="@TEST_DURATION@">
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[cleanupTestCase cleanupTestCase (null)]]>
</system-out>
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
</testcase>
</testsuite>
Reference in New Issue Copy Permalink