AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
bab660d124
qt5base-lts/tests/auto/testlib/selftests/expected_verbose1.junitxml

79 lines
4.5 KiB
Plaintext
Raw Normal View History

Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
<?xml version="1.0" encoding="UTF-8" ?>
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<testsuite name="tst_Counting" timestamp="@TEST_START_TIME@" hostname="@HOSTNAME@" tests="32" failures="8" errors="0" skipped="8" time="@TEST_DURATION@">
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
<properties>
testlib: Output JUnitXML attributes in right order The attributes are, like the elements, maintained in reverse order in the underlying QTestCoreList, so we need to iterate them backwards when printing out the resulting XML to reflect the order they were added. This results in e.g.: <testcase name="passingBenchmark" result="pass"> Instead of: <testcase result="pass" name="passingBenchmark"> Change-Id: Ic2eeab8de05ffedd0c41977358d5b40ff77878b1 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-21 23:38:42 +00:00
<property name="QTestVersion" value="@INSERT_QT_VERSION_HERE@"/>
<property name="QtVersion" value="@INSERT_QT_VERSION_HERE@"/>
<property name="QtBuild" value=""/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</properties>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="initTestCase" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testPassPass(row 1)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testPassPass(row 2)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testPassSkip(row 1)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testPassSkip(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testPassFail(row 1)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testPassFail(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testSkipPass(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testSkipPass(row 2)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testSkipSkip(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testSkipSkip(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testSkipFail(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testSkipFail(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailPass(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailPass(row 2)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testFailSkip(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailSkip(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skipping"/>
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailFail(row 1)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailFail(row 2)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="&apos;false&apos; returned FALSE. ()"/>
testlib: Skip test function if init() fails. Prior to this commit, the following statement in the qtestlib documentation was untrue: "If init() fails, the following testfunction will not be executed, the test will proceed to the next testfunction." If init() called QSKIP, the test function would be skipped, but if init() reported a failure, the test function would still be executed (even though doing so could be unsafe). This commit makes testlib skip a test function if init() reports a failure and enhances the selftests to cover skips and fails in both init() and cleanup(). Task-number: QTBUG-20371 Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-16 06:00:41 +00:00
</testcase>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
<testcase name="testFailInInit(before)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testFailInInit(fail)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="Fail in init()"/>
</testcase>
<testcase name="testFailInInit(after)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testFailInCleanup(before)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testFailInCleanup(fail)" classname="tst_Counting" time="@TEST_DURATION@">
<failure type="fail" message="Fail in cleanup()"/>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[This test function should execute and then QFAIL in cleanup()]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="testFailInCleanup(after)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testSkipInInit(before)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testSkipInInit(skip)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skip in init()"/>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="testSkipInInit(after)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testSkipInCleanup(before)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="testSkipInCleanup(skip)" classname="tst_Counting" time="@TEST_DURATION@">
testlib: Report skipped tests in JUnit reporter as <skipped> elements The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 15:37:58 +00:00
<skipped message="Skip in cleanup()"/>
testlib: Move stdout/stderr reporting into individual <testcase> elements The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-03 10:50:26 +00:00
<system-out>
<![CDATA[This test function should execute and then QSKIP in cleanup()]]>
</system-out>
testlib: Improve JUnit XML conformance The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-07-29 07:59:59 +00:00
</testcase>
<testcase name="testSkipInCleanup(after)" classname="tst_Counting" time="@TEST_DURATION@"/>
<testcase name="cleanupTestCase" classname="tst_Counting" time="@TEST_DURATION@"/>
Add testlib selftest for counting of test results. The test simply generates each possible pair of Pass, Fail and Skip results. At present the test simply serves to demonstrate the current shortcomings of testlib's plain text logging, namely: * If a test function passes for all data rows, that is counted as one pass, but each skipped or failed row counts as one skip or fail. * Only skipped and failed rows are reported individually in the test output. Passed rows are not reported, so it is impossible to see how many rows were executed. * A skip followed by a pass will be reported as an overall pass for the test function, but the same rows in reverse order will not report any overall result for the test function. Future commits will attempt to correct these problems. Task-number: QTBUG-22124 Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-31 04:07:58 +00:00
</testsuite>
Reference in New Issue Copy Permalink