Commit Graph

14 Commits

Author SHA1 Message Date
Edward Welbourne
16f5132b21 Add test-case for TAP logging glitch
The recently added test for QTestEventLoop produced a message which
the TAP test-logger mis-parses. Since that message shall soon go away
(when I fix the bug the new test exists to verify), modify one of
cmptest's QVERIFY2() messages to trip up the same bug, so we can spot
any regressions on this in future (after the imminent fix to this TAP
issue).

Change-Id: Ibbe9931c01c75df529c9571e2bbdbd34010dd9ec
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-09-22 17:34:51 +02:00
Edward Welbourne
54bcefb25c Test QTRY_COMPARE() and expand testing of QTRY_VERIFY*()
In the process, simplify the latter while adding some actual
time-variation for the QTRY_* loop to navigate round - based on the
extendedcompare test's ClassWithDeferredSetter. Testing remains
primitive, but is at least a bit more thorough.

Pick-to: 6.4
Change-Id: I40be8fb485f3f18f0a4f4bc62ad36cccac691979
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-08-12 20:32:30 +02:00
Volker Hilsheimer
92e696b4ba Use debug stream in QTest::toString's default fallback if possible
For built-in types, this is a compile-time assert - we should not have
any types in Qt for which we have neither debug streaming nor a
QTest::toString specialization implemented. A build of most of Qt
submodules passes with this change, after minor modifications to some
tests. We cannot declare QSizeHint::Policy as a metatype after the
QMetaType has already been instantiated for it, and the QDebug stream
operator for QElaspedTimer needs to be correctly declared within the
namespace.

Add a self-test function for a custom type, and update reference files
of the self-test.

Task-number: QTBUG-104867
Pick-to: 6.4
Change-Id: I2936db5933f4589fce45f47cf2f3224ed614d8c9
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
2022-07-13 15:14:35 +02:00
Volker Hilsheimer
54b276be0b testlib: Don't print QCOMPARE values if they lack string representation
Before 0681a2dd5a, QCOMPARE'ing types
for which no QTest::toString specialization exists did not output
Actual and Expected lines on failure, as that would only print <null>
for both values (which then look like the same value, confusingly).

Commit 0681a2dd5a changed that behavior,
and started printing the confusing <null> values.

Take care of the logic in the formatFailMessage function: if both values
are nullptr, then print only the variable names, but not the confusing
<null> text representation of the values.

Remove dead and duplicated code related to the formatting logic, add a
self-test function, and update the expected_cmptest files.

Fixes: QTBUG-104867
Pick-to: 6.4
Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-07-10 03:08:32 +00:00
Tor Arne Vestbø
308280e7a9 testlib: Move stdout/stderr reporting into individual <testcase> elements
The original Ant JUnit reporter only writes <system-err> and <system-out>
to the <testsuite>, but more modern reporters such as Maven Surefire
scopes output to each individual <testcase>.

This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.:

 https://github.com/jenkinsci/junit-plugin/commit/145eb5c98

Pick-to: 6.2
Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-10 18:06:36 +02:00
Tor Arne Vestbø
a2026e4116 testlib: Write failure details as <failure> content in JUnit reporter
Pick-to: 6.2
Change-Id: Ica48769e7dfcabdc4bc8f0ed058bc22e29a0b632
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-06 17:00:22 +02:00
Tor Arne Vestbø
fb16a66b71 testlib: Report skipped tests in JUnit reporter as <skipped> elements
The Apache Ant and Surefire Maven specs document a <skipped> element that
can be used to signify skipped test, with a corresponding total skipped
test attribute on the <testsuite>.

The element includes an optional message attribute, documented in the
Surefire spec, and in the Ant source code, but not yet documented in
the reverse-engineered Ant spec:

 https://github.com/windyroad/JUnit-Schema/pull/11

Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-05 03:58:50 +02:00
Tor Arne Vestbø
973e74399e testlib: Improve JUnit XML conformance
The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:

 https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java

Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:

 https://plugins.jenkins.io/junit/

Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:

 https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
 https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd

The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.

The following changes have been made to improve conformance:

  - The 'timestamp' attribute on <testsuite> is is now in ISO
    8601 local time, without any time zone specified
  - The 'hostname' attribute on <testsuite> is now included
  - The 'classname' attribute on <testcase> is now included
  - The non-standard 'result' attribute on <testcase> has
    been removed
  - The non-standard 'result' attribute on <failure> has
    been renamed to 'type'
  - The <system-out> element on <testsuite> is always included,
    even when empty
  - The non-standard 'tag' attribute on <failure> has been
    removed. Data-driven tests are now represented as individual
    <testcase> elements, e.g.:

      <testcase name="someTest(someData X)" ...>
      <testcase name="someTest(someData Y)" ...>
      <testcase name="someTest(someData Z)" ...>

The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.

Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-08-05 03:58:49 +02:00
Friedemann Kleint
ae02188233 QTestlib: Add formatting for QObject * in QCOMPARE
Output object name and class in QCOMPARE(). This should help
to debug flaky QWidget tests that for example check on focusWidget().

[ChangeLog][QtTestLib] QCOMPARE() now reports QObject * values by class and objectName().

Task-number: QTBUG-64446
Change-Id: Ife04e89bba04fc78d077c8f0f07af17a17c9cf8c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-05-26 11:02:48 +00:00
Christian Kandeler
b88f692d7d QTestLib: Show the values of unregistered enums
... on a failed QCOMPARE.

Change-Id: I653894927e49fad92c21409d03ed70880ca510f6
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2021-01-13 16:15:06 +01:00
Tor Arne Vestbø
af7a3430ed testlib: Add start time and test duration to JUnit XML
As defined by https://llg.cubic.org/docs/junit/

Change-Id: Ic7683f3d49c529674f8467d591528d4a65d3add8
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-24 00:52:01 +02:00
Tor Arne Vestbø
67c877562b testlib: Output JUnitXML attributes in right order
The attributes are, like the elements, maintained in reverse
order in the underlying QTestCoreList, so we need to iterate
them backwards when printing out the resulting XML to reflect
the order they were added.

This results in e.g.:

  <testcase name="passingBenchmark" result="pass">

Instead of:

  <testcase result="pass" name="passingBenchmark">

Change-Id: Ic2eeab8de05ffedd0c41977358d5b40ff77878b1
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-24 00:51:54 +02:00
Friedemann Kleint
d25589e052 QTestlib: Enable comparing QList against initializer lists/arrays
It is unnecessary to create a QList container just for comparison.

Split out helpers for comparing sequence sizes and sequences from
qCompare(QList) and add a template for an array with a non-type template
parameter for the size.

One can then write something like:

const int expected[] = {10, 12,...};
QCOMPARE(QFontDatabase.pointSizes(...), expected)

Unfortunately, any commas in such an array will be misread by macro expansion
as macro argument separators, so any expected array with more than one entry
needs an extra macro expanding __VA_ARGS__.

Change-Id: Ie7c8dc20bf669bbb25f6d7f8562455f8d03968c8
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2020-07-21 22:01:20 +02:00
Tor Arne Vestbø
27db9e458c testlib: Clarify that our XUnit reporter is actually a JUnit reporter
The reporter was probably named 'xunit' based on the historical use of
xUnit to refer to testing frameworks derived from Smalltalk's SUnit.
These frameworks typically added their own prefix, e.g. JUnit for Java,
RUnit for R, etc.

The most popular of these was the JUnit framework, and the corresponding
XML output produced by the Ant built tool became somewhat of a de facto
standard, which is probably why we chose to model our reporter after it.

Nowadays however, naming it 'xunit' is problematic as there is actually
a testing famework named xUnit.net, typically shortened to, you guessed
it: xunit.

Test report consumers will typically have a junit mode, and an xunit
mode, and the latter could easily be mistaken for what testlib outputs,
unless we clarify this.

The clarification also allows us to safely extend our support for the
JUnit XML format to incorporate some elements that are nowadays common,
but where we are lagging behind the standard.

[ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has
been renamed to what it actually is: a JUnit test reporter, and is now
triggered by passing -o junitxml to the test binary.

Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
2020-02-03 13:48:25 +01:00