Make the various versions of the failure message align consistently so
that it's a little easier to compare the actual and expected values. Of
course, the value won't align nicely unless the "actual" and "expected"
strings are the same length, but at least this commit makes that
consistent across all versions of the message.
Change-Id: If9ce231df3b5d279a06f6458fdb5da0aa4586068
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
If the expected output file was missing (e.g. not included in
selftests.qrc), tst_selftests would trigger an assert inside QList by
calling QList::at() on an empty list. Make tst_selftests detect this
error instead and give a meaningful error message.
When loading expected output for the crashes selftest, where there are
several alternative versions of the expected output, the code reused the
"exp" variable when loading the alternative test output files. This
caused the last file loaded to be used unintentionally if none of the
alternative files had the correct number of lines. Use a different
variable so that exp remains empty if none of the alternatives are
valid and a failure can be reported.
Change-Id: I35b2a3d905d069d3ee8dcb1447836eb68d5c8612
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as
"COMPARE()", making it impossible to see what was compared and difficult
to match the output to the source of a test containing many calls to
QCOMPARE.
This commit changes testlib's internal compare_helper API so that string
representations of the compared expressions are always passed to
QTestResult::compare() when available, and can thus be shown in the
verbose and XPASS output. The XPASS output has also been changed to
state explicitly that the comparison succeeded unexpectedly, bringing it
in line with the XPASS output resulting from a call to QVERIFY.
This commit also changes all calls to compare_helper() to call the
eight-argument version of the function, which simplifies much of the
calling code. The now obsolete four-argument version of
compare_helper() has been changed to output a warning that it is
obsolete. It will be removed once other modules have had some time to
catch up.
The improved XPASS and verbose output is demonstrated by the expectfail
and verbose2 selftests.
Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The verbose output for QCOMPARE is not very helpful. Make the verbose2
selftest (which reuses the counting selftest) demonstrate this, so that
a future commit can demonstrate improvement.
Change-Id: I6b3bc8f5199e984aa11d0a67b76a8c916be86380
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
These tests don't have their own source code but rather reuse the
counting selftest with additional command-line options.
Note that currently the -v1 switch only changes the plain text output,
and the expected xml output is identical to that of the counting test.
This may change in the future however.
This commit also restores a couple of lists to alphabetical order, where
the findtestdata selftest was not sorted into the list correctly.
Change-Id: Ie38e255f8029157b34162b3864b5fa66e137d74a
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Do this regardless of whether the event subclass
is public API or only used in examples. Examples
are examples, used by others as templates or even
copied verbatim, so they should also follow sound
engineering rules.
Anyway, there's only one in examples/...
Change-Id: I586ff16407a956c9e89288fdd4377eed73f45c0f
Reviewed-by: Samuel Rødal <samuel.rodal@nokia.com>
The private git history shows that the test program associated with this
data file was missing from the original commit and was never
subsequently added.
Change-Id: I3401724ac04168158a48eb06436db83d3557711f
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously the silent logging mode suppressed passes, skips and internal
testlib info messages, but did not suppress debugging output, making it
hard to see the fails in a noisy test. This commit changes silent mode
so that it suppresses all output except test failures and fatal errors,
making silent mode truly useful for seeing just the important test
output.
This commit also adds a selftest to verify the behaviour of silent mode.
Change-Id: I75420aead03682306210746a87e2a3b608b58fc6
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The meta-object format is going to change for Qt5. Use QMOB to
insulate the badxml test from such changes.
(It just so happens that the QFAIL("a failure") statement is still on
line 109 after the refactoring, so the expected_badxml.* files'
location tags did not have to be changed.)
Change-Id: I04421d13c4df71c8004fa71cafc4823a59079a41
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Prior to this commit, a benchmark test could report 0..n passes and 0..m
fails or skips, where n is the number of accumulation iterations used to
collect benchmark data and m is the number of times the test function
was invoked. Depending on the type of benchmark measurer being used,
this could result in a very large volume of test output and inconsistent
pass, fail and skip counts between test runs.
This commit changes the behaviour so that each benchmark test reports
one pass, fail or skip, regardless of the number of iterations used to
collect benchmark data.
This commit also prevents benchmark data being reported in the test
output if the benchmark test failed or skipped, as any benchmark data is
of dubious value in such cases.
The latter change in behaviour requires a minor modification to the
badxml selftest, which now tests quoting of literal strings in xml test
output for both passing and failing benchmarks.
Finally, this commit also adds a new selftest specifically for verifying
correct behaviour for benchmarks that fail or skip.
Task-number: QTBUG-24313
Change-Id: I3426dc659a7511b62fd183a031c7235bc753f497
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, when an unexpected pass occurred the test output
showed the message "XPASS : tst_foo::function() 'expr' returned FALSE",
where the problem was actually that the expression evaluated to true
when it was expected to be false.
This commit changes the output to make it clear that the expression
evaluated to true unexpectedly.
Task-number: QTBUG-22118
Change-Id: Id22c178073d3b75789675ca37a8ef019029b1f91
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Prior to this commit, the following statement in the qtestlib
documentation was untrue: "If init() fails, the following testfunction
will not be executed, the test will proceed to the next testfunction."
If init() called QSKIP, the test function would be skipped, but if
init() reported a failure, the test function would still be executed
(even though doing so could be unsafe).
This commit makes testlib skip a test function if init() reports a
failure and enhances the selftests to cover skips and fails in both
init() and cleanup().
Task-number: QTBUG-20371
Change-Id: Id1cc8464ae0b8c257ae1b74dbe9189a501f5366b
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
For data-driven tests, testlib previously counted one fail or skip for
each data row that failed or skipped, while it counted only one pass
for a test function where all rows passed and counted no passes for a
test function where some rows passed and some rows failed. A similar
problem also existed for benchmark tests, which could run multiple
iterations of the same test, with each fail and skip being counted but
only a single pass being counted for the entire series of iterations.
This commit makes testlib count one pass, fail or skip for each data
row. Test functions that are not data-driven count one result for the
test function, as before. Benchmark tests count one pass, fail or skip
per iteration.
A side-effect of this change is that the test output in plain text, xml
and light xml formats now shows a result for every data row and
benchmark iteration executed, allowing post-processors to correctly
calculate the total number of tests executed. Previously, individual
rows were not shown in the test output if they passed, making such
calculations impossible.
The only change to the xunitxml output format is to correct a bug where
no test result was recorded for a test function if the last data row
was skipped and all other rows passed -- in which case the overall
result should be a pass. Note that there is also a pre-existing bug
in the xunit logger, where no result is reported if all rows are
skipped; that bug is unaffected by this commit.
Task-number: QTBUG-21848
Task-number: QTBUG-22124
Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b
Reviewed-by: Ed Baak <ed.baak@nokia.com>
Set QT_TESTCOCOON_ACTIVE environment variable when the coverage is installed
for a test and unset it when the coverage data is saved. Tests that run when
QT_TESTCOCOON_ACTIVE is set are subtests and will not be considered as
stand-alone tests for the coverage.
When a test is run as a subtest its coverage data will not be saved for
itself but for the main test it is merged with. Also its status will not be
reported since only the status of the main test is expected in the test report,
e.g. the test tests/auto/testlib/selftests.
Change-Id: Icfdf99300aae18040e1a3441a8af21f68df4c0db
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Correct a bug in the test and use unique names for data rows. The bug
was that the test assumed that QCOMPARE for float values 100001 and
100002 would fail, but it actually succeeds. QCOMPARE for floats uses
qFuzzyCompare(), which succeeds if the numbers differ by no more than
1/100,000th of the smaller value. Thus QCOMPARE(100001, 100002) passes,
while QCOMPARE(99998, 99999) fails.
Change-Id: Ia35d3126c2e3ebe91d64daa309048514a365d9fb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Some of the subtests are only run with plain-text output format. For
those subtests, the other output formats were unused and gradually
becoming out-of-date.
Change-Id: I4c10f7f5bab2d2cc7d2d2ad641fbf5d4df02b798
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Our OSX 10.7 test machines send a SIGILL rather than a SIGSEGV to a
process which attempts to dereference a null pointer.
Change the "crashes" test to dereference an invalid pointer with a
value slightly greater than 0 so that we get the same crash behavior on
all (unix) platforms.
Change-Id: I700a2c7d654a9468af5e5996010a258695ed2ae5
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Previously, ignored messages were only cleared at the end of each test
function, i.e. after all data rows were finished. This meant that if a
data row in a data-driven test function didn't cause all of the expected
messages to be generated, the remaining messages would be carried over
to the next data row. This would result in errors about missing
messages being associated with the last data row rather than with the
correct data row.
This commit makes testlib check for missing ignored messages after
running each data row rather than only doing so after the last data
row.
This commit also adds a regression test to demonstrate that ignored
messages can no longer be carried over from one data row to another.
Change-Id: Ibee51aa6e96866fbcbcb4acee1a8340a86a6a4ba
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Previously, expected failures were only cleared at the end of each test
function, i.e. after all data rows were finished. This meant that if a
data-driven test function called QEXPECT_FAIL and then didn't perform
any further verification steps to trigger the expected failure, the
expected failure would be carried over to the next data row, probably
causing the first verification step in the test function to XPASS (with
a seemingly irrelevant error message) for the next data row.
This commit adds the new function QTestResult::finishedCurrentTestData()
to cleanup after each data row is executed. This function treats calls
to QEXPECT_FAIL without subsequent verification steps as a test failure.
This commit also adds a regression test to demonstrate that expected
failures can no longer be carried over from one data row to another. If
run against the previous version of testlib, the new test would report a
pass instead of an error.
Change-Id: Ida5c7f080815b0dca9531131fed582b0918334cb
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
- Needed to add toLower() for compare as c:\ and C:\ was causing
a failure.
Change-Id: Idd55774d118b7249e88362688166b4f9d9712c7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Rename some of the test functions to improve consistency, and change the
order so that all the xfail tests are together (some new xpass tests
will be added later).
Change-Id: Ice1ac5bf61bc26109bb0d08f72eb74e1a0424101
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The test simply generates each possible pair of Pass, Fail and Skip
results. At present the test simply serves to demonstrate the current
shortcomings of testlib's plain text logging, namely:
* If a test function passes for all data rows, that is counted as one
pass, but each skipped or failed row counts as one skip or fail.
* Only skipped and failed rows are reported individually in the test
output. Passed rows are not reported, so it is impossible to see how
many rows were executed.
* A skip followed by a pass will be reported as an overall pass for the
test function, but the same rows in reverse order will not report any
overall result for the test function.
Future commits will attempt to correct these problems.
Task-number: QTBUG-22124
Change-Id: If8c7ea15fc43ba9a1bccd0e881c1efc18e705b25
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test case for calling QEXPECT_FAIL in a data-driven test without
passing a specific data tag for the expected failure, in which the
expected failure should apply to the current data row.
Change-Id: I70a6592de77b1718c3a749fd2216e66f8f312805
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where a call to QEXPECT_FAIL nominates a
different data row from the current row, in which case the call to
QEXPECT_FAIL should be ignored.
Change-Id: Iff774b89f9374fa26105acdd8769177272a95229
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Add a test for the case where QEXPECT_FAIL is called twice in a row
without an intervening verification step, which is supposed to result in
a failure.
Change-Id: Ie004652a76a2c35c2679945670ff5561e8f5b62e
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Make the test fail if calling QEXPECT_FAIL with the mode set to Abort
does not cause the current test function to abort. Previously, such an
error would not cause the selftest to fail and someone would have to see
the warning message in the test output to know that there is a problem.
Change-Id: I2bd58fafe6b51ea0ab86fde7c5afb781e4534ee4
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The existing expectfail selftest did not test QEXPECT_FAIL with a
data-driven test function. This commit adds such a test.
Change-Id: I39fa9aa227b58779ce5268dd37bf55468e7269c5
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This test relies on being able to set a custom QLibraryInfo::PrefixPath
for the duration of the test. However, the test code neglected to
account for the following behavior on mac (from "Using qt.conf"
documentation):
On Mac OS X, the Prefix is relative to the Contents in the application
bundle.
Change-Id: Ie9b6d5ebfe8af7d7f332e827069b60a830d6c6f2
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- Changed to detect valgrind at runtime and skip test instead of failing.
- subprograms inherit QT_QPA_PLATFORM value from parent if set.
Change-Id: I280acee389df1ee74ee6758a0dd1601226e103c7
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
- All subprograms installed as part of test instead of separate installs
- Added installTestHelperApp() to testcase.prf
Change-Id: I02fbbb299f095054c9216ad0e5f91f574fb0fe3d
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
The previous change missed some headers from years prior to 2011, and a
few new files were merged after the previous change.
Change-Id: Ib7d1a2b7062228c2a5373da64242b2ee1f0981e1
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Changed selftests unittest to use specific classes of QtCore instead of
pulling in all of QtCore headers by using include <QtCore>
- Decreasing build time.
Change-Id: Ic53a70423ecafeb342cc05ae6d7dee795533d205
Reviewed-by: Kurt Korbatits <kurt.korbatits@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Changed these tests to use QFINDTESTDATA macro to detect location of testdata.
Checking for a specific file contained in the testdata so as not to be confused by empty directories
created during configure.
Change-Id: Iac2ac6304b6b9ac79e00886025b93ec0af5a8507
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
... and deprecate the old registerTimer() functions. The new pure-
virtual registerTimer() breaks source-compatibility. Subclasses cannot
be instantiated anymore, since the pure virtual function signature has
changed.
QAbstractEventDispatcher::TimerInfo is no longer a QPair. It is now a
struct with timerId, interval, and timerType members. This is a source
incompatibility that should only affect subclasses of
QAbstractEventDispatcher, which will need to pass 3 arguments to the
TimerInfo constructor instead of 2. If the subclass used QPair<int,int>
instead of the TimerInfo typedef, the QPair<int,int> declarations will
need to be replaced with TimerInfo.
Call the new registerTimer() function with the type from
QObject::startTimer(). Change all subclasses of QAbstractEventDispatcher
to reimplement the new virtual function. The type argument is unused at
the momemnt, except to ensure that registeredTimers() returns the type
each timer was registered with. Implementations for the various
dispatchers will be done in separate commits.
Author: Thiago Macieira <thiago.macieira@nokia.com>
Change-Id: Ia22697e0ab0847810c5d162ef473e0e5a17a904b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Introduce a specialization for qCompare(QImage,QImage) that
checks isNull, size and format and outputs verbose messages.
Check isNull, size similarly for QPixmap.
Add an autotest:
- Add test to cmptest and make it a GUI application since
QImage requires QGuiApplication.
- Make testlib/selftests capable of running X11-GUI applications
by passing DISPLAY.
- Ignore stderr output for cmptest
- Add test data
Change-Id: I2b29c7822fbeedf2b22c90889739ed7ff859ce92
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Required for launching the sub-processes. The test worked only
when launched locally, not from 'make check'.
Change-Id: I42c9202a7726c3135f94445fb336b2b8241535a4
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@nokia.com>
