The skipcleanup and failcleanup tests were actually testing skip and
fail in cleanupTestCase(), not in cleanup(). Add almost-duplicate
tests and clean up so that we now have {fail,skip}cleanup(,testcase}
tests to cover all four cases. Generated expected output. The new
tests (with old names) get their fail or skip - during cleanup() -
reported against the test instead of the cleanupTestCase function.
(Results for {init,cleanup}TestCase() are always reported, even when
these slots are not defined, as no-op passes.)
Pick-to: 6.4
Change-Id: I0988d1696b50c0e2f30c45ddc25e1bd0bfd2151a
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
[ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
macros. These new macros behave similarly to QVERIFY(a op b),
where 'op' is ==, !=, <, <=, >, >= respectively, but print
a formatted error message with argument values in case of failure.
The formatting is done lazily, which means that the strings will
be generated only when the comparison fails.
Also add a new test for tst_selftest and generate expected output
for it.
Fixes: QTBUG-98873
Task-number: QTBUG-98874
Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Multiple tests use qt_internal_add_resource that copies the
functionality that is already implemented inside the
qt_internal_add_test function. Simplify these test by replacing
the qt_internal_add_resource call with the new BUILTIN_TESTDATA
option.
Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This ensures that a command such as
$ ninja tst_qlocale && ninja tst_qlocale_check
will automagically build the syslocaleapp program that the test runs
from a subtest. Similar for testlib's selftests and tst_QProcess.
As a drive-by, pruned some legacy comments from when CMakeLists.txt
files were generated from .pro files.
Change-Id: I67691a8175aaef124d4104cf1898193993408bdf
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The JUnit reporter was initially named xunit, but the naming was inaccurate
and the reporter was renamed in 27db9e458c.
The corresponding test has now been renamed as well, and as an added bonus
we only run it for that reporter.
Pick-to: 6.2
Change-Id: I59cb7d949514cdf46a0199a53a7a3e39f833207c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
The MinGW resource compiler fails to handle compile definitions
with multiple values.
When the resource file is compiled as part of the main target rather
than a separate object library, the resource generation rule will
inherit all the compile definitions from the main target.
For the case of tst_selftests this causes errors like
gcc: error: badxml\: No such file or directory
gcc: error: benchlibcallgrind\: No such file or directory
gcc: error: benchlibcounting\: No such file or directory
gcc: error: benchlibeventcounter\: No such file or directory
gcc: error: benchliboptions\: No such file or directory
Limit the compile definition to the C++ language only, so the multiple
values are not passed to the resource compiler.
Pick-to: 6.2
Change-Id: Ie53666839272556323b50d79c090f0dc71745d11
Reviewed-by: Cristian Adam <cristian.adam@qt.io>
The emulation detection has been usable only on qtbase tests, move it to
QTest so that it can be used in other modules as well.
Pick-to: 6.1
Change-Id: I4b2321b7856414d7b1cfd5e6b1405a633c6bb878
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Modify special case locations to use the new API as well.
Clean up some stale .prev files that are not needed anymore.
Clean up some project files that are not used anymore.
Task-number: QTBUG-86815
Change-Id: I9947da921f98686023c6bb053dfcc101851276b5
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
It doesn't understand the arguments we usually pass to testlib.
Change-Id: Iea83d1d8c31a510b2bc442cbfc810eac631322e7
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Macros and the await helper function from qfunctions_winrt(_p).h are
needed in other Qt modules which use UWP APIs on desktop windows.
Task-number: QTBUG-84434
Change-Id: Ice09c11436ad151c17bdccd2c7defadd08c13925
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
We use the Catch2 testing framework to test Qt Testlib, which also opens
up the possibility of using it for other internal testing once it's made
available through the build system.
The test now has a --rebase mode which will write out the actual results
as new expected files. Once we add the required post-processing to the
results to remove timestamps and other testrun-specific data we can
remove the standalone python script generate_expected_output.py that
today has to be kept in sync with the test itself.
No attempt has been made to clean up the comparison-functions, but
these could all benefit from moving their logic from the comparison
to the sanitization step. This will both make the expected files
more generic, and will reduce the diff once a failure occurs, since
we're not seeing all the hunks that the comparison-functions ignored.
Change-Id: I1769d42e7958d56d1ad5da958db0e8fe3a2a3c23
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
The use of a single subdirectory only for the test binary, but not the
actual source file, just made things harder to discover when looking
for the test code. Now all subdirectories are actual sub-tests.
Change-Id: Ia3e308ba8d231cc8ead1491b10a34801f76669b0
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This patch adds the equivalent of testlib's selfcover.pri and is enabled
for both testlib and the respective selftests test.
This patch also fixes the selftests so that they can run without
FEATURE_testlib_selfcover enabled.
Change-Id: I15913de2d572ac79804ce3e652cee66de74318f8
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
This will help with future regeneration.
Amends 6d1c695b0e
Change-Id: I2454dc36cfc00f7aad72488fb7927d7807978fd8
Reviewed-by: Qt CMake Build Bot
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Actually build tst_selftest
Fix the subdirs of it not to be actually built as tests (so that ctest
doesn't pick them up and tries to run them)
Change-Id: Id9d069f44241027dbd103884d36551163382b216
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CMake Build Bot
It isn't being built in the current qmake build either (maybe because it
doesn't build because uses sleep())
Change-Id: Ie97287c6e45d1fcc32063bd9eb37ff2907dfa891
Reviewed-by: Tobias Hunger <tobias.hunger@qt.io>
