In selftests.qrc, imposed alphabetic order (on stem of name, then on
suffix, effectively treating . as sorting before any letter) while
removing old tests and adding new tests and data. Updated all non-csv
files and added many missing files. (Not clear on csv support status;
the script seems to have dropped it after 5.6, but the test still uses
it.)
Left expected_crashes* alone (no new files added, no update to old) as
I don't get results resembling those anticipated.
Omitted printdatatagswithglobaltags, printdatatags due to dangling
hspace on output lines, which upset sanity-bot. A change to the test
cpp is needed to make it viable to skip that dangling hspace.
Change-Id: Iab3fb626c44a91c249b2fb626c12c75ea0317098
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
We need to have a finer grained control over the tests
we skip in our CI system. This adds a blacklisting
mechanism that allows blacklisting individual test
functions (or even test data) using a set of predefined
matching keys for the operating system and some other
relevant variables.
QTestlib will search for a file called BLACKLIST in the test
directory and parse it if found. The file contains a simple
ini style list of functions to blacklist. For details see
qtestblacklist.cpp.
Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
The newly added generate_expected_output.py was used to get the expected
output into a more reproducible state.
Change-Id: I1ca75c8e0c5778d25c1df531bd298007aac0ff4a
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
"QTest" is the C++ namespace; "QtTest" is the library name
- Edited the logger output in qplaintestlogger.cpp
- Updated documentation
- Updated expected outputs for self-tests
Change-Id: I43c525c43221a8d4e843a00d6d55b0f06ef55fd7
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Prior to this commit, a benchmark test could report 0..n passes and 0..m
fails or skips, where n is the number of accumulation iterations used to
collect benchmark data and m is the number of times the test function
was invoked. Depending on the type of benchmark measurer being used,
this could result in a very large volume of test output and inconsistent
pass, fail and skip counts between test runs.
This commit changes the behaviour so that each benchmark test reports
one pass, fail or skip, regardless of the number of iterations used to
collect benchmark data.
This commit also prevents benchmark data being reported in the test
output if the benchmark test failed or skipped, as any benchmark data is
of dubious value in such cases.
The latter change in behaviour requires a minor modification to the
badxml selftest, which now tests quoting of literal strings in xml test
output for both passing and failing benchmarks.
Finally, this commit also adds a new selftest specifically for verifying
correct behaviour for benchmarks that fail or skip.
Task-number: QTBUG-24313
Change-Id: I3426dc659a7511b62fd183a031c7235bc753f497
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
