Hardcode the tests which were previously scanned from the source.
See qtbase/24e83de8d1924b8003c84f1df05b7befea2c5120.
Change-Id: I8fb05568977f86726b20948a9c2d1cfce5cba161
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
As discussed in the mailing list and in the Qt Contributor Summit 2019.
Tested on Linux, macOS and FreeBSD, showing these fallbacks:
OS environment setlocale() call
FreeBSD empty "C.UTF-8"
FreeBSD LC_ALL=C "C.UTF-8"
Linux empty "C.UTF-8"
Linux LC_ALL=C "C.UTF-8"
Linux LANG=en_US "en_US.UTF-8"
Linux LANG=de_DE@euro "de_DE.UTF-8"
Linux LANG=en_GB.iso885915 "en_GB.UTF-8"
Linux LANG=hy_AM.armscii8 "hy_AM.UTF-8"
Linux LANG=ja_JP.sjis "ja_JP.UTF-8"
Linux LANG=ru_RU.koi8r "ru_RU.UTF-8"
macOS empty "UTF-8"
macOS LC_ALL=C "UTF-8"
Versions tested: FreeBSD 12.1, Linux w/ glibc 2.30, macOS 10.14.2.
See
* https://wiki.qt.io/Qt_Contributor_Summit_2019_-_QtCore
* https://lists.qt-project.org/pipermail/development/2019-October/037791.html
Change-Id: Ia2aa807ffa8a4c798425fffd15d97ddb4f35b0ae
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
The test orchestrator shouldn't have to deal with the individual options
needed for each test.
Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The reporter was probably named 'xunit' based on the historical use of
xUnit to refer to testing frameworks derived from Smalltalk's SUnit.
These frameworks typically added their own prefix, e.g. JUnit for Java,
RUnit for R, etc.
The most popular of these was the JUnit framework, and the corresponding
XML output produced by the Ant built tool became somewhat of a de facto
standard, which is probably why we chose to model our reporter after it.
Nowadays however, naming it 'xunit' is problematic as there is actually
a testing famework named xUnit.net, typically shortened to, you guessed
it: xunit.
Test report consumers will typically have a junit mode, and an xunit
mode, and the latter could easily be mistaken for what testlib outputs,
unless we clarify this.
The clarification also allows us to safely extend our support for the
JUnit XML format to incorporate some elements that are nowadays common,
but where we are lagging behind the standard.
[ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has
been renamed to what it actually is: a JUnit test reporter, and is now
triggered by passing -o junitxml to the test binary.
Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
It has been completely untested for a while and saw some breakage.
So let's add a selftest for it. ('-vs' when running tests)
Change-Id: Ibfb5ac0a2d741de7c3f519d91202d4977996045e
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Regexes have long specified that a [ as the first character inside a
[...] is just a literal [, but apparently we need to escape it now, to
avoid a "nested set" FutureWarning.
Change-Id: I76a48c9aafb0684a1d6b0d5284fe9852c9ea0e43
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Samuel Gaist <samuel.gaist@idiap.ch>
Verify that it does cut in after the specified time has elapsed.
Task-number: QTPM-1385
Change-Id: Ib18e8d6af28339f79cca4d62b869287ce07b8cc1
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Match the environment tst_selftests.cpp uses for subtests more
faithfully. Extends b22e50acda. In the process, tweak how crashers
are handling, in preparation for the watchdog test.
Change-Id: I09a046460f6f3bff0b12069fad6c1437d89572ce
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
One test for bad data for the column, another for a bad QFETCH.
Incidentally extend blacklist testing by blacklisting them.
Reorganise a QEMU condition that needed extended as part of this.
Task-number: QTPM-1385
Change-Id: Iac72ada19760321c5c9264ddfff7740d1fdd0700
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The testlib selftest sets various things in the environment for
crashing tests; the generator for its expected output should set the
same things, as they affect what output is produced.
Change-Id: Iec2ed59982ea1043582573530c33619d8e8ed08e
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This supplements b1945604a7, which
removed the qrc file in favor of test/test.pro coding for it.
Change-Id: I15507c89ca14fa6e6b8223de671ffff7092272d0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When generate_expected_output.py is run for an in-source build, the
raw output contains no paths to the sources for the script to whittle
down, as it does for shadow builds, to just the path from qtbase down.
So kludge together some extra regexes that can fix that up and tweak
some relevant code to provide them with the data they need.
Change-Id: I656d7126087bd9ad20b2af6835fba314d90a171d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Otherwise random debug messages from Qt might mess up the expected vs
actual results.
The setting of QT_LOGGING_RULES in initTestcase has been removed, as
the selftest overrides that for each invocation of a subtests, via
the processEnvironment() function.
Task-number: QTQAINFRA-1631
Change-Id: I855d31274f8261f8b125df23409353f7101be0e4
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
On some systems (e.g. Debian/buster) a standard Qt package may install
a qtlogging.ini file, e.g. in /etc/xdg/QtProject/, which suppresses
QDEBUG output. A ~/.config/QtProject/qtlogging.ini could interfere
similarly. This can break the selftest, when run with expected_*
files generated without such interference. Likewise, if those
expected_* files are generated on a system such similar, they won't
work on CI. Given that this caused confusion and delay with
integrating the latest set of updates to the expected_* files, it
seemed best to save others from similar bafflement.
So set a standard value for QT_LOGGING_RULES in both the selftest and
the generator script, so we get consistency.
Change-Id: I649e2f1f6ead21edf8af051aaee286e369fed064
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Two more ways line numbers were making it through.
Corrected a doc-string to tell nearer to the truth.
Change-Id: I946aaeb936d47fffe50d7ec15e2524992cc9e428
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Document that the saved output is used by tst_selftests.cpp and use a
crude parse of it to get the list of subdirs that it actually tests.
Change-Id: I73023228c9e547f965b7749dd66de7ef09c3815e
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Restructured, separated the canonicalising of output lines out (into
an object that prepares the necessary regexes and replacements)
suppress more, changed the path-stripping to strip qtbase's parent
rather than os.getcwd() and took account of shadow builds (so both
source tree and build tree provide prefixes we want to strip from
paths). Also cope with $PWD potentially having symlinks in it, where
os.getcwd() is canonical.
It's possible some output might name files elsewhere in the source
tree; these won't be filtered by the prior cwd prefix removal; and, in
any case, the problem with cwd is only that the ancestry of qtbase is
apt to vary; paths relative to there should be consistent between test
runs. This change shall lead to a one-off rewrite of all expected_*
files; but it should now catch all paths. By stripping both build
root and source root (when different) it also avoids differences for
those doing out-of-source ("shadow") builds.
In our XML formats, any hyphens in root paths (e.g. I had Qt-5.6 in my
build root's path) got represented by a character entity, confounding
the replacement; so also do replacement that catches this. We may
discover other character entity subsitutions needed along with this.
Now filtering line numbers and timing information, including benchmark
results; these numbers all get replaced with 0 to avoid noisy diffs.
Also purging dangling hspace, to placate sanity-bot.
The module can now be imported - the code it runs is packaged as a
main() function that a __name__ == '__main__' stanza runs - and all
data is localised to where it's needed, rather than held in globals.
Tidied up and organized the existing regexes. There are doc-strings;
there is a short usage comment. Data is localised rather than global
and modern pythonic idioms get used where apt.
Regexes are compiled once instead of repeatedly. An object looks
after the list of patterns to apply and its construction handles all
anticipated problems. Failures are mediated by an exception.
The output file now gets written once, instead of twice (once before
editing, then over-write to edit), and Popen uses text mode, so that
write can do the same. Its command is delivered as an array, avoiding
the need to invoke a shell.
Instead of relying on qmake being in our path (which might give us a
bogus QT_VERSION if the one in path doesn't match our build tree), use
the relative path to qmake - we rely on being run in a specific
directory in the build tree, after all. Escape dots in the version
properly, so that 51730 doesn't get mistaken for 5.7.0 (for example),
and moved this check later in the sequence (matching a smaller target
makes it more likely to falsely match).
Overtly check we are in the right directory and tell the user what we
actually need, if run from the wrong place. Simplify handling of the
unsupported use-case for MS-Windows (but note what would be needed for
it).
Change-Id: Ibdff8f8cae173f6c31492648148cc345ae29022b
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
Conflicts:
qmake/library/qmakeevaluator.cpp
One side changed the iterator to use ranged-for, the other changed its
body; they only conflicted because the latter had to add braces around
the body, intruding on the for-line. Trivial resolution.
Change-Id: Ib487bc3bd6e3c5225db15f94b9a8f6caaa33456b
Otherwise, float numbers are formatted using a decimal comma
in German, causing huge diffs and failures.
Change-Id: Icd85a293d0564cac6be244eb0793611920d0c89c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Adapt the script to generate output for the new TeamCity format
added by fbd6acedac.
Change-Id: I9435382ec3daf80428c324c58434aa951841bf08
Reviewed-by: Borgar Øvsthus <borgar.ovsthus@fmcti.com>
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some
exceptions, see
http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/
Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one
(in those files which will be under GPL 3 with exceptions)
Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
Otherwise write() for my python (OS X 2.7) wants strings, which is incorrect.
Change-Id: Ibd9d050646d1039ba8370d121dd25756ceffdb7a
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
Qt copyrights are now in The Qt Company, so we could update the source
code headers accordingly. In the same go we should also fix the links to
point to qt.io.
Outdated header.LGPL removed (use header.LGPL21 instead)
Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing
combination. New header.LGPL-COMM taken in the use file which were
using old header.LGPL3 (src/plugins/platforms/android/extract.cpp)
Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license
combination
Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe
Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
Do not apply additional encoding when reading process output.
Fixes errors when encountering UTF-8:
Traceback (most recent call last):
File "./generate_expected_output.py", line 117, in <module>
generateTestData(path)
File "./generate_expected_output.py", line 106, in generateTestData
out.write(data.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 5485: ordinal not in range(128)
Change-Id: Ib827787a59a18b4d3d0601645856517f43c01fc3
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
Evaluate command line arguments and use directories only when empty.
Change-Id: I818ec13c686018a3f607e91174e57d8f8bbf15f7
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@digia.com>
The actual code in testlib now indents the duration tag when it is for a
function.
Change-Id: Iee62db9c81f11dc54e57f166bf9fb2b7012b7e03
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
This makes testlib a lot more hackable since when the output is changed
in a defined way we can regenerate the expected output (eg adding the
test duration which will be a follow-up patch).
Note that the script does not work properly on Windows and not all
benchmarks work, but at least it reduces the work of adapting files to a few lines.
Change-Id: I910387cd92ff82aa629747a3a3033dae17fbd711
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@digia.com>
