There are two possible interpretations of "expected failure": either
the test *must* fail (exactly the inverse of an ordinary test, with
success becoming failure and failure becoming success), or the test
*may* fail (with success intended, but failure possible in some
environments). Autotools had the second interpretation, which seems
more useful in practice, but Meson has the first.
Instead of using should_fail, we can put the tests in one of two new
suites: "flaky" is intended for tests that succeed or fail unpredictably
according to the test environment or chance, while "failing" is for
tests that ought to succeed but currently never do as a result of a
bug or missing functionality. With a sufficiently new version of Meson,
the flaky and failing tests are not run by default, but can be requested
by running a setup that does not exclude them, with a command like:
meson test --setup=x11_unstable --suite=flaky --suite=failing
As a bonus, now that we're setting up setups and their excluded suites
programmatically, the gsk-compare-broadway tests are also excluded by
default when running the test setup for a non-broadway backend.
When running the tests in CI, --suite=gtk overrides the default
exclude_suites, so we have to specify --no-suite=flaky and
--no-suite=failing explicitly.
This arrangement is inspired by GNOME/glib!2987, which was contributed
by Marco Trevisan.
Signed-off-by: Simon McVittie <smcv@debian.org>
This is supposed to test the most fallback GL stuff, so we might want to
set even more env vars here.
Also enable the run for the Fedora builder in CI.
The leak sanitizer causes on average 3-4 tests
to segfault during a testsuite run. Disable it
for now to see if we can get a successful
testsuite run with asan alone.
When running the testsuite with the address sanitizer,
many of our dependencies cause it to report cause it
to report memory leaks, causing tests to fail.
Therefore, point the leak sanitizer at a list of
suppressions. The list is kept in the lsan.supp
file in git.
The test setup mechanism in meson is not flexible
enough to let us run different suites depending
on setup, so just pass in explicitly which suites
we want to skip, depending on the backend.
We want the test names in the junit xml to be
unique across all the tests in a job, so we need
to include the backend in the test name.
And we also want to see the used backend in
the html report.
We are going to need to hide the report generation into the test runner
script, as we want the job to produce the reports even in case of
failure, instead of bailing out immediately.
We should have a single job for building both static and shared
versions of GTK. On the other hand, having a separate job for the
release build would be a plus.
Additionally, we shouldn't use an opaque script to build GTK; the only
step in the process that requires hand-holding is running the tests
suites under Xvfb, and having the build options visible from the YAML
file gives us a better idea of what kind of build we're running.
