gtk2/.gitlab-ci
Simon McVittie 957dd49ef7 testsuite: Use separate setups for unstable tests instead of should_fail
There are two possible interpretations of "expected failure": either
the test *must* fail (exactly the inverse of an ordinary test, with
success becoming failure and failure becoming success), or the test
*may* fail (with success intended, but failure possible in some
environments). Autotools had the second interpretation, which seems
more useful in practice, but Meson has the first.

Instead of using should_fail, we can put the tests in one of two new
suites: "flaky" is intended for tests that succeed or fail unpredictably
according to the test environment or chance, while "failing" is for
tests that ought to succeed but currently never do as a result of a
bug or missing functionality. With a sufficiently new version of Meson,
the flaky and failing tests are not run by default, but can be requested
by running a setup that does not exclude them, with a command like:

    meson test --setup=x11_unstable --suite=flaky --suite=failing

As a bonus, now that we're setting up setups and their excluded suites
programmatically, the gsk-compare-broadway tests are also excluded by
default when running the test setup for a non-broadway backend.

When running the tests in CI, --suite=gtk overrides the default
exclude_suites, so we have to specify --no-suite=flaky and
--no-suite=failing explicitly.

This arrangement is inspired by GNOME/glib!2987, which was contributed
by Marco Trevisan.

Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-24 13:46:33 +00:00
..
pages docs: Add a side bar to the docs.gtk.org landing page 2021-03-23 14:58:03 +00:00
.lock ci: Add a build with asan 2020-07-09 19:43:06 -04:00
clang-format-diff.py ci: Add a style check pass 2020-02-11 14:47:22 +00:00
fedora.Dockerfile ci: Update the Fedora image to Fedora 36 2022-05-14 12:23:02 -04:00
flatpak-build.sh flatpak-build.sh: change the buildtype to debugoptimized 2022-07-25 15:42:14 +03:00
meson-html-report.py CI: Include reftest nodes in artifacts 2022-05-18 19:54:34 +02:00
meson-junit-report.py Update references to master in the repository 2021-11-29 17:37:49 -05:00
README.md ci: Update the Fedora image to Fedora 36 2022-05-14 12:23:02 -04:00
run-docker.sh Rename base version argument in run-docker.sh 2021-03-11 16:37:29 +00:00
run-style-check-diff.sh ci: Improve style check script 2022-09-09 19:41:42 -04:00
run-tests.sh testsuite: Use separate setups for unstable tests instead of should_fail 2022-11-24 13:46:33 +00:00
show-info-linux.sh ci: Show OS release for our containers 2021-05-05 16:22:32 -04:00
show-info-osx.sh ci: Show OS release for our containers 2021-05-05 16:22:32 -04:00
test-docker.sh build: disable Vulkan by default 2021-07-20 14:00:25 -04:00
test-msvc.bat CI: bump meson version from 0.59 to 0.60.3 2022-05-22 11:46:54 +02:00
test-msys2.sh CI: remove workaround for some subproject builds failing on Windows 2022-08-04 22:26:04 +02:00

GTK CI infrastructure

GTK uses different CI images depending on platform and jobs.

The CI images are Docker containers, generated either using docker or podman, and pushed to the GitLab container registry.

Each Docker image has a tag composed of two parts:

  • ${image}: the base image for a given platform, like "fedora" or "debian-stable"
  • ${number}: an incremental version number, or latest

See the container registry for the available images for each branch, as well as their available versions.

Note that using latest as version number will overwrite the most recently uploaded image in the registry.

Checklist for Updating a CI image

  • Update the ${image}.Dockerfile file with the dependencies
  • Run ./run-docker.sh build --base ${image} --version ${number}
  • Run ./run-docker.sh push --base ${image} --version ${number} once the Docker image is built; you may need to log in by using docker login or podman login
  • Update the image keys in the .gitlab-ci.yml file with the new image tag
  • Open a merge request with your changes and let it run

Checklist for Adding a new CI image

  • Write a new ${image}.Dockerfile with the instructions to set up a build environment
  • Add the pip3 install meson incantation
  • Run ./run-docker.sh build --base ${image} --version ${number}
  • Run ./run-docker.sh push --base ${image} --version ${number}
  • Add the new job to .gitlab-ci.yml referencing the image
  • Open a merge request with your changes and let it run

Checklist for Adding a new dependency to a CI image

Our images are layered, and the base (called fedora-base) contains all the rpm payload. Therefore, adding a new dependency is a 2-step process:

  1. Build and upload fedora-base:$version+1
  2. Build and upload fedora:$version+1 based on fedora-base:version+1