AuroraMiddleware/gtk
1
0
Fork 1
mirror of https://gitlab.gnome.org/GNOME/gtk.git synced 2024-11-09 18:30:08 +00:00
Code Issues Projects Releases Wiki Activity
4d9d50d8e1
gtk/.gitlab-ci/run-tests.sh

106 lines
2.9 KiB
Bash
Raw Normal View History

ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
#!/bin/bash
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
set -x
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
set +e
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
srcdir=$( pwd )
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
builddir=$1
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
setup=$2
suite=$3
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
multiplier=${MESON_TEST_TIMEOUT_MULTIPLIER:-1}
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
ci: Set LSAN_OPTIONS for tests When running the testsuite with the address sanitizer, many of our dependencies cause it to report cause it to report memory leaks, causing tests to fail. Therefore, point the leak sanitizer at a list of suppressions. The list is kept in the lsan.supp file in git.
2020-07-09 02:19:32 +00:00
# Ignore memory leaks lower in dependencies
ci: Tweak asan options Allow the allocator to return NULL, so our g_try_malloc tests don't fail.
2023-06-06 01:09:51 +00:00
export LSAN_OPTIONS=suppressions=$srcdir/lsan.supp:print_suppressions=0:detect_leaks=0:allocator_may_return_null=1
Force malloc of g_slices in asan build Otherwise leaks could be missed.
2021-01-22 05:13:13 +00:00
export G_SLICE=always-malloc
ci: Set LSAN_OPTIONS for tests When running the testsuite with the address sanitizer, many of our dependencies cause it to report cause it to report memory leaks, causing tests to fail. Therefore, point the leak sanitizer at a list of suppressions. The list is kept in the lsan.supp file in git.
2020-07-09 02:19:32 +00:00
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
case "${setup}" in
x11*)
ci: Pass -norest to Xvfb This may help with the sporadic "No display" failures we are seeing when running tests under X11 in ci.
2020-08-04 15:59:32 +00:00
xvfb-run -a -s "-screen 0 1024x768x24 -noreset" \
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
--print-errorlogs \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--setup=${setup} \
--suite=${suite//,/ --suite=} \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--no-suite=${setup}_failing \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=flaky \
ci: Skip the headless tests Despite waiting for a long time, these still regularly time out on slow runners.
2023-09-21 16:17:47 +00:00
--no-suite=headless \
ci: Don't run opengl renderer tests on Broadway The test setup mechanism in meson is not flexible enough to let us run different suites depending on setup, so just pass in explicitly which suites we want to skip, depending on the backend.
2020-05-16 14:12:47 +00:00
--no-suite=gsk-compare-broadway
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
# Store the exit code for the CI run, but always
# generate the reports
exit_code=$?
;;
ci: Run tests with small textures
2023-03-19 14:16:19 +00:00
wayland*)
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
export XDG_RUNTIME_DIR="$(mktemp -p $(pwd) -d xdg-runtime-XXXXXX)"
tests: Disable idle locking in headless weston In my local testing i was burned by the default 300sec value here. I'm not sure if we actually hit this in the CI, but lets not take any chances.
2020-05-28 13:15:04 +00:00
weston --backend=headless-backend.so --socket=wayland-5 --idle-time=0 &
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
compositor=$!
export WAYLAND_DISPLAY=wayland-5
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
--print-errorlogs \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--setup=${setup} \
--suite=${suite//,/ --suite=} \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--no-suite=${setup}_failing \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=flaky \
ci: Skip the headless tests Despite waiting for a long time, these still regularly time out on slow runners.
2023-09-21 16:17:47 +00:00
--no-suite=headless \
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
--no-suite=gsk-compare-broadway
exit_code=$?
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
kill ${compositor}
;;
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
broadway*)
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
export XDG_RUNTIME_DIR="$(mktemp -p $(pwd) -d xdg-runtime-XXXXXX)"
${builddir}/gdk/broadway/gtk4-broadwayd :5 &
server=$!
export BROADWAY_DISPLAY=:5
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
--print-errorlogs \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--setup=${setup} \
--suite=${suite//,/ --suite=} \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--no-suite=${setup}_failing \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=flaky \
ci: Skip the headless tests Despite waiting for a long time, these still regularly time out on slow runners.
2023-09-21 16:17:47 +00:00
--no-suite=headless \
ci: Don't run opengl renderer tests on Broadway The test setup mechanism in meson is not flexible enough to let us run different suites depending on setup, so just pass in explicitly which suites we want to skip, depending on the backend.
2020-05-16 14:12:47 +00:00
--no-suite=gsk-compare-opengl
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
kill ${server}
;;
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
*)
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
echo "Failed to add ${setup} to .gitlab-ci/run-tests.sh"
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
exit 1
;;
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
esac
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
ci: Fix linking of the CI assets in the report The report is relative to the build directory, and so are the generated assets.
2020-02-12 18:03:48 +00:00
cd ${builddir}
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
$srcdir/.gitlab-ci/meson-junit-report.py \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--project-name=gtk \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--backend="${setup}" \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--job-id="${CI_JOB_NAME}" \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--output="report-${setup}.xml" \
"meson-logs/testlog-${setup}.json"
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
$srcdir/.gitlab-ci/meson-html-report.py \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--project-name=gtk \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--backend="${setup}" \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--job-id="${CI_JOB_NAME}" \
ci: Tweak the run-tests script The main change here is to pass the suites to run as an explicit argument. This is in preparation for running less tests.
2023-11-19 13:22:27 +00:00
--reftest-output-dir="testsuite/reftests/output/${setup}" \
--output="report-${setup}.html" \
"meson-logs/testlog-${setup}.json"
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
exit $exit_code
Reference in New Issue Copy Permalink