AuroraMiddleware/gtk
1
0
Fork 1
mirror of https://gitlab.gnome.org/GNOME/gtk.git synced 2024-12-28 06:21:14 +00:00
Code Issues Projects Releases Wiki Activity
bf87a40b20
gtk/.gitlab-ci/run-tests.sh

100 lines
2.6 KiB
Bash
Raw Normal View History

ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
#!/bin/bash
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
set +x
set +e
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
srcdir=$( pwd )
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
builddir=$1
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
backend=$2
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
multiplier=${MESON_TEST_TIMEOUT_MULTIPLIER:-1}
ci: Refactor the CI pipeline We should have a single job for building both static and shared versions of GTK. On the other hand, having a separate job for the release build would be a plus. Additionally, we shouldn't use an opaque script to build GTK; the only step in the process that requires hand-holding is running the tests suites under Xvfb, and having the build options visible from the YAML file gives us a better idea of what kind of build we're running.
2020-01-29 09:35:10 +00:00
ci: Set LSAN_OPTIONS for tests When running the testsuite with the address sanitizer, many of our dependencies cause it to report cause it to report memory leaks, causing tests to fail. Therefore, point the leak sanitizer at a list of suppressions. The list is kept in the lsan.supp file in git.
2020-07-09 02:19:32 +00:00
# Ignore memory leaks lower in dependencies
ci: More verbose output from asan
2022-12-20 04:36:19 +00:00
export LSAN_OPTIONS=suppressions=$srcdir/lsan.supp:print_suppressions=0:verbosity=1:log_threads=1
Force malloc of g_slices in asan build Otherwise leaks could be missed.
2021-01-22 05:13:13 +00:00
export G_SLICE=always-malloc
ci: Set LSAN_OPTIONS for tests When running the testsuite with the address sanitizer, many of our dependencies cause it to report cause it to report memory leaks, causing tests to fail. Therefore, point the leak sanitizer at a list of suppressions. The list is kept in the lsan.supp file in git.
2020-07-09 02:19:32 +00:00
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
case "${backend}" in
x11)
ci: Pass -norest to Xvfb This may help with the sporadic "No display" failures we are seeing when running tests under X11 in ci.
2020-08-04 15:59:32 +00:00
xvfb-run -a -s "-screen 0 1024x768x24 -noreset" \
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
--print-errorlogs \
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
--setup=${backend} \
ci: Add missing newline escape Otherwise we're going to run the accessibility test suite, and it's going to fail.
2020-01-29 11:31:00 +00:00
--suite=gtk \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
--no-suite=flaky \
ci: Don't run opengl renderer tests on Broadway The test setup mechanism in meson is not flexible enough to let us run different suites depending on setup, so just pass in explicitly which suites we want to skip, depending on the backend.
2020-05-16 14:12:47 +00:00
--no-suite=gsk-compare-broadway
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
# Store the exit code for the CI run, but always
# generate the reports
exit_code=$?
;;
ci: Run tests with small textures
2023-03-19 14:16:19 +00:00
wayland*)
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
export XDG_RUNTIME_DIR="$(mktemp -p $(pwd) -d xdg-runtime-XXXXXX)"
tests: Disable idle locking in headless weston In my local testing i was burned by the default 300sec value here. I'm not sure if we actually hit this in the CI, but lets not take any chances.
2020-05-28 13:15:04 +00:00
weston --backend=headless-backend.so --socket=wayland-5 --idle-time=0 &
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
compositor=$!
export WAYLAND_DISPLAY=wayland-5
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
--print-errorlogs \
--setup=${backend} \
--suite=gtk \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
--no-suite=flaky \
Skip some tests with small textures When textures are too large, we will scale them down to fit the max texture size, which will cause comparisons to fail.
2023-05-01 20:00:24 +00:00
--no-suite=${backend}_failing \
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
--no-suite=gsk-compare-broadway
exit_code=$?
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
kill ${compositor}
;;
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
broadway)
export XDG_RUNTIME_DIR="$(mktemp -p $(pwd) -d xdg-runtime-XXXXXX)"
${builddir}/gdk/broadway/gtk4-broadwayd :5 &
server=$!
export BROADWAY_DISPLAY=:5
meson test -C ${builddir} \
ci: Produce less terminal output Pass --quiet to meson test. Less is more, in this case.
2023-05-03 19:32:50 +00:00
--quiet \
Make run-test.sh easier to use locally Set a default timeout multiplier so we don't have to rememver obscure environment variables.
2023-05-02 10:36:18 +00:00
--timeout-multiplier "${multiplier}" \
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
--print-errorlogs \
--setup=${backend} \
--suite=gtk \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--no-suite=failing \
--no-suite=flaky \
ci: Don't run opengl renderer tests on Broadway The test setup mechanism in meson is not flexible enough to let us run different suites depending on setup, so just pass in explicitly which suites we want to skip, depending on the backend.
2020-05-16 14:12:47 +00:00
--no-suite=gsk-compare-opengl
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
kill ${server}
;;
testsuite: Add run for GLES This is supposed to test the most fallback GL stuff, so we might want to set even more env vars here. Also enable the run for the Fedora builder in CI.
2021-10-08 16:38:51 +00:00
*)
echo "Failed to add ${backend} to .gitlab-ci/run-tests.sh"
exit 1
;;
ci: Run tests with different backends Run the testsuite under x11, wayland and broadway in the fedora-x86_64 job. This requires us to use the v17 image which includes weston.
2020-05-15 13:44:24 +00:00
esac
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
ci: Fix linking of the CI assets in the report The report is relative to the build directory, and so are the generated assets.
2020-02-12 18:03:48 +00:00
cd ${builddir}
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
$srcdir/.gitlab-ci/meson-junit-report.py \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--project-name=gtk \
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
--backend="${backend}" \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--job-id="${CI_JOB_NAME}" \
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
--output="report-${backend}.xml" \
"meson-logs/testlog-${backend}.json"
$srcdir/.gitlab-ci/meson-html-report.py \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--project-name=gtk \
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
--backend="${backend}" \
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
--job-id="${CI_JOB_NAME}" \
ci: Stop running the unstable test setups This is just duplicating all our test runs, for no discernible gain.
2023-05-03 19:10:17 +00:00
--reftest-output-dir="testsuite/reftests/output/${backend}" \
--output="report-${backend}.html" \
"meson-logs/testlog-${backend}.json"
ci: Generate the report for the release builds We are going to need to hide the report generation into the test runner script, as we want the job to produce the reports even in case of failure, instead of bailing out immediately.
2020-01-29 13:36:23 +00:00
exit $exit_code
Reference in New Issue Copy Permalink