gtk/testsuite/meson.build

77 lines
2.0 KiB
Meson
Raw Normal View History

2017-10-10 04:18:21 +00:00
gtk_libexecdir = join_paths(gtk_prefix, get_option('libexecdir'))
installed_test_bindir = join_paths(gtk_libexecdir, 'installed-tests', 'gtk-4.0')
installed_test_datadir = join_paths(gtk_datadir, 'installed-tests', 'gtk-4.0')
common_env = [
'GIO_USE_VOLUME_MONITOR=unix',
'GSETTINGS_BACKEND=memory',
'GTK_CSD=1',
'G_ENABLE_DIAGNOSTIC=0',
'GSETTINGS_SCHEMA_DIR=@0@'.format(gtk_schema_build_dir),
'GDK_DEBUG=default-settings',
]
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
exclude_unstable = ['flaky', 'failing']
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
setups = [
{ 'backend': 'x11', 'if': x11_enabled, },
{ 'backend': 'wayland', 'if': wayland_enabled, 'is_default': true, },
{ 'name': 'wayland_gles',
'backend': 'wayland', 'if': wayland_enabled,
'env': ['GDK_DEBUG=gl-gles,default-settings',
'MESA_GLES_VERSION_OVERRIDE=2.0',
'MESA_EXTENSION_OVERRIDE=-GL_OES_vertex_array_object',
'GSK_MAX_TEXTURE_SIZE=1024',
], },
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
{ 'backend': 'win32', 'if': os_win32 },
{ 'backend': 'broadway', 'if': broadway_enabled, },
{ 'backend': 'win32', 'if': os_win32 },
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
]
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
foreach setup : setups
if setup.get('if')
backend = setup.get('backend')
name = setup.get('name', backend)
exclude = []
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
if backend != 'broadway'
exclude += 'gsk-compare-broadway'
endif
if name == 'wayland_gles'
exclude += 'wayland_gles_failing'
endif
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
env = common_env + [
'GDK_BACKEND=@0@'.format(backend),
] + setup.get('env', [])
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
add_test_setup(
name,
env: env + ['TEST_OUTPUT_SUBDIR=@0@'.format(name)],
exclude_suites: exclude_unstable + exclude,
is_default: setup.get('is_default', false),
)
testsuite: Use separate setups for unstable tests instead of should_fail There are two possible interpretations of "expected failure": either the test *must* fail (exactly the inverse of an ordinary test, with success becoming failure and failure becoming success), or the test *may* fail (with success intended, but failure possible in some environments). Autotools had the second interpretation, which seems more useful in practice, but Meson has the first. Instead of using should_fail, we can put the tests in one of two new suites: "flaky" is intended for tests that succeed or fail unpredictably according to the test environment or chance, while "failing" is for tests that ought to succeed but currently never do as a result of a bug or missing functionality. With a sufficiently new version of Meson, the flaky and failing tests are not run by default, but can be requested by running a setup that does not exclude them, with a command like: meson test --setup=x11_unstable --suite=flaky --suite=failing As a bonus, now that we're setting up setups and their excluded suites programmatically, the gsk-compare-broadway tests are also excluded by default when running the test setup for a non-broadway backend. When running the tests in CI, --suite=gtk overrides the default exclude_suites, so we have to specify --no-suite=flaky and --no-suite=failing explicitly. This arrangement is inspired by GNOME/glib!2987, which was contributed by Marco Trevisan. Signed-off-by: Simon McVittie <smcv@debian.org>
2022-11-23 19:13:32 +00:00
add_test_setup(
'@0@_unstable'.format(name),
env: env + ['TEST_OUTPUT_SUBDIR=@0@_unstable'.format(name)],
exclude_suites: exclude,
)
endif
endforeach
subdir('performance')
subdir('gdk')
subdir('gsk')
2016-08-21 16:01:14 +00:00
subdir('gtk')
subdir('css')
subdir('a11y')
subdir('tools')
subdir('reftests')
if build_gir
subdir('introspection')
endif
if wayland_enabled
subdir('headless')
endif