The batch test target name is a useful thing to obtain in various
target-generating scripts.
Change-Id: I9605cf860fe1485e48108eba7e93f9064209d8fb
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
An approach of test batching (joining multiple tests into a single
binary) has been taken, due to long linking times/binary size on certain
platforms, including WASM. This change adds a new feature
'batch_test_support' in Qt testlib. Based on the value of the feature,
test batching may become enabled with the -batch-tests switch.
Batching works for every target added via qt_internal_add_test. When
first such target is being processed, a new combined target for all of
the future test sources is created under the name of 'test_batch'.
CMake attempts to merge the parameters of each of the tests, and some
basic checks are run for parameter differences that are impossible to
reconcile.
On the C++ level, convenience macros instantiating the tests are
redefined when batch_tests is on. The new, changed behavior triggered
by the changes in the macros registers the tests in a central test
registry, where they are available for execution based solely on their
test name. The test name is interoperable with the names CMake is aware
of, so CTest is able to run the tests one by one in the combined binary.
Task-number: QTBUG-105273
Change-Id: I2b6071d58be16979bd967eab2d405249f5a4e658
Reviewed-by: Topi Reiniö <topi.reinio@qt.io>
It's needed for creating qmake build tests.
CMake / CTest has a limitation of not allowing to create single-config
tests when using a multi-config generator using the add_test(NAME)
signature.
Using add_test(NAME) forcefully creates per-config tests, which means
that it's not possible to just run ctest to execute tests, without
specifying a -C parameter, which we do in the CI.
qmake tests need to use the add_test(NAME) signature
to specify the WORKING_DIRECTORY option.
Because of the above limitation, a work around is to not use the
add_test(NAME) signature, but instead delegate the working directory
assignment to a generated cmake script, which
_qt_internal_create_command_script can already do.
Pick-to: 6.4
Task-number: QTBUG-96058
Change-Id: I6f439165994671724157f0edb7a71e351271e329
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
This ensures that tests can be executed on the simulator or device, by
doing the necessary steps like setting a bundle identifier, Info.plist
file, launch screen, etc.
This is done by calling _qt_internal_finalize_executable in the
implementation of all internal test adding functions.
The finalizers are limited only to iOS for now, as an incremental
step, and to ensure we don't accidentally break tests on other
platforms.
At least WebAssembly uses its own finalizers which would likely cause
duplicate calls if the _qt_internal_finalize_executable was
unconditional.
Pick-to: 6.4
Fixes: QTBUG-104754
Change-Id: I729d56385dd206b22c975fc2ce4e2c683e6e4e2c
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
CMakeLists.txt and .cmake files of significant size
(more than 2 lines according to our check in tst_license.pl)
now have the copyright and license header.
Existing copyright statements remain intact
Task-number: QTBUG-88621
Change-Id: I3b98cdc55ead806ec81ce09af9271f9b95af97fa
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
Warn projects not to use it because PUBLIC_LIBRARIES don't make
sense for executable targets and it also led to some issues in the
internal functions where some of them did not expect to receive
PUBLIC_LIBRARIES.
To ensure builds don't needlessly break, treat PUBLIC_LIBRARIES values
as regular LIBRARIES. In the future we might add an error instead.
Using PUBLIC_LIBRARIES in qt_internal_add_app, etc, accidentally
worked because the option name and the values following it were
parsed as values of the "previous" option, like SOURCES or
INCLUDE_DIRECTORIES or LIBRARIES, and when those got
passed through to qt_internal_extend_target, things magically worked.
We have a lot of projects using PUBLIC_LIBRARIES, mostly due to the
way qmake pro files were written and how pro2cmake converted them.
We'll have to clean up each repo.
Change-Id: I69e09d34afdf98f0d47c08d324643fc986f8131c
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Added a small helper function that also installs the script to
prefix builds. Additionally, leveraging the cmake option
CROSSCOMPILING_EMULATOR for the testrunner fits neatly in the rest
of our cmake code.
Change-Id: I75288e97c81b250ac3997f2e7a22bc7bd82b7b69
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Add a python script that allows us to run wasm tests in CI, along with
the necessary cmake logic to install the script and execute tests
accordingly.
Change-Id: I93b95c115538c4e27b2b833405acab8162be2a8a
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
CROSSCOMPILING_EMULATOR is a target property, not a test property.
Fixes: QTBUG-87864
Pick-to: 6.2 6.3
Change-Id: Icb07e9ed71a6bcbfceb7aa2116bf56eaa0a545c6
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Network autotests cannot connect to services on docker containers from
QNX qemu without all ports explicitly defined (both host and container)
side.
Pick-to: 6.2 6.3
Task-number: QTQAINFRA-4684
Change-Id: Iba44cfa17d42d43ecec3e29985e404f75d5fc232
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
This introduces a new helper function,
qt_internal_add_repo_local_defines and makes use of it in
qt_internal_add_{module,test,executable,benchmark,plugin}. That function
checks whether QT_EXTRA_INTERNAL_TARGET_DEFINES is set. If it is, the
defines listed in there will be aded to all targets passed to the
functions mentioned above.
The intended usage is that QT_EXTRA_INTERNAL_TARGET_DEFINES gets set
in the repository local .cmake.conf. This allows e.g. opting in to
source incompatible changes in leaf modules (as long as those are
guarded by some define).
Pick-to: 6.2 6.3
Fixes: QTBUG-101640
Change-Id: I06c3693ee69f46e95a48de724621f0c97e7cc3a8
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Multiple tests use qt_internal_add_resource that copies the
functionality that is already implemented inside the
qt_internal_add_test function. Simplify these test by replacing
the qt_internal_add_resource call with the new BUILTIN_TESTDATA
option.
Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
With Ninja Multi-Config generator it's necessary to provide the
'-C' option matching the build configuration when running
ctest. This patchset adds the missing option to the command line
of the '_check' targets.
Pick-to: 6.2 6.3
Change-Id: I42acfba26a09877a6d3a5be9230860f15f95a0cd
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
qt_internal_undefine_global_definition disables an internal global
definition that is defined by the qt_internal_add_global_definition
function for a specific target.
Remove the ability to set the custom "undefine" flag for the
definitions since it's hard to control it using the introduced
function.
Pick-to: 6.2 6.3
Task-number: QTBUG-100334
Change-Id: Ic1637d97aa51bbdd06c5b191c57a941aa208d4dc
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Looks like Integrity doesn't deal well with external test data either.
Pick-to: 6.3
Change-Id: I8beb5ac4d2440d40a7efdceae973e430a2354e9d
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Restore the 'QT_NO_JAVA_STYLE_ITERATORS' and
'QT_NO_NARROWING_CONVERSIONS_IN_CONNECT' definitions for Qt
targets.
Add the function that adds global definitions for Qt targets according
to the provided scope and the target property-based switch to disable
the definition for a specific target.
Pick-to: 6.2 6.3
Task-number: QTBUG-100295
Change-Id: I28697e81f9aabc45c48d79aae1e5caea141e04e1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Qt's WinRT support was removed long time ago.
Change-Id: I60b220e970072c3450e3793862d6f68801d2b5b3
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The result variable was expanded at configure time rather than at
script execution time due to missing escaping.
A tidbit of information, the result variable can contain not only an
exit code, but a string as well.
For example on arm macOS with a crashed test it contains
'SIGTRAP'.
Curiously if the crashing executable is executed directly without
CMake, 'Trace/BPT trap: 5' is shown instead, perhaps because of
the shell.
Amends 3ef6af024b
Pick-to: 6.2 6.3
Change-Id: I50e57922abfc6eccde205c6252eebfda510bad41
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
This might give a few insights into why the process has failed, at least
if distinct return values are used for different errors.
Change-Id: I61fe0ede812c4dda3d0cf0f2c96a479d198d340d
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
When building for and running the '_check' target in Android we need
to make sure that target apk is assembled and ready for deployment.
Pick-to: 6.2
Change-Id: Ifc63aac230f0bb8375d319ec443174c5a436485e
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Tests for QNX are run in qemu over ssh and setting WORKING_DIRECTORY
will cause created test wrapper to try change specified directory
which will fail (expects local directory) when test is invoked over
ssh.
Task-number: QTBUG-87628
Pick-to: 6.2
Change-Id: If991002398811cc9cb9cbac54da5d2cb0d8c3589
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The format was renamed in 27db9e458c.
Pick-to: 6.2
Change-Id: I53975c7467d8768dc9dc9ac2d89c42eefa12e22f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Currently in order to compile a Qt6 test standalone one needs to use
the qt-cmake-standalone-test script which will load the
Qt6BuildInternals/QtStandaloneTestTemplateProject/CMakeLists.txt project
with the current test source directory to create a complete CMake project.
This commit will allow a test to have these lines at top:
cmake_minimum_required(VERSION 3.16)
if (NOT QT_BUILD_STANDALONE_TESTS AND NOT QT_BUILDING_QT)
project(a_qt_test LANGUAGES C CXX ASM)
find_package(Qt6BuildInternals COMPONENTS STANDALONE_TEST)
endif()
and be standalone and work with any IDE capable of loading CMake projects.
Fixes: QTCREATORBUG-25389
Pick-to: 6.1
Change-Id: If3f878b7e560a8bfbb604a8f1aa30b7962742e66
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
QT_TESTCASE_BUILDDIR was implicitly defined by Qt::Test target for
library users in Qt5. By default, this definition will point to
CMAKE_CURRENT_BINARY_DIR. This logic works similarly to qmake logic.
From user's perspective it might be useful to not rely on standard
search paths, but specify own. This can be done by setting the
QT_TESTCASE_BUILDDIR property for the user's target. CMake will
substute the value of the QT_TESTCASE_BUILDDIR property into the
QT_TESTCASE_BUILDDIR definition.
The implicit QT_TESTCASE_SOURCEDIR also seems to be useful. According
to the current logic, it points to CMAKE_CURRENT_SOURCE_DIR.
Fixes: QTBUG-92079
Change-Id: I8a9065f08e859c713b3c8cc08142a9ced0677770
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When configuring a benchmark using the standalone-test script the
'benchmark' target is not available, causing a configure error.
Pick-to: 6.1 6.0
Change-Id: I8e480c9e72b47783c0910428187f0092049e89db
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Tests should follow their default output of txt as when run directly,
and any output configuration should be handled by the CI system.
Change-Id: Id0d72318df5edad1c58ac637e827d6043884e829
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Toni Saario <toni.saario@qt.io>
Normal Android apps require Qt::Gui dependency and since tests don't
need to handle any special cases for an app without Qt::Gui, let's add
it by default.
This also will allow us to remove some workarounds done on CI side to
run tests for Android.
Fixes: QTBUG-90870
Pick-to: 6.0 6.1
Change-Id: I845650c17a1b73e4c4977043f863ec44e50f06c3
Reviewed-by: Ville Voutilainen <ville.voutilainen@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: BogDan Vatra <bogdan@kdab.com>
Since test command variable may contain quotes, there is a warning
in generated test wrappers related to 'message(FATAL_ERROR', that
should display full command in log in case of error. To avoid any
complicated quoting of command to only display it, join command into
single string in wrapper script and pass resulting variable to
'message(FATAL_ERROR'.
Pick-to: 6.0
Change-Id: Ie990fc0b0bf2c19b119c7c4e2aeec092e5200103
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Use the QT_MKSPECS_DIR instead of QT_SOURCE_TREE to find out the
location for the testserver feature's data. QT_SOURCE_TREE points to
qtbase source location, which may not be always available. This allows
to use this feature by projects outside of qtbase.
Task-number: QTBUG-85034
Change-Id: If7fa6ef95777be9acf35d36f8d99776bf9568a79
Reviewed-by: Cristian Adam <cristian.adam@qt.io>
Introduce a new QT_TESTSERVER_COMPOSE_FILE variable for setting a custom
docker compose file path, which can be set by projects outside of qtbase
(e.g. QtCoap). If the variable is not set, fallback to the compose file
located in qtbase.
Task-number: QTBUG-85034
Change-Id: I4c0c9b161cad9787b089ef7e9da2aa00352a9159
Reviewed-by: Cristian Adam <cristian.adam@qt.io>
Wrap benchmark executable target with cmake script to run it on windows
without necessity of environment variables set.
Move common logic related to test and benchmark environment to
'qt_internal_collect_command_environment' function.
Make error message of wrapper script a bit more verbose.
Amends 61d5b01972
Task-number: QTBUG-89076
Pick-to: 6.0
Change-Id: I6ad3a027dc071176070cf3af4cf306f20d652039
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Refactor test adding procedure. It's expected that for each new
platform we prepare test executable and arguments first and then
pass them to common test adding section.
Rename '_qt_internal_wrap_test' to 'qt_internal_create_command_script'
Rework paramerters handling of 'qt_internal_create_command_script',
make them named.
Add 'qt_internal_create_test_script' to add tests and wrap them using
'qt_internal_create_command_script'.
Amends f19266bd02
Fixes: QTBUG-88053
Pick-to: 6.0
Change-Id: I812f4a295005bf3a85cdcb5b8c41180f8249dc96
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Use 'separate_arguments' to explicitly split arguments from environment
that form 'COMMAND' for execute_process. Enclose using bracket syntax
arguments, that propagated to wrapper script as pure strings.
Amends f19266bd02
Pick-to: 6.0
Change-Id: I858ddff7efa281f9cecfda656a02e1fd12361758
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
In some case CROSSCOMPILING_EMULATOR property is not INHERITED and is
set to 'NOTFOUND' by get_test_property. Force it to empty string in
this case.
Add missed 'name' argument to '_qt_internal_wrap_test'. 'name'
previously was used from parent scope.
Pick-to: 6.0
Fixes: QTBUG-88053
Change-Id: I5286bcb9b11659f638e178ce52172dfee994fc34
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Add custom targets with '_benchmark' suffixes to make run of
benchmarks using generators possible, e.g.:
$ ninja tst_bench_qudpsocket_benchmark
Extend '-[no]make' option to pass benchmark. Rework
'-[no]make' processing to unify these options processing.
Also looks like it doesn't make sense to enable benchmarks without
having test enabled. So '-DQT_BUILD_BENCHMARKS' enables test as own
dependency automatically.
Task-number: QTBUG-89076
Pick-to: 6.0
Change-Id: Ieee9eadaf6d75a1efec120242d6eb786ace1b071
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Wrap Qt tests to handle TESTRUNNER and TESTARGS environment variables.
Variables are handled according to qmake build procedure. All test
now are wrapped by generated CMake script.
Fixes: QTBUG-88053
Pick-to: 6.0
Change-Id: I339977ce6ce11ddd38a4bf0bd26eb8f2ae463ba3
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Without breaking currently generated CMakeLists
Fixes: QTBUG-88549
Change-Id: Ibda643e1374d9024bf693c12de8ec0ac46e09b7b
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
All usages should be replaced with qt_internal_add_executable instead.
This should fix configuration failure of benchmarks and manual tests.
Amends e0c62a48b8
Task-number: QTBUG-86815
Change-Id: I0791d849998ed9517e32f699d843367949b97cb9
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Offer compatibility wrapper functions until we update all of the Qt
repos to use the new names.
Task-number: QTBUG-86815
Change-Id: I5826a4116f52a8509db32601ef7c200f9bd331de
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
qt_add_test supports now QT_TEST_SERVER_LIST, which will add the test
servers as docker test fixtures.
The docker server will be started before the test is run, and stopped
after the test is run.
Running the tests in parallel is not supported.
Docker tests are currently only supported on Linux hosts.
Task-number: QTBUG-85034
Change-Id: If3cefe05c5dec19c14b05d2fa8b01a0b6d95e259
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
QtBuild.cmake is huge. Split it.
Move module, plugin, tools, executables and test related functions out
of QtBuild.cmake into separate files.
Do the same for many other things too.
An additional requirement is that all the new Helpers files only
define functions and macros.
No global variable definitions are allowed, nor execution of commands
with side effects.
Some notes:
qt_install_qml_files is removed because it's dead code.
Some functions still need to be figured out, because they are
interspersed and depend on various global state assignments.
Task-number: QTBUG-86035
Change-Id: I21d79ff02eef923c202eb1000422888727cb0e2c
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
