Commit Graph

10 Commits

Author SHA1 Message Date
Volker Hilsheimer
ad333b9b87 Baseline testing: make style name explicit
Mangling it into the checksum makes it hard to navigate the available
images.

Pick-to: 6.3
Change-Id: I54dcab5681e747ce1c5fe1b141ef6c4441d1f7eb
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
2022-03-14 23:38:09 +01:00
Volker Hilsheimer
c7539876f6 Baseline: Move the paintcommands code into the shared directory
It's used by the lancebench and the lance tool, and it will probably be
useful for writing some high-dpi related unit and baseline test cases,
so move it to the shared folder.

Change-Id: I969bab51c9504be13b4c192b4f29f69cd9102868
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
2022-01-20 22:56:11 +01:00
Volker Hilsheimer
ad4c0ac5fb Baseline test framework: follow rule of zero for PlatformInfo type
The compiler generated special functions are just fine.

Pick-to: 6.3 6.2
Change-Id: I64fba1fac59f55d2a82ab18e32c1a2b854df72f0
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
Reviewed-by: Axel Spoerl <axel.spoerl@qt.io>
2022-01-19 13:58:23 +01:00
Eirik Aavitsland
5cd35711a2 Baseline test framework: fix bug in internal data structure
The 'misc' data field was not copied in the assignment operator.

That field is normally not used, so this bug went undiscovered for a
long time. But in certain cases, the bug would cause an image size
mismatch to be reported as just a normal mismatch.

Fix the source of the problem by following the rule of zero - the
compiler generated special functions are just fine for this value
type.

Done-With: Volker Hilsheimer <volker.hilsheimer@qt.io>
Pick-to: 6.3 6.2
Change-Id: I8fc8d32d1b83b78cd4ef3f4ec9a8f22661b0e025
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-01-19 12:58:22 +00:00
Axel Spoerl
fd93c298d6 Prevent takeStandardSnapshots() from stopping on first mismatch
Force takeStandardSnapshots() to take and log all snapshots before
launching QFAIL. Macros QBASELINE_CHECK_DEFERRED and
QBASELINE_CHECK_DEFERRED_SUM have been added in qbaselinetest.h

Task-number: QTBUG-99790
Pick-to: 6.3
Change-Id: Ia015de808f354e842ac4029c5c84be18c4a4e209
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-01-17 21:24:27 +01:00
Eirik Aavitsland
7719fcea4b Add a couple of utility functions to the baseline test framework
Adds a function for setting the baseline Project name, which selects
the top level server directory and hence config settings.

Also adds a function that lets the client override the server config
setting for the project image keys (ItemPathKeys). This is the list of
properties that will be used to determine which clients are "the
same", i.e. which baseline to compare against. Overriding that is
handy since it allows experimentation and customization of clients
without involving changes in the project config on the baseline
server.

Pick-to: 6.3 6.2
Change-Id: Id3998356494a9a2cb71c009b43593d3dc1b6963a
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Axel Spoerl <axel.spoerl@qt.io>
2022-01-13 14:56:37 +01:00
Eirik Aavitsland
0f051815a0 Remove faulty #include from widgets baseline test
Since there is an actual qwidgetbaselinetest.h header file, one should
not include a .moc file. The build system will take care of it, and
currently warns about this #include.

Pick-to: 6.3
Change-Id: I4fbff9ef75c901fe3db4df54d6f3ff0d9307edce
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-01-04 05:53:30 +01:00
Volker Hilsheimer
969cd432f5 Share common code for baseline-testing widget UIs
Setting up the baseline tests, creating an appearance identifier,
and basic image-grabbing functionality doesn't need to be reinvented
for each test case that wants to use baseline testing of widget UIs.

As a drive-by, remove unneeded Qt 5 meta tags from .pri file.

Change-Id: I1562e1b377946305cac018e0f0f0175c2c07cd31
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
2021-11-26 22:40:54 +01:00
Eirik Aavitsland
8a883dea1c Update baseline testing framework
Merge in various minor changes and fixes that have been done to the
branched copy in qtquick3d, and clean out some outdated code.

Mostly just coding style fixes and cleanups, but also:

- adds -keeprunning command line parameter, intended for tests that by
  default exits early if it seems the platform is too unstable
  (e.g. crashing) for a meaningful testrun.

- Changes behaviour for fuzzy matches, from SKIP to PASS. The (mis)use
  of QSKIP was done to force log output; now the output is just
  printed by qInfo() instead.

Pick-to: 6.2
Change-Id: I46e77a94cc5b1980ac420086c2ae88dc9b84ef12
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2021-11-24 11:47:43 +01:00
Eirik Aavitsland
e8f93e38de Rename and restructure the baseline (lancelot) testing code
In preparation for addition of new baseline tests, establish a new
test category, "baseline". This is similar to the category
"benchmarks" in that it contains tests that use the QTest framework,
but conceptually are not unit tests, in contrast to those under auto/.

Move the existing QPainter baseline test, tst_lancelot, into this new
category, and rename it accordingly.

Baseline tests use the QBaselineTest extension to QTest. Move that
extension too into the tests/baseline directory, allowing the clean
out of the baselineserver directory.

Pick-to: 6.2
Change-Id: I1b527f5867c953b1d22be73798fcf7d1494712ea
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Eirik Aavitsland <eirik.aavitsland@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2021-11-16 14:01:50 +01:00