qt5base-lts/tests/auto/testlib/selftests

Running the QtTestLib selftests with Catch2
===========================================

Catch2 [1] is a header only test framework that we use to allow
testing QtTestLib without relying on any part of testlib itself.

To run the test suite, execute 'make check' or './tst_selftests'
as normal. This should print:

    ===================================================================
    All tests passed (2453 assertions in 5 test cases)

To run specific tests, first lists the available tests:

    ❯ ./tst_selftests -l
    All available test cases:
      Loggers support both old and new style arguments
      Loggers can output to both file and stdout
      Logging to file and stdout at the same time
      All loggers can be enabled at the same time
      Scenario: Test output of the loggers is as expected
    5 test cases

Then pass the name of the test in quotes as the first argument:

    ❯ ./tst_selftests "Loggers support both old and new style arguments"
    Filters: Loggers support both old and new style arguments
    ==================================================================
    All tests passed (96 assertions in 1 test case)

You can find the tests in the sources as individual TEST_CASE
entries. Note that each of these tests run the tests once per
logger, and in the case of the test log check also all sub tests,
so the amount of actual test assertions is much higher than the
five tests listed above.

To see what the tests is actually doing, pass the -s option.
This will result in very verbose output. Each leaf test is
prefixed with a heading:

    ---------------------------------------------------------------
          Given: The QTestLog::TAP logger
           When: Passing arguments with new style
    ---------------------------------------------------------------

You can choose a specific subtest by passing the -c option:

    ❯ ./tst_selftests "Scenario: Test output of the loggers is as expected" \
        -c "Given: The QTestLog::Plain logger" \
        -c 'And given: The "skip" subtest'

Note that on Windows, when using cmd.exe, you cannot use single
quotes, so the command above becomes:

    ❯ ./tst_selftests "Scenario: Test output of the loggers is as expected" \
        -c "Given: The QTestLog::Plain logger" \
        -c "And given: The ""skip"" subtest"

It's possible to pass only the first -c options, to e.g. run all
tests with the Plain logger, but it's unfortunately not possible
to pass only the last -c option, to run the 'skip' subtest with
all loggers.

If a test fails it will print the expected, actual, and difference.
The test results are also left in a temporary directory for closer
inspection.

Add new tests by modifying selftest.pri and CMakeLists.txt, adding
a new subprogram.

Generating new test expectations is done using the python script
in this directory (generate_expected_output.py). In the future this
will be done with the --rebase option to ./tst_selftest, but this
is not fleshed out yet.

[1] https://github.com/catchorg/Catch2