From 5a4106748c578da9f0dc62e77db9e37d01f186ca Mon Sep 17 00:00:00 2001 From: Edward Welbourne Date: Wed, 16 Mar 2022 10:54:14 +0100 Subject: [PATCH] Document some details of TAP's use of YAMLish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ... along with how QtTest soon shall be using it. Change-Id: I9c723ebdb8ee5be1c4152aa911ae7c846ea61b5c Reviewed-by: Tor Arne Vestbø --- src/testlib/qtaptestlogger.cpp | 68 +++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/src/testlib/qtaptestlogger.cpp b/src/testlib/qtaptestlogger.cpp index 9926836c34..dfe167974f 100644 --- a/src/testlib/qtaptestlogger.cpp +++ b/src/testlib/qtaptestlogger.cpp @@ -56,6 +56,71 @@ QT_BEGIN_NAMESPACE The \l{Test Anything Protocol} (TAP) is a simple plain-text interface between testing code and systems for reporting and analyzing test results. + Since QtTest doesn't build the table for a data-driven test until the test + is about to be run, we don't typically know how many tests we'll run until + we've run them, so we put The Plan at the end, rather than the beginning. + We summarise the results in comments following The Plan. + + \section1 YAMLish + + The TAP protocol supports inclusion, after a Test Line, of a "diagnostic + block" in YAML, containing supporting information. We use this to package + other information from the test, in combination with comments to record + information we're unable to deliver at the same time as a test line. By + convention, TAP producers limit themselves to a restricted subset of YAML, + known as YAMLish, for maximal compatibility with TAP consumers. + + YAML (see \c yaml.org for details) supports three data-types: mapping (hash + or dictionary), sequence (list, array or set) and scalar (string or number), + much as JSON does. It uses indentation to indicate levels of nesting of + mappings and sequences within one another. A line starting with a dash (at + the same level of indent as its context) introduces a list entry. A line + starting with a key (more indented than its context, aside from any earlier + keys at its level) followed by a colon introduces a mapping entry; the key + may be either a simple word or a quoted string (within which numeric escapes + may be used to indicate unicode characters). The value associated with a + given key, or the entry in a list, can appar after the key-colon or dasy + either on the same line or on a succession of subsequent lines at higher + indent. Thus + + \code + - first list item + - + second: list item is a mapping + with: + - two keys + - the second of which is a list + - back in the outer list, a third item + \endcode + + In YAMLish, the top-level structure should be a hash. The keys supported for + this hash, with the meanings for their values, are: + + \list + \li \c message: free text supplying supporting details + \li \c severity: how bad is it ? + \li \c source: source describing the test, as an URL (compare file, line) + \li \c at: identifies the function (with file and line) performing the test + \li \c datetime: when the test was run (ISO 8601 or HTTP format) + \li \c file: source of the test as a local file-name, when appropriate + \li \c line: line number within the source + \li \c name: test name + \li \c extensions: sub-hash in which to store random other stuff + \li \c actual: what happened (a.k.a. found; contrast expected) + \li \c expected: what was expected (a.k.a. wanted; contrast actual) + \li \c display: description of the result, suitable for display + \li \c dump: a sub-hash of variable values when the result arose + \li \c error: describes the error + \li \c backtrace: describes the call-stack of the error + \endlist + + In practice, only \c at, \c expected and \c actual appear to be generally + supported. + + We can also have several messages produced within a single test, so the + simple \c message / \c severity pair of top-level keys does not serve us + well. We therefore use \c extensions with a sub-tag \c messages in which to + package a list of messages. \sa QAbstractTestLogger */ @@ -171,7 +236,8 @@ void QTapTestLogger::addIncident(IncidentTypes type, const char *description, if (!ok) { // All failures need a diagnostics section to not confuse consumers - // The indent needs to be two spaces for maximum compatibility + // The indent needs to be two spaces for maximum compatibility. + // This matches the width of the "- " prefix on a list item's first line. #define YAML_INDENT " " outputString(YAML_INDENT "---\n");