skia2/dm/DMJsonWriter.h
scroggo 0ee2627026 When running DM, write test failures to json.
Add skiatest::Failure to keep track of data about a test failure.

Reporter::reportFailed and ::onReportFailed now take Failure as a
parameter. This allows the implementation to treat the failure as it
wishes. Provide a helper to format the failure the same as prior to
the change.

Update the macros for calling reportFailed (REPORTER_ASSERT etc) to
create a Failure object.

Convert a direct call to reportFailed to the macro ERRORF.

Write Failures to Json.
Sample output when running dm on the dummy test crrev.com/705723004:
{
   "test_results" : {
      "failures" : [
         {
            "condition" : "0 > 3",
            "file_name" : "../../tests/DummyTest.cpp",
            "line_no" : 10,
            "message" : ""
         },
         {
            "condition" : "false",
            "file_name" : "../../tests/DummyTest.cpp",
            "line_no" : 4,
            "message" : ""
         },
         {
            "condition" : "1 == 3",
            "file_name" : "../../tests/DummyTest.cpp",
            "line_no" : 5,
            "message" : "I can too count!"
         },
         {
            "condition" : "",
            "file_name" : "../../tests/DummyTest.cpp",
            "line_no" : 6,
            "message" : "seven is 7"
         },
         {
            "condition" : "1 == 3",
            "file_name" : "../../tests/DummyTest.cpp",
            "line_no" : 14,
            "message" : "I can too count!"
         }
      ]
   }
}

Report all of the failures from one test.
Previously, if one test had multiple failures, only one was reportered.
e.g:

Failures:
  test Dummy: ../../tests/DummyTest.cpp:6   seven is 7
  test Dummy2: ../../tests/DummyTest.cpp:10 0 > 3
  test Dummy3: ../../tests/DummyTest.cpp:14 I can too count!: 1 == 3
3 failures.

Now, we get all the messages:

Failures:
  test Dummy: ../../tests/DummyTest.cpp:4   false
        ../../tests/DummyTest.cpp:5 I can too count!: 1 == 3
        ../../tests/DummyTest.cpp:6 seven is 7
  test Dummy2: ../../tests/DummyTest.cpp:10 0 > 3
  test Dummy3: ../../tests/DummyTest.cpp:14 I can too count!: 1 == 3
3 failures.

(Note that we still state "3 failures" because 3 DM::Tasks failed.)

BUG=skia:3082
BUG=skia:2454

Review URL: https://codereview.chromium.org/694703005
2014-11-07 06:07:32 -08:00

51 lines
1.2 KiB
C++

/*
* Copyright 2014 Google Inc.
*
* Use of this source code is governed by a BSD-style license that can be
* found in the LICENSE file.
*/
#ifndef DMJsonWriter_DEFINED
#define DMJsonWriter_DEFINED
#include "SkString.h"
#include "Test.h"
namespace DM {
/**
* Class for collecting results from DM and writing to a json file.
* All methods are thread-safe.
*/
class JsonWriter {
public:
/**
* Info describing a single run.
*/
struct BitmapResult {
SkString name; // E.g. "ninepatch-stretch", "desk-gws_skp"
SkString config; // "gpu", "8888"
SkString mode; // "direct", "default-tilegrid", "pipe"
SkString sourceType; // "GM", "SKP"
SkString md5; // In ASCII, so 32 bytes long.
};
/**
* Add a result to the end of the list of results.
*/
static void AddBitmapResult(const BitmapResult&);
/**
* Add a Failure from a Test.
*/
static void AddTestFailure(const skiatest::Failure&);
/**
* Write all collected results to the file FLAGS_writePath[0]/dm.json.
*/
static void DumpJson();
};
} // namespace DM
#endif // DMJsonWriter_DEFINED