skia2/dm/DMWriteTask.cpp

216 lines
6.4 KiB
C++
Raw Normal View History

#include "DMWriteTask.h"
#include "DMUtil.h"
#include "SkColorPriv.h"
#include "SkCommonFlags.h"
#include "SkData.h"
#include "SkImageEncoder.h"
#include "SkMD5.h"
#include "SkMallocPixelRef.h"
#include "SkOSFile.h"
#include "SkStream.h"
#include "SkString.h"
DEFINE_bool(nameByHash, false, "If true, write .../hash.png instead of .../mode/config/name.png");
namespace DM {
Add support for reading a directory of images with --expectations (-r). DM writes out its images in a hierarchy that's a little different than GM, so this can't read GM's output. But it can read its own, written with -w. Example usage: $ out/Release/dm -w /tmp/baseline $ out/Release/dm -r /tmp/baseline -w /tmp/new (and optionally) $ mkdir /tmp/diff; out/Release/skdiff /tmp/baseline /tmp/new /tmp/diff GM's IndividualImageExpectationsSource and Expectations are a little too eager about decoding and hashing the expected images, so I took the opportunity to add DM::Expectations that mostly replaces skiagm::ExpectationsSource and skiagm::Expectations in DM. It mainly exists to move the image decoding and comparison off the main thread, which would otherwise be a major speed bottleneck. I tried to use skiagm code where possible. One notable place where I differed is in this new feature. When -r is a directory of images, DM does no hashing. It considerably faster to read the expected file into an SkBitmap and do a byte-for-byte comparison than to hash the two bitmaps and check those. The example usage above isn't quite working 100% yet. Expectations on some GMs fail, even with no binary change. I haven't pinned down whether this is due to - a bug in DM - flaky GMs - unthreadsafe GMs - flaky image decoding - unthreadsafe image decoding - something else but I intend to. Leon, Derek and I have suspected PNG decoding isn't threadsafe, but are as yet unable to prove it. I also seem to be able to cause malloc to fail on my laptop if I run too many configs at once, though I never seem to be using more than ~1G of RAM. Will track that down too. BUG= R=reed@google.com, bsalomon@google.com Author: mtklein@google.com Review URL: https://codereview.chromium.org/108963002 git-svn-id: http://skia.googlecode.com/svn/trunk@12596 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-12-10 14:53:16 +00:00
// Splits off the last N suffixes of name (splitting on _) and appends them to out.
// Returns the total number of characters consumed.
static int split_suffixes(int N, const char* name, SkTArray<SkString>* out) {
SkTArray<SkString> split;
SkStrSplit(name, "_", &split);
Add support for reading a directory of images with --expectations (-r). DM writes out its images in a hierarchy that's a little different than GM, so this can't read GM's output. But it can read its own, written with -w. Example usage: $ out/Release/dm -w /tmp/baseline $ out/Release/dm -r /tmp/baseline -w /tmp/new (and optionally) $ mkdir /tmp/diff; out/Release/skdiff /tmp/baseline /tmp/new /tmp/diff GM's IndividualImageExpectationsSource and Expectations are a little too eager about decoding and hashing the expected images, so I took the opportunity to add DM::Expectations that mostly replaces skiagm::ExpectationsSource and skiagm::Expectations in DM. It mainly exists to move the image decoding and comparison off the main thread, which would otherwise be a major speed bottleneck. I tried to use skiagm code where possible. One notable place where I differed is in this new feature. When -r is a directory of images, DM does no hashing. It considerably faster to read the expected file into an SkBitmap and do a byte-for-byte comparison than to hash the two bitmaps and check those. The example usage above isn't quite working 100% yet. Expectations on some GMs fail, even with no binary change. I haven't pinned down whether this is due to - a bug in DM - flaky GMs - unthreadsafe GMs - flaky image decoding - unthreadsafe image decoding - something else but I intend to. Leon, Derek and I have suspected PNG decoding isn't threadsafe, but are as yet unable to prove it. I also seem to be able to cause malloc to fail on my laptop if I run too many configs at once, though I never seem to be using more than ~1G of RAM. Will track that down too. BUG= R=reed@google.com, bsalomon@google.com Author: mtklein@google.com Review URL: https://codereview.chromium.org/108963002 git-svn-id: http://skia.googlecode.com/svn/trunk@12596 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-12-10 14:53:16 +00:00
int consumed = 0;
for (int i = 0; i < N; i++) {
// We're splitting off suffixes from the back to front.
Add support for reading a directory of images with --expectations (-r). DM writes out its images in a hierarchy that's a little different than GM, so this can't read GM's output. But it can read its own, written with -w. Example usage: $ out/Release/dm -w /tmp/baseline $ out/Release/dm -r /tmp/baseline -w /tmp/new (and optionally) $ mkdir /tmp/diff; out/Release/skdiff /tmp/baseline /tmp/new /tmp/diff GM's IndividualImageExpectationsSource and Expectations are a little too eager about decoding and hashing the expected images, so I took the opportunity to add DM::Expectations that mostly replaces skiagm::ExpectationsSource and skiagm::Expectations in DM. It mainly exists to move the image decoding and comparison off the main thread, which would otherwise be a major speed bottleneck. I tried to use skiagm code where possible. One notable place where I differed is in this new feature. When -r is a directory of images, DM does no hashing. It considerably faster to read the expected file into an SkBitmap and do a byte-for-byte comparison than to hash the two bitmaps and check those. The example usage above isn't quite working 100% yet. Expectations on some GMs fail, even with no binary change. I haven't pinned down whether this is due to - a bug in DM - flaky GMs - unthreadsafe GMs - flaky image decoding - unthreadsafe image decoding - something else but I intend to. Leon, Derek and I have suspected PNG decoding isn't threadsafe, but are as yet unable to prove it. I also seem to be able to cause malloc to fail on my laptop if I run too many configs at once, though I never seem to be using more than ~1G of RAM. Will track that down too. BUG= R=reed@google.com, bsalomon@google.com Author: mtklein@google.com Review URL: https://codereview.chromium.org/108963002 git-svn-id: http://skia.googlecode.com/svn/trunk@12596 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-12-10 14:53:16 +00:00
out->push_back(split[split.count()-i-1]);
consumed += out->back().size() + 1; // Add one for the _.
}
Add support for reading a directory of images with --expectations (-r). DM writes out its images in a hierarchy that's a little different than GM, so this can't read GM's output. But it can read its own, written with -w. Example usage: $ out/Release/dm -w /tmp/baseline $ out/Release/dm -r /tmp/baseline -w /tmp/new (and optionally) $ mkdir /tmp/diff; out/Release/skdiff /tmp/baseline /tmp/new /tmp/diff GM's IndividualImageExpectationsSource and Expectations are a little too eager about decoding and hashing the expected images, so I took the opportunity to add DM::Expectations that mostly replaces skiagm::ExpectationsSource and skiagm::Expectations in DM. It mainly exists to move the image decoding and comparison off the main thread, which would otherwise be a major speed bottleneck. I tried to use skiagm code where possible. One notable place where I differed is in this new feature. When -r is a directory of images, DM does no hashing. It considerably faster to read the expected file into an SkBitmap and do a byte-for-byte comparison than to hash the two bitmaps and check those. The example usage above isn't quite working 100% yet. Expectations on some GMs fail, even with no binary change. I haven't pinned down whether this is due to - a bug in DM - flaky GMs - unthreadsafe GMs - flaky image decoding - unthreadsafe image decoding - something else but I intend to. Leon, Derek and I have suspected PNG decoding isn't threadsafe, but are as yet unable to prove it. I also seem to be able to cause malloc to fail on my laptop if I run too many configs at once, though I never seem to be using more than ~1G of RAM. Will track that down too. BUG= R=reed@google.com, bsalomon@google.com Author: mtklein@google.com Review URL: https://codereview.chromium.org/108963002 git-svn-id: http://skia.googlecode.com/svn/trunk@12596 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-12-10 14:53:16 +00:00
return consumed;
}
inline static SkString find_base_name(const Task& parent, SkTArray<SkString>* suffixList) {
const int suffixes = parent.depth() + 1;
const SkString& name = parent.name();
const int totalSuffixLength = split_suffixes(suffixes, name.c_str(), suffixList);
return SkString(name.c_str(), name.size() - totalSuffixLength);
}
WriteTask::WriteTask(const Task& parent, const char* sourceType, SkBitmap bitmap)
: CpuTask(parent)
, fBaseName(find_base_name(parent, &fSuffixes))
, fSourceType(sourceType)
, fBitmap(bitmap)
, fData(NULL)
, fExtension(".png") {
}
WriteTask::WriteTask(const Task& parent,
const char* sourceType,
SkStreamAsset *data,
const char* ext)
: CpuTask(parent)
, fBaseName(find_base_name(parent, &fSuffixes))
, fSourceType(sourceType)
, fData(data)
, fExtension(ext) {
SkASSERT(fData.get());
SkASSERT(fData->unique());
}
void WriteTask::makeDirOrFail(SkString dir) {
if (!sk_mkdir(dir.c_str())) {
this->fail();
}
}
static SkString get_md5_string(SkMD5* hasher) {
SkMD5::Digest digest;
hasher->finish(digest);
SkString md5;
for (int i = 0; i < 16; i++) {
md5.appendf("%02x", digest.data[i]);
}
return md5;
}
static SkString get_md5(const void* ptr, size_t len) {
SkMD5 hasher;
hasher.write(ptr, len);
return get_md5_string(&hasher);
}
static bool write_asset(SkStreamAsset* input, SkWStream* output) {
return input->rewind() && output->writeStream(input, input->getLength());
}
static SkString get_md5(SkStreamAsset* stream) {
SkMD5 hasher;
write_asset(stream, &hasher);
return get_md5_string(&hasher);
}
struct JsonData {
SkString name; // E.g. "ninepatch-stretch", "desk-gws_skp"
SkString config; // "gpu", "8888"
SkString mode; // "direct", "default-tilegrid", "pipe"
SkString sourceType; // "GM", "SKP"
SkString md5; // In ASCII, so 32 bytes long.
};
SkTArray<JsonData> gJsonData;
SK_DECLARE_STATIC_MUTEX(gJsonDataLock);
void WriteTask::draw() {
SkString md5;
{
SkAutoLockPixels lock(fBitmap);
md5 = fData ? get_md5(fData)
: get_md5(fBitmap.getPixels(), fBitmap.getSize());
}
SkASSERT(fSuffixes.count() > 0);
SkString config = fSuffixes.back();
SkString mode("direct");
if (fSuffixes.count() > 1) {
mode = fSuffixes.fromBack(1);
}
JsonData entry = { fBaseName, config, mode, fSourceType, md5 };
{
SkAutoMutexAcquire lock(&gJsonDataLock);
gJsonData.push_back(entry);
}
SkString dir(FLAGS_writePath[0]);
#if defined(SK_BUILD_FOR_IOS)
if (dir.equals("@")) {
dir.set(FLAGS_resourcePath[0]);
}
#endif
this->makeDirOrFail(dir);
SkString path;
if (FLAGS_nameByHash) {
// Flat directory of hash-named files.
path = SkOSPath::Join(dir.c_str(), md5.c_str());
path.append(fExtension);
// We're content-addressed, so it's possible two threads race to write
// this file. We let the first one win. This also means we won't
// overwrite identical files from previous runs.
if (sk_exists(path.c_str())) {
return;
}
} else {
// Nested by mode, config, etc.
for (int i = 0; i < fSuffixes.count(); i++) {
dir = SkOSPath::Join(dir.c_str(), fSuffixes[i].c_str());
this->makeDirOrFail(dir);
}
path = SkOSPath::Join(dir.c_str(), fBaseName.c_str());
path.append(fExtension);
// The path is unique, so two threads can't both write to the same file.
// If already present we overwrite here, since the content may have changed.
}
SkFILEWStream file(path.c_str());
if (!file.isValid()) {
return this->fail("Can't open file.");
}
bool ok = fData ? write_asset(fData, &file)
: SkImageEncoder::EncodeStream(&file, fBitmap, SkImageEncoder::kPNG_Type, 100);
if (!ok) {
return this->fail("Can't write to file.");
}
}
SkString WriteTask::name() const {
SkString name("writing ");
for (int i = 0; i < fSuffixes.count(); i++) {
name.appendf("%s/", fSuffixes[i].c_str());
}
name.append(fBaseName.c_str());
return name;
}
bool WriteTask::shouldSkip() const {
return FLAGS_writePath.isEmpty();
}
void WriteTask::DumpJson() {
if (FLAGS_writePath.isEmpty()) {
return;
}
Json::Value root;
for (int i = 1; i < FLAGS_properties.count(); i += 2) {
root[FLAGS_properties[i-1]] = FLAGS_properties[i];
}
for (int i = 1; i < FLAGS_key.count(); i += 2) {
root["key"][FLAGS_key[i-1]] = FLAGS_key[i];
}
{
SkAutoMutexAcquire lock(&gJsonDataLock);
for (int i = 0; i < gJsonData.count(); i++) {
Json::Value result;
result["key"]["name"] = gJsonData[i].name.c_str();
result["key"]["config"] = gJsonData[i].config.c_str();
result["key"]["mode"] = gJsonData[i].mode.c_str();
result["options"]["source_type"] = gJsonData[i].sourceType.c_str();
result["md5"] = gJsonData[i].md5.c_str();
root["results"].append(result);
}
}
SkString path = SkOSPath::Join(FLAGS_writePath[0], "dm.json");
SkFILEWStream stream(path.c_str());
stream.writeText(Json::StyledWriter().write(root).c_str());
stream.flush();
Add support for reading a directory of images with --expectations (-r). DM writes out its images in a hierarchy that's a little different than GM, so this can't read GM's output. But it can read its own, written with -w. Example usage: $ out/Release/dm -w /tmp/baseline $ out/Release/dm -r /tmp/baseline -w /tmp/new (and optionally) $ mkdir /tmp/diff; out/Release/skdiff /tmp/baseline /tmp/new /tmp/diff GM's IndividualImageExpectationsSource and Expectations are a little too eager about decoding and hashing the expected images, so I took the opportunity to add DM::Expectations that mostly replaces skiagm::ExpectationsSource and skiagm::Expectations in DM. It mainly exists to move the image decoding and comparison off the main thread, which would otherwise be a major speed bottleneck. I tried to use skiagm code where possible. One notable place where I differed is in this new feature. When -r is a directory of images, DM does no hashing. It considerably faster to read the expected file into an SkBitmap and do a byte-for-byte comparison than to hash the two bitmaps and check those. The example usage above isn't quite working 100% yet. Expectations on some GMs fail, even with no binary change. I haven't pinned down whether this is due to - a bug in DM - flaky GMs - unthreadsafe GMs - flaky image decoding - unthreadsafe image decoding - something else but I intend to. Leon, Derek and I have suspected PNG decoding isn't threadsafe, but are as yet unable to prove it. I also seem to be able to cause malloc to fail on my laptop if I run too many configs at once, though I never seem to be using more than ~1G of RAM. Will track that down too. BUG= R=reed@google.com, bsalomon@google.com Author: mtklein@google.com Review URL: https://codereview.chromium.org/108963002 git-svn-id: http://skia.googlecode.com/svn/trunk@12596 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-12-10 14:53:16 +00:00
}
} // namespace DM