Commit Graph

99 Commits

Author SHA1 Message Date
epoger@google.com
a783f2bee8 rebaseline.py: if expectations dir contains JSON format results, update those instead of image files
Part of Step 3 in https://goto.google.com/ChecksumTransitionDetail

R=senorblanco@chromium.org

Review URL: https://codereview.chromium.org/18348018

git-svn-id: http://skia.googlecode.com/svn/trunk@9910 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-07-08 17:51:58 +00:00
epoger@google.com
34b73a2da1 rebaseline.py self-tests: prepare to add tests of JSON-format rebaselining
For now, rebaseline.py still does not support JSON-format rebaselining, so
the one self-test reports failure.  But this change will allow us to test
the JSON-format rebaselining once rebaseline.py supports it.

R=scroggo@google.com

Review URL: https://codereview.chromium.org/18348015

git-svn-id: http://skia.googlecode.com/svn/trunk@9896 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-07-03 17:43:11 +00:00
epoger@google.com
53953b457b Create jsondiff tool to capture diffs between two JSON expectations files.
This will be used to view diffs before committing rebaselines, or to review
exactly what changed between two baseline revisions in the past.

Part of Step 3 in https://goto.google.com/ChecksumTransitionDetail

R=scroggo@google.com

Review URL: https://codereview.chromium.org/18103005

git-svn-id: http://skia.googlecode.com/svn/trunk@9878 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-07-02 20:22:27 +00:00
epoger@google.com
8575b4df41 Fix rebaseline.py self-test to account for r9859
R=robertphillips@google.com

Review URL: https://codereview.chromium.org/18559002

git-svn-id: http://skia.googlecode.com/svn/trunk@9872 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-07-02 18:25:06 +00:00
scroggo@google.com
2b9424b342 Treat empty directory properly in skimage.
Instead of treating an empty directory as a file that failed to
decode, treat it as an empty directory.

Add self tests to check for this.

Builds upon from https://codereview.chromium.org/16866010,
which was reverted.

R=epoger@google.com

Review URL: https://codereview.chromium.org/17101029

git-svn-id: http://skia.googlecode.com/svn/trunk@9730 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-21 19:12:47 +00:00
epoger@google.com
99a8ec9705 rebaseline.py: split image-based rebaselining, which will go away soon, into its own script
(part of step 3 in https://goto.google.com/ChecksumTransitionDetail )

also adds new --expectations-root option

R=scroggo@google.com, senorblanco@chromium.org

Review URL: https://codereview.chromium.org/17379004

git-svn-id: http://skia.googlecode.com/svn/trunk@9689 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-19 18:56:59 +00:00
epoger@google.com
c7e1873c85 skimage: revert r9642
git-svn-id: http://skia.googlecode.com/svn/trunk@9647 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-18 02:53:16 +00:00
epoger@google.com
e0433f1ad3 skimage: if --readPath points at an empty dir, create an empty expectations file
TBR=scroggo

Review URL: https://codereview.chromium.org/17294006

git-svn-id: http://skia.googlecode.com/svn/trunk@9642 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-17 19:34:58 +00:00
epoger@google.com
74feb1593a skimage self-tests: confirm correctness of bitmap hash values
R=scroggo@google.com

Review URL: https://codereview.chromium.org/16866010

git-svn-id: http://skia.googlecode.com/svn/trunk@9591 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-13 19:12:05 +00:00
epoger@google.com
e78d20798d rebaseline.py: try to download images from Google Storage before skia-autogen
This will allow us to complete Step 1 of https://goto.google.com/ChecksumTransitionDetail

R=scroggo@google.com

Review URL: https://codereview.chromium.org/16311011

git-svn-id: http://skia.googlecode.com/svn/trunk@9534 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-12 17:44:14 +00:00
epoger@google.com
dad5310ab6 rebaseline.py: add --add-new option, and only add new expectations if it is set
R=senorblanco@chromium.org

Review URL: https://codereview.chromium.org/16363006

git-svn-id: http://skia.googlecode.com/svn/trunk@9527 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-12 14:25:30 +00:00
epoger@google.com
cc2e1cf13c rebaseline.py: improve readability of dry-run output
Split off of already-reviewed https://codereview.chromium.org/15789010/ ('rebaseline.py: --tests and --configs are now FILTERS within json results')

Review URL: https://codereview.chromium.org/16509006

git-svn-id: http://skia.googlecode.com/svn/trunk@9507 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-11 16:24:37 +00:00
epoger@google.com
82f317815e rebaseline.py: in option names, replace underscores with hyphens
Split off of already-reviewed https://codereview.chromium.org/15789010/ ('rebaseline.py: --tests and --configs are now FILTERS within json results')

Review URL: https://codereview.chromium.org/15904018

git-svn-id: http://skia.googlecode.com/svn/trunk@9504 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-11 15:45:46 +00:00
scroggo@google.com
cf5eb6ab25 Preparations for running skimage tool on bots.
skimage_main:
Specify the full name of the expectations file to write on the command line,
rather than the folder to write 'results.json'. This is consistent with
writeJsonSummaryPath in GM, as well as the format for reading the file.
Further, it allows more flexibility to name the file.
Do not force opaque when writing bitmaps. skimage does its comparison on
the raw bitmaps. The images are written for visual comparison, and mostly
transparent bitmaps (like resources/plane.png) aren't meaningful when
forced opaque.
Provide an option to to write only mismatches to a directory. Handled
similar to the writePath.
Report a failure if there is no expectations file. This allows writing
to the mismatchPath so we can create an initial baseline.
Pass SkBitmap by const&.

skimage_self_test:
Include the name of the json file in the parameter passed to
createExpectationsPath. The name has been changed to be more specific.

.gitignore:
Ignore the new name of the file created by skimage_self_test.

run.sh:
Run skimage_self_tests.py

BUG=http://code.google.com/p/skia/issues/detail?id=1241
R=epoger@google.com

Review URL: https://codereview.chromium.org/15813016

git-svn-id: http://skia.googlecode.com/svn/trunk@9474 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-07 12:43:15 +00:00
scroggo@google.com
e762dad67c Run skimage as part of run.sh
Test skimage to ensure that it does not break.

Split off from https://codereview.chromium.org/15813016/

BUG=http://code.google.com/p/skia/issues/detail?id=1241
R=epoger@google.com

Review URL: https://codereview.chromium.org/15774017

git-svn-id: http://skia.googlecode.com/svn/trunk@9473 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-07 12:40:35 +00:00
epoger@google.com
19e80c16a5 rebaseline.py self-tests: split test inputs and outputs into completely separate directories
Review URL: https://codereview.chromium.org/16557002

git-svn-id: http://skia.googlecode.com/svn/trunk@9459 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-06 15:11:11 +00:00
epoger@google.com
bf65378266 add more rebaseline test resources (no behavioral changes)
(SkipBuildbotRuns)

Review URL: https://codereview.chromium.org/15774019

git-svn-id: http://skia.googlecode.com/svn/trunk@9446 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-05 16:26:23 +00:00
epoger@google.com
99ba65a181 rebaseline.py: if --tests is not specified, get test list from actual-results.json
R=scroggo@google.com

Review URL: https://codereview.chromium.org/16093025

git-svn-id: http://skia.googlecode.com/svn/trunk@9443 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-05 15:43:37 +00:00
epoger@google.com
db29a310f7 rebaseline.py : add self-tests
R=senorblanco@chromium.org

Review URL: https://codereview.chromium.org/15660014

git-svn-id: http://skia.googlecode.com/svn/trunk@9426 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-06-04 14:58:47 +00:00
scroggo@google.com
4fed643cb9 Create a self test for skimage.
Runs skimage twice: once to create an expectations file, and a
second time comparing against the file. Uses the files in
resources as test files.

R=epoger@google.com

Review URL: https://codereview.chromium.org/14969007

git-svn-id: http://skia.googlecode.com/svn/trunk@9123 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-05-14 17:50:02 +00:00
epoger@google.com
bd9669ad27 bench_graph_svg: add r7618 results (NOT per-tile) to self-test source data
Review URL: https://codereview.appspot.com/7305098

git-svn-id: http://skia.googlecode.com/svn/trunk@7760 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-17 08:33:27 +00:00
epoger@google.com
aa6470af8e bench_graph_svg self-test: add non-tile bench data
TBR=benchen
Review URL: https://codereview.appspot.com/7317048

git-svn-id: http://skia.googlecode.com/svn/trunk@7750 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-15 16:42:21 +00:00
epoger@google.com
673816c44f tools/tests/run.sh: download test bench data from Google Storage rather than SVN
Review URL: https://codereview.appspot.com/7343045

git-svn-id: http://skia.googlecode.com/svn/trunk@7749 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-15 15:50:23 +00:00
epoger@google.com
2459a394b8 bench_graph_svg: generate output in consistently sorted order, to make diffs more useful
TBR=benchen
Review URL: https://codereview.appspot.com/7330044

git-svn-id: http://skia.googlecode.com/svn/trunk@7744 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-14 18:58:05 +00:00
epoger@google.com
9316d7a0cc disable new bench_graph_svg test in tools/tests/run.sh that fails on housekeeper
TBR=bensong
Review URL: https://codereview.appspot.com/7303092

git-svn-id: http://skia.googlecode.com/svn/trunk@7739 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-14 15:35:23 +00:00
epoger@google.com
f4eeeabcdc tools/tests/run.sh: add self-test for bench_graph_svg.py
Review URL: https://codereview.appspot.com/7319044

git-svn-id: http://skia.googlecode.com/svn/trunk@7735 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-02-14 15:10:24 +00:00
borenet@google.com
b00d670114 Enable per-tile logging in bench_pictures
- Modify bench_pictures buildstep to translate "key: True" --> "--key" rather
  than requiring "key: value" --> "--key value"
- Add whacky TileArgs helper which includes "timeIndividualTiles: True"
Review URL: https://codereview.appspot.com/7092046

git-svn-id: http://skia.googlecode.com/svn/trunk@7141 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-01-11 20:45:24 +00:00
rmistry@google.com
ee5a5eee12 * Extended skdiff to report alpha channel differences.
* Created a tools/tests/rebaseline.sh to copy output-actual into output-expected.
* Sample results are available here: http://www.corp.google.com/~rmistry/skia/gm-playback-windows/images-skdiff/
Review URL: https://codereview.appspot.com/7038048

git-svn-id: http://skia.googlecode.com/svn/trunk@7003 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-01-03 19:23:22 +00:00
epoger@google.com
3aa3358c03 Add more warnings about https://code.google.com/p/skia/issues/detail?id=677 ('make tools/tests/run.sh work cross-platform')
git-svn-id: http://skia.googlecode.com/svn/trunk@6965 2bbb7eff-a529-9590-31e7-b0007b416f81
2013-01-02 15:53:25 +00:00
borenet@google.com
3b98bfd402 Address comments for r6822: https://codereview.appspot.com/6946052/
Review URL: https://codereview.appspot.com/6943059

git-svn-id: http://skia.googlecode.com/svn/trunk@6847 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-12-17 17:21:04 +00:00
borenet@google.com
ddf36e73ac Fix bench_pictures.cfg and add sanity check
Review URL: https://codereview.appspot.com/6946052

git-svn-id: http://skia.googlecode.com/svn/trunk@6822 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-12-14 14:56:37 +00:00
bungeman@google.com
e3c8ddfd03 Update skdiff.
https://codereview.appspot.com/6850115/


git-svn-id: http://skia.googlecode.com/svn/trunk@6681 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-12-05 20:13:12 +00:00
epoger@google.com
a413a53cb2 Add a self-test for gm, similar to our self-test for skdiff.
Review URL: https://codereview.appspot.com/6827068

git-svn-id: http://skia.googlecode.com/svn/trunk@6380 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-11-12 18:04:51 +00:00
jkleinert@google.com
ed01f12d13 Adding checkboxes to skdiff output so a user can select a subset of images to rebaseline.
BUG=745
Review URL: https://codereview.appspot.com/6497054

git-svn-id: http://skia.googlecode.com/svn/trunk@5360 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-08-30 22:05:53 +00:00
epoger@google.com
71329d809a skdiff: recurse over subdirectories, unless --norecurse option is given
in anticipation of https://code.google.com/p/skia/issues/detail?id=743 ('move gm baselines outside of trunk, and modify naming convention')
Review URL: https://codereview.appspot.com/6465053

git-svn-id: http://skia.googlecode.com/svn/trunk@5121 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-08-16 13:42:13 +00:00
epoger@google.com
3af4ff46a7 skdiff output: indicate which result types would cause failure
Review URL: https://codereview.appspot.com/6419053

git-svn-id: http://skia.googlecode.com/svn/trunk@4676 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-07-19 17:35:04 +00:00
epoger@google.com
dfbf24e5e7 skdiff: replace --failonmismatches with repeatable --failonresult option
Review URL: https://codereview.appspot.com/6392064

git-svn-id: http://skia.googlecode.com/svn/trunk@4611 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-07-13 21:22:02 +00:00
epoger@google.com
46a45963fa skdiff: add --failonmismatches and --listfilename options, plus cleanup
These changes are needed in order to switch the buildbots from using "gm -r" to "skdiff" to compare gm results, and should be generally good for humans too.
Review URL: https://codereview.appspot.com/6392054

git-svn-id: http://skia.googlecode.com/svn/trunk@4579 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-07-12 18:16:02 +00:00
epoger@google.com
28060e7c94 Improve sorting of skdiff output, and make it consistent across platforms
BUG=https://code.google.com/p/skia/issues/detail?id=677
Review URL: https://codereview.appspot.com/6351045

git-svn-id: http://skia.googlecode.com/svn/trunk@4388 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-06-28 16:47:34 +00:00
epoger@google.com
be6188d647 skdiff: return nonzero exit value if there are any mismatches
This will allow us to replace "gm -r" in our buildbots with "skdiff".  This will speed things up, and work around http://code.google.com/p/skia/issues/detail?id=473 ('PDF gradtext gm image results are nondeterministic')
Review URL: https://codereview.appspot.com/6242071

git-svn-id: http://skia.googlecode.com/svn/trunk@4097 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-31 15:13:45 +00:00
epoger@google.com
8e4e51d69b test-only: add test coverage of skdiff over input with identical bits or pixels
Review URL: https://codereview.appspot.com/6260055

git-svn-id: http://skia.googlecode.com/svn/trunk@4084 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-30 19:12:45 +00:00
epoger@google.com
46256ea736 Add to skdiff: test images for bitwise equality before testing for pixel-wise equality
- we now check for bitwise equality of files before bothering with pixel 
  comparisons (this will help with formats for which we don't have good
  decoders, like PDF)
- unparsable files are now reported as differing, unless they are bitwise equal
  (before, they were always reported as matching!)
- "largest area mismatch" is now based on same-size image pairs only;
  previously, if any image pairs had mismatching size, it was 100%
- removed repetitive "image size mismatch, so no diff to display" messages
- changed format of leftmost table cells to be more readable

BUG=http://code.google.com/p/skia/issues/detail?id=473
Review URL: https://codereview.appspot.com/6208089

git-svn-id: http://skia.googlecode.com/svn/trunk@4027 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-22 13:45:35 +00:00
epoger@google.com
90cb9b325c skdiff test-only change: add test of --nodiffs mode, make tests validate return value
Review URL: https://codereview.appspot.com/6206085

git-svn-id: http://skia.googlecode.com/svn/trunk@3999 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-18 18:25:26 +00:00
epoger@google.com
a44435296a test-only change: update run.sh and rearrange dirs to ease addition of more tests
Review URL: https://codereview.appspot.com/6220044

git-svn-id: http://skia.googlecode.com/svn/trunk@3995 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-17 17:12:38 +00:00
epoger@google.com
b2aacb6dc2 test-only changes: add more image files to test more skdiff behavior
Tests output order for very vs. slightly different pixels, very vs. slightly different image sizes, etc.

TBR=tomhudson@google.com
Review URL: https://codereview.appspot.com/6213049

git-svn-id: http://skia.googlecode.com/svn/trunk@3977 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-16 18:06:54 +00:00
epoger@google.com
6600852667 skdiff: only write out diff images that are included in index.html
Not only are those files unnecessary, in some different-image-size cases, their contents are nondeterministic!
Review URL: https://codereview.appspot.com/6208073

git-svn-id: http://skia.googlecode.com/svn/trunk@3975 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-16 17:40:57 +00:00
epoger@google.com
a2b793c4d1 Add self-test for skdiff, and remove date/time from skdiff output so it passes.
BUG=http://code.google.com/p/skia/issues/detail?id=612
Review URL: https://codereview.appspot.com/6198074

git-svn-id: http://skia.googlecode.com/svn/trunk@3938 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-15 14:58:53 +00:00
epoger@google.com
15b5167a2d Rename tests/images dir as tests/skdiff
git-svn-id: http://skia.googlecode.com/svn/trunk@3930 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-14 20:19:42 +00:00
epoger@google.com
620c3b4a9f Images for testing skdiff tool
Review URL: https://codereview.appspot.com/6208056

git-svn-id: http://skia.googlecode.com/svn/trunk@3922 2bbb7eff-a529-9590-31e7-b0007b416f81
2012-05-14 15:38:57 +00:00