Add --writeJsonSummary option to gm

Add --writeJsonSummary option to gm

This writes out a JSON file containing expected and actual checksum values for every test.
Next step: allow the same checksum file to be compared against, instead of PNG files.
Committed: https://code.google.com/p/skia/source/detail?r=6843

[epoger, 2012-12-13 21:42:39]

+Edi/Ben in CC, just so they know what I've got cooking.

[epoger, 2012-12-13 21:45:16]

https://codereview.appspot.com/6940053/diff/1/gm/tests/outputs/compared-again...
File
gm/tests/outputs/compared-against-different-pixels/output-expected/json-summary.txt
(right):

https://codereview.appspot.com/6940053/diff/1/gm/tests/outputs/compared-again...
gm/tests/outputs/compared-against-different-pixels/output-expected/json-summary.txt:10:
"checksums" : [ 15161495552186645995 ],
The idea is that there is an ARRAY of allowed checksums for each test (in case
we know that it could be any one of several images for some reason)

[EricB, 2012-12-13 22:00:06]

Awesome!  Just a couple of comments.

https://codereview.appspot.com/6940053/diff/1/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.appspot.com/6940053/diff/1/gm/gmmain.cpp#newcode85
gm/gmmain.cpp:85: const static char kJsonKey_ExpectedResults_Ignore[]    =
"ignore";
Maybe "suppress-error" instead of "ignore"?  To me "ignore" implies that we
shouldn't even bother rendering the image. Or maybe "ignore-failure"?

https://codereview.appspot.com/6940053/diff/1/gm/tests/outputs/compared-again...
File
gm/tests/outputs/compared-against-different-pixels/output-expected/json-summary.txt
(right):

https://codereview.appspot.com/6940053/diff/1/gm/tests/outputs/compared-again...
gm/tests/outputs/compared-against-different-pixels/output-expected/json-summary.txt:10:
"checksums" : [ 15161495552186645995 ],
On 2012/12/13 21:45:16, epoger wrote:
> The idea is that there is an ARRAY of allowed checksums for each test (in case
> we know that it could be any one of several images for some reason)

+1000

https://codereview.appspot.com/6940053/diff/1/gm/tests/run.sh
File gm/tests/run.sh (right):

https://codereview.appspot.com/6940053/diff/1/gm/tests/run.sh#newcode1
gm/tests/run.sh:1: #!/bin/bash
How about a test which exercises the "checksum can be one of several" code path?

[epoger, 2012-12-14 18:26:03]

https://codereview.appspot.com/6940053/diff/1/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.appspot.com/6940053/diff/1/gm/gmmain.cpp#newcode85
gm/gmmain.cpp:85: const static char kJsonKey_ExpectedResults_Ignore[]    =
"ignore";
On 2012/12/13 22:00:07, EricB wrote:
> Maybe "suppress-error" instead of "ignore"?  To me "ignore" implies that we
> shouldn't even bother rendering the image. Or maybe "ignore-failure"?

Good idea; changed to "ignore-failure".

You'll see that I also changed the ActualResults section to be split into 3
parts: Failed, FailureIgnored, and Succeeded.  I figured that most consumers of
these results would only want to see the failures anyway, so why make them
examine every record just to find the failures?

Let me know if you think that change is problematic, or if you think of a more
clever way of dicing up the output.

https://codereview.appspot.com/6940053/diff/1/gm/tests/run.sh
File gm/tests/run.sh (right):

https://codereview.appspot.com/6940053/diff/1/gm/tests/run.sh#newcode1
gm/tests/run.sh:1: #!/bin/bash
On 2012/12/13 22:00:07, EricB wrote:
> How about a test which exercises the "checksum can be one of several" code
path?

Well, I haven't actually added the code yet that will read in a JSON file for
comparisons (instead of reading individual expected image files).  So for now,
we only compare each actual against a single expected.

But when I add that code, absolutely, I'll add a test.

https://codereview.appspot.com/6940053/diff/6001/gm/tests/outputs/compared-ag...
File
gm/tests/outputs/compared-against-empty-dir/output-expected/json-summary.txt
(right):

https://codereview.appspot.com/6940053/diff/6001/gm/tests/outputs/compared-ag...
gm/tests/outputs/compared-against-empty-dir/output-expected/json-summary.txt:13:
"checksums" : null,
This new test shows that a missing expected image is interpreted as: no known
checksum to compare against, and ignore any failure.

[EricB, 2012-12-14 20:02:22]

LGTM!

https://codereview.appspot.com/6940053/diff/6001/gm/tests/outputs/compared-ag...
File
gm/tests/outputs/compared-against-empty-dir/output-expected/json-summary.txt
(right):

https://codereview.appspot.com/6940053/diff/6001/gm/tests/outputs/compared-ag...
gm/tests/outputs/compared-against-empty-dir/output-expected/json-summary.txt:13:
"checksums" : null,
On 2012/12/14 18:26:03, epoger wrote:
> This new test shows that a missing expected image is interpreted as: no known
> checksum to compare against, and ignore any failure.

Nice!  This should make rebaselining super easy.