gm: Add ability to compare against checksums (as opposed to image files)

gm: Add ability to compare against checksums (as opposed to image files)
Committed: https://code.google.com/p/skia/source/detail?r=7724

[epoger, 2013-02-08 21:01:33]

I'll happily take a review from either one of you guys, whoever wants it...

https://codereview.appspot.com/7306071/diff/1/gm/gm_expectations.h
File gm/gm_expectations.h (right):

https://codereview.appspot.com/7306071/diff/1/gm/gm_expectations.h#newcode32
gm/gm_expectations.h:32: const static char kJsonKey_ActualResults[]   =
"actual-results";
moved into here from gmmain.cpp

https://codereview.appspot.com/7306071/diff/1/gm/gm_expectations.h#newcode264
gm/gm_expectations.h:264: SkAutoDataUnref
dataRef(inFile.readIntoSkData(inFile.getLength()));
I am adding this SkStream::readIntoSkData() method in
https://codereview.appspot.com/7300071 .  Depending on the resolution of that
CL, I may need to change it here.

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
File
gm/tests/outputs/compared-against-different-pixels-json/output-expected/command_line
(right):

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
gm/tests/outputs/compared-against-different-pixels-json/output-expected/command_line:1:
out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/different-pixels.json --writeJsonSummary
gm/tests/outputs/compared-against-different-pixels-json/output-actual/json-summary.txt
And, as expected, these results are very similar to the analogous image
operations:

$ diff -r --exclude=.*
gm/tests/outputs/compared-against-different-pixels-images/output-expected
gm/tests/outputs/compared-against-different-pixels-json/output-expected
diff -r --exclude='.*'
gm/tests/outputs/compared-against-different-pixels-images/output-expected/command_line
gm/tests/outputs/compared-against-different-pixels-json/output-expected/command_line
1c1
< out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/images/different-pixels --writeJsonSummary
gm/tests/outputs/compared-against-different-pixels-images/output-actual/json-summary.txt
---
> out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/different-pixels.json --writeJsonSummary
gm/tests/outputs/compared-against-different-pixels-json/output-actual/json-summary.txt
diff -r --exclude='.*'
gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout
gm/tests/outputs/compared-against-different-pixels-json/output-expected/stdout
1c1
< reading from gm/tests/inputs/images/different-pixels
---
> reading expectations from JSON summary file
gm/tests/inputs/json/different-pixels.json

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
File
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/command_line
(right):

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/command_line:1:
out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/identical-bytes.json --writeJsonSummary
gm/tests/outputs/compared-against-identical-bytes-json/output-actual/json-summary.txt
As expected...

$ diff -r --exclude=.*
gm/tests/outputs/compared-against-identical-bytes-images/output-expected
gm/tests/outputs/compared-against-identical-bytes-json/output-expected
diff -r --exclude='.*'
gm/tests/outputs/compared-against-identical-bytes-images/output-expected/command_line
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/command_line
1c1
< out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/images/identical-bytes --writeJsonSummary
gm/tests/outputs/compared-against-identical-bytes-images/output-actual/json-summary.txt
---
> out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/identical-bytes.json --writeJsonSummary
gm/tests/outputs/compared-against-identical-bytes-json/output-actual/json-summary.txt
diff -r --exclude='.*'
gm/tests/outputs/compared-against-identical-bytes-images/output-expected/stdout
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/stdout
1c1
< reading from gm/tests/inputs/images/identical-bytes
---
> reading expectations from JSON summary file
gm/tests/inputs/json/identical-bytes.json

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
File
gm/tests/outputs/compared-against-identical-pixels-json/output-expected/command_line
(right):

https://codereview.appspot.com/7306071/diff/1/gm/tests/outputs/compared-again...
gm/tests/outputs/compared-against-identical-pixels-json/output-expected/command_line:1:
out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/identical-pixels.json --writeJsonSummary
gm/tests/outputs/compared-against-identical-pixels-json/output-actual/json-summary.txt
As expected...

$ diff -r --exclude=.*
gm/tests/outputs/compared-against-identical-pixels-images/output-expected
gm/tests/outputs/compared-against-identical-pixels-json/output-expected
diff -r --exclude='.*'
gm/tests/outputs/compared-against-identical-pixels-images/output-expected/command_line
gm/tests/outputs/compared-against-identical-pixels-json/output-expected/command_line
1c1
< out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/images/identical-pixels --writeJsonSummary
gm/tests/outputs/compared-against-identical-pixels-images/output-actual/json-summary.txt
---
> out/Debug/gm --hierarchy --match dashing2 --config 8888 --config 565 -r
gm/tests/inputs/json/identical-pixels.json --writeJsonSummary
gm/tests/outputs/compared-against-identical-pixels-json/output-actual/json-summary.txt
diff -r --exclude='.*'
gm/tests/outputs/compared-against-identical-pixels-images/output-expected/stdout
gm/tests/outputs/compared-against-identical-pixels-json/output-expected/stdout
1c1
< reading from gm/tests/inputs/images/identical-pixels
---
> reading expectations from JSON summary file
gm/tests/inputs/json/identical-pixels.json

[EricB, 2013-02-08 22:23:19]

This LGTM with a comment/suggestion.

https://codereview.appspot.com/7306071/diff/1/gm/gm_expectations.h
File gm/gm_expectations.h (right):

https://codereview.appspot.com/7306071/diff/1/gm/gm_expectations.h#newcode115
gm/gm_expectations.h:115: SkASSERT(checksumElement.isIntegral());
I realize that most (probably all?) of the JSON will not be generated by hand,
but I wonder if we could have more informative error messages here.

[epoger, 2013-02-12 18:23:52]

PTAL at patchset 2...

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h
File gm/gm_expectations.h (right):

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h#newcode106
gm/gm_expectations.h:106: fprintf(stderr, "found non-boolean json value for key
'%s' in element '%s'\n",
much more detailed error messages

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h#newcode124
gm/gm_expectations.h:124: for (Json::ArrayIndex i=0; i<allowedChecksums.size();
i++) {
making this ArrayIndex instead of int fixes a warning I had missed before

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h#newcode269
gm/gm_expectations.h:269: /**
Copied this code in here so we are not blocked by
https://codereview.appspot.com/7300071

[EricB, 2013-02-12 19:58:17]

LGTM!

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h
File gm/gm_expectations.h (right):

https://codereview.appspot.com/7306071/diff/5001/gm/gm_expectations.h#newcode106
gm/gm_expectations.h:106: fprintf(stderr, "found non-boolean json value for key
'%s' in element '%s'\n",
On 2013/02/12 18:23:52, epoger wrote:
> much more detailed error messages

Thanks!

[rmistry, 2013-02-13 20:18:05]

https://codereview.appspot.com/7306071/diff/2003/gm/tests/outputs/compared-ag...
File
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/json-summary.txt
(right):

https://codereview.appspot.com/7306071/diff/2003/gm/tests/outputs/compared-ag...
gm/tests/outputs/compared-against-identical-bytes-json/output-expected/json-summary.txt:1:
{
What will the summary JSON look like when we do GM comparisons of 9k images, it
will list every single file we compare?
We could save some space by truncating this output in terms of whitespace, I am
not sure if the savings will be significant or not, we can always explore this
option when we get there.

https://codereview.appspot.com/7306071/diff/2003/gm/tests/run.sh
File gm/tests/run.sh (right):

https://codereview.appspot.com/7306071/diff/2003/gm/tests/run.sh#newcode47
gm/tests/run.sh:47: # - writing json summary into
$2/$OUTPUT_ACTUAL_SUBDIR/json-summary.txt
A better name for this is probably summary.json. Not a big deal here because its
for tests but in production we should end with .json