Doc: Contributor's: Update Regression tests

Doc: Contributor's: Update Regression tests

[Graham Percival (old account), 2010-06-08 12:50:23]

Looks mostly good.

http://codereview.appspot.com/1545043/diff/1/2
File Documentation/contributor/programming-work.itexi (right):

http://codereview.appspot.com/1545043/diff/1/2#newcode1305
Documentation/contributor/programming-work.itexi:1305: 
Yes, -jX CPU_COUNT=X would be useful, but I'd rather not reproduce that info
everywhere in the manual.  ITTWBB (... ok, that doesn't work.  I think it would
be better) if had had a @node in CG 3 called something like "Optimizing
compilation" or "Compiling with more cores" or something like that.

That said, since you've already written this stuff, and since programming-work
is very much a work in progress, we might as well leave it here.  I might try
cleaning up CG 3 in a month or two, so that cleanup can wait until then.

http://codereview.appspot.com/1545043/diff/1/3
File Documentation/contributor/regressions.itexi (right):

http://codereview.appspot.com/1545043/diff/1/3#newcode32
Documentation/contributor/regressions.itexi:32: 
I'd either leave the "output of the regtests is also automatically...", or else
remove the `automaticaly' and have "output of the regtests can be checked...".

Basically, the regtest comparison is generated by automatically comparing two
sets of lilypond output, but the comparison should be / could be checked
manually.  (hmm, maybe we should just add a third sentence, explain the whole
thing, and not try to condense it into a single sentence)

http://codereview.appspot.com/1545043/diff/1/3#newcode40
Documentation/contributor/regressions.itexi:40: 
Why @ignore this stuff?  It looks accurate, it looks good for an overview of the
situation.  If you don't want to explain it in the intro (I'm willing to listen
to reasons), then please remove it entirely.

http://codereview.appspot.com/1545043/diff/1/3#newcode55
Documentation/contributor/regressions.itexi:55: 
Link to latest (complete, non-comparison'd) regtests?

http://codereview.appspot.com/1545043/diff/1/3#newcode68
Documentation/contributor/regressions.itexi:68: the current release and the
prior release.  Each test that has
Somewhere... I think in the "releases" chapter... I noted that the comparison
only looks inside the bounding boxes within lilypond, so errors due to
ghostscript aren't found, nor are errors in lyrics+text.

I don't know exactly what I'm talking about, but it's probably adding a @warning
that not **all** of the changes will be shown.

http://codereview.appspot.com/1545043/diff/1/3#newcode157
Documentation/contributor/regressions.itexi:157: The special regression test
@file{test-output-distance.ly} will always
Please put this paragraph inside a @warning.

http://codereview.appspot.com/1545043/diff/1/4
File Documentation/included/compile.itexi (right):

http://codereview.appspot.com/1545043/diff/1/4#newcode858
Documentation/included/compile.itexi:858: 
Eh?  "make test" has nothing to do with installing lilypond, surely?

[Carl, 2010-06-08 19:55:36]

Thanks for the review.

I've posted a new version.

Thanks,

Carl

http://codereview.appspot.com/1545043/diff/1/2
File Documentation/contributor/programming-work.itexi (right):

http://codereview.appspot.com/1545043/diff/1/2#newcode1305
Documentation/contributor/programming-work.itexi:1305: 


I agree.  This is part of the stuff that was just "dumped somewhere", as you
asked in your earlier email.  I think it belongs in a separate chapter on the
build system, as I mentioned before, rather than buried in chapter 3.

http://codereview.appspot.com/1545043/diff/1/3
File Documentation/contributor/regressions.itexi (right):

http://codereview.appspot.com/1545043/diff/1/3#newcode32
Documentation/contributor/regressions.itexi:32: 
On 2010/06/08 12:50:23, Graham Percival wrote:
> I'd either leave the "output of the regtests is also automatically...", or
else
> remove the `automaticaly' and have "output of the regtests can be checked...".
> 
> Basically, the regtest comparison is generated by automatically comparing two
> sets of lilypond output, but the comparison should be / could be checked
> manually.  (hmm, maybe we should just add a third sentence, explain the whole
> thing, and not try to condense it into a single sentence)

I expanded to indicate three uses:  1) Successful completion; 2) Description
matches graphical output; 3) Comparison between different versions.  I also
indicated that the comparison is automatic, but the checking is manual.

http://codereview.appspot.com/1545043/diff/1/3#newcode40
Documentation/contributor/regressions.itexi:40: 
On 2010/06/08 12:50:23, Graham Percival wrote:
> Why @ignore this stuff?  It looks accurate, it looks good for an overview of
the
> situation.  If you don't want to explain it in the intro (I'm willing to
listen
> to reasons), then please remove it entirely.


I don't agree with the statement "The latter is recommended", and I hadn't yet
decided if I was going to pitch the whole thing, although I was leaning that
way.

I've adjusted this and removed the ignore.

http://codereview.appspot.com/1545043/diff/1/3#newcode55
Documentation/contributor/regressions.itexi:55: 
On 2010/06/08 12:50:23, Graham Percival wrote:
> Link to latest (complete, non-comparison'd) regtests?

Done.

http://codereview.appspot.com/1545043/diff/1/3#newcode68
Documentation/contributor/regressions.itexi:68: the current release and the
prior release.  Each test that has
On 2010/06/08 12:50:23, Graham Percival wrote:
> Somewhere... I think in the "releases" chapter... I noted that the comparison
> only looks inside the bounding boxes within lilypond, so errors due to
> ghostscript aren't found, nor are errors in lyrics+text.
> 
> I don't know exactly what I'm talking about, but it's probably adding a
@warning
> that not **all** of the changes will be shown.

Done.

http://codereview.appspot.com/1545043/diff/1/3#newcode157
Documentation/contributor/regressions.itexi:157: The special regression test
@file{test-output-distance.ly} will always
On 2010/06/08 12:50:23, Graham Percival wrote:
> Please put this paragraph inside a @warning.

Done.

[Graham Percival (old account), 2010-06-08 20:41:01]

Looks good to me!

[Neil Puttock, 2010-06-08 20:42:13]

Hi Carl,

LGTM too.

Cheers,
Neil

http://codereview.appspot.com/1545043/diff/6001/7001
File Documentation/contributor/programming-work.itexi (right):

http://codereview.appspot.com/1545043/diff/6001/7001#newcode1304
Documentation/contributor/programming-work.itexi:1304: @code{test-redo}?
In my experience, all of the above apart from `clean' (which is so quick, the
difference is negligible.)

http://codereview.appspot.com/1545043/diff/6001/7001#newcode1346
Documentation/contributor/programming-work.itexi:1346: @command{make} before
running @command{make test-redo}.  This will
It's been a while since I tested font changes, but doesn't

make -C mf/ clean

work?

http://codereview.appspot.com/1545043/diff/6001/7002
File Documentation/contributor/regressions.itexi (right):

http://codereview.appspot.com/1545043/diff/6001/7002#newcode100
Documentation/contributor/regressions.itexi:100: LilyPond bounding boxes, so
Ghostscript changes and changes in
And log files, so it picks up error/warning messages even if there are no
bounding box changes.

[Carl, 2010-06-09 03:00:39]

Having gotten approval from Graham and Neil, and having responded to Neil's
comments, I'll go ahead and push.  I'll also post the set that I'm pushing for
completeness.

Thanks,

Carl

http://codereview.appspot.com/1545043/diff/6001/7001
File Documentation/contributor/programming-work.itexi (right):

http://codereview.appspot.com/1545043/diff/6001/7001#newcode1346
Documentation/contributor/programming-work.itexi:1346: @command{make} before
running @command{make test-redo}.  This will
On 2010/06/08 20:42:13, Neil Puttock wrote:
> It's been a while since I tested font changes, but doesn't
> 
> make -C mf/ clean
> 
> work?

Haven't tried that, but I'm sure it does.  I do 

rm mf/out/*

This isn't new writing, it's just dumped from CG3.  I'll clean it up later;
Graham asked me to move in stages, focusing on Regression right now.

http://codereview.appspot.com/1545043/diff/6001/7002
File Documentation/contributor/regressions.itexi (right):

http://codereview.appspot.com/1545043/diff/6001/7002#newcode100
Documentation/contributor/regressions.itexi:100: LilyPond bounding boxes, so
Ghostscript changes and changes in
On 2010/06/08 20:42:13, Neil Puttock wrote:
> And log files, so it picks up error/warning messages even if there are no
> bounding box changes.
> 

Done.