Remove references to test-output-distance.ly.

Remove references to test-output-distance.ly.

It was removed in commit 8a74a0e ("Remove
input/regression/test-output-distance.ly")

[hahnjo, 2020-03-29 18:15:27]

https://codereview.appspot.com/549780044/diff/553810043/Documentation/contrib...
File Documentation/contributor/regressions.itexi (right):

https://codereview.appspot.com/549780044/diff/553810043/Documentation/contrib...
Documentation/contributor/regressions.itexi:240: The special regression test
@file{test-output-distance.ly} will always
additional reference

https://codereview.appspot.com/549780044/diff/553810043/Documentation/contrib...
Documentation/contributor/regressions.itexi:255: @file{test-output-distance.ly}.
additional reference

[hanwenn, 2020-03-29 18:45:00]

more refs

[Dan Eble, 2020-03-29 21:33:55]

> Remove references to test-output-distance.ly.
>
> It was removed in commit 8a74a0e ("Remove
> input/regression/test-output-distance.ly")

I must have slept through that.  Had I been paying attention, I would have
pointed out what the documentation you propose to remove says:

    This test changes each time it is run, and serves to verify that the
regression tests have, in fact, run.

test-output-distance was removed on the grounds that the self-test serves the
same purpose, but I don't see how it does.

[hanwenn, 2020-03-29 21:39:19]

On Sun, Mar 29, 2020 at 11:33 PM <nine.fierce.ballads@gmail.com> wrote:
>
> > Remove references to test-output-distance.ly.
> >
> > It was removed in commit 8a74a0e ("Remove
> > input/regression/test-output-distance.ly")
>
> I must have slept through that.  Had I been paying attention, I would
> have pointed out what the documentation you propose to remove says:
>
>     This test changes each time it is run, and serves to verify that the
> regression tests have, in fact, run.
>
> test-output-distance was removed on the grounds that the self-test
> serves the same purpose, but I don't see how it does.
>

Could you elaborate? What failure scenario are you worried about?

>
>
>
> https://codereview.appspot.com/549780044/



-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dan_faithful.be, 2020-03-29 21:45:59]

On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
>> test-output-distance was removed on the grounds that the self-test
>> serves the same purpose, but I don't see how it does.
> 
> Could you elaborate? What failure scenario are you worried about?

My question is, how does the self-test "verify that the regression tests have,
in fact, run"?  I don't see how it could do more than verify that itself has
run.
— 
Dan

[c_sorensen, 2020-03-30 00:38:35]

On 3/29/20, 3:39 PM, "lilypond-devel on behalf of Han-Wen Nienhuys"
<lilypond-devel-bounces+c_sorensen=byu.edu@gnu.org on behalf of
hanwenn@gmail.com> wrote:

    On Sun, Mar 29, 2020 at 11:33 PM <nine.fierce.ballads@gmail.com> wrote:
    >
    > > Remove references to test-output-distance.ly.
    > >
    > > It was removed in commit 8a74a0e ("Remove
    > > input/regression/test-output-distance.ly")
    >
    > I must have slept through that.  Had I been paying attention, I would
    > have pointed out what the documentation you propose to remove says:
    >
    >     This test changes each time it is run, and serves to verify that the
    > regression tests have, in fact, run.
    >
    > test-output-distance was removed on the grounds that the self-test
    > serves the same purpose, but I don't see how it does.
    >
    
    Could you elaborate? What failure scenario are you worried about?

As a novice in running the test suite, before test-output-distance was part of
the test suite, we could run tests and get no output.  So we thought that
everything was OK, but it wasn't.  Since adding test-output-distance, we always
get the result of at least one regression test failure.

I thought it was a good addition.

Thanks,

Carl
 

[hanwenn, 2020-03-30 21:35:08]

On Sun, Mar 29, 2020 at 11:45 PM Dan Eble <dan@faithful.be> wrote:
>
> On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> >> test-output-distance was removed on the grounds that the self-test
> >> serves the same purpose, but I don't see how it does.
> >
> > Could you elaborate? What failure scenario are you worried about?
>
> My question is, how does the self-test "verify that the regression tests have,
in fact, run"?  I don't see how it could do more than verify that itself has
run.

How does test-output-distance verify that the regression tests have run?

The verification doesn't come from the test file. It comes from the
fact that someone is looking at the test output.

The self-test gives us assurance that the test result index.html file
is making sense. You can verify that the tests were run by checking
that the index.html file is there.

For further context, I am interested in automating our test setup
further: I want to have CI that can post feedback ("test failed!") on
code reviews automatically. test-output-distance is a test that always
fails, which makes that more complicated.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dan_faithful.be, 2020-03-31 00:38:12]

On Mar 30, 2020, at 17:34, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> 
> On Sun, Mar 29, 2020 at 11:45 PM Dan Eble <dan@faithful.be> wrote:
>> 
>> On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
>>>> test-output-distance was removed on the grounds that the self-test
>>>> serves the same purpose, but I don't see how it does.
>>> 
>>> Could you elaborate? What failure scenario are you worried about?
>> 
>> My question is, how does the self-test "verify that the regression tests
have, in fact, run"?  I don't see how it could do more than verify that itself
has run.
> 
> How does test-output-distance verify that the regression tests have run?
> 
> The verification doesn't come from the test file. It comes from the
> fact that someone is looking at the test output.

(Rhetorically) How much verification comes from looking at the output since you
removed the test?

> The self-test gives us assurance that the test result index.html file
> is making sense.

Yes, and that does not overlap with the purpose of test-output-distance.

> You can verify that the tests were run by checking
> that the index.html file is there.

I think the only thing that the presence of index.html implies is that
output-distance.py was run.

> For further context, I am interested in automating our test setup
> further: I want to have CI that can post feedback ("test failed!") on
> code reviews automatically. test-output-distance is a test that always
> fails, which makes that more complicated.

This is what the description of the commit that removed test-output-distance
ought to have been.

I've been trying to point out that something of value was lost.  This last
statement of yours finally gives something to weigh that value against.
-- 
Dan

[hanwenn, 2020-04-09 17:31:30]

commit f342a35bb1f4ab7c85e13a344a126eece38993fb (test-output-dist)
Author: Han-Wen Nienhuys <hanwen@lilypond.org>
Date:   Sun Mar 29 18:52:50 2020 +0200

    Remove references to test-output-distance.ly.