Prevent filtered test runs from reporting previous runs' outcomes.

Delete and recreate ant-reports/tests/ before each test run.

This prevents results from tests run previously but not this time (due
to -Dtest.filter or brtest/nobrtest, or a test that was deleted) from
being included in the summary line or JUnit report, possibly resulting
in an announcement of failed (or passed) tests which were not actually
run.

[kpreid2, 2013-04-20 00:02:41]

Note: This has the inconvenient side effect of deleting reports when you 'ant
brserve', which could be fixed by making brserve less of a 'run tests, but not
really' gimmick.

[felix8a, 2013-04-20 01:26:05]

I think deleting results will annoy me a lot, please don't do this. I might be
ok with generating a new test report dir for each test run. (I'm already
planning on doing something like that for multiple browser testing.)

[kpreid2, 2013-04-20 01:33:03]

On 2013/04/20 01:26:05, felix8a wrote:
> I think deleting results will annoy me a lot, please don't do this.

Deleting at all, or deleting in brserve? If the latter, I'll fix brserve in this
change. If the former, I'm not clear on what the benefit is.

> I might be
> ok with generating a new test report dir for each test run. (I'm already
> planning on doing something like that for multiple browser testing.)

I would find new dirs annoying, because I would have to re-copy the results path
into a web browser rather than just reloading. But if we made some way for them
to optionally automatically be `open`/`xdg-open`ed on failure, that would do.

[felix8a, 2013-04-20 02:46:26]

for instance, if I'm working on something involving multiple test failures, I'd
like to use the test results as a reference for what to do next, and I'll
iterate over fixing one failure by rerunning just the specific test failure,
before moving onto the next.

If rerunning one test deletes other test results, then either I have to rerun
all tests (slow) or save a snapshot of the tests before iterating (awkward).

[kpreid2, 2013-04-20 02:57:27]

On 2013/04/20 02:46:26, felix8a wrote:
> for instance, if I'm working on something involving multiple test failures,
I'd
> like to use the test results as a reference for what to do next, and I'll
> iterate over fixing one failure by rerunning just the specific test failure,
> before moving onto the next.
> 
> If rerunning one test deletes other test results, then either I have to rerun
> all tests (slow) or save a snapshot of the tests before iterating (awkward).

I see. I've done that too, now that you mention it. New idea:

Two results directories. One is the current run, and one is the accumulated runs
(the current run is copied into it at the end). The totals and success/failure
output come from the current run, and the HTML report comes from the accumulated
runs.

On a full run (no filter), the accumulated run directory is cleared before the
copy, which gets rid of old bogus data.

Does that sound correct and usable?

[felix8a, 2013-04-20 06:48:41]

tentative yes, but is this actually a problem to solve? usually when I run full
tests I do it as 'ant clean runtests' anyway, since there are several potential
hazards in trusting test results from an unclean build, and the build from
scratch adds less than a minute to the total testing time.

[kpreid2, 2013-04-29 21:32:21]

Closing this review, will open a new one if I write a better solution.