worker/provisioner: make tests deterministic

worker/provisioner: make tests deterministic

Fix TestProvisionerRetriesTransientErrors which
was relying on non-deterministic behaviour of
the provisioner. Wait for machine-1 and then
machine-2 to be provisioned serially.

Fixes lp:1322037

https://code.launchpad.net/~axwalk/juju-core/lp1322037-workerprovisioner-deterministic/+merge/220668

(do not edit description out of merge proposal)

[axw, 2014-05-22 15:11:26]

Please take a look.

[menn0, 2014-05-22 22:24:54]

I *think* I understand what's going on here. Sometimes machine 3 or 4 could end
up being unintentionally up when the test requires that they are only on their
way up at the top of the test?

LGTM.

[axw, 2014-05-23 00:44:03]

On 2014/05/22 22:24:54, menn0 wrote:
> I *think* I understand what's going on here. Sometimes machine 3 or 4 could
end
> up being unintentionally up when the test requires that they are only on their
> way up at the top of the test?

There was non-determinism in the order in which m1 and m2 were provisioned. What
was happening was m1 and m2 were added, both picked up (in a batch) by the
provisioner, and the provisioner would provision them in a "random" order. It
was deterministic in Go 1.2, but not so in 1.3 or gccgo (or the order changed at
any rate).

My change simply waits for machines 1 before going on to 2, and then on to 3.

> LGTM.