Wait briefly for the API server to start.

Wait briefly for the API server to start.

With the bootstrap node, we need to wait briefly for the
api server to start so we don't have a race condition that
causes the machine agent to bounce along.

https://code.launchpad.net/~thumper/juju-core/task-race-on-api/+merge/184726

(do not edit description out of merge proposal)

[thumper, 2013-09-10 06:03:46]

Please take a look.

[jameinel, 2013-09-10 06:15:02]

For pragmatic purposes, I could be convinced this is ok.

Though I think I prefer my option (2). Instead of having an API blip be
immediately fatal, allow for at least 1 retry after the default retryDelay
(don't use DefaultDialOpts, but use a soft fail instead of a hard fail).

That should be approximately the same size change, and will also handle network
blips elsewhere, and not default us to always having a 3s delay on startup
(which likely effects the test suite as well).

https://codereview.appspot.com/13412047/diff/1/cmd/jujud/machine.go
File cmd/jujud/machine.go (right):

https://codereview.appspot.com/13412047/diff/1/cmd/jujud/machine.go#newcode131
cmd/jujud/machine.go:131: time.Sleep(apiServerWaitTime)
For pragmatic purposes, I could accept a sleep here.

However, couldn't we instead look at around like 147 where we are actually
connecting to the API, and include a sleep/retry loop there?

There are a few bits:

1) the ensureStateWorker call @ line 159 can go away. We've removed the
compatibility code when the state worker starts up, so all those comments can be
removed.

2) the code inside openAPIState says "We let the API dial fail immediately
because the runner's loop outside the caller of openAPIState will keep
retrying".

We know that the runner dying actually kills the state server in some cases. So
if we just change agent.go:142 

st, newPassword, err := agentConfig.OpenAPI(api.DialOpts{RetryDelay:retryDelay,
Timeout:retryDelay*3})

(I wouldn't recommend keeping the default 10min timeout waiting for the server
to start up, but a quick couple-of-seconds retry seems good.)

3) Eventually we want to get away from 'allFatal' so that failure in connecting
to the API doesn't kill all the other state tasks that are running.

That would be a change to machine.go:169 so instead of the API Worker having
allFatal, it would use some other signaling. (though it seems to me that API
connection failures should be all fatal for things running under the APIWorker,
it is just that failures for things under APIWorker shouldn't kill things
running as StateWorkers.)

[rog, 2013-09-10 10:49:58]

On 2013/09/10 06:03:46, thumper wrote:
> Please take a look.

NOT LGTM.

This does not address the actual problem, which is AFAICS that a
state worker going down will take the API server down
with it even if the state connection is still ok.

Also, the maas and azure changes look unrelated to the
CL description.

How about this for an alternative? https://codereview.appspot.com/13640043

[thumper, 2013-09-10 23:32:15]

Please take a look.