HA: system-identity file for all state servers

HA: system-identity file for all state servers

This is a fix for the following bug:
--------------------------------
I just tried running in HA mode:
$ juju run --machine 0,1,2 "cat /etc/hostname" -e amz-jam
- MachineId: "0"
  ReturnCode: 255
  Stderr: "Warning: Identity file /var/lib/juju/system-identity not accessible: No
    such file or directory.\nWarning: Permanently added 'ip-10-72-129-222.eu-west-1.compute.internal,10.72.129.222'
    (ECDSA) to the list of known hosts.\r\nPermission denied (publickey).\r\n"
  Stdout: ""

This was repeated for all 3 machines.

Looking at the machines only machine-0 has /var/lib/juju/system-identity

So the StateSecrets needs to include /var/lib/juju/system-identity as well as the Mongo and CA certs.
--------------------------------

The main idea of this patchset is passing of environment SSH key via agent.conf and creation of
/var/lib/juju/system-identity file on agent start.
For this purpose SystemIdentity field was added to params.StateServingInfo and agent.format_1_18Serialization structs.

https://code.launchpad.net/~klyachin/juju-core/104-juju-run-failure/+merge/216834

(do not edit description out of merge proposal)

[vladislav.klyachin, 2014-04-23 09:04:04]

Please take a look.

[fwereade, 2014-04-23 10:01:07]

This looks good, but AFAICT it won't help systems that have already upgraded to
1.18 -- don't we need a new set of upgrade steps that'll be run post-1.18?

[jameinel, 2014-04-23 12:21:55]

On 2014/04/23 10:01:07, fwereade wrote:
> This looks good, but AFAICT it won't help systems that have already upgraded
to
> 1.18 -- don't we need a new set of upgrade steps that'll be run post-1.18?

well, we didn't have HA in 1.18, so that part of the bug didn't need fixing. The
only upgrade changes are just because the naming got moved around a bit, so I
don't think we have to worry about upgrades here.

It is more that after EnsureAvailability we should end up with the secrets on
the new machines.

[jameinel, 2014-04-23 15:15:07]

LGTM

Sorry it took so long but it took me a while to figure out why it was working.

Essentially, the document in state is directly punned with the params object.
Which is something we explicitly don't do anywhere else. (state/* is supposed to
be internal model objects vs state/api/params being the RPC objects).

So the order is that during initial bootstrap, we right out the agent.conf file
and call "jujud bootstrap-state".
Which then comes up and writes the /var/lib/juju/system-identity thanks to
Vlad's patch and then already called
st.SetStateServingInfo(agentConfig.StateServingInfo())
So because the data and types are punned when we updated the params object to
hold the content in agent.conf it automatically got injected into the database.
Which then means that when a new API Server comes up it does:
		if job.NeedsState() {
			info, err := st.Agent().StateServingInfo()
...
			err = a.ChangeConfig(func(config agent.ConfigSetter) {
				config.SetStateServingInfo(info)
			})
and then
			agentConfig = a.CurrentConfig()
to re-read the one it just updated.

And again, because the types are punned when the State object gets the new
attributes, the API gets the new attributes, leading to the Agent on the other
side getting the new attributes.

The only other thing we need to worry about is if StateServingInfo was an API in
1.18 we shouldn't be changing its signature wantonly, but since it only existed
for HA we can still tweak it.

https://codereview.appspot.com/90580043/diff/1/agent/identity_test.go
File agent/identity_test.go (right):

https://codereview.appspot.com/90580043/diff/1/agent/identity_test.go#newcode54
agent/identity_test.go:54: }
I feel like it would be good to check the file has 0600 permissions, just to
avoid someone slipping and messing up that bit in the future.

[vladislav.klyachin, 2014-04-23 15:45:33]

Please take a look.