Added support for HA backups on restore

Added support for HA backups on restore

HA backups contain a set of file that reference
other state machines besides 0. Restore would
bring up the old db and agent.conf and peergrouper
would try to re-set the HA cluster.
Editing of agents.conf and deletion of some mongo
registries have been placed to set the restore as 
if the machine was a single one.

https://code.launchpad.net/~hduran-8/juju-core/1305026_add_restore_support_for_HA/+merge/219357

Requires: https://code.launchpad.net/~hduran-8/juju-core/upgrade_restore_for_replicaset/+merge/218998

(do not edit description out of merge proposal)

[hduran, 2014-05-13 12:33:02]

Please take a look.

[menn0, 2014-05-14 05:05:04]

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
File cmd/plugins/juju-restore/restore.go (right):

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:150: conf = { "_id" : "juju", "version" : 1,
"members" : [ { "_id" : 1, "host" : "{{ .PrivateAddress | printf "%s:37017" }}"
, "tags" : { "juju-machine-id" : "0" } }]}
This is somewhat tangential to your change but hardcoding the port number here
doesn't seem like such a great idea. 

At the very least, the state and API port numbers from the default config should
be substituted in to the script template.

Depending how close the restored system is supposed to be to the original it
might be more correct to extract the state and API ports  from agent.conf at the
same time the password is pulled out. These values would have to be used during
bootstrap. This is probably out the scope of the work you're doing though :)

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:181: REMOVECOUNT=$(grep -Ec "^-.*17070$"
/var/lib/juju/agents/machine-0/agent.conf )
Same as previous for this port number

[menn0, 2014-05-14 05:19:45]

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
File cmd/plugins/juju-restore/restore.go (right):

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:182: awk -v removecount=$REMOVECOUNT
'/\-.*17070$/{i++}i<1' machine-0/agent.conf > machine-0/agent.conf.new
This call to awk doesn't need removecount

[hduran, 2014-05-14 18:43:50]

Please take a look.

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
File cmd/plugins/juju-restore/restore.go (right):

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:150: conf = { "_id" : "juju", "version" : 1,
"members" : [ { "_id" : 1, "host" : "{{ .PrivateAddress | printf "%s:37017" }}"
, "tags" : { "juju-machine-id" : "0" } }]}
On 2014/05/14 05:05:04, menn0 wrote:
> This is somewhat tangential to your change but hardcoding the port number here
> doesn't seem like such a great idea. 
> 
> At the very least, the state and API port numbers from the default config
should
> be substituted in to the script template.
> 
> Depending how close the restored system is supposed to be to the original it
> might be more correct to extract the state and API ports  from agent.conf at
the
> same time the password is pulled out. These values would have to be used
during
> bootstrap. This is probably out the scope of the work you're doing though :) 

Done.

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:181: REMOVECOUNT=$(grep -Ec "^-.*17070$"
/var/lib/juju/agents/machine-0/agent.conf )
On 2014/05/14 05:05:04, menn0 wrote:
> Same as previous for this port number

Done.

https://codereview.appspot.com/92320043/diff/1/cmd/plugins/juju-restore/resto...
cmd/plugins/juju-restore/restore.go:182: awk -v removecount=$REMOVECOUNT
'/\-.*17070$/{i++}i<1' machine-0/agent.conf > machine-0/agent.conf.new
On 2014/05/14 05:19:45, menn0 wrote:
> This call to awk doesn't need removecount

Done.

[hduran, 2014-05-14 18:45:08]

Please take a look.

[menn0, 2014-05-14 21:33:30]

This is looking great. 

My only remaining concern is, what if environs.ConfigForName loads configuration
with different state and api port numbers than what's in agent.conf? The
environment would be bootstrapped with the port numbers from the .jenv but the
bash script would be working with the numbers in agent.conf. 

As far as I can tell, the port numbers should never be different between the 2
sources so perhaps just add a check to error out if they are.

Other than that LGTM (although you might want to get a second opinion since I'm
new to the team).

[fwereade, 2014-05-15 09:08:54]

So, just to clarify: this takes an env backup that may or may not be HA, and
restores it in non-HA mode?

[hduran, 2014-05-15 09:56:07]

On 2014/05/15 09:08:54, fwereade wrote:
> So, just to clarify: this takes an env backup that may or may not be HA, and
> restores it in non-HA mode?

Yes, we assume that when you restore you no longer have any master nodes up.

[hduran, 2014-05-22 19:58:55]

Please take a look.

[natefinch, 2014-05-22 20:11:27]

LGTM with some typo fixes

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt
File doc/backup_and_restore.txt (right):

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:13: The process gathers varios files relevant to the
server such as:
various

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:18: it less than ideal to be run as often as a backup
process should be.
I presume it also restarts the db, could you give an estimate of for how long
the db is down, and what the resulting user experience is like?  (For example,
attempting juju commands will fail, but running services will not be affected)

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:23: juju-restore is a go cmd plugin that, if no
state-server is present, will
s/go cmd/juju/

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:24: bootstrap a new node is safe mode then upload to
the tgz backup file
s/is/in/  also, what's safe mode?

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:29: * Untars the fs files ino the current machine
s/ino/into/

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:31: machine with the ones of the new one in the
relevant config files and also
s/the ones of the new one/those of the new machine/

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:32: on the db, if thi step is not preformed
peergrouper will kick our machine
s/on/in/    s/thi/this/    s/preformed/performed/

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:44: We asume, for now, that if you are running
restore is because you have
s/asume/assume/

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:48: State Servers since otherwise this will take them
out of line and possibly
s/out of line/offline/

[hduran, 2014-05-23 00:05:31]

Please take a look.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt
File doc/backup_and_restore.txt (right):

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:13: The process gathers varios files relevant to the
server such as:
On 2014/05/22 20:11:27, nate.finch wrote:
> various

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:18: it less than ideal to be run as often as a backup
process should be.
On 2014/05/22 20:11:26, nate.finch wrote:
> I presume it also restarts the db, could you give an estimate of for how long
> the db is down, and what the resulting user experience is like?  (For example,
> attempting juju commands will fail, but running services will not be affected)

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:23: juju-restore is a go cmd plugin that, if no
state-server is present, will
On 2014/05/22 20:11:27, nate.finch wrote:
> s/go cmd/juju/

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:24: bootstrap a new node is safe mode then upload to
the tgz backup file
On 2014/05/22 20:11:26, nate.finch wrote:
> s/is/in/  also, what's safe mode?

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:29: * Untars the fs files ino the current machine
On 2014/05/22 20:11:26, nate.finch wrote:
> s/ino/into/

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:31: machine with the ones of the new one in the
relevant config files and also
On 2014/05/22 20:11:27, nate.finch wrote:
> s/the ones of the new one/those of the new machine/

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:32: on the db, if thi step is not preformed
peergrouper will kick our machine
On 2014/05/22 20:11:27, nate.finch wrote:
> s/on/in/    s/thi/this/    s/preformed/performed/

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:44: We asume, for now, that if you are running
restore is because you have
On 2014/05/22 20:11:27, nate.finch wrote:
> s/asume/assume/

Done.

https://codereview.appspot.com/92320043/diff/60001/doc/backup_and_restore.txt...
doc/backup_and_restore.txt:48: State Servers since otherwise this will take them
out of line and possibly
On 2014/05/22 20:11:27, nate.finch wrote:
> s/out of line/offline/

Done.