Implement rsyslog TLS support

Implement rsyslog TLS support

This changes the rsyslog configuration
we generate to use TLS. We install the
rsyslog-gnutls package and generate a
new CA cert, server cert and key
specifically for rsyslog.

This completely changes the way rsyslog
configuration is managed. Now, instead
of writing at bootstrap time and having
an upgrade step, we have a worker that
writes the syslog config whenever syslog
parameters change. The state server will
generate certificates and propagate the
CA cert to other agents via environ config.

There are some other miscellaneous changes:
  - we now use reliable forwarding, as
    otherwise the machine agent and unit
    agent will restart rsyslog while
    log messages are buffered causing
    message loss
  - dedicated upgrades for rsyslog are
    redundant and removed. The new worker
    upgrades machine and unit agent rsyslog
    automatically.
  - syslog-port can now be changed, and must
    be changed to work around the privilege
    drop race in rsyslog 5.x (this is the
    sole motivation)
  - the ryslog config path is no longer
    populated into agent.conf, and we
    no longer use the existing value to
    perform cleanups. The worker is now
    responsible for cleaning up the config
    file on tear-down.
  - the local provider symlinks machine-0.log
    into /var/log/juju$namespace so that we
    do not need any configuration other than
    the existing namespace to determine log
    location

TODO(axw) tests in worker/rsyslog
TODO(axw) tests in state/api/rsyslog
TODO(axw) tests in state/apiserver/rsyslog

Fixes lp:1281071

https://code.launchpad.net/~axwalk/juju-core/lp1281071-rsyslog-tls/+merge/207889

(do not edit description out of merge proposal)

[axw, 2014-02-24 10:30:08]

Please take a look.

[thumper, 2014-02-24 21:48:28]

We already need to rewrite the syslog config files as part of the 1.18 upgrade
process as they are all out of date and wrong, so this is something we need to
address anyway.

I'd like to see us move from UDP to TCP for the logging.

I'm a little surprised that I have never been hit by the inability for a machine
to listen on the current port.

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go
File environs/cloudinit/cloudinit.go (right):

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.g...
environs/cloudinit/cloudinit.go:434: configRenderer.TLSCACertPath =
path.Join(cfg.LogDir, "ca.pem")
Do we really want to reuse the same cert?  I strongly suggest that we have
different certs / keys for different jobs.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go
File log/syslog/config.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode38
log/syslog/config.go:38: //          File="{{logDir}}/all-machines.log"
Argh... you can't just change this.

rsyslog has an apparmor profile that restricts where it can write files.  This
was a source of many hours of searching.

Needs to be /var/log/** somewhere.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode64
log/syslog/config.go:64: $ModLoad imudp
I think we should actually be using TCP not UDP for log messages anyway.  There
is a built in store and forward mechanism.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode65
log/syslog/config.go:65: $UDPServerRun {{portNumber}}
Why had I not been hit by this port number problem?

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode198
log/syslog/config.go:198: addressPrefix = "@@"
If we switch to TCP for all, we can avoid this.

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go
File log/syslog/testing/syslogconf.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf....
log/syslog/testing/syslogconf.go:25: $DefaultNetstreamDriverCAFile
/var/log/juju/ca.pem
Oh? so this isn't the same as the mongo cert?

[axw, 2014-02-25 02:08:50]

I'll start on a new worker, and do the certificate stuff at the same time.

If we don't mind installing rsyslog-gnutls on machine-0 for local, we can remove
the syslog-tls config attr and make TLS mandatory.

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go
File environs/cloudinit/cloudinit.go (right):

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.g...
environs/cloudinit/cloudinit.go:434: configRenderer.TLSCACertPath =
path.Join(cfg.LogDir, "ca.pem")
On 2014/02/24 21:48:29, thumper wrote:
> Do we really want to reuse the same cert?  I strongly suggest that we have
> different certs / keys for different jobs.

I will update the code to generate a new CA cert/key for rsyslog, and use that
to sign a new server cert. The CA cert will be populated into environment
config, which the rsyslog worker will watch for.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go
File log/syslog/config.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode38
log/syslog/config.go:38: //          File="{{logDir}}/all-machines.log"
On 2014/02/24 21:48:29, thumper wrote:
> Argh... you can't just change this.

I tested it and it works on both precise and trusty.

> rsyslog has an apparmor profile that restricts where it can write files.  This
> was a source of many hours of searching.

I know, and I did it knowingly. It is writing to /var/log/juju, on account of
the symlink.

> Needs to be /var/log/** somewhere.

If we hard-code it, then the SyslogConfig parameters ought to be changed.
Otherwise it looks like it's going to write to the configured log-dir, but
doesn't.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode64
log/syslog/config.go:64: $ModLoad imudp
On 2014/02/24 21:48:29, thumper wrote:
> I think we should actually be using TCP not UDP for log messages anyway. 
There
> is a built in store and forward mechanism.

Probably not a bad idea, but let's please leave that until after TLS is sorted.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode65
log/syslog/config.go:65: $UDPServerRun {{portNumber}}
On 2014/02/24 21:48:29, thumper wrote:
> Why had I not been hit by this port number problem?

I don't know the specifics of the bug. All I know is that it only hit me when I
was using TLS, and not UDP.

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go
File log/syslog/testing/syslogconf.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf....
log/syslog/testing/syslogconf.go:25: $DefaultNetstreamDriverCAFile
/var/log/juju/ca.pem
On 2014/02/24 21:48:29, thumper wrote:
> Oh? so this isn't the same as the mongo cert?

It is, it's just copied to another place to allow for using a different one.

[axw, 2014-02-25 14:22:14]

Please take a look.

[axw, 2014-02-25 14:27:20]

On 2014/02/25 14:22:14, axw wrote:
> Please take a look.

This CL has jumped the shark, I guess, but I wanted to get it out there to get
feedback.
If the direction is good I will try to pare it back, keeping the existing
environs/cloudinit and related code, and remove all that in a followup.

[axw, 2014-02-27 05:40:18]

On 2014/02/25 14:27:20, axw wrote:
> On 2014/02/25 14:22:14, axw wrote:
> > Please take a look.
> 
> This CL has jumped the shark, I guess, but I wanted to get it out there to get
> feedback.
> If the direction is good I will try to pare it back, keeping the existing
> environs/cloudinit and related code, and remove all that in a followup.

This is superseded by https://codereview.appspot.com/68930045