Parse and validate tools metadata

Parse and validate tools metadata

Tools metadata in simplestreams format can be parsed and
validated. A new environs/tools package is created, and the
existing environs/tools.go is moved there, and the new 
simplestreams support added also. The agent/tools package
is disambiguated as agenttools where there were clashes.

The metadata plugin is extended to support validating tools
metadata in the same way as image metadata. The next branches
will support using the tools metadata when finding tools to
bootstrap with.

https://code.launchpad.net/~wallyworld/juju-core/parse-tools-simplestreams/+merge/179845

Requires: https://code.launchpad.net/~wallyworld/juju-core/refactor-simplestreams/+merge/178878

(do not edit description out of merge proposal)

[wallyworld, 2013-08-13 05:03:03]

Please take a look.

[fwereade, 2013-08-13 17:13:18]

I feel like I need to do another pass, but that I can do it a bit later because
you're off tomorrow. In case you pass by I have a couple of preliminaries.

FWIW the moves/renames feel really noisy and make it rather harder to follow the
CL as a whole; where possible I'd generally appreciate seeing these sorts of
changes in their own CLs.

And while I don't quite follow why tools.go moved into the tools package (I
think the layering was previously quite clear) I'm not really that bothered; I
presume they're scheduled for deletion anyway, right?

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
File environs/simplestreams/testing/testing.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
environs/simplestreams/testing/testing.go:66: "com.ubuntu.juju:tools": {
Just checking: smoser's in agreement on how this will look, right?

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
environs/simplestreams/testing/testing.go:101: "1130preciseamd64": {
I'm a little bit twitchy about this. Can we be sure the potential collisions
will never cause upset?

https://codereview.appspot.com/12833043/diff/1/environs/tools/boilerplate.go
File environs/tools/boilerplate.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/tools/boilerplate.go#...
environs/tools/boilerplate.go:44: ToolsBinaryHash: tm.Hash,
Can we maybe call this SHA256 to (1) match what we do for charm validation and
(2) be able to swap the algorithm one day?

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
File environs/tools/simplestreams_test.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
environs/tools/simplestreams_test.go:112: {
}{{ lets you lose a level of indentation without losing clarity IMO. YMMV.

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
environs/tools/simplestreams_test.go:128: {
}, { similarly

[jameinel, 2013-08-14 10:28:33]

Maybe I'm misunderstanding what you are concerned about?

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
File environs/simplestreams/testing/testing.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
environs/simplestreams/testing/testing.go:101: "1130preciseamd64": {
On 2013/08/13 17:13:18, fwereade wrote:
> I'm a little bit twitchy about this. Can we be sure the potential collisions
> will never cause upset?

I assume you mean the short-name. Short names are guaranteed to be unique inside
a stream collection (by how the stream is generated) as part of the spec.

I don't think we want to count on them being 'stable', but they are a form of
length compression for the stream data.

[wallyworld, 2013-08-14 23:46:16]

Launchpad's merge proposal diffs handle the moves renames properly without the
noise and it's a lot less work to move the files and tweak a few lines therein
using the one merge proposal. If reitveld can't do this properly, perhaps we
should stick to just using Launchpad :-P

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
File environs/simplestreams/testing/testing.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
environs/simplestreams/testing/testing.go:66: "com.ubuntu.juju:tools": {
On 2013/08/13 17:13:18, fwereade wrote:
> Just checking: smoser's in agreement on how this will look, right?

It's based on previous discussions and I've also started a new email thread to
finalise it all.

https://codereview.appspot.com/12833043/diff/1/environs/simplestreams/testing...
environs/simplestreams/testing/testing.go:101: "1130preciseamd64": {
On 2013/08/13 17:13:18, fwereade wrote:
> I'm a little bit twitchy about this. Can we be sure the potential collisions
> will never cause upset?

This is made up data at the moment. But the keys in the items map do not need to
be globally unique. The scheme I used - versionseriesarch - is sufficient.

https://codereview.appspot.com/12833043/diff/1/environs/tools/boilerplate.go
File environs/tools/boilerplate.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/tools/boilerplate.go#...
environs/tools/boilerplate.go:44: ToolsBinaryHash: tm.Hash,
On 2013/08/13 17:13:18, fwereade wrote:
> Can we maybe call this SHA256 to (1) match what we do for charm validation and
> (2) be able to swap the algorithm one day?

Done.

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
File environs/tools/simplestreams_test.go (right):

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
environs/tools/simplestreams_test.go:112: {
On 2013/08/13 17:13:18, fwereade wrote:
> }{{ lets you lose a level of indentation without losing clarity IMO. YMMV.

Done.

https://codereview.appspot.com/12833043/diff/1/environs/tools/simplestreams_t...
environs/tools/simplestreams_test.go:128: {
On 2013/08/13 17:13:18, fwereade wrote:
> }, { similarly

Done.

[wallyworld, 2013-08-14 23:47:56]

Please take a look.

[wallyworld, 2013-08-15 04:02:15]

Please take a look.

[fwereade, 2013-08-19 10:55:34]

LGTM, sorry for the delay.