New environ ResourceCatalog

New environ ResourceCatalog

A ResourceCatalog is added, which knows how to persist
resource records. A resource record contains a path and
a hash and is used to catalog the data blobs stored in
the recently added ResourceStorage. A category attribute
is also provided and will be used later for defining
resources are tools vs charms etc. 

When a resource entry with the same path is saved, a new
record is not created but instead the existing one has
its ref count incremented. The opposite occurs with
removal. Removing the last record deletes it.

The above will be used by a new ManagedResource implemention
which will know how to store data for environs and users and
do things like de-duping using checksums etc.

https://code.launchpad.net/~wallyworld/juju-core/environ-resource-catalogs/+merge/221484

(do not edit description out of merge proposal)

[wallyworld, 2014-05-30 06:24:13]

Please take a look.

[fwereade, 2014-05-30 09:19:06]

Let's talk about this when you get back. I think it would be more profitable to
start by just straight-up implementing the Storage interface, backed by gridfs,
and leaving all considerations of deduplication out.

https://codereview.appspot.com/104720046/diff/1/state/interface.go
File state/interface.go (right):

https://codereview.appspot.com/104720046/diff/1/state/interface.go#newcode197
state/interface.go:197: // MongoTransactionRunner instances execute mongo
transactions.
/me freaks out, and it's not pretty. I really don't think we should let this
implementation detail leak out of state.

https://codereview.appspot.com/104720046/diff/1/state/storage/interface.go
File state/storage/interface.go (right):

https://codereview.appspot.com/104720046/diff/1/state/storage/interface.go#ne...
state/storage/interface.go:23: // Resources with the same Path value are not
duplicated; instead a reference count is incremented.
I think this is not quite at the correct level of abstraction. Path is a
distraction: it's the *content* that's important, and if the first cut puts the
emphasis on path I think it'll only make it harder to do the deduplication right
when we get to that phase.

https://codereview.appspot.com/104720046/diff/1/state/storage/resourcecatalog.go
File state/storage/resourcecatalog.go (right):

https://codereview.appspot.com/104720046/diff/1/state/storage/resourcecatalog...
state/storage/resourcecatalog.go:52: func NewResourceCatalog(collection
*mgo.Collection, txnRunner state.MongoTransactionRunner) ResourceCatalog {
OK, I can see why you're doing this, but I still don't really like it. It
*could* be fixed by layering -- ie putting the runTransaction functionality into
"state/txn", and having both State and ResourceCatalog use it -- but exposing it
to other random clients of state freaks me right out.

https://codereview.appspot.com/104720046/diff/1/state/storage/resourcecatalog...
state/storage/resourcecatalog.go:93: if ops, err := rc.resourceDecRefOps(id);
err == mgo.ErrNotFound {
Hmm. Is this definitely a "you already did this", or is it perhaps evidence that
everything's totally screwed?