Replace redis with mongoqueues.

Replace redis with mongoqueues.

Replaced all redis queues in jobs with Mongoqueues.
Updated tests accordingly.
Removed other calls to redis (e.g. reset-db)
Removed redis from sys deps.

There's more to do to take advantage or some parts of mongoqueues (e.g.
queue.error) but that is best done in subsequent work as the job system will
be refactored heavily for queue/worker interaction.

https://code.launchpad.net/~jcsackett/charmworld/mongoize-queue/+merge/146737

(do not edit description out of merge proposal)

[j.c.sackett, 2013-02-05 22:08:36]

Please take a look.

[j.c.sackett, 2013-02-05 22:42:56]

Please take a look.

[rharding, 2013-02-06 01:48:02]

please update from trunk. This branch removes the code abel and I landed
yesterday.

A couple of suggestions for test structure and such. 

Thanks for the changes. It looks good and makes a lot of sense. 

Holding a +1 just to make sure we get straight the removing of the code from the
day.

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/config.py
File charmworld/jobs/config.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/config.py#ne...
charmworld/jobs/config.py:26: LP_OUT_QUEUE = MongoQueue(db['bzr-fetch'],
'bzr-fetch-queue')
Is there a reaon the db collections are done as dicts vs object access?

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/config.py#ne...
charmworld/jobs/config.py:65: # deletion?
Looks like this was added in -r 53 but never tied anywhere that bzr grep can
find in the history. I'd say safe to delete.

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/tests/test_b...
File charmworld/jobs/tests/test_bzr.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/tests/test_b...
charmworld/jobs/tests/test_bzr.py:16: (in_queue, out_queue) =
self.get_mongo_queues(data=charm_data)
So with making the queues part of setup this would just be:
self.in_queue.put(data)

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py
File charmworld/models.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py#oldcode39
charmworld/models.py:39: 
please keep this code.

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py#oldcode212
charmworld/models.py:212: class CharmFileSet(object):
this code should not be removed as well.

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_models.py
File charmworld/test_models.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_models.py#ol...
charmworld/test_models.py:2: from os.path import dirname
Mode code that should stay.

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_utils.py
File charmworld/test_utils.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_utils.py#old...
charmworld/test_utils.py:9: import urllib
this is abels code that landed.

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__.py
File charmworld/testing/__init__.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:18: from charmworld.models import getfs
More code to keep

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:47: self.fs = getfs(self.db)
Why is this deleted? This is used in tests for the mongo file store.

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__.py
File charmworld/testing/__init__.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:105: def get_mongo_queues(self, data=None):
Is there any reason to not just add these to setUp/tearDown methods? The queues
can be made available as self.in_queue and self.out_queue.

[j.c.sackett, 2013-02-06 02:39:08]

Sorry about all the deleted code; I had a weird issue where it was citing the
merged tip in 'bzr st' after updating from trunk. I did another commit to get
rid of that and it didn't show as an issue. bzr st and bzr diff -r colo:trunk
never showed this deleted code either.

I'll certainly fix that before landing. I've marked the changes you were
concerned by as fallout where appropriate.

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/config.py
File charmworld/jobs/config.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/config.py#ne...
charmworld/jobs/config.py:26: LP_OUT_QUEUE = MongoQueue(db['bzr-fetch'],
'bzr-fetch-queue')
On 2013/02/06 01:48:02, rharding wrote:
> Is there a reaon the db collections are done as dicts vs object access?

Only that I want to make it explicit that they're arbitrarily named; that mongo
does a behind the scenes any attr becomes available access wasn't clear to me as
I was writing this. I can change it if you feel it's an issue.

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/tests/test_b...
File charmworld/jobs/tests/test_bzr.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/jobs/tests/test_b...
charmworld/jobs/tests/test_bzr.py:16: (in_queue, out_queue) =
self.get_mongo_queues(data=charm_data)
On 2013/02/06 01:48:02, rharding wrote:
> So with making the queues part of setup this would just be:
> self.in_queue.put(data)'

Not all tests using the JobTestClass are going to need queues; we're already
getting to the point where `make test` is reason for me to switch to another
window for a bit--I don't see the advantage of adding the setUp and tearDown for
everything.

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py
File charmworld/models.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py#oldcode39
charmworld/models.py:39: 
On 2013/02/06 01:48:02, rharding wrote:
> please keep this code.

This is all the result of a merge error; I'm not sure how it is I lost the merge
from trunk, but this is all fallout from that. I'll certainly fix it before
landing.

https://codereview.appspot.com/7310046/diff/3001/charmworld/models.py#oldcode212
charmworld/models.py:212: class CharmFileSet(object):
On 2013/02/06 01:48:02, rharding wrote:
> this code should not be removed as well.
Same response as above.

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_models.py
File charmworld/test_models.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_models.py#ol...
charmworld/test_models.py:2: from os.path import dirname
On 2013/02/06 01:48:02, rharding wrote:
> Mode code that should stay.
Yeah, this is more fallout.

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_utils.py
File charmworld/test_utils.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/test_utils.py#old...
charmworld/test_utils.py:9: import urllib
On 2013/02/06 01:48:02, rharding wrote:
> this is abels code that landed.
More fallout.

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__.py
File charmworld/testing/__init__.py (left):

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:18: from charmworld.models import getfs
On 2013/02/06 01:48:02, rharding wrote:
> More code to keep
More fallout.

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:47: self.fs = getfs(self.db)
On 2013/02/06 01:48:02, rharding wrote:
> Why is this deleted? This is used in tests for the mongo file store.

It's just fallout.

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__.py
File charmworld/testing/__init__.py (right):

https://codereview.appspot.com/7310046/diff/3001/charmworld/testing/__init__....
charmworld/testing/__init__.py:105: def get_mongo_queues(self, data=None):
On 2013/02/06 01:48:02, rharding wrote:
> Is there any reason to not just add these to setUp/tearDown methods? The
queues
> can be made available as self.in_queue and self.out_queue.

Alas, rietveld doesn't show these in order, or I would have answered this first.
As I said in the example you picked, not all tests require this--I don't see any
reason to add to the time for those tests, given I'm already starting to chafe
at `make test` time. As we refactor the test code, there should be more, not
less tests that don't need the queues.

[rharding, 2013-02-06 02:40:57]

lgtm once merge issues are resolved.

[j.c.sackett, 2013-02-06 16:12:25]

Please take a look.

[hazmat, 2013-02-06 16:40:44]

lgtm, land with changes.

in future we can cleanup the exception handling nicely with the job as context
manager.

https://codereview.appspot.com/7310046/diff/10001/charmworld/jobs/config.py
File charmworld/jobs/config.py (right):

https://codereview.appspot.com/7310046/diff/10001/charmworld/jobs/config.py#n...
charmworld/jobs/config.py:18: db = getdb(connection, DB_NAME)
instead of reintroducing a global mongo connection, it would be nicer if this
could instantiate the job.main to get db, and use the queue names here to get
the collections. perhaps just a helper method get_queue(name) to reduce the
redundancy.

This would also get rid of the current use of global instance variables bound to
function defaults.

[j.c.sackett, 2013-02-06 20:31:22]

Please take a look.

[rharding, 2013-02-06 22:04:39]

lgtm