Use the twisted client in the scan tool

Use the twisted client in the scan tool

[ekasper, 2014-01-28 20:29:57]

The purpose of this CL is to demonstrate an end-to-end twisted workflow. I think
async processing should pretty much compensate for removing multiprocessing -
though it depends on network delay, and is hard to measure precisely.

[Eran, 2014-01-29 09:43:50]

Given that some Twisted tests fail on every platform I've tried so far, can we
have the multiplocessing and twisted clients coexist?

[ekasper, 2014-01-29 10:25:33]

I would rather fix the failing tests :) They succeed on every platform I've
tried though, so I'm in the dark here. Do you think you could give me a failure
log along with platform and package info?

On 2014/01/29 09:43:50, Eran wrote:
> Given that some Twisted tests fail on every platform I've tried so far, can we
> have the multiplocessing and twisted clients coexist?

[Eran, 2014-02-04 11:35:50]

Odd, I thought I saw these failures on my Mac but can't reproduce. I'll review
the CL later today.

[Eran, 2014-02-13 12:59:41]

Overall LGTM once the comments are addressed.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
File src/python/ct/client/tools/scan.py (right):

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:39: chain = []
chain does not seem to be used anywhere in this function.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:60: # Has to be an IBodyProducer.
Nit: Document that the produces has to be an IBodyProducer by adding
documentation to this method.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:67: producer_done.addBoth(self.stop)
Seems like you should do error handling in self.stop if you add it as a callback
and errback.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:83: def loop(self):
Nit: Document that this is the entry point for scanning, returning a deferred
that is invoked when the scan is done.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:149: def scan():
Nit: Name start_scan or initiate_scan or something similar, as this method does
not really do any scanning itself.

https://codereview.appspot.com/50390045/diff/20001/src/python/ct/client/tools...
src/python/ct/client/tools/scan.py:153: d = client.get_sth()
Nit: Document that the get_entries_callback depends on get_sth completing
because it needs the size of the tree.

[Eran, 2014-02-13 13:40:02]

Also, can you have a look at the twisted-related errors I've sent the other
day?



On Thu, Feb 13, 2014 at 12:59 PM, <eranm@google.com> wrote:

> Overall LGTM once the comments are addressed.
>
>
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py
> File src/python/ct/client/tools/scan.py (right):
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode39
> src/python/ct/client/tools/scan.py:39: chain = []
> chain does not seem to be used anywhere in this function.
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode60
> src/python/ct/client/tools/scan.py:60: # Has to be an IBodyProducer.
> Nit: Document that the produces has to be an IBodyProducer by adding
> documentation to this method.
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode67
> src/python/ct/client/tools/scan.py:67: producer_done.addBoth(self.stop)
> Seems like you should do error handling in self.stop if you add it as a
> callback and errback.
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode83
> src/python/ct/client/tools/scan.py:83: def loop(self):
> Nit: Document that this is the entry point for scanning, returning a
> deferred that is invoked when the scan is done.
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode149
> src/python/ct/client/tools/scan.py:149: def scan():
> Nit: Name start_scan or initiate_scan or something similar, as this
> method does not really do any scanning itself.
>
> https://codereview.appspot.com/50390045/diff/20001/src/
> python/ct/client/tools/scan.py#newcode153
> src/python/ct/client/tools/scan.py:153: d = client.get_sth()
> Nit: Document that the get_entries_callback depends on get_sth
> completing because it needs the size of the tree.
>
> https://codereview.appspot.com/50390045/
>

[Eran, 2014-04-10 16:23:21]

LGTM 
Many apologies for the delay. These twisted errors has been due to an old
twisted version on my machine.
Feel free to commit at your leisure. I can commit on your behalf if you'd like.