Add queryview to m-lab.analysis

Add queryview to m-lab.analysis

[dominic-mlab, 2013-05-31 16:16:11]

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py
File queryview/queryview.py (right):

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py#newcode28
queryview/queryview.py:28: def bigquery_exec(query_string, output_filename,
verbose=False):
why not use the bigquery api instead of calling out to shell?

https://codereview.appspot.com/9763044/diff/1/queryview/sql/ndt-tmpl-minrtt.sql
File queryview/sql/ndt-tmpl-minrtt.sql (right):

https://codereview.appspot.com/9763044/diff/1/queryview/sql/ndt-tmpl-minrtt.s...
queryview/sql/ndt-tmpl-minrtt.sql:6:
NTH(90,QUANTILES(web100_log_entry.snap.MinRTT,101)) AS quantile_90,
the web100 var should be passed in to a generic web100 template

[stephenz-oti, 2013-05-31 17:09:56]

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py
File queryview/queryview.py (right):

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py#newcode28
queryview/queryview.py:28: def bigquery_exec(query_string, output_filename,
verbose=False):
On 2013/05/31 16:16:11, dominic-mlab wrote:
> why not use the bigquery api instead of calling out to shell?

That is certainly my intention and was my first hope.  I did not find the api as
easily exposed as I expected.  B/c the bq command encapsulates all the
authentication logic and boilerplate setup, as a first-pass, this was the right
balance of cost/benefit.

https://codereview.appspot.com/9763044/diff/1/queryview/sql/ndt-tmpl-minrtt.sql
File queryview/sql/ndt-tmpl-minrtt.sql (right):

https://codereview.appspot.com/9763044/diff/1/queryview/sql/ndt-tmpl-minrtt.s...
queryview/sql/ndt-tmpl-minrtt.sql:6:
NTH(90,QUANTILES(web100_log_entry.snap.MinRTT,101)) AS quantile_90,
On 2013/05/31 16:16:11, dominic-mlab wrote:
> the web100 var should be passed in to a generic web100 template

I've added a generic web100 template in the second patch set, with an example in
the README.

[dominic-mlab, 2013-05-31 19:56:04]

LGTM

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py
File queryview/queryview.py (right):

https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py#newcode28
queryview/queryview.py:28: def bigquery_exec(query_string, output_filename,
verbose=False):
On 2013/05/31 17:09:56, stephenz-oti wrote:
> On 2013/05/31 16:16:11, dominic-mlab wrote:
> > why not use the bigquery api instead of calling out to shell?
> 
> That is certainly my intention and was my first hope.  I did not find the api
as
> easily exposed as I expected.  B/c the bq command encapsulates all the
> authentication logic and boilerplate setup, as a first-pass, this was the
right
> balance of cost/benefit.

when you're ready, check out geoviz in this very repo. It uses the bq api to
pull in all the locations of ndt tests.

https://codereview.appspot.com/9763044/diff/6001/queryview/sql/ndt-tmpl-web10...
File queryview/sql/ndt-tmpl-web100.sql (right):

https://codereview.appspot.com/9763044/diff/6001/queryview/sql/ndt-tmpl-web10...
queryview/sql/ndt-tmpl-web100.sql:19: AND
IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
so can you now use this template instead of the minrtt one? if so, maybe there's
no need to have them all in. :)

[stephenz-oti, 2013-06-03 21:58:26]

On 2013/05/31 19:56:04, dominic-mlab wrote:
> LGTM
> 
> https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py
> File queryview/queryview.py (right):
> 
> https://codereview.appspot.com/9763044/diff/1/queryview/queryview.py#newcode28
> queryview/queryview.py:28: def bigquery_exec(query_string, output_filename,
> verbose=False):
> On 2013/05/31 17:09:56, stephenz-oti wrote:
> > On 2013/05/31 16:16:11, dominic-mlab wrote:
> > > why not use the bigquery api instead of calling out to shell?
> > 
> > That is certainly my intention and was my first hope.  I did not find the
api
> as
> > easily exposed as I expected.  B/c the bq command encapsulates all the
> > authentication logic and boilerplate setup, as a first-pass, this was the
> right
> > balance of cost/benefit.
> 
> when you're ready, check out geoviz in this very repo. It uses the bq api to
> pull in all the locations of ndt tests.
> 

I will. That looks great.

>
https://codereview.appspot.com/9763044/diff/6001/queryview/sql/ndt-tmpl-web10...
> File queryview/sql/ndt-tmpl-web100.sql (right):
> 
>
https://codereview.appspot.com/9763044/diff/6001/queryview/sql/ndt-tmpl-web10...
> queryview/sql/ndt-tmpl-web100.sql:19: AND
> IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
> so can you now use this template instead of the minrtt one? if so, maybe
there's
> no need to have them all in. :)

Yeah, I ended up pruning the sql files.

Cheers.