Add mapreduce framework to default branch.

This moves the mapreduce integration to the default branch.

The chromium branch uses already some mapreduce jobs. Since it is useful in default too this change adds the mapreduce integration to default.

To make merging easier I'd propose that the chromium specific jobs go into a separate Python file (for example admin_tasks_chromium.py). That would mean that we only need to merge mapreduce.yaml where all available jobs are configured.

[M-A, 2013-07-12 14:39:06]

https://codereview.appspot.com/11202043/diff/1/Makefile
File Makefile (right):

https://codereview.appspot.com/11202043/diff/1/Makefile#newcode91
Makefile:91: svn export
http://appengine-mapreduce.googlecode.com/svn/trunk/python/src/mapreduce
I'd recommend hardcoding a revision to be on the safe side. Currently -r 491.

I'd also prefer a proper checkout than an export.

https://codereview.appspot.com/11202043/diff/1/app.yaml
File app.yaml (right):

https://codereview.appspot.com/11202043/diff/1/app.yaml#newcode1
app.yaml:1: application: rvtests-hr
Ah?

[Andi, 2013-07-12 16:21:52]

Pin revision

[Andi, 2013-07-12 16:26:57]

PTAL

https://codereview.appspot.com/11202043/diff/1/Makefile
File Makefile (right):

https://codereview.appspot.com/11202043/diff/1/Makefile#newcode91
Makefile:91: svn export
http://appengine-mapreduce.googlecode.com/svn/trunk/python/src/mapreduce
On 2013/07/12 14:39:06, M-A wrote:
> I'd recommend hardcoding a revision to be on the safe side. Currently -r 491.
> 
> I'd also prefer a proper checkout than an export.

Done.

https://codereview.appspot.com/11202043/diff/1/app.yaml
File app.yaml (right):

https://codereview.appspot.com/11202043/diff/1/app.yaml#newcode1
app.yaml:1: application: rvtests-hr
On 2013/07/12 14:39:06, M-A wrote:
> Ah?

oh, that was just a gentle attempt to prove you that I've actually tested my
change :)

[M-A, 2013-07-12 16:59:00]

https://codereview.appspot.com/11202043/diff/7001/Makefile
File Makefile (right):

https://codereview.appspot.com/11202043/diff/7001/Makefile#newcode90
Makefile:90: mapreduce:
you should have it depend on a file in the checkout so it doesn't fail the
second time

[Andi, 2013-07-12 20:05:15]

Am Freitag, 12. Juli 2013 schrieb :

>
>
https://codereview.appspot.**com/11202043/diff/7001/**Makefile<https://codere...
> File Makefile (right):
>
>
https://codereview.appspot.**com/11202043/diff/7001/**Makefile#newcode90<http...
> Makefile:90: mapreduce:
> you should have it depend on a file in the checkout so it doesn't fail
> the second time


You're right. The intention was that instance maintainers can just run make
upgrade and developers can decide to run svn up and change the revision
number in the Makefile then.
But this doesn't solve the problem when instance maintainers should update
to the latest approved revision.

I'll upload a new CL later.


>
>
https://codereview.appspot.**com/11202043/<https://codereview.appspot.com/112...
>

[iannucci, 2013-07-12 22:36:54]

Isn't this patch missing mapreduce.yaml?

[Andi, 2013-07-13 08:42:22]

Add mapreduce.yaml, mapreduce.patch, correct hgignore

[Andi, 2013-07-13 08:44:12]

On Sat, Jul 13, 2013 at 12:36 AM,  <iannucci@chromium.org> wrote:
> Isn't this patch missing mapreduce.yaml?

Thanks! The hgignore regex for the mapreduce directory was wrong.
mapreduce.yaml and mapreduce.patch are added now.

>
> https://codereview.appspot.com/11202043/

[Andi, 2013-07-13 08:50:02]

https://codereview.appspot.com/11202043/diff/7001/Makefile
File Makefile (right):

https://codereview.appspot.com/11202043/diff/7001/Makefile#newcode90
Makefile:90: mapreduce:
On 2013/07/12 16:59:00, M-A wrote:
> you should have it depend on a file in the checkout so it doesn't fail the
> second time

How can it fail the second time? When the mapreduce directory exists nothing
should happen when "make mapreduce" is called. Do I miss something?

[M-A, 2013-07-13 13:57:05]

On 2013/07/13 08:50:02, Andi wrote:
> https://codereview.appspot.com/11202043/diff/7001/Makefile
> File Makefile (right):
> 
> https://codereview.appspot.com/11202043/diff/7001/Makefile#newcode90
> Makefile:90: mapreduce:
> On 2013/07/12 16:59:00, M-A wrote:
> > you should have it depend on a file in the checkout so it doesn't fail the
> > second time
> 
> How can it fail the second time? When the mapreduce directory exists nothing
> should happen when "make mapreduce" is called. Do I miss something?

No, I was confused. I wish there was a way in make to say "rerun this step if
svn revision for this directory != $X or svn status is not empty".

lgtm

[Andi, 2013-07-13 18:31:15]

On 2013/07/13 13:57:05, M-A wrote:
> On 2013/07/13 08:50:02, Andi wrote:
> > https://codereview.appspot.com/11202043/diff/7001/Makefile
> > File Makefile (right):
> > 
> > https://codereview.appspot.com/11202043/diff/7001/Makefile#newcode90
> > Makefile:90: mapreduce:
> > On 2013/07/12 16:59:00, M-A wrote:
> > > you should have it depend on a file in the checkout so it doesn't fail the
> > > second time
> > 
> > How can it fail the second time? When the mapreduce directory exists nothing
> > should happen when "make mapreduce" is called. Do I miss something?
> 
> No, I was confused. I wish there was a way in make to say "rerun this step if
> svn revision for this directory != $X or svn status is not empty".
> 
> lgtm

Committed as 97b385fffa35

Thanks for the review!

I hope merging this back to chromium isn't too painful.