Migrate testing related third parties to /appengine/third_party_local/.

Migrate testing related third parties to /appengine/third_party_local/.

These third parties are not used nor uploaded to the AppEngine instances. They
were under /appengine/components/ but I think it's worth raising at higher
level. More importantly, the name is intentionally 'third_party_local' to make
it clear that these packages are never to be uploaded to AppEngine. This gets
rid of the confusion between components/third_party and
components/components/third_party.

Removed a few sys.path manipulation along the way and renamed appengine/README
to README.md.

Imported libraries will be migrated in a follow up.

R=vadimsh@chromium.org
BUG=

Committed: https://code.google.com/p/swarming/source/detail?repo=default&r=3e2046aec942d7c460aebbf36ab9ba9be198a96e

[vadimsh, 2015-03-24 02:23:31]

What will happen with appengine/components/components/third_party? The current
feature of components is that it requires only two changes to be usable from
other apps:
1) Create "components" symlink.
2) In main.py hack sys.path once to make components/third_party importable.

If we start using symlinks instead of step 2, then ALL apps that use components
(there are like 6 of them in chromium repos) would need to know what symlinks to
use exactly. e.g. if we add a new components that uses a new third party
library, we would need to add symlink to this library everywhere. I prefer
sys.path hack done once, rather than symlinks that needs to be updated
everywhere.

[M-A, 2015-03-24 23:21:44]

On 2015/03/24 02:23:31, vadimsh wrote:
> What will happen with appengine/components/components/third_party? The current
> feature of components is that it requires only two changes to be usable from
> other apps:
> 1) Create "components" symlink.
> 2) In main.py hack sys.path once to make components/third_party importable.
> 
> If we start using symlinks instead of step 2, then ALL apps that use
components
> (there are like 6 of them in chromium repos) would need to know what symlinks
to
> use exactly. e.g. if we add a new components that uses a new third party
> library, we would need to add symlink to this library everywhere. I prefer
> sys.path hack done once, rather than symlinks that needs to be updated
> everywhere.

I agree and in theory it sounds awful. In practice it's not that much, we use
only 4 third parties that are uploaded to GAE: cloudstorage, gviz, mapreduce and
protobuf. I don't see that as a big deal, frankly.

We could still keep components/components/third_party, that doesn't change much
this CL IMHO. The main gain is that now what's remote and what's local is clear.

[vadimsh, 2015-03-25 00:00:41]

On 2015/03/24 23:21:44, M-A wrote:
> On 2015/03/24 02:23:31, vadimsh wrote:
> > What will happen with appengine/components/components/third_party? The
current
> > feature of components is that it requires only two changes to be usable from
> > other apps:
> > 1) Create "components" symlink.
> > 2) In main.py hack sys.path once to make components/third_party importable.
> > 
> > If we start using symlinks instead of step 2, then ALL apps that use
> components
> > (there are like 6 of them in chromium repos) would need to know what
symlinks
> to
> > use exactly. e.g. if we add a new components that uses a new third party
> > library, we would need to add symlink to this library everywhere. I prefer
> > sys.path hack done once, rather than symlinks that needs to be updated
> > everywhere.
> 
> I agree and in theory it sounds awful. In practice it's not that much, we use
> only 4 third parties that are uploaded to GAE: cloudstorage, gviz, mapreduce
and
> protobuf. I don't see that as a big deal, frankly.
> 
> We could still keep components/components/third_party, that doesn't change
much
> this CL IMHO. The main gain is that now what's remote and what's local is
clear.
This CL lgtm (assuming it works, haven't looked very carefully).

I'm worried about components/components/third_party.

I'm considering instruction for using "components" from GAE apps. Right now they
are:
1) Add a symlink to components/
2) Copy paste two lines of sys.path hack.
3) Done. 

If we switch to symlinking of individual third parties, they'll look like:
1) Add a symlink to components/
2) Add a symlink to protobuf/, gviz/ ("wtf? why I need to do this").
3) We may ask you in the future to add symlinks to other random third party
packages that we happen to use from components/.

Perhaps we can do some other horrible things like modiying sys.path from inside
components/__init__.py instead? I do not want users of "components" to create
symlinks for essentially internal implementation details of components.

[M-A, 2015-03-25 00:59:39]

On 2015/03/25 00:00:41, vadimsh wrote:
> On 2015/03/24 23:21:44, M-A wrote:
> > On 2015/03/24 02:23:31, vadimsh wrote:
> > > What will happen with appengine/components/components/third_party? The
> current
> > > feature of components is that it requires only two changes to be usable
from
> > > other apps:
> > > 1) Create "components" symlink.
> > > 2) In main.py hack sys.path once to make components/third_party
importable.
> > > 
> > > If we start using symlinks instead of step 2, then ALL apps that use
> > components
> > > (there are like 6 of them in chromium repos) would need to know what
> symlinks
> > to
> > > use exactly. e.g. if we add a new components that uses a new third party
> > > library, we would need to add symlink to this library everywhere. I prefer
> > > sys.path hack done once, rather than symlinks that needs to be updated
> > > everywhere.
> > 
> > I agree and in theory it sounds awful. In practice it's not that much, we
use
> > only 4 third parties that are uploaded to GAE: cloudstorage, gviz, mapreduce
> and
> > protobuf. I don't see that as a big deal, frankly.
> > 
> > We could still keep components/components/third_party, that doesn't change
> much
> > this CL IMHO. The main gain is that now what's remote and what's local is
> clear.
> This CL lgtm (assuming it works, haven't looked very carefully).
> 
> I'm worried about components/components/third_party.
> 
> I'm considering instruction for using "components" from GAE apps. Right now
they
> are:
> 1) Add a symlink to components/
> 2) Copy paste two lines of sys.path hack.
> 3) Done. 
> 
> If we switch to symlinking of individual third parties, they'll look like:
> 1) Add a symlink to components/
> 2) Add a symlink to protobuf/, gviz/ ("wtf? why I need to do this").
> 3) We may ask you in the future to add symlinks to other random third party
> packages that we happen to use from components/.
> 
> Perhaps we can do some other horrible things like modiying sys.path from
inside
> components/__init__.py instead? I do not want users of "components" to create
> symlinks for essentially internal implementation details of components.

Let's discuss that sitting around a CL.

[M-A, 2015-03-25 01:03:57]

Committed patchset #1 (id:20001) manually as
3e2046aec942d7c460aebbf36ab9ba9be198a96e (presubmit successful).