isolate.py: split commands hashtable and archive so each have different behavior

isolate.py: split commands hashtable and archive so each have different behavior

hashtable creates an hashtable in a directory. This is useful for archiving
files on a NFS/CIFS shared drive. It is a very simple way to share files.

archive uses the isolate server to upload the data. It is an active cache, so
older files will be deleted.

Add --isolate-server and --namespace to command archive for coherency with the
other tools.

Reduce --outdir to only the commands it makes sense.
Stop using a default value for --outdir for the command remap, it didn't make
sense.

R=vadimsh@chromium.org
BUG=

Committed: https://code.google.com/p/swarming/source/detail?repo=client&r=f9538ee

[M-A, 2014-01-24 16:47:34]

This will be tricky to roll, since --outdir becomes --isolate-server. This means
the canary will be broken until this is rolled into src/DEPS. It also depends on
https://codereview.chromium.org/145553004 and the masters to be restarted.

[M-A, 2014-01-24 21:03:37]

Rietveld will eventually be consistent so replying over email;

2014/1/24 <vadimsh@chromium.org>

> lgtm with questions
>

Anyway I can't commit until the masters are restarted.



> https://codereview.appspot.com/52050044/diff/50004/isolate.py#newcode1088
> isolate.py:1088: if not success and os.path.isfile(options.isolated):
> Hm.. I never seen this. Pretty aggressive behavior if 'isolate.py
> archive' is used manually by developer.
>

It was like this before. The rationale is:
- .isolated.state is not deleted, so no performance optimisation is lost.
- This makes the sha1 not show up in swarm_hashes, which is important for
https://codereview.chromium.org/139343011, so that the corresponding step
is not (!) triggered on swarm since it was not archived properly.



> https://codereview.appspot.com/52050044/diff/50004/isolate.py#newcode1144
> isolate.py:1144: if not success and os.path.isfile(options.isolated):
> I don't think it's necessary here?.. Swarm is not running on top of
> locally mapped files.
>

Same as above.



>  https://codereview.appspot.com/52050044/diff/50004/isolate.py#newcode1146
> isolate.py:1146: return not success
> return int(not success)? It's a process exit code... I don't want to
> think too hard about meaning of sys.exit(True)... Same in CMDarchive.
>

It works, but done.


>
> https://codereview.appspot.com/52050044/diff/50004/isolate.py#newcode1212
> isolate.py:1212: print('Remapping into %s' % options.outdir)
> Is it a fancy remapping with symlinks? What will happen if |outdir| is
> on another partition? I'm asking because 'run' now uses
> run_isolated.make_temp_dir that does check partitions.
>

Remap doesn't use make_temp_dir(). So in that case it will do normal file
copy.

M-A


> https://codereview.appspot.com/52050044/
>

[vadimsh, 2014-01-24 22:36:30]

Oh, I've just noticed that my comments are still marked as Drafts. Rietvled
sends email first (it even sent two identical emails!), and then modifies the
datastore. Cool...

[vadimsh, 2014-01-24 22:36:42]

On 2014/01/24 22:36:30, vadimsh wrote:
> Oh, I've just noticed that my comments are still marked as Drafts. Rietvled
> sends email first (it even sent two identical emails!), and then modifies the
> datastore. Cool...

[vadimsh, 2014-01-24 22:36:52]

On 2014/01/24 22:36:42, vadimsh wrote:
> On 2014/01/24 22:36:30, vadimsh wrote:
> > Oh, I've just noticed that my comments are still marked as Drafts. Rietvled
> > sends email first (it even sent two identical emails!), and then modifies
the
> > datastore. Cool...

And lgtm.

[M-A, 2014-01-30 15:44:03]

Committed patchset #5 manually as rf9538ee (presubmit successful).