Add SpamBayes performance test

Here is a straightforward performance test using the SpamBayes tokenizer and its scoring machinery.  This is my first attempt at writing a performance test so I'm a bit unclear on how much priming of the pump is needed.

[Skip, 2009-12-20 00:25:01]

PTAL

[alex.gaynor, 2009-12-20 08:44:14]

A few things I spotted (warning 3 AM review).

http://codereview.appspot.com/179110/diff/1/1002
File performance/spambayes/bm_spambayes.py (right):

http://codereview.appspot.com/179110/diff/1/1002#newcode17
performance/spambayes/bm_spambayes.py:17:
sys.path.append("/Users/skip/src/unladen-swallow/tests/performance")
Don't use your own computers paths.

http://codereview.appspot.com/179110/diff/1/1002#newcode25
performance/spambayes/bm_spambayes.py:25: for _ in xrange(iterations):
You shouldn't need to run N times to prime it, one or two runs should usually be
enough.

[Skip, 2009-12-20 14:46:12]

> Don't use your own computers paths.

Understood.  I couldn't tell how to get the performance directory into sys.path.
 It
wasn't there automagically as far as I could tell.  Collin said he would clean
up problems,
so I await his verdict on this.

> You shouldn't need to run N times to prime it, one or two runs should usually
be
> enough.

That's what I thought, but running it more before starting the timing seemed to
improve
the relative performance of the u-s version.  How does u-s know a routine is
"hot" if you
only execute it once?

[Skip, 2009-12-20 16:29:00]

> > You shouldn't need to run N times to prime it, one or two runs should
usually be
> > enough.
> 
> That's what I thought, but running it more before starting the timing seemed
to
> improve the relative performance of the u-s version.  How does u-s know a
routine
> is "hot" if you only execute it once?

Here's what I mean.  I removed the first for loop after the "prime the pump"
comment.
Before:

    0.881673 -> 0.735389: 1.1989x faster

After:

    0.885434 -> 2.112219: 2.3855x slower

That suggests to me that the CPU-intensive functions aren't hot in the later
case until
well after the timed runs begin.

If I add back in the loop and simply adjust the number of iterations the greater
the
number of iterations the better the performance.

[Collin Winter, 2009-12-21 18:00:10]

I believe the hotness discrepancy is due to a bug at line 38 of bm_spambayes.py.
I'll fix that.

I'm pulling this into my working copy and will be committing it shortly. Thanks
for putting this together, Skip!

[Collin Winter, 2009-12-21 19:48:41]

Committed as http://code.google.com/p/unladen-swallow/source/detail?r=945

I reduced the amount of pump-priming because I think the the lower amount is a
fairer test.

Thanks, Skip!

[skip_pobox.com, 2009-12-22 00:43:08]

    Collin> I believe the hotness discrepancy is due to a bug at line 38 of
    Collin> bm_spambayes.py. I'll fix that.

Well, that was dumb.  It's not the whole story though.  After fixing that
bug here's my pump primer:

    for _ in xrange(N):
        for msg in msgs:
            ntokens = len(list(tokenize(msg)))
            score = h.score(msg)

where I simply change N to suit: 1, 5, iterations, etc.

If N == 1:

    Min: 0.854196 -> 0.699381: 1.2214x faster
    Avg: 0.860160 -> 0.920951: 1.0707x slower
    Not significant
    Stddev: 0.01476 -> 0.64805: 43.9011x larger

If N == 5:

    Min: 0.855203 -> 0.693278: 1.2336x faster
    Avg: 0.858933 -> 0.797368: 1.0772x faster
    Not significant
    Stddev: 0.00783 -> 0.23453: 29.9475x larger

If N == iterations:

    Min: 0.853942 -> 0.694206: 1.2301x faster
    Avg: 0.857288 -> 0.708853: 1.2094x faster
    Significant (t=22.599881, a=0.95)
    Stddev: 0.00296 -> 0.04635: 15.6742x larger

So my assertion still stands.  Executing the key SpamBayes functions once is
not sufficient to mark them "hot".

Skip