code review 105990045: testing: make benchmarking faster

testing: make benchmarking faster

The number of estimated iterations required to reach the benchtime is multiplied by a safety margin (to avoid falling just short) and then rounded up to a readable number. With an accurate estimate, in the worse case, the resulting number of iterations could be 3.75x more than necessary: 1.5x for safety * 2.5x to round up (e.g. from 2eX+1 to 5eX).

This CL reduces the safety margin to 1.2x. Experimentation showed a diminishing margin of return past 1.2x, although the average case continued to show improvements down to 1.05x.

This CL also reduces the maximum round-up multiplier from 2.5x (from 2eX+1 to 5eX) to 2x, by allowing the number of iterations to be of the form 3eX.

Both changes improve benchmark wall clock times, and the effects are cumulative.

From 1.5x to 1.2x safety margin:

package		old s	new s	delta
bytes		163	125	-23%
encoding/json	27	21	-22%
net/http	42	36	-14%
runtime		463	418	-10%
strings		82	65	-21%

Allowing 3eX iterations:

package		old s	new s	delta
bytes		163	134	-18%
encoding/json	27	23	-15%
net/http	42	36	-14%
runtime		463	422	-9%
strings		82	72	-12%

Combined:

package		old s	new s	delta
bytes		163	112	-31%
encoding/json	27	20	-26%
net/http	42	30	-29%
runtime		463	346	-25%
strings		82	60	-27%

[josharian, 2014-06-13 21:53:01]

Hello golang-codereviews@googlegroups.com (cc: crawshaw@golang.org,
rsc@golang.org),

I'd like you to review this change to
https://code.google.com/p/go

[crawshaw, 2014-06-14 03:15:18]

LGTM but wait for rsc, as I have already misunderstood this code once.

[r, 2014-06-23 22:13:50]

LGTM
no need to wait for rsc

[rsc, 2014-06-23 23:00:31]

LGTM

Thanks for working through the rationale. We'll see what happens with the
builders...

[josharian, 2014-06-24 15:39:42]

*** Submitted as https://code.google.com/p/go/source/detail?r=829d3df73b1a ***

testing: make benchmarking faster

The number of estimated iterations required to reach the benchtime is multiplied
by a safety margin (to avoid falling just short) and then rounded up to a
readable number. With an accurate estimate, in the worse case, the resulting
number of iterations could be 3.75x more than necessary: 1.5x for safety * 2.5x
to round up (e.g. from 2eX+1 to 5eX).

This CL reduces the safety margin to 1.2x. Experimentation showed a diminishing
margin of return past 1.2x, although the average case continued to show
improvements down to 1.05x.

This CL also reduces the maximum round-up multiplier from 2.5x (from 2eX+1 to
5eX) to 2x, by allowing the number of iterations to be of the form 3eX.

Both changes improve benchmark wall clock times, and the effects are cumulative.

From 1.5x to 1.2x safety margin:

package		old s	new s	delta
bytes		163	125	-23%
encoding/json	27	21	-22%
net/http	42	36	-14%
runtime		463	418	-10%
strings		82	65	-21%

Allowing 3eX iterations:

package		old s	new s	delta
bytes		163	134	-18%
encoding/json	27	23	-15%
net/http	42	36	-14%
runtime		463	422	-9%
strings		82	72	-12%

Combined:

package		old s	new s	delta
bytes		163	112	-31%
encoding/json	27	20	-26%
net/http	42	30	-29%
runtime		463	346	-25%
strings		82	60	-27%

LGTM=crawshaw, r, rsc
R=golang-codereviews, crawshaw, r, rsc
CC=golang-codereviews
https://codereview.appspot.com/105990045