code review 5532076: go/scanner: 17% faster scanning

go/scanner: 17% faster scanning

- Changed the Scan API semantics slightly:
The token literal string is only returned
if the token is a literal, comment, semicolon,
or illegal character. In all other cases, the
token literal value is determined by the token
value.

Clients that care about the token literal value
when not present can always use the following
piece of code:

pos, tok, lit := scanner.Scan()
if lit == "" {
   lit = tok.String()
}

- Changed token.Lookup API to use a string instead
of a []byte argument.

- Both these changes were long-standing TODOs.

- Added BenchmarkScan.

This change permits a faster implementation of Scan
with much fewer string creations:

benchmark                old ns/op    new ns/op    delta
scanner.BenchmarkScan        74404        61457  -17.40%

[gri, 2012-01-11 22:10:42]

Hello golang-dev@googlegroups.com,

I'd like you to review this change to
https://go.googlecode.com/hg/

[rsc, 2012-01-11 22:14:59]

LGTM

[gri, 2012-01-11 22:20:37]

*** Submitted as http://code.google.com/p/go/source/detail?r=af2cb1911e0e ***

go/scanner: 17% faster scanning

- Changed the Scan API semantics slightly:
The token literal string is only returned
if the token is a literal, comment, semicolon,
or illegal character. In all other cases, the
token literal value is determined by the token
value.

Clients that care about the token literal value
when not present can always use the following
piece of code:

pos, tok, lit := scanner.Scan()
if lit == "" {
   lit = tok.String()
}

- Changed token.Lookup API to use a string instead
of a []byte argument.

- Both these changes were long-standing TODOs.

- Added BenchmarkScan.

This change permits a faster implementation of Scan
with much fewer string creations:

benchmark                old ns/op    new ns/op    delta
scanner.BenchmarkScan        74404        61457  -17.40%

R=golang-dev, rsc
CC=golang-dev
http://codereview.appspot.com/5532076