code review 11565043: go.text/encoding: add UTF-16 encodings.

go.text/encoding: add UTF-16 encodings.

There are some TODOs concerning the exact behavior for bad UTF-16, but
I'll address those after getting consensus on the broad-brush design.

candide-utf-16le.txt was generated by
iconv -f UTF-8 -t UTF-16LE < candide-utf-8.txt > candide-utf-16le.txt

[nigeltao, 2013-07-19 06:12:36]

Hello r@golang.org (cc: andybalholm@gmail.com, golang-dev@googlegroups.com,
mpvl@golang.org, rogpeppe@gmail.com),

I'd like you to review this change to
https://code.google.com/p/go.text

[r, 2013-07-19 06:23:16]

https://codereview.appspot.com/11565043/diff/5001/encoding/encoding_test.go
File encoding/encoding_test.go (right):

https://codereview.appspot.com/11565043/diff/5001/encoding/encoding_test.go#n...
encoding/encoding_test.go:279: // - inputs with multiple U+FEFF and U+FFFE
runes. Should occurences other
occurrences

https://codereview.appspot.com/11565043/diff/5001/encoding/encoding_test.go#n...
encoding/encoding_test.go:280: //   than at the start be converted to U+FFFD?
no

https://codereview.appspot.com/11565043/diff/5001/encoding/encoding_test.go#n...
encoding/encoding_test.go:283: //   return ErrShortSrc?
U+FFFD

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go
File encoding/utf16.go (right):

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode20
encoding/utf16.go:20: // the endianness used for decoding, and will instead be
output as UTF-8:
s/UTF-8/their standard UTF-8 encodings/

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode30
encoding/utf16.go:30: // the stream if the BOMPolicy is ExpectBOM. Otherwise, a
BOM will not be
s/stream/output/ or /output &/

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode60
encoding/utf16.go:60: // BigEndian is the big-endian form of UTF-16.
these have official names, UTF-16LE and UTF-16BE i think

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode76
encoding/utf16.go:76: return &utf16Decoder{endianness: u.endianness, bomPolicy:
u.bomPolicy}
it's not superimportant but i would check that the policy and endianness are
valid.
i would also consider making them boolean rather than integers just so they can
never be invalid and i can avoid the check.

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode91
encoding/utf16.go:91: return "UTF16(" + e + " Endian, " + b + " BOM)"
maybe UTF-16LE etc.

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode95
encoding/utf16.go:95: // TODO: does these need to be explicitly or implicitly
reset, if this
s/does/do/

https://codereview.appspot.com/11565043/diff/5001/encoding/utf16.go#newcode186
encoding/utf16.go:186: // If the previous input was invalid UTF-8, skip any
continuation bytes.
overkill. see what unicode/utf8 does

[nigeltao, 2013-07-19 07:05:45]

Trivial edits were done.

I'm leaning in favor of removing the UTF-8 synchronization code, but I'll let it
stew overnight (and while I'm on vacation).

[nigeltao, 2013-07-29 05:36:19]

PTAL.

I will remove the UTF-8 synchronization code, but I'll do that in a follow-up
CL, as that affects the existing charmap encodings, not just the new UTF-16
encodings that this CL adds.

[r, 2013-07-30 01:46:38]

LGTM
then delete the UTF-8 sync code

[nigeltao, 2013-07-30 01:53:42]

*** Submitted as
https://code.google.com/p/go/source/detail?r=c80cf5c1f0a2&repo=text ***

go.text/encoding: add UTF-16 encodings.

There are some TODOs concerning the exact behavior for bad UTF-16, but
I'll address those after getting consensus on the broad-brush design.

candide-utf-16le.txt was generated by
iconv -f UTF-8 -t UTF-16LE < candide-utf-8.txt > candide-utf-16le.txt

R=r
CC=andybalholm, golang-dev, mpvl, rogpeppe
https://codereview.appspot.com/11565043