go.net/unicode/encoding: new package.

go.text/unicode/{encoding,transform}: new packages.

[rog, 2013-06-07 12:35:27]

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.go
File unicode/encoding/encoding.go (right):

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:39: // dst should always be large enough to write
at least one rune.
i don't understand this. unless you mean "will" instead of "should".

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:42: // there is not enough source bytes to decode
the next rune. For example,
s/there is/there are/

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:124: buf *[2][bufSize]byte
i find this (and the code that uses it) a little hard to read. i *think* it
would be easier to understand if it were two [bufSize]byte fields named "dst"
and "src".

any particular reason it's a pointer BTW? alignment concerns?

[andybalholm, 2013-06-07 17:47:34]

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.go
File unicode/encoding/encoding.go (right):

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:58: // encoding; the rune returned in this case
should be ignored.
If DecodeRune returns n == 0, how does the caller know how many bytes were
occupied by the escape sequence that caused the state transition?

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:70: // Encode converts UTF-8 bytes in src to
encoded runes in src. It returns
s/encoded runes in src/encoded runes in dst/

[mpvl, 2013-06-12 15:51:40]

some high-level comments only.

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.go
File unicode/encoding/encoding.go (right):

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:42: // there is not enough source bytes to decode
the next rune. For example,
This assumption is not correct for all encodings. For example, Regional
Indicator Symbols are represented as multiple codepoints in Unicode, whereas
they are represented as a single character in Shift-JIS, IIUC.  At least they
should be encoded as a single unit.  There is a proposal to augment this
standard such that encoders should interject a Zero-Width Joiner for consecutive
RISs.  If this were to happen, the minimum character size is 11 bytes in UTF-8!
There could also be encodings where accents need to be translated into modifiers
and the like. In these cases, the minimum buffer size might be larger.  I can't
think of an example, though.

https://codereview.appspot.com/10085049/diff/5001/unicode/encoding/encoding.g...
unicode/encoding/encoding.go:68: DecodeRune(p []byte) (r rune, n int, enc
Encoding)
Not all characters from a source encoding can necessarily be encoded in a single
destination rune.

https://codereview.appspot.com/10085049/diff/10001/unicode/transform/transfor...
File unicode/transform/transform.go (right):

https://codereview.appspot.com/10085049/diff/10001/unicode/transform/transfor...
unicode/transform/transform.go:5: package transform
I would put package transform directly under go.text.  Transform will likely be
used throughout go.text and it is not specific to unicode.

[nigeltao, 2013-07-18 04:15:04]

*** Abandoned ***