encoding/pem: split line based logic into another package.

encoding/pem: split line based logic into another package.

I needed the lineWriter for OpenPGP's PEM-but-not-quite format. Since
the OpenPGP code is streaming based, I also needed a line reader so I
put that in there too.

[agl1, 2010-10-16 17:31:02]

(I looked around for a LineReader already in pkg/, but there don't seem to be
any exported.)

[r2, 2010-10-16 18:12:54]

The LineBreaker will be nice to have.  I'm uncertain this is the right home for
it, although I don't have a better idea.  The tabwriter package is something of
a model for how it should work and be named; perhaps they should also cohabit a
subdirectory.

LineBreaker as written uses bytes to define length.  That is parochial.  Is that
behavior an issue for PEM?

-rob

[agl1, 2010-10-16 18:44:14]

On Sat, Oct 16, 2010 at 2:12 PM, Rob 'Commander' Pike <r@google.com> wrote:
> The LineBreaker will be nice to have.  I'm uncertain this is the right home
for it, although I don't have a better idea.  The tabwriter package is something
of a model for how it should work and be named; perhaps they should also cohabit
a subdirectory.

In this case I picked encoding because, for the uses that we currently
have, it really is an encoding. It's a reversible transformation
needed to move data over some channel. In this case, email.

As for the naming, are you suggesting that the structs should be
lines.Reader and lines.Breaker? Happy to do that. Probably would have
had I not biased myself by starting with something called
"lineWriter".

> LineBreaker as written uses bytes to define length.  That is parochial.  Is
that behavior an issue for PEM?

Yea, I paused when I did that. In the case of PEM, the input is
base64-encoded, so all codepoints are single bytes. I looked around
for how to do it in a Unicode-friendly fashion and found
http://unicode.org/reports/tr14/, which scared me off.

I think encoding.lines is not the same as, say, text.lines. The latter
would be Unicode aware, break on word boundaries, optionally
hyphenate. I see encoding.lines as being about surviving channels
which add line breaks.


AGL

[r2, 2010-10-16 19:03:38]

On Oct 16, 2010, at 11:44 AM, Adam Langley wrote:

> On Sat, Oct 16, 2010 at 2:12 PM, Rob 'Commander' Pike <r@google.com> wrote:
>> The LineBreaker will be nice to have.  I'm uncertain this is the right home
for it, although I don't have a better idea.  The tabwriter package is something
of a model for how it should work and be named; perhaps they should also cohabit
a subdirectory.
> 
> In this case I picked encoding because, for the uses that we currently
> have, it really is an encoding. It's a reversible transformation
> needed to move data over some channel. In this case, email.
> 
> As for the naming, are you suggesting that the structs should be
> lines.Reader and lines.Breaker? Happy to do that. Probably would have
> had I not biased myself by starting with something called
> "lineWriter".

Yes. This package is like bufio.Reader: give it a Reader, get a Reader back.

>> LineBreaker as written uses bytes to define length.  That is parochial.  Is
that behavior an issue for PEM?
> 
> Yea, I paused when I did that. In the case of PEM, the input is
> base64-encoded, so all codepoints are single bytes. I looked around
> for how to do it in a Unicode-friendly fashion and found
> http://unicode.org/reports/tr14/, which scared me off.

All I'm asking is to count code points rather than bytes, although that is also
naive since some characters, such as Asian ideographs, are always wider than,
say, é, and the issue of combining characters is completely absent.  To do that
stuff right, though, is a graphics/font question combined with Unicode
normalization.  Both those topics are in our future.

> I think encoding.lines is not the same as, say, text.lines. The latter
> would be Unicode aware, break on word boundaries, optionally
> hyphenate. I see encoding.lines as being about surviving channels
> which add line breaks.

My question is: if you were to use code points rather than bytes to determine
line length, wouldn't this work just fine for your purposes?  Then at least most
major languages would get consistent line lengths (aside from combining
characters, which are relatively rare) with this code and it could be more
widely used.   Failing that, this code should have a big comment about being
ASCII only, with narrow intent.

-rob

[agl1, 2010-10-17 13:22:34]

On Sat, Oct 16, 2010 at 2:57 PM, Rob 'Commander' Pike <r@google.com> wrote:
> My question is: if you were to use code points rather than bytes to determine
line length, wouldn't this work just fine for your purposes?  Then at least most
major languages would get consistent line lengths (aside from combining
characters, which are relatively rare) with this code and it could be more
widely used.   Failing that, this code should have a big comment about being
ASCII only, with narrow intent.

Yep, I can count code points just as well. Let me rework this CL and
I'll PTAL when I'm ready.

Thanks


AGL

[rsc, 2010-10-18 15:48:13]

<bikeshed>

I'd prefer to call this package line, because most
package names are singular, and then the names
can be the less repetitive

line.Reader
line.Writer
line.NewReader
line.NewWriter

</bikeshed>

</russ>