Support streaming record length-encoded messages.

This commit introduces support for streaming from io.Reader and
streaming to io.Writer with messages that have been prefixed with
a varint-encoded length delimiter that indicates the size of the
subsequent encoded message body payload length.  There are many
applications of this practice, but binary logging is one of the most
notable ones.

Overall the practice described in http://goo.gl/9863Z has been
applied in a manner idiomatic to Go per the C++/Java reference
implementations of writeDelimitedTo and mergeDelimitedFrom.

[albert.strasheim, 2013-11-12 21:25:56]

Any interest in getting this patch merged? It would definitely help us.

Cheers

Albert

[dsymonds, 2013-11-12 21:41:01]

We aren't interested in merging this into goprotobuf. Protocol buffers are not
self-delimiting, and this CL is implementing a record format above protocol
buffers, which is fine but not really needed in goprotobuf.

[mtp, 2013-11-12 22:34:05]

On 2013/11/12 21:41:01, dsymonds wrote:
> We aren't interested in merging this into goprotobuf. Protocol buffers are not
> self-delimiting, and this CL is implementing a record format above protocol
> buffers, which is fine but not really needed in goprotobuf.

David, I do want to respectfully register my dissent in this, for both the C++
and Java reference implementations offer this functionality as first-class
feature.  Moreover, it is a useful one, too.

I have taken this merge proposal and hosted it in a separate go-get-able
repository for those who are interested:
http://godoc.org/github.com/matttproud/golang_protobuf_extensions/ext.  Albert,
consider using it from there.  :-)

[dsymonds, 2013-11-12 22:55:59]

Sure, but there's a lot of stuff in the C++ and Java implementations
that the Go implementation doesn't provide. I think that's okay.

[shanemhansen, 2013-11-28 05:52:28]

On 2013/11/12 22:55:59, dsymonds wrote:
> Sure, but there's a lot of stuff in the C++ and Java implementations
> that the Go implementation doesn't provide. I think that's okay.

Having one extra helper function is awesome, and it's opt in. It doesn't break
compatibility.

[dsymonds, 2013-11-28 05:54:41]

That's not my point. My point is that everything is "useful", but if
that's the defining criteria then you'll get a big bloated package
with an API that's hard to understand, and I'm drawing the line in a
way that excludes record support. My comment about the C++ and Java
implementations was mentioning that feature parity with those aren't
an important goal for goprotobuf.

[awalterschulze, 2014-03-14 08:47:55]

On 2013/11/28 05:54:41, dsymonds wrote:
> That's not my point. My point is that everything is "useful", but if
> that's the defining criteria then you'll get a big bloated package
> with an API that's hard to understand, and I'm drawing the line in a
> way that excludes record support. My comment about the C++ and Java
> implementations was mentioning that feature parity with those aren't
> an important goal for goprotobuf.

Something similar is in gogoprotobuf now.
http://code.google.com/p/gogoprotobuf/source/browse/#git%2Fio

gogoprotobuf: also known as the bloated goprotobuf ;)