code review 5040041: index/suffixarray: support for serialization

index/suffixarray: support for serialization

[gri, 2011-09-15 23:13:38]

Hello r@golang.org (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
https://go.googlecode.com/hg/

[r, 2011-09-15 23:16:22]

LGTM

[gri, 2011-09-15 23:21:27]

*** Submitted as http://code.google.com/p/go/source/detail?r=0e01bf9b2856 ***

index/suffixarray: support for serialization

R=r
CC=golang-dev
http://codereview.appspot.com/5040041

[rog, 2011-09-16 00:55:30]

any particular reason not to use gob for the encoding?
it would save space.

On 16 September 2011 00:21,  <gri@golang.org> wrote:
> *** Submitted as
> http://code.google.com/p/go/source/detail?r=0e01bf9b2856 ***
>
> index/suffixarray: support for serialization
>
> R=r
> CC=golang-dev
> http://codereview.appspot.com/5040041
>
>
> http://codereview.appspot.com/5040041/
>

[gri, 2011-09-16 01:06:19]

It would save space on disk, but I'm not sure it would be faster or
use less memory when reading it in (both of which are the primary
concern here as this is primarily for godoc on app-engine use). The
godoc index file for the GOROOT tree is > 100MB.

I think encoding/binary also makes an extra copy before
reading/writing from/to disk - which is not ideal either.

But it's a good point; I should try using gobs and compare (will have
to wait until tomorrow).
- Robert

On Thu, Sep 15, 2011 at 5:55 PM, roger peppe <rogpeppe@gmail.com> wrote:
> any particular reason not to use gob for the encoding?
> it would save space.
>
> On 16 September 2011 00:21,  <gri@golang.org> wrote:
>> *** Submitted as
>> http://code.google.com/p/go/source/detail?r=0e01bf9b2856 ***
>>
>> index/suffixarray: support for serialization
>>
>> R=r
>> CC=golang-dev
>> http://codereview.appspot.com/5040041
>>
>>
>> http://codereview.appspot.com/5040041/
>>
>

[rog, 2011-09-16 01:19:03]

the other thing gob gives you is future-proofing - it's trivial
to upgrade to 64 bits or another format if needed.

On 16 September 2011 02:06, Robert Griesemer <gri@golang.org> wrote:
> It would save space on disk, but I'm not sure it would be faster or
> use less memory when reading it in (both of which are the primary
> concern here as this is primarily for godoc on app-engine use). The
> godoc index file for the GOROOT tree is > 100MB.
>
> I think encoding/binary also makes an extra copy before
> reading/writing from/to disk - which is not ideal either.
>
> But it's a good point; I should try using gobs and compare (will have
> to wait until tomorrow).
> - Robert
>
> On Thu, Sep 15, 2011 at 5:55 PM, roger peppe <rogpeppe@gmail.com> wrote:
>> any particular reason not to use gob for the encoding?
>> it would save space.
>>
>> On 16 September 2011 00:21,  <gri@golang.org> wrote:
>>> *** Submitted as
>>> http://code.google.com/p/go/source/detail?r=0e01bf9b2856 ***
>>>
>>> index/suffixarray: support for serialization
>>>
>>> R=r
>>> CC=golang-dev
>>> http://codereview.appspot.com/5040041
>>>
>>>
>>> http://codereview.appspot.com/5040041/
>>>
>>
>

[gri, 2011-09-19 23:08:10]

FYI: I've been playing with this. The gob encoded suffixarrays (of
sufficiently large size) happen to be larger than the corresponding
binary-encoded ones (because large 32bit int values tend to use 5
bytes instead of 4), but gob-encoding appears to be faster (more than
2x, from a first hand-stopped test). I suspect this is due to the fact
that binary encoding does an interface call (little vs big endian) for
each int32 it encodes, while gob's int encoding is fixed.

Anyway, a CL for a better code (that also does away with the huge
extra copy) is forthcoming once some gob issues are fixed.

- gri

On Thu, Sep 15, 2011 at 6:19 PM, roger peppe <rogpeppe@gmail.com> wrote:
> the other thing gob gives you is future-proofing - it's trivial
> to upgrade to 64 bits or another format if needed.
>
> On 16 September 2011 02:06, Robert Griesemer <gri@golang.org> wrote:
>> It would save space on disk, but I'm not sure it would be faster or
>> use less memory when reading it in (both of which are the primary
>> concern here as this is primarily for godoc on app-engine use). The
>> godoc index file for the GOROOT tree is > 100MB.
>>
>> I think encoding/binary also makes an extra copy before
>> reading/writing from/to disk - which is not ideal either.
>>
>> But it's a good point; I should try using gobs and compare (will have
>> to wait until tomorrow).
>> - Robert
>>
>> On Thu, Sep 15, 2011 at 5:55 PM, roger peppe <rogpeppe@gmail.com> wrote:
>>> any particular reason not to use gob for the encoding?
>>> it would save space.
>>>
>>> On 16 September 2011 00:21,  <gri@golang.org> wrote:
>>>> *** Submitted as
>>>> http://code.google.com/p/go/source/detail?r=0e01bf9b2856 ***
>>>>
>>>> index/suffixarray: support for serialization
>>>>
>>>> R=r
>>>> CC=golang-dev
>>>> http://codereview.appspot.com/5040041
>>>>
>>>>
>>>> http://codereview.appspot.com/5040041/
>>>>
>>>
>>
>