code review 6345044: cmd/gc: reduce memory consumption

cmd/gc: reduce memory consumption

This change introduces a new struct FuncNode and moves fields from
Node to FuncNode. Node and FuncNode are logically the same object,
thus for each FuncNode it holds true that (n->funcnode->node == n)
and (aFuncNode->node->funcnode == aFuncNode).

The fact that Node and FuncNode are logically the same object means
that the semantics of the original C code remains unchanged.
Reduced memory consumption is achieved because FuncNode is allocated
only if it is needed.

The new function funcnode() will allocate a FuncNode on demand. It
is recommended to always call this function when going from a Node to
to its FuncNode, unless circumstances are against this recommendation.

Similarly, the new struct UncommonNode contains fields of Node which
are specific to a small number of Node instances. The accessor function
unode() will allocate an UncommonNode on demand.

Although this patch affects y.tab.h and y.tab.c, these two files are
omitted from the patch.

[atom, 2012-06-26 21:23:58]

Hello dave@cheney.net, rsc@golang.org (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
https://go.googlecode.com/hg/

[bradfitz, 2012-06-26 21:39:31]

Numbers?

On Tue, Jun 26, 2012 at 2:23 PM, <0xE2.0x9A.0x9B@gmail.com> wrote:

> Reviewers: dfc, rsc,
>
> Message:
> Hello dave@cheney.net, rsc@golang.org (cc: golang-dev@googlegroups.com),
>
> I'd like you to review this change to
> https://go.googlecode.com/hg/
>
>
> Description:
> cmd/gc: reduce memory consumption
>
> This change introduces a new struct FuncNode and moves fields from
> Node to FuncNode. Node and FuncNode are logically the same object,
> thus for each FuncNode it holds true that (n->funcnode->node == n)
> and (aFuncNode->node->funcnode == aFuncNode).
>
> The fact that Node and FuncNode are logically the same object means
> that the semantics of the original C code remains unchanged.
> Reduced memory consumption is achieved because FuncNode is allocated
> only if it is needed.
>
> The new function funcnode() will allocate a FuncNode on demand. It
> is recommended to always call this function when going from a Node to
> to its FuncNode, unless circumstances are against this recommendation.
>
> In addition to the above, this change uses 1-bit fields for boolean
> variables in struct Node.
>
> Although this patch affects y.tab.h and y.tab.c, these two files are
> omitted from the patch.
>
> Please review this at
http://codereview.appspot.com/**6345044/<http://codereview.appspot.com/6345044/>
>
> Affected files:
>  M src/cmd/5g/gg.h
>  M src/cmd/5g/ggen.c
>  M src/cmd/5g/reg.c
>  M src/cmd/6g/gg.h
>  M src/cmd/6g/ggen.c
>  M src/cmd/6g/reg.c
>  M src/cmd/8g/gg.h
>  M src/cmd/8g/ggen.c
>  M src/cmd/8g/reg.c
>  M src/cmd/gc/closure.c
>  M src/cmd/gc/dcl.c
>  M src/cmd/gc/esc.c
>  M src/cmd/gc/export.c
>  M src/cmd/gc/fmt.c
>  M src/cmd/gc/gen.c
>  M src/cmd/gc/go.h
>  M src/cmd/gc/go.y
>  M src/cmd/gc/init.c
>  M src/cmd/gc/inl.c
>  M src/cmd/gc/lex.c
>  M src/cmd/gc/order.c
>  M src/cmd/gc/pgen.c
>  M src/cmd/gc/sinit.c
>  M src/cmd/gc/subr.c
>  M src/cmd/gc/swt.c
>  M src/cmd/gc/typecheck.c
>  M src/cmd/gc/walk.c
>
>
>

[0xe2.0x9a.0x9b_gmail.com, 2012-06-26 21:43:42]

On Tuesday, June 26, 2012 11:39:30 PM UTC+2, Brad Fitzpatrick wrote:
>
> Numbers?
>
 
10% reduction (from 110MB to 100MB) when compiling godoc with 8g.

[lvd, 2012-06-27 08:51:05]

is gc memory usage a problem?

On Tue, Jun 26, 2012 at 11:43 PM, ⚛ <0xe2.0x9a.0x9b@gmail.com> wrote:

> On Tuesday, June 26, 2012 11:39:30 PM UTC+2, Brad Fitzpatrick wrote:
>>
>> Numbers?
>>
>
> 10% reduction (from 110MB to 100MB) when compiling godoc with 8g.
>

[dfc, 2012-06-27 09:00:12]

It can be, I can't run a go builder reliably on my ARMv5 host any more, which is
a shame. The folks coming up with Pi systems are running 256mb or less. 

On 27/06/2012, at 18:51, Luuk van Dijk <lvd@google.com> wrote:

> is gc memory usage a problem?
> 
> On Tue, Jun 26, 2012 at 11:43 PM, ⚛ <0xe2.0x9a.0x9b@gmail.com> wrote:
> On Tuesday, June 26, 2012 11:39:30 PM UTC+2, Brad Fitzpatrick wrote:
> Numbers?
>  
> 10% reduction (from 110MB to 100MB) when compiling godoc with 8g.
> 

[rsc, 2012-06-27 13:42:09]

I already said that I would like to have a discussion about memory
reduction strategies when I am back in a few weeks. I do not plan to
look at this CL before having that discussion.

[dfc, 2012-10-11 00:33:30]

On 2012/06/27 13:42:09, rsc wrote:
> I already said that I would like to have a discussion about memory
> reduction strategies when I am back in a few weeks. I do not plan to
> look at this CL before having that discussion.

Hello,

Would it be possible to separate the changes to the 1bit fields into an
independent CL? 

Cheers

Dave

[atom, 2012-10-12 17:00:10]

On 2012/10/11 00:33:30, dfc wrote:
> On 2012/06/27 13:42:09, rsc wrote:
> > I already said that I would like to have a discussion about memory
> > reduction strategies when I am back in a few weeks. I do not plan to
> > look at this CL before having that discussion.
> 
> Hello,
> 
> Would it be possible to separate the changes to the 1bit fields into an
> independent CL?

http://codereview.appspot.com/6650054

[dfc, 2012-10-13 00:32:03]

Thank you.

On Sat, Oct 13, 2012 at 4:00 AM,  <0xE2.0x9A.0x9B@gmail.com> wrote:
> On 2012/10/11 00:33:30, dfc wrote:
>>
>> On 2012/06/27 13:42:09, rsc wrote:
>> > I already said that I would like to have a discussion about memory
>> > reduction strategies when I am back in a few weeks. I do not plan to
>> > look at this CL before having that discussion.
>
>
>> Hello,
>
>
>> Would it be possible to separate the changes to the 1bit fields into
>
> an
>>
>> independent CL?
>
>
> http://codereview.appspot.com/6650054
>
> https://codereview.appspot.com/6345044/

[atom, 2012-10-15 13:55:46]

Hello dave@cheney.net, rsc@golang.org, bradfitz@golang.org, lvd@google.com (cc:
golang-dev@googlegroups.com),

Please take another look.

[rsc, 2012-11-01 18:44:19]

I am still interested in this CL, but I would like to see a memory
profile showing what memory usage looks like before we start fixing
things. One way to get a memory profile would be to hook it up on
Linux to the C++ pprof (http://code.google.com/p/gperftools).

[atom, 2012-11-03 09:25:48]

Hello dave@cheney.net, rsc@golang.org, bradfitz@golang.org, lvd@google.com (cc:
golang-dev@googlegroups.com),

Please take another look.

[atom, 2012-11-03 09:44:56]

On 2012/11/01 18:44:19, rsc wrote:
> I am still interested in this CL, but I would like to see a memory
> profile showing what memory usage looks like before we start fixing
> things. One way to get a memory profile would be to hook it up on
> Linux to the C++ pprof (http://code.google.com/p/gperftools).

I used Valgrind. When compiling godoc with 8g, the call counts are:

nod(): 357782 calls, 357782 calls to mal()
funcnode(): 21654 calls, 17648 mal()
unode(): 4470 calls, 3690 mal()

Sizes in bytes (8g):

sizeof(Node) = 180 (old size: 216)
sizeof(FuncNode) = 40
sizeof(UncommonNode) = 20

The theoretical difference in memory usage when compiling godoc is approximately
340000*(216-180) = 12MiB. Actual difference as measured by /usr/bin/time is
10MiB (=131-121).

---

Note: A new struct UncommonNode and function unode() has been added to the
changeset. Maybe we could move additional uncommon fields from Node to
UncommonNode before the decision to merge this changeset into Go is made.