code review 5490053: runtime: make more build-friendly

runtime: make more build-friendly

Collapse the arch,os-specific directories into the main directory
by renaming xxx/foo.c to foo_xxx.c, and so on.

There are no substantial edits here, except to the Makefile.
The assumption is that the Go tool will #define GOOS_darwin
and GOARCH_amd64 and will make any file named something
like signals_darwin.h available as signals_GOOS.h during the
build.  This replaces what used to be done with -I$(GOOS).

There is still work to be done to make runtime build with
standard tools, but this is a big step.  After this we will have
to write a script to generate all the generated files so they
can be checked in (instead of generated during the build).

[rsc, 2011-12-16 06:02:52]

Hello r (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
https://go.googlecode.com/hg/

[rsc, 2011-12-16 06:04:27]

I welcome better suggestions than the ugly arch.h etc
files, but if everything has to be in one directory
(and that is the convention we have established)
then I don't see any other way to make #include "arch.h"
do the right thing.

[r, 2011-12-16 06:08:27]

it might be nice if the new ugly foo.h files failed in place if no #ifdef was
satisfied instead of some later, mysterious place.

other people will have to do this sort of thing and it's O(N) at the best of
times. can the go command not do this for you? if there's a rule for _$GOOS.go,
why not _$GOOS.[ch]?

[iant2, 2011-12-16 06:26:43]

Russ Cox <rsc@golang.org> writes:

> I welcome better suggestions than the ugly arch.h etc
> files, but if everything has to be in one directory
> (and that is the convention we have established)
> then I don't see any other way to make #include "arch.h"
> do the right thing.

I can understand the constraint about putting everything into one
directory, but I'm somewhat concerned about not providing any way to
automatically generate a file which is then compiled.  Lots of code does
this for perfectly good reasons.  Can we think of a way to make that
work, like a comment at the start of the file which gives a command to
run?  Or is there a good reason not to do that?

Ian

[r2, 2011-12-16 06:45:18]

On Dec 15, 2011, at 10:26 PM, Ian Lance Taylor wrote:

> Russ Cox <rsc@golang.org> writes:
> 
>> I welcome better suggestions than the ugly arch.h etc
>> files, but if everything has to be in one directory
>> (and that is the convention we have established)
>> then I don't see any other way to make #include "arch.h"
>> do the right thing.
> 
> I can understand the constraint about putting everything into one
> directory, but I'm somewhat concerned about not providing any way to
> automatically generate a file which is then compiled.  Lots of code does
> this for perfectly good reasons.  Can we think of a way to make that
> work, like a comment at the start of the file which gives a command to
> run?  Or is there a good reason not to do that?

We've got to be careful this doesn't turn into make, but I'd like to understand
the plan better.

-rob

[rsc, 2011-12-16 12:24:00]

On Fri, Dec 16, 2011 at 01:08,  <r@golang.org> wrote:
> it might be nice if the new ugly foo.h files failed in place if no
> #ifdef was satisfied instead of some later, mysterious place.
>
> other people will have to do this sort of thing and it's O(N) at the
> best of times. can the go command not do this for you? if there's a rule
> for _$GOOS.go, why not _$GOOS.[ch]?

I believe those rules exist, as far as file selection is concerned,
but C is not as helpful as Go here.  In Go you can take a bunch
of files and hand them all to the compiler, and as long as
everything is there somewhere, the compiler figures it out.
In contrast, C needs every file to refer to the same #include
file by name so they agree on the declarations.  If that #include
file happens to be per-architecture, there is no good name.

We could define that the go tool takes every foo_$GOOS.h file
and copies it into the work directory as foo.h automatically,
but that would be a C-specific rule in a Go tool.  The ugliness
here contains the C rules in the runtime directory.

Russ

[rsc, 2011-12-16 13:04:44]

On Fri, Dec 16, 2011 at 01:26, Ian Lance Taylor <iant@google.com> wrote:
> I can understand the constraint about putting everything into one
> directory, but I'm somewhat concerned about not providing any way to
> automatically generate a file which is then compiled.  Lots of code does
> this for perfectly good reasons.  Can we think of a way to make that
> work, like a comment at the start of the file which gives a command to
> run?  Or is there a good reason not to do that?

In package runtime and in the few other packages that have generated
code, I expect that there will be a makefile describing the generation
of the code, and that when you edit stuff there, you have to run make
to update that code before you run the go tool.

I am explicitly not implementing the case where files get generated
during a build by 'go'.  We do not implement that case in goinstall
either, and so far that has been just fine.  In fact, I think it is preferable.
Without that case, people can use whatever tools they like to write
the programs for them, but then those generated programs have to
be checked into the repository.  There's a tradeoff here: this is a little
more work for the developer of a package, who has to check in the
generated files, but it is significantly less work for the users of the
package, who do not have to find, build, and install the tools that the
developer happens to use.  I think the internet has tipped the scales
in favor of making life easier for the people installing the package.
If we do this, all the effort that goes into things like configure scripts
to decide that you have the right version of yacc evaporates.

This has worked pretty well in package syscall, where we decided
early on that we could not depend on the inputs to the code generators
being available on every machine that would need to build Go.
It is more work for us when editing syscall but significantly less work
than trying to make something that every user can run.
It is also similar to checking in both the .tmpl and .html files for our
documentation instead of requiring the web server to know how to
do that conversion on the fly.

Just to take one more example, suppose that the Google APIs for
interacting with Docs and so on used protocol buffers (thankfully they
do not!).  As a client of a package for interacting with Docs, I don't care.
I don't want to know.  I just want to install that package and then call
the functions that it provides.  If installing that package requires that
I first go find out what a "protocol compiler" is, and then that I download
and install it (oh, and g++ too!), then I'm not going to use that package.
This is not entirely hypothetical: installing the protocol compiler is
difficult enough that I think we all agree it is beyond the scope of the
go tool, and yet I have written packages that interact with web services
using protocol buffers.  Checking in the pb.go is far easier on the
users of those packages than forcing them to have the same tools I do.

That's really just the argument for not invoking tools during the build.
If we turned around and decided we did want to invoke tools during
package builds, then we would also have to solve all the usual problems
that arise when building a make.  Those are moot as long as we decide
not to invoke tools in the first place.

Russ

[rsc, 2011-12-16 13:14:57]

On Fri, Dec 16, 2011 at 01:08,  <r@golang.org> wrote:
> it might be nice if the new ugly foo.h files failed in place if no
> #ifdef was satisfied instead of some later, mysterious place.

Done + uploaded.

> other people will have to do this sort of thing and it's O(N) at the
> best of times. can the go command not do this for you? if there's a rule
> for _$GOOS.go, why not _$GOOS.[ch]?

In addition to what I wrote earlier, I do not believe other people
will have to do this sort of thing.  The ugliness here is all caused
by C code, and we want other people to be writing Go code.
Runtime is special in almost every way.

Russ

[iant2, 2011-12-16 14:49:40]

Russ Cox <rsc@golang.org> writes:

> I believe those rules exist, as far as file selection is concerned,
> but C is not as helpful as Go here.  In Go you can take a bunch
> of files and hand them all to the compiler, and as long as
> everything is there somewhere, the compiler figures it out.
> In contrast, C needs every file to refer to the same #include
> file by name so they agree on the declarations.  If that #include
> file happens to be per-architecture, there is no good name.
>
> We could define that the go tool takes every foo_$GOOS.h file
> and copies it into the work directory as foo.h automatically,
> but that would be a C-specific rule in a Go tool.  The ugliness
> here contains the C rules in the runtime directory.

I'm not necessarily recommending this but there is another way.  In your
.c file write

#include ARCH_H

and compile the file with -DARCH_H=\"arch_i386.h\".

Ian

[lucio, 2011-12-16 15:30:31]

i just spotted this thread and I'd like to contribute just a small
factor that I'm sure was not overlooked, but has not been raised:
there is a multiplicative factor arising from the OS platforms and
architectures on which Go is able to be used.  If we continue to
expand the range of header files within a single directory, we're
liable to get lost.  It certainly does not seem to encourage expanding
the range of platforms and that ought to be a bad thing.

Also, it is my understanding that in Plan 9 the approach to
multiplicity of architectures (and platforms are hopefully some sort
of extension here) is to use the target as the development directory
and the "port" directory as an input (../port/*).

I have noticed that the combination of (OS/arch)es would give Plan 9
indigestion too, despite its traditionally cleaner approach, but I do
believe the problem of proliferation ought to be resolved as early as
possible.  Naturally I'm assuming that dumping everything in one
directory is not what anyone would choose, I'm hoping that there is a
better way that will be discovered if enough minds are applied to the
problem (and the problem is formulated correctly).

Lucio.

On 12/16/11, Ian Lance Taylor <iant@google.com> wrote:
> Russ Cox <rsc@golang.org> writes:
>
>> I believe those rules exist, as far as file selection is concerned,
>> but C is not as helpful as Go here.  In Go you can take a bunch
>> of files and hand them all to the compiler, and as long as
>> everything is there somewhere, the compiler figures it out.
>> In contrast, C needs every file to refer to the same #include
>> file by name so they agree on the declarations.  If that #include
>> file happens to be per-architecture, there is no good name.
>>
>> We could define that the go tool takes every foo_$GOOS.h file
>> and copies it into the work directory as foo.h automatically,
>> but that would be a C-specific rule in a Go tool.  The ugliness
>> here contains the C rules in the runtime directory.
>
> I'm not necessarily recommending this but there is another way.  In your
> .c file write
>
> #include ARCH_H
>
> and compile the file with -DARCH_H=\"arch_i386.h\".
>
> Ian
>

[rsc, 2011-12-16 20:18:59]

PTAL.

Now the behavior expected from the Go tool is
to compile/assemble with

-DGOOS_$(GOOS) -DGOARCH_$(GOARCH)

and

cp foo_$(GOOS).h foo_GOOS.h
cp foo_$(GOARCH).h foo_GOARCH.h
cp foo_$(GOOS)_$(GOARCH).h foo_GOOS_GOARCH.h

for all possible foo.

Russ

[r, 2011-12-16 20:25:03]

LGTM

it turned out well.

[rsc, 2011-12-16 20:34:04]

*** Submitted as http://code.google.com/p/go/source/detail?r=479bba71fb07 ***

runtime: make more build-friendly

Collapse the arch,os-specific directories into the main directory
by renaming xxx/foo.c to foo_xxx.c, and so on.

There are no substantial edits here, except to the Makefile.
The assumption is that the Go tool will #define GOOS_darwin
and GOARCH_amd64 and will make any file named something
like signals_darwin.h available as signals_GOOS.h during the
build.  This replaces what used to be done with -I$(GOOS).

There is still work to be done to make runtime build with
standard tools, but this is a big step.  After this we will have
to write a script to generate all the generated files so they
can be checked in (instead of generated during the build).

R=r, iant, r, lucio.dere
CC=golang-dev
http://codereview.appspot.com/5490053

[lucio, 2011-12-17 04:58:15]

I need to adjust my Plan 9 build infrastructure to match, and this
seems a very practical approach.  I really can't measure the impact
before I've tried that, but it does seem to be heading towards greater
Makefile simplicity and I do like that.

Lucio.


On 12/16/11, rsc@golang.org <rsc@golang.org> wrote:
> *** Submitted as
> http://code.google.com/p/go/source/detail?r=479bba71fb07 ***
>
> runtime: make more build-friendly
>
> Collapse the arch,os-specific directories into the main directory
> by renaming xxx/foo.c to foo_xxx.c, and so on.
>
> There are no substantial edits here, except to the Makefile.
> The assumption is that the Go tool will #define GOOS_darwin
> and GOARCH_amd64 and will make any file named something
> like signals_darwin.h available as signals_GOOS.h during the
> build.  This replaces what used to be done with -I$(GOOS).
>
> There is still work to be done to make runtime build with
> standard tools, but this is a big step.  After this we will have
> to write a script to generate all the generated files so they
> can be checked in (instead of generated during the build).
>
> R=r, iant, r, lucio.dere
> CC=golang-dev
> http://codereview.appspot.com/5490053
>
>
> http://codereview.appspot.com/5490053/
>