code review 6036054: gc, go/parser: accept .go files with UTF-8 BOM, ignore ...

gc, go/parser: accept .go files with UTF-8 BOM, ignore that bytes before compile and parse.

Some text editors, especially in Windows, get used to add UTF-8 BOM to file header.
Currently, the BOM are "invalid characters" for gc and go/parser, which should be
ignored to make sure that the files are compiled and parsed properly.
After this CL, cmd/go godoc gofmt and other go tools works well with UTF-8 BOM, too.

[liigo, 2012-04-15 17:37:47]

Hello golang-dev@googlegroups.com,

I'd like you to review this change to
https://code.google.com/p/go

[0xjnml, 2012-04-15 18:24:20]

On Sunday, April 15, 2012 7:37:47 PM UTC+2, Liigo Zhuang wrote:
>
> Description:
> gc, go/parser: accept .go files with UTF-8 BOM, ignore that bytes before
> compile and parse.
>
I believe this has been discussed at least once before at golang-nuts. I 
suggest to reject this CL, it's IMO a bad idea to support broken editors.

[r, 2012-04-15 21:34:04]

Strictly speaking, a BOM is legal in UTF-8 but only as a marker for
the type of the data stream, a magic number if you will. Since Go
source code is required to be UTF-8, a BOM is never necessary and
arguably erroneous. We've come this far without accepting BOMS and I'd
like to keep it that way.

-rob

[liigo, 2012-04-16 07:17:53]

BOM is commonly used to help text editors identifying encoding of a file.
Some editors add BOM impliedly when saving utf-8 encoded files. If there is
no BOM, text editors maybe try guessing it when opening a file, and error
maybe occurs. See wikipedia how the BOMs appear and how they are widely
used:  http://en.wikipedia.org/wiki/Byte_order_mark

So, a .go source files with utf-8 BOM is a valid utf-8 encoded files. But
the go's parser and compiler didn't accept these files, which is very
strange. If someone stick to not accept utf-8 BOM in .go files, I would
like to suggest modifying Go's language specification to clarity that
explicitly.


2012/4/16 Rob 'Commander' Pike <r@golang.org>

> Strictly speaking, a BOM is legal in UTF-8 but only as a marker for
> the type of the data stream, a magic number if you will. Since Go
> source code is required to be UTF-8, a BOM is never necessary and
> arguably erroneous. We've come this far without accepting BOMS and I'd
> like to keep it that way.
>
> -rob
>



-- 
by *Liigo*, http://blog.csdn.net/liigo/
Google+  https://plus.google.com/105597640837742873343/

[kardia, 2012-04-16 15:00:00]

Microsoft Notepad adds a BOM to files, as does Microsoft's Visual Studio 
Editor. BUT, those are the only products I know of that do that. Notepad++ 
does not, nor does Sublime Text 2, ... I have not opinion on this topic, 
but in my experience, the BOM automatic insertions is only with Microsoft's 
products. 

On Monday, April 16, 2012 12:17:52 AM UTC-7, Liigo Zhuang wrote:
>
> BOM is commonly used to help text editors identifying encoding of a file. 
> Some editors add BOM impliedly when saving utf-8 encoded files. If there is 
> no BOM, text editors maybe try guessing it when opening a file, and error 
> maybe occurs. See wikipedia how the BOMs appear and how they are widely 
> used:  http://en.wikipedia.org/wiki/Byte_order_mark
>
> So, a .go source files with utf-8 BOM is a valid utf-8 encoded files. But 
> the go's parser and compiler didn't accept these files, which is very 
> strange. If someone stick to not accept utf-8 BOM in .go files, I would 
> like to suggest modifying Go's language specification to clarity that 
> explicitly.
>
>
> 2012/4/16 Rob 'Commander' Pike <r@golang.org>
>
>> Strictly speaking, a BOM is legal in UTF-8 but only as a marker for
>> the type of the data stream, a magic number if you will. Since Go
>> source code is required to be UTF-8, a BOM is never necessary and
>> arguably erroneous. We've come this far without accepting BOMS and I'd
>> like to keep it that way.
>>
>> -rob
>>
>
>
>
> -- 
> by *Liigo*, http://blog.csdn.net/liigo/
> Google+  https://plus.google.com/105597640837742873343/
>
>