Issue 905: Gracefully ignore UTF-8 BOM in the middle of a file

Issue 905: Gracefully ignore UTF-8 BOM in the middle of a file

[lemzwerg, 2011-08-15 18:14:21]

Could you please tell me what this patch is good for?  A BOM not at the
beginning of a file is no longer a BOM...

I don't oppose to emitting a warning if U+FEFF is encountered, and we
subsequently ignore it (since its use as zero width no-break space is
deprecated), but only within strings...

What am I missing?

[Reinhold, 2011-08-15 18:36:53]

On 2011/08/15 18:14:21, lemzwerg wrote:
> Could you please tell me what this patch is good for?  A BOM not at the
> beginning of a file is no longer a BOM...
> 
> I don't oppose to emitting a warning if U+FEFF is encountered, and we
> subsequently ignore it (since its use as zero width no-break space is
> deprecated), but only within strings...
> 
> What am I missing?

RFC 3629 says that U+FEFF is a zero-width non-breakable space, which is also
used as BOM. It also says: 
" This character
   can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text,"
...
"  It is important to understand that the character U+FEFF appearing at
   any position other than the beginning of a stream MUST be interpreted
   with the semantics for the zero-width non-breaking space, and MUST
   NOT be interpreted as a signature."

Also, our lilypond files are text, so I would understand this that we should
treat the U+FEFF inside the file contents as normal whitespace.

[pacovila, 2011-08-15 20:07:48]

On 2011/08/15 18:14:21, lemzwerg wrote:
> Could you please tell me what this patch is good for?  A BOM not at the
> beginning of a file is no longer a BOM...
> 
> I don't oppose to emitting a warning if U+FEFF is encountered, and we
> subsequently ignore it (since its use as zero width no-break space is
> deprecated), but only within strings...
> 
> What am I missing?

The BOM is invisible and you can not be sure whether it is inside a string or
anywhere else.  In my experience, it is very common that Windows users
create/open/modify lilypond input files with the Notepad  accessory, maybe
inserting new charachters at the very beginning, thus inadvertently moving the
BOM and obtaining failed compiles.  Moreover, they are not able to fix the file
because the BOM is invisible.  So, the patch would be very useful in that makes
the BOM transparent for the lilypond syntax and the user could forget it at
last.

[lemzwerg, 2011-08-15 22:08:24]

OK.

Please make the warning message more verbose, though.

[pkx166h, 2011-08-24 21:27:01]

passes make and reg tests