Fix identifier normalization to work under JDK1.5

Most browsers that have a Sun Java plugin with JRE1.5.

My recent changes to ParserBase to check that identifiers
are properly normalized use APIs added only in JRE1.6.

This consolidates the normalized identifier checks, so
that they work in reduced mode on JRE1.5.  Any identifiers
rejected under JRE1.6 will be rejected under JRE1.5, but
so will any identifiers containing a non-ASCII codepoint.

[MarkM, 2008-12-06 22:48:14]

LGTM

[MikeSamuel, 2008-12-06 22:55:13]

@3100

[DavidSarah, 2008-12-07 01:57:31]

http://codereview.appspot.com/10061/diff/1/2
File src/com/google/caja/parser/ParserBase.java (right):

http://codereview.appspot.com/10061/diff/1/2#newcode180
Line 180: if (s.charAt(i) >= 127) { return false; }
Is it worth accepting ISO-Latin-1? That is,
if (s.charAt(i) > 255) { return false; }

(Note that it is not necessary for this check to reject control characters --
those should be rejected based on IDENTIFIER_OR_KEYWORD_RE.)

[MikeSamuel, 2008-12-07 02:16:28]

http://codereview.appspot.com/10061/diff/1/2
File src/com/google/caja/parser/ParserBase.java (right):

http://codereview.appspot.com/10061/diff/1/2#newcode180
Line 180: if (s.charAt(i) >= 127) { return false; }
On 2008/12/07 01:57:32, DavidSarah wrote:
> Is it worth accepting ISO-Latin-1? That is,
> if (s.charAt(i) > 255) { return false; }

I don't know enough about NFC to know whether a character in [127,255] can
appear in a string that is not normalized occurring to NFC.

> (Note that it is not necessary for this check to reject control characters --
> those should be rejected based on IDENTIFIER_OR_KEYWORD_RE.)

Yep.

[DavidSarah, 2008-12-07 03:28:12]

http://codereview.appspot.com/10061/diff/1/2
File src/com/google/caja/parser/ParserBase.java (right):

http://codereview.appspot.com/10061/diff/1/2#newcode180
Line 180: if (s.charAt(i) >= 127) { return false; }
From http://unicode.org/reports/tr15/ :

Legacy character sets are classified into three categories based on their
normalization behavior with accepted transcoders.

Prenormalized. Any string in the character set is already in Normalization Form
X.
For example, ISO 8859-1 is prenormalized in NFC.