Reafactored Lexer class to allow chaining. The full chain when parsing #if directive looks like thi…

Reafactored Lexer class to allow chaining. The full chain when parsing #if directive looks like this: Preprocessor -> MacroExpander -> DirectiveHandler -> MacroExpander -> DefinedFilter -> Tokenizer. This chain dynamically changes depending on the current context.

Also added an incomplete implementation of #if handling and ExpressionParser to illustrate how this design is supposed to work.
Committed: https://code.google.com/p/angleproject/source/detail?r=1084

[Alok Priyadarshi, 2012-05-09 20:52:37]

Sorry for the huge patch. I have marked files that have just been renamed.

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
File src/compiler/preprocessor/new/Tokenizer.h (right):

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
src/compiler/preprocessor/new/Tokenizer.h:10: #include "Input.h"
No need to review: This is the original Lexer.h

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
File src/compiler/preprocessor/new/Tokenizer.l (right):

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
src/compiler/preprocessor/new/Tokenizer.l:26: #include "Tokenizer.h"
No need to review: This is the original pp.l

[kbr1, 2012-05-09 21:04:14]

LGTM after one cleanup.

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
File src/compiler/preprocessor/new/DirectiveParser.cpp (right):

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
src/compiler/preprocessor/new/DirectiveParser.cpp:44: if (token->type == '#')
parseDirective(token);
All of these character and string constants should be defined somewhere.
Probably in Token.h.

[Alok Priyadarshi, 2012-05-09 21:25:03]

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
File src/compiler/preprocessor/new/DirectiveParser.cpp (right):

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
src/compiler/preprocessor/new/DirectiveParser.cpp:44: if (token->type == '#')
parseDirective(token);
On 2012/05/09 21:04:14, kbr1 wrote:
> All of these character and string constants should be defined somewhere.
> Probably in Token.h.

Agree about the string constants. They will eventually become keywords so I
would not need to compare strings.

I am not sure what defining character constants would accomplish? In fact I find
it clearer to use the actual character literals everywhere.

[kbr1, 2012-05-09 21:35:15]

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
File src/compiler/preprocessor/new/DirectiveParser.cpp (right):

http://codereview.appspot.com/6203060/diff/2001/src/compiler/preprocessor/new...
src/compiler/preprocessor/new/DirectiveParser.cpp:44: if (token->type == '#')
parseDirective(token);
On 2012/05/09 21:25:03, Alok Priyadarshi wrote:
> I am not sure what defining character constants would accomplish? In fact I
find
> it clearer to use the actual character literals everywhere.

Leave this to your discretion in that case. Still think it would reduce the
possibility of typos if they were constants rather than literals.