RFC: language beginnings

This includes most of a grammar for a PP domain-specific language.
Syntactically and behaviorally it is very much like C, with reduced types
(int, string, bool, no structs, no enums).  Most of the PP-specific actions
are simple function calls.

Major departures from C are:
  - dynamic typing, with optional runtime checking
  - function literals and anonymous functions
  - list/tuple literals
  - symbols are private unless declard public
  - no headers

I have converted all of the existing device code to the new syntax, with a few
bits of hand-waving to work out still.  The actual interpreter for the
language is yet to be written. :)

I'm sending this change as a chance to get some feedback.

Tim


 Makefile           |   78 
 grammar.l          | 1012 +++++
 grammar.y          |  882 +++++
 identifier.h       |   78 
 language.cpp       |  304 +
 language.h         |  146 
 lexer_test.cpp     |  838 ++++
 main_lex.cpp       |   36 
 main_parse.cpp     |   20 
 pipe_file.h        |  244 +
 pp-files/amd_k8.pp | 9168 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 pp-files/cpu.pp    |   51 
 pp-files/cpuid.pp  | 3543 ++++++++++++++++++++
 pp-files/msr.pp    |  924 +++++
 pp-files/pci.pp    | 6018 ++++++++++++++++++++++++++++++++++
 pp-files/pp.pp     |  267 +
 string_file.h      |  116 
 variable.h         |  218 +
 variable_test.cpp  |  152 
 19 files changed, 24095 insertions(+)

[mjterave, 2009-11-12 03:59:59]

I just glanced at the lex/yacc stuff. I'll read through more later.

http://codereview.appspot.com/154047/diff/1/10
File language/grammar.l (right):

http://codereview.appspot.com/154047/diff/1/10#newcode2
language/grammar.l:2: DEC			[0-9]
[:digit:] ?

http://codereview.appspot.com/154047/diff/1/10#newcode5
language/grammar.l:5: SPACE			[ \t\n\v\f\r]
Why can't you use [:space:] ?

http://codereview.appspot.com/154047/diff/1/10#newcode69
language/grammar.l:69: /* Support C-style comments, even nested. */
Argh. Don't do this. :)

Why deviate from the C-style rule for comments? It's what people are used to.

http://codereview.appspot.com/154047/diff/1/10#newcode86
language/grammar.l:86: 0[bB]{DEC}+		{ dump(yyscanner); return
int_literal(yyscanner); }
What is this token? This is weird to me.
I don't think you want to match 0b34.

http://codereview.appspot.com/154047/diff/1/19
File language/grammar.y (right):

http://codereview.appspot.com/154047/diff/1/19#newcode88
language/grammar.y:88: | string_literal { fprintf(stderr, "%d primary_expression
<- string_literal\n", lex_lineno()); }
Agh; why is the casing different for different literals :(

It's annoying to have rules "string_literal" and "STRING_LITERAL".

http://codereview.appspot.com/154047/diff/1/19#newcode112
language/grammar.y:112: // TODO: '123()' parses to a function_call_expression. 
Make sure to validate
Do you really want this to allow <primary_expression>()?
I even find "{return 0;}()" questionable above; I was expecting this to be
restricted to identifiers for a symbol table lookup.

Note: You also allow 123()().

http://codereview.appspot.com/154047/diff/1/19#newcode122
language/grammar.y:122: // We allow dangling commas for convenience.
Blech. Okay, I guess.

http://codereview.appspot.com/154047/diff/1/19#newcode129
language/grammar.y:129: | IDENTIFIER ':' assignment_expression { fprintf(stderr,
"%d argument <- IDENTIFIER ':' assignment_expression\n", lex_lineno()); }
Why do you want to allow this?
(b:a=5) ? Clearly I'm missing something here.

http://codereview.appspot.com/154047/diff/1/19#newcode161
language/grammar.y:161: : '+' { fprintf(stderr, "%d unary_operator <- '+'\n",
lex_lineno()); }
Why allow + as a unary operator?

http://codereview.appspot.com/154047/diff/1/19#newcode173
language/grammar.y:173: : cast_expression { fprintf(stderr, "%d
multiplicative_expression <- cast_expression\n", lex_lineno()); }
Presumably this is for order of expressions? It seems like there must be a
better way.