Parser: make optional arguments compatible with lookahead

Parser: make optional arguments compatible with lookahead

Basically reimplementing the MYBACKUP and MYREPARSE macros based on an
experimentally verified understanding of the LALR(1) parser produced
by Bison, this makes it possible to greatly simplify the grammar while
making optional funciton arguments (and particularly the following
non-optional arguments even when optional arguments are being skipped)
behave much more predictable.

Further simplifications are feasible: this is a first pitch.  Consists
of the following commits in reverse order:


Adapt documentation for optional arguments now that lookahead is permissible


Parser: properly disambiguate pitches from music in function arguments

In many instances, function arguments were parsed greedily without
looking at the function argument predicates first.  This did not allow
using subsequent function arguments of type pitch and duration since
they were combined into one music expression.

Parser: remove command_element and command_event synonyms of tempo_event


Parser: remove pitch_arg and PITCH_ARG


Parser: inline a few "common" expressions occuring only once now


Remove all closed music expressions


Permit event functions to use a non-closed argument list.


Add location data to extra_tokens_ stack in lexer.


Parser: let MYREPARSE and MYBACKUP back up even in presence of lookahead

This implies that the lookahead token must not yet have made an impact
on the state stack other than causing the reduction of the current
rule, or switching the lookahead token while Bison is mulling it over
will cause trouble.

Since an LALR(1) works on the top of the stack, this can usually be
guaranteed.  The "Too much lookahead" message is not produced.  If
these constructs are misused, the parser should typically complain
about the replacement token being unexpected.

The advantage is the ability to vastly simplify syntax rules that up
to now have to avoid lookahead tokens with a lot of drudgework.

Move the extra_token mechanism out of the lexer proper

Calling lex->pop_extra_token explicitly in the parser's yylex function
makes things considerably more robust.  It also allows pushing and
popping the EOF condition, necessary for backing up right at the end
of a file, like with ##{ \relative { c d e f } #} where the music
expression is checked against being a pitch, and then is backed up.

[lemzwerg, 2013-12-13 06:13:32]

As usual, thanks for working on such simplifications and clean-ups!  Inspecting
the code (without really understanding it), I haven't seen any gross
irregularities, and this is probably the only thing I can check...

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy
File lily/parser.yy (right):

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy#newcode1073
lily/parser.yy:1073: if (!unsmob_music ($$))
Looks like a slightly incorrect indentation...

[dak, 2013-12-13 06:35:19]

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy
File lily/parser.yy (right):

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy#newcode1073
lily/parser.yy:1073: if (!unsmob_music ($$))
On 2013/12/13 06:13:32, lemzwerg wrote:
> Looks like a slightly incorrect indentation...

How so?  This file is (see first line) basically indented in the Linux style. 
It's true that this tends to place the brace on the same line as the if.  If it
is on a separate line, however, it is lined up with the closing brace.

I certainly can put the brace in the same line, but that's not correcting
indentation, merely compressing the code a bit more.

What's more annoying is that the copy&paste passages often use spaces instead of
tabs, while the global indentation style is supposed to use tabs.  This partly
stems from a brief period where this file got edited while project-wide
directory file variables told Emacs not to use tabs.

The inconsistency is somewhat annoying, but in the interest of letting "git 
blame" work, gratuitous changes while moving lines around are not a good idea.

[lemzwerg, 2013-12-13 06:47:29]

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy
File lily/parser.yy (right):

https://codereview.appspot.com/41720043/diff/1/lily/parser.yy#newcode1073
lily/parser.yy:1073: if (!unsmob_music ($$))
> What's more annoying is that the copy&paste passages often use spaces
> instead of tabs, while the global indentation style is supposed to use
> tabs.

This is what I'm referring to.  I find such issues irritating, but maybe this is
only me.

> The inconsistency is somewhat annoying, but in the interest of letting
> "git blame" work, gratuitous changes while moving lines around are not
> a good idea.

Yep.  There are more such issues in the file, BTW.  I'm quite unhappy that  we
can't fix this easily without obfuscating results of `blame' command...

[dak, 2013-12-13 06:57:13]

On 2013/12/13 06:47:29, lemzwerg wrote:
> https://codereview.appspot.com/41720043/diff/1/lily/parser.yy
> File lily/parser.yy (right):
> 
> https://codereview.appspot.com/41720043/diff/1/lily/parser.yy#newcode1073
> lily/parser.yy:1073: if (!unsmob_music ($$))
> > What's more annoying is that the copy&paste passages often use spaces
> > instead of tabs, while the global indentation style is supposed to use
> > tabs.
> 
> This is what I'm referring to.  I find such issues irritating, but maybe this
is
> only me.
> 
> > The inconsistency is somewhat annoying, but in the interest of letting
> > "git blame" work, gratuitous changes while moving lines around are not
> > a good idea.
> 
> Yep.  There are more such issues in the file, BTW.  I'm quite unhappy that  we
> can't fix this easily without obfuscating results of `blame' command...

One can "fix" indentation in a separate commit not doing other things.
In the particular case of lily/lexer.ll and lily/parser.yy however,
there is no automatic indentation available.  Indentation Linux with tabs
merely means that you have to manually indent start and end of a syntax
rule and most lines inside of a rule will match that.

Since we have no automatic indentation engine that will work for those files,
it seems pointless to do reorganizations manually that will not stay around.

There is a comment at the start of those files essentially expressing this
rationale.

I have not actively invested thought in the tabs/spaces in this patch: the
Emacs settings here imply that new code will get tab-spaced, so the
space-spacing definitely stems from copy&paste passages.  Where they are
really copy&paste rather than cut&paste, git will likely not correlate
them anyway, but given the overall state of the file, it seems silly to
try making a large effort.