Eliminate MARKUP_IDENTIFIER and MARKUPLIST_IDENTIFIER

Eliminate MARKUP_IDENTIFIER and MARKUPLIST_IDENTIFIER

Those and their LYRIC variants are no longer generated by the lexer.
Instead, SCM_IDENTIFIER takes over their function.  One advantage is
that standard Scheme strings and lists of strings (including the empty
list!) are no longer receiving special syntactic status when used as
identifier.

Another is that markups and markup lists can now also be produced and
interpreted by define-scheme-function.  The ultimate goal, of course,
is to have try_special_identifiers return nothing except
SCM_IDENTIFIER.  If that is the case, $... ("immediate Scheme") does
no longer need to get evaluated in the lexer and can be retitled into
"active Scheme".  That would get rid of the Scheme timing problems and
make basically _everything_ doable by define-scheme-function.

This issue is a two-patch issue with the first patch being named:

parser.yy: remove "embedded_scm" as one production for "assignment"

This saved a few lines in the grammar for the price of clarity.

[janek, 2012-09-28 06:22:40]

The description looks really nice, but unfortunately i think i don't grasp the
consequences of this change.  Could you give some before/after input examples?
Does this have anything to do with what you proposed in "[GLISS] turn xxx.yyy
into ("xxx" "yyy")"?

[dak, 2012-09-29 13:10:09]

On 2012/09/28 06:22:40, janek wrote:
> The description looks really nice, but unfortunately i think i don't grasp the
> consequences of this change.  Could you give some before/after input examples?

Embarrassingly, not really.  The point of this change is to move forward in
removing the special-casing of various *_IDENTIFIER token types.  The target is
to arrive at only a single SCM_IDENTIFIER type.  Once that is achieved, it is no
longer necessary to evaluate identifiers in the lexer, removing a source for
awkward timing problems like

xxx = { c4 c4 c4 c4 }
\xxx

But as long as there remains more than one category of identifier, this can't be
done.  In a similar vein, it would be nice to be able to use a scheme function
call anywhere you use a markup: if we arrive there, markup and markup list
commands become mostly uninteresting for defining commands intended to be
"top-level" markups, usually called with \markup \my-command ...  In that case,
you can just use \my-command directly without leading \markup and define it as a
scheme function.

At the current point of time, this does not work at most places where markups
are allowed: you can use an identifier there (which basically is an
SCM_IDENTIFIER now) but you can't use a Scheme function as their argument list
parsing is a huge source of parser conflicts.  Once Scheme/music functions are
called somewhat transparently, they can be used pretty much everywhere, also in
and as markup.

This patch is part of preparing for that change, but it does not provide
significant improvements in usability by itself.  The main point is to cut down
on special cases provided by the lexer without _losing_ functionality.  But the
actual awards for that will only come after a number of similar patches.

> Does this have anything to do with what you proposed in "[GLISS] turn xxx.yyy
> into ("xxx" "yyy")"?

Not really.  It likely helps a bit.  It also begs the question whether the .
syntax should work just for building string lists, or whether it should be open
to building arbitrary markup lists (of which string lists are a special case). 
But that decision does not really require advance consideration: it will more or
less be hardwired into the parser actions without requiring to be anchored in
the grammar itself.

[janek, 2012-09-29 13:37:28]

On Sat, Sep 29, 2012 at 3:10 PM,  <dak@gnu.org> wrote:
> Embarrassingly, not really.  The point of this change is to move forward
> in removing the special-casing of various *_IDENTIFIER token types.  The
> target is to arrive at only a single SCM_IDENTIFIER type.  Once that is
> achieved, it is no longer necessary to evaluate identifiers in the
> lexer, removing a source for awkward timing problems like

So the idea is to make the grammar simpler (use less different types)
and have things happen on higher level?  Sounds reasonable, i guess...

> In a similar vein, it would be nice to be able to use a
> scheme function call anywhere you use a markup: if we arrive there,
> markup and markup list commands become mostly uninteresting for defining
> commands intended to be "top-level" markups, usually called with \markup
> \my-command ...  In that case, you can just use \my-command directly
> without leading \markup and define it as a scheme function.

oh, that'd be nice.

Thanks for explanation!
LTGM.
Janek

[dak, 2012-09-29 14:30:57]

On 2012/09/29 13:37:28, janek wrote:
> On Sat, Sep 29, 2012 at 3:10 PM,  <mailto:dak@gnu.org> wrote:
> > Embarrassingly, not really.  The point of this change is to move forward
> > in removing the special-casing of various *_IDENTIFIER token types.  The
> > target is to arrive at only a single SCM_IDENTIFIER type.  Once that is
> > achieved, it is no longer necessary to evaluate identifiers in the
> > lexer, removing a source for awkward timing problems like
> 
> So the idea is to make the grammar simpler (use less different types)
> and have things happen on higher level?  Sounds reasonable, i guess...

Actually, it is not really "simplifying" the grammar.  At the current point of
time, the lexer has to make decisions about the type of identifiers (and
functions) at a point of time where those decisions are actually premature.  As
a result of those decisions, different tokens are generated for identifiers and
functions.  I am moving towards generating only one kind of token.  The same
kind of decisions as before still have to be made, and since the lexer no longer
makes them, the decisions are instead taken in the actions of the parser.  The
results of those decisions are only apparent in the parser rules when those
decisions lead to "synthetic" tokens (similar to the previous "real" ones) being
generated in the parser actions.

So the distinctions that were previously made are still there, they are just
done at a later point of time, and result in different actions through different
mechanisms.  Of course, once the lexer only delivers identifiers in a single
way, there are some simplifications possible, but the overall architecture is
still a worse fit to the Bison/Flex pairing of parser/scanner generator than the
state from two years ago, as that pairing is based on lookahead principles with
rigid tokenization rules that are not really a perfect fit for LilyPond's sort
of dynamic typing.

So overall, the parser is not getting simpler.  I can straighten out some stuff
when passing certain milestones, "boiling down" complexity again, but since I am
moving functionality from where it is "intended to be" from the view of the
basic Bison/Flex architecture, the best it can be is a consistent subsystem not
looking more complex than the task it is taking over.

[janek, 2012-09-29 15:57:27]

On Sat, Sep 29, 2012 at 4:30 PM,  <dak@gnu.org> wrote:
> Actually, it is not really "simplifying" the grammar.  At the current
> point of time, the lexer has to make decisions about the type of
> identifiers (and functions) at a point of time where those decisions are
> actually premature.  As a result of those decisions, different tokens
> are generated for identifiers and functions.  I am moving towards
> generating only one kind of token.  The same kind of decisions as before
> still have to be made, and since the lexer no longer makes them, the
> decisions are instead taken in the actions of the parser.  The results
> of those decisions are only apparent in the parser rules when those
> decisions lead to "synthetic" tokens (similar to the previous "real"
> ones) being generated in the parser actions.

ok.  That's what i meant when i said "have things happen on higher
level" (i expect that it's not a technically correct statement, of
course :) )
thanks for explanations!
Janek