Make arguments like Context.GrobName accessible as symbol lists

Make arguments like Context.GrobName accessible as symbol lists

This is a rather awkward case for a Rietveld review since it consists
of 20 separate commits (listed below).  The principal target was to
allow things like \omit Staff.TimeSignature to work as expected.  The
same treatment has been given to \alterBroken, \hide,
\overrideProperty and \shape, as well as to \accidentalStyle, and \footnote has had its interface overhauled (at the current point of time, the German documentation is in pre-2.16 state and the invalid uses of \footnote trigger errors after conversion).

For improving backward compatibility, dotted _LilyPond_ strings like
"Staff.TimeSignature" are also accepted in lieu of symbol list.  A
rather surprising corollary is that

  \override Lyrics.LyricText #'font-shape = #'italic

now works without needing to offset . with spaces.  While
Lyrics.LyricText is indeed run into a single LilyPond string in this
case, \override will happily pick that string apart again.

LilyPond strings are also acceptable for symbol? predicates.  The
conversions will only be attempted if the string is not accepted by a
predicate in the form of a string.

This iteration of the code allows \override and \revert to understand dotted notation even when there is a dot between grob specification and grob property path, like with

\override Accidental.color = #red

The following commits are involved:
Bump version for checking purposes


Run scripts/auxiliar/update-with-convert-ly.sh


Add more extensive rules for \tweak, \override, \revert, \overrideProperty


Change \footnote user interface and behavior to match \hide/\omit etc.

This does the following changes to the footnote user interface for
footnotes in music rather than in markups:

Time-based footnote syntax:

    \footnote [mark] offset #'Grob text \default
->  \footnote [mark] offset text [ Context. ] Grob

Like with any override, you now need to specify Context if it is not
supposed to be Bottom, like with Staff.TimeSignature.

Music-based footnote syntax:

    \footnote [mark] offset text music
->  stays identical

    \footnote [mark] offset #'Grob text music
->  \single\footnote [mark] offset text Grob music

Specifying Context.Grob is possible, but Context gets ignored.


Let \overrideProperty accept nested properties


Change \tweak syntax to symbol syntax

Instead of \tweak #'color, \tweak color is used.
Instead of \tweak Accidental #'color, \tweak Accidental.color is used.


convert-ly rule for \alterbroken, \accidentalStyle, \overrideProperty

This is required since not all of the old call forms can be
automatically recognized anymore, in particular not when the
Scheme-only form #"Context.GrobName" is being used.


Change \accidentalStyle to get a symbol list as argument

This changes the syntax of \accidentalStyle to accept specs like
Voice.piano-cautionary .


Make \alterBroken, \hide, \omit, \overrideProperty, \shape take unquoted Context.Grob specs

This changes the syntax of the named commands to accept
Voice.Accidental and similar specifications without any quoting.


Replace the rather fuzzy list-or-symbol? with symbol-list-or-symbol?

list-or-symbol? was previously used in the meaning of
symbol-list-or-symbol? only, and there is no point in not checking the
list members for actually being symbols, in order to avoid ugly
surprises later.


Add symbol-list-or-music? predicate

This is of interest for commands like \hide which accept either music
(to see an override) or a grob specification like Accidental or
Voice.Accidental.


Allow the string "xxx.yyy" to parse as a symbol list

This changes the parser to let xxx.yyy be a valid way of specifying a
function argument required to be #'(xxx yyy).


Allow property and grob paths to be constructed from strings

This allows use of either
\override Accidental color = #red
or
\override Accidental.color = #red
(in addition to existing forms), both for context modifications or for
property overrides in music, the latter with optional leading context
spec defaulting to Bottom. .  The same holds for reverts.


Add symbol-list? predicate


Allow LilyPond strings to stand in for symbols when parsing

Bare words and quote-enclosed words meeting word syntax can now be
used when a function argument expects to see a symbol.


Let \accidentalStyle detect its optional context argument by looking at its letter case

This is a fishy stopgap measure to let \accidentalStyle able to
distinguish #'Voice "default" from "default" alone while we generate
symbols from from #'Voice as well as "default".  If the first letter
is uppercase, the symbol is considered to be a context specification.


Factor SCM_IDENTIFIER out from embedded_scm_arg*

SCM_IDENTIFIER indicates "active Scheme" introduced with $ that is
subject to copying and reinterpretation.  While the parser had its own
*_IDENTIFIER category for everything interpreted specially, the
fallback SCM_IDENTIFIER itself never needed special or different
treatment from material generated with # (SCM_TOKEN).  As the various
*_IDENTIFIER types are faded out, this is changing.  As a consequence,
SCM_IDENTIFIER no longer shares the non-terminals with other
Scheme-type definitions.


parser.yy: make is_regular_identifier match the lexer definition

is_regular_identifier checks now for valid identifiers employing the
definition of words that is also used in the lexer: letters, and
characters outside of the ASCII range, interspersed with single - and
_ characters.


parser.yy: FRACTION does not require a closed expression before it

FRACTION items can't trail any other valid expression without
separating '*' or similar, so the argument list before them does not
need to be of "closed" type.


Assignments can't contain literal lyric music (wrong lexer mode)

Since assignments can't happen in lyrics mode (only in INITIAL mode or
the initial music mode), there is no point in supporting
non-mode-switching items on the right side of assignments that can
only occur in lyrics mode.

[lemzwerg, 2012-10-09 05:04:50]

LGTM, without testing, and without really understanding the change.  However,
simplifications and generalizations are always a good thing.

http://codereview.appspot.com/6635050/diff/1/Documentation/de/notation/pitche...
File Documentation/de/notation/pitches.itely (right):

http://codereview.appspot.com/6635050/diff/1/Documentation/de/notation/pitche...
Documentation/de/notation/pitches.itely:1529: \accidentalStyle
StaffGroup."voice"
This looks strange.  I don't expect double quotes after a dot.  Any better
representation possibility?

[dak, 2012-10-09 07:36:52]

On 2012/10/09 05:04:50, lemzwerg wrote:
> LGTM, without testing, and without really understanding the change.  However,
> simplifications and generalizations are always a good thing.
> 
>
http://codereview.appspot.com/6635050/diff/1/Documentation/de/notation/pitche...
> File Documentation/de/notation/pitches.itely (right):
> 
>
http://codereview.appspot.com/6635050/diff/1/Documentation/de/notation/pitche...
> Documentation/de/notation/pitches.itely:1529: \accidentalStyle
> StaffGroup."voice"
> This looks strange.  I don't expect double quotes after a dot.  Any better
> representation possibility?

Agreed.  _All_ of the following will work:
\accidentalStyle "StaffGroup.voice"
\accidentalStyle StaffGroup.voice
\accidentalStyle "StaffGroup"."voice"

whereas everything involving #"Staffgroup..." will _not_ work (only LilyPond
strings autoconvert to symbols).

I think I'd go for
\accidentalStyle Staffgroup.piano-cautionary
myself as then the temptation to employ Scheme strings by accident is smaller,
but then it seems inconsistent to use
\accidentalStyle "piano-cautionary"
so I'd crank open another convert-ly rule to turn those into
\accidentalStyle piano-cautionary
as well.  Which makes for worse backward-compatibility of scores.

What do people prefer to see?

[t.daniels_treda.co.uk, 2012-10-09 08:14:55]

<dak@gnu.org>: Tuesday, October 09, 2012 8:36 AM

> On 2012/10/09 05:04:50, lemzwerg wrote:
>>
http://codereview.appspot.com/6635050/diff/1/Documentation/de/notation/pitche...
>> Documentation/de/notation/pitches.itely:1529: \accidentalStyle
>> StaffGroup."voice"
>> This looks strange.  I don't expect double quotes after a dot.  Any
> better
>> representation possibility?
> 
> Agreed.  _All_ of the following will work:
> \accidentalStyle "StaffGroup.voice"
> \accidentalStyle StaffGroup.voice
> \accidentalStyle "StaffGroup"."voice"
> 
> whereas everything involving #"Staffgroup..." will _not_ work (only
> LilyPond strings autoconvert to symbols).
> 
> I think I'd go for
> \accidentalStyle Staffgroup.piano-cautionary
> myself as then the temptation to employ Scheme strings by accident is
> smaller, but then it seems inconsistent to use
> \accidentalStyle "piano-cautionary"
> so I'd crank open another convert-ly rule to turn those into
> \accidentalStyle piano-cautionary
> as well.  Which makes for worse backward-compatibility of scores.
> 
> What do people prefer to see?

As the 2.17 releases appear to be majoring on syntax changes I'd prefer
to go for the most intuitive easily-learned syntax rather than backward
compatibility.  Let's get all the syntax changes incorporated in 2.17,
then we can agree an immutable set ready for release 3.

Trevor

[dak, 2012-10-09 08:38:34]

On 2012/10/09 08:14:55, t.daniels_treda.co.uk wrote:
> <dak@gnu.org>: Tuesday, October 09, 2012 8:36 AM

> > so I'd crank open another convert-ly rule to turn those into
> > \accidentalStyle piano-cautionary
> > as well.  Which makes for worse backward-compatibility of scores.
> > 
> > What do people prefer to see?
> 
> As the 2.17 releases appear to be majoring on syntax changes I'd prefer
> to go for the most intuitive easily-learned syntax rather than backward
> compatibility.

Here is the rub: what we are currently talking about is a choice between several
convert-ly rules that all lead to valid programs after this change.

Whether you write
\override TextSpanner #'(bound-details left stencil-align-dir-y) = #-2
or
\override Bottom.TextSpanner #'(bound-details left stencil-align-dir-y) = #-2
or
\override #'(Bottom TextSpanner) bound-details.left.stencil-align-dir-y = #-2
it is now all the same to LilyPond.

Now it is nice to have found underlying principles making it possible to
obliterate fine distinctions and remove the necessity of using Scheme for
specifying a grob property path to modify, with mostly cosmetic backward
compatibility consequences.

But since LilyPond is now able to _read_ a lot of different phrasings of the
same material, the question is how we want to be _writing_ this material.

I'll likely go for the quoteless \accidentalStyle form generally here.

I think I'll prepare a radical convert-ly-only patch on top of this patch series
that demonstrates what "now valid" syntax we _could_ be using/advertising as
input if we wanted to.

We can get rid of a _lot_ of #' style thingies with this patch series.

> Let's get all the syntax changes incorporated in 2.17,
> then we can agree an immutable set ready for release 3.

"Immutable" as "we guarantee we will be able to process this" rather than "we
guarantee we will write our scores to have input looking like that"?

[t.daniels_treda.co.uk, 2012-10-09 10:03:59]

dak@gnu.org: Tuesday, October 09, 2012 9:38 AM


> Here is the rub: what we are currently talking about is a choice between
> several convert-ly rules that all lead to valid programs after this
> change.
> 
> Whether you write
> \override TextSpanner #'(bound-details left stencil-align-dir-y) = #-2
> or
> \override Bottom.TextSpanner #'(bound-details left stencil-align-dir-y)
> = #-2
> or
> \override #'(Bottom TextSpanner) bound-details.left.stencil-align-dir-y
> = #-2

and presumably even
\override Voice.TextSpanner   bound-details.left.stencil-align-dir-y = #-2
(which I prefer)

> it is now all the same to LilyPond.
> 
> Now it is nice to have found underlying principles making it possible to
> obliterate fine distinctions and remove the necessity of using Scheme
> for specifying a grob property path to modify, with mostly cosmetic
> backward compatibility consequences.
> 
> But since LilyPond is now able to _read_ a lot of different phrasings of
> the same material, the question is how we want to be _writing_ this
> material.

Indeed.  We need to define a canonical form for the bulk of the manuals,
although we should explain the flexibility and available variants once.
 
> I'll likely go for the quoteless \accidentalStyle form generally here.

Good.
 
> I think I'll prepare a radical convert-ly-only patch on top of this
> patch series that demonstrates what "now valid" syntax we _could_ be
> using/advertising as input if we wanted to.

It's only right and just that you get first go at suggesting a canonical form,
and a patch would be a good way of expressing your preference.
(Regexp tutorial, anyone?  Although we can always apply the patch and 
run it on the manuals.)

> We can get rid of a _lot_ of #' style thingies with this patch series.

The more the better.

>> Let's get all the syntax changes incorporated in 2.17,
>> then we can agree an immutable set ready for release 3.
> 
> "Immutable" as "we guarantee we will be able to process this" rather
> than "we guarantee we will write our scores to have input looking like
> that"?

I meant the former; the latter would be just an aspiration.  As with 
the documentation, this will need an agreement on the canonical form.

Trevor

[dak, 2012-10-09 10:33:36]

On 2012/10/09 10:03:59, t.daniels_treda.co.uk wrote:
> dak@gnu.org: Tuesday, October 09, 2012 9:38 AM
> 
> > I think I'll prepare a radical convert-ly-only patch on top of this
> > patch series that demonstrates what "now valid" syntax we _could_ be
> > using/advertising as input if we wanted to.
> 
> It's only right and just that you get first go at suggesting a
> canonical form, and a patch would be a good way of expressing your
> preference.

Well, I am a programmer.  I know what symbols are and what strings are
and can deal with the differences just fine.

The main point of such a radical convert-ly-only patch (which is going
to convert valid syntax into also valid syntax) is to provide one
extreme outlook as a reference point for discussion.  The main
obstacles will be the need to redocument, to rethink, to reeducate,
and to talk with those who have to deal with scores written by others,
and with those who teach others about LilyPond.

In all of those areas my voice carries less rather than more weight
than that of others.

> >> Let's get all the syntax changes incorporated in 2.17,
> >> then we can agree an immutable set ready for release 3.
> > 
> > "Immutable" as "we guarantee we will be able to process this" rather
> > than "we guarantee we will write our scores to have input looking like
> > that"?
> 
> I meant the former; the latter would be just an aspiration.  As with 
> the documentation, this will need an agreement on the canonical form.

Perhaps \displayLilyMusic and its underlying primitives at some point
of time will need to allow specifying a preferred printing style to
use.

At least those that use \displayMusic and its ilk for harvesting
LilyPond's interpretation of an input file will not be affected.

[wl_gnu.org, 2012-10-09 12:28:25]

> \accidentalStyle StaffGroup.voice

This is my preferred form.  However, some time ago we decided (more or
less) to say e.g.

  \clef "bass"

In the light of your syntax improvements/simplifications, shouldn't we
start to use

  \clef bass

instead, making data in double quotes really user strings only?


    Werner

[dak, 2012-10-09 12:32:42]

Werner LEMBERG <wl@gnu.org> writes:

>> \accidentalStyle StaffGroup.voice
>
> This is my preferred form.  However, some time ago we decided (more or
> less) to say e.g.
>
>   \clef "bass"
>
> In the light of your syntax improvements/simplifications, shouldn't we
> start to use
>
>   \clef bass
>
> instead, making data in double quotes really user strings only?

\clef bass_8 won't work, neither will \clef treble^8, and I see no
sensible way to make them work.  As long as that is our naming choice,
using quotes for clefs is the only way to keep the syntax consistent.

-- 
David Kastrup

[lemzwerg, 2012-10-09 16:44:16]

LGTM.

http://codereview.appspot.com/6635050/diff/7002/ly/music-functions-init.ly
File ly/music-functions-init.ly (right):

http://codereview.appspot.com/6635050/diff/7002/ly/music-functions-init.ly#ne...
ly/music-functions-init.ly:105: (ly:input-warning location (_ "not a spanner
name, `~a'") name)
Wouldn't

  "`~a' is not a spanner name"

be more appropriate?

[dak, 2012-10-09 18:24:22]

http://codereview.appspot.com/6635050/diff/7002/ly/music-functions-init.ly
File ly/music-functions-init.ly (right):

http://codereview.appspot.com/6635050/diff/7002/ly/music-functions-init.ly#ne...
ly/music-functions-init.ly:105: (ly:input-warning location (_ "not a spanner
name, `~a'") name)
On 2012/10/09 16:44:16, lemzwerg wrote:
> Wouldn't
> 
>   "`~a' is not a spanner name"
> 
> be more appropriate?

No idea, I just copied the existing error message.  And the corresponding
regression test already expects this warning.

[janek, 2012-10-09 21:08:47]

I'm just skimming the discussion (the patch is big and non-trivial, so
i will have problems reviewing it), but i spotted one interesting
sentence...

On Tue, Oct 9, 2012 at 10:38 AM,  <dak@gnu.org> wrote:
> We can get rid of a _lot_ of #' style thingies with this patch series.

Ahhhh!!!  So this is the patch i've been waiting for for about 5 years now! :D
As Trevor said, the more the better.  Pity that \clef treble_8 won't
work... if you stumble upon an idea for supporting something like
this, let us know!

best,
Janek

[dak, 2012-10-10 16:34:21]

On 2012/10/09 21:08:47, janek wrote:
> I'm just skimming the discussion (the patch is big and non-trivial, so
> i will have problems reviewing it), but i spotted one interesting
> sentence...
> 
> On Tue, Oct 9, 2012 at 10:38 AM,  <mailto:dak@gnu.org> wrote:
> > We can get rid of a _lot_ of #' style thingies with this patch series.
> 
> Ahhhh!!!  So this is the patch i've been waiting for for about 5 years now! :D

Well, I uploaded a version of the patch with a heavy-handed conversion rule for
demonstration purposes.  Now you can probably wait for about 5 years before the
patch downloads...

[Keith, 2012-10-13 05:16:00]

I like it.

http://codereview.appspot.com/6635050/diff/15002/Documentation/notation/chang...
File Documentation/notation/changing-defaults.itely (right):

http://codereview.appspot.com/6635050/diff/15002/Documentation/notation/chang...
Documentation/notation/changing-defaults.itely:4202: \tweak NoteHead.stencil
#ly:text-interface::print

The dotted form is very nice.  

This way we always have two arguments, each of which can be expanded to a dotted
form to clarify where to find the thing we want LilyPond to change.  This
more-regular syntax is easier to remember than optional arguments.

{ \override Script X-offset = #-0.5
  d'->
  d'-\tweak X-offset #-0.5 ->
  \override Staff.Stem X-offset = #0.7
  << g' \\ b >>
  \tweak Stem.X-offset #0.25 g' }

http://codereview.appspot.com/6635050/diff/15002/lily/footnote-engraver.cc
File lily/footnote-engraver.cc (left):

http://codereview.appspot.com/6635050/diff/15002/lily/footnote-engraver.cc#ol...
lily/footnote-engraver.cc:49: IMPLEMENT_TRANSLATOR_LISTENER (Footnote_engraver,
footnote);
I don't see now the new syntax avoids the need for a listener -- is this
clean-up of unused code ?

[dak, 2012-10-13 07:04:08]

http://codereview.appspot.com/6635050/diff/15002/Documentation/notation/chang...
File Documentation/notation/changing-defaults.itely (right):

http://codereview.appspot.com/6635050/diff/15002/Documentation/notation/chang...
Documentation/notation/changing-defaults.itely:4202: \tweak NoteHead.stencil
#ly:text-interface::print
On 2012/10/13 05:16:00, Keith wrote:
> 
> The dotted form is very nice.  
> 
> This way we always have two arguments, each of which can be expanded to a
dotted
> form to clarify where to find the thing we want LilyPond to change.  This
> more-regular syntax is easier to remember than optional arguments.
> 
> { \override Script X-offset = #-0.5
>   d'->
>   d'-\tweak X-offset #-0.5 ->
>   \override Staff.Stem X-offset = #0.7
>   << g' \\ b >>
>   \tweak Stem.X-offset #0.25 g' }

The mailing list discussion pointed out the inconsistency between the use of
dots between tweaks and overrides.  If tweaking of nested properties is
implemented at some point of time, recognizing the specification of an optional
context when at the same time optional trailing path components might be
available is not really feasible.

Several proposals for changing this have been put forward.  Have to think about
this.

http://codereview.appspot.com/6635050/diff/15002/lily/footnote-engraver.cc
File lily/footnote-engraver.cc (left):

http://codereview.appspot.com/6635050/diff/15002/lily/footnote-engraver.cc#ol...
lily/footnote-engraver.cc:49: IMPLEMENT_TRANSLATOR_LISTENER (Footnote_engraver,
footnote);
On 2012/10/13 05:16:00, Keith wrote:
> I don't see now the new syntax avoids the need for a listener -- is this
> clean-up of unused code ?

The new syntax allows a grob specification _with_ context, allowing to turn a
"time-based" footnote into a proper override instead of an event.  Since the
tweak-like interpretation of the "footnote-music" field was already in place
previously, this works just as well.

The disadvantage is that you have to specify the context in case of things like
Staff.TimeSignature, just like you would have to do with any override.

The advantage is that this is predictably similar to other operations and does
not require you to juggle the Footnote_engraver between different contexts in
order to have it do its job in difficult circumstances.

So this change of operation is not forced by the new syntax, but rather
facilitated.

Since there is no longer anything generating footnote music events for any
purpose except sticking it into the footnote-music property of grobs, using a
Music data structure at all for storing the footnote data is pretty arbitrary. 
It might be subject to later cleanup, but I wanted to keep the changes
reasonably confined.

This is probably something that is easier to examine in the scope of its change
as a commit in the dev/syntax branch