Make several special characters with or without backslash "non-special"

Issue 3487: Make several special characters with or without backslash "shorthands"


This is made from the following commits:

Make tempo range \tempo 20~30 be input as \tempo 20-30 instead

\tempo's use of ~ was rather untypical for LilyPond.  Letting it
rather use @code{-} for ranges leaves just a single use for ties,
making it feasible to make @code{~} definable by the user in a later
commit.


Replace staccatissimo shorthand -| with -!

The bar line character is used too prominently, and ! seems more appropriate.


Issue 3487: Make several special characters with or without backslash "shorthands"    
Single non-alphanumeric ASCII characters not requiring special
treatment in lexer or parser can now be redefined like escaped
identifiers.  The same holds for escaped non-alphanumeric ASCII
characters.  The identifying name you use for redefining them is the
string corresponding to the full shorthand, in contrast to escaped
identifiers where the identifying name omits the initial backslash.

Notable shorthands not treated specially in the parser (some of them
newly so) can be seen in the following definitions from
scm/declarations-init.ly:

"|" = #(make-music 'BarCheck)
"[" = #(make-span-event 'BeamEvent START)
"]" = #(make-span-event 'BeamEvent STOP)
"~" = #(make-music 'TieEvent)
"(" = #(make-span-event 'SlurEvent START)
")" = #(make-span-event 'SlurEvent STOP)
"\\!" = #(make-span-event 'CrescendoEvent STOP)
"\\(" = #(make-span-event 'PhrasingSlurEvent START)
"\\)" = #(make-span-event 'PhrasingSlurEvent STOP)
"\\>" = #(make-span-event 'DecrescendoEvent START)
"\\<" = #(make-span-event 'CrescendoEvent START)
"\\[" = #(make-span-event 'LigatureEvent START)
"\\]" = #(make-span-event 'LigatureEvent STOP)
"\\~" = #(make-music 'PesOrFlexaEvent)
"\\\\" = #(make-music 'VoiceSeparator)


Run scripts/auxiliar/update-with-convert-ly.sh


Make regtest for shorthands (Issue 3487)

[dak, 2013-08-04 13:03:19]

Better convert-ly rules

[dak, 2013-08-04 13:33:12]

Fix botched conversion inside of texidoc string manually

[lemzwerg, 2013-08-05 10:56:49]

LGTM.  Very nice!

[janneke_gnu.org, 2013-08-05 11:31:11]

> LGTM.  Very nice!
>
> https://codereview.appspot.com/12432043/

Nice.  Looking at your remarks

   Make tempo range \tempo 20~30 be input as \tempo 20-30 instead
   Dash is not particularly nice here

I would expect something like \tempo #20 and \tempo #'(20 30); however
I'm not sure if that would be in scope for this patch, and how much
we want to expose guile into .ly for normal use?

Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

[dak, 2013-08-05 12:03:38]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

>> LGTM.  Very nice!
>>
>> https://codereview.appspot.com/12432043/
>
> Nice.  Looking at your remarks
>
>    Make tempo range \tempo 20~30 be input as \tempo 20-30 instead
>    Dash is not particularly nice here
>
> I would expect something like \tempo #20 and \tempo #'(20 30); however
> I'm not sure if that would be in scope for this patch, and how much
> we want to expose guile into .ly for normal use?

If you allow a number or a pair, either 20 or 20/30 will be acceptable.
However, we had a discussion about this about a year ago, and there was
a definite lack of enthusiam about using what looks like a fraction
here.  Now one need not go overboard for avoiding Scheme: after all,
\override/\revert only very recently got a non-Scheme input method.

Nevertheless, I don't think that there would be a lot of enthusiasm for
requiring Scheme here.

One drawback I have not mentioned is the accessibility of those symbols
from Scheme.  It turns out that (string->symbol "\\(") cranks out
something like #{\\\(}# in Guile 1 and #{\\x28;}# (which actually seems
wrong) in Guile 2.  So the "straight" input syntax is sort of iffy, but
people usually won't need it.  Just using the string is easier to
remember than working with things like escapedParenOpenSymbol.

There is also a slight inconsistency in that definition/use pairs are

"xxx" = ... called via \xxx
"|" = ... called via |
"\\|" = ... called via \|

So for symbols, the backslash becomes part of the symbol name while it
doesn't for proper words.  Again, this seems natural, but it also means
that symbols like "+" or "-" or "/" or "*" seem rather off limits unless
they get encoded with something more intricate than string->symbol.  Now
"-", "*", "/" are all special in the parser anyway, but one might still
want to keep this in mind.

-- 
David Kastrup

[Keith, 2013-08-05 19:21:44]

Looks sensible.
The comments are confusing.
The commit message says almost the opposite of what you do.  Maybe
"Recognize )(][~| and \! etc. in the lexer.
Treat the ) in c4) like the \p command in c4\p but without the need for an
escape character, and similarly for '\)' ']' '\!' etc.  This requires the lexer
to recognize these characters and their escaped versions, but simplifies the
parser, and allows ")" to be (re-)defined in LilyPond code.
Two characters, '~' and '|', had a second meaning in some contexts, formerly
distinguished in the parser, so we change \tempo 20~30 to \tempo 20-30 and the
staccatissimo indication to -! "

Changing staccatissimo to -! should get a short1 entry in 'changes.tely'

https://codereview.appspot.com/12432043/diff/6001/input/regression/articulati...
File input/regression/articulation-syntax-change.ly (right):

https://codereview.appspot.com/12432043/diff/6001/input/regression/articulati...
input/regression/articulation-syntax-change.ly:3: texidoc = "The syntactic class
of articulations can be changed within
It is confusing to put it this way. 

We could always redefine post events to create an empty chord
trill = <>\trill
Now we can redefine ) as well

It is not clear what functionality you are protecting from regression.  Probably
this added regression test is not needed, because the usual definition and use
of ")" tests the new code.

https://codereview.appspot.com/12432043/diff/6001/lily/lexer.ll
File lily/lexer.ll (right):

https://codereview.appspot.com/12432043/diff/6001/lily/lexer.ll#newcode153
lily/lexer.ll:153: SPECIAL		\\.|[][|()~]
'special' is vague. Maybe POST_EVENT_CHAR

I wonder why you didn't make an explicit list of the escaped single-characters,
POST_EVENT_CHAR    [][|)(~]
POST_EVENT_COMMAND \\[][)(><~!\\]
so you catch errors like \= as an unexpected '=' rather than undefined command
"=".

https://codereview.appspot.com/12432043/diff/6001/python/convertrules.py
File python/convertrules.py (right):

https://codereview.appspot.com/12432043/diff/6001/python/convertrules.py#newc...
python/convertrules.py:3614: str = re.sub (r"([-^_])\||" + matchstring +
r"|[-^_][-^_]", subnonstring, str)
convert-ly rules are unlikely to be maintained, but this one is so non-obvious
that you might still want a comment: 
# Match strings, and articulations that end in -^_
# So that we avoid leave alone -| in quoted strings and c4--|

[dak, 2013-08-06 10:30:44]

I can't really work with "the comments are confusing" and I don't really see
that the characterization of the commit messages is accurate.  I can try to
reword them in a manner that puts less stress on the state before the change.

https://codereview.appspot.com/12432043/diff/6001/input/regression/articulati...
File input/regression/articulation-syntax-change.ly (right):

https://codereview.appspot.com/12432043/diff/6001/input/regression/articulati...
input/regression/articulation-syntax-change.ly:3: texidoc = "The syntactic class
of articulations can be changed within
On 2013/08/05 19:21:45, Keith wrote:
> It is confusing to put it this way. 

Well, I can try my hand of writing less gobbledygook here.

> We could always redefine post events to create an empty chord
> trill = <>\trill
> Now we can redefine ) as well

I'd prefer not to drag unrelated commands into the description, even if just as
an example.

> It is not clear what functionality you are protecting from regression.  

Redefining ) in a manner that changes it from post-event to standalone event:
previous to that patch, the syntax of ) was fixed.

> Probably
> this added regression test is not needed, because the usual definition and use
> of ")" tests the new code.

When it is used for providing the old semantics, yes.  We have more or less a
policy that new functionality should at least be visible in the regtests.

Should I rather move this to a snippet?

https://codereview.appspot.com/12432043/diff/6001/lily/lexer.ll
File lily/lexer.ll (right):

https://codereview.appspot.com/12432043/diff/6001/lily/lexer.ll#newcode153
lily/lexer.ll:153: SPECIAL		\\.|[][|()~]
On 2013/08/05 19:21:45, Keith wrote:
> 'special' is vague. Maybe POST_EVENT_CHAR

The whole point is that those characters behave like a backslashed identifier:
they will be postfix if defined as a postfix event, or standalone when defined
as standalone event.  So POST_EVENT_CHAR would be quite the opposite from what
this is about.  For example, this makes | and ~ very much equivalent from the
view of lexer and parser, but due to the assigned default definitions, ~ is
post-event and | is not.


I can dig into TeX terminology and call this "active", but I thought "special"
fit rather well.  Emacs syntax tables call this "symbol".  In contrast to TeX's
active characters, they also combine with backslash.

> I wonder why you didn't make an explicit list of the escaped
single-characters,
> POST_EVENT_CHAR    [][|)(~]
> POST_EVENT_COMMAND \\[][)(><~!\\]
> so you catch errors like \= as an unexpected '=' rather than undefined command
> "=".

Why would I want to do that when part of the motivation is to make the user able
to define \= ?  It makes more sense to define those as _inverted_ sets, namely
_exclude_ every character that the parser needs to see as a separate token
class.  For \x, this is more or less the case implicitly.

For straightforward non-letter characters such as @, this code is not
really yet doing that.  You ask why the \x set is not closed down.  I'd rather
go into the direction of opening up the x set...

https://codereview.appspot.com/12432043/diff/6001/python/convertrules.py
File python/convertrules.py (right):

https://codereview.appspot.com/12432043/diff/6001/python/convertrules.py#newc...
python/convertrules.py:3614: str = re.sub (r"([-^_])\||" + matchstring +
r"|[-^_][-^_]", subnonstring, str)
On 2013/08/05 19:21:45, Keith wrote:
> convert-ly rules are unlikely to be maintained, but this one is so non-obvious
> that you might still want a comment: 
> # Match strings, and articulations that end in -^_
> # So that we avoid leave alone -| in quoted strings and c4--|

Can do (though I'll reword the second line).

[Keith, 2013-08-07 02:44:07]

On 2013/08/06 10:30:44, dak wrote:
> I can't really work with "the comments are confusing" 

Sorry, I meant the commit message and texidoc.

> I don't really see
> that the characterization of the commit messages is accurate.  

It was the "make special characters 'non-special'" that confused me.  The first
thing I saw was that these characters are newly matched by a pattern called
SPECIAL and act like \command, except that they don't need a backslash, and if
you do type a backslash that character is treated as part of the command name,
which all seems pretty special to me.  I began to wonder if you reversed the
arguments to 'diff' when creating the patchfile.

The removal of these special characters as special-cases in the parser is very
nice, of course.

> > It is not clear what functionality you are protecting from regression.  
>
> Redefining ) in a manner that changes it from post-event to standalone event:
> previous to that patch, the syntax of ) was fixed.

Or redefining ) at all.  Probably a regression test is appropriate, then,
showing a couple cases to illustrate where the \s go.
 "\\!" = <>\!
 "\\@" = -\espressivo
 "|" = {\bar"|" \set Timing.measurePosition = #(ly:make-moment 0)}
 {c'4\< g'\@  f' \! | e'1}

It was only the "changing the syntax class of articulations" that confused me. 
I thought 'syntax' referred to the language structure that the parser uses.  
  trill = <>\trill  {<>\prall c2 \trill c}
doesn't change syntax of articulations, it changes semantics of \trill. 
Originally \trill was defined to be the trill articulation, but can be redefined
to mean a chord with a trill, waiting for notes.

> https://codereview.appspot.com/12432043/diff/6001/lily/lexer.ll#newcode153
> lily/lexer.ll:153: SPECIAL		\\.|[][|()~]
> On 2013/08/05 19:21:45, Keith wrote:
> > 'special' is vague.

> I can dig into TeX terminology and call this "active", but I thought "special"
> fit rather well.  

Yes SPECIAL is probably fine.  It was mostly the conflict with my interpretation
of the commit message caused me confusion.

Maybe separate the escaped form into its own pattern, for the sake of human
readers.
It is challenging to follow in the code where all \-characters go 
  \command  produces a STRING-token "command"
  )         produces a STRING-token ")"
  \)        produces a STRING-token "\)"

[dak, 2013-08-09 16:57:34]

Rework along Keith's suggestions

[dak, 2013-08-09 17:19:38]

Fix merge conflict residue in changes.tely

[janek, 2013-08-14 11:41:43]

Hi,

i'm late with my comments but i'll publish them anyway.

A general question: what about using -' or -, for staccatissimo?  For me they
seem to be somewhat more intuitive.

Janek

https://codereview.appspot.com/12432043/diff/23001/lily/parser.yy
File lily/parser.yy (right):

https://codereview.appspot.com/12432043/diff/23001/lily/parser.yy#newcode2719
lily/parser.yy:2719: $$ = scm_from_locale_string ("Bang");
Why is this "Bang" and not "Exclaim"?

https://codereview.appspot.com/12432043/diff/23001/ly/script-init.ly
File ly/script-init.ly (right):

https://codereview.appspot.com/12432043/diff/23001/ly/script-init.ly#newcode9
ly/script-init.ly:9: dashBang = "staccatissimo"
ditto

[janek, 2013-08-14 11:43:08]

On 2013/08/14 11:41:43, janek wrote:
> A general question: what about using -' or -, for staccatissimo?  For me they
> seem to be somewhat more intuitive.

Additional advantage would be that they don't require Shift to produce on
standard keyboard.

Janek

[dak, 2013-08-14 12:01:06]

On 2013/08/14 11:43:08, janek wrote:
> On 2013/08/14 11:41:43, janek wrote:
> > A general question: what about using -' or -, for staccatissimo?  For me
they
> > seem to be somewhat more intuitive.
> 
> Additional advantage would be that they don't require Shift to produce on
> standard keyboard.
> 
> Janek

I don't really agree.  staccato is -. so it seems natural for me to have
staccatissimo as -! (I mean, it was -| before).  In addition, ' and , are
slanted in most fonts and that does not really match the visuals of
staccatissimo.

I crosschecked with the parser and it would not object to either.  The only
argument _for_ such a change would be more visual similarity rather than
mnemonic value and that rules out -, pretty definitely.  One could talk about -'
at most.  It would be sort of a nuisance to change that including all
documentation again but there would be no technical problem doing that.

[janek, 2013-08-14 12:18:12]

2013/8/14  <dak@gnu.org>:
> I don't really agree.  staccato is -. so it seems natural for me to have
> staccatissimo as -! (I mean, it was -| before).  In addition, ' and ,
> are slanted in most fonts and that does not really match the visuals of
> staccatissimo.
>
> I crosschecked with the parser and it would not object to either.  The
> only argument _for_ such a change would be more visual similarity rather
> than mnemonic value and that rules out -, pretty definitely.  One could
> talk about -' at most.  It would be sort of a nuisance to change that
> including all documentation again but there would be no technical
> problem doing that.

Hmm.
would there be a technical problem to have /both/ -! and -' as
shorthands for staccato?

Janek

[dak, 2013-08-14 13:30:24]

On 2013/08/14 12:18:12, janek wrote:
> 2013/8/14  <dak@gnu.org>:

> > I crosschecked with the parser and it would not object to either.  The
> > only argument _for_ such a change would be more visual similarity rather
> > than mnemonic value and that rules out -, pretty definitely.  One could
> > talk about -' at most.  It would be sort of a nuisance to change that
> > including all documentation again but there would be no technical
> > problem doing that.
> 
> Hmm.
> would there be a technical problem to have /both/ -! and -' as
> shorthands for staccato?

Technical?  Not really except that we'd have to make a choice of convert-ly
rule.  But I don't think we help anybody by providing two shorthands for one
particular articulation.  Anybody else with an opinion regarding -! or -' (and
of course the corresponding _! ^! _' ^' too) as the shorthand for staccatissimo,
replacing the previous -| (which would not work due to technical reasons).

[thomasmorley651, 2013-08-14 17:34:56]

On 2013/08/14 13:30:24, dak wrote:

> I don't think we help anybody by providing two shorthands for one
> particular articulation.  
+1

> Anybody else with an opinion regarding -! or -' (and
> of course the corresponding _! ^! _' ^' too) as the shorthand for
staccatissimo,
> replacing the previous -| (which would not work due to technical reasons).

I'd prefer -! to -' or -,

[Ian Hulin, 2013-08-15 09:22:12]

Hi David and Janek,
On 14/08/13 14:30, dak@gnu.org wrote:
> On 2013/08/14 12:18:12, janek wrote:
>> 2013/8/14  <dak@gnu.org>:
> 
>> would there be a technical problem to have /both/ -! and -' as 
>> shorthands for staccato?
> 
> Technical?  Not really except that we'd have to make a choice of 
> convert-ly rule.  But I don't think we help anybody by providing
> two shorthands for one particular articulation.
1+
> Anybody else with an opinion regarding -! or -' (and of course the
> corresponding _! ^! _' ^' too) as the shorthand for staccatissimo,
> replacing the previous -| (which would not work due to technical
> reasons).
> 
1+ for -'
  (' is visually mnemonic for notation staccatissimo,
   also
   ' is ergonomic on my keyboard -
     no Shift key involved, both keys on right-hand side of
     main keyboard)
Cheers,
Ian

[dak, 2013-08-15 09:33:54]

On 2013/08/15 09:22:12, Ian Hulin wrote:
> Hi David and Janek,
> On 14/08/13 14:30, mailto:dak@gnu.org wrote:
> > On 2013/08/14 12:18:12, janek wrote:
> >> 2013/8/14  <dak@gnu.org>:
> > 
> >> would there be a technical problem to have /both/ -! and -' as 
> >> shorthands for staccato?
> > 
> > Technical?  Not really except that we'd have to make a choice of 
> > convert-ly rule.  But I don't think we help anybody by providing
> > two shorthands for one particular articulation.
> 1+
> > Anybody else with an opinion regarding -! or -' (and of course the
> > corresponding _! ^! _' ^' too) as the shorthand for staccatissimo,
> > replacing the previous -| (which would not work due to technical
> > reasons).
> > 
> 1+ for -'
>   (' is visually mnemonic for notation staccatissimo,
>    also
>    ' is ergonomic on my keyboard -
>      no Shift key involved, both keys on right-hand side of
>      main keyboard)
> Cheers,
> Ian

Sigh.  What do people think countdowns are for?  This was a merge commit with
three different topics and a single application of convert-ly.

[Keith, 2013-08-15 17:18:39]

On Thu, 15 Aug 2013 02:33:55 -0700, <dak@gnu.org> wrote:
> On 2013/08/15 09:22:12, Ian Hulin wrote:
>
>> 1+ for -'
>>    (' is visually mnemonic for notation staccatissimo,
>>     also
>>     ' is ergonomic on my keyboard -
>>       no Shift key involved, both keys on right-hand side of
>>       main keyboard)

>
> Sigh.  What do people think countdowns are for?

You can't please everyone.  Don't worry about it.

'!' also looks like a staccatissimo,
is a stop character so conceptually related to the '.' for staccato,
conflicts less with the use of ''' in pitches:
   e'4-! cs-! bf-! g-! e'-! cs-! \tuplet3/2{bf-> g-> e->}
   e'4-' cs-' bf-' g-' e'-' cs-' \tuplet3/2{bf-> g-> e->}
indicates an exclamation which "Hey!" is conceptually similar to a
staccatissimo,
is easier to type on key layouts that use ''' for accents (like mine, US
deadkeys on Windows).

''' would look ridiculous when using octave checks: e'='-'

[dak, 2013-08-15 19:35:07]

"Keith OHara" <k-ohara5a5a@oco.net> writes:

> '!' also looks like a staccatissimo,
> is a stop character so conceptually related to the '.' for staccato,
> conflicts less with the use of ''' in pitches:
>   e'4-! cs-! bf-! g-! e'-! cs-! \tuplet3/2{bf-> g-> e->}
>   e'4-' cs-' bf-' g-' e'-' cs-' \tuplet3/2{bf-> g-> e->}
> indicates an exclamation which "Hey!" is conceptually similar to a
staccatissimo,
> is easier to type on key layouts that use ''' for accents (like mine,
> US deadkeys on Windows).
>
> ''' would look ridiculous when using octave checks: e'='-'

Well, we can have eis'!='-! or even eis!=-! as well, but then
octave-less octave checks always look like an oddity.  But forced
accidentals are usually less frequent than octave marks.

And I agree that in the non-contrived interlinear comparison above, the variant
using ! is less of an overall distraction.

[Ian Hulin, 2013-08-15 23:45:45]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Keith
On 15/08/13 18:18, Keith OHara wrote:
> On Thu, 15 Aug 2013 02:33:55 -0700, <dak@gnu.org> wrote:
>> On 2013/08/15 09:22:12, Ian Hulin wrote:
>> 
>>> 1+ for -' (' is visually mnemonic for notation staccatissimo, 
>>> also ' is ergonomic on my keyboard - no Shift key involved,
>>> both keys on right-hand side of main keyboard)
> 
>> 
>> Sigh.  What do people think countdowns are for?
> 
> You can't please everyone.  Don't worry about it.
> 
> '!' also looks like a staccatissimo, is a stop character so
> conceptually related to the '.' for staccato, conflicts less with
> the use of ''' in pitches: e'4-! cs-! bf-! g-! e'-! cs-!
> \tuplet3/2{bf-> g-> e->} e'4-' cs-' bf-' g-' e'-' cs-'
> \tuplet3/2{bf-> g-> e->} indicates an exclamation which "Hey!" is
> conceptually similar to a staccatissimo, is easier to type on key
> layouts that use ''' for accents (like mine, US deadkeys on
> Windows).
> 
> ''' would look ridiculous when using octave checks: e'='-'
> 
> 
OK.  I'm convinced, 1+ for the -! staccatissimo.

Cheers,
Ian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSDWgnAAoJEBqidDirZqASlAAH/0PCUnyVMqH/Bq90cegVHFT+
L+bd+eU4/YDDG0hKFSm7niBHvXomkJ1tOVnWYx3BaH4fy2AbHHcyV93XUxnh2i1N
5x77q1GiQaljfptWl4REkVIp4eXioM+3B3/T0pvVShvHUeF6qhwt+SjDN1KYDLRl
IidvdR8aZpcSt7uSm7cX1yEaUb8V38fSRmU47AWizbk6GAmi0n/MuYhv22dcsybz
jETXb76u8NBIS2KXx79KI+9Nl/LfAttZ/foaqoTLfOzQOQp6lHrsLwuWTiWL1lFy
iTbXCsKLXminSM6gAfqWFDzJcfferi/tKaf93k5z5S4yI++yNbMoTNWEkUcybVI=
=aBdM
-----END PGP SIGNATURE-----

[janek, 2013-08-16 22:39:59]

2013/8/15  <dak@gnu.org>:
> "Keith OHara" <k-ohara5a5a@oco.net> writes:
>
>> '!' also looks like a staccatissimo,
>> is a stop character so conceptually related to the '.' for staccato,
>> conflicts less with the use of ''' in pitches:
>>    e'4-! cs-! bf-! g-! e'-! cs-! \tuplet3/2{bf-> g-> e->}
>>    e'4-' cs-' bf-' g-' e'-' cs-' \tuplet3/2{bf-> g-> e->}
>> indicates an exclamation which "Hey!" is conceptually similar to a
>> staccatissimo,
>> is easier to type on key layouts that use ''' for accents (like mine,
>> US deadkeys on Windows).
>> ''' would look ridiculous when using octave checks: e'='-'
>
> Well, we can have eis'!='-! or even eis!=-! as well, but then
> octave-less octave checks always look like an oddity.  But forced
> accidentals are usually less frequent than octave marks.
>
> And I agree that in the non-contrived interlinear comparison above, the
> variant using ! is less of an overall distraction.

ok, i'm convinced.  And my question was of the "musing-kind" anyway.

Jaenk