New alist to replace special characters.

New alist to replace special characters.

[Bertrand Bordage, 2011-05-23 10:28:34]

A feature that helps text typesetting.

Bertrand

[Bertrand Bordage, 2011-05-23 10:38:44]

This prints every character with its shorthand, but works bad because of '\t',
'\v' and '\n' :

\version "2.15.0"
\include "special-characters.ly"
#(define-markup-list-command (show-special-characters layout props) ()
                             (interpret-markup-list layout props
                               (map (lambda (pair)
                                            (markup #:fill-line (
                                                #:null
                                                #:override
                                                 
'(replacement-string-max-length . 0)
                                                (car pair)
                                                (car pair)
                                                #:null)))
                                    special-characters-alist)))
\markuplines \show-special-characters

[mike_apollinemike.com, 2011-05-23 10:57:01]

On May 23, 2011, at 12:38 PM, bordage.bertrand@gmail.com wrote:

> This prints every character with its shorthand, but works bad because of
> '\t', '\v' and '\n' :
> 
> \version "2.15.0"
> \include "special-characters.ly"
> #(define-markup-list-command (show-special-characters layout props) ()
>                             (interpret-markup-list layout props
>                               (map (lambda (pair)
>                                            (markup #:fill-line (
>                                                #:null
>                                                #:override
> 
> '(replacement-string-max-length . 0)
>                                                (car pair)
>                                                (car pair)
>                                                #:null)))
>                                    special-characters-alist)))
> \markuplines \show-special-characters


Good work!

A couple things :

(1)	If I want to write "\emdash is a command in lilypond", that won't work (I
think) with this patch.  You need to create either a ("\\\\" . "\\") special
character or another workaround for the backslash.
(2)	For ligatures that arrive at the end of the word like in some German texts,
you'll need to figure out where the ligature falls in the word.  Writing a
single exception to do this in C++ is easy, but if ligatures are any hairier
than this (i.e. if in Nabakov's Zemblan there is an ie ligature except after c),
then you'd be better off porting the whole thing to Scheme and doing it via
regular expressions or some other sorta text replacement algorithm.  The added
advantage of this is that users can specify their own rules.
(3)	Newbs will invariably learn about this feature, want to use it, and then
complain that the ligature they want is in a different font than Century
Schoolbook in the PDF.  Perhaps lilypond should issue a warning if a character
is found in a font that is not the one you currently have selected like
"Warning: Charater `foo' found, but not in the current font."

Cheers,
MS

[Bertrand Bordage, 2011-05-23 11:08:48]

(1) You can do it using \override #'(replacement-string-max-length . 0)
    But I agree, there's a problem with backslashes.
(2) What if we define something like "\tz" when we want 'ꜩ' and just write "tz"
when no ligature is needed ? This isn' t so constraining.
(3) I know, this is another change to be done. And when this patch will be
pushed, I'll take time to add a doc entry.

Bertrand

[mike_apollinemike.com, 2011-05-23 11:14:04]

On May 23, 2011, at 1:08 PM, bordage.bertrand@gmail.com wrote:

> (1) You can do it using \override #'(replacement-string-max-length . 0)
>    But I agree, there's a problem with backslashes.

I am the worst person in the world for advice on the most straightforward way to
do something...I'll let others chime in on how to solve this problem.  You may
want to note as well in the docs that if someone puts:

\markup { \italic { foo \emdash bar } }

Lilypond will think \emdash is a function.  They'd need to do: \markup { \italic
{ foo "\emdash" bar } }

> (2) What if we define something like "\tz" when we want 'ꜩ' and just
> write "tz" when no ligature is needed ? This isn' t so constraining.

This sounds like a good idea!

> (3) I know, this is another change to be done. And when this patch will
> be pushed, I'll take time to add a doc entry.
> 

Good stuff.

Cheers,
Mike

[Reinhold, 2011-05-23 11:49:04]

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly
File ly/special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode1
ly/special-characters.ly:1: #(define special-characters-alist
Is there any (public) command to add a replacement to that alist (or remove
one)?
Or do you have to repeat the whole list if you want to add/remove just one
entry?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode14
ly/special-characters.ly:14: ("\nbhyphen" . "‑")
\\nbhyphen?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode36
ly/special-characters.ly:36: ("_" . " ") ; thinspace
Will this work with words that have a wanted underscore? Or will the underscore
always be replaced with a thinspace?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode41
ly/special-characters.ly:41: ("\textbullet" . "•")
\\textbullet ?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode80
ly/special-characters.ly:80: ("\ng" . "ŋ")
Should that be \\ng?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode85
ly/special-characters.ly:85: ("\\OE" . "Œ")
Is the replacement always done on the longest string found? In particular, if
you wan to write ØE (Øe is a danish name apparently), you'll have to use \\OE,
which might or might not be matched with the sequence for Πinstead...

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode88
ly/special-characters.ly:88: ("\th" . "þ")
\\th?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode94
ly/special-characters.ly:94: ("\times" . "×")
\\times

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode97
ly/special-characters.ly:97: ("\twosuperior" . "²")
\\twosupererior

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode98
ly/special-characters.ly:98: ("\threesuperior" . "³")
\\threesuperior ?

[Carl, 2011-05-23 14:01:57]

http://codereview.appspot.com/4553056/diff/1/lily/text-interface.cc
File lily/text-interface.cc (right):

http://codereview.appspot.com/4553056/diff/1/lily/text-interface.cc#newcode40
lily/text-interface.cc:40: int max_length = scm_to_int (ly_chain_assoc_get
(ly_symbol2scm ("replacement-string-max-length"),
Why is string-max-length needed?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly
File ly/special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode1
ly/special-characters.ly:1: #(define special-characters-alist
I think this should be a .scm file, rather than a .ly file.

I'm concerned about the default position of moving from UTF-* to ascii for
special characters.  I think that's moving in the wrong direction.

I do think the ligature replacement is very good.

But I could easily be persuaded that this is the right thing to do.

[Bertrand Bordage, 2011-05-23 21:32:26]

Response to Reinhold and Carl's concerns.

http://codereview.appspot.com/4553056/diff/1/lily/text-interface.cc
File lily/text-interface.cc (right):

http://codereview.appspot.com/4553056/diff/1/lily/text-interface.cc#newcode40
lily/text-interface.cc:40: int max_length = scm_to_int (ly_chain_assoc_get
(ly_symbol2scm ("replacement-string-max-length"),
It is needed line 50.
This can be inferred from "replacement-alist".
But with this property we can disable every shorthand or just those longer than
replacement-string-max-length.

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly
File ly/special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode1
ly/special-characters.ly:1: #(define special-characters-alist
@Reinhold :
Of course, otherwise it would be seriously annoying :o)
It is called "additional-replacement" in ly/parper-defaults-init.ly.
There is an example in the regression test.

@Carl :
I agree for the .scm file.
Why do you think it is going in the wrong direction ?
The main issue I see is that it makes the syntax messy.
This can easily be improved:
see the following response to Reinhold.

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode14
ly/special-characters.ly:14: ("\nbhyphen" . "‑")
No, because '\' is sometimes an escape character (in '\t', '\n' and '\v').
\markup "\n" si the same as (markup "\n")
whereas \markup "\g (or anything else)" that means (markup "\\g").
This is why there's only one backslash on the lines you commented.

There's also other problems with this use of the backslash :
as Mike reported, \markup { \solidus } or \lyricmode { \solidus } won't work.
LilyPond will interpret \solidus as being a markup command.

And this will be hard for the user to understand the difference between a
command and an alias.

That's why I suggest this : use another escape character for these special
characters. Do you think '@' is a good idea?
Beside this, I will introduce this idea of "alias" in the doc.
Do you agree ?

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode36
ly/special-characters.ly:36: ("_" . " ") ; thinspace
No, but there's '\_' line 55.
If I remember well, thinspace is '~' in LaTeX.
But the tilde is also used for tied-lyrics...
A better idea would be welcome !

http://codereview.appspot.com/4553056/diff/1/ly/special-characters.ly#newcode85
ly/special-characters.ly:85: ("\\OE" . "Œ")
Yes, the replacement always takes the longest string found.

For these characters, I just copied LaTeX since I don't use them (except æ and œ
who already have a shortcut in linux's french keyboard).
But LaTeX may be wrong for this.
The international community will have to improve and complete this table.

Another detail : every character in this alist is defined in Century Schoolbook.

[c_sorensen, 2011-05-23 21:54:53]

On 5/23/11 3:32 PM, "bordage.bertrand@gmail.com"
<bordage.bertrand@gmail.com> wrote:

> http://codereview.appspot.com/4553056/diff/1/lily/text-interface.cc#newcode40
> lily/text-interface.cc:40: int max_length = scm_to_int
> (ly_chain_assoc_get (ly_symbol2scm ("replacement-string-max-length"),
> It is needed line 50.
> This can be inferred from "replacement-alist".
> But with this property we can disable every shorthand or just those
> longer than replacement-string-max-length.

The ideas of disabling shorthands by the length of the replacement string
seems very hackish to me.  It seems unlikely to me that I would ever decide
which replacement strings to disable by the length.

To disable all replacement strings, we should have something like the
following functionality

#(define default-string-replacement-alist '(;  This would be your list
                            ))


to enable replacements:

\override TextScript #'replacement-alist =
#'default-string-replacement-alist

to disable replacements:

\override TextScript #'replacement-alist = #'()_


> 
> @Carl :
> I agree for the .scm file.
> Why do you think it is going in the wrong direction ?

Because we are making up our own syntax for characters that are already
defined as UTF characters.  We should avoid making up syntax if at all
possible.

For the ligatures, I'm supportive; we don't have a text processor that will
automatically handle ligatures.

For the special characters like trademark, and super2, we should just have
them inserted into the file as UTF characters, IMO.

I don't think we should be trying to (for example) handle TeX input in
LilyPond strings.  I see that as a maintenance pain in the neck.

We're moving in the direction (if we're not there already) of correctly
handling UTF text.  Why not just let the person who wants to use UTF text do
so?  What is the advantage to typing \\aa instead of å?

Thanks,

Carl

[Bertrand Bordage, 2011-05-23 22:15:38]

Yes.
Knowing this, I suggest we keep whitespaces, punctuation, quotes and word
dividers (with some small changes).

There's still something that bothers me : isn't there some special characters
that you can't do with you keyboard ?
Even on linux I can't type some symbols like ſ or • without copying/pasting.
So, to your question "What is the advantage to typing \\aa instead of å?" I
answer "I don't have any key for this".

Cheers,
Bertrand

[Carl, 2011-05-23 22:34:53]

On 2011/05/23 22:15:38, Bertrand Bordage wrote:
> Yes.
> Knowing this, I suggest we keep whitespaces, punctuation, quotes and word
> dividers (with some small changes).
> 
> There's still something that bothers me : isn't there some special characters
> that you can't do with you keyboard ?
> Even on linux I can't type some symbols like ſ or • without copying/pasting.
> So, to your question "What is the advantage to typing \\aa instead of å?" I
> answer "I don't have any key for this".

See the Wikipedia entry on Unicode input for ways to get characters that aren't
on your keyboard:

 <http://en.wikipedia.org/wiki/Unicode_input>


HTH,

Carl

[Bertrand Bordage, 2011-05-23 23:11:33]

...definitely not user-friendly!
I totally agree it's better to type in UTF and that's what I always did with
LilyPond. But this REALLY wastes time when we type several symbols and
ligatures.
Making this easier should be the OS's job.
I'll think more about this and make another patch set.

Bertrand

[Carl, 2011-05-23 23:28:02]

On 2011/05/23 23:11:33, Bertrand Bordage wrote:
> Making this easier should be the OS's job.

Yes, I agree.  That's why I'm not in favor of making it part of LilyPond for us
to maintain.

Having the facility to do the general substitution as part of LilyPond is fine,
IMO.

Having the list of all the Unicode characters is not a good idea.  Having that
list as an LSR snippet would be OK.  That way it's not part of the contract we
have with users to maintain it.

Thanks,

Carl

[Bertrand Bordage, 2011-06-03 11:11:57]

New patch set with Carl's ideas.
I don't know if replacement-alist should be defined by default.
Nor if '&' is a good escape character. I chose it because it's quicker to type
on a french keyboard than '@'.

Bertrand.

[Keith, 2011-06-03 18:42:23]

Philosophically, utf-8 is the One True Way, but I am sympathetic to laxy
typists.

On 2011/06/03 11:11:57, Bertrand Bordage wrote:
> I don't know if replacement-alist should be defined by default.
The shorter the default list, the fewer surprises.
The keys should not include any possibly-desired input.

> Nor if '&' is a good escape character.
It is reasonable, but draw from an existing string-replacement set like HTML and
use the closing ;.  
Then we will be less-likely confused when I reference a&s records (or when you
reference r&s records) in a markup on a score.

[Keith, 2011-06-03 18:47:24]

http://codereview.appspot.com/4553056/diff/5002/scm/output-lib.scm
File scm/output-lib.scm (right):

http://codereview.appspot.com/4553056/diff/5002/scm/output-lib.scm#newcode959
scm/output-lib.scm:959: ("..." . "…")
("…" . "…")
("&emdash;" . "–")
etc.

http://codereview.appspot.com/4553056/diff/5002/scm/output-lib.scm#newcode974
scm/output-lib.scm:974: ("&ordM" . "º")
("º" . "º")
etc.

http://codereview.appspot.com/4553056/diff/5002/scm/output-lib.scm#newcode979
scm/output-lib.scm:979: ("&s" . "ſ")))
HTML just uses the unicode number ſ for long s, so maybe just &s;

[Bertrand Bordage, 2011-06-05 11:54:31]

Ok, I'll change these. What about '№' ? '№', 'N°' or '&N°;' ?

[Bertrand Bordage, 2011-06-06 12:13:48]

Done !
Regression tests are ok.

[Carl, 2011-06-07 03:34:43]

Patch LGTM.  I still have concerns about  maintaining a list of ascii
definitions for UTF-8 characters, as described in the comments.

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
File input/regression/markup-special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
input/regression/markup-special-characters.ly:7: #(define-markup-list-command
(show-special-characters layout props) ()
If we're going to do this, this code should be added to the appendices of the
Notation Reference, so it will be included in the documentation.  I don't really
think it should be part of the regression test suite, but I could be convinced
otherwise.

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm
File scm/output-lib.scm (right):

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm#newcode958
scm/output-lib.scm:958: ; Punctuation
I'm still not convinced that we should get in the business of defining these
ascii-based characters instead of using UTF-8 directly as part of the core
lilypond distribution.  But if nobody else is opposed to it then I will not
maintain my objection and I will agree with the patch.

I have *no* objections to including the functionality with whitespace defined in
the default-string-replacement-alist, and having an LSR snippet with additional
replacements.

[Bertrand Bordage, 2011-06-07 09:45:12]

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
File input/regression/markup-special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
input/regression/markup-special-characters.ly:7: #(define-markup-list-command
(show-special-characters layout props) ()
It was a way to remember this piece of code. I will not do all the documentation
before deciding if the list remains.
Anyway, there are other examples of such lists in regression testing.
For example, 'chord-name-entry' or 'woodwind-diagrams-empty'.

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm
File scm/output-lib.scm (right):

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm#newcode958
scm/output-lib.scm:958: ; Punctuation
I understand your concerns and I'm a little embarrassed to introduce a new
syntax for the risk of disrupting the users understanding. But it also lacks a
lot of people.
In the list, I added the characters which are not allocated in the French Linux
keyboard (except the ellipsis, the non-breaking space and the dashes).
It does bother me however to remove this list.

[MikeSol, 2011-06-07 09:53:26]

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm
File scm/output-lib.scm (right):

http://codereview.appspot.com/4553056/diff/9003/scm/output-lib.scm#newcode958
scm/output-lib.scm:958: ; Punctuation
On 2011/06/07 09:45:12, Bertrand Bordage wrote:
> I understand your concerns and I'm a little embarrassed to introduce a new
> syntax for the risk of disrupting the users understanding. But it also lacks a
> lot of people.
> In the list, I added the characters which are not allocated in the French
Linux
> keyboard (except the ellipsis, the non-breaking space and the dashes).
> It does bother me however to remove this list.

It may be worth testing this out on multiple platforms - I've sometimes had
difficulty opening up my .ly files on Windows machines if they contain special
characters.  To what extent is UTF-8 easy to work with outside of a posix
environment?

In general, one way to work this into LilyPond may be to include the basic
functionality, send the string-replacement-alist out to lilypond-user, and see
to what extent people desire to use it.  I've done this for a couple things and
been able to scrap/add functionalities accordingly.  If it seems like it's
something that many people would want to use, then it should probably be part of
the distro.

[nicolas.sceaux, 2011-06-07 20:19:56]

A replacement function for text is a very good idea, and would be very useful.

General comments:
A regtest for lyrics might be missing.

The current implementation of replacement function, in the C++ part, need some
rework.

An alternate design might be:
Define in the guile part the replacement function. The C++ code would just call
that function on the string. By default, the replacement function could do
exactly the job that is done in current LilyPond version (replacing line feeds,
etc, by spaces).
Another replacement function would be proposed, doing the work you propose here,
e.g. compiling and applying a regexp, instead of manually implementing the
replacement (in a possibly less efficient way than the built-in would do).
And a \useTheFancyTextReplacementThing switch could select the later replacement
function.
(just brainstorming)

If the second replacement function has not much impact on performances, it may
even be the default.

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
File input/regression/markup-special-characters-shorthands.ly (right):

http://codereview.appspot.com/4553056/diff/9003/input/regression/markup-speci...
input/regression/markup-special-characters-shorthands.ly:15:
(additional-replacements text-font-defaults
The interface for adding replacements/shortcuts seems a bit complicated.
Maybe a function call modifying text-font-defaults behind the scene would be
better?

#(add-text-replacements! '(...))

http://codereview.appspot.com/4553056/diff/9003/lily/text-interface.cc
File lily/text-interface.cc (right):

http://codereview.appspot.com/4553056/diff/9003/lily/text-interface.cc#newcode46
lily/text-interface.cc:46: for (int j = 0; j < scm_to_int (scm_length
(replacement_alist)); j++)
This is not the way to loop over a list: the idiom can be found many times in
the C++ code. scm_length and scm_list_ref are O(n).

What's more, at this point the `str' argument is not yet used. So maybe this
could be cached to avoid being computed again each time.

http://codereview.appspot.com/4553056/diff/9003/lily/text-interface.cc#newcode62
lily/text-interface.cc:62: str->replace (i, j, ligature);
We see three embedded loops: while, for, and a call to ly_assoc_get... The
previous add only one loop.
What is the impact on performance, wrt to current version?

Have you considered compiling (and caching) a regexp and use the appropriate
builtin function which will do all the replacement work?

Or maybe... get this remplacement part out of C++ code, and define in the scheme
part a replacement function, that will be called here. A default replacement
function might do the \n, \t \v replacement alone (as on current version). An
other replacement function would be proposed which does more complicated things.

[nicolas.sceaux, 2011-06-07 20:49:47]

http://codereview.appspot.com/4553056/diff/9003/lily/text-interface.cc
File lily/text-interface.cc (right):

http://codereview.appspot.com/4553056/diff/9003/lily/text-interface.cc#newcode62
lily/text-interface.cc:62: str->replace (i, j, ligature);
On 2011/06/07 20:19:56, nicolas.sceaux wrote:
> Have you considered compiling (and caching) a regexp and use the appropriate
> builtin function which will do all the replacement work?

oh, probably there is no builtin regexp replacement function that modifies its
argument string to avoid the consing... sorry for the noise.

[Bertrand Bordage, 2011-07-19 18:33:20]

Patch updated.
I abandonned the idea of including a special characters list.

[Carl, 2011-07-19 19:19:03]

LGTM.

You could add a .ly file that defined a more comprehensive set of special
characters as an optional item that the user includes with \include.  That would
allow its use without requiring its use.

THanks,

Carl

[Bertrand Bordage, 2011-07-19 20:22:49]

A new file in ly with definitions using "add-text-replacements!"?
Like this?

germanHTML = #(add-text-replacements! '(("&ss;" . "ß")
                                        [...])

englishLaTeX = [...]

This way, users can easily stack special characters with this syntax:
\paper {
  \englishHTML
  \frenchLigatures
}

Besides, I made some performance tests.
With a big replacement alist, building Nicolas Sceaux's huge Atys takes 0.7s
more than without it.

Thanks,
Bertrand

[pkx166h, 2011-07-19 20:50:26]

passes make and reg tests

[Bertrand Bordage, 2011-07-31 17:41:21]

I updated the patch.

There's now a list of special characters and a \replace command for markups.

The escape character is now '§'.  It's the only one that works great with
lyrics.  And it isn't used elsewhere in LilyPond's syntax.

The syntax for using the list of special characters
#(include-special-characters) is a bit ackward but I didn't found anything
better.

Regards,
Bertrand

[pkx166h, 2011-07-31 19:11:52]

passes make and reg tests

[c_sorensen, 2011-07-31 22:37:42]

On 7/31/11 11:41 AM, "bordage.bertrand@gmail.com"
<bordage.bertrand@gmail.com> wrote:

> I updated the patch.
> 
> There's now a list of special characters and a \replace command for
> markups.
> 
> The escape character is now '§'.  It's the only one that works great
> with lyrics.  And it isn't used elsewhere in LilyPond's syntax.

This escape is not available on US keyboards, so it's as much work to use
the escape character as to put in the UTF character directly.  For me, as a
US user, that seems to negate the purpose for the patch.

What was the problem with the previous escape characters?

Thanks,

Carl

[Bertrand Bordage, 2011-07-31 23:09:51]

Ouch...

The problem with '&' is that it fails on lyrics:
this works:
\new Lyrics \lyricmode { a&s; }

but this doesn't:
\new Lyrics \lyricmode { &s; }

There is the same kind of issues with almost every easy-to-type special
character:
@ % $ # \ / < > ^ ~ + = * ; ( ) [ ] { }

This is the exact list of every possible escape characters for a french keyboard
under linux:
§ £ µ € ¤ ¹ ² ' ` © ¨ ø Ø ≤ ≥ « » “ ” ↓ ¬ × ÷ ¿ ¡ ∕ ⋅ …

Of course, only a few of them are available under windows:
§ £ µ € ¤ ' `

If we consider US, british, german and spanish keyboards, this only gives these
two very bad characters:
' `

The best solution may be to solve this lyric problem first :\

Bertrand

[Bertrand Bordage, 2011-08-04 13:06:59]

lily/text-interface.cc updated.

[Bertrand Bordage, 2011-09-15 22:14:50]

New patch set.
'&' and '@' have been added to the lexer.
'&' is therefore a perfectly working escape character in lyrics.

Bertrand

[pkx166h, 2011-09-16 21:31:25]

Hello, passes make but fails make check

please see

http://code.google.com/p/lilypond/issues/detail?id=1891#c1

James

[pkx166h, 2011-09-16 21:34:37]

On 2011/09/16 21:31:25, J_lowe wrote:
> Hello, passes make but fails make check
> 
Sorry ignore this, I just realised I made a mistake in my make check. I'll redo
now.

James

[pkx166h, 2011-09-16 21:48:28]

passes make and reg tests

[MikeSol, 2011-09-20 05:22:16]

Hey Bertrand,

Could you show an example, in the regtest, of how one would write:

–

(or whatever) such that – appeared (meaning the string "–" - not the
replacement n-dash).

[Bertrand Bordage, 2011-09-20 10:18:41]

Hey!

I'm currently writing a doc entry that explains how to use replacements. I have
a few questions:

Where do you think I should put it? In NR 1.8.1 or 1.8.2?

Do you think I have to move the table from the regtest to the Appendix A (and
keep the rest of the regtest as a regtest)?

Bertrand

PS: I'm also writing a changelog entry.

[pkx166h, 2011-09-20 12:07:31]

On 2011/09/20 10:18:41, Bertrand Bordage wrote:
> Hey!
> 
> I'm currently writing a doc entry that explains how to use replacements. I
have
> a few questions:
> 
> Where do you think I should put it? In NR 1.8.1 or 1.8.2?

Hmm.. I'd say 3.3.3 actually

http://lilypond.org/doc/v2.14/Documentation/notation/text-encoding

1.8.x are how to format text (bold italic etc) not to use special chars which
this seems to be and which NR 3.3.3 is all about.

> 
> Do you think I have to move the table from the regtest to the Appendix A (and
> keep the rest of the regtest as a regtest)?

I am not experienced enough to answer this, but which Appendix A did you intend?
A new one or add to an existing as I cannot see where this would be appropriate.
> 
> Bertrand
> 
> PS: I'm also writing a changelog entry.

[Bertrand Bordage, 2011-09-20 12:11:46]

On 2011/09/20 12:07:31, J_lowe wrote:
> > Where do you think I should put it? In NR 1.8.1 or 1.8.2?
> 
> Hmm.. I'd say 3.3.3 actually

Oh, yes! This is better.

> > Do you think I have to move the table from the regtest to the Appendix A
(and
> > keep the rest of the regtest as a regtest)?
> 
> I am not experienced enough to answer this, but which Appendix A did you
intend?
> A new one or add to an existing as I cannot see where this would be
appropriate.

In a new one called "Special characters", just after "Text markup list
commands".

[Bertrand Bordage, 2011-09-20 16:26:22]

New patch set including some documentation work.

[Trevor Daniels, 2011-09-20 16:54:08]

A few comments following a quick look at the doc changes

Trevor

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
File Documentation/notation/input.itely (right):

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1591: @cindex UTF-8
Better to index these under the appropriate subsubsec

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1597: @end menu
I think I'd prefer the order to be:

* Text encoding:: 
* Unicode:: 
* ASCII aliases:: 

That way we work from the general to the more specific,
rather than the other way round.

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1625: See the complete @ref{List of special
characters}.
Should also have a See also:

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1636: @end lilypond
Didn't we discuss earlier using \include?
If this works it should be mentioned too.

[Bertrand Bordage, 2011-09-20 17:09:47]

Hmm, I don't understand one comment, but I agree with the others.

Thanks,
Bertrand

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
File Documentation/notation/input.itely (right):

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1636: @end lilypond
What do you mean?

[Trevor Daniels, 2011-09-20 21:14:45]

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
File Documentation/notation/input.itely (right):

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1636: @end lilypond
On 2011/09/20 17:09:48, Bertrand Bordage wrote:
> What do you mean?

See comments 25 and 26.

If this works (I presume it does):

germanHTML = #(add-text-replacements! '(("&ss;" . "ß")
                                        [...]))
frenchLigatures = [...]

\paper {
  \germanHTML
  \frenchLigatures
}

it would be good to use this form in the snippet and
comment that the definitions can be placed in ly files 
which can be \included.

[t.daniels_treda.co.uk, 2011-09-20 22:08:56]

<bordage.bertrand@gmail.com>
Tuesday, September 20, 2011 11:18 AM

> Do you think I have to move the table from the regtest to the 
> Appendix A
> (and keep the rest of the regtest as a regtest)?

Yes.  Between the current A10 and A11 looks a good place.

Trevor



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3908 - Release Date: 09/20/11

[Bertrand Bordage, 2011-09-20 22:11:15]

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
File Documentation/notation/input.itely (right):

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/inpu...
Documentation/notation/input.itely:1636: @end lilypond
Yes, unfortunately, this isn't simple. I didn't find any technical way to
achieve this.

[Graham Percival (old account), 2011-09-20 22:37:37]

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/nota...
File Documentation/notation/notation-appendices.itely (right):

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/nota...
Documentation/notation/notation-appendices.itely:909: @example
blank line before @example and after @end example

http://codereview.appspot.com/4553056/diff/103001/Documentation/notation/nota...
Documentation/notation/notation-appendices.itely:912: to a @code{\paper} block.
See @ref{ASCII aliases} for more details.
We're not allowed to write
  see @ref{} for more details
according to the texinfo spec.  Please change to:
  For more details, see @ref{}.

Come to think of it, rewrite the whole paragraph to keep the @example after the
text:

The following special characters references can be used; for more details, see
@ref{ASCII aliases}.

@example
...
@end example

[Bertrand Bordage, 2011-09-21 16:16:23]

New patch set.
I hope this is ready for to be pushed, now.

[Neil Puttock, 2011-09-21 21:10:09]

http://codereview.appspot.com/4553056/diff/106003/Documentation/included/spec...
File Documentation/included/special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/106003/Documentation/included/spec...
Documentation/included/special-characters.ly:1: \version "2.15.12"
2.15.13

http://codereview.appspot.com/4553056/diff/106003/Documentation/included/spec...
Documentation/included/special-characters.ly:8: #(define-markup-list-command
(show-special-characters layout props) ()
fix indentation (nearly every line)

http://codereview.appspot.com/4553056/diff/106003/Documentation/included/spec...
Documentation/included/special-characters.ly:13: #:override '(replacement-alist
. ()) (car pair)
indentation is artificially compressed (aligns with the dangling parenthesis)

http://codereview.appspot.com/4553056/diff/106003/Documentation/notation/inpu...
File Documentation/notation/input.itely (right):

http://codereview.appspot.com/4553056/diff/106003/Documentation/notation/inpu...
Documentation/notation/input.itely:1762: \new Staff { \repeat unfold 9 a' }
a'4

http://codereview.appspot.com/4553056/diff/106003/Documentation/notation/nota...
File Documentation/notation/notation-appendices.itely (right):

http://codereview.appspot.com/4553056/diff/106003/Documentation/notation/nota...
Documentation/notation/notation-appendices.itely:912: The rest of them are
inspired of LaTeX.
inspired by @LaTeX{}

http://codereview.appspot.com/4553056/diff/106003/input/regression/markup-spe...
File input/regression/markup-special-characters.ly (right):

http://codereview.appspot.com/4553056/diff/106003/input/regression/markup-spe...
input/regression/markup-special-characters.ly:1: \version "2.15.12"
2.15.13

http://codereview.appspot.com/4553056/diff/106003/input/regression/markup-spe...
input/regression/markup-special-characters.ly:4: A list of special characters
ASCII aliases can be easily included.
character

http://codereview.appspot.com/4553056/diff/106003/input/regression/markup-spe...
input/regression/markup-special-characters.ly:5: This works for markups and
lyrics.
remove leading spaces

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc
File lily/text-interface.cc (right):

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:43: if (!to_boolean (scm_list_p (replacement_alist))
!ly_is_list

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:44: || to_boolean (scm_null_p (replacement_alist)))
scm_is_null

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:51: (scm_string_length (scm_caar (s))));
fix indent (run through fixcc.py)

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:56: for (int j = max_length; j > 0; j--)
(vsize j = max_length; j--;)

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:60: (ly_assoc_get (ly_string2scm (dummy),
fix indent

http://codereview.appspot.com/4553056/diff/106003/lily/text-interface.cc#newc...
lily/text-interface.cc:61: replacement_alist, SCM_BOOL_F), "");
fix indent

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly
File ly/text-replacements.ly (right):

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:20: #(define (add-text-replacements! alist)
fix indentation

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:26: (add-text-replacements!
fix indentation

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:27: '(; Punctuation
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:36: ; French, German and English quotes open/close
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:50: ; Word dividers
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:60: ; General typography
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:77: ; Diacritics
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:88: ; Non-ASCII Letters (Excluding Accented Letters)
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:110: ; Mathematical symbols
;;

http://codereview.appspot.com/4553056/diff/106003/ly/text-replacements.ly#new...
ly/text-replacements.ly:127: ; Currency symbols
;;

http://codereview.appspot.com/4553056/diff/106003/scm/define-grob-properties.scm
File scm/define-grob-properties.scm (right):

http://codereview.appspot.com/4553056/diff/106003/scm/define-grob-properties....
scm/define-grob-properties.scm:685: The key is a string of the pattern to be
replaced. The value is a
two spaces following full stop/period

http://codereview.appspot.com/4553056/diff/106003/scm/define-grob-properties....
scm/define-grob-properties.scm:686: string of what should be displayed. Useful
for ligatures.")
two spaces following full stop/period

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands.scm
File scm/define-markup-commands.scm (right):

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:2161: layout
indent (one space)

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:2163: `((baseline-skip . ,(* magnification
ref-baseline))
indent (one space)

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:2187: layout
indent

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:2189: `((baseline-skip . ,(* baseline-skip
(magstep increment)))
indent

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:3714: layout
indent

http://codereview.appspot.com/4553056/diff/106003/scm/define-markup-commands....
scm/define-markup-commands.scm:3716: props
indent

http://codereview.appspot.com/4553056/diff/106003/scm/output-lib.scm
File scm/output-lib.scm (right):

http://codereview.appspot.com/4553056/diff/106003/scm/output-lib.scm#newcode996
scm/output-lib.scm:996: ;; text replacements
These are misplaced here.  output-lib.scm is for grob callbacks.

http://codereview.appspot.com/4553056/diff/106003/scm/output-lib.scm#newcode999
scm/output-lib.scm:999: '(; Whitespaces
;; (`;' is for margin comments)

fix indent

http://codereview.appspot.com/4553056/diff/106003/scm/output-lib.scm#newcode1007
scm/output-lib.scm:1007: (append dummy-replacements alist)))
indent

http://codereview.appspot.com/4553056/diff/106003/scm/output-lib.scm#newcode1008
scm/output-lib.scm:1008: (prepend-alist-chain 'replacement-alist
new-replacements props)))
indent (aligns with `e' in let*)

[Bertrand Bordage, 2011-09-21 22:13:12]

Thanks a lot, Neil.
Could you have a last look at the Scheme files?
I'm not sure of the indentation.

I created a new scm/text.scm file for the definitions I couldn't put elsewhere.

Bertrand

[Bertrand Bordage, 2011-09-24 10:50:00]

Pushed as 688f5f1711d8ca07338385a2ae0191b1a8aae315.