Remove `-dgs-load-fonts' in lilypond-book

Issue 4882: Remove `-dgs-load-fonts' in lilypond-book

Some fonts cannot use with `-dgs-load-fonts' option
(also `-dgs-load-lily-fonts' option).

[trueroad, 2016-06-11 08:40:12]

Remove `-dgs-load-fonts' in lilypond-book

[lemzwerg, 2016-06-11 08:47:31]

LGTM – please update the title of this Rietveld issue

[dak, 2016-06-11 08:56:19]

On 2016/06/11 08:40:12, trueroad wrote:
> Remove `-dgs-load-fonts' in lilypond-book

The problem I have with that behavior is the ridiculous amount of disk space it
consumes.  I believe we use this for building the LilyPond docs already, and I
need about 4GB of free disk space to run a build with docs right now.  4GB. 
Several small snippets chip in with 20MB of size.

So the combination of embedding all fonts and apparently not subsetting them is
causing very large disk space requirements.  I don't think this is a long-term
feasible solution and I would not be surprised if a lot of automatically running
package builders would abort building LilyPond because of suspiciously high
memory requirements.

So the question is whether we can get gs-load-fonts and/or font subsetting
working in a manner where the end product does not contain multiple font copies.

One option I could imagine would be writing a multipage PS file getting
converted into a multipage PDF file which gets included once and then just has
the individual objects referenced as they occur.  In that case, we would no
longer be working with the "database" approach, meaning that the special case of
multiple translations of documents containing a large amount of repeated
snippets would result in quite more LilyPond work, but quite less Ghostscript
work.  That would be a net loss of performance but offset by making things
considerably simpler conceptually.

Another option is working subsetting of fonts such that they _will_ combine
properly.  I mean, other people manage, don't they?

[lemzwerg, 2016-06-11 09:36:23]

What about creating intermediate PDFs that don't embed fonts and deleting the
intermediate EPS files?

[trueroad, 2016-06-11 14:08:06]

Without `--bigpdfs`, intermediate PDFs and final PDFs are embedded subsetting
fonts.
They are not embedded full fonts.

In other words, the PDFs file size is not affected with or without
`-dgs-load-fonts` option.
Here is the total size of intermediate PDFs.

---Original (with -dgs-load-fonts, without --bigpdfs)---
$ find . -name "lily-*.pdf" -print0 | du -hc --files0-from=- | tail
12K     ./out/lybook-testdb/bd/lily-aba28542.pdf
12K     ./out/lybook-testdb/c8/lily-a527a091-1.pdf
12K     ./out/lybook-testdb/c8/lily-a527a091.pdf
12K     ./out/lybook-testdb/c9/lily-08ef596f-1.pdf
12K     ./out/lybook-testdb/c9/lily-08ef596f.pdf
12K     ./out/lybook-testdb/d4/lily-b457fc5b-1.pdf
12K     ./out/lybook-testdb/d4/lily-b457fc5b.pdf
12K     ./out/lybook-testdb/ea/lily-33e3543e-1.pdf
12K     ./out/lybook-testdb/ea/lily-33e3543e.pdf
265M    total
--------------------------------------------------------

---Patch Set 2 (without -dgs-load-fonts, without --bigpdfs)---
$ find . -name "lily-*.pdf" -print0 | du -hc --files0-from=- | tail
12K     ./out/lybook-testdb/bd/lily-aba28542.pdf
12K     ./out/lybook-testdb/c8/lily-a527a091-1.pdf
12K     ./out/lybook-testdb/c8/lily-a527a091.pdf
12K     ./out/lybook-testdb/c9/lily-08ef596f-1.pdf
12K     ./out/lybook-testdb/c9/lily-08ef596f.pdf
12K     ./out/lybook-testdb/d4/lily-b457fc5b-1.pdf
12K     ./out/lybook-testdb/d4/lily-b457fc5b.pdf
12K     ./out/lybook-testdb/ea/lily-33e3543e-1.pdf
12K     ./out/lybook-testdb/ea/lily-33e3543e.pdf
269M    total
--------------------------------------------------------------

Ofcource, the intermediate EPSs file size is affected with or without
`-dgs-load-fonts` option.
Here is the total size of intermediate EPSs.

---Original (with -dgs-load-fonts, without --bigpdfs)---
$ find . -name "lily-*.eps" -print0 | du -hc --files0-from=- | tail
80K     ./out/lybook-testdb/ff/lily-46213804-1.eps
80K     ./out/lybook-testdb/ff/lily-46213804-2.eps
84K     ./out/lybook-testdb/ff/lily-46213804.eps
164K    ./out/lybook-testdb/ff/lily-907f83e3-1.eps
160K    ./out/lybook-testdb/ff/lily-907f83e3-2.eps
160K    ./out/lybook-testdb/ff/lily-907f83e3-3.eps
176K    ./out/lybook-testdb/ff/lily-907f83e3.eps
160K    ./out/lybook-testdb/ff/lily-fcaab206-1.eps
160K    ./out/lybook-testdb/ff/lily-fcaab206.eps
899M    total
--------------------------------------------------------

---Patch Set 2 (without -dgs-load-fonts, without --bigpdfs)---
$ find . -name "lily-*.eps" -print0 | du -hc --files0-from=- | tail
148K    ./out/lybook-testdb/ff/lily-46213804-1.eps
148K    ./out/lybook-testdb/ff/lily-46213804-2.eps
148K    ./out/lybook-testdb/ff/lily-46213804.eps
336K    ./out/lybook-testdb/ff/lily-907f83e3-1.eps
332K    ./out/lybook-testdb/ff/lily-907f83e3-2.eps
332K    ./out/lybook-testdb/ff/lily-907f83e3-3.eps
348K    ./out/lybook-testdb/ff/lily-907f83e3.eps
224K    ./out/lybook-testdb/ff/lily-fcaab206-1.eps
224K    ./out/lybook-testdb/ff/lily-fcaab206.eps
4.1G    total
--------------------------------------------------------------

[dak, 2016-06-11 20:21:35]

On 2016/06/11 08:56:19, dak wrote:
> On 2016/06/11 08:40:12, trueroad wrote:
> > Remove `-dgs-load-fonts' in lilypond-book
> 
> The problem I have with that behavior is the ridiculous amount of disk space
it
> consumes.  I believe we use this for building the LilyPond docs already, and I
> need about 4GB of free disk space to run a build with docs right now.  4GB. 
> Several small snippets chip in with 20MB of size.

Ok, so I've taken a look at log files, and I have had to adjust my beliefs.  The
way it appears to me, the docs are currently built (at least partially) using
-dgs-load-fonts .  I haven't seen --bigpdf actually used.  So is gs-load-fonts
unable to do proper subsetting and causes large PDF size?

Sorry, I am completely without clue here.  The ideal behavior to me would be
that no intermediate files include any fonts at all and that the final
lilypond-book results incorporate complete but subsetted fonts, once.  But
obviously I have no clue how to achieve that state.  And I also have no clue
what our current options actually do, if any.  All I know is that the current
situation takes ridiculous amounts of disk space to build the docs.

[trueroad, 2016-06-12 10:22:28]

> Ok, so I've taken a look at log files, and I have had to adjust my beliefs. 
The
> way it appears to me, the docs are currently built (at least partially) using
> -dgs-load-fonts .  I haven't seen --bigpdf actually used.  So is gs-load-fonts
> unable to do proper subsetting and causes large PDF size?

`-dgs-load-fonts` does not manage PDF font embedding.
It also does not manage font subsetting.

It manages EPS output whether embedding font filename or embedding font glyphs.
Both EPSs are converted to PDFs which is embedded subset font glyphs.

> Sorry, I am completely without clue here.  The ideal behavior to me would be
> that no intermediate files include any fonts at all and that the final
> lilypond-book results incorporate complete but subsetted fonts, once.  But
> obviously I have no clue how to achieve that state.  And I also have no clue
> what our current options actually do, if any.  All I know is that the current
> situation takes ridiculous amounts of disk space to build the docs.

Sorry, I have no idea for achieving your ideal behavior.

But, would you like to continue to use `-dgs-load-fonts`?

It cannot load many fonts.
The intermediate PDFs and the final PDFs file size is not affected by whether
using `-dgs-load-fonts` or not.
However, the intermediate EPSs file size is affected.

[trueroad, 2016-06-13 14:00:57]

I came up with another idea.

Remove `-dgs-load-fonts` from lilypond-book default.
During LilyPond and its documents are built, `-dgs-load-lily-fonts` option
instead of `-dgs-load-fonts` is used.

`-dgs-load-lily-fonts` means "Load only the LilyPond fonts via Ghostscript."

"LilyPond fonts" mean the fonts in the LilyPond data directory.
i.e. Emmentaler, TeX Gyre Schola, TeX Gyre Heros, and TeX Gyre Cursor, in
default.

For LilyPond document building,
other font glyphs are embedded to intermediate EPSs.
But most fonts in LilyPond documents are Emmentaler and TeX Gyre.
So their glyphs are not embedded to the EPSs.

How about this?

[pkx166h, 2016-06-13 15:00:04]

On 2016/06/13 14:00:57, trueroad wrote:
> I came up with another idea.
> 
> Remove `-dgs-load-fonts` from lilypond-book default.
> During LilyPond and its documents are built, `-dgs-load-lily-fonts` option
> instead of `-dgs-load-fonts` is used.
> 
> `-dgs-load-lily-fonts` means "Load only the LilyPond fonts via Ghostscript."
> 
> "LilyPond fonts" mean the fonts in the LilyPond data directory.
> i.e. Emmentaler, TeX Gyre Schola, TeX Gyre Heros, and TeX Gyre Cursor, in
> default.
> 
> For LilyPond document building,
> other font glyphs are embedded to intermediate EPSs.
> But most fonts in LilyPond documents are Emmentaler and TeX Gyre.
> So their glyphs are not embedded to the EPSs.
> 
> How about this?

We would have to be careful none of the snippets or even @lilypond examples used
other fonts - I am thinking perhaps if there are any 'Cyrillic' or 'Hebrew' type
fonts. As doc building does actually build a 'snippet' document but not the
entire LSR, this might cause us problems or end up with 'missing' fonts in the
snippets. I don't know how to 'quickly' check this and I am not sure if I would
(when testing patches) see any errors for missing fonts while compiling docs. I
seem to recall some fix we had a few months ago where the 'Pi' (?) Character was
not being displayed properly - or something like that.

[wl_gnu.org, 2016-06-14 04:08:06]

>> Remove `-dgs-load-fonts` from lilypond-book default.  During
>> LilyPond and its documents are built, `-dgs-load-lily-fonts` option
>> instead of `-dgs-load-fonts` is used.  `-dgs-load-lily-fonts` means
>> "Load only the LilyPond fonts via Ghostscript."
>>
>> "LilyPond fonts" mean the fonts in the LilyPond data directory.
>> i.e. Emmentaler, TeX Gyre Schola, TeX Gyre Heros, and TeX Gyre
>> Cursor, in default.
>>
>> For LilyPond document building, other font glyphs are embedded to
>> intermediate EPSs.  But most fonts in LilyPond documents are
>> Emmentaler and TeX Gyre.  So their glyphs are not embedded to the
>> EPSs.
>> 
>> How about this?

Mhmm.  As James writes, this can easily break.

I can imagine another approach.  Since lilypond itself converts all
fonts to PostScript resources, why not writing those resources to a
`fontresource' directory instead of embedding?  We could add a
checksum to the resource name, just to be sure that, say, `foo.tff'
and `foo.otf' will be rejected.

Ideally, all intermediate PDFs also refer to this `fontresource'
directory, and only in the last step PDFs with subsetted, embedded
fonts are created.


    Werner

[trueroad, 2016-06-14 13:20:48]

> Mhmm.  As James writes, this can easily break.

My proposal never change the font selection method.

It changes the font embedding method for intermediate EPS.

Current "make doc" works as follows:

For all fonts, the font file name is embedded.
In other words, the font glyphs are not embedded.
So EPS file size can be small.

My proposal works as follows:

For "LilyPond fonts", the font file name is embedded.
For other fonts, the font glyphs are embedded.
In other words, most fonts file name is embedded. (Emmentaler and TeX Gyre)
The glyphs of few fonts are embedded. (e.g. Linux Libertine etc.)
So EPS file size does not almost increase.

Either way, intermediate and final PDF is embedded subset fonts in the same way.

I think that my proposal is temporary solution.

It can solve the problem that lilypond-book can not use OTC fonts.
In other words, "make doc" can use OTC fonts.
Required disk space for "make doc" does not almost increase.

If quickly the ideal can be achieved, my proposal is not required.

[dak, 2016-06-14 13:32:09]

On 2016/06/14 13:20:48, trueroad wrote:
> > Mhmm.  As James writes, this can easily break.
> 
> My proposal never change the font selection method.
> 
> It changes the font embedding method for intermediate EPS.
> 
> Current "make doc" works as follows:
> 
> For all fonts, the font file name is embedded.
> In other words, the font glyphs are not embedded.
> So EPS file size can be small.
> 
> My proposal works as follows:
> 
> For "LilyPond fonts", the font file name is embedded.
> For other fonts, the font glyphs are embedded.
> In other words, most fonts file name is embedded. (Emmentaler and TeX Gyre)
> The glyphs of few fonts are embedded. (e.g. Linux Libertine etc.)
> So EPS file size does not almost increase.
> 
> Either way, intermediate and final PDF is embedded subset fonts in the same
way.
> 
> I think that my proposal is temporary solution.
> 
> It can solve the problem that lilypond-book can not use OTC fonts.
> In other words, "make doc" can use OTC fonts.
> Required disk space for "make doc" does not almost increase.
> 
> If quickly the ideal can be achieved, my proposal is not required.

We are currently using about 4GB of disk space for building in spite of
-dgs-load-fonts being used.  Embedding/subsetting non-LilyPond fonts seems like
a plausible step but would further increase disk usage.  Does anybody have an
idea _why_ we currently use so much disk space even while using -dgs-load-fonts
and how we could get this under control?

[wl_gnu.org, 2016-07-22 15:16:10]

About a month ago we discussed how to reduce the disk space necessary
for builing the lilypond documentation.

> [...]  Since lilypond itself converts all fonts to PostScript
> resources, why not writing those resources to a `fontresource'
> directory instead of embedding?  We could add a checksum to the
> resource name, just to be sure that, say, `foo.tff' and `foo.otf'
> will be rejected.
> 
> Ideally, all intermediate PDFs also refer to this `fontresource'
> directory, and only in the last step PDFs with subsetted, embedded
> fonts are created.

Such an approach has now been discussed on the the pdftex mailing
list, cf.

  http://tug.org/pipermail/pdftex/2016-July/009045.html

and the following e-mails in this thread.

I've just tested successfully the following method, except item 1,
which I've executed manually.

1. Instead of embedding font resources into the file, lilypond writes
   them to a font resource directory and uses the `.loadfont' operator
   in its PS output file.  For simplicity, the font resources should
   have the PS font name as its file name (regardless of the font
   format), e.g. `TeXGyraSchola-Regular'; we then don't need a font
   map for ghostscript.

2. Lilypond's PS files are converted to PDFs with the additional gs
   option `-dEmbedAllFonts=false' (to be added to `postscript->PDF' in
   file `backend-library.scm').

3. Both xetex and pdftex accept the fontless PDFs without complaints;
   they simply embed them into its output file without alterations,
   AFAICS.

4. After the output PDF is built, a call to

     ps2pdf -I<fontresourcedir> \
            -dNOSAFER -P \
            Fontless.pdf WithEmbeddedFonts.pdf

   creates the final document.

Comparing the `--bigpdfs' method with the fontless PDF approach as
outlined above, the latter creates a final output file about 30%
smaller (at least in my small test).


     Werner

[wl_gnu.org, 2016-07-22 15:27:10]

>      ps2pdf -I<fontresourcedir> \
>             -dNOSAFER -P \
>             Fontless.pdf WithEmbeddedFonts.pdf

This should rather be

  ps2pdf -I<fontresourcedir> \
          -dNOSAFER \
          Fontless.pdf WithEmbeddedFonts.pdf

I've tested with the fonts in the current directory, thus `-P'.


    Werner

[dak, 2016-07-22 15:57:30]

Werner LEMBERG <wl@gnu.org> writes:

> About a month ago we discussed how to reduce the disk space necessary
> for builing the lilypond documentation.
>
>> [...]  Since lilypond itself converts all fonts to PostScript
>> resources, why not writing those resources to a `fontresource'
>> directory instead of embedding?  We could add a checksum to the
>> resource name, just to be sure that, say, `foo.tff' and `foo.otf'
>> will be rejected.
>> 
>> Ideally, all intermediate PDFs also refer to this `fontresource'
>> directory, and only in the last step PDFs with subsetted, embedded
>> fonts are created.
>
> Such an approach has now been discussed on the the pdftex mailing
> list, cf.
>
>   http://tug.org/pipermail/pdftex/2016-July/009045.html
>
> and the following e-mails in this thread.
>
> I've just tested successfully the following method, except item 1,
> which I've executed manually.
>
> 1. Instead of embedding font resources into the file, lilypond writes
>    them to a font resource directory and uses the `.loadfont' operator
>    in its PS output file.  For simplicity, the font resources should
>    have the PS font name as its file name (regardless of the font
>    format), e.g. `TeXGyraSchola-Regular'; we then don't need a font
>    map for ghostscript.
>
> 2. Lilypond's PS files are converted to PDFs with the additional gs
>    option `-dEmbedAllFonts=false' (to be added to `postscript->PDF' in
>    file `backend-library.scm').
>
> 3. Both xetex and pdftex accept the fontless PDFs without complaints;
>    they simply embed them into its output file without alterations,
>    AFAICS.
>
> 4. After the output PDF is built, a call to
>
>      ps2pdf -I<fontresourcedir> \
>             -dNOSAFER -P \
>             Fontless.pdf WithEmbeddedFonts.pdf
>
>    creates the final document.
>
> Comparing the `--bigpdfs' method with the fontless PDF approach as
> outlined above, the latter creates a final output file about 30%
> smaller (at least in my small test).

A 30% reduction in the final output file size sounds nice.  Personally,
I find the prospect of not having 4GB of disk usage for running
lilypond-patchy-staging quite more compelling, and I would seriously
suspect that all the amount of font juggling and merging subsetted fonts
will not just take quite a bit of disk space but also of processing
time.  So if we could successfully pull this off and have it work
reliably for lilypond-book, I consider it likely to end up as a real
boon in resource usage.

-- 
David Kastrup

[wl_gnu.org, 2016-07-22 16:06:29]

>> Comparing the `--bigpdfs' method with the fontless PDF approach as
>> outlined above, the latter creates a final output file about 30%
>> smaller (at least in my small test).
> 
> A 30% reduction in the final output file size sounds nice.

This is an *additional* 30%, since `--bigpdfs' already makes the
final output file much smaller.

> Personally, I find the prospect of not having 4GB of disk usage for
> running lilypond-patchy-staging quite more compelling, and I would
> seriously suspect that all the amount of font juggling and merging
> subsetted fonts will not just take quite a bit of disk space but
> also of processing time.  So if we could successfully pull this off
> and have it work reliably for lilypond-book, I consider it likely to
> end up as a real boon in resource usage.

Me too.  The writing of font resources shouldn't be too difficult to
implement, at least for persons who are fluent with Scheme...


    Werner

[c_sorensen, 2016-07-22 17:48:22]

On 7/22/16 9:35 AM, "lilypond-devel on behalf of David Kastrup"
<lilypond-devel-bounces+c_sorensen=byu.edu@gnu.org on behalf of
dak@gnu.org> wrote:

>
>A 30% reduction in the final output file size sounds nice.  Personally,
>I find the prospect of not having 4GB of disk usage for running
>lilypond-patchy-staging quite more compelling, and I would seriously
>suspect that all the amount of font juggling and merging subsetted fonts
>will not just take quite a bit of disk space but also of processing
>time.  So if we could successfully pull this off and have it work
>reliably for lilypond-book, I consider it likely to end up as a real
>boon in resource usage.

I expect this to be a real boon as well.

However, without doing a careful study, I expect that the 4GB of disk
space used for running lilypond-patchy-staging largely related to the
large number of snippet files, each of which rather inefficiently uses
disk allocation units.  So I'm not sure I see a huge reduction in this
overall disk usage for running patchy.

But this is just a hunch, not a demonstrated reality.

Thanks,

Carl

[pkx, 2016-07-22 18:35:55]

hello,


On 22/07/16 18:48, Carl Sorensen wrote:
>
> On 7/22/16 9:35 AM, "lilypond-devel on behalf of David Kastrup"
> <lilypond-devel-bounces+c_sorensen=byu.edu@gnu.org on behalf of
> dak@gnu.org> wrote:
>
>> A 30% reduction in the final output file size sounds nice.  Personally,
>> I find the prospect of not having 4GB of disk usage for running
>> lilypond-patchy-staging quite more compelling, and I would seriously
>> suspect that all the amount of font juggling and merging subsetted fonts
>> will not just take quite a bit of disk space but also of processing
>> time.  So if we could successfully pull this off and have it work
>> reliably for lilypond-book, I consider it likely to end up as a real
>> boon in resource usage.
> I expect this to be a real boon as well.
>
> However, without doing a careful study, I expect that the 4GB of disk
> space used for running lilypond-patchy-staging largely related to the
> large number of snippet files, each of which rather inefficiently uses
> disk allocation units.  So I'm not sure I see a huge reduction in this
> overall disk usage for running patchy.
>
> But this is just a hunch, not a demonstrated reality.

After my last Patchy run (just now) I get this (ive edited so just the 
largest dirs in each case are shown):

NB> It seems that the Japanese docs are built using its own 'build' dir 
(i.e. it has its own out-www build doc and doesn't seem to share any of 
the files the other docs do while building, but I am probably not 
understanding this).

Are we wasting build time and space with the way we build the Japanese 
docs compared to the other languages?

Anyway...

jlowe@jloweDesktop ~/lilypond-git/build $  du -smh * | sort -hr
1.9G    input
1.4G    Documentation
773M    out
592M    out-www
171M    lily
9.5M    mf
7.5M    flower
1.2M    po
..

jlowe@jloweDesktop ~/lilypond-git/build/Documentation $  du -smh * | 
sort -hr
425M    out-www /<--- English Docs/
341M    ja /<---- why is this so big compared to the other language docs?/
233M    de
133M    es
90M    it
89M    fr
39M    ca
21M    hu
20M    out
17M    cs
13M    nl
8.3M    pictures
3.6M    ly-examples
1.8M    po
1.7M    snippets

...

jlowe@jloweDesktop ~/lilypond-git/build/Documentation/out-www $  du -smh 
* | sort -hr
50M    notation
36M    notation.pdf
36M    notation.it.pdf
36M    notation.fr.pdf
36M    notation.es.pdf
34M    notation.de.pdf
25M    learning
18M    internals
15M    music-glossary
11M    snippets.pdf
9.3M    web
7.7M    usage
6.7M    contributor
5.3M    snippets
5.3M    learning.pdf
5.3M    learning.fr.pdf
5.3M    learning.es.pdf
5.3M    learning.de.pdf
5.2M    learning.it.pdf
4.9M    learning.ca.pdf
4.7M    learning.nl.pdf
4.1M    notation-big-page.fr.html
4.1M    notation-big-page.es.html
4.0M    notation-big-page.ja.html
4.0M    notation-big-page.it.html
3.9M    notation-big-page.de.html
3.8M    notation-big-page.html

jlowe@jloweDesktop ~/lilypond-git/build/input $  du -smh * | sort -hr
1.9G    regression
8.0K    out-www
8.0K    out
4.0K    GNUmakefile

jlowe@jloweDesktop ~/lilypond-git/build/input/regression $  du -smh * | 
sort -hr
705M    out-test-baseline
705M    out-test
238M    out-www
155M    musicxml
44M    midi
32M    abc2ly
20M    lilypond-book
468K    collated-files.texi2pdf.log


jlowe@jloweDesktop ~/lilypond-git/build/Documentation/ja $  du -smh * | 
sort -hr
512M    out-www
8.0K    out
8.0K    notation.splittexi.log
8.0K    notation.bigtexi.log

...

jlowe@jloweDesktop ~/lilypond-git/build/Documentation/ja/out-www $ du 
-smh * | sort -hr
70M    34
52M    37
38M    ef
38M    3c
33M    bf
21M    f1
20M    8b
20M    4e
19M    9a
9.6M    notation
  ....

James
>

[dak, 2016-07-22 19:06:38]

James <pkx@gnu.org> writes:

> NB> It seems that the Japanese docs are built using its own 'build'
> dir (i.e. it has its own out-www build doc and doesn't seem to share
> any of the files the other docs do while building, but I am probably
> not understanding this).
>
> Are we wasting build time and space with the way we build the Japanese
> docs compared to the other languages?
>
> Anyway...
>
> jlowe@jloweDesktop ~/lilypond-git/build $  du -smh * | sort -hr
> 1.9G    input
> 1.4G    Documentation
> 773M    out
> 592M    out-www
> 171M    lily
> 9.5M    mf
> 7.5M    flower
> 1.2M    po
> ..
>
> jlowe@jloweDesktop ~/lilypond-git/build/Documentation $  du -smh * |
> sort -hr
> 425M    out-www /<--- English Docs/
> 341M    ja /<---- why is this so big compared to the other language
> docs?/

Because if thousands of snippets include a Japanese font over and over
again rather than include some Western font over and over again, the
total size is quite larger.

-- 
David Kastrup

[c_sorensen, 2016-07-22 20:02:08]

On 7/22/16 12:35 PM, "James" <pkx@gnu.org> wrote:

>
>jlowe@jloweDesktop ~/lilypond-git/build/input $  du -smh * | sort -hr
>1.9G    regression
>8.0K    out-www
>8.0K    out
>4.0K    GNUmakefile

Do we turn off point and click when doing the doc builds?

Carl

[trueroad, 2016-07-24 13:36:05]

On 2016/07/22 15:27:10, wl_gnu.org wrote:
> >      ps2pdf -I<fontresourcedir> \
> >             -dNOSAFER -P \
> >             Fontless.pdf WithEmbeddedFonts.pdf
> 
> This should rather be
> 
>   ps2pdf -I<fontresourcedir> \
>           -dNOSAFER \
>           Fontless.pdf WithEmbeddedFonts.pdf
> 
> I've tested with the fonts in the current directory, thus `-P'.
> 
> 
>     Werner

Unfortunately, Ghostscript (ps2pdf) discards PDF destination names.
So remote PDF links do not work fine.

http://bugs.ghostscript.com/show_bug.cgi?id=696943
http://bugs.ghostscript.com/show_bug.cgi?id=695760

Moreover, Ghostscript developers seem to want that `.loadfont' is not used.

http://bugs.ghostscript.com/show_bug.cgi?id=696823
http://bugs.ghostscript.com/show_bug.cgi?id=696824

[wl_gnu.org, 2016-07-24 15:27:47]

> Moreover, Ghostscript developers seem to want that `.loadfont' is
> not used.
> 
> http://bugs.ghostscript.com/show_bug.cgi?id=696823
> http://bugs.ghostscript.com/show_bug.cgi?id=696824

This is something we can trustfully ignore, I think, given that we
resolve TTCs to Type42 and OTCs to bare CFF resources by ourselves.

> Unfortunately, Ghostscript (ps2pdf) discards PDF destination names.
> So remote PDF links do not work fine.
> 
> http://bugs.ghostscript.com/show_bug.cgi?id=696943
> http://bugs.ghostscript.com/show_bug.cgi?id=695760

This is much more depressing :-(

Isn't it possible to modify `texinfo.tex' so that it creates `invalid'
remote links that only become valid after calling `ps2pdf'?  Maybe
this is a naive question since I don't know PDF format details.

On the other hand, I can imagine to unify all separate PDF
documentation files into a single `lilypond.pdf'; we then wouldn't
have problems with remote PDF links.  This file would have far more
than 1000 pages, but due to the improved font handling it would be
actually smaller than the current `notation.pdf', I think.

While working on this we could also try to convince the texinfo
maintainers to implement `parts', one level higher than chapters...


    Werner

[trueroad, 2016-07-26 15:25:45]

> > Unfortunately, Ghostscript (ps2pdf) discards PDF destination names.
> > So remote PDF links do not work fine.
> > 
> > http://bugs.ghostscript.com/show_bug.cgi?id=696943
> > http://bugs.ghostscript.com/show_bug.cgi?id=695760
> 
> This is much more depressing :-(
> 
> Isn't it possible to modify `texinfo.tex' so that it creates `invalid'
> remote links that only become valid after calling `ps2pdf'?  Maybe
> this is a naive question since I don't know PDF format details.

I have another idea.

Extracting page mode and named destinations as PDFmark from the font less PDF.
Both the PDF and the PDFmark are mixed by Ghostscript.

https://github.com/trueroad/extractpdfmark/

i.e.

$ extractpdfmark Fontless.pdf > Fontless.pdfmark.ps
$ gs -I<fontresourcedir> -dNOSAFER \
     -q -dBATCH -dNOPAUSE -sDEVICE=pdfwrite \
     -sOutputFile=WithEmbeddedFonts.pdf Fontless.pdf Fontless.pdfmark.ps

[wl_gnu.org, 2016-07-26 15:48:34]

> Extracting page mode and named destinations as PDFmark from the font
> less PDF.  Both the PDF and the PDFmark are mixed by Ghostscript.

Aaah!  Very nice idea.

If I understand the issue correctly, your solution overcomes the last
obstacle with ghostscript, so we could actually start with an
implementation :-)


    Werner

[trueroad, 2016-08-05 13:17:21]

> If I understand the issue correctly, your solution overcomes the last
> obstacle with ghostscript, so we could actually start with an
> implementation :-)

Unfortunately, Ghostscript 9.19 (newest release version) and prior versions
cannot handle some PDF destination names.
http://bugs.ghostscript.com/show_bug.cgi?id=696974

So the next GS release (something like GS 9.20) would be required.

[trueroad_trueroad.jp, 2016-10-25 15:28:40]

> About a month ago we discussed how to reduce the disk space necessary
> for builing the lilypond documentation.
> 
>> [...]  Since lilypond itself converts all fonts to PostScript
>> resources, why not writing those resources to a `fontresource'
>> directory instead of embedding?  We could add a checksum to the
>> resource name, just to be sure that, say, `foo.tff' and `foo.otf'
>> will be rejected.
>> 
>> Ideally, all intermediate PDFs also refer to this `fontresource'
>> directory, and only in the last step PDFs with subsetted, embedded
>> fonts are created.
> 
> Such an approach has now been discussed on the the pdftex mailing
> list, cf.
> 
>   http://tug.org/pipermail/pdftex/2016-July/009045.html
> 
> and the following e-mails in this thread.
> 
> I've just tested successfully the following method, except item 1,
> which I've executed manually.
> 
> 1. Instead of embedding font resources into the file, lilypond writes
>    them to a font resource directory and uses the `.loadfont' operator
>    in its PS output file.  For simplicity, the font resources should
>    have the PS font name as its file name (regardless of the font
>    format), e.g. `TeXGyraSchola-Regular'; we then don't need a font
>    map for ghostscript.
> 
> 2. Lilypond's PS files are converted to PDFs with the additional gs
>    option `-dEmbedAllFonts=false' (to be added to `postscript->PDF' in
>    file `backend-library.scm').
> 
> 3. Both xetex and pdftex accept the fontless PDFs without complaints;
>    they simply embed them into its output file without alterations,
>    AFAICS.
> 
> 4. After the output PDF is built, a call to
> 
>      ps2pdf -I<fontresourcedir> \
>             -dNOSAFER -P \
>             Fontless.pdf WithEmbeddedFonts.pdf
> 
>    creates the final document.
> 
> Comparing the `--bigpdfs' method with the fontless PDF approach as
> outlined above, the latter creates a final output file about 30%
> smaller (at least in my small test).

I've tried this way.
I use Ghostscript 9.20 and LilyPond 2.19.49.

`sample.tar.gz` is my test files.

`final-no-embed.png` is the generated file by this way.
Some glyphs are broken.

`final-embed-subset.png` is the generated file by conventional way.
It is no problem. No glyphs are broken.

You can get the results by the following command in `sample.tar.gz`.

$ ./makepdfs.sh

It seems that non-Latin glyphs in TrueType (TTF and TTC) fonts are broken.
In the case of Japanese TrueType fonts etc.,
most glyphs (also Latin glyphs) are broken.

Type 1 fonts and Open Type (OTF) fonts seem no problem.

Note:
Ghostscript does not find Japanese OTF fonts in -I<fontresourcedir>.
It find them in <fontresourcedir>/CIDFont/.

[wl_gnu.org, 2016-10-25 21:18:09]

>> I've just tested successfully the following method, except item 1,
>> which I've executed manually.  [...]
>
> I've tried this way.  [...]

Thank you for your thorough testing, which obviously covers more
situations than my limited tries.

BTW, I see that your `extractpdfmark' tool has already reached version
1.0.1 – congratulations!

  https://github.com/trueroad/extractpdfmark

Do you have time to integrate this into lilypond in the near future?



    Werner

[trueroad_trueroad.jp, 2016-10-26 11:36:52]

>>> I've just tested successfully the following method, except item 1,
>>> which I've executed manually.  [...]
>>
>> I've tried this way.  [...]
> 
> Thank you for your thorough testing, which obviously covers more
> situations than my limited tries.
> 
> BTW, I see that your `extractpdfmark' tool has already reached version
> 1.0.1 – congratulations!
> 
>   https://github.com/trueroad/extractpdfmark

Thank you.

> Do you have time to integrate this into lilypond in the near future?

I would like so.
My GUB environment can compile it for GUB inner using.
(v1.0.0 could not compile.)

But, I think that the integration requires a lot of time.

[wl_gnu.org, 2016-10-26 11:40:14]

>> Do you have time to integrate this into lilypond in the near future?
> 
> I would like so.  My GUB environment can compile it for GUB inner
> using.  (v1.0.0 could not compile.)
> 
> But, I think that the integration requires a lot of time.

Why?  Please elaborate.


    Werner

[trueroad_trueroad.jp, 2016-10-26 13:03:13]

>>> Do you have time to integrate this into lilypond in the near future?
>> 
>> I would like so.  My GUB environment can compile it for GUB inner
>> using.  (v1.0.0 could not compile.)
>> 
>> But, I think that the integration requires a lot of time.
> 
> Why?  Please elaborate.

At least, in order to avoid broken glyphs,
we will need to change whether or not embedding by the font type.

i.e.
TrueType   : no embedding
Type1 / OTF: embedding

So `-dEmbedAllFonts=false` cannot be used.
Perhaps, `/NeverEmbed` and `/AlwaysEmbed` can be used.
However, I've never tried it. It is just an idea.

Moreover, we will need to change font resource directory
depending on the font type.

i.e.
Japanese OTF: <fontresourcedir>/CIDFont/
Other fonts?: <fontresourcedir>

But, I have another idea.
It does not need the font resource directory.
To use a PostScript file which only contains font resources.
It is one PostScript file and includes

`%%BeginFont`
~
`%%EndFont`

blocks of all necessary fonts.
I'm trying it.

Anyway, I think there will be a lot of things to consider.

[wl_gnu.org, 2016-10-26 13:10:56]

> At least, in order to avoid broken glyphs, we will need to change
> whether or not embedding by the font type.

OK, so the issue is not integrating `extractpdfmark' into lilypond but
adjusting lilypond's font handling to make it work.

> Moreover, we will need to change font resource directory
> depending on the font type.
> 
> i.e.
> Japanese OTF: <fontresourcedir>/CIDFont/
> Other fonts?: <fontresourcedir>

Looks sensible.

> But, I have another idea.
> It does not need the font resource directory.
> To use a PostScript file which only contains font resources.
> It is one PostScript file and includes
> 
> `%%BeginFont`
> ~
> `%%EndFont`
> 
> blocks of all necessary fonts.
> I'm trying it.

Interesting!  And thanks for trying.


    Werner