Allow parallelism in input/regression/lilypond-book/

Allow parallelism in input/regression/lilypond-book/

This is a 3x improvement, bringing down processing time at -j4 to ~20
seconds, down from 1 minute.

[dak, 2020-02-23 09:49:24]

Stupid question: does the database design of lilypond-book even allow for
uncoordinated parallel runs?

[hanwenn, 2020-02-23 10:04:46]

On 2020/02/23 09:49:24, dak wrote:
> Stupid question: does the database design of lilypond-book even allow for
> uncoordinated parallel runs?

It's not a stupid question; it's a good question.  

Writing files atomically (open temp file, write, close, rename), seems a lot of
work.

I've considered adding an advisory lock on the DB directory. Since we max out
CPUs when we run lp-book, it shouldn't slow down things, but I was worried about
doing file locks cross-platform.

Maybe I can do flock(1), which at least lets limit the worry to OSX vs. Linux.

[hanwenn, 2020-02-23 10:29:53]

On 2020/02/23 10:04:46, hanwenn wrote:
> On 2020/02/23 09:49:24, dak wrote:
> > Stupid question: does the database design of lilypond-book even allow for
> > uncoordinated parallel runs?
> 
> It's not a stupid question; it's a good question.  
> 
> Writing files atomically (open temp file, write, close, rename), seems a lot
of
> work.
> 
> I've considered adding an advisory lock on the DB directory. Since we max out
> CPUs when we run lp-book, it shouldn't slow down things, but I was worried
about
> doing file locks cross-platform.
> 
> Maybe I can do flock(1), which at least lets limit the worry to OSX vs. Linux.

to answer your original question: I think the \sourcefileline statements can
differ between snippets written from different lp-book instances, and this can
trigger a consistency check failure.

[dak, 2020-02-23 11:00:18]

On 2020/02/23 10:29:53, hanwenn wrote:
> On 2020/02/23 10:04:46, hanwenn wrote:
> > On 2020/02/23 09:49:24, dak wrote:
> > > Stupid question: does the database design of lilypond-book even allow for
> > > uncoordinated parallel runs?
> > 
> > It's not a stupid question; it's a good question.  
> > 
> > Writing files atomically (open temp file, write, close, rename), seems a lot
> of
> > work.
> > 
> > I've considered adding an advisory lock on the DB directory. Since we max
out
> > CPUs when we run lp-book, it shouldn't slow down things, but I was worried
> about
> > doing file locks cross-platform.
> > 
> > Maybe I can do flock(1), which at least lets limit the worry to OSX vs.
Linux.
> 
> to answer your original question: I think the \sourcefileline statements can
> differ between snippets written from different lp-book instances, and this can
> trigger a consistency check failure.

So wouldn't it appear that the way to exploit parallelism with lilypond-book,
short of writing its own jobserver, is to use CPU_COUNT like we did before?

[hanwenn, 2020-02-23 11:08:48]

On Sun, Feb 23, 2020 at 12:00 PM <dak@gnu.org> wrote:
> > to answer your original question: I think the \sourcefileline
> statements can
> > differ between snippets written from different lp-book instances, and
> this can
> > trigger a consistency check failure.
>
> So wouldn't it appear that the way to exploit parallelism with
> lilypond-book, short of writing its own jobserver, is to use CPU_COUNT
> like we did before?

I think we should do both: the lilypond runs in lp-book should be
protected by some sort of lock, and we should use both CPU_COUNT=M and
-jN.

then worst case, you have M lilypond processes and N-1 other jobs.

BTW,  you run at -j8 with only 4 CPUs, right? If you have an SSD
drive, this is probably doesn't bring benefits, as there is no I/O
delay any more that you can paper over with CPU bound tasks.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dak, 2020-02-23 11:16:47]

Han-Wen Nienhuys <hanwenn@gmail.com> writes:

> On Sun, Feb 23, 2020 at 12:00 PM <dak@gnu.org> wrote:
>> > to answer your original question: I think the \sourcefileline
>> statements can
>> > differ between snippets written from different lp-book instances, and
>> this can
>> > trigger a consistency check failure.
>>
>> So wouldn't it appear that the way to exploit parallelism with
>> lilypond-book, short of writing its own jobserver, is to use CPU_COUNT
>> like we did before?
>
> I think we should do both: the lilypond runs in lp-book should be
> protected by some sort of lock, and we should use both CPU_COUNT=M and
> -jN.
>
> then worst case, you have M lilypond processes and N-1 other jobs.
>
> BTW,  you run at -j8 with only 4 CPUs, right? If you have an SSD
> drive, this is probably doesn't bring benefits, as there is no I/O
> delay any more that you can paper over with CPU bound tasks.

I run with -j9, using 1 CPU for I/O latency and 4 CPUs for
hyperthreading.  I wish there were no I/O delay any more, that would be
nice.

-- 
David Kastrup

[dan_faithful.be, 2020-02-23 13:59:31]

On Feb 23, 2020, at 06:08, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> I think we should do both: the lilypond runs in lp-book should be
> protected by some sort of lock, and we should use both CPU_COUNT=M and
> -jN.
> 
> then worst case, you have M lilypond processes and N-1 other jobs.

What would you recommend to a developer who doesn't want to run more than J = M
+ N - 1 concurrent jobs due to lilypond development?  What values of M and N
would serve best?

On Sun, Feb 23, 2020 at 12:00 PM <dak@gnu.org> wrote:
> So wouldn't it appear that the way to exploit parallelism with
> lilypond-book, short of writing its own jobserver, is to use CPU_COUNT
> like we did before?

Making lilypond-book a client of the GNU make job server sounds like an option.

  "Sharing Job Slots with GNU make"
  https://www.gnu.org/software/make/manual/html_node/Job-Slots.html

Examples (good or bad? I haven't looked):

  "Add GNU make jobserver client support"
  https://github.com/ninja-build/ninja/pull/1140

  "Add a GNU make jobserver implementation to Cargo"
  https://github.com/rust-lang/cargo/pull/4110

— 
Dan

[hanwenn, 2020-02-23 14:04:25]

On Sun, Feb 23, 2020 at 2:59 PM Dan Eble <dan@faithful.be> wrote:
>
> On Feb 23, 2020, at 06:08, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> > I think we should do both: the lilypond runs in lp-book should be
> > protected by some sort of lock, and we should use both CPU_COUNT=M and
> > -jN.
> >
> > then worst case, you have M lilypond processes and N-1 other jobs.
>
> What would you recommend to a developer who doesn't want to run more than J =
M + N - 1 concurrent jobs due to lilypond development?  What values of M and N
would serve best?

Normally M=N= #cpus should be OK. A bit of extra parallelism doesn't
hurt, especially if you have hyperthreaded CPUs.

> > So wouldn't it appear that the way to exploit parallelism with
> > lilypond-book, short of writing its own jobserver, is to use CPU_COUNT
> > like we did before?
>
> Making lilypond-book a client of the GNU make job server sounds like an
option.
>
>   "Sharing Job Slots with GNU make"
>   https://www.gnu.org/software/make/manual/html_node/Job-Slots.html

This looks hairy, because it's not lilypond-book but lilypond itself
that starts up the jobs, so you have plumb through the jobserver
support.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dak, 2020-02-23 14:11:30]

Dan Eble <dan@faithful.be> writes:

> On Feb 23, 2020, at 06:08, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
>> I think we should do both: the lilypond runs in lp-book should be
>> protected by some sort of lock, and we should use both CPU_COUNT=M and
>> -jN.
>> 
>> then worst case, you have M lilypond processes and N-1 other jobs.
>
> What would you recommend to a developer who doesn't want to run more
> than J = M + N - 1 concurrent jobs due to lilypond development?  What
> values of M and N would serve best?
>
> On Sun, Feb 23, 2020 at 12:00 PM <dak@gnu.org> wrote:
>> So wouldn't it appear that the way to exploit parallelism with
>> lilypond-book, short of writing its own jobserver, is to use CPU_COUNT
>> like we did before?
>
> Making lilypond-book a client of the GNU make job server sounds like an
option.
>
>   "Sharing Job Slots with GNU make"
>   https://www.gnu.org/software/make/manual/html_node/Job-Slots.html

But that still doesn't solve the problem that the database approach of
lilypond-book does not work for running multiple lilypond-book jobs in
parallel.

-- 
David Kastrup

[dan_faithful.be, 2020-02-23 14:36:44]

On Feb 23, 2020, at 09:04, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> 
>> What would you recommend to a developer who doesn't want to run more than J =
M + N - 1 concurrent jobs due to lilypond development?  What values of M and N
would serve best?
> 
> Normally M=N= #cpus should be OK. A bit of extra parallelism doesn't
> hurt, especially if you have hyperthreaded CPUs.

While that qualifies as guidance, I'm not sure you've heard me.  Extra
parallelism is unwelcome by definition in this case.

Also, don't forget that although the parallelism is (sadly only) governed by the
number of jobs, each job uses resources other than CPU time that the developer
might be concerned about.

Regards,
— 
Dan

[hanwenn, 2020-02-23 14:40:09]

On Sun, Feb 23, 2020 at 3:36 PM Dan Eble <dan@faithful.be> wrote:
>
> On Feb 23, 2020, at 09:04, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
> >
> >> What would you recommend to a developer who doesn't want to run more than J
= M + N - 1 concurrent jobs due to lilypond development?  What values of M and N
would serve best?
> >
> > Normally M=N= #cpus should be OK. A bit of extra parallelism doesn't
> > hurt, especially if you have hyperthreaded CPUs.
>
> While that qualifies as guidance, I'm not sure you've heard me.  Extra
parallelism is unwelcome by definition in this case.
>
> Also, don't forget that although the parallelism is (sadly only) governed by
the number of jobs, each job uses resources other than CPU time that the
developer might be concerned about.

You use Docker for working on LilyPoind, right? If you are concerned
with too much resource use, you can constrain the amount of CPU that
docker hands out. This is more reliable than whatever makefile trick
we employ.

>
> Regards,
> —
> Dan
>


-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dan_faithful.be, 2020-02-23 14:41:29]

On Feb 23, 2020, at 09:11, David Kastrup <dak@gnu.org> wrote:
> 
>>  "Sharing Job Slots with GNU make"
>>  https://www.gnu.org/software/make/manual/html_node/Job-Slots.html
> 
> But that still doesn't solve the problem that the database approach of
> lilypond-book does not work for running multiple lilypond-book jobs in
> parallel.

Maybe I haven't devoted enough time to understand this situation well. 
Basically, my thought was that whatever currently uses CPU_COUNT could
communicate with the job server instead so that one wouldn't need to repeat -jN
CPU_COUNT=N.
— 
Dan

[dak, 2020-02-23 14:51:51]

Dan Eble <dan@faithful.be> writes:

> On Feb 23, 2020, at 09:11, David Kastrup <dak@gnu.org> wrote:
>> 
>>>  "Sharing Job Slots with GNU make"
>>>  https://www.gnu.org/software/make/manual/html_node/Job-Slots.html
>> 
>> But that still doesn't solve the problem that the database approach of
>> lilypond-book does not work for running multiple lilypond-book jobs in
>> parallel.
>
> Maybe I haven't devoted enough time to understand this situation well.
> Basically, my thought was that whatever currently uses CPU_COUNT could
> communicate with the job server instead so that one wouldn't need to
> repeat -jN CPU_COUNT=N.

It is LilyPond itself that gets started with a -djob-count argument and
a long commandline of files which it then distributes to the named jobs.

make/lilypond-vars.make:LILYPOND_JOBS=$(if
$(CPU_COUNT),-djob-count=$(CPU_COUNT),)

dak@lola:/usr/local/tmp/lilypond$ git grep LILYPOND_JOBS
make/lilypond-vars.make:LILYPOND_JOBS=$(if
$(CPU_COUNT),-djob-count=$(CPU_COUNT),)
make/lilypond-vars.make:$(LILYPOND_JOBS) \
make/lysdoc-targets.make:       $(MAKE)
LILYPOND_BOOK_LILYPOND_FLAGS="-dbackend=eps --formats=ps $(LILYPOND_JOBS)
-dseparate-log-files -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -I
$(top-src-dir)/Documentation/included/ -ddump-profile -dcheck-internal-types
-ddump-signatures -danti-alias-factor=1
-dfont-export-dir=$(top-build-dir)/out-fonts -O TeX-GS" LILYPOND_BOOK_WARN=
$(outdir)/collated-files.html LYS_OUTPUT_DIR=$(top-build-dir)/out/lybook-testdb
make/lysdoc-targets.make:       $(MAKE)
LILYPOND_BOOK_LILYPOND_FLAGS="-dbackend=eps --formats=ps $(LILYPOND_JOBS)
-dseparate-log-files -dinclude-eps-fonts -dgs-load-lily-fonts --header=texidoc
-I $(top-src-dir)/Documentation/included/ -ddump-profile -dcheck-internal-types
-ddump-signatures -danti-alias-factor=1" LILYPOND_BOOK_WARN=
$(outdir)/collated-files.html LYS_OUTPUT_DIR=$(top-build-dir)/out/lybook-testdb
scripts/auxiliar/build-coverage.sh:  make conf=cov test-clean OUT_TEST=testcov
LILYPOND_JOBS= && \
scripts/auxiliar/build-coverage.sh:  make conf=cov test OUT_TEST=testcov
LILYPOND_JOBS='-dtrace-scheme-coverage '

While we could try asking parallel Make for job slots, the problem
remains that only one lilypond-book job (per backend/database) could
possibly run at a time.

We could, however, conceivably parallelize a lilypond-book job with PNG
backend and a lilypond-book job with PDF backend.  I don't think that
those would share the same database (correct me if I am wrong).  I have
no good idea how to make this parallelisation work in anything remotely
resembling a reproducible/reliable manner, however.  At the time the
first lilypond-book job is started, it needs to get told how many
processors it should be using, and by that time the job server has no
idea how many other lilypond-book jobs may appear in parallel.

-- 
David Kastrup
My replies have a tendency to cause friction.  To help mitigating
damage, feel free to forward problematic posts to me adding a subject
like "timeout 1d" (for a suggested timeout of 1 day) or "offensive".

[hanwenn, 2020-02-23 15:00:25]

On Sun, Feb 23, 2020 at 3:51 PM David Kastrup <dak@gnu.org> wrote:
> We could, however, conceivably parallelize a lilypond-book job with PNG
> backend and a lilypond-book job with PDF backend.  I don't think that
> those would share the same database (correct me if I am wrong).  I have

they share the same directory. Hopefully, we're not building the PNG
and PDF separate, because we'd have to run over all of the snippets
twice. Both the PNG and PDF gets generated from within lilypond by
running EPS output through GS.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[hanwenn, 2020-02-27 10:48:32]

superseeded by https://codereview.appspot.com/555360043/