Use $(MAKE) instead of 'make' in lilypond-book regtests

Use $(MAKE) instead of 'make' in lilypond-book regtests

This enables parallel builds of lilypond-book tests

[hanwenn, 2020-02-22 21:43:42]

all problems

[lemzwerg, 2020-02-22 22:00:33]

LGTM

[Dan Eble, 2020-02-22 22:27:05]

How does this interact with CPU_COUNT, which is documented in the Contributor's
Guide thus (emphasis mine)?

    The most time consuming task for building the documentation is
    running LilyPond to build images of music, and *** there cannot be
    several simultaneously running lilypond-book instances, *** so the
    -j make option does not significantly speed up the build
    process. To help speed it up, the makefile variable CPU_COUNT may
    be set in local.make or on the command line to the number of .ly
    files that LilyPond should process simultaneously, e.g. on a
    bi-processor or dual core machine:

        make -j3 CPU_COUNT=3 doc

This is how I always build.  Is it going to have the same effect now, or will it
do something new, e.g. attempt 9 concurrent jobs?

[hanwenn, 2020-02-22 23:17:50]

On 2020/02/22 22:27:05, Dan Eble wrote:
> How does this interact with CPU_COUNT, which is documented in the
Contributor's
> Guide thus (emphasis mine)?
> 
>     The most time consuming task for building the documentation is
>     running LilyPond to build images of music, and *** there cannot be
>     several simultaneously running lilypond-book instances, *** so the
>     -j make option does not significantly speed up the build
>     process. To help speed it up, the makefile variable CPU_COUNT may
>     be set in local.make or on the command line to the number of .ly
>     files that LilyPond should process simultaneously, e.g. on a
>     bi-processor or dual core machine:
> 
>         make -j3 CPU_COUNT=3 doc
> 
> This is how I always build.  Is it going to have the same effect now, or will
it
> do something new, e.g. attempt 9 concurrent jobs?

if you were doing 

  make -j3 CPU_COUNT=3 

you were already potentially in a state where you have 3 jobs each spawning 3
lp-book invocations,
as all almost everything is already doing $(MAKE) instead of 'make'.

As you can see, this only affects the regtests for lilypond-book, and not even
that, since parallelism 
is explicitly switched off for this directory (facepalm) by the statement I
marked.

There is a better reason to avoid multiple lp-book processes, though: I am
pretty sure lilypond doesn't write 
files atomically, so multiple lp-book invocations might try to write the same
files at the same time.

We should probably add locking on the .ly snippet DB so this doesn't happen.

[hanwenn, 2020-02-22 23:18:43]

On 2020/02/22 23:17:50, hanwenn wrote:
> you were already potentially in a state where you have 3 jobs each spawning 3
> lp-book invocations,

whoops, I mean: 3 lp-book invocations spawning 3 lilypond subprocesses each, for
9 in total.

[hanwenn, 2020-02-22 23:18:49]

https://codereview.appspot.com/569400043/diff/549600043/input/regression/lily...
File input/regression/lilypond-book/GNUmakefile (right):

https://codereview.appspot.com/569400043/diff/549600043/input/regression/lily...
input/regression/lilypond-book/GNUmakefile:42: .NOTPARALLEL:
here.

[dak, 2020-02-22 23:29:15]

On 2020/02/22 23:18:43, hanwenn wrote:
> On 2020/02/22 23:17:50, hanwenn wrote:
> > you were already potentially in a state where you have 3 jobs each spawning
3
> > lp-book invocations,
> 
> whoops, I mean: 3 lp-book invocations spawning 3 lilypond subprocesses each,
for
> 9 in total.

Considering that I build with CPU_COUNT=9 make -j9 I am pretty sure that I'd
have noticed if that were happening so far.  I may have a nice number of cores
(4) and memory considered plentiful (16GB) for a laptop, but 81 parallel
instances of LilyPond would certainly bring this system to its knees.

[hanwenn, 2020-02-22 23:56:23]

On Sun, Feb 23, 2020 at 12:29 AM <dak@gnu.org> wrote:
>
> On 2020/02/22 23:18:43, hanwenn wrote:
> > On 2020/02/22 23:17:50, hanwenn wrote:
> > > you were already potentially in a state where you have 3 jobs each
> spawning 3
> > > lp-book invocations,
> >
> > whoops, I mean: 3 lp-book invocations spawning 3 lilypond subprocesses
> each, for
> > 9 in total.
>
> Considering that I build with CPU_COUNT=9 make -j9 I am pretty sure that
> I'd have noticed if that were happening so far.  I may have a nice
> number of cores (4) and memory considered plentiful (16GB) for a laptop,
> but 81 parallel instances of LilyPond would certainly bring this system
> to its knees.

In practice, this doesn't happen, because our build system doesn't
parallelize all that well, and we don't have much large lp-book
documents anyway

In particular, we do

define make_subdirs
#       enforce order to avoid surprises due to implicit dependencies
[ "$(1)" = "$(filter $(1),$(SUBDIRS))" ] || (echo "*** {$(1)} is not a
subset of ordered set {$(SUBDIRS)}" 1>&2 && false)
+$(foreach d, $(1), $(MAKE) -C $(d) $(2) &&) true

ie. we only recurse into one subdirectory at a time.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dak, 2020-02-23 00:10:31]

On 2020/02/22 23:56:23, hanwenn wrote:
> On Sun, Feb 23, 2020 at 12:29 AM <mailto:dak@gnu.org> wrote:
> >
> > On 2020/02/22 23:18:43, hanwenn wrote:
> > > On 2020/02/22 23:17:50, hanwenn wrote:
> > > > you were already potentially in a state where you have 3 jobs each
> > spawning 3
> > > > lp-book invocations,
> > >
> > > whoops, I mean: 3 lp-book invocations spawning 3 lilypond subprocesses
> > each, for
> > > 9 in total.
> >
> > Considering that I build with CPU_COUNT=9 make -j9 I am pretty sure that
> > I'd have noticed if that were happening so far.  I may have a nice
> > number of cores (4) and memory considered plentiful (16GB) for a laptop,
> > but 81 parallel instances of LilyPond would certainly bring this system
> > to its knees.
> 
> In practice, this doesn't happen, because our build system doesn't
> parallelize all that well, and we don't have much large lp-book
> documents anyway

But this patch attempts to make it happen, right?

[hanwenn, 2020-02-23 19:29:00]

On 2020/02/23 00:10:31, dak wrote:
> On 2020/02/22 23:56:23, hanwenn wrote:
> > On Sun, Feb 23, 2020 at 12:29 AM <mailto:dak@gnu.org> wrote:
> > >
> > > On 2020/02/22 23:18:43, hanwenn wrote:
> > > > On 2020/02/22 23:17:50, hanwenn wrote:
> > > > > you were already potentially in a state where you have 3 jobs each
> > > spawning 3
> > > > > lp-book invocations,
> > > >
> > > > whoops, I mean: 3 lp-book invocations spawning 3 lilypond subprocesses
> > > each, for
> > > > 9 in total.
> > >
> > > Considering that I build with CPU_COUNT=9 make -j9 I am pretty sure that
> > > I'd have noticed if that were happening so far.  I may have a nice
> > > number of cores (4) and memory considered plentiful (16GB) for a laptop,
> > > but 81 parallel instances of LilyPond would certainly bring this system
> > > to its knees.
> > 
> > In practice, this doesn't happen, because our build system doesn't
> > parallelize all that well, and we don't have much large lp-book
> > documents anyway
> 
> But this patch attempts to make it happen, right?

abandoning this for 
https://codereview.appspot.com/555360043/