Keep a staff alive with multiple layers

Keep a staff alive with multiple layers

This allows the `VerticalAxisGroup.remove-layer' property to
accept a list of values. The layer will stay alive with any
other member of the Keep_alive_together_engrave group with a
remove-layer value in that list.

The principal reason for this patch was to allow the use of
MarkLine contexts in a Frenched score, where the context should
stay alive with any single staff in a StaffGroup. This
implementation should also allow additional flexibility with ossia
and divisi staves.

[dak, 2016-08-02 11:27:22]

Personally, I think that this "remove-layer is either a layer number for
ourselves or a list of supportive layer numbers who can keep us alive unless
this list is empty in which case we can be kept alive by our own
keep-alive-events" is just a bit too much meaning crammed into one property.

I've taken a look a

[dak, 2016-08-02 11:45:28]

On 2016/08/02 11:27:22, dak wrote:
> Personally, I think that this "remove-layer is either a layer number for
> ourselves or a list of supportive layer numbers who can keep us alive unless
> this list is empty in which case we can be kept alive by our own
> keep-alive-events" is just a bit too much meaning crammed into one property.
> 
> I've taken a look a

Sorry, wrong key.  I've taken a look at the code and it does not appear like
there is a danger for infinite loops: the checks for suicide relying on other
axis groups are not called recursively.  The only real danger we are dealing
with here is inefficiency, and particularly undefined behavior since a context
may decide to commit suicide based on the status of a context which already
suicided on the expectation of the first context staying alive.  If the order of
checks in such a situation are reversed, the outcome may be different.  Should
we declare this "somebody else's problem"?

At the low level, dependencies are decided by make-dead-when and keep-alive-with
properties (containing vertical axis groups) which are populated based on the
remove-layer setting.  If we want to provide the underlying flexibility, we'd
likely need a remove-layer property and _two_ properties carrying lists of layer
numbers.  Or _one_ property with a list where the _sign_ of the layers indicates
friend or foe.

However, using the sign will _fix_ layer numbers to be numbers.  When going
general like that, going via key-list? seems plausible, and then we'll require
two additional lists anyway since a symbol cannot provide a sign.

So would it make sense to provide the full hog that the low-level decision
engine supports?  Basically, we want the kind of user interface where people
need to think the least but still can do everything they want to.

[mark_opus11.net, 2016-08-02 13:55:56]

On 2016/08/02 11:45:28, dak wrote:
> I've taken a look at the code and it does not appear like there is a danger
> for infinite loops: the checks for suicide relying on other axis groups are
> not called recursively.  The only real danger we are dealing with here is
> inefficiency, and particularly undefined behavior since a context may decide
> to commit suicide based on the status of a context which already suicided on
> the expectation of the first context staying alive.  If the order of checks in
> such a situation are reversed, the outcome may be different.  Should we
> declare this "somebody else's problem"?

As far as I can tell, the iteration in Keep_alive_together_engraver::finalize
and Hara_kiri_group_spanner::request_suicide is always from top to bottom (as
presented in the \score block) and so should be predictable in that the first
encountered context will decide first. But I've not been able to construct a
situation where anything unintended happens.

> At the low level, dependencies are decided by make-dead-when and
> keep-alive-with properties (containing vertical axis groups) which are
> populated based on the remove-layer setting.  If we want to provide the
> underlying flexibility, we'd likely need a remove-layer property and _two_
> properties carrying lists of layer numbers.  Or _one_ property with a list
> where the _sign_ of the layers indicates friend or foe.
> 
> However, using the sign will _fix_ layer numbers to be numbers.  When going
> general like that, going via key-list? seems plausible, and then we'll require
> two additional lists anyway since a symbol cannot provide a sign.

So this would essentially be two user settable properties which provide indirect
access to the make-dead-when and keep-alive-with internal properties. With the
additional intervening logic of numerical ordering in the engraver taking care
of purely numerical cases.
 
> So would it make sense to provide the full hog that the low-level decision
> engine supports?  Basically, we want the kind of user interface where people
> need to think the least but still can do everything they want to.

Does this make the multi-level divisi situation that Abraham raised any easier?
http://lists.gnu.org/archive/html/lilypond-devel/2016-07/msg00227.html
Probably a bit.

New property names: "keep-alive-with-layers", "make-dead-with-layers"? Or
perhaps rename the internal properties to those and reclaim the shorter names
for the user properties?

[dak, 2016-08-02 14:41:47]

On 2016/08/02 13:55:56, mark_opus11.net wrote:
> On 2016/08/02 11:45:28, dak wrote:
> > However, using the sign will _fix_ layer numbers to be numbers.  When going
> > general like that, going via key-list? seems plausible, and then we'll
require
> > two additional lists anyway since a symbol cannot provide a sign.
> 
> So this would essentially be two user settable properties which provide
indirect
> access to the make-dead-when and keep-alive-with internal properties. With the
> additional intervening logic of numerical ordering in the engraver taking care
> of purely numerical cases.

I'd have those extra lists default to #f, implying the numerical behavior.  When
they are lists (including '()) then they'll behave strictly as lists.

> > So would it make sense to provide the full hog that the low-level decision
> > engine supports?  Basically, we want the kind of user interface where people
> > need to think the least but still can do everything they want to.
> 
> Does this make the multi-level divisi situation that Abraham raised any
easier?
> http://lists.gnu.org/archive/html/lilypond-devel/2016-07/msg00227.html
> Probably a bit.
> 
> New property names: "keep-alive-with-layers", "make-dead-with-layers"? Or
> perhaps rename the internal properties to those and reclaim the shorter names
> for the user properties?

remove-friends and remove-foes ?

Maybe this needs to be removal-friends, removal-foes, and removal-layer ? 
"remove" as a category name for "removal-related" in compound names just scans
badly.

[mark_opus11.net, 2016-08-02 15:05:15]

On 2016/08/02 14:41:47, dak wrote:
> On 2016/08/02 13:55:56, http://mark_opus11.net wrote:
> > New property names: "keep-alive-with-layers", "make-dead-with-layers"? Or
> > perhaps rename the internal properties to those and reclaim the shorter
names
> > for the user properties?
> 
> remove-friends and remove-foes ?
> 
> Maybe this needs to be removal-friends, removal-foes, and removal-layer ? 
> "remove" as a category name for "removal-related" in compound names just scans
> badly.

OK. I'll do something along these lines for the next patchset.

[mark_opus11.net, 2016-08-12 14:52:42]

Use key? (integer or symbol)

[mark_opus11.net, 2016-08-12 14:55:06]

On 2016/08/12 14:52:42, mark_opus11.net wrote:
> Use key? (integer or symbol)

I looked at doing this, as David suggested, by introducing two new
properties (removal-friends and removal-foes) but found the
resultant logical permutations too confusing to either document
clearly or be useful to a user.

So I have returned to a cleaner implementation of Patch Set 1, in
which remove-layer will accept a key with the following
behaviours:

- integer: layer removed when larger than smallest retained
- 'any: layer kept alive by any other layer
- #f: layer independent of the Keep_alive_together_engraver
- '(): layer does not participate in the layering decisions

Using a single polymorphic property avoids any possibility of
setting contradictory values for any single layer. The use of the
key type allows the potential for other behaviours to be added in
the future.

[dak, 2016-08-12 22:12:43]

On 2016/08/12 14:55:06, mark_opus11.net wrote:
> On 2016/08/12 14:52:42, http://mark_opus11.net wrote:
> > Use key? (integer or symbol)
> 
> I looked at doing this, as David suggested, by introducing two new
> properties (removal-friends and removal-foes) but found the
> resultant logical permutations too confusing to either document
> clearly or be useful to a user.

I'm somewhat surprised since I would have thought the _semantics_ reasonably
straightforward.  If the _use_ turns out to be awkward, it could probably
amended with a few scheme functions delivering appropriate context modifications
or possibly some music function.

I haven't actually bothered doing so so I might be overlooking something here. 
Can you illustrate some of the occuring problems/awkwardnesses?  I would have
thought the proposal comparatively straightforward to describe and use and so
I'd be interested to know where I was being too optimistic.

[mark_opus11.net, 2016-08-16 09:09:25]

On 2016/08/12 22:12:43, dak wrote:
> I'm somewhat surprised since I would have thought the _semantics_ reasonably
> straightforward.  If the _use_ turns out to be awkward, it could probably
> amended with a few scheme functions delivering appropriate context
modifications
> or possibly some music function.
> 
> I haven't actually bothered doing so so I might be overlooking something here.

> Can you illustrate some of the occuring problems/awkwardnesses?  I would have
> thought the proposal comparatively straightforward to describe and use and so
> I'd be interested to know where I was being too optimistic.

Well, whilst it is relatively straightforward to implement, it
also creates the possibility of setting up a situation where the
result is essentially unknowable without fully understanding the
implementation (or trial and error).

For example, what should happen here?

- Staff A
    - removal-layer = 2
    - removal-friends = #'(b)
- Staff B
    - removal-layer = #'b
- Staff C
    - removal-layer = 1

Since A's index is greater than C's, C is set as a foe.
Hara_kiri_group_spanner::request_suicide checks foes before
friends, therefore A will be removed whenever C is present,
regardless of B's existence.

Of course this _can_ be documented and explained (that
integer-based layering will trump symbol), but I'm just not sure
what we gain from allowing this mediated access to the friends and
foes lists. Whilst adding some considerable complexity to the
codebase.

For example, in the multi-level divisi situation mentioned here
http://lists.gnu.org/archive/html/lilypond-devel/2016-07/msg00227.html,
the difficulty is unrelated to the setting of the layer priorities
- the current integer prioritising works perfectly here - but is
in finding a way to specify multiple levels of "trickyness" in the
music itself.

Essentially, I have made the multi-list approach work, but since I
don't have a use case for it I don't feel confident that it is
done in a useful way.

[dak, 2016-08-16 16:21:26]

On 2016/08/16 09:09:25, mark_opus11.net wrote:
> On 2016/08/12 22:12:43, dak wrote:
> > I'm somewhat surprised since I would have thought the _semantics_ reasonably
> > straightforward.  If the _use_ turns out to be awkward, it could probably
> > amended with a few scheme functions delivering appropriate context
> modifications
> > or possibly some music function.
> > 
> > I haven't actually bothered doing so so I might be overlooking something
here.
> 
> > Can you illustrate some of the occuring problems/awkwardnesses?  I would
have
> > thought the proposal comparatively straightforward to describe and use and
so
> > I'd be interested to know where I was being too optimistic.
> 
> Well, whilst it is relatively straightforward to implement, it
> also creates the possibility of setting up a situation where the
> result is essentially unknowable without fully understanding the
> implementation (or trial and error).
> 
> For example, what should happen here?
> 
> - Staff A
>     - removal-layer = 2
>     - removal-friends = #'(b)
> - Staff B
>     - removal-layer = #'b
> - Staff C
>     - removal-layer = 1
> 
> Since A's index is greater than C's,

and since A.removal-foes is at its default value of #f

> C is set as a foe.

More like "considered a foe": the property stays #f of course.  But effectively
yes.

> Hara_kiri_group_spanner::request_suicide checks foes before
> friends, therefore A will be removed whenever C is present,
> regardless of B's existence.

Yes.

> Of course this _can_ be documented and explained (that
> integer-based layering will trump symbol),

Uh no?  Foes trump friends (if we want to avoid the unsavory message, we can try
"poison" and "food" instead which also makes the priorities more obvious, or
something else altogether), but integers and symbols are not pitted against one
another.  It's just that integers have natural friends and foes (when the
respective list is #f) and symbols don't.

> but I'm just not sure
> what we gain from allowing this mediated access to the friends and
> foes lists. Whilst adding some considerable complexity to the
> codebase.

I don't think it adds considerable complexity to the _codebase_ itself (at least
I think I could do it with comparatively straightforward changes), and I don't
consider the behavement just as irregular as you paint it.  It will complicate
the description, and it will create somewhat complicated use cases for
situations that were not possible before.

At any rate, our evaluations are not all that different and you are the one who
actually tried working with the code so I lean towards trusting your judgment. 
I just hope that we don't get hit by use cases which will again not yield to
this code while it might have been approachable by what _looks_ like a somewhat
more general scheme.  I really don't have much more than a gut feeling to work
with here.

[mark_opus11.net, 2016-08-16 17:38:05]

On 2016/08/16 16:21:26, dak wrote:
> At any rate, our evaluations are not all that different and you
> are the one who actually tried working with the code so I lean
> towards trusting your judgment. I just hope that we don't get
> hit by use cases which will again not yield to this code while
> it might have been approachable by what _looks_ like a somewhat
> more general scheme.  I really don't have much more than a gut
> feeling to work with here.

Perhaps I should advertise on the user list for some more
potential use cases for this. That could yield some situations
that I can ensure are satisfied before finalising this.

[dak, 2016-08-16 18:31:50]

On 2016/08/16 17:38:05, mark_opus11.net wrote:
> On 2016/08/16 16:21:26, dak wrote:
> > At any rate, our evaluations are not all that different and you
> > are the one who actually tried working with the code so I lean
> > towards trusting your judgment. I just hope that we don't get
> > hit by use cases which will again not yield to this code while
> > it might have been approachable by what _looks_ like a somewhat
> > more general scheme.  I really don't have much more than a gut
> > feeling to work with here.
> 
> Perhaps I should advertise on the user list for some more
> potential use cases for this. That could yield some situations
> that I can ensure are satisfied before finalising this.

That sounds like a good idea.

[mark_opus11.net, 2016-08-21 16:22:14]

Group lower-level contexts

[mark_opus11.net, 2016-08-21 16:23:57]

On 2016/08/21 16:22:14, mark_opus11.net wrote:
> Group lower-level contexts

This introduces a `VerticalAxisGroup.keep-alive-group' property                 
                
which can be set to a symbol to associate a subset of contexts                  

under the control of the same Keep_alive_together_engraver.                     
      
This group can further be set to operate only up to a maximum                
layer level with the `keep-alive-group-layer' property.

[dak, 2016-08-23 19:43:14]

On 2016/08/21 16:23:57, mark_opus11.net wrote:
> On 2016/08/21 16:22:14, http://mark_opus11.net wrote:
> > Group lower-level contexts
> 
> This introduces a `VerticalAxisGroup.keep-alive-group' property               
 
>                 
> which can be set to a symbol to associate a subset of contexts                
 
> 
> under the control of the same Keep_alive_together_engraver.                   
 
>       
> This group can further be set to operate only up to a maximum                
> layer level with the `keep-alive-group-layer' property.

Well, this doesn't seem to do more than the removal-friends/removal-foes
proposal (or does it?) and complicates things thoroughly.
So let me suggest yet-another-proposal that might require less intricate
planning (and the intricate planning aspect was the thing speaking against
removal-friends/removal-foes and, to be realistic, already against the currently
implemented code which nobody who has a need for it would seem to actually
understand well enough to actually use).

We create a new context, say RemovalGroup.  It has no engravers apart from the
Keep_alive_together_engraver.  And the Keep_alive_together_engraver does not
descend into contexts already having a Keep_alive_together_engraver, instead
letting _that_ Keep_alive_together_engraver make its decisions and, uh, consider
itself alive if any of its subcontexts is alive?  That way its layer number
could be assigned independently from the layer numbers of its subcontexts,
making it possible not to have to keep a global tally of subgroups but rather
use group numbers fresh at each layer.  Which would greatly facilitate providing
context mods for assigning all the layer numbers and stuff as they would not
need to take into account lower or higher groupings.

This would sound _almost_ as powerful as your plan and considerably simpler in
use.  The downside is that if you wanted to have a group with non-contiguous
elements, you'd have to put them contiguous into the source and juggle with
alignBelowContext and alignAboveContext for rearranging them.  But is that a
common use case?  I hope not, but I might be overlooking something in the
context of lyrics.

[mark_opus11.net, 2016-08-24 08:11:03]

On 2016/08/23 19:43:14, dak wrote:
> On 2016/08/21 16:23:57, http://mark_opus11.net wrote:
> > On 2016/08/21 16:22:14, http://mark_opus11.net wrote:
> > > Group lower-level contexts
> >
> > This introduces a `VerticalAxisGroup.keep-alive-group'
> > property which can be set to a symbol to associate a
> > subset of contexts under the control of the same
> > Keep_alive_together_engraver. This group can further be
> > set to operate only up to a maximum layer level with the
> > `keep-alive-group-layer' property.
>
> Well, this doesn't seem to do more than the
> removal-friends/removal-foes proposal (or does it?) and
> complicates things thoroughly.

True, although it does it via an interface which is more
closely related to the structure of the score and easier to
understand (IMO).

> We create a new context, say RemovalGroup.  It has no
> engravers apart from the Keep_alive_together_engraver.
> And the Keep_alive_together_engraver does not descend into
> contexts already having a Keep_alive_together_engraver,
> instead letting _that_ Keep_alive_together_engraver make
> its decisions and, uh, consider itself alive if any of its
> subcontexts is alive?  That way its layer number could be
> assigned independently from the layer numbers of its
> subcontexts, making it possible not to have to keep a
> global tally of subgroups but rather use group numbers
> fresh at each layer.  Which would greatly facilitate
> providing context mods for assigning all the layer numbers
> and stuff as they would not need to take into account
> lower or higher groupings.

Yes, I considered this also, but...

> This would sound _almost_ as powerful as your plan and
> considerably simpler in use.  The downside is that if you
> wanted to have a group with non-contiguous elements, you'd
> have to put them contiguous into the source and juggle
> with alignBelowContext and alignAboveContext for
> rearranging them.  But is that a common use case?

... it is specifically one of the use cases that was raised
on the list (which led me to the keep-alive-group proposal).
Unfortunately in these cases alignAboveContext /
alignBelowContext don't work to rearrange non-contiguous
contexts into potentially non-existent system-start-braces.

So I think that having non-contiguous sub-groups is pretty
much vital to the usefulness of the feature.

I agree that the `keep-alive-group-layer` property is not
particularly elegant, and somewhat redundant as it is just
max(remove-layer for layers in group). These values could be
found automatically by adding a loop through the Grobs to
populate a map of (group-name, max-remove-layer-values)
before the main loop. I didn't bother doing this until the
concept was worked out.

> I hope not, but I might be overlooking something in the
> context of lyrics.

Actually in the process of working on Kieren's example I did
find some issues involving Lyrics. Everything works, but
things do become rather verbose due to the need to specify
\overrides for each Lyrics context. Perhaps I'm missing
something here, but it seems an explicitly created Lyrics
context requires a named voice to align to using \lyricsto -
this is difficult to specify when using e.g. \partcombine to
create a staff.

A thought of how to deal with this would be to have a
remove-layer symbol value #'above, which would keep the
context alive with, and only with, the context above it.
This could then be applied in a global layout block to all
Lyrics contexts. It could be also useful for Dynamics and
similar supplementary contexts. Perhaps #'below could be
defined also for symmetry. Implementation is essentially:

if (scm_is_eq (this_layer, ly_symbol2scm ('above') and
    i == j + 1))

So, a proposal: The keep-alive-group stuff should in any
case be separated into another patch and discussed further.
I suggest returning to Patch Set 2, as described in
Comment #8, with the addition of the #'above symbol, as a
first step.

[mark_opus11.net, 2016-09-03 15:04:05]

Remove keep-alive-group properties, add above/below symbols