Add option to use Ghostscript API instead of forking

Add option to use Ghostscript API instead of forking

This is much more efficient than the current scheme when converting
many PostScript files, for example when building the documentation.
For the Notation Reference, lilypond-book now takes ~2m30s instead
of 5m to compile all snippets from the notation.tely file. Other
manuals benefit less, but still the time for 'make doc' on my system
improves by one third from around 33m to 22m.

An alternative would have been to always call gsapi_init_with_args
with the same arguments used until now. This indeed works and avoids
the overhead of forking, but the instance cannot be reused. Calling
gsapi_new_instance -> gsapi_init_with_args -> gsapi_delete_instance
for every conversion still means a lot of overhead and a prototype
suggested only a small gain compared to the previous solution.

This is not the default because recent versions of Ghostscript are
distributed under the AGPL and it's unclear what the implications of
linking to the library is. If built with support for the API, use
-dgs-api=#f to still fork the gs command.

[hahnjo, 2020-04-30 19:15:23]

This probably goes without saying, but touching such core functionality should
receive very rigid testing. I hope I got most things covered, but especially
manual feedback would be very much appreciated!

[dak, 2020-04-30 19:37:56]

On 2020/04/30 19:15:23, hahnjo wrote:
> This probably goes without saying, but touching such core functionality should
> receive very rigid testing. I hope I got most things covered, but especially
> manual feedback would be very much appreciated!

Sounds pretty good.  A great long-term perspective would be to use Cairo for
everything (or almost everything).  Last time I looked, the PDF document model
was insufficient for getting all LilyPond PDF features supported, but that may
have changed.

LilyPond's internal graphical data structures are certainly worth replacing with
something more compact.  If stencils were just Cairo data...

[lemzwerg, 2020-04-30 19:52:09]

LGTM, thanks!  Looks very good, indeed.

[hanwenn, 2020-05-01 06:28:56]

Technologically speaking, this is a brilliant idea, and I am all in favor it.

However, I think we can't enable this by default.

Ghostscript is licensed under AGPL, and linking it in makes LilyPond a derived
work, putting it under AGPL as well. That would be effectively a license change
of LilyPond, which would need consent of the current authors, and I think the
Scorio folks would not be happy with.

I suggest we make this an option that you have enable explicitly. If it is
enabled, we'd have to change the --license output to say AGPL as well.

[hanwenn, 2020-05-01 06:42:43]

https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
File lily/general-scheme.cc (right):

https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ") run";
This doesn't hook very deeply into GS internals or how we arrange the page.
Could we get the same speedup by putting the batching into an encompassing .ps
file, and calling GS on that file once?

Also: is there a GPL'd version of Ghostscript for which this works?

[hahnjo, 2020-05-01 07:58:32]

disclaimer: I'm not a lawyer, this is just my understanding of the licenses.

On 2020/05/01 06:28:56, hanwenn wrote:
> Technologically speaking, this is a brilliant idea, and I am all in favor it.
> 
> However, I think we can't enable this by default.
> 
> Ghostscript is licensed under AGPL, and linking it in makes LilyPond a derived
> work, putting it under AGPL as well. That would be effectively a license
change
> of LilyPond, which would need consent of the current authors, and I think the
> Scorio folks would not be happy with.

To put this on a solid basis, here's a quote from
https://www.gnu.org/licenses/gpl-3.0.html

13. Use with the GNU Affero General Public License.

Notwithstanding any other provision of this License, you have permission to link
or combine any covered work with a work licensed under version 3 of the GNU
Affero General Public License into a single combined work, and to convey the
resulting work. The terms of this License will continue to apply to the part
which is the covered work, but the special requirements of the GNU Affero
General Public License, section 13, concerning interaction through a network
will apply to the combination as such.

So I disagree that this is a license change as the "terms of this License will
continue to apply to the part which is the covered work". LilyPond stays
licensed under GPLv3. But yes "the special requirements of the GNU Affero
General Public License, section 13, concerning interaction through a network
will apply to the combination as such".

And here's the point that many have long argued about and continue to disagree:
What is "combination"? Linking to a library clearly is (and it's even spelled
out explicitly), but what about other "control flows" like calling GhostScript.
Here's another quote:

The “Corresponding Source” for a work in object code form means all the source
code needed to generate, install, and (for an executable work) run the object
code and to modify the work, including scripts to control those activities.
However, it does not include the work's System Libraries, or general-purpose
tools or generally available free programs which are used unmodified in
performing those activities but which are not part of the work. For example,
Corresponding Source includes interface definition files associated with source
files for the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require, such as by
intimate data communication or control flow between those subprograms and other
parts of the work.

So the first sentence and the example seem to include runtime dependencies
("subprograms"), but others (like System Libraries) are excluded. I don't know
whether this applies to GhostScript, but I'd argue that it's not a "major
essential component" of the OS. So in my opinion, LilyPond + GS already is AGPL
if you're using a recent version of the software.


> I suggest we make this an option that you have enable explicitly. If it is
> enabled, we'd have to change the --license output to say AGPL as well.

I'm open to making this change if people prefer. The major benefit will be for
documentation builds anyway, so the "normal" usage of LilyPond is fine with
forking.

[hanwenn, 2020-05-01 09:49:35]

On 2020/05/01 07:58:32, hahnjo wrote:
> disclaimer: I'm not a lawyer, this is just my understanding of the licenses.
> 
> On 2020/05/01 06:28:56, hanwenn wrote:
> > Technologically speaking, this is a brilliant idea, and I am all in favor
it.
> > 
> > However, I think we can't enable this by default.
> > 
> > Ghostscript is licensed under AGPL, and linking it in makes LilyPond a
derived
> > work, putting it under AGPL as well. That would be effectively a license
> change
> > of LilyPond, which would need consent of the current authors, and I think
the
> > Scorio folks would not be happy with.
> 
> To put this on a solid basis, here's a quote from
> https://www.gnu.org/licenses/gpl-3.0.html
> 
> 13. Use with the GNU Affero General Public License.
> 
> Notwithstanding any other provision of this License, you have permission to
link
> or combine any covered work with a work licensed under version 3 of the GNU
> Affero General Public License into a single combined work, and to convey the
> resulting work. The terms of this License will continue to apply to the part
> which is the covered work, but the special requirements of the GNU Affero
> General Public License, section 13, concerning interaction through a network
> will apply to the combination as such.
> 
> So I disagree that this is a license change as the "terms of this License will
> continue to apply to the part which is the covered work". LilyPond stays
> licensed under GPLv3. But yes "the special requirements of the GNU Affero
> General Public License, section 13, concerning interaction through a network
> will apply to the combination as such".
> 
> And here's the point that many have long argued about and continue to
disagree:
> What is "combination"? Linking to a library clearly is (and it's even spelled
> out explicitly), but what about other "control flows" like calling
GhostScript.
> Here's another quote:
> 
> The “Corresponding Source” for a work in object code form means all the source
> code needed to generate, install, and (for an executable work) run the object
> code and to modify the work, including scripts to control those activities.
> However, it does not include the work's System Libraries, or general-purpose
> tools or generally available free programs which are used unmodified in
> performing those activities but which are not part of the work. For example,
> Corresponding Source includes interface definition files associated with
source
> files for the work, and the source code for shared libraries and dynamically
> linked subprograms that the work is specifically designed to require, such as
by
> intimate data communication or control flow between those subprograms and
other
> parts of the work.
> 
> So the first sentence and the example seem to include runtime dependencies
> ("subprograms"), but others (like System Libraries) are excluded. I don't know
> whether this applies to GhostScript, but I'd argue that it's not a "major
> essential component" of the OS. So in my opinion, LilyPond + GS already is
AGPL
> if you're using a recent version of the software.

LilyPond can also output postscript with embedded fonts, which you should be
able to send to a Postscript printer (I have a PS printer too, but it's a 2001
model, which probably is too old to deal with OTF/CFF fonts). In this scenario,
you don't need Ghostscript, so LilyPond per se doesn't depend on GS.


> > I suggest we make this an option that you have enable explicitly. If it is
> > enabled, we'd have to change the --license output to say AGPL as well.
> 
> I'm open to making this change if people prefer. The major benefit will be for
> documentation builds anyway, so the "normal" usage of LilyPond is fine with
> forking.

Yeah, I think this is the better option. 

I might have a look at Cairo one of these days.

[hahnjo, 2020-05-01 09:52:38]

On 2020/05/01 09:49:35, hanwenn wrote:
> LilyPond can also output postscript with embedded fonts, which you should be
> able to send to a Postscript printer (I have a PS printer too, but it's a 2001
> model, which probably is too old to deal with OTF/CFF fonts). In this
scenario,
> you don't need Ghostscript, so LilyPond per se doesn't depend on GS.

Sure, but web applications where AGPL is of primary concern are more likely to
provide PDFs or PNGs, both of which are handled via GhostScript.

[dak, 2020-05-01 11:15:55]

On 2020/05/01 07:58:32, hahnjo wrote:
> disclaimer: I'm not a lawyer, this is just my understanding of the licenses.
> 
> On 2020/05/01 06:28:56, hanwenn wrote:
> > Technologically speaking, this is a brilliant idea, and I am all in favor
it.
> > 
> > However, I think we can't enable this by default.
> > 
> > Ghostscript is licensed under AGPL, and linking it in makes LilyPond a
derived
> > work, putting it under AGPL as well. That would be effectively a license
> change
> > of LilyPond, which would need consent of the current authors, and I think
the
> > Scorio folks would not be happy with.
> 
> To put this on a solid basis, here's a quote from
> https://www.gnu.org/licenses/gpl-3.0.html
> 
> 13. Use with the GNU Affero General Public License.
> 
> Notwithstanding any other provision of this License, you have permission to
link
> or combine any covered work with a work licensed under version 3 of the GNU
> Affero General Public License into a single combined work, and to convey the
> resulting work. The terms of this License will continue to apply to the part
> which is the covered work, but the special requirements of the GNU Affero
> General Public License, section 13, concerning interaction through a network
> will apply to the combination as such.
> 
> So I disagree that this is a license change as the "terms of this License will
> continue to apply to the part which is the covered work". LilyPond stays
> licensed under GPLv3. But yes "the special requirements of the GNU Affero
> General Public License, section 13, concerning interaction through a network
> will apply to the combination as such".
> 
> And here's the point that many have long argued about and continue to
disagree:
> What is "combination"? Linking to a library clearly is (and it's even spelled
> out explicitly), but what about other "control flows" like calling
GhostScript.
> Here's another quote:
> 
> The “Corresponding Source” for a work in object code form means all the source
> code needed to generate, install, and (for an executable work) run the object
> code and to modify the work, including scripts to control those activities.
> However, it does not include the work's System Libraries, or general-purpose
> tools or generally available free programs which are used unmodified in
> performing those activities but which are not part of the work. For example,
> Corresponding Source includes interface definition files associated with
source
> files for the work, and the source code for shared libraries and dynamically
> linked subprograms that the work is specifically designed to require, such as
by
> intimate data communication or control flow between those subprograms and
other
> parts of the work.
> 
> So the first sentence and the example seem to include runtime dependencies
> ("subprograms"), but others (like System Libraries) are excluded. I don't know
> whether this applies to GhostScript, but I'd argue that it's not a "major
> essential component" of the OS. So in my opinion, LilyPond + GS already is
AGPL
> if you're using a recent version of the software.
> 
> 
> > I suggest we make this an option that you have enable explicitly. If it is
> > enabled, we'd have to change the --license output to say AGPL as well.
> 
> I'm open to making this change if people prefer. The major benefit will be for
> documentation builds anyway, so the "normal" usage of LilyPond is fine with
> forking.

From Artifex' web site regaring licensing::

If you meet certain criteria, then we will not consider your distribution of
your application in an executable form to be in violation of AGPL, even though
you ship an executable product that includes your application and Ghostscript.
Here are the criteria:

    It is conspicuous and clear to the end user that he/she is getting access to
two separate pieces of software (i.e., AGPL Ghostscript in addition to the
application using this product).
    The end user has the ability to opt out of installing the AGPL version of
our products during the install process.
    Each AGPL module is separable and replaceable within the build.
    The available source code for the AGPL modules must be for the build that
corresponds with your binaries.

Any other redistribution is not licensed under AGPL. If you cannot meet the
requirements of AGPL, please fill out the form below to inquire about a
commercial license.

It is pretty likely that if software like Scorio.com wanted to avail themselves
of the advantages of the Ghostscript API performance, they would need to adhere
to the AGPL terms or take out a commercial license of Ghostscript.  We
definitely should not be distributing LilyPond in a form where it would only
work using the GhostScript API, but with a separate option that is off by
default, I don't see much of a problem.

[hahnjo, 2020-05-01 11:18:11]

https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
File lily/general-scheme.cc (right):

https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ") run";
On 2020/05/01 06:42:43, hanwenn wrote:
> This doesn't hook very deeply into GS internals or how we arrange the page.
> Could we get the same speedup by putting the batching into an encompassing .ps
> file, and calling GS on that file once?

I think this would require substantial refactoring of how LilyPond works. As far
as I understand, currently each input file is handled separately and temporary
files are deleted after processing. To get similar speedup, we have to reuse the
interpreter for the complete run of LilyPond.

> Also: is there a GPL'd version of Ghostscript for which this works?

GPL Ghostscript 9.06 from 2012 is apparently the last GPL version. You probably
don't want to stick with dependencies of that age ;-)

[hanwenn, 2020-05-01 11:31:23]

On Fri, May 1, 2020 at 1:18 PM <jonas.hahnfeld@gmail.com> wrote:
>
>
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
> File lily/general-scheme.cc (right):
>
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
> lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ")
> run";
> On 2020/05/01 06:42:43, hanwenn wrote:
> > This doesn't hook very deeply into GS internals or how we arrange the
> page.
> > Could we get the same speedup by putting the batching into an
> encompassing .ps
> > file, and calling GS on that file once?
>
> I think this would require substantial refactoring of how LilyPond
> works. As far as I understand, currently each input file is handled
> separately and temporary files are deleted after processing. To get
> similar speedup, we have to reuse the interpreter for the complete run
> of LilyPond.

We can hide it behind a facade, in much the same way that your patch
introduces hidden state. That would only work if we don't postprocess
the GS output after it's generated, but IIRC, we don't do that.

> > Also: is there a GPL'd version of Ghostscript for which this works?
>
> GPL Ghostscript 9.06 from 2012 is apparently the last GPL version. You
> probably don't want to stick with dependencies of that age ;-)




-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[hahnjo, 2020-05-01 11:35:26]

On 2020/05/01 11:31:23, hanwenn wrote:
> On Fri, May 1, 2020 at 1:18 PM <mailto:jonas.hahnfeld@gmail.com> wrote:
> >
> >
> >
> https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
> > File lily/general-scheme.cc (right):
> >
> >
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
> > lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ")
> > run";
> > On 2020/05/01 06:42:43, hanwenn wrote:
> > > This doesn't hook very deeply into GS internals or how we arrange the
> > page.
> > > Could we get the same speedup by putting the batching into an
> > encompassing .ps
> > > file, and calling GS on that file once?
> >
> > I think this would require substantial refactoring of how LilyPond
> > works. As far as I understand, currently each input file is handled
> > separately and temporary files are deleted after processing. To get
> > similar speedup, we have to reuse the interpreter for the complete run
> > of LilyPond.
> 
> We can hide it behind a facade, in much the same way that your patch
> introduces hidden state. That would only work if we don't postprocess
> the GS output after it's generated, but IIRC, we don't do that.

No, but LilyPond deletes the temporary PS unless the user wants to have it. This
would also need to be deferred after the batching .ps has been processed.
Instead this patch keeps the processing where it was before (right after
producing the PS), it just saves the interpreter for reuse.

[dak, 2020-05-01 11:41:13]

On 2020/05/01 11:18:11, hahnjo wrote:
> https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
> File lily/general-scheme.cc (right):
> 
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
> lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ") run";
> On 2020/05/01 06:42:43, hanwenn wrote:
> > This doesn't hook very deeply into GS internals or how we arrange the page.
> > Could we get the same speedup by putting the batching into an encompassing
.ps
> > file, and calling GS on that file once?
> 
> I think this would require substantial refactoring of how LilyPond works. As
far
> as I understand, currently each input file is handled separately and temporary
> files are deleted after processing. To get similar speedup, we have to reuse
the
> interpreter for the complete run of LilyPond.
> 
> > Also: is there a GPL'd version of Ghostscript for which this works?
> 
> GPL Ghostscript 9.06 from 2012 is apparently the last GPL version. You
probably
> don't want to stick with dependencies of that age ;-)

It would need to be GPLv3+ anyway.  Haven't checked, though.

preview-latex <https://www.gnu.org/software/auctex/preview-latex.html> works by
feeding scraps of PostScript control through stdin.  The main thing is that
startup and font loading are not once per graphics but per document.  I have
little enough clue how this would work for us, but obviously any way of using a
single Ghostscript session for multiple snippet conversions, including just a
large batch of input files generating lots of small output files with a
predictable naming scheme (this is what LaTeX/Ghostscript under control of
preview.sty can do) will get us significant benefits.  The Ghostscript API is
just one way of organising this.

[hahnjo, 2020-05-01 11:44:05]

On 2020/05/01 11:41:13, dak wrote:
> On 2020/05/01 11:18:11, hahnjo wrote:
> >
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
> > File lily/general-scheme.cc (right):
> > 
> >
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
> > lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ")
run";
> > On 2020/05/01 06:42:43, hanwenn wrote:
> > > This doesn't hook very deeply into GS internals or how we arrange the
page.
> > > Could we get the same speedup by putting the batching into an encompassing
> .ps
> > > file, and calling GS on that file once?
> > 
> > I think this would require substantial refactoring of how LilyPond works. As
> far
> > as I understand, currently each input file is handled separately and
temporary
> > files are deleted after processing. To get similar speedup, we have to reuse
> the
> > interpreter for the complete run of LilyPond.
> > 
> > > Also: is there a GPL'd version of Ghostscript for which this works?
> > 
> > GPL Ghostscript 9.06 from 2012 is apparently the last GPL version. You
> probably
> > don't want to stick with dependencies of that age ;-)
> 
> It would need to be GPLv3+ anyway.  Haven't checked, though.
> 
> preview-latex <https://www.gnu.org/software/auctex/preview-latex.html> works
by
> feeding scraps of PostScript control through stdin.  The main thing is that
> startup and font loading are not once per graphics but per document.  I have
> little enough clue how this would work for us, but obviously any way of using
a
> single Ghostscript session for multiple snippet conversions, including just a
> large batch of input files generating lots of small output files with a
> predictable naming scheme (this is what LaTeX/Ghostscript under control of
> preview.sty can do) will get us significant benefits.  The Ghostscript API is
> just one way of organising this.

Sure, we could also open an "interactive" gs command and keep it around for the
process. I didn't consider this different from using the API, but if the license
is happy as long as it's a separate process...

[dak, 2020-05-01 12:12:40]

On 2020/05/01 11:44:05, hahnjo wrote:
> On 2020/05/01 11:41:13, dak wrote:
> > On 2020/05/01 11:18:11, hahnjo wrote:
> > >
> https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.cc
> > > File lily/general-scheme.cc (right):
> > > 
> > >
> >
>
https://codereview.appspot.com/548030043/diff/583830043/lily/general-scheme.c...
> > > lily/general-scheme.cc:783: command += "(" + ly_scm2string (input) + ")
> run";
> > > On 2020/05/01 06:42:43, hanwenn wrote:
> > > > This doesn't hook very deeply into GS internals or how we arrange the
> page.
> > > > Could we get the same speedup by putting the batching into an
encompassing
> > .ps
> > > > file, and calling GS on that file once?
> > > 
> > > I think this would require substantial refactoring of how LilyPond works.
As
> > far
> > > as I understand, currently each input file is handled separately and
> temporary
> > > files are deleted after processing. To get similar speedup, we have to
reuse
> > the
> > > interpreter for the complete run of LilyPond.
> > > 
> > > > Also: is there a GPL'd version of Ghostscript for which this works?
> > > 
> > > GPL Ghostscript 9.06 from 2012 is apparently the last GPL version. You
> > probably
> > > don't want to stick with dependencies of that age ;-)
> > 
> > It would need to be GPLv3+ anyway.  Haven't checked, though.
> > 
> > preview-latex <https://www.gnu.org/software/auctex/preview-latex.html> works
> by
> > feeding scraps of PostScript control through stdin.  The main thing is that
> > startup and font loading are not once per graphics but per document.  I have
> > little enough clue how this would work for us, but obviously any way of
using
> a
> > single Ghostscript session for multiple snippet conversions, including just
a
> > large batch of input files generating lots of small output files with a
> > predictable naming scheme (this is what LaTeX/Ghostscript under control of
> > preview.sty can do) will get us significant benefits.  The Ghostscript API
is
> > just one way of organising this.
> 
> Sure, we could also open an "interactive" gs command and keep it around for
the
> process. I didn't consider this different from using the API, but if the
license
> is happy as long as it's a separate process...

I haven't looked too closely at the details of Ghostscript but the cursory
license information from Artifex I have seen was not specifically talking about
the API.  I have not looked closer though.  This is more of a question of "what
are we prepared to get sued over".  The legal situation with APIs and/or linking
is at best murky.  Many proprietary offerings work around that by casting
themselves in contract form, basically stating "you may not even use the
software unless you agree to those terms", and those define what is considered a
combined/derived work in the sense of the contract rather than the sense of the
law.  For us the line likely is not as much the law but when Artifex is likely
to spring in action.

That being said, the situation regarding Scorio, a proprietary entity using Free
Software as a component of delivering a web-based service with non-disclosed
components, is _exactly_ the reason Artifex chose the AGPL as a basis for their
business model selling commercial Ghostscript licenses.

So they quite likely are prepared to threaten legal action over it.  We just
have to make sure that they aren't going to target us, and preferably that it's
actually easy to get a commercially licensed Ghostscript version working in that
context so that parties like Scorio.com have the option to do so and know the
conditions where it makes sense for them to take out that option.

I don't see the API as the trigger point judging from the Ghostscript licensing
page, but I may well be overlooking something here.

[Valentin Villenave, 2020-05-01 21:58:47]

On 2020/05/01 12:12:40, dak wrote:
> That being said, the situation regarding Scorio, a proprietary entity using
Free
> Software as a component of delivering a web-based service with non-disclosed
> components, is _exactly_ the reason Artifex chose the AGPL as a basis for
their
> business model selling commercial Ghostscript licenses.

And since there is no (as far as I know) contractual, or even simply practical,
obligation tying us to Scorio, this is _also_ exactly why LilyPond itself ought
to follow the same way IMO.

Relicensing LilyPond as AGPL for the future 2.22 branch (or even 3.0 if that,
combined with all current optimization work, may be as good a reason as any for
a major version bump) should be on a the table, even independently of the
GhostScript situation. (That goes for many free-software programs actually,
which date back from before the SaaS trend and should now consider switching to
AGPL; there even were some talks of merging AGPL into GPLv4 in the future,
though the past few months have put many things on hold.)

Sure, that’d mean getting consent from as many contributors as we can track
down, but history has proved us it’s quite achievable -- remember the move to
GPLv3? It was basically hand-waved with a "are there any objections? Counting
once, counting twice, sold."

Just my two cents; carry on.

V.

[hanwenn, 2020-05-01 22:19:38]

On Fri, May 1, 2020 at 11:58 PM <v.villenave@gmail.com> wrote:
>
> On 2020/05/01 12:12:40, dak wrote:
> > That being said, the situation regarding Scorio, a proprietary entity
> using Free
> > Software as a component of delivering a web-based service with
> non-disclosed
> > components, is _exactly_ the reason Artifex chose the AGPL as a basis
> for their
> > business model selling commercial Ghostscript licenses.
>
> And since there is no (as far as I know) contractual, or even simply
> practical, obligation tying us to Scorio, this is _also_ exactly why
> LilyPond itself ought to follow the same way IMO.
>
> Relicensing LilyPond as AGPL for the future 2.22 branch (or even 3.0 if
> that, combined with all current optimization work, may be as good a
> reason as any for a major version bump) should be on a the table, even
> independently of the GhostScript situation. (That goes for many
> free-software programs actually, which date back from before the SaaS
> trend and should now consider switching to AGPL; there even were some
> talks of merging AGPL into GPLv4 in the future, though the past few
> months have put many things on hold.)

I don't want to relicense LilyPond under the AGPL.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[dak, 2020-05-01 22:38:02]

On 2020/05/01 22:19:38, hanwenn wrote:
> On Fri, May 1, 2020 at 11:58 PM <mailto:v.villenave@gmail.com> wrote:
> >
> > On 2020/05/01 12:12:40, dak wrote:
> > > That being said, the situation regarding Scorio, a proprietary entity
> > using Free
> > > Software as a component of delivering a web-based service with
> > non-disclosed
> > > components, is _exactly_ the reason Artifex chose the AGPL as a basis
> > for their
> > > business model selling commercial Ghostscript licenses.
> >
> > And since there is no (as far as I know) contractual, or even simply
> > practical, obligation tying us to Scorio, this is _also_ exactly why
> > LilyPond itself ought to follow the same way IMO.
> >
> > Relicensing LilyPond as AGPL for the future 2.22 branch (or even 3.0 if
> > that, combined with all current optimization work, may be as good a
> > reason as any for a major version bump) should be on a the table, even
> > independently of the GhostScript situation. (That goes for many
> > free-software programs actually, which date back from before the SaaS
> > trend and should now consider switching to AGPL; there even were some
> > talks of merging AGPL into GPLv4 in the future, though the past few
> > months have put many things on hold.)
> 
> I don't want to relicense LilyPond under the AGPL.
> 
> -- 
> Han-Wen Nienhuys - mailto:hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

I am actually with Han-Wen here.  Ultimately, a license should draw the line you
are willing to insist on.  Pursuing GPL breaches is already a tricky proposition
(and essentially only reasonable for significant copyright holders: the FSF
cannot pitch in here since they have no standing).  Tracking down distributions
of LilyPond in a legally convincing manner is hard enough.  Evidence for running
as a service is just too ephemeral.  I don't want to go there myself.

[hahnjo, 2020-05-02 09:45:01]

Add option to use Ghostscript API instead of forking

[hahnjo, 2020-05-02 09:49:44]

On 2020/05/01 06:28:56, hanwenn wrote:
> I suggest we make this an option that you have enable explicitly.

done

> If it is enabled, we'd have to change the --license output to say AGPL as
well.

I thought about this and decided against it:
1. LilyPond stays under GPL, even if the whole may be AGPL.
2. I don't see a good option to find out the license of the library we link to.
3. It's not our business of giving legal advice. Instead I added a warning to
the configure option "Beware of licensing implications!"

[dak, 2020-05-02 10:04:14]

On 2020/05/02 09:49:44, hahnjo wrote:
> On 2020/05/01 06:28:56, hanwenn wrote:
> > I suggest we make this an option that you have enable explicitly.
> 
> done
> 
> > If it is enabled, we'd have to change the --license output to say AGPL as
> well.
> 
> I thought about this and decided against it:
> 1. LilyPond stays under GPL, even if the whole may be AGPL.
> 2. I don't see a good option to find out the license of the library we link
to.
> 3. It's not our business of giving legal advice. Instead I added a warning to
> the configure option "Beware of licensing implications!"

If it is a compile time option that actually links stuff (and the headers
suggest that), we should add something like

This version of LilyPond has been compiled and linked with a version
of Ghostscript licensed under the AGPL.

to the --license text.  The implications are then for the user to find out.
We should also add this to the license information in the documentation
(namely that --license should give you the information whether there is
the Ghostscript API linked in which case additional terms and conditions apply).

Yes, this kind of stuff is a pain.  But it's probably less pain in the end to
be proactive about it.  Our main duty is making sure that users know what terms
they are dealing with so that the buck stops at the user rather than at us.

[hahnjo, 2020-05-02 10:09:33]

On 2020/05/02 10:04:14, dak wrote:
> On 2020/05/02 09:49:44, hahnjo wrote:
> > On 2020/05/01 06:28:56, hanwenn wrote:
> > > I suggest we make this an option that you have enable explicitly.
> > 
> > done
> > 
> > > If it is enabled, we'd have to change the --license output to say AGPL as
> > well.
> > 
> > I thought about this and decided against it:
> > 1. LilyPond stays under GPL, even if the whole may be AGPL.
> > 2. I don't see a good option to find out the license of the library we link
> to.
> > 3. It's not our business of giving legal advice. Instead I added a warning
to
> > the configure option "Beware of licensing implications!"
> 
> If it is a compile time option that actually links stuff (and the headers
> suggest that), we should add something like
> 
> This version of LilyPond has been compiled and linked with a version
> of Ghostscript licensed under the AGPL.

Please re-read point 2 above. Old Ghostscript versions are not AGPL and other
implementations may not be either.

[hanwenn, 2020-05-02 10:12:45]

On Sat, May 2, 2020 at 12:09 PM <jonas.hahnfeld@gmail.com> wrote:

> > This version of LilyPond has been compiled and linked with a version
> > of Ghostscript licensed under the AGPL.
>
> Please re-read point 2 above. Old Ghostscript versions are not AGPL and
> other implementations may not be either.

it's easy to discover if GS is AGPL or not (assuming that the command
line binary is the same as the API we're linking):

$ gs
GPL Ghostscript 9.27 (2019-04-04)
Copyright (C) 2018 Artifex Software, Inc.  All rights reserved.
This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:
see the file COPYING for details.



(BTW, I've been unable to find the GS API on fedora 31 :-/ )
-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

[hanwenn, 2020-05-02 10:15:40]

https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.cc
File lily/general-scheme.cc (right):

https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.c...
lily/general-scheme.cc:778: free (a);
the code mixes setting up the GS instance (memory management etc) with handling
the file. Does it have to be this way? Can we have a 

  class Ghostscript {
     process(string file, string device);
     close();
  };

  Ghostscript *get_gs(vector<string> args);

instead?

I think it should be possible to construct the API such that we always have
ly:gs , and that it falls back to shelling out to GS if the API is not
available.

[hahnjo, 2020-05-02 10:20:25]

On 2020/05/02 10:12:45, hanwenn wrote:
> On Sat, May 2, 2020 at 12:09 PM <mailto:jonas.hahnfeld@gmail.com> wrote:
> 
> > > This version of LilyPond has been compiled and linked with a version
> > > of Ghostscript licensed under the AGPL.
> >
> > Please re-read point 2 above. Old Ghostscript versions are not AGPL and
> > other implementations may not be either.
> 
> it's easy to discover if GS is AGPL or not (assuming that the command
> line binary is the same as the API we're linking):
> 
> $ gs
> GPL Ghostscript 9.27 (2019-04-04)
> Copyright (C) 2018 Artifex Software, Inc.  All rights reserved.
> This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:
> see the file COPYING for details.

That's a bold claim that may often hold, but not always. Feel free to do this
change after this one landed, but I'm certainly not going to give imperfect
legal advice.

> (BTW, I've been unable to find the GS API on fedora 31 :-/ )

https://apps.fedoraproject.org/packages/libgs-devel/ ?

[hahnjo, 2020-05-02 10:22:15]

https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.cc
File lily/general-scheme.cc (right):

https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.c...
lily/general-scheme.cc:778: free (a);
On 2020/05/02 10:15:40, hanwenn wrote:
> the code mixes setting up the GS instance (memory management etc) with
handling
> the file. Does it have to be this way? Can we have a 
> 
>   class Ghostscript {
>      process(string file, string device);
>      close();
>   };
> 
>   Ghostscript *get_gs(vector<string> args);
> 
> instead?
> 
> I think it should be possible to construct the API such that we always have
> ly:gs , and that it falls back to shelling out to GS if the API is not
> available.

No, because there are two types of arguments when using the API: args and
device_args where the latter is added to command below. This uses a different
syntax and some properties are called differently (HWResolution vs -r for
example).

[hahnjo, 2020-05-02 10:48:20]

add license information

[hahnjo, 2020-05-02 10:49:11]

On 2020/05/02 10:48:20, hahnjo wrote:
> add license information

$ lilypond -w
GNU LilyPond 2.21.2

Copyright (c) 1996--2020 by
  Han-Wen Nienhuys <hanwen@xs4all.nl>
  Jan Nieuwenhuizen <janneke@gnu.org>
  and others.

    This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as 
published by the Free Software Foundation, either version 3 of
the License, or (at your option) any later version.

    This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.

    You should have received a copy of the
GNU General Public License along with this program; if not, write to
the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
Boston, MA 02111-1307, USA.

linked against Ghostscript:
GPL Ghostscript 9.52 (2020-03-19)
Copyright (C) 2020 Artifex Software, Inc.  All rights reserved.
This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:
see the file COPYING for details.

[hanwenn, 2020-05-02 11:05:21]

https://codereview.appspot.com/548030043/diff/548050047/lily/main.cc
File lily/main.cc (right):

https://codereview.appspot.com/548030043/diff/548050047/lily/main.cc#newcode345
lily/main.cc:345: printf ("%s", (_ ("linked against Ghostscript:").c_str ()));
excellent, thanks for bearing with us.

[hanwenn, 2020-05-02 11:07:12]

LGTM

https://codereview.appspot.com/548030043/diff/548050047/lily/general-scheme.cc
File lily/general-scheme.cc (right):

https://codereview.appspot.com/548030043/diff/548050047/lily/general-scheme.c...
lily/general-scheme.cc:777: for (char *a : passed_args)
I'll try to clean this up a bit in a follow-up change.

[hahnjo, 2020-05-03 09:46:56]

rebase + allow -dgs-api=#f to still fork

[dak, 2020-05-03 10:40:25]

On 2020/05/03 09:46:56, hahnjo wrote:
> rebase + allow -dgs-api=#f to still fork

I decided to look up the terms of the Affero GPL version 3.  The relevant
differences are:

> Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users interacting with
it remotely through a computer network (if your version supports such
interaction) an opportunity to receive the Corresponding Source of your version
by providing access to the Corresponding Source from a network server at no
charge, through some standard or customary means of facilitating copying of
software. This Corresponding Source shall include the Corresponding Source for
any work covered by version 3 of the GNU General Public License that is
incorporated pursuant to the following paragraph.

Note that we don't modify Ghostscript (I hope).

> Notwithstanding any other provision of this License, you have permission to
link or combine any covered work with a work licensed under version 3 of the GNU
General Public License into a single combined work, and to convey the resulting
work. The terms of this License will continue to apply to the part which is the
covered work, but the work with which it is combined will remain governed by
version 3 of the GNU General Public License.

That does not sound like LilyPond would get "infected" by the AGPL.  Not does it
sound like any conditions would trigger unless someone modifies Ghostscript. 
Frankly, this almost sounds like the license was written explicitly in a manner
where it would not get in the hair of GhostScript API users.

However, what this does mean is that we very much should avoid to be modifying
the Ghostscript sources in any manner, including any patches.  As long as we
don't modify Ghostscript, it sounds like we (and, by the way, Scorio.com) should
be free of other obligations (except of course the normal GPLv3 terms) even when
linking with it.

Of course, we should still prominently mention that this binary contains AGPL
parts.  But I think we could link with the GS API by default and enable it if
that makes sense.

[dak, 2020-05-03 10:44:24]

On 2020/05/02 10:22:15, hahnjo wrote:
> https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.cc
> File lily/general-scheme.cc (right):
> 
>
https://codereview.appspot.com/548030043/diff/559960055/lily/general-scheme.c...
> lily/general-scheme.cc:778: free (a);
> On 2020/05/02 10:15:40, hanwenn wrote:
> > the code mixes setting up the GS instance (memory management etc) with
> handling
> > the file. Does it have to be this way? Can we have a 
> > 
> >   class Ghostscript {
> >      process(string file, string device);
> >      close();
> >   };
> > 
> >   Ghostscript *get_gs(vector<string> args);
> > 
> > instead?
> > 
> > I think it should be possible to construct the API such that we always have
> > ly:gs , and that it falls back to shelling out to GS if the API is not
> > available.
> 
> No, because there are two types of arguments when using the API: args and
> device_args where the latter is added to command below. This uses a different
> syntax and some properties are called differently (HWResolution vs -r for
> example).

It may well be that calling them "HWResolution" instead of -r would work fine
and -r is just provided as a courtesy abbreviation.  I don't know whether this
holds true, just bringing this up in case it might make for a different view of
the situation.

[hahnjo, 2020-05-03 10:46:49]

On 2020/05/03 10:40:25, dak wrote:
> On 2020/05/03 09:46:56, hahnjo wrote:
> > rebase + allow -dgs-api=#f to still fork
> 
> I decided to look up the terms of the Affero GPL version 3.  The relevant
> differences are:
> [...]

I think you're not looking at the right direction, ie we're not integrating GPL
(LilyPond) into AGPL (Ghostscript), but the other way around. In that case
Section 13 of the GPL 3.0 is relevant:

> Notwithstanding any other provision of this License, you have permission to
link or combine any covered work with a work licensed under version 3 of the GNU
Affero General Public License into a single combined work, and to convey the
resulting work. The terms of this License will continue to apply to the part
which is the covered work, but the special requirements of the GNU Affero
General Public License, section 13, concerning interaction through a network
will apply to the combination as such.

Note there is no mention of modifying the work, AGPL applies "to the combination
as such".

[dak, 2020-05-03 11:38:56]

On 2020/05/03 10:46:49, hahnjo wrote:
> On 2020/05/03 10:40:25, dak wrote:
> > On 2020/05/03 09:46:56, hahnjo wrote:
> > > rebase + allow -dgs-api=#f to still fork
> > 
> > I decided to look up the terms of the Affero GPL version 3.  The relevant
> > differences are:
> > [...]
> 
> I think you're not looking at the right direction, ie we're not integrating
GPL
> (LilyPond) into AGPL (Ghostscript), but the other way around. In that case
> Section 13 of the GPL 3.0 is relevant:
> 
> > Notwithstanding any other provision of this License, you have permission to
> link or combine any covered work with a work licensed under version 3 of the
GNU
> Affero General Public License into a single combined work, and to convey the
> resulting work. The terms of this License will continue to apply to the part
> which is the covered work, but the special requirements of the GNU Affero
> General Public License, section 13, concerning interaction through a network
> will apply to the combination as such.
> 
> Note there is no mention of modifying the work, AGPL applies "to the
combination
> as such".

Well, if it states "the special requirements of the AGPL apply to the
combination" and the AGPL explicitly says that the combination remains licensed
individually, it definitely sounds to me that the the requirement to distribute
the source of _both_ components triggers when the AGPL component gets modified.

Artifex states on its web site <https://artifex.com/licensing/agpl/>:

Network Use

> If you modify our software and make the functionality of the software
available to users interacting with it remotely through a computer network (such
as a SaaS or web-based application) you must share your application source code.

> Any modified versions must also be made available to those interacting with
the software remotely.

To me that sounds like modification of the GPLed part does not count as
modification of the AGPLed part.  If we distribute a combined binary package, of
course we have to state that the package is not, as a whole, governed by the
rules of the GPL since any modification of the AGPLed part would trigger the
requirement to make the combined source available to network users of the
software.

The FSF's GPL FAQ (which of course is not legally binding) states
<https://www.gnu.org/licenses/gpl-faq.html#AGPLv3InteractingRemotely>:
In AGPLv3, what counts as “interacting with [the software] remotely through a
computer network?” (#AGPLv3InteractingRemotely)

    If the program is expressly designed to accept user requests and send
responses over a network, then it meets these criteria. Common examples of
programs that would fall into this category include web and mail servers,
interactive web-based applications, and servers for games that are played
online.

    If a program is not expressly designed to interact with a user through a
network, but is being run in an environment where it happens to do so, then it
does not fall into this category. For example, an application is not required to
provide source merely because the user is running it over SSH, or a remote X
session.

Again, that sounds to me like the AGPL requirements of using Ghostscript as an
integrated component of web-based PDF generation would not be different when
using Ghostscript unmodified from the command line as opposed to linked
unmodified into a command-line version (modified or not) of LilyPond.  There
would be a conceivable difference if the server itself were linked with the GS
API.

At any rate: my take on this involved evaluating legalese and making a judgment
on it and on the associated risks and possible interpretations.  But that's not
my job, and it shouldn't be ours.

The main thing we can do here is be transparent.  That basically leaves the
responsibility of making decisions with or without contact to Artifex to the
respective users wanting to provide a web service.  And to be honest: I don't
really see that using the GS API is making a significant difference in that
regard.  I think that would have been a requirement anyway.

[hahnjo, 2020-05-06 18:33:23]

In case you're interested how I found out about this:
https://www.hahnjo.de/blog/2020/05/06/recursively-timeing-processes.html

before this change (commit 8fa5191cbf)
/usr/bin/gsc called 18100 times, total of 2044s
lilypond called 937 times, total of 1120s

after an early version of this change:
/usr/bin/gsc is obviously gone; lilypond is still called 937 times with a total
of 1545s
-> difference of 1619s

[Valentin Villenave, 2020-05-06 22:53:49]

On 5/6/20, jonas.hahnfeld@gmail.com <jonas.hahnfeld@gmail.com> wrote:
> In case you're interested how I found out about this:
> https://www.hahnjo.de/blog/2020/05/06/recursively-timeing-processes.html

Good to know, thanks! (And the resulting improvement is indeed impressive.)

Cheers,
-- V.

[hanwenn, 2020-05-09 09:52:22]

https://codereview.appspot.com/548030043/diff/559960069/scm/ps-to-png.scm
File scm/ps-to-png.scm (right):

https://codereview.appspot.com/548030043/diff/559960069/scm/ps-to-png.scm#new...
scm/ps-to-png.scm:184: (if (not (= 1 anti-alias-factor))
please submit this as is, but for a follow-up, it would be interesting to get
rid of the homebuilt anti-aliasing, and use the GS DownScaleFactor variable
instead.