Issue 553700043: scripts/build/scan-mf-deps: script to generate MF dependencies (Closed)
|
Created:
5 years ago by hanwenn Modified:
4 years, 11 months ago CC:
lilypond-devel_gnu.org Visibility:
Public. |
Description mf/GNUmakefile: move logic to scripts
This is a small enabling step for moving to a different build system
* Metafont dependency scanning uses advanced GNU Make scripting. Use a
python script instead
* The mf2pt invocation is intricate. Use a shell script instead
* Take the .log and .tfm files from the MetaPost run. (We were
compiling all fonts twice.)
* Remove some unused variables and rules
Patch Set 1 #Patch Set 2 : more cleanup #
Total comments: 1
Patch Set 3 : /bin/dash #
Total comments: 1
Patch Set 4 : analyze log file #Patch Set 5 : realpath #Patch Set 6 : realpath 2 #Patch Set 7 : /bin/sh #
Total comments: 6
Patch Set 8 : oops - restored overly zealous deletions #Patch Set 9 : local-clean #
Total comments: 12
Patch Set 10 : -recorder #
Total comments: 4
Patch Set 11 : rename. #Patch Set 12 : rename (2nd try) #
Total comments: 4
Patch Set 13 : touch multiple files #
Total comments: 1
Patch Set 14 : jonas' suggestion #
Total comments: 16
Patch Set 15 : Final comments. #Patch Set 16 : oops #
Total comments: 2
Patch Set 17 : Jonas' nits #
MessagesTotal messages: 63
more cleanup
Sign in to reply to this message.
this looks like a good idea, thanks! https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:4: mf2pt1=$(realpath $1) I only know `realpath` as a GNU make command. Ditto for other commands below in this script. Is this script work in progress?
Sign in to reply to this message.
/bin/dash
Sign in to reply to this message.
Why would we want to move away from GNU make as a build system?
Sign in to reply to this message.
On 2020/03/14 23:46:17, dak wrote: > Why would we want to move away from GNU make as a build system? The current build system is a per directory system, which is fundamentally flawed. (See http://aegis.sourceforge.net/auug97.pdf). This has several implications: * we can't get enough parallelism in the doc build * there are several underspecified dependencies (eg. if you change something scripts/build/ , a build in mf/ will not pick up the change) then there are problems with stepmake * because each subdirectory has its own output directory (eg. mf/out/ ), a significant part of our build is involved with copying/symlinking files so we don't have out-www/ in our HTML links. * our build system relies heavily on GNU make magic. As a programming language, GNU Make just isn't very good. It's not expressive, and it has no typesystem. This makes it hard for people to work on the build system. * stepmake tried to be generic Make templating system (but did a bad job at it). At the same time, the genericity is causing complexity I have been experimenting with Ninja (https://ninja-build.org/), but Ninja has a very restricted model. Make functions, wildcards, etc. are not possible. In order to advance my experiments, I'm trying to clean up the build so I can port more of the build over. But even if we don't migrate to something else, cleaning up the build makes it easier to maintain. See https://github.com/hanwen/lilypond/tree/ninja if you want to try it out. Moving to something el
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/545710043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/545710043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:4: mf2pt1=$(realpath $1) Second try. According to https://unix.stackexchange.com/questions/136494/whats-the-difference-between-... https://unix.stackexchange.com/questions/101080/realpath-command-not-found it seems that today a `realpath` binary is available on most systems. However, it is not mentioned in the suite of programs that autoconf uses universally. I thus recommend that we add a `configure` test for it – or at least mention it in the list of LilyPond build requisites.
Sign in to reply to this message.
On 2020/03/14 23:57:48, hanwenn wrote: > I have been experimenting with Ninja (https://ninja-build.org/), but Ninja has a > very restricted model. Make functions, wildcards, etc. are not possible. In > order to advance my experiments, I'm trying to clean up the build so I can port > more of the build over. From the website: "it [Ninja] is designed to have its input files generated by a higher-level build system". And that's to be taken literally, the syntax is so low-level that it's no fun to read it, let alone write. So big NACK from me for switching to Ninja. Both Meson and CMake are able to generate files for Ninja, but Meson is pretty new and I'm not sure yet whether it will become one of those projects that goes away after a few years (see Scons). CMake would certainly work, but has its own problems and corner cases. In sum we should definitely cleanup the current build system and see where it gets us. What I'm not really happy with is replacing all of this with custom Python scripts, however small they are. Skimming over scan-mf-deps.py, that looks totally doable in around half the lines with a sh script. What's the motivation to use Python for all of this?
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 12:05 PM <jonas.hahnfeld@gmail.com> wrote: > > On 2020/03/14 23:57:48, hanwenn wrote: > > I have been experimenting with Ninja (https://ninja-build.org/), but > Ninja has a > > very restricted model. Make functions, wildcards, etc. are not > possible. In > > order to advance my experiments, I'm trying to clean up the build so I > can port > > more of the build over. > > From the website: "it [Ninja] is designed to have its input files > generated by a higher-level build system". And that's to be taken > literally, the syntax is so low-level that it's no fun to read it, let > alone write. So big NACK from me for switching to Ninja. Have you looked at the code I wrote? I know about Ninja, and in fact I personally know its creator. Using Ninja means generating a Ninja file. I looked for a bit at CMake and Meson, but I'm not convinced they'll be a win; for one I'd have to learn yet another poorly conceived and limited language. The LilyPond build is highly custom (metafont, lilypond-book, texi2html), so it's also not obvious that either they'll bring much to the table. So I'm simply writing the build.ninja file from a Python script directly, without any attempt to make it generic. We'll see how far that brings me. > Both Meson and CMake are able to generate files for Ninja, but Meson is > pretty new and I'm not sure yet whether it will become one of those > projects that goes away after a few years (see Scons). CMake would > certainly work, but has its own problems and corner cases. In sum we > should definitely cleanup the current build system and see where it gets > us. > > What I'm not really happy with is replacing all of this with custom > Python scripts, however small they are. Skimming over scan-mf-deps.py, > that looks totally doable in around half the lines with a sh script. > What's the motivation to use Python for all of this? What's the motivation to use sh for all this? You can certainly make it shorter with shell, but will it be significantly simpler, easier to understand or easier to maintain? I decided for Python in this case, because recursion seems like something I don't want to do in a sh script. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
On 3/15/20, 8:39 AM, "lilypond-devel on behalf of Han-Wen Nienhuys" <lilypond-devel-bounces+c_sorensen=byu.edu@gnu.org on behalf of hanwenn@gmail.com> wrote: On Sun, Mar 15, 2020 at 12:05 PM <jonas.hahnfeld@gmail.com> wrote: > > > What I'm not really happy with is replacing all of this with custom > Python scripts, however small they are. Skimming over scan-mf-deps.py, > that looks totally doable in around half the lines with a sh script. > What's the motivation to use Python for all of this? What's the motivation to use sh for all this? You can certainly make it shorter with shell, but will it be significantly simpler, easier to understand or easier to maintain? I decided for Python in this case, because recursion seems like something I don't want to do in a sh script. Personally, I'm totally in favor of using python for creating ninja files. We have a well-documented, well-maintained, powerful language. No different flavors (bash, sh, dash, csh, ksh...). Reading the docs, ninja looks promising as a build system. Seeing that it's used to build Chrome gives me some warm and fuzzy feelings. Years ago I hoped we could use Scons to smooth things out, but it got swamped in performance issues. We have some hope that Ninja won't suffer the same fate. I'm excited to see what comes out of this. Thanks, Carl
Sign in to reply to this message.
>> What's the motivation to use Python for all of this?
>
> What's the motivation to use sh for all this?
>
> You can certainly make it shorter with shell, but will it be
> significantly simpler, easier to understand or easier to maintain?
>
> I decided for Python in this case, because recursion seems like
> something I don't want to do in a sh script.
I agree with Han-Wen.
Jonas, the rule is very simple: Use whatever you think is best. A
contributor invests time and has to make sure that the code works and
is well-documented. But why should we enforce a certain programming
language (of the ones we already use, that is)?
For example, if I contributed a more complicated script, it would be
written in Perl...
Werner
Sign in to reply to this message.
Werner LEMBERG <wl@gnu.org> writes: >>> What's the motivation to use Python for all of this? >> >> What's the motivation to use sh for all this? >> >> You can certainly make it shorter with shell, but will it be >> significantly simpler, easier to understand or easier to maintain? >> >> I decided for Python in this case, because recursion seems like >> something I don't want to do in a sh script. > > I agree with Han-Wen. > > Jonas, the rule is very simple: Use whatever you think is best. A > contributor invests time and has to make sure that the code works and > is well-documented. But why should we enforce a certain programming > language (of the ones we already use, that is)? > > For example, if I contributed a more complicated script, it would be > written in Perl... I don't think "Use whatever you think is best." makes a lot of sense for a community-maintained project. It ends up leaving the project in an incoherent state with lots of different approaches that made personal sense to the person having started them. Makefiles are an established technique well-known to most contributors. Recursive python scripts for writing human-unreadable controlling files in some build system that is the current fad of the year incompatible with anything else: who is going to maintain that five years from now? -- David Kastrup
Sign in to reply to this message.
On 2020/03/15 14:39:06, hanwenn wrote: > On Sun, Mar 15, 2020 at 12:05 PM <mailto:jonas.hahnfeld@gmail.com> wrote: > > On 2020/03/14 23:57:48, hanwenn wrote: > > > I have been experimenting with Ninja (https://ninja-build.org/), but > > Ninja has a > > > very restricted model. Make functions, wildcards, etc. are not > > possible. In > > > order to advance my experiments, I'm trying to clean up the build so I > > can port > > > more of the build over. > > > > From the website: "it [Ninja] is designed to have its input files > > generated by a higher-level build system". And that's to be taken > > literally, the syntax is so low-level that it's no fun to read it, let > > alone write. So big NACK from me for switching to Ninja. > > Have you looked at the code I wrote? I know about Ninja, and in fact > I personally know its creator. > > Using Ninja means generating a Ninja file. I looked for a bit at CMake > and Meson, but I'm not convinced they'll be a win; for one I'd have to > learn yet another poorly conceived and limited language. The LilyPond > build is highly custom (metafont, lilypond-book, texi2html), so it's > also not obvious that either they'll bring much to the table. > > So I'm simply writing the build.ninja file from a Python script > directly, without any attempt to make it generic. We'll see how far > that brings me. I did so now and ... does this mean you want to write yet another meta build system? That sounds horrible to maintain. > > What's the motivation to use Python for all of this? > > What's the motivation to use sh for all this? Let me explain with my background of porting to Python 3: It was a nightmare which leads me to the conclusion that Python scripts are not easy to maintain over an extended period of time. I recently read https://gregoryszorc.com/blog/2020/01/13/mercurial's-journey-to-and-reflectio... and this very much confirms this point. sh instead is pretty much the same since decades and *very* portable. On 2020/03/15 15:10:05, wl_gnu.org wrote: > Jonas, the rule is very simple: Use whatever you think is best. A > contributor invests time and has to make sure that the code works and > is well-documented. But why should we enforce a certain programming > language (of the ones we already use, that is)? Yes, and others might disagree with a particular choice. For build scripts, I think it's the wrong language (see above).
Sign in to reply to this message.
Carl Sorensen <c_sorensen@byu.edu> writes: > On 3/15/20, 8:39 AM, "lilypond-devel on behalf of Han-Wen Nienhuys" > <lilypond-devel-bounces+c_sorensen=byu.edu@gnu.org on behalf of > hanwenn@gmail.com> wrote: > > On Sun, Mar 15, 2020 at 12:05 PM <jonas.hahnfeld@gmail.com> wrote: > > > > > > What I'm not really happy with is replacing all of this with custom > > Python scripts, however small they are. Skimming over scan-mf-deps.py, > > that looks totally doable in around half the lines with a sh script. > > What's the motivation to use Python for all of this? > > What's the motivation to use sh for all this? > > You can certainly make it shorter with shell, but will it be > significantly simpler, easier to understand or easier to maintain? > > I decided for Python in this case, because recursion seems like > something I don't want to do in a sh script. > > Personally, I'm totally in favor of using python for creating ninja > files. We have a well-documented, well-maintained, powerful language. > No different flavors (bash, sh, dash, csh, ksh...). Regarding the "no different flavors" mantra, you might want to speak with the people who were (and still are) involved with the Python2 to Python3 porting effort. -- David Kastrup
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 5:04 PM <jonas.hahnfeld@gmail.com> wrote: > > On 2020/03/15 14:39:06, hanwenn wrote: > > On Sun, Mar 15, 2020 at 12:05 PM <mailto:jonas.hahnfeld@gmail.com> > wrote: > > > On 2020/03/14 23:57:48, hanwenn wrote: > > > > I have been experimenting with Ninja (https://ninja-build.org/), > but > > > Ninja has a > > > > very restricted model. Make functions, wildcards, etc. are not > > > possible. In > > > > order to advance my experiments, I'm trying to clean up the build > so I > > > can port > > > > more of the build over. > > > > > > From the website: "it [Ninja] is designed to have its input files > > > generated by a higher-level build system". And that's to be taken > > > literally, the syntax is so low-level that it's no fun to read it, > let > > > alone write. So big NACK from me for switching to Ninja. > > > > Have you looked at the code I wrote? I know about Ninja, and in fact > > I personally know its creator. > > > > Using Ninja means generating a Ninja file. I looked for a bit at CMake > > and Meson, but I'm not convinced they'll be a win; for one I'd have to > > learn yet another poorly conceived and limited language. The LilyPond > > build is highly custom (metafont, lilypond-book, texi2html), so it's > > also not obvious that either they'll bring much to the table. > > > > So I'm simply writing the build.ninja file from a Python script > > directly, without any attempt to make it generic. We'll see how far > > that brings me. > > I did so now and ... does this mean you want to write yet another meta > build system? That sounds horrible to maintain. It doesn't have to be a *meta* build system, because it doesn't have to cater to anything but LilyPond. That makes me confident that it will be much simpler than any of the existing systems. > > > What's the motivation to use Python for all of this? > > > > What's the motivation to use sh for all this? > > Let me explain with my background of porting to Python 3: It was a > nightmare which leads me to the conclusion that Python scripts are not > easy to maintain over an extended period of time. I recently read > https://gregoryszorc.com/blog/2020/01/13/mercurial's-journey-to-and-reflectio... > and this very much confirms this point. > sh instead is pretty much the same since decades and *very* portable. I very much agree with you, but out of the existing language within our tree (Perl, Python, Shell, C++, Make), it is the least worst option. Make is too limited (see the mess that our makefiles are today), Shell has all the problems of Make. We can't use C++ in the build system, and Perl is a godawful mess of line-noise. What do you suggest instead? For all the code where I have a choice of language, I nowadays use Go, and I'd be happy to do a (meta)build system in Go. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <dak@gnu.org> wrote: > Makefiles are an established technique well-known to most contributors. > Recursive python scripts for writing human-unreadable controlling files What recursive script are you talking about ? > in some build system that is the current fad of the year incompatible > with anything else Are you talking about Ninja? You realize that it is the standard infrastructure of virtually all modern build systems, including Meson/CMake. The chrome browser is built with Ninja, as is the Android platform, and the Clang compiler. Ninja was introduced in 2012, so it's not exactly the fad of the year. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
On 2020/03/15 16:16:06, hanwenn wrote: > On Sun, Mar 15, 2020 at 5:04 PM <mailto:jonas.hahnfeld@gmail.com> wrote: > > I did so now and ... does this mean you want to write yet another meta > > build system? That sounds horrible to maintain. > > It doesn't have to be a *meta* build system, because it doesn't have > to cater to anything but LilyPond. That makes me confident that it > will be much simpler than any of the existing systems. It's a (build) system that generates files for a build system. I'd call this "meta" but this is likely a matter of definition. > I very much agree with you, but out of the existing language within > our tree (Perl, Python, Shell, C++, Make), it is the least worst > option. Make is too limited (see the mess that our makefiles are > today), Shell has all the problems of Make. We can't use C++ in the > build system, and Perl is a godawful mess of line-noise. > > What do you suggest instead? Not doing this at all. In cases like this, I begin to question the very fundamental assumptions. In this case: Do we really need to dynamically generate the dependencies if there's really no tool for it? My answer is "no" after seeing that the last change of 'include' statements was in 2013 (if you count a2f44bbf0e; otherwise maybe earlier). As such, how about just making them static? See https://codereview.appspot.com/555440062 to get the idea.
Sign in to reply to this message.
On 2020/03/15 18:11:44, hanwenn wrote: > On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <mailto:dak@gnu.org> wrote: > > Makefiles are an established technique well-known to most contributors. > > Recursive python scripts for writing human-unreadable controlling files > > What recursive script are you talking about ? > > > in some build system that is the current fad of the year incompatible > > with anything else > > Are you talking about Ninja? You realize that it is the standard > infrastructure of virtually all modern build systems, including > Meson/CMake. The chrome browser is built with Ninja, as is the Android > platform, and the Clang compiler. Begin a LLVM committer, let me disagree with this one: LLVM/Clang uses CMake which is *able* to generate Ninja. Or Makefiles. Or files for VisualStudio. (Did I forget one "generator" target?)
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 7:14 PM <jonas.hahnfeld@gmail.com> wrote: > > On Sun, Mar 15, 2020 at 5:04 PM <mailto:jonas.hahnfeld@gmail.com> > wrote: > > > I did so now and ... does this mean you want to write yet another > meta > > > build system? That sounds horrible to maintain. > > > > It doesn't have to be a *meta* build system, because it doesn't have > > to cater to anything but LilyPond. That makes me confident that it > > will be much simpler than any of the existing systems. > > It's a (build) system that generates files for a build system. I'd call > this "meta" but this is likely a matter of definition. Python + Ninja seems like its a system with less surprises, and therefore, better maintainability than autoconf + make. > > I very much agree with you, but out of the existing language within > > our tree (Perl, Python, Shell, C++, Make), it is the least worst > > option. Make is too limited (see the mess that our makefiles are > > today), Shell has all the problems of Make. We can't use C++ in the > > build system, and Perl is a godawful mess of line-noise. > > > > What do you suggest instead? > > Not doing this at all. > In cases like this, I begin to question the very fundamental > assumptions. In this case: Do we really need to dynamically generate the > dependencies if there's really no tool for it? My answer is "no" after > seeing that the last change of 'include' statements was in 2013 (if you > count a2f44bbf0e; otherwise maybe earlier). As such, how about just > making them static? See https://codereview.appspot.com/555440062 to get > the idea. FWIW, at some point the dependency generation was part of mf-to-table.py, which generated from the log file (using Python). I don't know why it was moved to be a make macro. If we think we're not going to change the mf files, why not just check in the OTF files? (tongue in cheek, but you get the idea). In general leaving functionality intact leads to less discussion, which lets me move faster. In this case in particular, though, I think that as long as we're building the fonts from mf using make, we should generate correct dependencies. The build system is kind of a mess, but I think hard-coding hardcoding dependencies is strictly worse. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 7:14 PM <jonas.hahnfeld@gmail.com> wrote: > > I very much agree with you, but out of the existing language within > > our tree (Perl, Python, Shell, C++, Make), it is the least worst > > option. Make is too limited (see the mess that our makefiles are > > today), Shell has all the problems of Make. We can't use C++ in the > > build system, and Perl is a godawful mess of line-noise. > > > > What do you suggest instead? > > Not doing this at all. > In cases like this, I begin to question the very fundamental > assumptions. In this case: Do we really need to dynamically generate the You didn't answer my question, but I realize it was imperfectly stated. My question is: what overall architecture do you propose for the build system? -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
Han-Wen Nienhuys <hanwenn@gmail.com> writes: > On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <dak@gnu.org> wrote: >> Makefiles are an established technique well-known to most contributors. >> Recursive python scripts for writing human-unreadable controlling files > > What recursive script are you talking about ? It is you who stated the following: > I decided for Python in this case, because recursion seems like > something I don't want to do in a sh script. So apparently you want to use Python for recursive scripting, or why bring it up? >> in some build system that is the current fad of the year incompatible >> with anything else > > Are you talking about Ninja? You realize that it is the standard > infrastructure of virtually all modern build systems, including > Meson/CMake. It is one backend of several for those, and the more common CMake predates Ninja. > The chrome browser is built with Ninja, as is the Android platform, The Ninja website itself states "parts of Android". It also states "Where other build systems are high-level languages Ninja aims to be an assembler." and "it is designed to have its input files generated by a higher-level build system,". So your Python scripts are intended to eventually constitute another higher-level build system instead of reusing an existing one, with Ninja operating at the assembler level? This just screams like being slated to become a very person-bound dependency in the form of an idiosyncratic build system designed from scratch in essential parts, and eventually a liability. The problems with Make you stated you wanted to address with that are more problems with the _current_ Makefile architecture and dependency setup rather than inherent with Make. While a complete switch of tools has the advantage of making it easier to abandon parts of the design that are equal parts legacy and liability, the principal problems of incomplete dependencies particularly cross-directory, incomplete job control, and somewhat incrutable setup of what rules belong where in the stepmake system are not an inherent problem of Make but of what we have built around it. -- David Kastrup
Sign in to reply to this message.
On 2020/03/15 18:52:27, hanwenn wrote: > > Not doing this at all. > > In cases like this, I begin to question the very fundamental > > assumptions. In this case: Do we really need to dynamically generate the > > dependencies if there's really no tool for it? My answer is "no" after > > seeing that the last change of 'include' statements was in 2013 (if you > > count a2f44bbf0e; otherwise maybe earlier). As such, how about just > > making them static? See https://codereview.appspot.com/555440062 to get > > the idea. > > FWIW, at some point the dependency generation was part of > mf-to-table.py, which generated from the log file (using Python). I > don't know why it was moved to be a make macro. > > If we think we're not going to change the mf files, why not just check > in the OTF files? > > (tongue in cheek, but you get the idea). IMHO you need to differentiate between changes to mf files (which I think we should enable) and changes to the *structure* of the mf files. The latter gets slightly more complicated with static dependencies. But I don't expect many of them, much to the contrary of '#include's in C++. _If_ dependencies become a problem in the future _and_ we're still using make that point, we can of course revisit this issues. On 2020/03/15 18:54:44, hanwenn wrote: > On Sun, Mar 15, 2020 at 7:14 PM <mailto:jonas.hahnfeld@gmail.com> wrote: > > > I very much agree with you, but out of the existing language within > > > our tree (Perl, Python, Shell, C++, Make), it is the least worst > > > option. Make is too limited (see the mess that our makefiles are > > > today), Shell has all the problems of Make. We can't use C++ in the > > > build system, and Perl is a godawful mess of line-noise. > > > > > > What do you suggest instead? > > > > Not doing this at all. > > In cases like this, I begin to question the very fundamental > > assumptions. In this case: Do we really need to dynamically generate the > > You didn't answer my question, but I realize it was imperfectly stated. > > My question is: what overall architecture do you propose for the build system? Using whatever makes sense from the GNU build system and reduce the maintenance burden of the build system as much as possible. FWIW I fully agree with David that the current shortcomings originate from what LilyPond builds around Autoconf, GNU make et al. Using standard solutions (aclocal, possibly Automake for dependencies of C++ files) should enable us to reuse what others have already solved.
Sign in to reply to this message.
analyze log file
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 8:44 PM <jonas.hahnfeld@gmail.com> wrote: > On 2020/03/15 18:54:44, hanwenn wrote: > > On Sun, Mar 15, 2020 at 7:14 PM <mailto:jonas.hahnfeld@gmail.com> > wrote: > > > > I very much agree with you, but out of the existing language > within > > > > our tree (Perl, Python, Shell, C++, Make), it is the least worst > > > > option. Make is too limited (see the mess that our makefiles are > > > > today), Shell has all the problems of Make. We can't use C++ in > the > > > > build system, and Perl is a godawful mess of line-noise. > > > > > > > > What do you suggest instead? > > > > > > Not doing this at all. > > > In cases like this, I begin to question the very fundamental > > > assumptions. In this case: Do we really need to dynamically generate > the > > > > You didn't answer my question, but I realize it was imperfectly > stated. > > > > My question is: what overall architecture do you propose for the build > system? > > Using whatever makes sense from the GNU build system and reduce the > maintenance burden of the build system as much as possible. FWIW I fully > agree with David that the current shortcomings originate from what > LilyPond builds around Autoconf, GNU make et al. Using standard > solutions (aclocal, possibly Automake for dependencies of C++ files) > should enable us to reuse what others have already solved. I don't disagree with the analysis of the root cause, but getting rid of the recursive make for a non-recursive one means a complete rewrite, and not one that can be done incrementally. When you do a complete rewrite, you might as well reconsider the fundamental assumptions, one of which is "use the GNU build stack". I have plenty of experience with Automake, GNU make, building lilypond, and build systems in general. My conclusion of this experience is that it is a pretty poor stack, and for that reason I am experimenting with something that is completely different. I have simplfied the change under review. I'm sure there is a sed incantation that could achieve the same, but I don't want to bother learning it. If you can suggest one, I'd be happy to use it. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
On Sun, Mar 15, 2020 at 8:32 PM David Kastrup <dak@gnu.org> wrote: > > I decided for Python in this case, because recursion seems like > > something I don't want to do in a sh script. > > So apparently you want to use Python for recursive scripting, or why > bring it up? I'm talking about a recursive function. I'm not sure what you mean by recursive scripting. Note that the current Make file already has recursion, # Recursively scan the metafont .mf file $(1) for "input X;" # and return all dependencies. scan-mf = \ $(foreach f, $(shell test -f $(1) && sed -ne "/^[[:space:]]*input[[:space:]]/s/^[[:space:]]*input\([^.;]*\)\(.mf;\|;\)/\1.mf/p" $(1)), \ $(call find-mf,$(f)) \ $(call scan-mf,$(call find-mf,$(f))) \ ) -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
realpath
Sign in to reply to this message.
realpath 2
Sign in to reply to this message.
/bin/sh
Sign in to reply to this message.
On 2020/03/14 20:26:45, lemzwerg wrote: > this looks like a good idea, thanks! > > https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh > File mf/invoke-mf2pt.sh (right): > > https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh#ne... > mf/invoke-mf2pt.sh:4: mf2pt1=$(realpath $1) > I only know `realpath` as a GNU make command. Ditto for other commands below in > this script. > > Is this script work in progress? I added a shell function.
Sign in to reply to this message.
LGTM https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:18: # which no longer dump a .mem file We could probably get rid of the .mem file support, which is obsolete since about 10 years. https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:31: --fullname=$name \ spaces vs. tabs?
Sign in to reply to this message.
On 2020/03/15 21:46:38, hanwenn wrote: > On Sun, Mar 15, 2020 at 8:44 PM <mailto:jonas.hahnfeld@gmail.com> wrote: > > Using whatever makes sense from the GNU build system and reduce the > > maintenance burden of the build system as much as possible. FWIW I fully > > agree with David that the current shortcomings originate from what > > LilyPond builds around Autoconf, GNU make et al. Using standard > > solutions (aclocal, possibly Automake for dependencies of C++ files) > > should enable us to reuse what others have already solved. > > I don't disagree with the analysis of the root cause, but getting rid > of the recursive make for a non-recursive one means a complete > rewrite, and not one that can be done incrementally. When you do a > complete rewrite, you might as well reconsider the fundamental > assumptions, one of which is "use the GNU build stack". > > I have plenty of experience with Automake, GNU make, building > lilypond, and build systems in general. My conclusion of this > experience is that it is a pretty poor stack, ... which has led to the invention of Stepmake in the past ... > and for that reason I am > experimenting with something that is completely different. So we start another cycle of writing a custom build system. Fabulous! </sarcasm>
Sign in to reply to this message.
On Mon, Mar 16, 2020 at 8:29 AM <jonas.hahnfeld@gmail.com> wrote: > > On 2020/03/15 21:46:38, hanwenn wrote: > > On Sun, Mar 15, 2020 at 8:44 PM <mailto:jonas.hahnfeld@gmail.com> > wrote: > > > Using whatever makes sense from the GNU build system and reduce the > > > maintenance burden of the build system as much as possible. FWIW I > fully > > > agree with David that the current shortcomings originate from what > > > LilyPond builds around Autoconf, GNU make et al. Using standard > > > solutions (aclocal, possibly Automake for dependencies of C++ files) > > > should enable us to reuse what others have already solved. > > > > I don't disagree with the analysis of the root cause, but getting rid > > of the recursive make for a non-recursive one means a complete > > rewrite, and not one that can be done incrementally. When you do a > > complete rewrite, you might as well reconsider the fundamental > > assumptions, one of which is "use the GNU build stack". > > > > I have plenty of experience with Automake, GNU make, building > > lilypond, and build systems in general. My conclusion of this > > experience is that it is a pretty poor stack, > > ... which has led to the invention of Stepmake in the past ... > > > and for that reason I am > > experimenting with something that is completely different. > > So we start another cycle of writing a custom build system. Fabulous! > </sarcasm> I am not asking you for a review of a new build system. I am asking for the review of this change. I think we can do without the sarcasm about my capabilities, given that you are working on a project that I largely wrote, and has survived for 22 years. -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
Han-Wen Nienhuys <hanwenn@gmail.com> writes: > I am not asking you for a review of a new build system. I am asking > for the review of this change. > > I think we can do without the sarcasm about my capabilities, given > that you are working on a project that I largely wrote, and has > survived for 22 years. With all due respect, I think characterizing the phases of the project without your input as mere survival, even if you consider yourself the sole relevant author for those of the times you were active on it, is not exactly being a fair characterization of the work of dozens of contributors. -- David Kastrup
Sign in to reply to this message.
oops - restored overly zealous deletions
Sign in to reply to this message.
local-clean
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:18: # which no longer dump a .mem file On 2020/03/16 06:16:22, lemzwerg wrote: > We could probably get rid of the .mem file support, which is obsolete since > about 10 years. sure. Followup change? https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:31: --fullname=$name \ On 2020/03/16 06:16:22, lemzwerg wrote: > spaces vs. tabs? no idea. Whatever?
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:18: # which no longer dump a .mem file > > We could probably get rid of the .mem file support, > > which is obsolete since about 10 years. > > sure. Followup change? ok – no hurry :-) https://codereview.appspot.com/553700043/diff/555460043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:31: --fullname=$name \ > > spaces vs. tabs? > > no idea. Whatever? Bad formulation, sorry. I meant that there might be a tabs-vs-spaces issue, and that tabs should be replaced with spaces (and/or the indentation should be fixed).
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode36 mf/GNUmakefile:36: $(src-dir)/invoke-mf2pt.sh $(buildscript-dir)/mf2pt1 $< $@ $(METAFONT_QUIET) Should this script be called invoke-mf2pt1.sh? The package is also called mf2pt1 https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode39 mf/GNUmakefile:39: $(UPDATE_TARGET) Can we list .tfm and .log files directly as output of above rule that produces .pfb? As far as I understand, mf2pt1 produces all three of them, right? https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:4: realpath() { AFAIK sh syntax would be realpath() ( ... ) Curly braces are not part of POSIX https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py File mf/scan-mf-deps.py (right): https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:1: #!/usr/bin/python This should be #!@PYTHON@ so that setting PYTHON in configure is taken into account. https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:10: re.sub("\\(%s/?([^\n )]*\\.mf)" % srcdir, lambda m: deps.append(m.group(1)), input) Now that should be very possible with sed in .sh plus you already have all the information available and don't need to parse it from sys.argv
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh File mf/invoke-mf2pt.sh (right): https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh#ne... mf/invoke-mf2pt.sh:4: realpath() { On 2020/03/17 07:41:25, hahnjo wrote: > AFAIK sh syntax would be > realpath() > ( > ... > ) > > Curly braces are not part of POSIX Are you sure? I have never seen that. What isn't POSIX is bash's "function" keyword, but Han-Wen does not use it here.
Sign in to reply to this message.
On 2020/03/17 07:54:13, dak wrote: > https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh > File mf/invoke-mf2pt.sh (right): > > https://codereview.appspot.com/553700043/diff/567370043/mf/invoke-mf2pt.sh#ne... > mf/invoke-mf2pt.sh:4: realpath() { > On 2020/03/17 07:41:25, hahnjo wrote: > > AFAIK sh syntax would be > > realpath() > > ( > > ... > > ) > > > > Curly braces are not part of POSIX > Are you sure? I have never seen that. What isn't POSIX is bash's "function" > keyword, but Han-Wen does not use it here. Ah pardon me: { } are part of POSIX and create a compound statement. I confused this with the portable way of scoping variables, which is done via a subshell enclosed by ( )
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py File mf/scan-mf-deps.py (right): https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:12: target = target.replace(srcdir, "").lstrip("/") This doesn't work for out-of-tree builds where srcdir != builddir. target is naturally below builddir and srcdir is not found, so it only lstrip's / which leads to an absolute path without the leading slash.
Sign in to reply to this message.
-recorder
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode36 mf/GNUmakefile:36: $(src-dir)/invoke-mf2pt.sh $(buildscript-dir)/mf2pt1 $< $@ $(METAFONT_QUIET) On 2020/03/17 07:41:25, hahnjo wrote: > Should this script be called invoke-mf2pt1.sh? The package is also called mf2pt1 Done. https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode39 mf/GNUmakefile:39: $(UPDATE_TARGET) On 2020/03/17 07:41:25, hahnjo wrote: > Can we list .tfm and .log files directly as output of above rule that produces > .pfb? As far as I understand, mf2pt1 produces all three of them, right? this is tempting, but t1 t2 : src recipe is short for t1 : src recipe t2 : src recipe this means that the value of $@ depends on which thing gets built first. There is also a related &: operator, but it also doesn't work for the same reason. https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py File mf/scan-mf-deps.py (right): https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:1: #!/usr/bin/python On 2020/03/17 07:41:25, hahnjo wrote: > This should be #!@PYTHON@ so that setting PYTHON in configure is taken into > account. Acknowledged. https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:10: re.sub("\\(%s/?([^\n )]*\\.mf)" % srcdir, lambda m: deps.append(m.group(1)), input) On 2020/03/17 07:41:25, hahnjo wrote: > Now that should be very possible with sed in .sh plus you already have all the > information available and don't need to parse it from sys.argv Acknowledged. https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... mf/scan-mf-deps.py:12: target = target.replace(srcdir, "").lstrip("/") On 2020/03/17 08:08:37, hahnjo wrote: > This doesn't work for out-of-tree builds where srcdir != builddir. target is > naturally below builddir and srcdir is not found, so it only lstrip's / which > leads to an absolute path without the leading slash. I have a brilliant alternative. See next patchset.
Sign in to reply to this message.
On 2020/03/17 07:41:26, hahnjo wrote: > https://codereview.appspot.com/553700043/diff/567370043/mf/scan-mf-deps.py#ne... > mf/scan-mf-deps.py:10: re.sub("\\(%s/?([^\n )]*\\.mf)" % srcdir, lambda m: > deps.append(m.group(1)), input) > Now that should be very possible with sed in .sh plus you already have all the > information available and don't need to parse it from sys.argv Admittedly two sed's: diff --git a/mf/invoke-mf2pt.sh b/mf/invoke-mf2pt.sh index ea2afe771f..aa5740240a 100755 --- a/mf/invoke-mf2pt.sh +++ b/mf/invoke-mf2pt.sh @@ -8,7 +8,8 @@ realpath() { set -eu mf2pt1=$(realpath $1) src=$(realpath $2) -target=$(realpath $3) +target=$3 +target_path=$(realpath $target) srcdir=$(dirname ${src}) name=$(basename ${src} .mf) @@ -16,7 +17,7 @@ name=$(basename ${src} .mf) # # the soft link for mf2pt1.mp is for recent mpost versions # which no longer dump a .mem file -tmp=$(dirname $target)/tmp.$(basename $target) +tmp=$(dirname $target_path)/tmp.$(basename $target) rm -rf $tmp mkdir $tmp cd $tmp @@ -31,7 +32,10 @@ ${mf2pt1} --rounding=0.0001 \ --fullname=$name \ --name=$name $src -python ${srcdir}/scan-mf-deps.py ${srcdir} ${target} ${name}.log > ${name}.dep +# First put every possible inclusion starting with ( into a separate line. Then +# print only the included .mf files (sed -n) and munge the rest of that line. +dependencies=$(cat ${name}.log | sed "s|(|\n(|g" | sed -n "s|(\\$srcdir/*\([^ )]*\.mf\).*| \1 \\\\|pg") +echo "$target:$dependencies" > ${name}.dep mv *.pfb *.tfm *.log *.dep .. cd ..
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/547760043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/547760043/mf/GNUmakefile#newcode37 mf/GNUmakefile:37: $(src-dir)/invoke-mf2pt.sh $(buildscript-dir)/mf2pt1 $< $@ $(METAFONT_QUIET) Needs updating! https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:35: grep '^INPUT.*mf$' ${name}.fls | sed "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >> ${name}.dep grep: feta11.fls: No such file or directory
Sign in to reply to this message.
rename.
Sign in to reply to this message.
rename (2nd try)
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/547760043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/547760043/mf/GNUmakefile#newcode37 mf/GNUmakefile:37: $(src-dir)/invoke-mf2pt.sh $(buildscript-dir)/mf2pt1 $< $@ $(METAFONT_QUIET) On 2020/03/17 09:29:28, hahnjo wrote: > Needs updating! Done. https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:35: grep '^INPUT.*mf$' ${name}.fls | sed "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >> ${name}.dep On 2020/03/17 09:29:28, hahnjo wrote: > grep: feta11.fls: No such file or directory did you rebuild the scripts/build/ before ? Does "mf -help" list a -recorder flag? (the feature was introduced in 2001, so it should)
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/547770043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/547770043/mf/GNUmakefile#newcode40 mf/GNUmakefile:40: $(UPDATE_TARGET) I see spurious rebuilds if I re-run make and I think this is because the touch in UPDATE_TARGET is apparently only executed once. https://codereview.appspot.com/553700043/diff/547770043/mf/GNUmakefile#newcode67 mf/GNUmakefile:67: EXTRA_DIST_FILES += README mf2pt1.mp invoke-mf2pt.sh r 'r'? https://codereview.appspot.com/553700043/diff/547770043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/547770043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:32: --name=$name $src Still mixes tabs and spaces https://codereview.appspot.com/553700043/diff/547770043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:34: echo -n "$target : " > ${name}.dep This doesn't work for me because $target is an absolute path. On top of an older version I had the following: diff --git a/mf/invoke-mf2pt.sh b/mf/invoke-mf2pt.sh index ea2afe771f..aa5740240a 100755 --- a/mf/invoke-mf2pt.sh +++ b/mf/invoke-mf2pt.sh @@ -8,7 +8,8 @@ realpath() { set -eu mf2pt1=$(realpath $1) src=$(realpath $2) -target=$(realpath $3) +target=$3 +target_path=$(realpath $target) srcdir=$(dirname ${src}) name=$(basename ${src} .mf) @@ -16,7 +17,7 @@ name=$(basename ${src} .mf) # # the soft link for mf2pt1.mp is for recent mpost versions # which no longer dump a .mem file -tmp=$(dirname $target)/tmp.$(basename $target) +tmp=$(dirname $target_path)/tmp.$(basename $target) rm -rf $tmp mkdir $tmp cd $tmp
Sign in to reply to this message.
On 2020/03/17 20:10:51, hanwenn wrote: > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh > File mf/invoke-mf2pt1.sh (right): > > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh#n... > mf/invoke-mf2pt1.sh:35: grep '^INPUT.*mf$' ${name}.fls | sed > "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >> ${name}.dep > On 2020/03/17 09:29:28, hahnjo wrote: > > grep: feta11.fls: No such file or directory > > did you rebuild the scripts/build/ before ? Yes, but I must confess that I called "git apply" while in mf/, so the update to scripts/build didn't get applied - my bad! Works now, sorry for the noise. On 2020/03/17 09:13:02, hanwenn wrote: > https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode39 > mf/GNUmakefile:39: $(UPDATE_TARGET) > On 2020/03/17 07:41:25, hahnjo wrote: > > Can we list .tfm and .log files directly as output of above rule that produces > > .pfb? As far as I understand, mf2pt1 produces all three of them, right? > > this is tempting, but > > t1 t2 : src > recipe > > is short for > > t1 : src > recipe > > t2 : src > recipe Not entirely, recipe is still only evaluated once. > this means that the value of $@ depends on which thing gets built first. There > is also a related &: operator, but it also doesn't work for the same reason. This answer suggests the use of $*: https://stackoverflow.com/a/3113546/10606944 So using $(outdir)/$*.pfb instead of $@ should always evaluate to the right value?
Sign in to reply to this message.
On 2020/03/17 20:47:22, hahnjo wrote: > On 2020/03/17 20:10:51, hanwenn wrote: > > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh > > File mf/invoke-mf2pt1.sh (right): > > > > > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh#n... > > mf/invoke-mf2pt1.sh:35: grep '^INPUT.*mf$' ${name}.fls | sed > > "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >> ${name}.dep > > On 2020/03/17 09:29:28, hahnjo wrote: > > > grep: feta11.fls: No such file or directory > > > > did you rebuild the scripts/build/ before ? > > Yes, but I must confess that I called "git apply" while in mf/, so the update to > scripts/build didn't get applied - my bad! Works now, sorry for the noise. > > On 2020/03/17 09:13:02, hanwenn wrote: > > > https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode39 > > mf/GNUmakefile:39: $(UPDATE_TARGET) > > On 2020/03/17 07:41:25, hahnjo wrote: > > > Can we list .tfm and .log files directly as output of above rule that > produces > > > .pfb? As far as I understand, mf2pt1 produces all three of them, right? > > > > this is tempting, but > > > > t1 t2 : src > > recipe > > > > is short for > > > > t1 : src > > recipe > > > > t2 : src > > recipe > > Not entirely, recipe is still only evaluated once. > > > this means that the value of $@ depends on which thing gets built first. There > > is also a related &: operator, but it also doesn't work for the same reason. > > This answer suggests the use of $*: https://stackoverflow.com/a/3113546/10606944 > So using $(outdir)/$*.pfb instead of $@ should always evaluate to the right > value? looks it does, but I can't get it to work. I get things like: [hanwen@localhost mf]$ make clean ; make -j1000 out/emmentaler-20.otf .. Making mf/out/feta-flags20.log < mf Making mf/out/feta-alphabet20.log < mf .. Done making mf/out/feta-flags20.log < mf Making mf/out/feta-flags20.lisp < log Done making mf/out/feta-alphabet20.log < mf Traceback (most recent call last): File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 246, in <module> (g, m, deps) = parse_logfile (filenm) File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 56, in parse_logfile autolines, deps = read_log_file (fn) File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 29, in read_log_file str = open (fn).read () FileNotFoundError: [Errno 2] No such file or directory: 'out/feta-flags20.log' make: *** [GNUmakefile:132: out/feta-flags20.lisp] Error 1 the $(UPDATE_TARGET) hack works if I compile with -j1000.
Sign in to reply to this message.
touch multiple files
Sign in to reply to this message.
On 2020/03/17 21:18:50, hanwenn wrote: > On 2020/03/17 20:47:22, hahnjo wrote: > > On 2020/03/17 20:10:51, hanwenn wrote: > > > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh > > > File mf/invoke-mf2pt1.sh (right): > > > > > > > > > https://codereview.appspot.com/553700043/diff/547760043/mf/invoke-mf2pt1.sh#n... > > > mf/invoke-mf2pt1.sh:35: grep '^INPUT.*mf$' ${name}.fls | sed > > > "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >> ${name}.dep > > > On 2020/03/17 09:29:28, hahnjo wrote: > > > > grep: feta11.fls: No such file or directory > > > > > > did you rebuild the scripts/build/ before ? > > > > Yes, but I must confess that I called "git apply" while in mf/, so the update > to > > scripts/build didn't get applied - my bad! Works now, sorry for the noise. > > > > On 2020/03/17 09:13:02, hanwenn wrote: > > > > > > https://codereview.appspot.com/553700043/diff/567370043/mf/GNUmakefile#newcode39 > > > mf/GNUmakefile:39: $(UPDATE_TARGET) > > > On 2020/03/17 07:41:25, hahnjo wrote: > > > > Can we list .tfm and .log files directly as output of above rule that > > produces > > > > .pfb? As far as I understand, mf2pt1 produces all three of them, right? > > > > > > this is tempting, but > > > > > > t1 t2 : src > > > recipe > > > > > > is short for > > > > > > t1 : src > > > recipe > > > > > > t2 : src > > > recipe > > > > Not entirely, recipe is still only evaluated once. > > > > > this means that the value of $@ depends on which thing gets built first. > There > > > is also a related &: operator, but it also doesn't work for the same reason. > > > > This answer suggests the use of $*: > https://stackoverflow.com/a/3113546/10606944 > > So using $(outdir)/$*.pfb instead of $@ should always evaluate to the right > > value? > > looks it does, but I can't get it to work. I get things like: > > [hanwen@localhost mf]$ make clean ; make -j1000 out/emmentaler-20.otf > .. > Making mf/out/feta-flags20.log < mf > Making mf/out/feta-alphabet20.log < mf > .. > Done making mf/out/feta-flags20.log < mf > Making mf/out/feta-flags20.lisp < log > Done making mf/out/feta-alphabet20.log < mf > Traceback (most recent call last): > File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 246, in > <module> > (g, m, deps) = parse_logfile (filenm) > File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 56, in > parse_logfile > autolines, deps = read_log_file (fn) > File "/home/hanwen/vc/lilypond/scripts/build/out/mf-to-table", line 29, in > read_log_file > str = open (fn).read () > FileNotFoundError: [Errno 2] No such file or directory: 'out/feta-flags20.log' > make: *** [GNUmakefile:132: out/feta-flags20.lisp] Error 1 diff --git a/mf/GNUmakefile b/mf/GNUmakefile index c6fec04631..fc42c0ace7 100644 --- a/mf/GNUmakefile +++ b/mf/GNUmakefile @@ -32,13 +32,9 @@ $(outdir)/%.dvi: %.mf mv $(basename $<).dvi $(outdir) rm $(basename $<).*gf -$(outdir)/%.pfb: %.mf $(outdir)/mf2pt1.mem invoke-mf2pt1.sh - $(call ly_progress,Making,$@,< mf) - $(src-dir)/invoke-mf2pt1.sh $(buildscript-dir)/mf2pt1 $< $@ $(METAFONT_QUIET) - -$(outdir)/%.tfm $(outdir)/%.log: $(outdir)/%.pfb - $(UPDATE_TARGET) - +$(outdir)/%.pfb $(outdir)/%.tfm $(outdir)/%.log: %.mf $(outdir)/mf2pt1.mem invoke-mf2pt1.sh + $(call ly_progress,Making,$(outdir)/$*.pfb,< mf) + $(src-dir)/invoke-mf2pt1.sh $(buildscript-dir)/mf2pt1 $< $(outdir)/$*.pfb $(METAFONT_QUIET) # since recent mpost versions no longer create a mem file, we create a dummy # file to satisfy the dependency (which gets overwritten in case an older works for me with 'make -C mf/ -j1000 out/emmentaler-20.otf', but maybe my laptop just has too little hardware parallelism...
Sign in to reply to this message.
The latest version fails for me, see my inline comment. The following still
works and I think it is correct to specify multiple outputs:
diff --git a/mf/GNUmakefile b/mf/GNUmakefile
index f21d1650d5..da5330e93d 100644
--- a/mf/GNUmakefile
+++ b/mf/GNUmakefile
@@ -32,11 +32,9 @@ $(outdir)/%.dvi: %.mf
mv $(basename $<).dvi $(outdir)
rm $(basename $<).*gf
-$(outdir)/%.pfb: %.mf $(outdir)/mf2pt1.mem invoke-mf2pt1.sh
- $(call ly_progress,Making,$@,< mf)
- $(src-dir)/invoke-mf2pt1.sh $(buildscript-dir)/mf2pt1 $< $@
$(METAFONT_QUIET)
-
-$(outdir)/%.tfm $(outdir)/%.log: $(outdir)/%.pfb
+$(outdir)/%.pfb $(outdir)/%.tfm $(outdir)/%.log: %.mf $(outdir)/mf2pt1.mem
invoke-mf2pt1.sh
+ $(call ly_progress,Making,$(outdir)/$*.pfb,< mf)
+ $(src-dir)/invoke-mf2pt1.sh $(buildscript-dir)/mf2pt1 $<
$(outdir)/$*.pfb $(METAFONT_QUIET)
# since recent mpost versions no longer create a mem file, we create a dummy
# file to satisfy the dependency (which gets overwritten in case an older
@@ -62,7 +60,7 @@ $(outdir)/emmentaler-%.genpe:
$(buildscript-dir)/gen-emmentaler-scripts
$< --dir=$(outdir) --design-size=$(patsubst
emmentaler-%.genpe,%,$(notdir $@))
-EXTRA_DIST_FILES += README mf2pt1.mp invoke-mf2pt.sh r
+EXTRA_DIST_FILES += README mf2pt1.mp invoke-mf2pt.sh
STAFF_SIZES = 11 13 14 16 18 20 23 26
BRACES = a b c d e f g h i
@@ -127,45 +125,36 @@ $(outdir)/emmentaler-brace.otf-table: $(foreach x, a b c d
e f g h i,$(outdir)/f
#
# 2. are not included with teTeX
#
-$(outdir)/%.lisp: $(outdir)/%.log $(outdir)/%.tfm
- $(call ly_progress,Making,$@,< log)
+$(outdir)/%.lisp $(outdir)/%.otf-gtable $(outdir)/%.enc: $(outdir)/%.log
$(outdir)/%.tfm
+ $(call ly_progress,Making,$(outdir)/$*.lisp,< log)
$(buildscript-dir)/mf-to-table \
--global-lisp=$(outdir)/$(<F:.log=.otf-gtable) \
--lisp=$(outdir)/$(<F:.log=.lisp) \
--outdir=$(outdir) \
--enc $(outdir)/$(<F:.log=.enc) \
$<
- touch $(outdir)/$(<F:.log=.otf-gtable) \
- $(outdir)/$(<F:.log=.lisp) \
- $(outdir)/$(<F:.log=.enc)
-
-$(outdir)/%.otf-gtable $(outdir)/%.enc: $(outdir)/%.lisp
-
-$(outdir)/emmentaler-%.otf: $(outdir)/emmentaler-%.genpe \
- $(outdir)/feta%.pfb \
- $(outdir)/feta-noteheads%.pfb \
- $(outdir)/feta-flags%.pfb \
- $(outdir)/feta-alphabet%.pfb \
- $(outdir)/parmesan%.pfb \
- $(outdir)/parmesan-noteheads%.pfb \
- $(outdir)/feta%.otf-table \
- $(outdir)/feta%.otf-gtable
- $(call ly_progress,Making,$@,)
- cd $(outdir) && $(FONTFORGE) -script $(notdir $<)
- touch $(outdir)/emmentaler-%.svg $(outdir)/emmentaler-%.woff
$(outdir)/emmentaler-%.otf
-$(outdir)/emmentaler-%.svg $(outdir)/emmentaler-%.woff:
$(outdir)/emmentaler-%.otf
+$(outdir)/emmentaler-%.otf $(outdir)/emmentaler-%.svg
$(outdir)/emmentaler-%.woff: \
+ $(outdir)/emmentaler-%.genpe \
+ $(outdir)/feta%.pfb \
+ $(outdir)/feta-noteheads%.pfb \
+ $(outdir)/feta-flags%.pfb \
+ $(outdir)/feta-alphabet%.pfb \
+ $(outdir)/parmesan%.pfb \
+ $(outdir)/parmesan-noteheads%.pfb \
+ $(outdir)/feta%.otf-table \
+ $(outdir)/feta%.otf-gtable
+ $(call ly_progress,Making,$(outdir)/emmentaler-$*.otf,)
+ cd $(outdir) && $(FONTFORGE) -script $(notdir $<)
-$(outdir)/emmentaler-brace.otf: $(outdir)/emmentaler-brace.pe\
- $(foreach s,$(BRACES),$(outdir)/feta-braces-$(s).pfb) \
- $(outdir)/emmentaler-brace.otf-table \
- $(outdir)/emmentaler-brace.otf-gtable \
- $(outdir)/emmentaler-brace.subfonts
+$(outdir)/emmentaler-brace.otf $(outdir)/emmentaler-brace.svg
$(outdir)/emmentaler-brace.woff: \
+ $(outdir)/emmentaler-brace.pe \
+ $(foreach s,$(BRACES),$(outdir)/feta-braces-$(s).pfb) \
+ $(outdir)/emmentaler-brace.otf-table \
+ $(outdir)/emmentaler-brace.otf-gtable \
+ $(outdir)/emmentaler-brace.subfonts
$(call ly_progress,Making,$@,)
cd $(outdir) && $(FONTFORGE) -script emmentaler-brace.pe
- touch $(outdir)/emmentaler-brace.svg $(outdir)/emmentaler-brace.woff
$(outdir)/emmentaler-brace.otf
-
-$(outdir)/emmentaler-brace.svg $(outdir)/emmentaler-brace.woff:
$(outdir)/emmentaler-brace.otf
default: tree-regen \
$(outdir)/fonts.conf
diff --git a/mf/invoke-mf2pt1.sh b/mf/invoke-mf2pt1.sh
index 58d71ae01c..f57f379048 100755
--- a/mf/invoke-mf2pt1.sh
+++ b/mf/invoke-mf2pt1.sh
@@ -34,9 +34,6 @@ ${mf2pt1} --rounding=0.0001 \
echo -n "$target : " > ${name}.dep
grep '^INPUT.*mf$' ${name}.fls | sed "s|INPUT||;s|${srcdir}/||" | tr -d '\n' >>
${name}.dep
-# Touch all to have the same timestamp.
-touch *.pfb *.tfm *.log *.dep
-
mv *.pfb *.tfm *.log *.dep ..
cd ..
rm -rf ${tmp}
https://codereview.appspot.com/553700043/diff/575830048/mf/GNUmakefile
File mf/GNUmakefile (right):
https://codereview.appspot.com/553700043/diff/575830048/mf/GNUmakefile#newcode39
mf/GNUmakefile:39: $(outdir)/%.tfm $(outdir)/%.log: $(outdir)/%.pfb
This doesn't work for me: After a 'make clean' / deleting the build directory
and re-running configure, I get the following:
make: *** No rule to make target 'out/emmentaler-11.otf', needed by
'tree-regen'. Stop.
I traced this back to the following not working for a clean out/ directory:
make: *** No rule to make target 'out/feta11.log'. Stop.
The reason is that the marked line only declares dependencies of the %.log
files, but contains no rule that make could execute.
Sign in to reply to this message.
Am Mittwoch, den 18.03.2020, 06:27 -0700 schrieb jonas.hahnfeld@gmail.com: > The latest version fails for me, see my inline comment. The following > still works and I think it is correct to specify multiple outputs: > > [...] Rietveld screwed my diff, attached correctly here. > https://codereview.appspot.com/553700043/diff/575830048/mf/GNUmakefile > > File mf/GNUmakefile (right): > > https://codereview.appspot.com/553700043/diff/575830048/mf/GNUmakefile#newcode39 > > mf/GNUmakefile:39: $(outdir)/%.tfm $(outdir)/%.log: $(outdir)/%.pfb > This doesn't work for me: After a 'make clean' / deleting the build > directory and re-running configure, I get the following: > make: *** No rule to make target 'out/emmentaler-11.otf', needed by > 'tree-regen'. Stop. > > I traced this back to the following not working for a clean out/ > directory: > make: *** No rule to make target 'out/feta11.log'. Stop. > > The reason is that the marked line only declares dependencies of the > %.log files, but contains no rule that make could execute. > > https://codereview.appspot.com/553700043/
Sign in to reply to this message.
jonas' suggestion
Sign in to reply to this message.
On Tue, Mar 17, 2020 at 10:31 PM <jonas.hahnfeld@gmail.com> wrote: > works for me with 'make -C mf/ -j1000 out/emmentaler-20.otf', but maybe > my laptop just has too little hardware parallelism... thanks, I see the problem now. > https://codereview.appspot.com/553700043/ -- Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
Sign in to reply to this message.
The latest version builds, but still has many things wrong that I've mentioned before plus some new. https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcode36 mf/GNUmakefile:36: $(call ly_progress,Making,$@,< mf) Should be $(outdir)/%.pfb instead of $@ https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcode63 mf/GNUmakefile:63: EXTRA_DIST_FILES += README mf2pt1.mp invoke-mf2pt.sh r Extraneous 'r' at the end https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:124: $(call ly_progress,Making,$@,< log) Should (probably) be $(outdir)/%.lisp instead of $@ https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:141: $(call ly_progress,Making,$@,) $(outdir)/emmentaler-$*.otf https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:149: $(call ly_progress,Making,$@,) $(outdir)/emmentaler-brace.otf https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:24: ln -s ../../mf2pt1.mp . This link is not correct for out-of-tree builds. However MetaPost still finds mf2pt1.mp correctly in $srcdir because MFINPUTS is set below. I think all of this linking can be removed. https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:34: echo -n "$target : " > ${name}.dep As I said before this doesn't work because $target is an absolute path. My suggestion is to assign target=$3 and introduce target_path=$(realpath $target) https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:38: touch *.pfb *.tfm *.log *.dep Should not be needed anymore
Sign in to reply to this message.
Final comments.
Sign in to reply to this message.
oops
Sign in to reply to this message.
The changes largely LGTM and I've verified that it finally works on my system. https://codereview.appspot.com/553700043/diff/577680043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/577680043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:19: # which no longer dump a .mem file You might want to delete this comment as we're not doing this anymore. https://codereview.appspot.com/553700043/diff/577680043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:31: --name=$name $src The last two lines are still indented with tabs, please cleanup.
Sign in to reply to this message.
Jonas' nits
Sign in to reply to this message.
https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile File mf/GNUmakefile (right): https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcode36 mf/GNUmakefile:36: $(call ly_progress,Making,$@,< mf) On 2020/03/19 08:29:59, hahnjo wrote: > Should be $(outdir)/%.pfb instead of $@ Done. https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcode63 mf/GNUmakefile:63: EXTRA_DIST_FILES += README mf2pt1.mp invoke-mf2pt.sh r On 2020/03/19 08:29:59, hahnjo wrote: > Extraneous 'r' at the end Done. https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:124: $(call ly_progress,Making,$@,< log) On 2020/03/19 08:29:59, hahnjo wrote: > Should (probably) be $(outdir)/%.lisp instead of $@ Done. https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:141: $(call ly_progress,Making,$@,) On 2020/03/19 08:29:59, hahnjo wrote: > $(outdir)/emmentaler-$*.otf Done. https://codereview.appspot.com/553700043/diff/545720043/mf/GNUmakefile#newcod... mf/GNUmakefile:149: $(call ly_progress,Making,$@,) On 2020/03/19 08:29:59, hahnjo wrote: > $(outdir)/emmentaler-brace.otf Done. https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh File mf/invoke-mf2pt1.sh (right): https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:24: ln -s ../../mf2pt1.mp . On 2020/03/19 08:29:59, hahnjo wrote: > This link is not correct for out-of-tree builds. However MetaPost still finds > mf2pt1.mp correctly in $srcdir because MFINPUTS is set below. I think all of > this linking can be removed. Done. https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:34: echo -n "$target : " > ${name}.dep On 2020/03/19 08:29:59, hahnjo wrote: > As I said before this doesn't work because $target is an absolute path. My > suggestion is to assign target=$3 and introduce target_path=$(realpath $target) Done. https://codereview.appspot.com/553700043/diff/545720043/mf/invoke-mf2pt1.sh#n... mf/invoke-mf2pt1.sh:38: touch *.pfb *.tfm *.log *.dep On 2020/03/19 08:29:59, hahnjo wrote: > Should not be needed anymore Done.
Sign in to reply to this message.
commit c2eb8beac769a1bf30de357781601322d70095a6 Author: Han-Wen Nienhuys <hanwen@lilypond.org> Date: Sun Mar 15 22:35:28 2020 +0100 mf/GNUmakefile: move logic to scripts This simplifies the build, and removes some GNU Make specific idioms. * The mf2pt invocation is intricate. Use a shell script instead * Metafont dependency scanning uses advanced GNU Make scripting. Use the Metafont option -recorder instead. * Take the .log and .tfm files from the MetaPost run. (We were compiling all fonts twice.) * Remove some unused variables and rules https://sourceforge.net/p/testlilyissues/issues/5844 http://codereview.appspot.com/553700043
Sign in to reply to this message.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
