Final redirection of texi output

I've opened a new issue to avoid confusion.  AFAICS this redirects all the output from texi2pdf, makeinfo and tex2html to logfiles.  I've used Julien and David's suggestion of getting rid of --batch and --quiet, and it turns of the < /dev/null isn't needed when texi2pdf is run like this.  make; make doc is good.  If I edit notation.tely to put a load of random @ \ in, make doc fails with this on the terminal:

extract_texi_filenames.py: Processing out-www/notation.texi
writing: /media/IntelSSD/lilypond/lilypond-git/build/./out-www/xref-maps/notation.xref-map
lilypond-book.py (GNU LilyPond) 2.15.30

Please check the logfile notation.texi2pdf.log for errors

make[2]: *** [out-www/notation.pdf] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/media/IntelSSD/lilypond/lilypond-git/build/Documentation'
make[1]: *** [WWW-1] Error 2
make[1]: Leaving directory `/media/IntelSSD/lilypond/lilypond-git/build'
make: *** [doc-stage-1] Error 2

The contents of the logfile are:

cat notation.texi2pdf.log
This is pdfTeX, Version 3.1415926-1.40.10 (TeX Live 2009/Debian)
entering extended mode
(./notation.texi (/home/phil/lilypond-git/tex/texinfo.tex
Loading texinfo [version 2009-08-14.15]: pdf, fonts, markup, glyphs,
page headings, tables, conditionals, indexing, sectioning, toc, environments,
defuns, macros, cross references, insertions,
(/usr/share/texmf-texlive/tex/generic/epsf/epsf.tex
This is `epsf.tex' v2.7.3 <23 July 2005>
) localization, formatting, and turning on texinfo input format.)
(./notation.aux) (/home/phil/lilypond-git/tex/txi-en.tex)
Runaway argument?
@@\\@\\\@@@@\ 
./notation.texi:17: Paragraph ended before @\ was complete.
<to be read again> 
                   @par 
l.17 
     
? 
./notation.texi:17: Emergency stop.
<to be read again> 
                   @par 
l.17 
     
./notation.texi:17:  ==> Fatal error occurred, no output PDF file produced!
Transcript written on notation.log.
/usr/bin/texi2dvi: pdfetex exited with bad status, quitting.

I'm hoping this is this part of GOP 9 complete.

[PhilEHolmes, 2012-02-11 17:15:17]

Latest GOP 9 make doc reduction - please review.

[dak, 2012-02-11 17:20:40]

PhilEHolmes@googlemail.com writes:

> Reviewers: dak, Graham Percival, Julien Rioux,
>
> Message:
> Latest GOP 9 make doc reduction - please review.
>
> Description:
> I've opened a new issue to avoid confusion.  AFAICS this redirects all
> the output from texi2pdf, makeinfo and tex2html to logfiles.  I've used
> Julien and David's suggestion of getting rid of --batch and --quiet, and
> it turns of the < /dev/null isn't needed when texi2pdf is run like this.
>  make; make doc is good.

You might be mistaken here.  make -j... calls some processes with closed
stdin, but it is not predictable which processes these are.  So before
you come to this conclusion, try it with a single-job make.

-- 
David Kastrup

[mail_philholmes.net, 2012-02-11 17:56:59]

----- Original Message ----- 
From: "David Kastrup" <dak@gnu.org>
To: <PhilEHolmes@googlemail.com>
Cc: <reply@codereview-hr.appspotmail.com>; <julien.rioux@gmail.com>; 
<lilypond-devel@gnu.org>
Sent: Saturday, February 11, 2012 5:20 PM
Subject: Re: Final redirection of texi output (issue 5650064)


> PhilEHolmes@googlemail.com writes:
>
>> Reviewers: dak, Graham Percival, Julien Rioux,
>>
>> Message:
>> Latest GOP 9 make doc reduction - please review.
>>
>> Description:
>> I've opened a new issue to avoid confusion.  AFAICS this redirects all
>> the output from texi2pdf, makeinfo and tex2html to logfiles.  I've used
>> Julien and David's suggestion of getting rid of --batch and --quiet, and
>> it turns of the < /dev/null isn't needed when texi2pdf is run like this.
>>  make; make doc is good.
>
> You might be mistaken here.  make -j... calls some processes with closed
> stdin, but it is not predictable which processes these are.  So before
> you come to this conclusion, try it with a single-job make.
>
> -- 
> David Kastrup


Great catch, thanks.  without -jX it froze, as you say, waiting keyboard 
input.  New patch set uploaded including /dev/null which does not have this 
problem.

--
Phil Holmes

[Graham Percival, 2012-02-13 10:53:43]

LGTM

http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.sh
File scripts/build/run-and-check.sh (right):

http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.s...
scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
I know this has already been discussed, but I'm sorry to see no comments for
this line.  I have no clue what it's doing, so if any problem arose in this part
of the build system, I'd have to start reading some unix shell script tutorial. 
That might cause maintainability problems down the road.

Could we have a brief hint?

[dak, 2012-02-13 11:05:56]

http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.sh
File scripts/build/run-and-check.sh (right):

http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.s...
scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
On 2012/02/13 10:53:44, Graham Percival wrote:
> I know this has already been discussed, but I'm sorry to see no comments for
> this line.  I have no clue what it's doing, so if any problem arose in this
part
> of the build system, I'd have to start reading some unix shell script
tutorial. 
> That might cause maintainability problems down the road.
> 
> Could we have a brief hint?

Actually, it is totally basic shell scripting here.

Personally, instead of
run-and-check.sh 'some quoted shell command' logfile

I'd tend to write

some quoted shell command >logfile 2>&1 || { echo "Check" logfile "for errors"
>&2 ; exit 1; }

each time round.  I don't see that the extra indirection is really helpful. 
Possibly a Makefile macro can be used to write the above shorter.

[mail_philholmes.net, 2012-02-13 11:17:44]

----- Original Message ----- 
From: <graham@percival-music.ca>
To: <PhilEHolmes@googlemail.com>; <dak@gnu.org>; <julien.rioux@gmail.com>; 
<mail@philholmes.net>
Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
Sent: Monday, February 13, 2012 10:53 AM
Subject: Re: Final redirection of texi output (issue 5650064)


> LGTM
>
>
> http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.sh
> File scripts/build/run-and-check.sh (right):
>
>
http://codereview.appspot.com/5650064/diff/4001/scripts/build/run-and-check.s...
> scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
> I know this has already been discussed, but I'm sorry to see no comments
> for this line.  I have no clue what it's doing, so if any problem arose
> in this part of the build system, I'd have to start reading some unix
> shell script tutorial.  That might cause maintainability problems down
> the road.
>
> Could we have a brief hint?
>
> http://codereview.appspot.com/5650064/
>

Actually, this specific line hasn't been discussed before it's a much 
simplified version of the previous version which was littered with comments. 
I'll add a comment as requested.

@David: think I'd prefer to stick with the shell script now, having got this 
far down the road.

--
Phil Holmes

[PhilEHolmes, 2012-02-13 11:26:52]

Comments added describing eval line.

[Graham Percival, 2012-02-13 11:41:31]

On Mon, Feb 13, 2012 at 11:05:56AM +0000, dak@gnu.org wrote:
> 
> scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
> 
> Actually, it is totally basic shell scripting here.

Well, apparently you don't need "basic" shell scripting for 7
years of full-time linux use and 4 years of macosx use, and
thousands of hours of open-source work for multiple projects.
I've called programs, I've used for loops, but anything fancier
than that is done as a makefile or python.

My rule of thumb is that if I don't understand something easily,
then at least 50% of future lilypond contributors won't understand
it either.

- Graham

[Julien Rioux, 2012-02-13 13:56:09]

Looks good, I just have a few suggestions to avoid clashing log file names and I
wonder how this part of GOP 9 might be implemented down the road:
"All other logfiles will go in the @file{build/logfiles/} directory."

http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile
File Documentation/GNUmakefile (right):

http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile#new...
Documentation/GNUmakefile:234: $(buildscript-dir)/run-and-check "$(MAKEINFO)
-I$(src-dir) -I$(outdir) --output=$(dir $@) --docbook $<"  "$*.makeinfo.log"
We might need .makeinfoxml.log here, or something similar, to distinguish the
log files.

http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile#new...
Documentation/GNUmakefile:238: $(buildscript-dir)/run-and-check "$(MAKEINFO)
-I$(src-dir) -I$(outdir) --output=$(dir $@) --docbook $<"  "$*.makeinfo.log"
again, .makeinfoxml.log

http://codereview.appspot.com/5650064/diff/5003/make/doc-i18n-root-rules.make
File make/doc-i18n-root-rules.make (right):

http://codereview.appspot.com/5650064/diff/5003/make/doc-i18n-root-rules.make...
make/doc-i18n-root-rules.make:10: $(buildscript-dir)/run-and-check
"DEPTH=$(depth)/../ $(TEXI2HTML) $(TEXI2HTML_SPLIT) $(TEXI2HTML_FLAGS)
--output=$(outdir)/$* $<" "$*.splittexi.log"
I suggest adding $(ISOLANG) to the name of the log files generated from this
Makefile, eg:
$*.$(ISOLANG).splittexi.log

http://codereview.appspot.com/5650064/diff/5003/stepmake/stepmake/texinfo-rul...
File stepmake/stepmake/texinfo-rules.make (right):

http://codereview.appspot.com/5650064/diff/5003/stepmake/stepmake/texinfo-rul...
stepmake/stepmake/texinfo-rules.make:75: $(buildscript-dir)/run-and-check
"$(MAKEINFO) -I$(src-dir) -I$(outdir) --no-split --no-headers --output $@ $<" 
"$*.makeinfo.log"
.makeinfotxt.log

[dak, 2012-02-13 15:30:49]

Graham Percival <graham@percival-music.ca> writes:

> On Mon, Feb 13, 2012 at 11:05:56AM +0000, dak@gnu.org wrote:
>> 
>> scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
>> 
>> Actually, it is totally basic shell scripting here.
>
> Well, apparently you don't need "basic" shell scripting for 7
> years of full-time linux use and 4 years of macosx use, and
> thousands of hours of open-source work for multiple projects.
> I've called programs, I've used for loops, but anything fancier
> than that is done as a makefile or python.
>
> My rule of thumb is that if I don't understand something easily,
> then at least 50% of future lilypond contributors won't understand
> it either.

eval is ugly and a little-travelled corner, and it might be necessary to
write

eval "$1" > "$2" 2>&1

instead, though the amount of quoting required for working with shell
internals varies and I don't remember in all too fine detail.  The $1
quite likely wants quotes, the $2 will possibly do without.

But the actually "hard" thing is the eval (easy to find in the man page,
though).  And it is only needed when the command to be executed is a
_shell_ command.  So I am really quite doubtful that we reduce
complexity with this shell script which, after all, also needs to gets
called with the correct path.

-- 
David Kastrup

[Graham Percival, 2012-02-13 16:01:01]

On Mon, Feb 13, 2012 at 12:47:22PM +0100, David Kastrup wrote:
> Graham Percival <graham@percival-music.ca> writes:
> 
> > My rule of thumb is that if I don't understand something easily,
> > then at least 50% of future lilypond contributors won't understand
> > it either.
> 
> eval is ugly and a little-travelled corner, and it might be necessary to
> write
...
> So I am really quite doubtful that we reduce
> complexity with this shell script which, after all, also needs to gets
> called with the correct path.

I'm not talking about reducing complexity by changing the code.
I'm talking about dumping a two-line

# eval calls the command $1 and redirects it to the log file $2;
# the &>!2 makes sure that both stderr and stdout are saved
# to that log file.

(if that's what's actually happening)

It takes fewer key presses and less time that we've spent
discussing this to add such a comment, yet it may save somebody an
hour or more down the road.  I think it's worth doing.

Sure, you could argue that anbody who doesn't understand that much
about shell scripting shouldn't be modifying our build scripts...
but history has shown that often the only people who have time to
work on our build scripts know very little about shell scripts
(and build systems in general).  The fact that we have Julien and
Phil working on it, with active reviews from you and Reinhold, is
a historical aberration.

- Graham

[Jean-Charles, 2012-02-13 18:18:57]

Le 13/02/2012 14:56, julien.rioux@gmail.com disait :
> Looks good, I just have a few suggestions to avoid clashing log file
> names and I wonder how this part of GOP 9 might be implemented down the
> road:
> "All other logfiles will go in the @file{build/logfiles/} directory."
>
>
>
http://codereview.appspot.com/5650064/diff/5003/make/doc-i18n-root-rules.make...
>
> make/doc-i18n-root-rules.make:10: $(buildscript-dir)/run-and-check
> "DEPTH=$(depth)/../ $(TEXI2HTML) $(TEXI2HTML_SPLIT) $(TEXI2HTML_FLAGS)
> --output=$(outdir)/$* $<" "$*.splittexi.log"
> I suggest adding $(ISOLANG) to the name of the log files generated from
> this Makefile, eg:
> $*.$(ISOLANG).splittexi.log
>

Too bad ! I'm now used to see MANUAL.bigtexi.log and 
MANUAL.splittexi.log directly in Documentation/ISOLANG/

Very useful by the way for checking bad x-refs (size > 0 means something 
is wrong, except for web which normally contains a single line: 
"Processing web site: [ISOLANG]")
THANKS for this.

Cheers,
Jean-Charles

[mail_philholmes.net, 2012-02-14 11:13:53]

----- Original Message ----- 
From: "David Kastrup" <dak@gnu.org>
To: "Graham Percival" <graham@percival-music.ca>
Cc: <PhilEHolmes@googlemail.com>; <julien.rioux@gmail.com>; 
<mail@philholmes.net>; <lilypond-devel@gnu.org>; 
<reply@codereview-hr.appspotmail.com>
Sent: Monday, February 13, 2012 11:47 AM
Subject: Re: Final redirection of texi output (issue 5650064)


> Graham Percival <graham@percival-music.ca> writes:
>
>> On Mon, Feb 13, 2012 at 11:05:56AM +0000, dak@gnu.org wrote:
>>>
>>> scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
>>>
>>> Actually, it is totally basic shell scripting here.
>>
>> Well, apparently you don't need "basic" shell scripting for 7
>> years of full-time linux use and 4 years of macosx use, and
>> thousands of hours of open-source work for multiple projects.
>> I've called programs, I've used for loops, but anything fancier
>> than that is done as a makefile or python.
>>
>> My rule of thumb is that if I don't understand something easily,
>> then at least 50% of future lilypond contributors won't understand
>> it either.
>
> eval is ugly and a little-travelled corner, and it might be necessary to
> write
>
> eval "$1" > "$2" 2>&1
>
> instead, though the amount of quoting required for working with shell
> internals varies and I don't remember in all too fine detail.  The $1
> quite likely wants quotes, the $2 will possibly do without.

In this instance, it doesn't appear to need that.  If problems do arise, 
I'll try to remember the suggestion.

> But the actually "hard" thing is the eval (easy to find in the man page,
> though).  And it is only needed when the command to be executed is a
> _shell_ command.  So I am really quite doubtful that we reduce
> complexity with this shell script which, after all, also needs to gets
> called with the correct path.

I'm not trying to reduce complexity - rather to make "make doc" produce less 
output.

--
Phil Holmes

[dak, 2012-02-14 11:28:29]

"Phil Holmes" <mail@philholmes.net> writes:

> ----- Original Message ----- 
> From: "David Kastrup" <dak@gnu.org>
> To: "Graham Percival" <graham@percival-music.ca>
> Cc: <PhilEHolmes@googlemail.com>; <julien.rioux@gmail.com>;
> <mail@philholmes.net>; <lilypond-devel@gnu.org>;
> <reply@codereview-hr.appspotmail.com>
> Sent: Monday, February 13, 2012 11:47 AM
> Subject: Re: Final redirection of texi output (issue 5650064)
>
>
>> Graham Percival <graham@percival-music.ca> writes:
>>
>>> On Mon, Feb 13, 2012 at 11:05:56AM +0000, dak@gnu.org wrote:
>>>>
>>>> scripts/build/run-and-check.sh:2: eval $1 > $2 2>&1
>>>>
>>>> Actually, it is totally basic shell scripting here.
>>>
>>> Well, apparently you don't need "basic" shell scripting for 7
>>> years of full-time linux use and 4 years of macosx use, and
>>> thousands of hours of open-source work for multiple projects.
>>> I've called programs, I've used for loops, but anything fancier
>>> than that is done as a makefile or python.
>>>
>>> My rule of thumb is that if I don't understand something easily,
>>> then at least 50% of future lilypond contributors won't understand
>>> it either.
>>
>> eval is ugly and a little-travelled corner, and it might be necessary to
>> write
>>
>> eval "$1" > "$2" 2>&1
>>
>> instead, though the amount of quoting required for working with shell
>> internals varies and I don't remember in all too fine detail.  The $1
>> quite likely wants quotes, the $2 will possibly do without.
>
> In this instance, it doesn't appear to need that.

If any kind of quotes or wildcards are passed to the shell function, you
likely don't want them interpreted twice and/or prematurely.  Typical
candidates would be calling the shell function with something like
"cd xxx; echo *"
in which case you don't want to have * expanded before the cd is
executed.

If you create a shell function instead of a Makefile macro, it should do
what it is written for not just when called from inside of the Makefile.

If $1 happens to contain an absolute path in its contents, it is the
whim of the user creating the directory just how many metacharacters
will appear in it.

> If problems do arise, I'll try to remember the suggestion.

Good luck.

> I'm not trying to reduce complexity - rather to make "make doc"
> produce less output.

If you want to record one thing, but execute another, something like

	@echo x
        @x-with-lots-of-garbage

may be an option.  Of course, it might make it harder to reproduce the
last action causing a problem.

-- 
David Kastrup

[mail_philholmes.net, 2012-02-15 11:50:56]

----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <PhilEHolmes@googlemail.com>; <dak@gnu.org>; <graham@percival-music.ca>; 
<mail@philholmes.net>
Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
Sent: Monday, February 13, 2012 1:56 PM
Subject: Re: Final redirection of texi output (issue 5650064)


> Looks good, I just have a few suggestions to avoid clashing log file
> names and I wonder how this part of GOP 9 might be implemented down the
> road:
> "All other logfiles will go in the @file{build/logfiles/} directory."

Hmm.  Had forgotten that.  In practice I think the way it's actually working 
currently is better.  logfiles associated with, say, French translations go 
into the Doc/fr directory, and therefore translators can see them at a 
glance, rather than having to sort through them.  This patch isn't the last 
word, but my thought would be to let them go where they do now and see 
whether it actually causes problems.

> http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile
> File Documentation/GNUmakefile (right):
>
>
http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile#new...
> Documentation/GNUmakefile:234: $(buildscript-dir)/run-and-check
> "$(MAKEINFO) -I$(src-dir) -I$(outdir) --output=$(dir $@) --docbook $<"
> "$*.makeinfo.log"
> We might need .makeinfoxml.log here, or something similar, to
> distinguish the log files.
>
>
http://codereview.appspot.com/5650064/diff/5003/Documentation/GNUmakefile#new...
> Documentation/GNUmakefile:238: $(buildscript-dir)/run-and-check
> "$(MAKEINFO) -I$(src-dir) -I$(outdir) --output=$(dir $@) --docbook $<"
> "$*.makeinfo.log"
> again, .makeinfoxml.log

Done both of these in latest patch.

> http://codereview.appspot.com/5650064/diff/5003/make/doc-i18n-root-rules.make
> File make/doc-i18n-root-rules.make (right):
>
>
http://codereview.appspot.com/5650064/diff/5003/make/doc-i18n-root-rules.make...
> make/doc-i18n-root-rules.make:10: $(buildscript-dir)/run-and-check
> "DEPTH=$(depth)/../ $(TEXI2HTML) $(TEXI2HTML_SPLIT) $(TEXI2HTML_FLAGS)
> --output=$(outdir)/$* $<" "$*.splittexi.log"
> I suggest adding $(ISOLANG) to the name of the log files generated from
> this Makefile, eg:
> $*.$(ISOLANG).splittexi.log

Not necessary while the logfiles go into the relevant diretory for the 
translation.

>
http://codereview.appspot.com/5650064/diff/5003/stepmake/stepmake/texinfo-rul...
> File stepmake/stepmake/texinfo-rules.make (right):
>
>
http://codereview.appspot.com/5650064/diff/5003/stepmake/stepmake/texinfo-rul...
> stepmake/stepmake/texinfo-rules.make:75:
> $(buildscript-dir)/run-and-check "$(MAKEINFO) -I$(src-dir) -I$(outdir)
> --no-split --no-headers --output $@ $<"  "$*.makeinfo.log"
> .makeinfotxt.log
>
> http://codereview.appspot.com/5650064/
>

Done.

--
Phil Holmes

[Julien Rioux, 2012-02-15 14:43:09]

On 2012/02/15 11:50:56, mail_philholmes.net wrote:
> ----- Original Message ----- 
> From: <mailto:julien.rioux@gmail.com>
> > Looks good, I just have a few suggestions to avoid clashing log file
> > names and I wonder how this part of GOP 9 might be implemented down the
> > road:
> > "All other logfiles will go in the @file{build/logfiles/} directory."
> 
> Hmm.  Had forgotten that.  In practice I think the way it's actually working 
> currently is better.  logfiles associated with, say, French translations go 
> into the Doc/fr directory, and therefore translators can see them at a 
> glance, rather than having to sort through them.  This patch isn't the last 
> word, but my thought would be to let them go where they do now and see 
> whether it actually causes problems.
> 

I don't care so much where the log files end up, I was just pointing out what
GOP 9 says.

LGTM

Regards,
Julien