Issue 5494069: Build translated manuals: Don't rebuild everything each time unless necessary.

Rietveld

Moving towards a build system which tracks dependencies for every file, update
timestamps correctly and so on. These are all the fixes for the translated
manuals.
Please review. I tested make, make check, make doc, and do not get any problem.
However, with the previous versions I also did not get any problem, yet I broke
staging. I would be grateful if someone with a fast machine could test a
parallel make doc. I would not be comfortable pushing to staging without this
test.

Regards,
Julien
----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <julien.rioux@gmail.com>
Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
Sent: Wednesday, January 11, 2012 2:39 PM
Subject: Re: Build translated manuals: Don't rebuild everything each 
timeunless necessary. (issue 5494069)

> Please review. I tested make, make check, make doc, and do not get any
> problem. However, with the previous versions I also did not get any
> problem, yet I broke staging. I would be grateful if someone with a fast
> machine could test a parallel make doc. I would not be comfortable
> pushing to staging without this test.
>
> Regards,
> Julien
>
> http://codereview.appspot.com/5494069/

On it.  Just need to work out how to download and apply a patch from 
Rietveld.  This'll take me 1/2 hour or so, because I need to build a fresh 
instance to get a baseline of filenames, then clean, apply patch and rebuild 
and check.  I've got this automated so it's not much hassle.

--
Phil Holmes

----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <julien.rioux@gmail.com>
Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
Sent: Wednesday, January 11, 2012 2:39 PM
Subject: Re: Build translated manuals: Don't rebuild everything each 
timeunless necessary. (issue 5494069)

> Please review. I tested make, make check, make doc, and do not get any
> problem. However, with the previous versions I also did not get any
> problem, yet I broke staging. I would be grateful if someone with a fast
> machine could test a parallel make doc. I would not be comfortable
> pushing to staging without this test.
>
> Regards,
> Julien

On my fast system, make doc takes 12min 30 approx and currently produces a 
6.5 meg logfile.  I've run a system patched with this, and it ran for over 
20 minutes, with no apparent updates to the output, which was "only" about 1 
meg.  Only one CPU was being used (of 8) and this was 100% processing make. 
make output is available if you want to see what I have.

--
Phil Holmes

On Wed, Jan 11, 2012 at 1:50 PM, Phil Holmes <mail@philholmes.net> wrote:
> ----- Original Message ----- From: <julien.rioux@gmail.com>
> To: <julien.rioux@gmail.com>
> Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
> Sent: Wednesday, January 11, 2012 2:39 PM
> Subject: Re: Build translated manuals: Don't rebuild everything each
> timeunless necessary. (issue 5494069)
>
>
>> Please review. I tested make, make check, make doc, and do not get any
>> problem. However, with the previous versions I also did not get any
>> problem, yet I broke staging. I would be grateful if someone with a fast
>> machine could test a parallel make doc. I would not be comfortable
>> pushing to staging without this test.
>>
>> Regards,
>> Julien
>
>
>
> On my fast system, make doc takes 12min 30 approx and currently produces a
> 6.5 meg logfile.  I've run a system patched with this, and it ran for over
> 20 minutes, with no apparent updates to the output, which was "only" about 1
> meg.  Only one CPU was being used (of 8) and this was 100% processing make.
> make output is available if you want to see what I have.
>
> --
> Phil Holmes
>
>

Thanks for testing. Did make doc ever finish? Maybe it decided that
things need to be done in series, which could take up to 1h40m based
on your fast parallel build on 8 CPUs. You can send the output
privately.

Thanks,
Regards,
Julien

----- Original Message ----- 
From: "Julien Rioux" <julien.rioux@gmail.com>
To: "Phil Holmes" <mail@philholmes.net>
Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
Sent: Wednesday, January 11, 2012 9:41 PM
Subject: Re: Build translated manuals: Don't rebuild everything each 
timeunless necessary. (issue 5494069)

On Wed, Jan 11, 2012 at 1:50 PM, Phil Holmes <mail@philholmes.net> wrote:
> ----- Original Message ----- From: <julien.rioux@gmail.com>
> To: <julien.rioux@gmail.com>
> Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
> Sent: Wednesday, January 11, 2012 2:39 PM
> Subject: Re: Build translated manuals: Don't rebuild everything each
> timeunless necessary. (issue 5494069)
>
>
>> Please review. I tested make, make check, make doc, and do not get any
>> problem. However, with the previous versions I also did not get any
>> problem, yet I broke staging. I would be grateful if someone with a fast
>> machine could test a parallel make doc. I would not be comfortable
>> pushing to staging without this test.
>>
>> Regards,
>> Julien
>
>
>
> On my fast system, make doc takes 12min 30 approx and currently produces a
> 6.5 meg logfile. I've run a system patched with this, and it ran for over
> 20 minutes, with no apparent updates to the output, which was "only" about 
> 1
> meg. Only one CPU was being used (of 8) and this was 100% processing make.
> make output is available if you want to see what I have.
>
> --
> Phil Holmes
>
>

Thanks for testing. Did make doc ever finish? Maybe it decided that
things need to be done in series, which could take up to 1h40m based
on your fast parallel build on 8 CPUs. You can send the output
privately.

Thanks,
Regards,
Julien

================================================

I terminated it before it got that far.  I'll run it again tomorrow to see.

--
Phil Holmes

2012/1/11  <julien.rioux@gmail.com>:
> Please review. I tested make, make check, make doc, and do not get any
> problem. However, with the previous versions I also did not get any
> problem, yet I broke staging. I would be grateful if someone with a fast
> machine could test a parallel make doc. I would not be comfortable
> pushing to staging without this test.
>
> Regards,
> Julien
>
> http://codereview.appspot.com/5494069/

I have applied it and then tested twice the following

time ( make -j3 CPU_COUNT=3 all && make -j3 CPU_COUNT=3 doc )

runs successfully and the second time it lasts for less than a minute,
also successfully.

-- 
Francisco Vila. Badajoz (Spain)
www.paconet.org , www.csmbadajoz.com

----- Original Message ----- 
From: "Phil Holmes" <mail@philholmes.net>
To: "Julien Rioux" <julien.rioux@gmail.com>
Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
Sent: Wednesday, January 11, 2012 9:52 PM
Subject: Re: Build translated manuals: Don't rebuild everything 
eachtimeunless necessary. (issue 5494069)

> ----- Original Message ----- 
> From: "Julien Rioux" <julien.rioux@gmail.com>
> To: "Phil Holmes" <mail@philholmes.net>
> Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
> Sent: Wednesday, January 11, 2012 9:41 PM
> Subject: Re: Build translated manuals: Don't rebuild everything each 
> timeunless necessary. (issue 5494069)
>
>
> On Wed, Jan 11, 2012 at 1:50 PM, Phil Holmes <mail@philholmes.net> wrote:
>> ----- Original Message ----- From: <julien.rioux@gmail.com>
>> To: <julien.rioux@gmail.com>
>> Cc: <reply@codereview-hr.appspotmail.com>; <lilypond-devel@gnu.org>
>> Sent: Wednesday, January 11, 2012 2:39 PM
>> Subject: Re: Build translated manuals: Don't rebuild everything each
>> timeunless necessary. (issue 5494069)
>>
>>
>>> Please review. I tested make, make check, make doc, and do not get any
>>> problem. However, with the previous versions I also did not get any
>>> problem, yet I broke staging. I would be grateful if someone with a fast
>>> machine could test a parallel make doc. I would not be comfortable
>>> pushing to staging without this test.
>>>
>>> Regards,
>>> Julien
>>
>>
>>
>> On my fast system, make doc takes 12min 30 approx and currently produces 
>> a
>> 6.5 meg logfile. I've run a system patched with this, and it ran for over
>> 20 minutes, with no apparent updates to the output, which was "only" 
>> about 1
>> meg. Only one CPU was being used (of 8) and this was 100% processing 
>> make.
>> make output is available if you want to see what I have.
>>
>> --
>> Phil Holmes
>>
>>
>
> Thanks for testing. Did make doc ever finish? Maybe it decided that
> things need to be done in series, which could take up to 1h40m based
> on your fast parallel build on 8 CPUs. You can send the output
> privately.
>
> Thanks,
> Regards,
> Julien
>
> ================================================
>
> I terminated it before it got that far.  I'll run it again tomorrow to 
> see.
>
> --
> Phil Holmes
>
>
>
> _______________________________________________
> lilypond-devel mailing list
> lilypond-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/lilypond-devel
>

Let it run for almost 4 hours tonight.  It produces output for about 12 
minutes and then takes a single CPU for 100% load, with no console output. 
I've zipped the 1 meg output file and sent it to Julien off-list.  This is 
with a core i7 - 4 processor hyperthreaded - running as make -j9 
CPU_COUNT=9.

--
Phil Holmes

Phil, could you please put this patch to test again?
Thanks,
Julien
----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <mail@philholmes.net>; <paconet.org@gmail.com>
Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
Sent: Saturday, February 18, 2012 7:59 PM
Subject: Re: Build translated manuals: Don't rebuild everything each time 
unless necessary. (issue 5494069)

> Phil, could you please put this patch to test again?
> Thanks,
> Julien
>
> http://codereview.appspot.com/5494069/
>

Willdo tomorrow.

--
Phil Holmes

----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <mail@philholmes.net>; <paconet.org@gmail.com>
Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
Sent: Saturday, February 18, 2012 7:59 PM
Subject: Re: Build translated manuals: Don't rebuild everything each time 
unless necessary. (issue 5494069)

> Phil, could you please put this patch to test again?
> Thanks,
> Julien
>
> http://codereview.appspot.com/5494069/
>

Ran without problems.  There are a number of differences in the list of 
files produced in /build, but I'm not sure of the significance of this.  To 
explain, as part of my build sequence I list all the files in the build 
directory to a text file and compare them to the previous text file with 
diff.  The text files are each about 5 megs but will probably zip well.  Are 
you interested in the output from make doc -s, plus the file listings and 
diffs?

--
Phil Holmes

On Sun, Feb 19, 2012 at 6:47 AM, Phil Holmes <mail@philholmes.net> wrote:
> ----- Original Message ----- From: <julien.rioux@gmail.com>
> To: <mail@philholmes.net>; <paconet.org@gmail.com>
> Cc: <lilypond-devel@gnu.org>; <reply@codereview-hr.appspotmail.com>
> Sent: Saturday, February 18, 2012 7:59 PM
> Subject: Re: Build translated manuals: Don't rebuild everything each time
> unless necessary. (issue 5494069)
>
>
>> Phil, could you please put this patch to test again?
>> Thanks,
>> Julien
>>
>> http://codereview.appspot.com/5494069/
>>
>
> Ran without problems.  There are a number of differences in the list of
> files produced in /build, but I'm not sure of the significance of this.  To
> explain, as part of my build sequence I list all the files in the build
> directory to a text file and compare them to the previous text file with
> diff.  The text files are each about 5 megs but will probably zip well.  Are
> you interested in the output from make doc -s, plus the file listings and
> diffs?
>
> --
> Phil Holmes
>
>

Thanks for your test. I expect some diff in
Documentation/LANG/out-www, so as long as the diff occur only in these
directories I don't think I need to look at them.

Regards,
Julien

LGTM

Issue 5494069: Build translated manuals: Don't rebuild everything each time unless necessary. (Closed)

Description

Patch Set 1 #

Patch Set 2 : Two parts were not committed, here are reworked, more conservative versions #

Patch Set 3 : Rebase to latest master and try to avoid cyclic dependencies #

Messages