Brain surgery on the build system, first stage

Brain surgery on the build system, first stage

Sorry to do this just before 2.16 branching out, but if you remember
nightmares about non-distributed files and the like because of the
desire to include last-minute changes to translations and whatnot, IMO
the part of this patch regarding "make dist" is really worth the
trouble.  If you like, I can also include a merge of stepmake/stepmake
and make/, but I thought this would really clutter the review of other
changes, and this merge is trivial if you compare with the patch of
this issue, so that merge will be submitted for review after this
review issue settled down.

Commits are viewable separately on dev/jmandereau:
http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=shortlog;h=refs/heads/dev/jmandereau
I recommend inspecting changes locally with git with "git log -p -M"
to sort out real changes in moved files.

Patchy test is welcome, but is not enough.  People who have reasonable
horsepower (or many hours of CPU time to allocate for this, typically
12 hours on a Pentium 4 with 2GB RAM) and about five gigabytes of free
space might want to run the following test.

I checked which files disappeared or appeared with these changes in
both tarball and installation of binary and docs, with the following
"scripts" (they are very dirty, but I'm not sure it's worth adding
installation and distribution checks to Patchy), assuming you've set
LILYPOND_GIT and LILYPOND_BUILD_DIR as in
http://lilypond.org/doc/v2.15/Documentation/contributor/environment-variables
(please set them without using the tilde character '~') and you start
from master branch checked out and freshly pulled in $LILYPOND_GIT:

export VERSION=2.15.43
export MAKE_FLAGS=-j3 CPU_COUNT=3
./autogen.sh --noconfigure && \
rm -rf $LILYPOND_BUILD_DIR && \
mkdir -p $LILYPOND_BUILD_DIR && \
cd $LILYPOND_BUILD_DIR && \
${LILYPOND_GIT}/configure && \
mkdir -p $HOME/lily-base && \
make dist &>$HOME/lily-base/make-dist.log && \
mv out/lilypond-${VERSION}.tar.gz $HOME/lily-base && \
pushd $HOME/lily-base && tar -xzf lilypond-${VERSION}.tar.gz && popd && \
make distclean && \
${LILYPOND_GIT}/configure --prefix=$HOME/lily-base && \
make ${MAKE_FLAGS} &>$HOME/lily-base/make.log && \
make ${MAKE_FLAGS} test &>$HOME/lily-base/make-test.log && \
make ${MAKE_FLAGS} doc &>$HOME/lily-base/make-doc.log && \
make install &>$HOME/lily-base/make-install.log && \
make install-doc &>$HOME/lily-base/make-install-doc.log

Cd to $LILYPOND_GIT, apply patch -- or checkout dev/jmandereau, which is
most easily done with

git checkout origin/dev/jmandereau

then make sure you don't have anything valuable in files and
directories ignored by Git (because of the "git clean" command below),
then do

git clean -f -d -X && \ 
./autogen.sh --noconfigure && \
rm -rf $LILYPOND_BUILD_DIR && \
mkdir -p $LILYPOND_BUILD_DIR && \
cd $LILYPOND_BUILD_DIR && \
${LILYPOND_GIT}/configure && \
mkdir -p $HOME/lily-new && \
make dist &>$HOME/lily-new/make-dist.log && \
mv out/lilypond-${VERSION}.tar.gz $HOME/lily-new && \
pushd $HOME/lily-new && tar -xzf lilypond-${VERSION}.tar.gz && popd && \
make distclean && \
${LILYPOND_GIT}/configure --prefix=$HOME/lily-new && \
make ${MAKE_FLAGS} &>$HOME/lily-new/make.log && \
make ${MAKE_FLAGS} test &>$HOME/lily-new/make-test.log && \
make ${MAKE_FLAGS} doc &>$HOME/lily-new/make-doc.log && \
make install &>$HOME/lily-new/make-install.log && \
make install-doc &>$HOME/lily-new/make-install-doc.log

Cd to $HOME, check the output of
LANG= diff -qr lily-base lily-new |grep -i only
then test in-tree build (first line is optional, it's merely
here to free some disk space):

rm -rf $LILYPOND_BUILD_DIR && \
cd $LILYPOND_GIT && \
./configure && \
mkdir -p $HOME/lily-new-intree && \
make dist &>$HOME/lily-new-intree/make-dist.log && \
mv out/lilypond-${VERSION}.tar.gz $HOME/lily-new-intree && \
pushd $HOME/lily-new-intree && tar -xzf lilypond-${VERSION}.tar.gz && popd && \
make distclean && \
./configure --prefix=$HOME/lily-new-intree && \
make ${MAKE_FLAGS} &>$HOME/lily-new-intree/make.log && \
make ${MAKE_FLAGS} test &>$HOME/lily-new-intree/make-test.log && \
make ${MAKE_FLAGS} doc &>$HOME/lily-new-intree/make-doc.log && \
make install &>$HOME/lily-new-intree/make-install.log && \
make install-doc &>$HOME/lily-new-intree/make-install-doc.log

Again, cd to $HOME and see the output of
LANG= diff -qr lily-base lily-new-intree |grep -i only

You should in either case (lily-new, lily-new-intree) see differences
that match either deleted or moved files in the tree extracted from
tarball (these differences should match the ones in the commits being
reviewed), or lilypond-book hashing.


commit 2651085fa4642f802ac0ad3f6992eda548219ba9
Author: John Mandereau <john.mandereau@gmail.com>
Date:   Fri Aug 3 15:03:31 2012 +0200

    Make targets: improve *clean, remove bin, cvs-clean and maintainerclean
    
    * distclean was usable from any directory in theory, make it
    toplevel-only and call clean, doc-clean, test-clean and log-clean
    (previous behaviour can now be obtained with make clean && make
    local-distclean),
    
    * remove maintainerclean, which was a broken distclean-like, and
    cvs-clean that can be substituted by some "git clean" incantation,
    
    * remove bin, which does not work in a clean tree, and that can be
    replaced by  make bin-clean && make all in an already built tree,
    
    * update help target.

commit 10467285b3014ff14611718c60c25aa6aa844815
Author: John Mandereau <john.mandereau@gmail.com>
Date:   Thu Aug 2 23:25:54 2012 +0200

    Remove StepMake as a separate package
    
    Rationale:
    * there are no publicly-known current usage cases of StepMake outside
    LilyPond,
    * StepMake installation has been disabled for a decade,
    * generic templates (stepmake/stepmake) have been so heavily
    customized lately for our heavy documentation that they can no longer
    be qualifed as "generic",
    * only a few build scripts have remained in stepmake/bin, whereas
    most of them now live in scripts/build.
    
    This commit removes everything in stepmake/ except included makefiles;
    scripts that are still used are moved to config/ (Autoconf helpers) or
    scripts/build (build and install scripts).
    
    Remove config make target to avoid potential conflict with new
    config/ directory.
    
    For clarity, a merge of all stepmake templates, from stepmake/stepmake
    and make/ into stepmake/ (or similar), will be done in another commit.

commit 4fac59f99e2f5bbe0f78946aa9ed0f96d687b7aa
Author: John Mandereau <john.mandereau@gmail.com>
Date:   Wed Aug 1 12:22:43 2012 +0200

    Make distributed tarball from Git file list
    
    This removes the requirement of having a GNUmakefile
    in each and every directory of the source tree; this commit
    also deletes GNUmakefiles made unnecessary this way.
    
    dist-toplevel-txt-files toplevel target is also removed,
    because it is unneeded outside of dist target and it
    prevents linking of toplevel generated txt docs to
    distdir when these docs are already generated.
    
    In addition, on suggestions from Han-Wen, if dist is made with a
    source directory tracked by Git, then
    * the source directory is required to have no uncommitted changes nor
    non-ignored untracked files,
    * the time stamp of all files in the tarball is set to the time stamp
    of the head of the checked-out branch.

commit 4281dfd99d068e4c3ab53fa2343f5f3659fd8292
Author: John Mandereau <john.mandereau@gmail.com>
Date:   Wed Aug 1 12:14:50 2012 +0200

    mass-link: handle filesystem crossing and support OS without link support

[dak, 2012-08-09 18:32:28]

http://codereview.appspot.com/6446096/diff/2001/aclocal.m4
File aclocal.m4 (right):

http://codereview.appspot.com/6446096/diff/2001/aclocal.m4#newcode1350
aclocal.m4:1350: +    if test "$CYGWIN$MINGW32" = "nono"; then
This does not look like aclocal.m4 at all, more like the output of git log -p. 
What is up with this?

[John Mandereau, 2012-08-09 21:06:44]

http://codereview.appspot.com/6446096/diff/2001/aclocal.m4
File aclocal.m4 (right):

http://codereview.appspot.com/6446096/diff/2001/aclocal.m4#newcode1350
aclocal.m4:1350: +    if test "$CYGWIN$MINGW32" = "nono"; then
On 2012/08/09 18:32:28, dak wrote:
> This does not look like aclocal.m4 at all, more like the output of git log -p.

> What is up with this?

This looks like a bug in git-cl and/or side-by-side view or Rietveld.  Unified
diff view is correct, though.

[dak, 2012-08-17 10:27:08]

http://codereview.appspot.com/6446096/diff/2001/stepmake/stepmake/generic-tar...
File stepmake/stepmake/generic-targets.make (left):

http://codereview.appspot.com/6446096/diff/2001/stepmake/stepmake/generic-tar...
stepmake/stepmake/generic-targets.make:1: .PHONY : all clean bin-clean config
default dist exe help html lib TAGS\
The commit message states "Remove config make target to avoid potential conflict
with new config/ directory." but that seems nonsensical since "config" was
listed explicitly as a .PHONY target and so Make would not even have looked at
the config directory.