sort input files

sort input files

to have reproducible link order
so that the lilypond binary always contains its functions in the same order.

When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output,
thus without the patch, builds (in disposable VMs) would differ.

See https://reproducible-builds.org/ for why this matters.

There exists a patch for GNU make to make all wildcard globs sorted again, but they claim the spec does not guarantee any stable order so it is not yet merged:
https://savannah.gnu.org/bugs/index.php?52076

[dak, 2018-05-18 13:32:26]

https://codereview.appspot.com/347770043/diff/1/stepmake/stepmake/c++-vars.make
File stepmake/stepmake/c++-vars.make (right):

https://codereview.appspot.com/347770043/diff/1/stepmake/stepmake/c++-vars.ma...
stepmake/stepmake/c++-vars.make:14: CC_FILES := $(sort $(call
src-wildcard,*.cc))
src-wildcard actually uses the wildcard function of Make.  The documentation of
it does not mention sorting in any manner.  However, since it is modelled after
shell globs, it would seem weird for it not to sort.  So the question is whether
this should rather be addressed in Make or whether this is a symptom of a
different problem (like LC_COLLATE having different settings on different
systems, something likely also affecting the outcome of the "sort" function). 
Short of that, wouldn't it make more sense to redefine src-wildcard to always
sort?

It's defined as
stepmake/stepmake/generic-vars.make:src-wildcard = $(subst
$(src-dir)/,,$(wildcard $(src-dir)/$(1)))

[bmwiedemann, 2018-05-18 13:41:44]

On 2018/05/18 13:32:26, dak wrote:
> src-wildcard actually uses the wildcard function of Make.  The documentation
of
> it does not mention sorting in any manner.  However, since it is modelled
after
> shell globs, it would seem weird for it not to sort.  So the question is
whether
> this should rather be addressed in Make

See also https://savannah.gnu.org/bugs/index.php?52076 on that topic.

somewhat related: python globs are also surprisingly not sorted
http://bugs.python.org/issue30461

[Dan Eble, 2018-05-18 21:15:39]

https://codereview.appspot.com/347770043/diff/1/stepmake/stepmake/c++-vars.make
File stepmake/stepmake/c++-vars.make (right):

https://codereview.appspot.com/347770043/diff/1/stepmake/stepmake/c++-vars.ma...
stepmake/stepmake/c++-vars.make:14: CC_FILES := $(sort $(call
src-wildcard,*.cc))
This seems fragile.  If the order of arguments to the linker is what matters,
the call to sort should be in the linker command.  For example, one could break
this by adding CC_FILES += $(wildcard another-dir/*.cc) in an included makefile.

[bmwiedemann, 2018-05-23 12:22:09]

sorting .o files instead of .cc

[Dan Eble, 2018-05-26 13:04:12]

On 2018/05/23 12:22:09, bmwiedemann wrote:
> sorting .o files instead of .cc

This is not quite what I meant.  If it works for you, fine, but I think it is
more likely to break eventually than if you found the commands that are being
invoked and sorted their arguments.  For example,

    $(CXX) -o $@ $(sort $(O_FILES)) blah blah blah

This puts the sort where it really matters and leaves no room for modifying
O_FILES after the sort.  It would still be a good idea to include a short
comment on the reason for the sort, because It isn't obvious.

[lilypond-pkx, 2018-06-07 15:45:58]

On 2018/05/26 13:04:12, Dan Eble wrote:
> On 2018/05/23 12:22:09, bmwiedemann wrote:
> > sorting .o files instead of .cc
> 
> This is not quite what I meant.  If it works for you, fine, but I think it is
> more likely to break eventually than if you found the commands that are being
> invoked and sorted their arguments.  For example,
> 
>     $(CXX) -o $@ $(sort $(O_FILES)) blah blah blah
> 
> This puts the sort where it really matters and leaves no room for modifying
> O_FILES after the sort.  It would still be a good idea to include a short
> comment on the reason for the sort, because It isn't obvious.

Dan et al.

I am not seeing any feedback from Bernhard here on or on the tracker. I don't
want to waste good work, but it seems to me that this patch is still not ready
for pushing yet. I'll set the issue back down to 'review' for one more countdown
before I will set it to 'needs_work', but if you or someone else reading this
(preferably Bernhard) can help me understand if this can be pushed or not, then
I'd rather not have to make the unilateral decision myself to take it off the
countdown (even though I am Patch Meister).

James

[Dan Eble, 2018-06-07 20:37:28]

https://codereview.appspot.com/347770043/diff/20001/stepmake/stepmake/c++-var...
File stepmake/stepmake/c++-vars.make (right):

https://codereview.appspot.com/347770043/diff/20001/stepmake/stepmake/c++-var...
stepmake/stepmake/c++-vars.make:23: O_FILES+=$(addprefix $(outdir)/, $(sort
$(CC_FILES:.cc=.o) $(LL_FILES:.ll=.o) $(YY_FILES:.yy=.o)))
After closer inspection, I take back "if it works for you, fine."  I don't think
this patch accomplishes what is intended.  This line mentions objects produced
from .ll and .yy files, neither of which are sorted.

I reiterate my earlier suggestion.  If you want to be sure that the arguments of
a specific command are sorted, call sort on that very line.