robots.txt belongs in the root dir (issue 3430).

robots.txt belongs in the root dir (issue 3430).

Same goes for favicon.ico, both of them should go to the root directory of the website.

See http://code.google.com/p/lilypond/issues/detail?id=3430

[PhilEHolmes, 2013-07-08 11:13:00]

A small but important comment.

https://codereview.appspot.com/10983043/diff/1/make/website.make
File make/website.make (right):

https://codereview.appspot.com/10983043/diff/1/make/website.make#newcode158
make/website.make:158: $(OUT)/tweets.xml \
I've checked, and tweets.xml should be in the /website/ directory, not the root
directory.  (That's where it is now, and editing it directly on the web server
is effective at chaging the text displayed).

[Julien Rioux, 2013-07-08 12:44:13]

Leave tweets alone.

[PhilEHolmes, 2013-07-08 13:00:25]

LGTM.

FWIW you can check this is working on the live server within less than 2 hours
of the change hitting master.  If you were to add a comment to robots.txt as
part of the patch, the updated make website should be picked up from git and
used in a new make of the website - you'd then see the change made to robots.txt
replicated on the website.

[Julien Rioux, 2013-07-08 13:19:26]

On 2013/07/08 13:00:25, PhilEHolmes wrote:
> LGTM.
> 
> FWIW you can check this is working on the live server within less than 2 hours
> of the change hitting master.  If you were to add a comment to robots.txt as
> part of the patch, the updated make website should be picked up from git and
> used in a new make of the website - you'd then see the change made to
robots.txt
> replicated on the website.

Actually, I am confused now, because tweets.xml is obviously already in the root
dir:
http://lilypond.org/tweets.xml
but according to the Makefile it goes into the website subdir. Phil, do you
understand this?

--
Julien

[email_philholmes.net, 2013-07-08 13:38:19]

----- Original Message ----- 
From: <julien.rioux@gmail.com>
To: <phileholmes@googlemail.com>
Cc: <reply@codereview-hr.appspotmail.com>
Sent: Monday, July 08, 2013 2:19 PM
Subject: Re: robots.txt belongs in the root dir (issue 3430). (issue 
10983043)


> On 2013/07/08 13:00:25, PhilEHolmes wrote:
>> LGTM.
>
>> FWIW you can check this is working on the live server within less than
> 2 hours
>> of the change hitting master.  If you were to add a comment to
> robots.txt as
>> part of the patch, the updated make website should be picked up from
> git and
>> used in a new make of the website - you'd then see the change made to
> robots.txt
>> replicated on the website.
>
> Actually, I am confused now, because tweets.xml is obviously already in
> the root dir:
> http://lilypond.org/tweets.xml
> but according to the Makefile it goes into the website subdir. Phil, do
> you understand this?
>
> --
> Julien
>
> https://codereview.appspot.com/10983043/

Gonna have to look at the way the directories translate between the server 
and the make system.  Sorting out some music for tomorrow right now, but 
will get on it asap.

--
Phil Holmes 

[Graham Percival, 2013-07-11 02:42:22]

I believe that the current version is correct.  Phil, see
http://lilypond.org/doc/v2.17/Documentation/contributor/uploading-and-security
  ### make-website.sh
(halfway down the page)
and then poke around on the server a bit.

[email_philholmes.net, 2013-07-11 08:42:07]

----- Original Message ----- 
From: <graham@percival-music.ca>
To: <julien.rioux@gmail.com>; <phileholmes@googlemail.com>; 
<email@philholmes.net>
Cc: <reply@codereview-hr.appspotmail.com>
Sent: Thursday, July 11, 2013 3:42 AM
Subject: Re: robots.txt belongs in the root dir (issue 3430). (issue 
10983043)


>I believe that the current version is correct.  Phil, see
>
http://lilypond.org/doc/v2.17/Documentation/contributor/uploading-and-security
>   ### make-website.sh
> (halfway down the page)
> and then poke around on the server a bit.
>
> https://codereview.appspot.com/10983043/


My priority for today is 
http://code.google.com/p/lilypond/issues/detail?id=3432

However, I am currently certain that the current system isn't working. 
Compare http://lilypond.org/website/robots.txt with 
http://lilypond.org/robots.txt

--
Phil Holmes 

[Graham Percival, 2013-07-11 10:04:50]

On 2013/07/11 08:42:07, email_philholmes.net wrote:
> However, I am currently certain that the current system isn't working. 
> Compare http://lilypond.org/website/robots.txt with 
> http://lilypond.org/robots.txt

I see.  I poked around a bit, and yes, make-website.sh doesn't put anything in
the ~/web/ directory on the server; it only copies stuff to the ~/web/website/
directory.  I can't remember if there's any automated way of updating files in
~/web/ .  I have a sneaky suspicion that due to security concerns and the
infrequent nature of such updates, I didn't add any such mechanism.

Regardless, I think the stage this should happen is in make-website.sh rather
than make/website.make.  I added the out-website/website/ dir specifically
because texi2html dumps a lot of crap in the "build directory" (all the xref-map
stuff).  The robots.txt in out-wbsite/website/robots.txt looks fine (it has your
extra comment); the only issue is getting that file from the build dir to the
specific directory setup on lilypond.org

[email_philholmes.net, 2013-07-11 15:53:00]

----- Original Message ----- 
From: <graham@percival-music.ca>
To: <julien.rioux@gmail.com>; <phileholmes@googlemail.com>; 
<email@philholmes.net>
Cc: <reply@codereview-hr.appspotmail.com>
Sent: Thursday, July 11, 2013 11:04 AM
Subject: Re: robots.txt belongs in the root dir (issue 3430). (issue 
10983043)


> On 2013/07/11 08:42:07, email_philholmes.net wrote:
>> However, I am currently certain that the current system isn't working.
>
>> Compare http://lilypond.org/website/robots.txt with
>> http://lilypond.org/robots.txt
>
> I see.  I poked around a bit, and yes, make-website.sh doesn't put
> anything in the ~/web/ directory on the server; it only copies stuff to
> the ~/web/website/ directory.  I can't remember if there's any automated
> way of updating files in ~/web/ .  I have a sneaky suspicion that due to
> security concerns and the infrequent nature of such updates, I didn't
> add any such mechanism.
>
> Regardless, I think the stage this should happen is in make-website.sh
> rather than make/website.make.  I added the out-website/website/ dir
> specifically because texi2html dumps a lot of crap in the "build
> directory" (all the xref-map stuff).  The robots.txt in
> out-wbsite/website/robots.txt looks fine (it has your extra comment);
> the only issue is getting that file from the build dir to the specific
> directory setup on lilypond.org
>
> https://codereview.appspot.com/10983043/

So it looks like all that's necessary is to add

cp $DEST/website/robots.txt $DEST

to make-website.sh?  What about favicon.ico?

BTW - you'll have noticed I've updated the 2 shell scripts that run as cron 
jobs to allow me to check their output.  You want this updated in the CG?

--
Phil Holmes 

[Graham Percival, 2013-07-11 20:36:33]

On Thu, Jul 11, 2013 at 04:52:49PM +0100, Phil Holmes wrote:
> So it looks like all that's necessary is to add
> 
> cp $DEST/website/robots.txt $DEST
> 
> to make-website.sh?  What about favicon.ico?

I guess so, and favico.ico should also be copied.

> BTW - you'll have noticed I've updated the 2 shell scripts that run
> as cron jobs to allow me to check their output.  You want this
> updated in the CG?

Yes please; anything that reduces our "oral culture" is a good
thing.

Cheers,
- Graham