LIB Fix b/10302420: Archive feeds to a local directory, if configured.

This change adds a new optional configuration property, feed.archiveDirectory.
If specified, it should be a pathname to a directory in which the all
feeds successfully and unsuccessfully sent to the GSA are archived.
Failed feeds are tagged with FAILED in the archive feed file name.

[Brett, 2014-06-21 18:24:08]

I am liking the interface to the feed file archiver less and less.  The biggest
problem is saveDocidsFeed(), which is also called when feeding ACLs.  I am
thinking I should minimally change the interface to:

void saveFeed(String dataSource, String feedXml);  
void saveGroupsFeed(String dataSource, String feedXml);
void saveFailedFeed(String dataSource, String feedXml);

I could also dispense with the special handling of groups feeds and just do:

void saveFeed(String dataSource, String feedXml);  
void saveFailedFeed(String dataSource, String feedXml);

[pjo, 2014-06-24 23:29:38]

Thank you.  This looks great.

For the interface.  How about:
   saveRecordsFeed()
   saveGroupsFeed()
   saveFailedRecordsFeed()
   saveFailedGroupsFeed()

(1) The ACLs and DocIds are both records in feed file
(2) Keep the failed groups and records feeds seperate

Thank you

https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
File src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java (right):

https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java:40: ? null : new
File(archiveDirectory);
please log

[brett.michael.johnson_gmail.com, 2014-06-25 16:21:45]

One of the reasons I was favoring the much simpler
interface, was to open it up to the graph traversers.
Making the class public would allow the FS adaptor
to archive its directory listing feeds.

—
Brett M. Johnson

On Jun 24, 2014, at 4:29 PM, pjo@google.com wrote:

> Thank you.  This looks great.
> 
> For the interface.  How about:
>   saveRecordsFeed()
>   saveGroupsFeed()
>   saveFailedRecordsFeed()
>   saveFailedGroupsFeed()
> 
> (1) The ACLs and DocIds are both records in feed file
> (2) Keep the failed groups and records feeds seperate
> 
> Thank you
> 
> 
>
https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
> File src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java (right):
> 
>
https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
> src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java:40: ? null :
> new File(archiveDirectory);
> please log
> 
> https://codereview.appspot.com/108100043/

[pjo, 2014-06-25 17:52:30]

Sounds good.  Two functions and making accessible via
AdaptorContext


- technology's compounding interest
-


On Wed, Jun 25, 2014 at 9:21 AM, Brett Johnson <
brett.michael.johnson@gmail.com> wrote:

> One of the reasons I was favoring the much simpler
> interface, was to open it up to the graph traversers.
> Making the class public would allow the FS adaptor
> to archive its directory listing feeds.
>
> --
> Brett M. Johnson
>
> On Jun 24, 2014, at 4:29 PM, pjo@google.com wrote:
>
> > Thank you.  This looks great.
> >
> > For the interface.  How about:
> >   saveRecordsFeed()
> >   saveGroupsFeed()
> >   saveFailedRecordsFeed()
> >   saveFailedGroupsFeed()
> >
> > (1) The ACLs and DocIds are both records in feed file
> > (2) Keep the failed groups and records feeds seperate
> >
> > Thank you
> >
> >
> >
>
https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
> > File src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java (right):
> >
> >
>
https://codereview.appspot.com/108100043/diff/1/src/com/google/enterprise/ada...
> > src/com/google/enterprise/adaptor/GsaFeedFileArchiver.java:40: ? null :
> > new File(archiveDirectory);
> > please log
> >
> > https://codereview.appspot.com/108100043/
>
>

[Brett, 2014-06-25 23:16:35]

Simplified interface, archiver available via AdaptorContext

[pjo, 2014-06-25 23:44:55]

LGTM.  Thank you.

[Brett, 2014-06-25 23:48:05]

Committed 25 June 2014 to the Adaptor Library

To https://code.google.com/p/plexi/
   0e947a3..a15692a  master -> master

[ejona, 2014-06-26 22:40:09]

On 2014/06/25 16:21:45, brett.michael.johnson_gmail.com wrote:
> One of the reasons I was favoring the much simpler
> interface, was to open it up to the graph traversers.
> Making the class public would allow the FS adaptor
> to archive its directory listing feeds.

The FS Adaptor will use (Async)DocIdPusher which uses DocIdSender which will
automatically use the Feed Archiver. Am I missing something?

There is no need to archive the HTML listings since those can be obtained easily
when needed for debugging. Feeds were special in that you couldn't really see
what was sent to the GSA.

[pjo, 2014-06-27 19:43:34]

Brett and I were thinking of the HtmlWriter's html listings.

You're right that those could be gotten by just going to
the URL.  Not sure if this means they shouldn't be saved
tho.  I'll ask on the bug ticket for advice. Let me know if
you feel strongly HTML should not be possible to save,
or should at least not be a library feature.

Thank you,
PJ



- technology's compounding interest
-


On Thu, Jun 26, 2014 at 3:40 PM, <ejona@google.com> wrote:

> On 2014/06/25 16:21:45, brett.michael.johnson_gmail.com wrote:
>
>> One of the reasons I was favoring the much simpler
>> interface, was to open it up to the graph traversers.
>> Making the class public would allow the FS adaptor
>> to archive its directory listing feeds.
>>
>
> The FS Adaptor will use (Async)DocIdPusher which uses DocIdSender which
> will automatically use the Feed Archiver. Am I missing something?
>
> There is no need to archive the HTML listings since those can be
> obtained easily when needed for debugging. Feeds were special in that
> you couldn't really see what was sent to the GSA.
>
> https://codereview.appspot.com/108100043/
>

[ejona, 2014-06-27 20:34:15]

On Fri, Jun 27, 2014 at 12:43 PM, PJ <pjo@google.com> wrote:

> Brett and I were thinking of the HtmlWriter's html listings.
>

Ah, okay.

You're right that those could be gotten by just going to
> the URL.  Not sure if this means they shouldn't be saved
> tho.  I'll ask on the bug ticket for advice. Let me know if
> you feel strongly HTML should not be possible to save,
> or should at least not be a library feature.
>

I don't think it is a good idea to go down that road. Storing the feeds is
only done because you can't download them from the GSA. It was not
possible. If it was just painful, I didn't think we would bother.

For the HTML you have other avenues. It isn't really solving anything for
the HTML case. In addition, using the FeedArchiver API would be a pain for
FsAdaptor. Right now, writing directory listing is fully-streamed, and we
should probably keep it so. Also, the file would be named "blah.xml", which
is strange given it is HTML.

And you would have to support the extra API surface. If the GSA gets
support to download the feed files (the proper place), then we couldn't
actually remove the code from the Adaptor library.

[pjo, 2014-06-27 20:39:59]

I'm pretty convinced.  How about you Brett?



- technology's compounding interest
-


On Fri, Jun 27, 2014 at 1:33 PM, Eric Anderson <ejona@google.com> wrote:

> On Fri, Jun 27, 2014 at 12:43 PM, PJ <pjo@google.com> wrote:
>
>> Brett and I were thinking of the HtmlWriter's html listings.
>>
>
> Ah, okay.
>
> You're right that those could be gotten by just going to
>> the URL.  Not sure if this means they shouldn't be saved
>> tho.  I'll ask on the bug ticket for advice. Let me know if
>> you feel strongly HTML should not be possible to save,
>> or should at least not be a library feature.
>>
>
> I don't think it is a good idea to go down that road. Storing the feeds is
> only done because you can't download them from the GSA. It was not
> possible. If it was just painful, I didn't think we would bother.
>
> For the HTML you have other avenues. It isn't really solving anything for
> the HTML case. In addition, using the FeedArchiver API would be a pain for
> FsAdaptor. Right now, writing directory listing is fully-streamed, and we
> should probably keep it so. Also, the file would be named "blah.xml", which
> is strange given it is HTML.
>
> And you would have to support the extra API surface. If the GSA gets
> support to download the feed files (the proper place), then we couldn't
> actually remove the code from the Adaptor library.
>

[ejona, 2014-06-27 20:58:55]

BTW: the CL did look great. The feature should be very useful for non-Google
employees.

[brett.michael.johnson_gmail.com, 2014-06-27 22:10:48]

I am not super attached to opening it up to the adaptors.
Customers have long expressed a desire to track *everything*
we send to the GSA, and especially those that we fail to send,
process, or index.  They have wanted end-to-end tracking, and
this doesn’t even come close to that.

I’m not too concerned about the HTML/XML
bit, since we can make sure HtmlWriter produces XHTML.

—
Brett M. Johnson

On Jun 27, 2014, at 1:39 PM, PJ <pjo@google.com> wrote:

> I'm pretty convinced.  How about you Brett?
> 
> 
> 
> - technology's compounding interest
> -
> 
> 
> On Fri, Jun 27, 2014 at 1:33 PM, Eric Anderson <ejona@google.com> wrote:
> On Fri, Jun 27, 2014 at 12:43 PM, PJ <pjo@google.com> wrote:
> Brett and I were thinking of the HtmlWriter's html listings.
> 
> Ah, okay.
> 
> You're right that those could be gotten by just going to 
> the URL.  Not sure if this means they shouldn't be saved
> tho.  I'll ask on the bug ticket for advice. Let me know if
> you feel strongly HTML should not be possible to save,
> or should at least not be a library feature.
> 
> I don't think it is a good idea to go down that road. Storing the feeds is
only done because you can't download them from the GSA. It was not possible. If
it was just painful, I didn't think we would bother.
> 
> For the HTML you have other avenues. It isn't really solving anything for the
HTML case. In addition, using the FeedArchiver API would be a pain for
FsAdaptor. Right now, writing directory listing is fully-streamed, and we should
probably keep it so. Also, the file would be named "blah.xml", which is strange
given it is HTML.
> 
> And you would have to support the extra API surface. If the GSA gets support
to download the feed files (the proper place), then we couldn't actually remove
the code from the Adaptor library.
> 

[ejona, 2014-06-27 22:14:39]

On Fri, Jun 27, 2014 at 3:10 PM, Brett Johnson <
brett.michael.johnson@gmail.com> wrote:

> I’m not too concerned about the HTML/XML
> bit,
>

I'm not super-concerned with it either. But it does add to the "should we
use this for that?"

since we can make sure HtmlWriter produces XHTML.
>

That seems like an anti-goal. That sort of change doesn't help anyone. And
with the advent of HTML 5, XHTML seems dead and forever a dead-end.