[Buildbot-devel] html.Waterfall robots.txt support patch

[Justin Mason]

Tobi Vollebregt writes:
> > Does anyone on the list actually want a robots.txt besides
> > "User-agent: *\nDisallow: /\n"?  Is there any reason that shouldn't
> > just be hard-coded into the server?  (That was my lazy solution.)
> 
> My reason to do it this way, was that I could just look at how it was
> done for the css and the favicon and copy that ;-)
> 
> Also, IMHO, it (almost) never hurts to provide more functionality then
> strictly necessary for 90% of the users. You never know: someone might
> want their home brewn log crawler to index the logs, but refuse other
> search engines. Or in the future buildbot is extended to show other
> pages which make more sense to index?

yes -- bear in mind that a robot which scrapes the status pages to (let's
say) transcode and display on a cell-phone screen, *should* be respecting
robots.txt too.  in other words, not all robots are search engine spiders.

--j.