[Buildbot-commits] buildbot/docs buildbot.texinfo,1.124,1.125

[Brian Warner]

Update of /cvsroot/buildbot/buildbot/docs
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6673/docs

Modified Files:
	buildbot.texinfo 
Log Message:
[project @ update robots.txt, document it a bit better]

Original author: warner at lothar.com
Date: 2007-09-30 19:41:34+00:00

Index: buildbot.texinfo
===================================================================
RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v
retrieving revision 1.124
retrieving revision 1.125
diff -u -d -r1.124 -r1.125
--- buildbot.texinfo	30 Sep 2007 09:24:03 -0000	1.124
+++ buildbot.texinfo	30 Sep 2007 19:42:24 -0000	1.125
@@ -5992,6 +5992,14 @@
 c['status'].append(WebStatus(8080))
 @end example
 
+Note that the initial robots.txt file has Disallow lines for all of
+the dynamically-generated buildbot pages, to discourage web spiders
+and search engines from consuming a lot of CPU time as they crawl
+through the entire history of your buildbot. If you are running the
+buildbot behind a reverse proxy, you'll probably need to put the
+robots.txt file somewhere else (at the top level of the parent web
+server), and replace the URL prefixes in it with more suitable values.
+
 In addition, if you are familiar with twisted.web @emph{Resource
 Trees}, you can write code to add additional pages at places inside
 this web space. Just use @code{webstatus.putChild} to place these