[Buildbot-devel] RFC: more flexible web status display

[Brian Warner]

I've got another branch where I'm working on improving the web status
delivery code. Specifically, I'm trying to set up an architecture where you
can mix-and-match a variety of specialized status pages, of which the
existing Waterfall display would be just one possibility.

This framework would make it a lot easier to have a page which, e.g., has a
single line of text per build, or a page of just the builds that occurred on
the trunk, or a dynamically-generated PNG image showing the status of the
last build (which could then be IMG-tag included from some other page). It
also should make it a lot easier to serve robots.txt and CSS files.

Please take a look at the preliminary documentation on this feature and tell
me what you think. I'm still trying to figure out how to implement this all,
but first I want to make sure that you all think it will be
useable/configurable with the proposed interface.

thanks,
 -Brian



@heading NEW STUFF

@cindex WebStatus

The Waterfall object is actually a special case of the more
generalized @code{WebStatus} target. By using @code{WebStatus}
directly, you can construct arbitrary trees of web pages, mixing
static HTML pages with generated status displays of various types.

@menu
* Web Data Model::              
* Static Resources::            
* Buildbot Web Resources::      
* Special-Purpose Web Resources::  
@end menu

@node Web Data Model, Static Resources, HTML Waterfall, HTML Waterfall
@subsubsection Web Data Model

The model used by @code{WebStatus} is that of a @emph{resource
tree}@footnote{note that this is the same model used by the underlying
@code{twisted.web} framework, and all the status pages described here
are actually subclasses of @code{twisted.web.resource.Resource}}. In
this model, the web site is made up of a tree of nodes, and URLs
represent a slash-separated list of names that describes the path you
take from the root node down to the designated child.

The @code{WebStatus} object holds both the root node and any children.
You add children to any node by calling @code{putChild}. Each child
node is a @code{Resource}, and there are a variety of @code{Resource}
types that each render themselves in various ways.

For example, to create a simple web site that only has the root page
and a child page named @code{child.html}, you would do the following:

@example
from twisted.status import html
s = html.WebStatus(port=8080)
s.setRoot(html.Data("This is the root page.\n", "text/plain"))
s.putChild("child.html", html.Data("I am the child.\n", "text/plain"))
c['status'].append(s)
@end example

If the buildmaster were running on a host named
@code{buildbot.example.org}, then you would see the following behavior
from the web site (using the @command{GET} tool from libwww-perl):

@example
% GET http://buildbot.example.org:8080/
This is the root page.
% GET http://buildbot.example.org:8080/child.html
I am the child.
@end example

To create grand-child pages (which have URLs that contain two or more
slashes), you attach children to the child nodes:

@example
from twisted.status import html
s = html.WebStatus(port=8080)
s.setRoot(html.Data("This is the root page.\n", "text/plain"))
child1 = html.Data("I am the child.\n", "text/plain")
child2 = html.Data("Grandchild here. Goo goo gah.\n", "text/plain")
s.putChild("child", child1)
child1.putChild("baby", child2)
c['status'].append(s)
@end example

@example
% GET http://buildbot.example.org:8080/
This is the root page.
% GET http://buildbot.example.org:8080/child
I am the child.
% GET http://buildbot.example.org:8080/child/baby
Grandchild here. Goo goo gah.
@end example

@node Static Resources, Buildbot Web Resources, Web Data Model, HTML Waterfall
@subsubsection Static Resources

There are two resource types that are useful for simple static pages.
These do not utilize any status data from the Buildbot, but instead
are useful for things like a welcome page, providing CSS or image
files, and for serving a @file{robots.txt} page to discourage web
crawlers from sucking out gigabytes of logfiles.

All these resource types are defined in @code{buildbot.status.html},
so to create a @code{File} node, you should create an instance of
@code{buildbot.status.html.File}.

@table @code

@item Data(data, mimetype)

This produces a simple, statically-defined page. The data must be
provided as a string in the @code{data} argument, so it is most useful
for short pieces of text.

@example
robots_txt = \
"""User-agent: *
Crawl-Delay: 30
"""
s.putChild("robots.txt", html.Data(robots_txt, "text/plain"))
@end example

@item File(path)

This resource serves files from the disk. When the @code{path}
argument is a filename, it serves a single file. When @code{path} is a
directory, it serves a whole tree of files and directories from disk.
This works a lot like the @file{~/public_html} served by many Apache
web servers. @code{path} is interpreted relative to the buildmaster's
base directory.

It is a nuisance to add children to the interior of a directory served
this way, so @code{File} should probably be used as a leaf node.

@example
s.putChild("static", html.File("buildbot_html"))
@end example

In this example, accessing
@code{http://buildbot.example.org/static/foo/bar.html} will produce
the contents of @code{BASEDIR/buildbot_html/foo/bar.html}.

@end table

@node Buildbot Web Resources, Special-Purpose Web Resources, Static Resources, HTML Waterfall
@subsubsection Buildbot Web Resources

Of course, the interesting thing about having a webserver inside the
buildmaster is its ability to deliver pages that contain build status
information. The resources described here all provide various views of
the buildbot's status.

@table @code

@item TimelineOfEverything

This provides a chronologically-oriented display of the activity of
all builders. It is the same display used by the Waterfall display.

@item SlaveStatusTimeline

This provides a chronological display of configuration and operational
events: master startup/shutdown, slave connect/disconnect, and
config-file changes. When a config-file reload is abandoned because of
an error in the config file, the error is displayed on this page.

This page does not show any builds.


@item LastBuild

This shows one box per Builder, showing the results of the most recent
complete build. It does not show the individual steps, or the current
status.

There are some options to limit the boxes displayed:

@itemize @bullet
@item
branches: only show builds of specific branches
@item only show specific builders
@end itemize


@item LastBuildImage

This returns a PNG image that describes the results of the most recent
build, which can be referenced in an IMG tag by other pages, perhaps
from a completely different site. Use it as you would a webcounter.

@item OneLinePerBuild(numbuilds=20)

This produces a simple text page which contains a one-line summary for
each of the last N builds (where N defaults to 20). The most recent
build is at the bottom. Each line of text is colored in red, orange,
or green, as appropriate. at footnote{Apparently this is the same way
http://buildd.debian.org displays build status}


@end table


@node Special-Purpose Web Resources,  , Buildbot Web Resources, HTML Waterfall
@subsubsection Special-Purpose Web Resources

The last batch of web-status resources are intended for use by other
programs, rather than humans.

@table @code

@item RSS

This provides an RSS feed of recent builds.

@item XMLRPC

This runs an XML-RPC server which can be used to query status
information about various builds.

@end table