[Buildbot-devel] New, more efficient web stuff

[Dustin J. Mitchell]

I'm working on some major changes to both the web subsystem and to the
storage backend, because in my configuration the buildbot web server
is horrendously slow, due mostly to inefficient storage of historical
state.

Changes will be available at
  http://darcs.r.igoro.us/buildbot/speedup
although they're still under development at the moment, and not checked in.

A brief overview of the changes I'm making:

* status is just the current state -- what's in the configuration, and
what's going on right now.  It also generates events for subscribers.
* storage keeps the historical record, in a fairly straightforward
object-persistence architecture with pluggable backends.
* the web is no longer a status subscriber, since it's not in the
business of receiving events, but of displaying the historical record
and the current status.  Instead, it's configured as an
"extra_service" -- basically, a Twisted service that gets (re)started
by the buildmaster, and does its own thing from there on out.
* HTML generation is strictly the domain of the web service -- objects
themselves no longer generate HTML (>>ahem<<, Changes.asHTML), nor is
the buildmaster in the business of generating URLs.
* Twisted interfaces and adapters are used heaviliy to convert various
business objects (status or storage) into web pages (IPage), snippets
of HTML (ISnippet) for inclusion in a larger page, or URLs (IURL) for
links.

At this point, I have a "dummy" backend storage which just provides a
fixed set of builders, builds, slaves, etc., and I'm working on
rewriting the web interface against that.  Once that much is done,
I'll  record a patch and let you all know.  The next steps will be:
* write the "write" side of the storage interfaces -- how data gets
*into* storage -- an implement it
* write a real storage implementation or two -- one or more of a
pickle-based storage, SQLite-based, or using Twisted Enterprise, the
DB abstraction layer
* find a way to re-integrate the "control" hierarchy

Comments are welcome!

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com