[Buildbot-devel] embedding buildbot?

[Brian Warner]

> I am running a twisted application already, which provides the core of 
> some specific functionality I need. I want to run buildbot but ideally 
> i'd want to have a single twisted daemon, and run a master as part of my 
> main app.
> 
> Can this be done easily? How would I start a master from another twisted 
> application ( I use a .tac )? Can I integrate the waterfall status 
> output / interaction with my own nevow web interface?

Since we moved from .tap to .tac, embedding is pretty easy. The buildmaster
is just a (twisted.application.service.)MultiService, so you can put it as
the child of your existing Application instance. The buildbot.tac that modern
buildbot versions use just creates the two instances and associates them
like:

 application = service.Application('buildmaster')
 BuildMaster(basedir, configfile).setServiceParent(application)

Your program should create the BuildMaster instance the same way, but then
setServiceParent() it to your own Application (or other MultiService)
instead. Everything else the buildmaster does (listening ports, etc) is done
as a sub-child Service or MultiService, and stopService causes all persistent
data to be saved. The buildmaster will read its config file as part of
startService().

The only un-neighborly thing that the BuildMaster might do in this context
would be installing a SIGHUP handler. You may wish to comment that out (in
buildbot.master.BuildMaster.startService) if it interferes with your existing
application. Let me know if this is a problem and I'll add a flag so you can
disable the SIGHUP handler without editing buildbot code.

As for the Waterfall page, it isn't easy right now, because we're starting a
new web server inside the buildmaster. There is a html.StatusResource that
ought to be useable in other web servers running in the same application, but
1) it isn't particularly easy to get a reference to, and 2) it probably has
some absolute-URL issues that would cause problems if the Resource lives
anywhere but the root of the URL tree. I hope to fix these soon (quite
possibly for 0.7.0, because the new Scheduler approach requires some new HTML
status pages). The goal will be to let the buildbot config file build the
Resource tree however you like: html.Waterfall will produce the old
waterfall-only site, html.WebStatus will produce a site where the Waterfall
page is one of a couple of options (and a top-level "welcome to this
buildbot" page has links to them all), and html.WebStatus(root=myresource)
will let you start a web server with whatever arrangement of status resources
you want, maybe something like:

 w = html.WaterfallResource()
 c = html.ChangeDispositionResource()
 r = resource.Resource()
 r.addChild("waterfall", w)
 r.addChild("allchanges", w)
 c['status'].append(html.WebStatus(root=r))

To support that last mode, it will need to be possible to create Resource
instances for the various buildbot status pages; you should be able to use
these instances from your enclosing application to embed buildbot status HTML
in your existing web site. I haven't figured out the details yet: each
Resource needs access to the IStatus reference, so somehow they need to be
informed about it, and I also want to maintain the property that re-reading
the config file (without anything in it changing) will not interrupt the
running services, so we'll need to be able to compare the old and new
resource trees to tell if anything changed.

Anyway, that's the plan. It might also be interesting to switch to nevow for
buildbot HTML generation, I'm not sure yet. The upcoming Nevow release will
require twisted-2.0, so I'm hesitant to jump in feet first, but maybe in a
couple of months that wouldn't be so bad of a requirement.

cheers,
 -Brian