[Buildbot-devel] Newbie Seeking Best Practices

[Charles Lepple]

On Dec 10, 2011, at 10:03 AM, Tom Browder wrote:

> 1. Are there any advantages to installing the buildbot and
> buildbot-slave programs in a virtual environment if they can be
> installed globally by the root user?

Two reasons spring to mind: multiple *simultaneously installed* virtual Python environments, and making it easier to track newer development versions of Buildbot.

Since I don't think you would need different versions of Python installed to test brlcad (I admit I haven't looked at the brlcad code in many years), the first reason probably isn't very compelling.

In the second case, depending on your uptime requirements, you could have one virtual environment with a released version of Buildbot, and a second with code pulled from Git. This would be handy if you were adding a feature or tracking down a bug in the Buildbot code itself.

I was a little skeptical of the utility of Python virtual environments, but it's nice to be able to use a newer version of Buildbot than whatever your OS distribution has - and it's a lot easier to archive that virtual environment if you need something to fall back on after a botched upgrade.

> 2. Are there any problems setting up and running the buildmaster as a
> multi-master with sqlite as a database program?

I don't have direct experience with multi-master mode, but I have a feeling that if you need multiple masters, you probably want a network-accessible database process to make it easier to scale up.

Also note that if you are using multiple masters just to keep the width of the waterfall display manageable, there are category attributes for builders, and you can pass parameters to the waterfall to only show a single category. I help manage a Buildbot installation which probably has over fifty builders, and at any given time, only a handful of those builders are active. Since this setup is actually implemented in parallel on many independent masters, we just have a top-level HTML dashboard that links to the active category on each master.

> 3. Has anyone used the Django buildbot setup discussed here
> (http://jacobian.org/writing/buildbot/ci-is-hard/)?  If so, do you
> recommend it for a system that may grow to more than a few slaves?

Haven't tried that specifically, but my gut feeling is that you would need more than a few slaves and dynamic configuration changes to make that worth the effort. Some of the techniques are useful on their own - with the Network UPS Tools buildbot configuration[*], we make use of Python loops and dictionaries to mechanize the construction of the configuration (adding certain steps only if necessary for a given builder, etc). To make it easier to keep track of these configuration changes, the configuration is stored in a separate source control repository separate from the source code itself. I haven't been brave enough to automatically reload buildbot on pushing a change to the configuration.

[*] http://buildbot.networkupstools.org/public/nut/waterfall

http://boxster.ghz.cc/projects/misc/browser/trunk/svnautobuilder/buildbot-master-ocelot/master.cfg#L146 (no longer matches the running configuration, which has been moved from SVN to Git)

On the other end, if you want to manage the build products from Buildbot, I can definitely recommend using Django. On another decently-sized Buildbot installation, we added a database record for each packaged build that included attributes such as the version, branch name, target OS, language runtime requirements, path of the installer on disk, and any special properties entered on the "force build" form. We then embedded an update URL into the application, which would query Django for the latest build on the same branch that the application was built on. The database also allowed the QA team to mark an intermediate build as "worth keeping" when we went through and garbage-collected old buggy builds (after they were appropriately logged in the bug tracker). Buildbot does have facilities for garbage-collecting old build logs, but since it's up to each site to decide where they want to store build products, it can be a little different if you are used to other CI systems which store logs and build products adjacent to each other.

-- 
Charles Lepple
clepple at gmail