[Buildbot-devel] Going beyond the tutorial; ideas for simplifying larger configurations

[John Carr]

Hi Dan,

I've ended up with a couple of different hammers for this particular problem.

I was faced with lots of projects under a single master and there was a large amount of copy and paste. There were lots of stakeholders who might want to update the configuration but they don't (and can't) have access. So I wrote this:

https://github.com/Jc2k/buildbot_travis

It 'works for me' but I'd be reluctant to tell other people to use it for more than ideas at this stage. There is a central YAML file that lists project names and where their code is. Everything else is detailed in a .travis.yml file in the project's code repository. That includes how to build/test, what branches, when to send emails, etc. Each command gets its own BuildStep. And because it's a simplified case of buildbot I can remove a lot of stuff from the web UI - scroll down that link to see where I ended up.

I've ended up adapting and reusing one of the functions from travis in a couple of projects. Basically, given a repository, project name and list of steps you could have a function that sets up a BuildFactory, BuilderConfig, Poller, Scheduler, ChangeFilter and all the other boring stuff. So I have started:

https://github.com/Jc2k/buildbot_patterns

There is barely any code at the moment (certainly nothing useful) and the README is now out of date for what I'm actually writing which is something like this:

b = BuildTemplate().when("on_commit").when("nightly", hour=5, minutes=30)
b = b.how(steps=["virtualenv .", "./bin/python setup.py develop"])
b = b.where("slave1", "slave2", "slave3")

b.what(name="patterns", repository="git://github.com/Jc2k/buildbot_patterns").configure(c)
b.what(name="travis", repository="svn//github.com/Jc2k/buildbot_travis").configure(c)

First of all I define a reusable bit (all my projects will have a nightly and an on-commit bit and are tested just by installing them) and then i implement that twice - once for a git repo and once for an svn repo. Until you call configure on the Buildbot configuration object BuildTemplate hasn't done anything but collect the settings it needs. For this example, i'd expect configure to create 1 BuilderConfig but 2 schedulers. But I also imagine use cases where you define multiple groups of "where" - so it can create a windows builder, and an osx builder and so on.

For more advanced users, the list of steps can take BuildStep objects as well as strings. For the super simple use cases I see single-function wrappers like in the README:

build_on_commit(c,
    name = "buildbot",
    repository = "git://github.com/buildbot/buildbot",
    steps = [
        "virtualenv --no-site-packages .",
        "bin/python setup.py develop",
        "bin/pip install mock",
        "bin/trial buildbot",
        ]
    slaves = ['windows-1', 'windows-2']
    )

We use buildout heavily so I deploy our buildbots with buildout. When I want to hack on one i "svn co && python bootstrap.py && ./bin/buildout && ./bin/cluster start" and i have a working buildbot. The buildout recipe I use gives me ".d" directories allowing me to easily break up complicated configs. Configs are tagged as code in a repository and we run buildout to deploy them then reconfig or restart depending on the change. However we are also looking at a web UI to poke new projects directly into buildbot_travis without restarting. It basically works, but there is some glitching in WebStatus until a full restart :-/

John


On 21 Aug 2012, at 07:59, Dan Kegel <dank at kegel.com> wrote:

> The buildbot manual is good - better than, say, the cmake manual -
> but it's still hard for beginners when faced with largish buildbot
> configurations to make sense of them.
> The tutorial helps, but it stops at a very simple configuration.
> 
> I just joined a group that maintains eight or so related
> projects (one core library, and seven apps that use it).
> They use one buildbot for each of the projects.
> Over time, evidently the configuration got unruly,
> so they wrote a central configuration library that did the
> bulk of the work for master.cfg.
> Adding a new project to the central config is
> difficult because that central configuration ilbrary
> needs changes in about eight different areas to add
> one new project.
> Also, there was no simple script that automated
> creating/installing the many masters and slaves.
> (As I whittle away at these problems, I get the feeling that
> I could take what I'm learning and do a larger example
> for the tutorial.  No promises, though.)
> 
> Before I dive in and turn our central config library inside out,
> has anyone got advice about structuring medium-sized
> buildbot configurations?  I've been looking at the Chromium buildbot
> configuration for ideas
> http://www.chromium.org/developers/testing/chromium-build-infrastructure
> For instance, http://lackingrhoticity.blogspot.com/2011/08/fixing-trouble-with-buildbot.html
> describes one of their ideas.
> Any other good sources?
> - Dan
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20120821/7ebe259f/attachment.html>