[Buildbot-devel] Multi-distro Buildbot success story

[John Morris]

The dust hasn't completely settled, but I'm excited to show off the 
multi-distro, multi-arch Buildbot we use to test contributions to the 
Machinekit open-source project (http://www.machinekit.io).  I thought 
our instance might be interesting in this forum since it's publicly 
viewable and is perhaps a fairly complex example.  See it here:

http://buildbot.dovetail-automata.com/

And the configuration (still a mess):

https://github.com/zultron/machinekit-buildbot

The Buildbot currently builds for four distros, Wheezy, EL6, and Fedora 
19+20; Jessie will be added soon.  It builds 32- and 64- bit x86 
architectures, with ARM on the way.  The unit test build slaves also run 
one of two real-time kernels, with a third to be added.

The project is Machinekit, a machine controller based on LinuxCNC, 
requiring a real-time thread environment.  We recently extended 
Machinekit to support not only the RTAI RT kernel that LinuxCNC 
originally supports, but also PREEMPT_RT, Xenomai userland and Xenomai 
kernel threads, and non-RT POSIX threads, corresponding to the '*-tst' 
builders on the Buildbot (RTAI currently absent).  The RTOS work 
included revamping the build and run-time systems to build loadable 
modules for all available RT environments in a single './configure && 
make', and run-time scripts detect the running RT environment and load 
the appropriate thread modules.

This so-called 'Universal Build' enables an interesting Buildbot 
workflow.  The 'builder0' slave is configured with 8 CPUs, and runs the 
build in a chroot environment for each of the eight distro+arch 
combinations ('*-bld' builders), building modules for four threads 
systems.  Each of these build results is then sent to two matching test 
slaves, one running RT_PREEMPT, the other Xenomai kernels, which then 
run unit tests to cover all thread systems.  The 'builder0' slave also 
builds packages for each distro+arch ('*-pkg' builders), as well as a 
large documentation tree for each distro (arch-independent '*-doc' 
builders).

This Buildbot is specially important to the Machinekit project, which 
uses the C4 process (http://rfc.zeromq.org/spec:22) to minimize hurdles 
of contribution submission and acceptance in hopes of scaling up 
community development.  By C4, patches only need to follow a few simple, 
formal rules to be accepted, including compiling and testing cleanly. 
With the variety of platforms requiring testing, Buildbot saves time and 
effort for developers.  Also because C4 standards don't require value 
judgements by reviewers, unlike standards of many other projects, 
comprehensive testing on all supported platforms is relatively more 
critical.

Please feel free to poke around the Buildbot web GUI and configuration. 
  Suggestions/criticisms are very welcome!  (Please CC: me on list replies.)

Thanks-

	John