[Buildbot-devel] Different source types?

[Brian Warner]

> Since we already have bonsai set up to get email notifications for CVS 
> changes, my admin asked if we could hook into that instead of running a 
> separate FreshCVS daemon.  This means I have to write my own kind of 
> mail-based "change source" that connects to the build master.


Correct. The fundamental building block is a ChangeSource object (i.e.
anything that implements the IChangeSource "Interface"/"protocol"). This is a
piece of code that lives inside the buildmaster and does anything it likes to
find out that something has changed in your source tree. When it detects a
change has occurred, it creates a Change object to describe the addition /
deletion / modification, and submits that Change object to the ChangeMaster.
The object is just a data-holder (like a struct) and has no particular
behavior attached to it. I use a separate object for Changes out of
convenience: they are passed around a lot after being given to the
ChangeMaster (for example the summaries retrieved by following the links in
the "Changes" column of the waterfall display are built directly from the
Change object).

There are three current built-in approaches to getting these Change objects
into the buildmaster.

 1: set up a FreshCVS listener (by creating a FreshCVSSource). This will
    connect to a FreshCVS daemon (and attempt to reconnect if the connection
    is ever lost), and subscribe to hear about all changes. This is the most
    direct approach, but requires that you have CVSToys installed on the
    machine that hosts your CVS repository (and is hampered at the moment by
    some Twisted version-skew issues between the latest releases of CVSToys,
    Buildbot, and Twisted itself).

 2: get changes by email. There are two orthogonal issues:
    2a: how do you receive mail?
    2b: how should we parse the each mail message?

    For 2a, there is built-in support for watching qmail-style Maildirs. This
    is a simple protocol that involves a trio of directories. New messages
    are created in a temporary directory (BASE/tmp/) and then, when complete,
    moved atomically into the incoming directory (BASE/new/). When the
    message has been processed (i.e. when the receiving application commits
    to accepting the message), it is moved to the old directory (BASE/cur/, I
    think, maybe BASE/old/).

    You can create a MaildirSource object (actually an FCMaildirSource or a
    SyncmailMaildirSource), give it the base directory as an argument, and
    the buildbot code will do the rest. It will poll (or use linux's DNotify
    scheme) the maildir every once in a while to look for new messages.

    Note that you do not have to run qmail to use maildirs. It is trivial to
    create them by hand (or use the 'maildirmake' utility distributed with
    qmail): 'mkdir BASE BASE/cur BASE/new BASE/tmp'. Len Budney has written a
    small utility called 'safecat' which safely takes a message on stdin and
    drops a file into the maildir (doing all the necessary error-checking
    that is the whole point of using maildirs in the first place). You can
    put a pipe-to-safecat command in your non-qmail .forward file and be done
    with it.

    Safecat is available from:
      http://www.pobox.com/~lbudney/linux/software/safecat.html

    For 2b, each mail source has a parsing function which is used to scan the
    message body and generate a set of Change objects. Buildbot is
    distributed with two such functions: parseFreshCVSMail() parses the
    messages created by the MailNotification target built-in to FreshCVS,
    while parseSyncmail() handles the mail format generated by syncmail (a
    commonly-used CVS change-message distribution tool).

    Take the dot-product of the possibilities for 2a and 2b, and you've got a
    set (well, a pair) of change source classes which can be used in your
    buildmaster configuration:

     FCMaildirSource(basedir): receive FreshCVS.MailNotification messages in
                               a maildir located at 'basedir'

     SyncmailMaildirSource(basedir): receive Syncmail messages in a maildir
                                     located at 'basedir'

    If you receive CVS mail messages (or Subversion messages or whatever) in
    a different format than MailNotification or Syncmail, you can write a
    parser function based upon the code in buildbot/changes/mail.py and then
    create a new MaildirSource subclass. If you receive mail messages through
    something other than a maildir, you can write a new class that behaves
    the same way as MaildirSource does (i.e., calls the .messageReceived
    method with a filename from which the message can be read).

 3: push changes into the buildmaster via RPC (using Twisted's native RPC
    system named Perspective Broker, abbreviated "PB")

    Inside the buildmaster, the various change sources all deliver the Change
    objects to a piece of code called the ChangeMaster. The ChangeMaster also
    listens on a TCP port (actually it shares the same TCP port with the
    BotMaster, the debug port, and the remote status client port), and
    external programs can inject Change objects through this port.

    This port is enabled by default. (in buildbot-0.3.5 you could disable it
    when you call makeApp, via the 'services' argument) (in the upcoming
    release, there is an unimplemented 'services' configuration variable that
    would, if it were implemented, let you control whether the port is
    enabled or not. This feature is under development, it will probably
    acquire some password-protection and source-IP-address restrictions
    within a release or two). It listens on the same TCP port as everything
    else.

    There is a sample client in contrib/fakechange.py, which I use to
    initiate new builds during local testing. You can adapt this program to,
    for example, collect text from stdin, parse the message to extract a set
    of (username, filenames, change comments), assemble them into a
    dictionary, and fire off the remote method call to inject the change.

    The contrib/svn_buildbot.py script also uses this approach. This script
    is meant to be run from the Subversion repository's post-commit hook, and
    connects to the buildmaster's change port to inject the Change object.


If none of these approaches suffice, you can always write a new kind of
ChangeSource. If you already have a cvs-commits mailing list, the easiest
approach is probably to watch a maildir with SyncmailMaildirSource and use
.forward or .procmail to 'safecat' the messages into the maildir. If you can
install CVSToys on your repository machine, then use FreshCVSSource (the
latency will be much lower.. watching for email involves both the email
propagation delay and the polling interval).

A hint for all those eager Buildbot developers out there :) : a nice addition
to contrib/ would be a short script that parses, say, Syncmail messages from
stdin and sends the resulting Change objects over PB to the change port. All
the code is there already (parseSyncmail and config/fakechange.py), it just
needs to be shuffled into a new tool. This new program could be called
directly by .forward/.procmail and let non-qmail non-CVSToys installations
bypass the maildir/safecat part.


good luck,
 -Brian