[Buildbot-devel] source change for sf.net hosted CVS

[Brian Warner]

Stefan Kost <ensonic at hora-obscura.de> writes:

> Hi Brian,
>
> the mail idea wont work well for sf.net hosted project as there is a sync
> delay between the developer cvs and the read only cvs. I was looking at
>   http://buildbot.cvs.sourceforge.net/buildbot/buildbot/contrib/

Yeah, I ran into that problem myself with the metabuildbot[1]. For a while I
was running it against developer CVS (by giving the buildslave an SSH key),
and then I switched to an even more complicated scheme in which the email
triggers an external privileged process (with an SSH key) that uses tailor to
copy the changes from sf.net CVS into some other VC system (I think it was
Arch at the time), then use a commit hook on that other system to notify the
buildmaster.

These days I'm developing buildbot in Darcs, and use a commit hook there (on
a repository that runs on my own machine). The commit hook has two jobs:
notify the buildmaster, and push the changes into sf.net CVS. The buildslaves
then pull directly from a read-only Darcs repository, bypassing sf.net
altogether. The account where these commit hooks run still needs a
suitably-capable SSH key to let it write to sf.net, of course.

That sync delay is a real hassle :).

> We already use the sf.net syncmail to send commit mails. I see two more
> questions with this approach

> * the bot needs a mail address, needs to be subscribed to the mailing list
>   and fetch mails regularily (who will cleanup his inbox)

Yes, I recommend creating a new account for the buildmaster, which would make
it easier to give it a separate email address. (alternatively, if you use
qmail or postfix, you can use an extension address of your own address, and
redirect those messages to a different place where the buildmaster can see
them.. I might use warner-buildbotchanges at example.org and then have a
~/.qmail-buildbotchanges to redirect that mail).

The buildmaster knows how to read email out of a maildir (moving it from the
'new' subdirectory to the 'cur' subdirectory as it does so). Then you can
just have a cron job that deletes everything in the 'cur' directory nightly
to do cleanup.

Alternatively, you could have a .forward or .qmail or whatever that pipes the
message into a command, and that command could parse the message and contact
the buildmaster. This approach wouldn't require that the email account be on
the same machine as the buildmaster. I'm not sure how fully supported this
approach is, though.. I know I intended it to be a possibility, but I forget
if I actually finished the code to do it.

Unfortunately, we don't currently have code to, say, poll a POP or IMAP
mailbox for these messages. I have another project (petmail.lothar.com) which
has such a poller, so if there's a strong demand for it I could probably port
the code over into buildbot.

> * we have multiple modules, will the parse understand that and trigger the
>   right build?

That part is trickier. The maildir-watching ChangeSource looks for messages
that appear in a given maildir and turns each one into a Change object (or
decides to ignore it), and each of those Change objects contains the usual
fields (author, date, comments, branchname, list of files changed). For
convenience, most ChangeSources have a 'prefix' setting that lets you specify
what counts as the "top" of this particular project's tree: any files that
live outside this tree are ignored, any files that live inside this tree have
this prefix removed.

So depending upon how your CVS modules work, the prefix might let you
construct a ChangeSource that only produces Changes for the module you are
interested in. Or, it might not. You can subclass the ChangeSource to modify
the way that filenames are handled, to add more specific logic that knows how
your project uses modules. Buildbot has a fairly simplistic notion[2] of how
trees and repositories and Changes work, so usually the job is to present
your own tree in a simplified form that matches it.

hope that helps,
 -Brian



[1]: and yes, I'm still planning on making the metabuildbot (which runs the
     buildbot's own unit test suite) visible.. I'm setting up a new server to
     host it and the machine isn't quite ready yet. My goal is to get that
     done within a month.

[2]: http://buildbot.sourceforge.net/manual-0.7.5.html#Version-Control-Systems