[Buildbot-devel] SVN repository changes notification by mail

[Brian Warner]

> The svn_buildbot.py post-commit hook script does its job pretty well. It 
> works for us when the SVN repository can connect to the build master 
> with a non-standard destination port.

What do you mean by "non-standard" in this context? portnum != 80?

> For example, we can't execute a post-commit script that make an external
> connection with a non-standard destination port. Moreover, we don't want to
> make our buildmaster public, and we can't redirect connections made on port
> 9899 on our router to our buildmaster.

So it sounds like your buildmaster and buildslaves are behind a firewall, and
your SVN server is behind a *different* firewall. (Most of the
private-buildmaster installations I've seen so far had both behind the same
firewall). The SVN server's firewall must allow at least SVN connections in,
otherwise the buildslaves wouldn't be able to get at the source code.

Let me throw a couple of ideas out and see if any of them might be useful:

 1: you can have the buildmaster listen on any port you like. (except that
    low-numbered ports might be a problem). If you can find a port that the
    SVN server machine *is* allowed to connect to, you can have the
    buildmaster listen on that one instead. There's nothing magic about 9899,
    most of the port numbers in the sample config files are just ones I made
    up in the hope that they were unique enough that they could be used
    unmodified.

 2: if you set up a port-forwarding proxy somewhere, then the port number that
    svn_buildbot.py connects to doesn't need to have any relationship to the
    port that the buildmaster is listening on. You could even have
    svn_buildbot.py connect to port 80, then have a proxy on the buildmaster
    that forwarded that back to port 9899, if the SVN server's firewall
    allows connections to "standard" ports like that.

 3: you could use ssh to connect from your buildmaster host to the SVN
    server, and then configure a reverse port forwarding to send a port from
    the SVN server back to the buildmaster.. something like:
      ssh -R7000:localhost:9899 svn.example.com
    And then your svn_buildbot.py could use --bbhost=localhost --bbport=7000

 4: In the CVS world, the 'FreshCVS' daemon works through a connection in
    the direction you want: the buildmaster connects to the FreshCVS daemon,
    on the CVS server, and then the CVS post-commit hook script talks to the
    daemon through a unix-domain socket. I don't currently know of a similar
    daemon-based approach for the SVN world, but it wouldn't be too crazy to
    go and write one.

> So, the only solution i can come up with is parsing the notification 
> emails sent by an existing post-commit hook script, and then crafting 
> something similar as the message sent by the svn_buildbot.py to the 
> buildmaster.

That would work, although of course the machine that received and parsed the
mail would need to have better access to the buildmaster than the SVN machine
did.

It's pretty common to have the buildmaster's account subscribed to the commit
emails directly, and let the buildmaster do the parsing. There are several
ChangeSources for doing exactly this, and all you have to do is write the
mail-message parser. Take a look in buildbot/changes/mail.py for details. I
think most of the mail-parsers in there are aimed at common CVS mail formats,
but it's about time we added some standard SVN versions in there.

The commit-email approach involves two orthogonal issues. The first is how to
parse the email, and that gets encapsulated into a single function that
accepts an open file handle with the message, and returns a Change instance.
The second is how to receive the email message in the first place, and the
buildbot provides a "maildir poller" class that will watch a simple
maildir-style inbox for new messages. (you could conceivable write a POP/IMAP
poller to pull these messages down, and my 'petmail' project has some code
you could grab to do this, but usually it's easier to get the email to arrive
directly at the buildmaster host).

FYI, maildirs come from the "qmail" MTA, and are just a trio of sibling
directories: new/, cur/, and tmp/, and when a new message arrives, you write
it into a uniquely-named file in tmp/, and when you're all done you do an
atomic 'mv tmp/1223344 new/'. The 'safecat' utility will accept a message on
stdin and write it into a maildir correctly.

When I set up a maildir-based ChangeSource, and if the host's MTA provides
direct maildir delivery (qmail for sure, and I think exim and postfix offer
it too), then I just make a ~/maildir-buildbot/ and point my
buildbot.changes.mail.FooParserMaildirSource at it. If the MTA doesn't, I set
up a .forward that pipes the message into safecat and let safecat deliver it
to the maildir.


hope that helps,
 -Brian