[Buildbot-devel] SVN repository changes notification by mail

[Brian Warner]

> >What do you mean by "non-standard" in this context? portnum != 80?
> >
> Yes, and more generally, a port that is not filtered by corporate firewalls.

> We access our SVN server through HTTP, which is one of the few protocols 
> that are almost always accepted in.

My condolences. It's always frustrating to be stuck behind a firewall that
has been closed off so tightly as to prevent new, beneficial uses of the
internet. It's especially unfortunate when the network/IT admins don't
recognize engineering departments as their "customers" to whom they are
obligated to fix any problems that get in the way of their efforts.

> > 3: you could use ssh to connect from your buildmaster host to the SVN
> >
> We already thought about that, but the tunnel could go down sometimes.

Yeah, you'd have to construct the tunnel from a loop that keeps bringing it
back up each time it goes down. There are a bunch of projects floating around
to do this, written to address this specific need of an unnecessarily
restrictive firewall.

> > 4: In the CVS world, the 'FreshCVS' daemon works through a connection in
> >    the direction you want: the buildmaster connects to the FreshCVS daemon,
> >
> Correct me if i'm wrong, but this would mean having an additional daemon 
> running on our SVN server, and listening to another port (standard or 
> not). That would still place another dependency on the admins willing to 
> give access to this port or not.

Yes, you are correct.

> I have written such a svn version.

Excellent! Then you are most of the way there. If you can get the email to be
delivered directly to the buildmaster's account, then just plug your function
into a maildir-source (following the examples in buildbot/changes/mail.py).

> >The second is how to receive the email message in the first place, and the
> >buildbot provides a "maildir poller" class that will watch a simple
> >maildir-style inbox for new messages. (you could conceivable write a
> >POP/IMAP poller to pull these messages down, and my 'petmail' project has
> >some code you could grab to do this, but usually it's easier to get the
> >email to arrive directly at the buildmaster host).
> >  
> >
> Is it documented somewhere ?

The Petmail IMAP poller is on http://petmail.lothar.com , grab the latest
release and look in petmail/transport/imap.py . (be aware that polling an
IMAP server is stupidly complicated). The Petmail POP poller is incomplete.

The buildbot maildir poller has (brief) usage documentation in the user's
manual, section 5.1 . There isn't any documentation on building a new one,
per se, but the code in buildbot/changes/mail.py shouldn't be too hard to
follow. The basic rule is that all ChangeSources are instances of
twisted.application.service.Service (if they are 'leaves' of the Service
hierarchy) or of service.MultiService (if they need to have child Services of
their own, such as a service.TimerService, which may be useful for building
something that polls on a regular basis). They should implement all the
methods of the IChangeSource interface (in buildbot/interfaces.py).

> Ok, what i've come up so far is a twisted daemon (TimerService) that 
> polls a POP3 acount every n seconds. If it has new e-mail, it retreives 
> them, parses them and create a "SvnNotificationEmail" instance from each 
> of them. Then it pushes the changes to the master in a way similar to 
> the svn_buildbot.py.

Excellent. If you just create a Change instance instead of that
SvnNotificationEmail instance, and call self.changemaster.addChange(change)
with it, then you can move this code into the buildmaster directly, instead
of needing to run it in a separate process.

Or, if you find it easier to run it in a separate process, that works too.

> So far, it seems to be working fine. I'll do an IMAP implementation too. 
> Do you think that this approach makes sense ? If not ,why ?

Yes, I think it makes sense. I would hope that most installations do not have
to go through as many hoops as you have to, but when those are the
constraints you have to work within, a POP/IMAP poller may well be the only
way to get change notifications.

(incidentally, you might look at the patches on
http://sf.net/projects/buildbot/ in case there are any you find useful..
SF#1072845 has a parser for SVN's "commit-email.pl" script)

Also, somewhere on my TODO list is an entry to merge the "viewcvs poller"
that was contributed a while back. This would be a similar polling approach,
except instead of polling a mailbox being fed by commit emails, it would poll
a publically visible repository-browsing web page, looking to see if anything
had changed recently. For certain environments, such a ChangeSource could be
a lot easier to use than anything that involved creating or receiving email.

> I'll dig more into your proposed idea (a SVN's email parser and a 
> Maildir watcher with a postfix setup) and, if i can, will send you any 
> patch i could come up with.

Thanks!

cheers,
 -Brian