[Buildbot-devel] SVN repository changes notification by mail

[Julien Gilli]

Hello,

Brian Warner wrote:

>>The svn_buildbot.py post-commit hook script does its job pretty well. It 
>>works for us when the SVN repository can connect to the build master 
>>with a non-standard destination port.
>>    
>>
>
>What do you mean by "non-standard" in this context? portnum != 80?
>
>  
>
Yes, and more generally, a port that is not filtered by corporate firewalls.

>>For example, we can't execute a post-commit script that make an external
>>connection with a non-standard destination port. Moreover, we don't want to
>>make our buildmaster public, and we can't redirect connections made on port
>>9899 on our router to our buildmaster.
>>    
>>
>
>So it sounds like your buildmaster and buildslaves are behind a firewall, and
>your SVN server is behind a *different* firewall. (Most of the
>private-buildmaster installations I've seen so far had both behind the same
>firewall). The SVN server's firewall must allow at least SVN connections in,
>otherwise the buildslaves wouldn't be able to get at the source code.
>  
>
We access our SVN server through HTTP, which is one of the few protocols 
that are almost always accepted in.

>Let me throw a couple of ideas out and see if any of them might be useful:
>
> 1: you can have the buildmaster listen on any port you like. (except that
>    low-numbered ports might be a problem). If you can find a port that the
>    SVN server machine *is* allowed to connect to, you can have the
>    buildmaster listen on that one instead. There's nothing magic about 9899,
>    most of the port numbers in the sample config files are just ones I made
>    up in the hope that they were unique enough that they could be used
>    unmodified.
>  
>
It's true, but our buildmaster being behind a firewall, on a private 
network, we would have to redirect the stream. We can't do that because 
it would take a very long time before we get in touch with the right 
admin that could do that on the right firewall.

> 2: if you set up a port-forwarding proxy somewhere, then the port number that
>    svn_buildbot.py connects to doesn't need to have any relationship to the
>    port that the buildmaster is listening on. You could even have
>    svn_buildbot.py connect to port 80, then have a proxy on the buildmaster
>    that forwarded that back to port 9899, if the SVN server's firewall
>    allows connections to "standard" ports like that.
>  
>
Same thing as above.

> 3: you could use ssh to connect from your buildmaster host to the SVN
>    server, and then configure a reverse port forwarding to send a port from
>    the SVN server back to the buildmaster.. something like:
>      ssh -R7000:localhost:9899 svn.example.com
>    And then your svn_buildbot.py could use --bbhost=localhost --bbport=7000
>  
>
We already thought about that, but the tunnel could go down sometimes.

> 4: In the CVS world, the 'FreshCVS' daemon works through a connection in
>    the direction you want: the buildmaster connects to the FreshCVS daemon,
>    on the CVS server, and then the CVS post-commit hook script talks to the
>    daemon through a unix-domain socket. I don't currently know of a similar
>    daemon-based approach for the SVN world, but it wouldn't be too crazy to
>    go and write one.
>  
>
Correct me if i'm wrong, but this would mean having an additional daemon 
running on our SVN server, and listening to another port (standard or 
not). That would still place another dependency on the admins willing to 
give access to this port or not.

>>So, the only solution i can come up with is parsing the notification 
>>emails sent by an existing post-commit hook script, and then crafting 
>>something similar as the message sent by the svn_buildbot.py to the 
>>buildmaster.
>>    
>>
>
>That would work, although of course the machine that received and parsed the
>mail would need to have better access to the buildmaster than the SVN machine
>did.
>  
>
Actually, the software that parses the e-mails can be running on the 
build master.

>It's pretty common to have the buildmaster's account subscribed to the commit
>emails directly, and let the buildmaster do the parsing. There are several
>ChangeSources for doing exactly this, and all you have to do is write the
>mail-message parser. Take a look in buildbot/changes/mail.py for details. I
>think most of the mail-parsers in there are aimed at common CVS mail formats,
>but it's about time we added some standard SVN versions in there.
>
>  
>
I have written such a svn version.

>The commit-email approach involves two orthogonal issues. The first is how to
>parse the email, and that gets encapsulated into a single function that
>accepts an open file handle with the message, and returns a Change instance.
>  
>
I think i could adapt my current implementation to this scheme.

>The second is how to receive the email message in the first place, and the
>buildbot provides a "maildir poller" class that will watch a simple
>maildir-style inbox for new messages. (you could conceivable write a POP/IMAP
>poller to pull these messages down, and my 'petmail' project has some code
>you could grab to do this, but usually it's easier to get the email to arrive
>directly at the buildmaster host).
>  
>
Is it documented somewhere ?

>When I set up a maildir-based ChangeSource, and if the host's MTA provides
>direct maildir delivery (qmail for sure, and I think exim and postfix offer
>it too), then I just make a ~/maildir-buildbot/ and point my
>buildbot.changes.mail.FooParserMaildirSource at it. If the MTA doesn't, I set
>up a .forward that pipes the message into safecat and let safecat deliver it
>to the maildir.
>  
>
Ok, what i've come up so far is a twisted daemon (TimerService) that 
polls a POP3 acount every n seconds. If it has new e-mail, it retreives 
them, parses them and create a "SvnNotificationEmail" instance from each 
of them. Then it pushes the changes to the master in a way similar to 
the svn_buildbot.py.

So far, it seems to be working fine. I'll do an IMAP implementation too. 
Do you think that this approach makes sense ? If not ,why ?

I'll dig more into your proposed idea (a SVN's email parser and a 
Maildir watcher with a postfix setup) and, if i can, will send you any 
patch i could come up with.

Thank you very much for your kind help, once more :-).

All the best,

-- 
Julien Gilli