[Buildbot-devel] Extending ShellCommand

[Brian Warner]

> Today I was doing some maintenance, and decided to upgrade BuildBot to 
> 0.5.0. First thing I noticed is that all Shell test cases fail. There 
> was a thread about this problem and you blamed Twisted 1.1.0 and said 
> that in the end all these failures are not critical. The funny thing is 
> that I am using Twisted 1.3.0. I am running up to date Debian Sarge. 
> Anyways I went ahead and upgraded BuildBot, disregarding unittest 
> failures and it seems to work fine so far.

Yeah, I really should have updated those tests before releasing 0.5.0 . Oops
:). Rest assured that I'll make sure all the tests pass before releasing the
next one (probably "0.6.0" because of all the restructuring).

>     Sidenote: I had to develop simplistic debian package, and I wrote 
> init.d script for running buildbot (though it is not included in deb 
> yet). If anyone is willing to adopt and polish it I can send sources. It 
> would be really nice to have official debian package.

YES! That would be wonderful! I've played a little bit with making .debs (I
have a simple one for my Petmail system) but I've never tried to do anything
with init.d scripts. Are you using the whole dh_* scheme?

I'd love to have your sources, and I will certainly integrate them into the
next release. The only issue in my mind is how to help people configure it:
it should be possible to run just a slave, just a master, both, multiple
masters, etc. But for starters, one which ran a single master and a single
slave (and had some /etc/default/buildbot settings to disable either) would
be plenty.

> ...
> to my build step, but it did not help either. I don't have any ideas how 
> to tackle this problem, without digging into guts of BuildBot, which is 
> probably not a good idea, since you are performing major refactorings. 
> Maybe I should wait until next release and hope that your code 
> restructuring would squash this bug.

Yes, take a look at the current CVS sources, because both sides of the wire
have changed significantly (although the actual wire protocol remains pretty
much the same).

On the slave side, the remote "SlaveCommand" instances can run zero or more
shell commands as they see fit (the CVS/SVN steps change their behavior
depending upon whether they see an updatable working directory, for example).
Each one is started by creating a new ShellCommand instance (with the command
to run), and a Deferred is returned to let you hook a callback to fire when
the command completes. Each command sends stdout/stderr status up to the
master, where it all gets added to the same LogFile. There's a flag to set
whether you want the ShellCommand to send the exit code (since you should
only send it once), so if your SlaveCommand uses multiple ShellCommands then
it should either send the 'RC' status itself or ask the last ShellCommand to
do it.

On the master side, each BuildStep can run zero or more RemoteCommands, each
of which corresponds to exactly one SlaveCommand. The most common type (in
fact the only one used) is the RemoteShellCommand, which creates a single
LogFile that collects everything the SlaveCommand sends back.

The status-reporting overhaul imposes the following scheme:

 Each Step gets a single box in the waterfall display. It may have multiple
 LogFiles. It has a single "color", a single "result" (SUCCESS, WARNINGS,
 FAILURE, SKIPPED), and a single "text" array of short strings. It also has a
 .text2 array of strings which are added to the overall build's .text .

 Each Build is made up of a series of Steps. The Build has an overall status,
 a "color", a .text array of strings, and (eventually) a list of LogFiles.


Most of the BuildSteps derive from ShellCommand because there is a lot of
reusable status-reporting code in it. If your command only runs one
RemoteCommand then it should do what you need. The actual SlaveCommand that
gets executed depends upon what the .cmd attribute contains when .start() is
invoked: any kind of RemoteCommand will do (e.g. the CVS step uses a
LoggedRemoteCommand("cvs", args) to invoke the CVS slavecommand).

> I would love to contribute unit test for the problem, but looks like 
> this task is not easy -- I was expecting some test already tests the 
> whole status update interaction, but I can't find any.

Yeah, I haven't come up with a good way to test that part. I think the code
in CVS probably has some coverage now, but I agree that there probably isn't
anything to specifically test that the status updates occur the way we think
they ought to.


> If you have any suggestions, I would love to help you with
> testing/reproducing on my end, just let me know. If you want I can send you
> my config file and all local extensions via private e-mail.

Sure thing, I'd be happy to take a look at the class you're using and I'll
see if I can suggest a way to make it work. Start by looking at current CVS,
since everything I suggest will probably be based upon that :).

> Thanks again for a nice program, I like it a lot! While I am at it, do 
> you have any plans on adding e-mail notification feature? That should be 
> relatively easy, and would add a lot of value.

Funny you should mention that, take a look at buildbot/status/mail.py . It
only sends mail to a static list of recipients, but it's a start. Let me know
if you can think of some handy features to add to it.

cheers,
 -Brian