[Buildbot-devel] writing custom sources?

[Brian Warner]

> What's the easiest way to write a custom source? For instance I need to
> trigger a build when a logfile gets a new entry..

The best way is probably to copy the FreshCVSSource class from
buildbot/changes/freshcvs.py . I really need to get a better handle on the
way Twisted uses "Interfaces" and create an IChangeSource to define what
those classes are required to do, but basically it should be an object that
defines the following methods:

 def start(self):
   """Called when the buildmaster starts up. Can be used to establish a
   connection to a source control daemon, or start a timer to poll
   something."""

 def stop(self):
   """Called when the buildmaster shuts down. Connections should be torn
   down, timers should be canceled."""

 def describe(self):
   """Should return a string which describes this source and a brief summary
   of its status (like [OFFLINE]). Will be used in an HTML status page."""

Then, once the source is up and running, it should create Change objects
(buildbot/changes/changes.py) and feed them to:

  self.changemaster.addChange(c)

Each Change object represents one person making changes to one or more files
or directories, all with the same checkin comments.


I can think of two ways to watch a logfile for new entries. One involves
opening the file for reading, consuming everything that is available, then
waking up when the file becomes readable again (meaning someone else has
appended data to it): the "tail -f" approach. To do this you'd want to
create a class derived from twisted.internet.abstract.FileDescriptor, call
the startReading() method, and have a doRead() method that pulls off a line
and submits a Change.

The other way is a hack but might be easier: the "poll it" approach. In your
change source class, have start() create a timer with reactor.callLater.
When the timer fires, stat the logfile and remember its size. When the size
changes, read data from it and construct/submit the Change object. Make sure
to .cancel() the timer in the stop() method. It might also be useful
(eventually) to have the describe() text indicate how long until the next
poll takes place ("%d" % (self.timer.getTime() - time.time())).

In either case, feel free to create "fake" Change objects. They were
designed to accomodate CVS-style change notification, but if your build
process doesn't care about which files are being modified, and you aren't
trying to associate people with those changes, then you could just do:

 c = Change("logfile", [], "the logfile was changed", 0)
 self.changemaster.addChange(c)

to trigger the build timer. The waterfall page would indicate "logfile" as
the user in the Changes column, and no files would be listed.

Or, of course, if your logfile contains information resembling source code
control system change messages, you can parse them and create a more
accurate Change object.


Hope that helps.. please let me know how it goes, I am very hungry for
feedback about the internal interfaces, as I want to make sure they're
organized in such a way that it is easy to customize to do what you want.
The only use cases I've had so far are the ones I've come up with myself,
and that won't let me refactor it in the best way possible.

thanks,
 -Brian