[Buildbot-devel] Buildbot 1.0: The Shimmering Vision

[Brian Warner]

(I apologize for the brief response and for the top-posting.. I'm on  
vacation this week and am reading email through a cellphone..)

This all sounds great! I really like the of a clear, attainable goal  
for a 1.0 release.

I'd be happy with a 1.0 that just had the DB-backed status storage and  
"stateless" job scheduler, since they will enable so much more. I'll  
back away from some of the other, more grandiose milestones I used to  
have in mind ("Problem" tracking, IM status delivery, test-case  
parsing, and some others listed on the Trac "roadmap" page).

Both of those projects, as well as any other features that we want to  
put into this release, are going to require a non-zero amount of work,  
and my day job is keeping me too busy to let me write a lot of  
Buildbot code these days. But I can give direction and offer support.  
I have some specific ideas about how to proceed on both the status-db  
and the job-management.. if we can put together a "status-db strike  
force" then I think the coding shouldn't actually be too hard.

Status-db: as we discussed at the user's group meeting as well as on  
the mailing list, the basic question is sqlite (probably in the form  
of Axiom or some other OOPish front-end) or a proper remote SQL scheme  
via twisted.enterprise . The latter probably involves more manual work  
(and more thought about upgrades, etc) but also represents less work  
to switch to other backends in the future and to achieve better  
scaling for large systems. Log files will obviously be kept outside  
the DB, but that makes remote databases trickier to manage. My  
immediate inclination is to go with a local sqlite approach, with  
logfiles sitting next to the db file, but the criticism that this will  
not scale is a valid one.

My expectation is that we can leave the existing IStatus interfaces  
alone and just change the internal implementation. (that's the whole  
reason for IStatus anyways). We'd need to decide if we want to change  
the method definitions to return Deferreds (instead of their current  
synchronous form), though.. sync is fine for sqlite, but not for  
postgres.

For job control, I'm thinking of a separate local database that  
remembers which builds are currently running. The Schedulers would be  
changed to keep their state in this jobdb/buildsdb, and would re- 
examine it at boot time (so restarting the buildmaster won't forget  
about the pending build). The jobdb will track running builds, so a  
restart won't interrupt builds either. The buildslaves already queue  
their command status messages and don't retire them until the master  
acks them.. we just need to enhance this to re-send messages on  
reconnect instead of killing the current command.

In short, I think both of these goals are feasible 1.0 material, and  
if we can organize some people with SQL experience to write the code,  
we should be able to get this stuff done within a couple months.

Many thanks to Dustin for helping us keep up the 1.0 momentum.. let's  
do it!

cheers,
  -Brian



On Aug 12, 2008, at 2:29 PM, "Dustin J. Mitchell" <dustin at zmanda.com>  
wrote:

> On Tue, Aug 12, 2008 at 2:10 PM, Ben Hearsum <bhearsum at mozilla.com>  
> wrote:
>> Stateless client/server protocol would let us restart the master  
>> without
>> killing slave jobs, right? I totally freaking support that. +++++++
>
> /me counts plusses..
>
> Yes, and it would also probably allow non-python buildslaves.  I'm not
> totally sold on the idea -- Buildbot's existing use of PB is *really*
> solid and effective, and I had hoped to keep it.  I wonder if we could
> do both?  Eep.
>
> Dustin
>
> -- 
> Storage Software Engineer
> http://www.zmanda.com
>
> --- 
> ----------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel