[Buildbot-devel] CtuhuluBot (Was: Re: Fwd: [Buildbot] #641: New Default CSS)

[Marcus Lindblom]

On 2009-12-10 14:44, Almad wrote:
> On Thursday 10 of December 2009 10:55:34 Marcus Lindblom wrote:
>> But having a separate web server and BuildBot as a web service backend
>> is an interesting idea. But I think we have a slight collision course
>> with Brians DB work there, as that will bring the data into a SQL DB,
>> from which several processes/threads can work.
>
> I'm actually working on that. My approach is:
>
>   - django is used as main framework (this thing must first work inhouse, and
> we're djangonauts)
>   - configuration is stored in rdbms, as it's convenient to use
>   - build results are retrieved via status plugin and stored in mongodb
> database (via [1])
>
>   - mongodb is used as it should be faster then rdbms, it's easier to work with
> (build results are json anyway). Asynchronous writes are also problem,
> compatible python library shold be on it's way [2]

FWIW, I'm a Django fan too :) but the buildbot's core data flows are 
better suited for Twisted's async style. The web parts of twisted, OTOH, 
are a bit clunky, IMO, but that's what we're working on.

(Mongodb looks cool, I'll have to dig into that a bit.. :)

> Project is called CthulhuBot and is available at [3]. It's still WIP, but
> we're using it inhouse and it mostly works. However, it's not yet vcs-agnostic
> and friends (it's still kind of written for our git-based workflow) and just
> wrapping buildbot is not it goal.
>
> Additionally, there is simple "buildmaster client" cthylla [4] and set of
> "slave" setup.py plugins citools [5].
>
> My idea/hope is to enpower common users to create their own buildslaves as
> they need them, as well as create new project. So first thing on my plate is to
> have better configuration (so you write a plugin for company that says "create
> debian package according to our policy and upload it to repository, and user
> only says on web "OK, I want this for my project". This works now), but I'm
> assuming company-like network (= master has ssh access to slaves).

Interesting. Thinking further, a public repository of web-configurable 
plugins would be cool, as you currently have to be quite python savvy to 
set up a build master. (and there doesn't seem to be that much sharing 
of custom build steps, which I think would be a good thing to make it 
obvious that buildbot can be pretty useful for more things than just 
reporting fail/success and logging stdio)

> Second plan is to make better reporting, I'd like to have "developer
> dashboard" displaying all commits done by logged user and their test results
> (kind of filtered tgrid).

Aye. We need to make the web interface more accessible and extendable. I 
hope this will be possible if/when the data goes into SQL, making the 
"clients" of said database language/library/buildbot agnostic. I.e. 
empower users to write clients that do their favourite stuff in their 
favourite langugage/framework.

> Also, RSS/pubsubhubhub should be super easy by then.

Uhmm? :)

> I'm now in phase "first step: it's crufty, but works for us", so I'm now
> writing docs and preparing to polish things and stuff. My thoughts were to
> announce project when I'll have neat tutorial "how to automatically build your
> python package in virtualenv with few clicks" ;)

That'd be cool. :)

> Comments welcomed, althrough we should probably move to separate topic/thread.

Done. ;)

Cheers,
/Marcus

> [1] http://github.com/ella/buildbot-mongodb-status
> [2] http://github.com/fiorix/mongo-async-python-driver
> [3] http://github.com/ella/cthulhubot
> [4] http://github.com/ella/cthylla
> [5] http://github.com/ella/citools