[Buildbot-devel] database-backed status/scheduler-state project

[Almad]

Dne čtvrtek 03 Září 2009 00:56:05 Brian Warner napsal(a):
> Hi everybody, it's me again.

Hi,

> I've taken on a short-term contract with Mozilla to make some
> scaling/usability improvements on Buildbot that will be suitable for
> merging upstream. The basic pieces are:
>
>  * persistent (database-backed) scheduling state
>  * DB-backed status information
>  * ability to split buildmaster into multiple load-balanced pieces

Awesome!

> DATABASES:

> The statusdb (as opposed to the schedulerdb) may be implemented as a
> buildbot status plugin, leaving the existing pickle files alone, but
> exporting a copy of everything to an external database as the builds
> progress.

As I said on IRC, I'm storing configuration in rdbms and build results in 
mongo database and I'm happy with the results so far.

Additionally, mongo's "capped collections" feature looks fine for auto-
managing database load/size. 

> DEPENDENCIES:
> What do people think about the 0.8.0 buildmaster potentially requiring
> sqlalchemy? Would that annoy you? Annoy new users? Make it hard to
> upgrade your environment?

As for me, I'm okay with deps laying on disc unused ;)

> HELP!:
>
> I'm looking to hear about other folk's experiences with this sort of
> project. We've been talking about this for years, and some prototypes
> have been built, so I'd like to hear about them (I've been briefed on
> many of the mozilla efforts already).

Well, I'm now working on something similar, however my goals are different: 
I'd like to manage mediocre number of projects (~ 15-50) using web interface 
and have automatic management of port locks and slaves (i.e. adding new 
computer should be only SSH key setup, software should handle slave creation 
et al.)

It's still in somehow early stage of development, but we'll be relying on it 
heavily, so at least for package creation is should be ready soon. Progress 
can be viewed at http://github.com/ella/cthulhubot/tree/master ...but I have 
not bootstrapped yet :-( But I'll also try to reuse it as part of my thesis, 
so it should be documented well in the future ;)

My approach is to have django application that stores configuration in any 
rdbms via it's orm - and receiving build progress from mongo database. In the 
future, it'll also work with repositories, to be compilant with our 'continous 
versioning/deployment' approach.

As described, althrough I'm sharing some problems, I have other goals and 
working on top of buildbot instead of messing with it :)

>
> cheers,
>  -Brian



>
> ===== PROJECT PROPOSAL =====


>   * phase 1.1: Implement the Postgres database connector, and the
>     master.cfg options necessary to control which db type/location to
>     use for scheduler state. Test a buildmaster running with a remote
>     schedulerdb.

I'd like database connector to have own interface (i.e. not only orm-like 
datastore interface). This would allow people to build non-sql 
connectors...I'll probably try this as I think mongo will give better results 
than rdbms and is also simpler/better suited for thi spurpose.


------


All in all, I'm excited - it looks like your project will solve a lot of our 
problems :)

Almad