[Buildbot-devel] Updating scheduler to 0.8.x, pretty lost

[Dustin J. Mitchell]

First, I'm in the process of rewriting the schedulers to use
SQLAlchemy, and I don't see much rhyme or reason to the way schedulers
are written and subclassed.  We just uncovered a bug in Mozilla RelEng
where it turns out certain scheduler functions are run in a thread,
and will fail due to using the wrong SQLite Connection object.  That's
not at all clear from the interface.  So this will be the first place
that I'm writing out an actual, well-defined API against which
subclasses should be written.  Yay!  Of course, the downside is that
scheduler subclasses will need to be rewritten against the new API in
0.8.4.  It's lots cleaner (not to mention well-defined and written
down), but not backward-compatible.

On Mon, Dec 27, 2010 at 6:47 AM, Axel Hecht <l10n.moz at googlemail.com> wrote:
> Any recommendation on how to do BuildSet.waitUntilFinished() in a scheduler?

This is the purpose of the scheduler_subscribe_to_buildset method.
However, I think this only gets the scheduler loop run when that
buildset completes - you still need to poll its status from the
database on every run of the loop.  Look at how the triggerable
scheduler polls in its _run method.  Please *don't* look at the use of
Deferreds in a non-main thread in its _trigger method.  Yikes!

> The follow up question I'll have is, my code likely adds some hundred
> buildsets, same builder with different properties. Any recommendations on
> how to issue those? Reading the code it looks like 100 times 10 db
> connections or something like that, if just calling into db.create_buildset?

Yes, the existing scheduler infrastructure is horrendously
inefficient.  It's not that many connections, but it is that many
queries.  We should be able to reduce that substantially with the
SQLAlchemy changes, although each BuildSet still needs to have
multiple BuildRequest rows inserted as well.

Dustin