[Buildbot-devel] Updating scheduler to 0.8.x, pretty lost

[Axel Hecht]

Sounds like a good idea I asked :-)

I'll put that attempt on hold then, and if you need someone to smoketest
your API, drop me a note.

Axel

2010/12/28 Dustin J. Mitchell <dustin at v.igoro.us>

> First, I'm in the process of rewriting the schedulers to use
> SQLAlchemy, and I don't see much rhyme or reason to the way schedulers
> are written and subclassed.  We just uncovered a bug in Mozilla RelEng
> where it turns out certain scheduler functions are run in a thread,
> and will fail due to using the wrong SQLite Connection object.  That's
> not at all clear from the interface.  So this will be the first place
> that I'm writing out an actual, well-defined API against which
> subclasses should be written.  Yay!  Of course, the downside is that
> scheduler subclasses will need to be rewritten against the new API in
> 0.8.4.  It's lots cleaner (not to mention well-defined and written
> down), but not backward-compatible.
>
> On Mon, Dec 27, 2010 at 6:47 AM, Axel Hecht <l10n.moz at googlemail.com>
> wrote:
> > Any recommendation on how to do BuildSet.waitUntilFinished() in a
> scheduler?
>
> This is the purpose of the scheduler_subscribe_to_buildset method.
> However, I think this only gets the scheduler loop run when that
> buildset completes - you still need to poll its status from the
> database on every run of the loop.  Look at how the triggerable
> scheduler polls in its _run method.  Please *don't* look at the use of
> Deferreds in a non-main thread in its _trigger method.  Yikes!
>
> > The follow up question I'll have is, my code likely adds some hundred
> > buildsets, same builder with different properties. Any recommendations on
> > how to issue those? Reading the code it looks like 100 times 10 db
> > connections or something like that, if just calling into
> db.create_buildset?
>
> Yes, the existing scheduler infrastructure is horrendously
> inefficient.  It's not that many connections, but it is that many
> queries.  We should be able to reduce that substantially with the
> SQLAlchemy changes, although each BuildSet still needs to have
> multiple BuildRequest rows inserted as well.
>
> Dustin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101230/d8538df7/attachment.html>