[Buildbot-devel] Buildbot Summit - Out of process

[Axel Hecht]

Jumping in at the end, not as a real reply to this message.

There are two items that I'd like to add to the list:

- performance
- flexible development

Performance is something one needs to really pay attention to. My own status
app on top of my own status db takes a dive at a few hundred builds.
Mozilla's tbpl easily shows a thousand builds. An real overview of our
nightly builds shows a good 1000 builds, too.

Flexible development is important, too, in particular for the webparts. I
for one am horrible at both web coding and SQL, so I use the tool that
enables me to do both. I don't think that we should impact the design
decisions of the front end by synchronous vs async etc. Unless we're jumping
on the NodeJS train ;-) .

My personal conclusion is to get the status as descriptive as possible out
of buildbot/twisted and into a db, and then have competing and maybe even
customizable front ends based on that. Pulse etc is nice, but it's not
status, so folks hacking on presenting status will only care about the
status store.

FTR, one thing I found confusing in the thread is the question of sync vs
async, and it's impact on schedulers and such.

Axel

2010/11/24 Dustin J. Mitchell <dustin at v.igoro.us>

> On Wed, Nov 24, 2010 at 3:05 AM, Marcus Lindblom <macke at yar.nu> wrote:
> > Or am I missing some important point here?
>
> The key point that you didn't mention is that the in-Python status API
> is currently synchronous, and that it will have to become asynchronous
> when it's backed by a DB.
>
> So every foo.getBar() call will return a deferred.  This isn't so bad
> for, say, /builders/mybuilder or the like, but it represents a
> complete rewrite of the more complex synthesis displays - (t)grid,
> waterfall, and console.  The JSON calls, in the final analysis, will
> basically map an HTTP request to a DB query, and use the usual
> Deferred chaining to translate the query results back into JSON.  So
> by pushing this stuff out of process and onto the client, we get a few
> wins:
>
>  - no long synchronous rendering procedures on the master
>  - no need for a second master-side process
>  - a distinct (and distinctly licensable) web-status project that
> others can hack
>  - only one rewrite :)
>
> Even if we moved the WebStatus to another Python process, without a
> significant amount of work on the synthesis displays, it would still
> not work - out of process, all of those getBar invocations would be PB
> calls, and thus return a Deferred.  If you've ever looked at the
> waterfall code, this should make you break out in a cold sweat.
>
> By the way, within the buildmaster, I intend to support the status API
> in a few languages.  That is, you can make the same call via JSON,
> from out-of-process Python via PB, or from in-process Python via
> regular asynchronous function calls (modulo language differences, of
> course).  This will give a nice separation between status consumers
> and the buildmaster, allowing us to write e.g., status listeners that
> can run in-process or out-of-process.
>
> Also, creating a well-defined API gives users a clear indication of
> what we are committing to support in future versions.  We'll need more
> APIs than this one (in particular, what the heck can a BuildStep
> subclass do? What *can't* it do?), but this is a big step.
>
> I'm still very open to suggestions here, especially if they come with
> an indication that someone other than me will do the work! :)
>
> Dustin
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101125/285e1bbd/attachment.html>