[Buildbot-devel] Buildbot Summit - Out of process

Thu Nov 25 18:05:17 UTC 2010

Le 25 novembre 2010 10:01, Axel Hecht <l10n.moz at googlemail.com> a écrit :

> Jumping in at the end, not as a real reply to this message.
>
> There are two items that I'd like to add to the list:
>
> - performance
> - flexible development
>
> Performance is something one needs to really pay attention to. My own
> status app on top of my own status db takes a dive at a few hundred builds.
> Mozilla's tbpl easily shows a thousand builds. An real overview of our
> nightly builds shows a good 1000 builds, too.
>
> Flexible development is important, too, in particular for the webparts. I
> for one am horrible at both web coding and SQL, so I use the tool that
> enables me to do both. I don't think that we should impact the design
> decisions of the front end by synchronous vs async etc. Unless we're jumping
> on the NodeJS train ;-) .
>
> My personal conclusion is to get the status as descriptive as possible out
> of buildbot/twisted and into a db, and then have competing and maybe even
> customizable front ends based on that. Pulse etc is nice, but it's not
> status, so folks hacking on presenting status will only care about the
> status store.
>

I'll try to answer with commits.

And for non-Moz folks, Pulse = pulse.mozilla.org. I was confused with the other
pulse <http://zutubi.com/products/pulse/>.

M-A

> FTR, one thing I found confusing in the thread is the question of sync vs
> async, and it's impact on schedulers and such.
>
> Axel
>
> 2010/11/24 Dustin J. Mitchell <dustin at v.igoro.us>
>
> On Wed, Nov 24, 2010 at 3:05 AM, Marcus Lindblom <macke at yar.nu> wrote:
>> > Or am I missing some important point here?
>>
>> The key point that you didn't mention is that the in-Python status API
>> is currently synchronous, and that it will have to become asynchronous
>> when it's backed by a DB.
>>
>> So every foo.getBar() call will return a deferred.  This isn't so bad
>> for, say, /builders/mybuilder or the like, but it represents a
>> complete rewrite of the more complex synthesis displays - (t)grid,
>> waterfall, and console.  The JSON calls, in the final analysis, will
>> basically map an HTTP request to a DB query, and use the usual
>> Deferred chaining to translate the query results back into JSON.  So
>> by pushing this stuff out of process and onto the client, we get a few
>> wins:
>>
>>  - no long synchronous rendering procedures on the master
>>  - no need for a second master-side process
>>  - a distinct (and distinctly licensable) web-status project that
>> others can hack
>>  - only one rewrite :)
>>
>> Even if we moved the WebStatus to another Python process, without a
>> significant amount of work on the synthesis displays, it would still
>> not work - out of process, all of those getBar invocations would be PB
>> calls, and thus return a Deferred.  If you've ever looked at the
>> waterfall code, this should make you break out in a cold sweat.
>>
>> By the way, within the buildmaster, I intend to support the status API
>> in a few languages.  That is, you can make the same call via JSON,
>> from out-of-process Python via PB, or from in-process Python via
>> regular asynchronous function calls (modulo language differences, of
>> course).  This will give a nice separation between status consumers
>> and the buildmaster, allowing us to write e.g., status listeners that
>> can run in-process or out-of-process.
>>
>> Also, creating a well-defined API gives users a clear indication of
>> what we are committing to support in future versions.  We'll need more
>> APIs than this one (in particular, what the heck can a BuildStep
>> subclass do? What *can't* it do?), but this is a big step.
>>
>> I'm still very open to suggestions here, especially if they come with
>> an indication that someone other than me will do the work! :)
>>
>> Dustin
>>
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
>> Tap into the largest installed PC base & get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101125/276b5113/attachment.html>