[Buildbot-devel] the aftermath (was Re: SF bay area get-together?)

[Axel Hecht]

Here's my take, randomly tacked anywhere in the thread here:

I looked at separating the django orm stuff from the rest, and found
that hard. I might not have looked hard enough, but it's defnitely not
something where you end up with "oh, just install django-orm and you
won't have your master cluttered with all that django web stuff".

Regarding just hand-coding SQL, I know jack, but I saw that sqlite has
rather rudimentary support for things like dates, and that text is not
length limited. Both seem rather strong constraints when trying to
write portable code.

I'm not so worried about buildmaster write performance, i.e., having
db writes being blocking. Most masters don't write that often, and in
those setups where they do, things like having the db master locally
and let the web interface work on remote slave dbs or something sounds
like the more standard way to attack that problem then to start doing
bookkeeping asynch.

I think I pasted this before,
http://hg.mozilla.org/users/axel_mozilla.com/tooling/index.cgi/file/ad58028e948a/mozilla/tools/buildbotcustom/buildbotcustom/
is my current django approach, on tip, rather not-tested, with builds/
having the model stuff, and status/dbbuilds.py being the status
observer to fill the db. I have similar stuff based on slqalchemy on
l10ndb.py, though that's a hand-tailored db for my stuff so far.

One thing that I found really hard to think about was trying to port
the feature of incremental log display into a remote web front end. I
figured one could have a specific back-door for this in the master,
but I have no idea how to get the callback oriented data delivery
ported over into something like django, which assumes to get the data
all at once.

Ideas?

Axel

2008/8/6 Marcus Lindblom <macke at yar.nu>:
> Jean-Paul Calderone wrote:
>
>> I'm not sure what a "disconnected design" is - I suppose it's something
>> like loose coupling - but I don't think it will help avoid the pain of
>> using an ORM in a Twisted applications.
>
> It's simply that the components don't depend on eachother.
>
> I.e. you could use any subset of Django's ORM, HTTP or template system,
> and/or replace any one of those with one of your own design in your app.
>
> There is of course a small 'shortcut' api that helps with the common use
> case, but each part can still be used independently. (Which is not
> always the case.)
>
>> The problem is a very basic one:
>> in an ORM, the API for accessing the database is primary attribute-
>> oriented; accessing the database is a blocking operation.  If you block
>> every time you need to get data from or put data in the database, then
>> you'd better be using threads or you're going to have pretty unreasonably
>> bad performance. Using threads ubiquitously enough to keep reasonable
>> performance and using Twisted in the same application is going to be
>> messy and unpleasant.  There are two possible ways I know to avoid this:
>> make sure the ORM is backed by a database that is fast and local (ie, not
>> on the other side of a socket) and never make any queries which can't be
>> satisfied in O(1) time or better; or, use an ORM which doesn't make its
>> primary APIs blocking.  For the former, you can use things like SQLite,
>> but don't expect to ever be able to switch to PostgreSQL or other remote
>> databases.  The latter is still an open question; I don't know of any ORMs
>> for Python which make it feasible.
>
> (I'm a bit of a newbie on async RPC applications)
>
> My naive intuition (coupled with a likewise rather weak insight in
> buildbot/twisted internals) tells me that a web app and buildbot really
> isn't that much different, and what works performance wise for one
> should work for the other.
>
> But, if I understand you correctly, BuildBot/Twisted uses very few
> threads, whereas a typical web server uses many (one per request or
> something simmilar). Thus a web server circumvents the problem while the
> Twisted arcitecture suffers when using blocking DB ops?
>
> I should probably go and read up on Twisted a bit.
>
> Cheers,
> /Marcus
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>