[Buildbot-devel] the aftermath (was Re: SF bay area get-together?)

[Marcus Lindblom]

Jean-Paul Calderone wrote:

> I'm not sure what a "disconnected design" is - I suppose it's something
> like loose coupling - but I don't think it will help avoid the pain of
> using an ORM in a Twisted applications.

It's simply that the components don't depend on eachother.

I.e. you could use any subset of Django's ORM, HTTP or template system, 
and/or replace any one of those with one of your own design in your app.

There is of course a small 'shortcut' api that helps with the common use 
case, but each part can still be used independently. (Which is not 
always the case.)

> The problem is a very basic one:
> in an ORM, the API for accessing the database is primary attribute-
> oriented; accessing the database is a blocking operation.  If you block
> every time you need to get data from or put data in the database, then
> you'd better be using threads or you're going to have pretty unreasonably
> bad performance. Using threads ubiquitously enough to keep reasonable
> performance and using Twisted in the same application is going to be
> messy and unpleasant.  There are two possible ways I know to avoid this:
> make sure the ORM is backed by a database that is fast and local (ie, not
> on the other side of a socket) and never make any queries which can't be
> satisfied in O(1) time or better; or, use an ORM which doesn't make its
> primary APIs blocking.  For the former, you can use things like SQLite,
> but don't expect to ever be able to switch to PostgreSQL or other remote
> databases.  The latter is still an open question; I don't know of any ORMs
> for Python which make it feasible.

(I'm a bit of a newbie on async RPC applications)

My naive intuition (coupled with a likewise rather weak insight in 
buildbot/twisted internals) tells me that a web app and buildbot really 
isn't that much different, and what works performance wise for one 
should work for the other.

But, if I understand you correctly, BuildBot/Twisted uses very few 
threads, whereas a typical web server uses many (one per request or 
something simmilar). Thus a web server circumvents the problem while the 
Twisted arcitecture suffers when using blocking DB ops?

I should probably go and read up on Twisted a bit.

Cheers,
/Marcus