[Buildbot-devel] the aftermath (was Re: SF bay area get-together?)

[Jean-Paul Calderone]

On Tue, 05 Aug 2008 09:18:58 +0200, Marcus Lindblom <macke at yar.nu> wrote:
>Dustin J. Mitchell wrote:
>> On Mon, Aug 4, 2008 at 11:56 AM, Marcus Lindblom <macke at yar.nu> wrote:
>>> In fact, Django's model-api is so easy & cool I think it would be very
>>> easy to add extra data (metrics, whatnot) to builds from custom build
>>> steps or other tools.
>>>
>>> Would that be feasible? We could still use Twisted for the HTTP part,
>>> but use DB/template system from django for storage & presentation?
>>>
>>> Or am I completely out of line? ;)
>>
>> I've been reading about and playing with Django, too, and I agree that
>> it has a great design -- and not just in its database API.  What
>> Twisted gives us that Django does *not*, however, is the asynchronous
>> communication paradigm and the excellent RPC library.  I don't know
>> how feasible it would be to mix the two, but I imagine that it would
>> cause some pain.
>
>Actually, I think not. Django's manuals continously stress that they've
>striven for a disconnected design, and my impression from 10-12 days of
>hacking confirms that.
>

I'm not sure what a "disconnected design" is - I suppose it's something
like loose coupling - but I don't think it will help avoid the pain of
using an ORM in a Twisted applications.  The problem is a very basic one:
in an ORM, the API for accessing the database is primary attribute-
oriented; accessing the database is a blocking operation.  If you block
every time you need to get data from or put data in the database, then
you'd better be using threads or you're going to have pretty unreasonably
bad performance.  Using threads ubiquitously enough to keep reasonable
performance and using Twisted in the same application is going to be
messy and unpleasant.  There are two possible ways I know to avoid this:
make sure the ORM is backed by a database that is fast and local (ie, not
on the other side of a socket) and never make any queries which can't be
satisfied in O(1) time or better; or, use an ORM which doesn't make its
primary APIs blocking.  For the former, you can use things like SQLite,
but don't expect to ever be able to switch to PostgreSQL or other remote
databases.  The latter is still an open question; I don't know of any ORMs
for Python which make it feasible.

Jean-Paul