[Buildbot-devel] Acceptable dependency versions for patches?

[Jean-Paul Calderone]

On Sun, 16 Sep 2007 05:21:52 -0400, David Bolen <db3l.net at gmail.com> wrote:
>Jean-Paul Calderone <exarkun at divmod.com> writes:
>
>> You can also break the call chain like this (untested):
>>
>>     def lotsOfDeferreds(listOfStuff):
>>         overallResult = Deferred()
>>         def oneDeferred(ignored):
>>             if listOfStuff:
>>                 try:
>>                     oneResult = processOne(listOfStuff.pop())
>>                 except:
>>                     overallResult.errback()
>>                 else:
>>                     oneResult.addCallback(oneDeferred)
>>                     oneResult.addErrback(overallResult.errback)
>>             else:
>>                 overallResult.callback(None)
>>         oneDeferred()
>>         return overallResult
>>
>> The advantage of this approach over using callLater is primarily
>> that it is simpler to test.
>
>Slick - yep, that looks like it would be another approach as well.  I
>do also agree with the prior poster that callLater(0,...) is an
>alternative approach.
>
>I have to say I had to think a bit to assure myself that the above
>broke the chain (I guess I found it too easy to notice the addCallback
>and neglect to notice that oneResult is never percolating back).

It's a good thing to put into a function someplace and forget about :)

>
>One risk with the above code is if the deferred returned by processOne
>can be fired already, in which case this can theoretically turn into a
>synchronous recursive loop.  That can also be an issue for the
>callLater approach since you'd end up creating loads of delayed calls
>waiting in the reactor - not catastrophic, but not really desirable
>either.

Yep, that is a problem with this code.  If many Deferreds can have
fired when they are returned, considering something else is probably
a good idea.

>
>Interestingly enough, given your testing comment, that's where I've
>run into issues like that the most in the past - in tests where
>simulated activities may finish instantly or stub routines return
>defer.succeed() or defer.fail().
>
>I know the generator approaches in the Twisted library are coded
>against that scenario.  If I wanted to truly have a non-generator
>approach in my code I'd probably use that as a model, but just change
>out the assumption that the underlying function was a generator.  Then
>again, the generator approaches are convenient for purposes such as
>this (as long as you don't get carried away trying to use them to hide
>the asynchronous nature of the code, IMO).

Of course, you could always implement the generator API with another
kind of iterator.  This would be easiest with cooperator, since it only
uses __iter__ and next().  defgen would probably be next easiest (maybe
exactly as easy, actually).  inlineCallbacks would require send() and
throw().

>
>BTW, if you are really using code like this, wouldn't you want to
>create local bindings for listOfStuff and overallResult within
>oneDeferred, e.g.:
>
>    def oneDeferred(ignored,
>                    listOfStuff=listOfStuff, overallResult=overallResult):
>
>Otherwise, within the body of oneDeferred, those references will refer
>back to the containing function variables, but are dereferenced when
>the code runs and not locally bound at definition time.  So if there
>are multiple users of your lotsOfDeferred function at the same time,
>they'll likely step over each other (everyone will see the values for
>those bindings from the most recent call).

Each call to lotsOfDeferreds gets its own distinct locals, so there's
no shared state even with multiple concurrent calls.  The approach is
very similar to defining a class with a oneDeferred method and the
closed over variables as attributes on self.  It's just a bit less
typing (not without a cost - you lose convenient introspection).

Jean-Paul