[Buildbot-devel] Acceptable dependency versions for patches?

[David Bolen]

Jean-Paul Calderone <exarkun at divmod.com> writes:

> You can also break the call chain like this (untested):
>
>     def lotsOfDeferreds(listOfStuff):
>         overallResult = Deferred()
>         def oneDeferred(ignored):
>             if listOfStuff:
>                 try:
>                     oneResult = processOne(listOfStuff.pop())
>                 except:
>                     overallResult.errback()
>                 else:
>                     oneResult.addCallback(oneDeferred)
>                     oneResult.addErrback(overallResult.errback)
>             else:
>                 overallResult.callback(None)
>         oneDeferred()
>         return overallResult
>
> The advantage of this approach over using callLater is primarily
> that it is simpler to test.

Slick - yep, that looks like it would be another approach as well.  I
do also agree with the prior poster that callLater(0,...) is an
alternative approach.

I have to say I had to think a bit to assure myself that the above
broke the chain (I guess I found it too easy to notice the addCallback
and neglect to notice that oneResult is never percolating back).

One risk with the above code is if the deferred returned by processOne
can be fired already, in which case this can theoretically turn into a
synchronous recursive loop.  That can also be an issue for the
callLater approach since you'd end up creating loads of delayed calls
waiting in the reactor - not catastrophic, but not really desirable
either.

Interestingly enough, given your testing comment, that's where I've
run into issues like that the most in the past - in tests where
simulated activities may finish instantly or stub routines return
defer.succeed() or defer.fail().

I know the generator approaches in the Twisted library are coded
against that scenario.  If I wanted to truly have a non-generator
approach in my code I'd probably use that as a model, but just change
out the assumption that the underlying function was a generator.  Then
again, the generator approaches are convenient for purposes such as
this (as long as you don't get carried away trying to use them to hide
the asynchronous nature of the code, IMO).

BTW, if you are really using code like this, wouldn't you want to
create local bindings for listOfStuff and overallResult within
oneDeferred, e.g.:

    def oneDeferred(ignored,
                    listOfStuff=listOfStuff, overallResult=overallResult):

Otherwise, within the body of oneDeferred, those references will refer
back to the containing function variables, but are dereferenced when
the code runs and not locally bound at definition time.  So if there
are multiple users of your lotsOfDeferred function at the same time,
they'll likely step over each other (everyone will see the values for
those bindings from the most recent call).

-- David