[Buildbot-devel] Dependent schedulers not being triggered

[Brian Warner]

Kenneth Lareau <Ken.Lareau at nominum.com> writes:

> Follow-up to my previous message: I have no idea if I'm going in the
> right direction here, but I decided to run a few commands from within
> a manhole connection:

(I'm running out the door for a week out of town before PyCon, so I apologize
for the brevity of my response)

I can't think of an immediate cause for this problem, but I'll throw out a
couple of things that might help steer you in the right direction.

'expectations' is a red herring, it has to do with ETA calculation, and is
present if the Builder instance had a previous Build to extract timing
information from. It will probably be None if this is the first Build that's
been done since the buildmaster was started (since expectations aren't
persistent across a restart).

2007/01/02 05:20 PST [-] maybeStartBuild:[<buildbot.process.base.BuildRequest instance at 0xb1ce1b8c>] [<buildbot.process.builder.SlaveBuilder instance at 0xb32e9f8c>]

this maybeStartBuild log line emits two lists. The first is the .buildable
attribute: all the BuildRequests that are ready to go. The second is .slaves:
a list of all the SlaveBuilder instances (one per buildslave) which it knows
about. Some of these SlaveBuilders might be busy doing other builds.. that
test is done a few lines after this log message is emitted (the first line in
buildbot/process/builder.py, Builder.maybeStartBuild(), about line 500).

Dependent schedulers work by having the "downstream" scheduler subscribe to
the "upstream" scheduler, to watch for all builds which succeed. Each time
one does, the downstream scheduler extracts the SourceStamp from the
successful build, then submits its own set of BuildRequests with the same
SourceStamp. This happens in buildbot/scheduler.py:Dependent.upstreamBuilt(),
about line 307.

I'd look more carefully at the Dependent scheduler: make sure it is pointing
at the correct upstream Scheduler, make sure that the upstream scheduler is
actually the one scheduling these builds (if a different Scheduler is
responsible for them, then your downstream Dependents won't ever notice
them). I haven't looked carefully at what happens when the config file is
reloaded w.r.t. Scheduler identity: I suspect there could be a problem in
which a reconfig event that modified one of the Schedulers but not the other
could leave you in a situation where your downstream Dependent is pointing at
an unused copy of the upstream, in which case it would never get triggered.
Restarting the buildmaster would clear this, at least until the next reconfig
that modified one and not the other.

 (if this turns out to be the case, please file a bug on it.. I've run into
 this problem in other parts of the config file and it is a confusing and
 annoying issue, which is very satisfying to get fixed :).

To investigate this, look at the list returned by master.allSchedulers(),
find the Dependent one, look at its .upstream instance, and make sure the
upstream is actually on the master's list. If it points to something that
isn't on that list, that's a sign of this bug. (note that it would probably
point to something with all the same attributes as something on that list,
but the id() value would be different). If a Scheduler isn't on the list
returned by master.allSchedulers(), then it isn't going to hear about Changes
and probably won't ever trigger a build.


ok, gotta run.. hope this helped,
 -Brian