[Buildbot-devel] Out of order steps in waterfall?

[Brian Warner]

> But I have existing steps after the new steps I'm inserting. This 
> causes the waterfall to show steps not in the order they got executed 
> but in the order they got added to the build status object.
> 
> Any way of fixing this?

Not directly, but you might be able to accomplish what you want by
conditionally "skipping" the other steps. If you know ahead of time what the
other steps might be, but it's just that you aren't sure whether you'll need
to run them or not, then put all the steps (both the always-run and the
sometimes-run) in the Build, then make the sometimes-run steps return SKIPPED
from their start() methods if they aren't needed on that particular build.
(there's an example in process.step_twisted.HLint). This involves creating
subclasses for those steps, and overriding their start() method to look at
their parent Build to see whether they should be run or not.


If you know even less about what those steps might be (for example, maybe you
run some command to get a list of modules that passed their unit tests, and
then you have to run a shell command for each of those modules), things are
trickier. The specific problem is that the communication path from the Build
object to its related BuildStatus object is one-way. The code to look at is
buildbot.process.base.Build.setupBuild().

Build.setupBuild (called when the build actually starts) creates all of its
Step objects at once. At the same time, it also creates BuildStatus objects
for each of them. They're all done together so that the BuildStatus knows how
many steps are going to be executed (so it can compute an ETA for the build,
display a list of upcoming steps, etc). The Build calls
self.build_status.addStep(step) for each newly created step, which is an
append-only operation: you'd need some sort of insert operation to make sure
the BuildStatus had the correct sequence. In your later steps (or preferably
in your Build subclass), when you decided you need to add some steps, you'd
need to do something like:

 newstep = factory(build=self, **args)
 newstep.name = something_unique
 self.steps.insert(newstep, where)
 self.build_status.insertStep(newstep, where)

The status.builder.BuildStatus object publishes the list of steps via its
getSteps() method, but that method specifically states that the list of steps
might change over time, so inserting is feasible (it would just require a new
method in BuildStatus). It might also make sense to replace addStep and
insertStep with a single setSteps() method that just replaces the previous
list of steps. However, you'd have to implement this in such a way that the
status from already-started steps was not lost (maybe by walking the new list
of steps and only creating new BuildStepStatus objects for the ones you
haven't seen before).

Dunno.. it sounds feasible, but a bit involved.

The other thing to pay attention to is the Progress tracking. To calculate an
ETA, the Build has a BuildProgress object that needs to know about all the
StepProgress objects maintained by the BuildSteps, so inserting new steps
will throw off the ETA calculations. It might not be feasible to provide an
ETA when you don't even know how much work you've got left, so I guess doing
self.useProgress=False is always an option.

What is the nature of the variable steps? Could they be handled by the
'return SKIPPED' mechanism?

hope that helps,
 -Brian