[Buildbot-devel] implementing build dependencies

[Brian Warner]

[this message is in response to one I may not have sent to the list as I
intended.. scroll down for the context]


I've had a new realization about Dependencies.. they are really talking about
SourceStamps, not Changes. Rather than paying attention to the disposition of
each Change, the downstream scheduler should just subscribe to hear about
successful BuildSets from the upstream scheduler. For each of these, there is
a specific SourceStamp that describes what exactly was successfully built.
Then the downstream scheduler can just do the same.

This cleans up the architecture considerably. The new version of the "Build
Dependencies" page of the user's manual is here:

==== manual start

4.4.1 Build Dependencies
------------------------

It is common to wind up with one kind of build which should only be
performed if the same source code was successfully handled by some
other kind of build first. An example might be a packaging step: you
might only want to produce .deb or RPM packages from a tree that was
known to compile successfully and pass all unit tests. You could put
the packaging step in the same Build as the compile and testing steps,
but there might be other reasons to not do this (in particular you
might have several Builders worth of compiles/tests, but only wish to
do the packaging once). Another example is if you want to skip the
"full" builds after a failing "quick" build of the same source code.
Or, if one Build creates a product (like a compiled library) that is
used by some other Builder, you'd want to make sure the consuming
Build is run _after_ the producing one.

   You can use `Dependencies' to express this relationship to the
Buildbot. Each Change-based Scheduler can be told that it is
"dependent" upon one or more other Schedulers. For each Change that
arrives at such a Scheduler, it will ask the "upstream" Schedulers
what _they_ did with that Change. If the upstream says that the the
Change was involved in a Build which failed, the "downstream"
Scheduler will put that Change aside for a while (specifically: it
will not trigger a new build, but _will_ be included in the next
Build that happens for other reasons.. the Change is treated as
`unimportant', just as if the Scheduler's `isFileImportant' method
had decided the Change was not worth triggering a new build).
Changes that are involved in a successful upstream build will trigger
a downstream build normally. The downstream Scheduler will wait until
the upstream Change is involved in a build before starting its own.
(Changes which are ignored upstream are treated as successful, to
avoid surprises when the two Schedulers have different ideas about
which files are important).

   Schedulers may depend upon more than one upstream scheduler at the
same time. The downstream build will wait until _all_ of the upstream
builds have succeeded (or until the first one fails) before deciding
the fate of a given Change. When an upstream Scheduler triggers
Builds on multiple Builders, the same logic is applied (wait for
complete success or the first failure).

     from buildbot import scheduler
     tests = scheduler.Scheduler("tests", None, 5*60,
                                 ["full-linux", "full-netbsd", "full-OSX"])
     package = scheduler.Scheduler("package", None, 5*60,
                                   ["make-tarball", "make-deb", "make-rpm"],
                                   dependencies=[tests])
     c['schedulers'] = [tests, package]

   Note that the `dependencies' argument is given a list of upstream
`Scheduler' _instances_, not names. This makes it impossible to
create circular dependencies in the config file.

==== manual end

I want to show per-Change disposition status, but I'm more concerned about
having the build-triggering architecture be clean. I'm planning on giving
each Scheduler a status object, to which it will publish information about
what it's doing (change N is ignored/queued/built-successfully/failed), and I
think from that we can create the change-disposition table. My hunch is that
it will be better to derive this table than to let it drive the scheduling
process.

let me know what you think,
 -Brian


[previous message follows]

> From: Brian Warner <warner-buildbot at lothar.com>
> Subject: Re: [Buildbot-devel] Interlocked forever
> Date: Wed, 06 Jul 2005 16:19:57 -0700 (PDT)

> > Then if you click on that change in the web view you would hopefully be
> > able to see each builder's status with regard to that change; the same goes
> > for clicking on the builder, where you should be able to see all the
> > changes pending/waiting/whatever.
> 
> Ooh, I like that. There are some implementation issues that I don't yet know
> how to solve, but I definitely agree that it should be possible to get that
> sort of status out of the system and onto the Waterfall page.
> 
> I think I had something like this in the back of my mind when writing the
> original Interlock implementation.. the name= argument was to provide
> something that could be pointed to on the web page. But the idea evaporated
> while I was coding, and I ended up with those Deferreds instead of something
> more useable. I was also referring to everything by name (instead of by
> explicit Builder/Scheduler references, like I'm using now) because I had this
> crazy idea that it should be possible to configure the buildmaster entirely
> through a web page, and it seemed that using names would make that easier. I
> don't know how crazy that idea is, but I don't think that avoiding direct
> references would make it any more difficult to achieve.
> 
> Hm. This sort of change-tracking might be an appropriate task for the
> ChangeMaster, except that I was kind of hoping to get rid of it and merge its
> remaining functionality into the BuildMaster. We could have Changes track
> this information themselves, except that this would make them less pleasant
> to serialize (at the moment they are pure data: no references to anything
> else). We could put it in the Schedulers, which would have the benefit that
> non-Change-driven Schedulers (like the existing Periodic and the hypothetical
> WatchOtherBuildMaster) could cleanly opt out of the whole thing, but there
> are other uglinesses with that approach (particularly because Schedulers
> aren't all obligated to subscribe to the same set of ChangeSources).
> 
> Hrm, I will have to think about this more. I'm pleased with the configuration
> syntax of Dependencies that I'm aiming towards right now (I'll attach the
> user manual page below), but the implementation doesn't have to follow the
> same pattern.. we could have a centralized ChangeTracker object that follows
> everything without having to expose it in the config file.
> 
> thanks,
>  -Brian
> 
> 
> 
> 4.4.1 Build Dependencies
> ------------------------
> 
> It is common to wind up with one kind of build which should only be
> performed if the same source code was successfully handled by some
> other kind of build first. An example might be a packaging step: you
> might only want to produce .deb or RPM packages from a tree that was
> known to compile successfully and pass all unit tests. You could put
> the packaging step in the same Build as the compile and testing steps,
> but there might be other reasons to not do this (in particular you
> might have several Builders worth of compiles/tests, but only wish to
> do the packaging once). Another example is if you want to skip the
> "full" builds after a failing "quick" build of the same source code.
> Or, if one Build creates a product (like a compiled library) that is
> used by some other Builder, you'd want to make sure the consuming
> Build is run _after_ the producing one.
> 
>    You can use `Dependencies' to express this relationship to the
> Buildbot. Each Change-based Scheduler can be told that it is
> "dependent" upon one or more other Schedulers. For each Change that
> arrives at such a Scheduler, it will ask the "upstream" Schedulers
> what _they_ did with that Change. If the upstream says that the the
> Change was involved in a Build which failed, the "downstream"
> Scheduler will put that Change aside for a while (specifically: it
> will not trigger a new build, but _will_ be included in the next
> Build that happens for other reasons.. the Change is treated as
> `unimportant', just as if the Scheduler's `isFileImportant' method
> had decided the Change was not worth triggering a new build).
> Changes that are involved in a successful upstream build will trigger
> a downstream build normally. The downstream Scheduler will wait until
> the upstream Change is involved in a build before starting its own.
> (Changes which are ignored upstream are treated as successful, to
> avoid surprises when the two Schedulers have different ideas about
> which files are important).
> 
>    Schedulers may depend upon more than one upstream scheduler at the
> same time. The downstream build will wait until _all_ of the upstream
> builds have succeeded (or until the first one fails) before deciding
> the fate of a given Change. When an upstream Scheduler triggers
> Builds on multiple Builders, the same logic is applied (wait for
> complete success or the first failure).
> 
>      from buildbot import scheduler
>      tests = scheduler.Scheduler("tests", None, 5*60,
>                                  ["full-linux", "full-netbsd", "full-OSX"])
>      package = scheduler.Scheduler("package", None, 5*60,
>                                    ["make-tarball", "make-deb", "make-rpm"],
>                                    dependencies=[tests])
>      c['schedulers'] = [tests, package]
> 
>    Note that the `dependencies' argument is given a list of upstream
> `Scheduler' _instances_, not names. This makes it impossible to
> create circular dependencies in the config file.
>