[users at bb.net] Build has less changes than expected

[Lennart Sauerbeck]

Hi again,

I managed to work around the issues.

Instead of doing all the work in a custom BuildStep, I additionally created a custom Scheduler which does most of the work on the incoming changes. In `addBuildsetForChanges()` I iterate over all `changeids` and get the required change dictionaries using `self.master.db.changes.getChange(changeid)`. Based on the properties of these changes I can bundle up source stamps in cohesive groups. Each group gets passed to `addBuildsetForSourceStamps` and becomes a build for my custom BuildStep. 

In the builder using my custom BuildStep I now still only have one change (even though each source stamp is associated with a change), but I don't care. Since I am sure that all source stamps belong to the same group, I just have to figure out which triggerable Scheduler to trigger for my current problem set. Once that is done, the custom builder finishes and awaits the next buildset added by the custom Scheduler.

Let me know if anyone is running into similar problems, I can go into more detail.

Best Regards
Lennart Sauerbeck

 

Von: users [mailto:users-bounces at buildbot.net] Im Auftrag von Lennart Sauerbeck
Gesendet: Montag, 18. Dezember 2017 16:15
An: users at buildbot.net
Betreff: [users at bb.net] Build has less changes than expected

Hi folks,

first of all thanks to the team and contributors for working on Buildbot. I enjoyed deploying it in version 0.8 and really dig the new Web interface of 0.9.

However, during the port of one of my custom builders I hit a problem concerning changes. I read the documentation many times, dived into the source code and tried many things, but fear I can't solve this by myself. So any help would be really appreciated.

In order to avoid the X/Y-problem, I'd like to describe my usecase first:
I use Buildbot to run pre-merge continuous integration checks on a range of projects backed by Gerrit. A subclassed `GerritChangeSource` reacts on published reviews, adds all these as Buildbot changes to the database which triggers a custom builder. This builder now looks at all incoming changes, does some dependency calculations (due to projects with submodules) and triggers builds on multiple schedulers, each with a subset of the incoming changes (and some sourcestamps generated due to dependencies). This allows for each published Gerrit change  to be built as stand-alone as possible.

All these classes (`GerritChangeSource` subclass and the buildstep for the custom builder) were written for Buildbot 0.8 and worked great. With 0.9.14 (after rewriting them according to `Upgrading to Nine`, not so much. Since there's a short and a long version of the problem, I'll give you both.

The short version:
In version 0.8 each build got all the changes that were added by the connected `ChangeSource` (using `self.build.allChanges()`). In version 0.9.14 this only ever returns one change.
How can I get all changes the `ChangeSource` added?

The long version:
I subclass `GerritChangeSource` to receive Buildbot changes from Gerrit. Subclassing is mostly done to have a way of forcing rebuilds for a Gerrit change. Whenever a Gerrit change is published, I can see in the twistd.log that the expected number of changes are added to the database. So for two Gerrit changes that are published I see two Buildbot changes added to the database according to twistd.log. There only is one scheduler which reacts on these Gerrit changes (using `util.GerritChangeFilter` as `change_filter`). This scheduler only has one connected builder, which has `collapseRequests` set to `False`. This builder is triggered by the `ChangeSource` and its one and only step starts. This custom step implements `buildstep.ShellMixin` and `buildstep.Buildstep`. Inside the run method I now try to access all changes added by the ChangeSource for this build. Now here's the problem.
- `self.build.allChanges()` only ever returns a list of size 1, containing only the latest change.
- `self.build.requests()` also only ever returns a list of size 1 (and each request only ever has one change in each of the codebases).
- `self.build.build_status.getChanges()` only ever returns a list of size 1, containing only the latest change.
How can I access all changes that were added by the `ChangeSource`? They are sometimes listed on the `Changes` tab in the Web-UI (sometimes the same commit is shown multiple times here), but I cannot seem to get at them from inside the buildstep itself.

Did I miss any vital information? Is there anything I can explain in more detail?

Any help in this matter would be greatly appreciated.

Best Regards
Lennart Sauerbeck