[Buildbot-devel] Brainstorming about extending Git step to support multiple repos

[Jared Grubb]

I have been brainstorming about something I’ve wanted a while and wanted to get feedback and ideas.

I want to solve two problems:
1. share a “reference” repo across multiple builders
2. support named remote repos

Note that when I say “Git step”, that could either mean the existing Git step, or perhaps a subclass.

== Reference Repo
The current Git step already has a “reference” argument, which maps to the git-clone argument. However, I’m not sure how that reference repo should be maintained. It seems to me that, each build, I’d first want to fetch into that one, and then fetch into my local copy. So, I’d like to add direct support for updating the reference repo.

I’d like to make reference repos easier to maintain. For example, I have 34 builders that reference one particular git repo, which means that each buildslave has 34 independent local copies of each git repo. Using a reference repo will mean I still hit the network 34 times, but only download a revision once.

== Named Upstreams
I’d like to add multiple repourl’s to a Git step, each with a name. For example, you could have “origin -> official github” and then “jgrubb -> jgrubb’s mirror”, “djmitche -> his mirror”, etc. Then on ForceScheduler, you could make the “Repository” field a dropdown with the values {origin, jgrubb, djmitche}. (Note this is an example that mirrors my actual usage; I’m not proposing buildbot do that on its buildbot)

My buildbot currently does this by making repourl a renderable that chooses the repourl based on the sourcestamp. This has worked pretty well, but I’d like this to more natively supported. For example, I have a ShellCommand step that (based on a build property) may create a git tag and push to origin; however, if the checkout was not from the actual “origin”, then it will push to the wrong repo. Giving the repos fixed, known names let me build ShellCommand’s that can push to the correct repo later on. 

==
Anyway, has anyone else needed similar functionality? Do you think this warrants enhancement of the Git step? Do you think support a multi-repo scenario suggests a subclass of Git instead? Any thoughts?

Jared