[Buildbot-devel] Brainstorming about extending Git step to support multiple repos

[Jared Grubb]

Dustin,

I’m glad you said this. I was studying the Git class and was realizing how stingy it really is in what it tries to pull. I was having trouble fitting the new operations on top the “_fetch” and “_clone” methods. After reading your message it dawned on me that I probably want a full fetch in order to mirror correctly.

As for your locking, I was wondering if that would be needed. In theory, I think git does locking itself. Do you think it would be necessary to have a Lock class managing this too?

Jared

> On Dec 30, 2014, at 21:36, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
> 
> I can see a lot of use from a Git step that can check out a shared
> reference directory -- one that might be accessed by multiple builds
> at the same time and thus has some locking involved.
> 
> I'm not sure if that'd be a "regular" Git step (which doesn't do a
> bare repository, and is pretty careful to only fetch the branches it
> needs) or a custom class of some sort.
> 
> The regular Git step's `reference` parameter should be enough to point
> to the updated reference repo, I think.
> 
> Dustin
> 
> On Sun, Dec 28, 2014 at 4:25 PM, Vitali Lovich <vlovich at gmail.com> wrote:
>> 
>>> On Dec 27, 2014, at 11:24 AM, Jared Grubb <jared.grubb at gmail.com> wrote:
>>> 
>>> I have been brainstorming about something I’ve wanted a while and wanted to get feedback and ideas.
>>> 
>>> I want to solve two problems:
>>> 1. share a “reference” repo across multiple builders
>>> 2. support named remote repos
>>> 
>>> Note that when I say “Git step”, that could either mean the existing Git step, or perhaps a subclass.
>>> 
>>> == Reference Repo
>>> The current Git step already has a “reference” argument, which maps to the git-clone argument. However, I’m not sure how that reference repo should be maintained. It seems to me that, each build, I’d first want to fetch into that one, and then fetch into my local copy. So, I’d like to add direct support for updating the reference repo.
>>> 
>>> I’d like to make reference repos easier to maintain. For example, I have 34 builders that reference one particular git repo, which means that each buildslave has 34 independent local copies of each git repo. Using a reference repo will mean I still hit the network 34 times, but only download a revision once.
>>> 
>>> == Named Upstreams
>>> I’d like to add multiple repourl’s to a Git step, each with a name. For example, you could have “origin -> official github” and then “jgrubb -> jgrubb’s mirror”, “djmitche -> his mirror”, etc. Then on ForceScheduler, you could make the “Repository” field a dropdown with the values {origin, jgrubb, djmitche}. (Note this is an example that mirrors my actual usage; I’m not proposing buildbot do that on its buildbot)
>>> 
>>> My buildbot currently does this by making repourl a renderable that chooses the repourl based on the sourcestamp. This has worked pretty well, but I’d like this to more natively supported. For example, I have a ShellCommand step that (based on a build property) may create a git tag and push to origin; however, if the checkout was not from the actual “origin”, then it will push to the wrong repo. Giving the repos fixed, known names let me build ShellCommand’s that can push to the correct repo later on.
>>> 
>>> ==
>>> Anyway, has anyone else needed similar functionality? Do you think this warrants enhancement of the Git step? Do you think support a multi-repo scenario suggests a subclass of Git instead? Any thoughts?
>> Hey Jared.  #1 is done in the CI system I built out.  You know where that repo lives.  Basically, the way it works is that it creates a mirror folder above the builder so that multiple builders on the same machine can share.
>> The first step is to always just update that mirror.  Then new clones are created with git-new-workdir (IIRC) & the appropriate revision checked out.
>> 
>> It doesn’t use the git steps as those tend to be very locked into the version they check out the first time (which prevents me from having a separate working repository for automatic merges that doesn’t impact incremental compilation).  I’ve also found git-new-workdir to be faster & use less disk-space than the reference functionality of git clone (I suspect the disk space is just a measuring error because reference uses hard links whereas git-new-workdir uses symlinks IIRC).
>> 
>> This of course still suffers the disk-space problem as you still end up with 34 repositories: if the repository is ~1GB checked out (without the .git), you’re looking at 34GB.  I’m not sure how to solve that, but I suspect it would be at the filesystem level via compression.  I’ve punted on that problem - it doesn’t seem important enough to solve at the moment for my purposes.
>> 
>> I have no input on your second point.
>> 
>> -Vitali
>> 
>>> 
>>> Jared
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming! The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>>> look and join the conversation now. http://goparallel.sourceforge.net
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> 
>> 
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming! The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel