[Buildbot-devel] The case for git commits and pushes from buildbot

Fri Jun 27 16:16:57 UTC 2014

I wouldn't recommend running master in production at this point - it
has some known-missing features, and probably a lot of unknown missing
features.

However, the git step hasn't changed much since 0.8.x, so if you make
some modifications there, they should forward-port pretty easily to
master.

Dustin

On Fri, Jun 27, 2014 at 1:52 AM, Kay Hayen <kay.hayen at gmail.com> wrote:
> Hello,
>
> this is just a response to myself, to clarify, what I found out. So I was
> using Buildbot 0.8.8 and had used to "Git" build step as from the tutorial.
> This is an old style step. It has these kinds of problems, e.g. that it has
> no "origin" and no "tracking" of the branch being built.
>
> When I updated to 0.8.9, I didn't consider that the new style git step would
> be all that different, which is totally untrue. The new style git step in
> fact solves all these issues that I had. It is also much more space
> efficient, as it only has a "build" directory, but no "source".
>
> So, that is great. I am only have a hard time with submodules that have
> submodules that got rebased. I wish, I could have better subcontrol over
> which submodules are checked out anyway, as most of the time, only one or
> two submodules are necessary. In fact, my submodules are more like test
> cases.
>
> For example "tests/CPython26" is only ever of interest for some specific
> builders that execute my Python compiler with that test suite. Now it is
> being checked out, for everything. And so far, I have attempted to take
> liberty to rebase the test suite, so it represents minimal changes.
>
> Obviously, kind of user error-ish. On the other hand, "git submodule update
> --remote" is kind of some fallback that I would like to see happen there.
>
> I am now contemplating checking out nine, because I have looked at the old
> and new git steps, and kind of found that they are approachable. If git
> master of buildbot turns out to be good enough, I might add the ability to
> specify "submodules" as a list instead of a boolean. A non-empty list is
> "True" anyway, an empty counts as "False", so as an interface change it will
> be great, in that it's automatic. Let me see how that goes.
>
> And many thanks to Dustin who educated me about the git step mess.
>
> Yours,
> Kay
>
>
>
> 2014-06-15 8:30 GMT+02:00 Kay Hayen <kay.hayen at gmail.com>:
>
>>
>> Hello,
>>
>> this is one of the examples, where I think it illustrates the benefit of
>> git commits from buildbot, and enhanced support would be useful for it.
>>
>> I am driving my release process more and more from buildbot. Some builders
>> are scheduler, whenever "master" or "develop" branch of my public git
>> repository are updated. One set builds Debian packages. These packages are
>> built with a version number as it comes from the debian/changelog file. I am
>> using "reprpro", which itself has no means to track the build number. When
>> finished, I push the build to the server, and it's useable as an apt source.
>>
>> Now, while many pushes to develop and master change the version number,
>> not all do. Specifically when I make minor commits that do not warrant a
>> release to develop, the build jobs run again, and overwrite the Debian
>> packages.
>>
>> So, I am tasked with either storing the build number, or retriving it from
>> the server that holds the latest release.
>>
>> I prefer to store information locally. So my solution is as follows:
>>
>> I am adding a git repository specifically dedicated to the Debian
>> releases. In this, I am going to have 2 submodules, one being the softwares
>> that are released, and one being the repositories that have been created
>> from it so far. Accessing that "so far released", I can determine the commit
>> last built, and the debian package build number to use, then use "dch" to
>> tune the changelog to increment if necessary with automatic entries.
>>
>> The submodules for the software won't update automatically to latest
>> version of the branch. I need to  automate this. In fact, in many uses of
>> submodules for publishing, that will be the case. Providing environment
>> variables to Buildbot, that enable it to commit under its name, will be
>> working. But when it comes to "push", I don't know how to find the "origin",
>> as in "build" at least, there is no origin. I have yet to check "source", if
>> Buildbot has one there, but I would be surprised, as "build" is a copy,
>> which is done with "cp", right?
>>
>> Next, the changelog update. Either I record the build number to use, or
>> the generated repository, I need to update the submodule and push it as
>> well. Similar as above.
>>
>> Were I not to store the created repository locally (which may be a good
>> idea, as it allows me to quickly revert to previous versions, and gives me a
>> history of the published stuff, but might be wasteful on storage and
>> performance), then I would record this information NOT in a database, but
>> instead in a file that is under git control.
>>
>> There are other such cases. I am e.g. recording performance data, for the
>> speedcenter subdomain of my website, in a git repository already. Scripts
>> need to update them.
>>
>> The downloads page updates triggered by the release builders completing,
>> as well as nightly, and it updates a ".rst" file, which should be commited.
>> Before "deploy", of Nikola results, I tend to want to commit to
>> "output.git", which I check for surprising changes, but doing the deploy
>> manual, or triggered via buildbot. Again, it would be nice to track the
>> deployed state with e.g. tags, or maybe better a branch.
>>
>> All of this is feasible with current Buildbot, IF and only IF the "push"
>> target is hardcoded. The commits itself, I found, are better done within the
>> scripts that know if it's necessary, and what it's about.
>>
>> So, is there a way to provide the triggered repo in Buildbot. Specifally,
>> I would want to have the remote "origin". I can understand how it's a good
>> measure of non-corruption, to have the builders unable to access it,
>> normally.
>>
>> Using environment variables for the task is kind of suboptimal, as I would
>> like to also be able to run the scripts from standard git clones manually.
>> Currently, I am thinking of plastering the builds that need it, with "git
>> remote add" shell commands, but that ought to be part of the git build
>> steps, and should work for submodules in an automatic way too.
>>
>> Yours,
>> Kay
>>
>>
>
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>