[Buildbot-devel] The case for git commits and pushes from buildbot

[Kay Hayen]

Hello,

this is just a response to myself, to clarify, what I found out. So I was
using Buildbot 0.8.8 and had used to "Git" build step as from the tutorial.
This is an old style step. It has these kinds of problems, e.g. that it has
no "origin" and no "tracking" of the branch being built.

When I updated to 0.8.9, I didn't consider that the new style git step
would be all that different, which is totally untrue. The new style git
step in fact solves all these issues that I had. It is also much more space
efficient, as it only has a "build" directory, but no "source".

So, that is great. I am only have a hard time with submodules that have
submodules that got rebased. I wish, I could have better subcontrol over
which submodules are checked out anyway, as most of the time, only one or
two submodules are necessary. In fact, my submodules are more like test
cases.

For example "tests/CPython26" is only ever of interest for some specific
builders that execute my Python compiler with that test suite. Now it is
being checked out, for everything. And so far, I have attempted to take
liberty to rebase the test suite, so it represents minimal changes.

Obviously, kind of user error-ish. On the other hand, "git submodule update
--remote" is kind of some fallback that I would like to see happen there.

I am now contemplating checking out nine, because I have looked at the old
and new git steps, and kind of found that they are approachable. If git
master of buildbot turns out to be good enough, I might add the ability to
specify "submodules" as a list instead of a boolean. A non-empty list is
"True" anyway, an empty counts as "False", so as an interface change it
will be great, in that it's automatic. Let me see how that goes.

And many thanks to Dustin who educated me about the git step mess.

Yours,
Kay



2014-06-15 8:30 GMT+02:00 Kay Hayen <kay.hayen at gmail.com>:

>
> Hello,
>
> this is one of the examples, where I think it illustrates the benefit of
> git commits from buildbot, and enhanced support would be useful for it.
>
> I am driving my release process more and more from buildbot. Some builders
> are scheduler, whenever "master" or "develop" branch of my public git
> repository are updated. One set builds Debian packages. These packages are
> built with a version number as it comes from the debian/changelog file. I
> am using "reprpro", which itself has no means to track the build number.
> When finished, I push the build to the server, and it's useable as an apt
> source.
>
> Now, while many pushes to develop and master change the version number,
> not all do. Specifically when I make minor commits that do not warrant a
> release to develop, the build jobs run again, and overwrite the Debian
> packages.
>
> So, I am tasked with either storing the build number, or retriving it from
> the server that holds the latest release.
>
> I prefer to store information locally. So my solution is as follows:
>
> I am adding a git repository specifically dedicated to the Debian
> releases. In this, I am going to have 2 submodules, one being the softwares
> that are released, and one being the repositories that have been created
> from it so far. Accessing that "so far released", I can determine the
> commit last built, and the debian package build number to use, then use
> "dch" to tune the changelog to increment if necessary with automatic
> entries.
>
> The submodules for the software won't update automatically to latest
> version of the branch. I need to  automate this. In fact, in many uses of
> submodules for publishing, that will be the case. Providing environment
> variables to Buildbot, that enable it to commit under its name, will be
> working. But when it comes to "push", I don't know how to find the
> "origin", as in "build" at least, there is no origin. I have yet to check
> "source", if Buildbot has one there, but I would be surprised, as "build"
> is a copy, which is done with "cp", right?
>
> Next, the changelog update. Either I record the build number to use, or
> the generated repository, I need to update the submodule and push it as
> well. Similar as above.
>
> Were I not to store the created repository locally (which may be a good
> idea, as it allows me to quickly revert to previous versions, and gives me
> a history of the published stuff, but might be wasteful on storage and
> performance), then I would record this information NOT in a database, but
> instead in a file that is under git control.
>
> There are other such cases. I am e.g. recording performance data, for the
> speedcenter subdomain of my website, in a git repository already. Scripts
> need to update them.
>
> The downloads page updates triggered by the release builders completing,
> as well as nightly, and it updates a ".rst" file, which should be commited.
> Before "deploy", of Nikola results, I tend to want to commit to
> "output.git", which I check for surprising changes, but doing the deploy
> manual, or triggered via buildbot. Again, it would be nice to track the
> deployed state with e.g. tags, or maybe better a branch.
>
> All of this is feasible with current Buildbot, IF and only IF the "push"
> target is hardcoded. The commits itself, I found, are better done within
> the scripts that know if it's necessary, and what it's about.
>
> So, is there a way to provide the triggered repo in Buildbot. Specifally,
> I would want to have the remote "origin". I can understand how it's a good
> measure of non-corruption, to have the builders unable to access it,
> normally.
>
> Using environment variables for the task is kind of suboptimal, as I would
> like to also be able to run the scripts from standard git clones manually.
> Currently, I am thinking of plastering the builds that need it, with "git
> remote add" shell commands, but that ought to be part of the git build
> steps, and should work for submodules in an automatic way too.
>
> Yours,
> Kay
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140627/722c7746/attachment.html>