[Buildbot-devel] The case for git commits and pushes from buildbot

[Kay Hayen]

Hello,

this is one of the examples, where I think it illustrates the benefit of 
git commits from buildbot, and enhanced support would be useful for it.

I am driving my release process more and more from buildbot. Some 
builders are scheduler, whenever "master" or "develop" branch of my 
public git repository are updated. One set builds Debian packages. These 
packages are built with a version number as it comes from the 
debian/changelog file. I am using "reprpro", which itself has no means 
to track the build number. When finished, I push the build to the 
server, and it's useable as an apt source.

Now, while many pushes to develop and master change the version number, 
not all do. Specifically when I make minor commits that do not warrant a 
release to develop, the build jobs run again, and overwrite the Debian 
packages.

So, I am tasked with either storing the build number, or retriving it 
from the server that holds the latest release.

I prefer to store information locally. So my solution is as follows:

I am adding a git repository specifically dedicated to the Debian 
releases. In this, I am going to have 2 submodules, one being the 
softwares that are released, and one being the repositories that have 
been created from it so far. Accessing that "so far released", I can 
determine the commit last built, and the debian package build number to 
use, then use "dch" to tune the changelog to increment if necessary with 
automatic entries.

The submodules for the software won't update automatically to latest 
version of the branch. I need to  automate this. In fact, in many uses 
of submodules for publishing, that will be the case. Providing 
environment variables to Buildbot, that enable it to commit under its 
name, will be working. But when it comes to "push", I don't know how to 
find the "origin", as in "build" at least, there is no origin. I have 
yet to check "source", if Buildbot has one there, but I would be 
surprised, as "build" is a copy, which is done with "cp", right?

Next, the changelog update. Either I record the build number to use, or 
the generated repository, I need to update the submodule and push it as 
well. Similar as above.

Were I not to store the created repository locally (which may be a good 
idea, as it allows me to quickly revert to previous versions, and gives 
me a history of the published stuff, but might be wasteful on storage 
and performance), then I would record this information NOT in a 
database, but instead in a file that is under git control.

There are other such cases. I am e.g. recording performance data, for 
the speedcenter subdomain of my website, in a git repository already. 
Scripts need to update them.

The downloads page updates triggered by the release builders completing, 
as well as nightly, and it updates a ".rst" file, which should be 
commited. Before "deploy", of Nikola results, I tend to want to commit 
to "output.git", which I check for surprising changes, but doing the 
deploy manual, or triggered via buildbot. Again, it would be nice to 
track the deployed state with e.g. tags, or maybe better a branch.

All of this is feasible with current Buildbot, IF and only IF the "push" 
target is hardcoded. The commits itself, I found, are better done within 
the scripts that know if it's necessary, and what it's about.

So, is there a way to provide the triggered repo in Buildbot. 
Specifally, I would want to have the remote "origin". I can understand 
how it's a good measure of non-corruption, to have the builders unable 
to access it, normally.

Using environment variables for the task is kind of suboptimal, as I 
would like to also be able to run the scripts from standard git clones 
manually. Currently, I am thinking of plastering the builds that need 
it, with "git remote add" shell commands, but that ought to be part of 
the git build steps, and should work for submodules in an automatic way too.

Yours,
Kay