[Buildbot-devel] The case for git commits and pushes from buildbot

Sun Jun 22 15:21:25 UTC 2014

First, apologies for taking a week to get back to you -- this took
some time to read through :)

>From what I can tell, you've found situations where you need to be
able to push to the checked-out git repositories in your builds, but
the way Buildbot is performing the checkout is hiding the necessary
git remote.

It seems you're using mode=full and method=copy.  That uses 'cpdir' to
copy the source directory to the build directory, which *should*
include .git.  And that's copied from a srcdir which is checked out
with, effectively, mode=incremental.

The behavior there, as far as buildslave commands, is pretty similar
to the first bit of
  https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/unit/test_steps_source_git.py#L713
which does a pretty plain-vanilla 'git clone'

As far as I can see on the metabuildbot, this ends up with the remote
in .git/config as expected:

http://buildbot.buildbot.net/builders/www/builds/267/steps/git/logs/stdio
knuth bbot # cd /var/lib/buildslave/bbot/www/build
knuth build # grep -A1 '\[remote' .git/config
[remote "origin"]
  url = git://github.com/buildbot/buildbot.git

So, it sounds like you're seeing a bug here, but we'll need some more
detail to track it down.  If it is a bug, can you file a trac ticket?

Dustin

On Sun, Jun 15, 2014 at 2:30 AM, Kay Hayen <kay.hayen at gmail.com> wrote:
>
> Hello,
>
> this is one of the examples, where I think it illustrates the benefit of
> git commits from buildbot, and enhanced support would be useful for it.
>
> I am driving my release process more and more from buildbot. Some
> builders are scheduler, whenever "master" or "develop" branch of my
> public git repository are updated. One set builds Debian packages. These
> packages are built with a version number as it comes from the
> debian/changelog file. I am using "reprpro", which itself has no means
> to track the build number. When finished, I push the build to the
> server, and it's useable as an apt source.
>
> Now, while many pushes to develop and master change the version number,
> not all do. Specifically when I make minor commits that do not warrant a
> release to develop, the build jobs run again, and overwrite the Debian
> packages.
>
> So, I am tasked with either storing the build number, or retriving it
> from the server that holds the latest release.
>
> I prefer to store information locally. So my solution is as follows:
>
> I am adding a git repository specifically dedicated to the Debian
> releases. In this, I am going to have 2 submodules, one being the
> softwares that are released, and one being the repositories that have
> been created from it so far. Accessing that "so far released", I can
> determine the commit last built, and the debian package build number to
> use, then use "dch" to tune the changelog to increment if necessary with
> automatic entries.
>
> The submodules for the software won't update automatically to latest
> version of the branch. I need to  automate this. In fact, in many uses
> of submodules for publishing, that will be the case. Providing
> environment variables to Buildbot, that enable it to commit under its
> name, will be working. But when it comes to "push", I don't know how to
> find the "origin", as in "build" at least, there is no origin. I have
> yet to check "source", if Buildbot has one there, but I would be
> surprised, as "build" is a copy, which is done with "cp", right?
>
> Next, the changelog update. Either I record the build number to use, or
> the generated repository, I need to update the submodule and push it as
> well. Similar as above.
>
> Were I not to store the created repository locally (which may be a good
> idea, as it allows me to quickly revert to previous versions, and gives
> me a history of the published stuff, but might be wasteful on storage
> and performance), then I would record this information NOT in a
> database, but instead in a file that is under git control.
>
> There are other such cases. I am e.g. recording performance data, for
> the speedcenter subdomain of my website, in a git repository already.
> Scripts need to update them.
>
> The downloads page updates triggered by the release builders completing,
> as well as nightly, and it updates a ".rst" file, which should be
> commited. Before "deploy", of Nikola results, I tend to want to commit
> to "output.git", which I check for surprising changes, but doing the
> deploy manual, or triggered via buildbot. Again, it would be nice to
> track the deployed state with e.g. tags, or maybe better a branch.
>
> All of this is feasible with current Buildbot, IF and only IF the "push"
> target is hardcoded. The commits itself, I found, are better done within
> the scripts that know if it's necessary, and what it's about.
>
> So, is there a way to provide the triggered repo in Buildbot.
> Specifally, I would want to have the remote "origin". I can understand
> how it's a good measure of non-corruption, to have the builders unable
> to access it, normally.
>
> Using environment variables for the task is kind of suboptimal, as I
> would like to also be able to run the scripts from standard git clones
> manually. Currently, I am thinking of plastering the builds that need
> it, with "git remote add" shell commands, but that ought to be part of
> the git build steps, and should work for submodules in an automatic way too.
>
> Yours,
> Kay
>
>
>
> ------------------------------------------------------------------------------
> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
> Find What Matters Most in Your Big Data with HPCC Systems
> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
> http://p.sf.net/sfu/hpccsystems
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel