[users at bb.net] Suggestions for Configuration

[Povilas Kanapickas]

Hi Jan-Benedict,

> `---> ...or how is something "like that" done The Buildbot Way? Hints
> to docs or actual configurations are welcome, as well as hints to show
> up completely different concepts or approaches to my builts.

The reason why there are few known best practices is that Buildbot is
flexible, as the master configuration is just regular Python code. So
most of the time it just mirrors the requirements of the user which has
a huge variation.

I will reply to the rest of the email inline

>  Goals for a Buildbot Setup
> ============================
>   * Preferrably keep the scheme of job-generating jobs.

Buildbot master configuration is just a regular Python script, so as
long as it doesn't run any long-running tasks, it can run any code
needed to build the set of jobs that can be run. You can parse files and
so on.

>   * Use multiple worker hosts. (All shall be similar, ie. have access
>     to the very identical Docker containers or NetBSD Qemu VMs.)

This is easy, just run multiple workers on multiple hosts.

>   * All jobs are ment to build single-threaded (to keep the log output
>     as stable as possible), so on an usual assigned worker host, there
>     can be many worker instances running in parallel, each offering
>     one dockerized build environment or a Qemu VM (for running NetBSD.)

The simpliest is to run e.g. 8 workers on a single host and limit access
to them via locks.

>   * Put the generated Docker containers somewhere central and let the
>     workers pull from there. (Probably easy, but I've never done
>     that.)

Yes, just use docker-registry.

>   * Clone GIT repos via network instead from a locally-available
>     directory. Alternatively: Mirror all repos to the build hosts?

Cloning via network is the default. It's possible to configure Buildbot
to try to reuse as much as possible of existing checkouts, or to get rid
of full repository every time.

>   * Centrally host the "thick" artifacts (binaries) as well as the
>     small stuff (test results etc.)

This is out of scope of Buildbot. The buildbot builds can add URLs to
the web UI so any content hosted by a third-party can be easily accessed.

>   * Give workers a way to fetch the "thick" artifacts (ie. the Linux
>     builds need access to the toolchain tarballs.)

There are Buildbot build steps that do exactly that. It's also always
possible to just wget from within a build script.

>  Nice-to-have Goals
> ~~~~~~~~~~~~~~~~~~~~
> With my current setup, I find lots of issues. Most often, I report
> them to the person that "broke" it, or publically replying to the
> submitted patch. I've got some scripts to help with tracking down
> offending commits as I usually am too lazy to look through the commits
> since the last successful build if that's more than a few commits.
> 
>   * For GCC/gas/binutils/gdb/simh, I've got a script to hand in a
>     last-known-good and a known-bad commit id along with a modulo
>     number. That way, every n'th commit between [good,bad] is queued
>     as kind of a slightly parallelized `git bisect`. (Actually, I'm
>     often using `git bisect` along with this.)

Are the failures already reproducible on stock build configuration or do
they need additional custom setup, e.g. a patch containing a new test
that properly reproduces a bug. Both of these could be supported by
Buildbot.

The bisect functionality itself would not live within Buildbot itself
thought and would instead be just a regular script on user machine,
which would then interact with Buildbot via its API.

Though it would be interesting project to implement an inbuilt bisect
functionality into Buildbot itself.

> 
>   * For Linux jobs, I can queue in jobs in a similar way, but for
>     either Linux kernel commit IDs _or_ for a list of self-built
>     compilers to test. (That's why all "thick" build results have the
>     commit id in their filename and a symlink is done IFF this is a
>     fresh master build.)

Buildbot could handle these by using custom path or download URL in case
it's passed as a build parameter.

>   * How to I build variants of a generic job?
> 
>   * How can I run the build script inside a Docker container (that's a
>     fresh one on every invocation?)

While buildbot does not have inbuilt steps for this functionality, one
can just create a `ShellCommand` which runs `docker run --rm ...`. Would
that suffice?

>   * How do I spread /n/ instances on /m/ hosts? (With possibly
>     different /n/ as there might be a different number of CPUs or
>     insufficient RAM to serve as many jobs as there are CPUs.)

One would create as many workers as the number of concurrent builds
needed and then setup locks. For example (pseudocode, will probably
contain syntax errors)

host_locks = {
    'host1': util.MasterLock("host1", maxCount=32),
    'host2': util.MasterLock("host2", maxCount=16),
    'host3': util.MasterLock("host2", maxCount=8),
}

Then, when declaring a builder:

@util.renderer
def get_builder_locks(props):
    return [l.access('counting')
            for l in host_locks.get(props.getProperty("host_name"), [])]

...

builder = BuilderConfig(..., locks=get_builder_locks)

>   * How can I prepare Docker containers centrally and push/pull them
>     to the worker nodes?

docker-registry.

Hope the above cleared things up.

Thanks,
Povilas