[Buildbot-devel] Playing nice on loaded buildslaves - sharing buildmasters between projects?

[Dan Kegel]

On Fri, Oct 19, 2012 at 2:43 AM, tom fogal <tfogal at sci.utah.edu> wrote:
> I think it's better to serialize, not parallelize: when a commit
> comes and you need to run 5 builds (on 5 slaves/VMs), start a VM, build,
> kill it, start next VM, build ... lather, rinse, repeat.

Probably.  I guess I'll man up and look at doing this with latent buildslaves
at some point.  I already asked for a clue about that on this list,
and was pointed in the right direction to start.

> ... disable Linux's overcommit policy.
>
> ... enable kernel samepage merging

Excellent suggestions.

Quick test of samepage merging on my ubuntu 12.04 host
with 4GB RAM.  I did
$ echo 1 > /sys/kernel/mm/ksm/run
$ watch "free; cat /sys/kernel/mm/ksm/pages_shared; uptime"
then forced six builds, each in a different ephemeral lxc container.
Load average climbed to 15;
swap space used started at 72080 and only climbed to 95000, but
shared pages stayed pegged at zero.  I'd like to understand
why shared pages didn't kick in, but that's beyond the scope
of this list.  (I will report back if I figure it out.)

Re my idea
> - make the build script use -j1 instead of -n$ncpus when load is high
gnu make already has a -l option which does this dynamically.
Switching my build scripts to use -j$ncores -l$ncores instead of just -j$ncores
kept the load average rather lower.

But the result of my little stress test was - nothing fell over.
Maybe I'm worrying too much; my bot seems overprovisioned
enough for the moment.  I'll keep an eye on it as I load it up
with more builds.

Thanks, all!
- Dan