[Buildbot-devel] Dev / test / production instances of Buildbot?

[Philippe McLean]

buildbot credibility is very important, and there are a few things I've
done to keep it 'the gold standard':

- test master.cfg with a parallel master. Allocate 1 slave that reports to
it. Use vms.
Debug the  master.cfg before you commit to version control. All masters
keep master.cfg under version control.
Use python to check the syntax.

- simplify the build process, scripts, and environment as much as possible.
Keep slaves clean. Use vms. Avoid putting any non-trivial code in
master.cfg.

- don't restart the build master when builds are running (hit it when it's
idle). otherwise you will get purple 'interrupted exception' builds.

- ruthlessly investigate and correct buildbot 'differences' or
non-determinism.

- hound developers who break the build

Buildbot will send out a lot of email, and some people will simply ignore
it, or mistrust it ("there's a bug in the compiler"). It's partly a
cultural shift, as well as doing everything we can to make the build simple
and reliable.



On Mon, Jun 18, 2012 at 8:46 AM, Trask <trask at techsoft3d.com> wrote:

> In the past 6 months my company has successfully used Buildbot to
> implement continuous builds upon check-in.  It has been a boon to
> turn-around time with build-breakages.  It is a Big Deal.  Responsible
> authors to commits get an email if they break the build.  It didn't take
> long for us fiddling with build steps or file uploads to cause a build
> to fail without it actually being the engineers fault.  Soon the worry
> was Buildbot turning into the "boy who cried wolf" and having engineers
> ignore the email notifications.
>
> Ever since then I have been much more cautious about changes to
> Buildbot; so cautious, in fact, that it hampers productivity.
>
> Has anyone set up parallel instances of their production Buildbot
> instance in order to test changes before using them in production?  I'd
> love to hear how people have done this
>
> My brain-storming with a co-worker lead me to this plan.  It seems a
> little inelegant but should serve the purpose:
>
> My proposed steps to make a "test" instance of Buildbot:
> - install a second slave on every slave that currently exists; increment
> port numbers by 1.
> - install a second master in the directory "master_test" beside the
> existing "master" location.
> - Modify master.cfg to include logic to determine if "isTestInstance."
> Right now I just plan to look at the cwd of master.cfg and see if
> "master_test" is where it lives.  If so, then it is the test instance.
> Within the rest of master.cfg, occasionally the logic will differ
> whether it is the "production" or "test" instance.  For example:
> ... email notifications enabled in production; disabled in test
> ... production: schedulers accept changeSources from git triggers; test:
> force builds only
> ... port numbers for slaves will differ
> ... webstatus port must differ; etc
>
>
> We control our Buildbot source files with git.  The "master_test"
> directory would just be another clone of our Buildbot source (primarily
> just "master.cfg" changes).  Pulling changes from the test branches to
> the master branch would be clean.  This method is not ideal for testing
> because the same config file will have different code-paths if it is on
> the test instance versus the production instance, but it is definitely
> better than testing everything in production!
>
> Does anyone have any suggestions before I implement this?  Has anyone
> else done this already?
>
> ~Trask
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20120618/77642e7e/attachment.html>