[Buildbot-devel] Dev / test / production instances of Buildbot?

[Philippe McLean]

more suggestions:

- log everything in build steps
- give builders meaningful names
- give build steps meaningful names
- make build steps executable outside buildbot
- teach developers how to navigate buildbot and track down failures
- look for and prevent common build failures (forgetting to check in
changed files, forgetting to add new files)
- treat the build and buildbot as you would any other mission-critical
online software component


On Mon, Jun 18, 2012 at 9:29 AM, Philippe McLean
<philippe.mclean at gmail.com>wrote:

> buildbot credibility is very important, and there are a few things I've
> done to keep it 'the gold standard':
>
> - test master.cfg with a parallel master. Allocate 1 slave that reports to
> it. Use vms.
> Debug the  master.cfg before you commit to version control. All masters
> keep master.cfg under version control.
> Use python to check the syntax.
>
> - simplify the build process, scripts, and environment as much as
> possible. Keep slaves clean. Use vms. Avoid putting any non-trivial code in
> master.cfg.
>
> - don't restart the build master when builds are running (hit it when it's
> idle). otherwise you will get purple 'interrupted exception' builds.
>
> - ruthlessly investigate and correct buildbot 'differences' or
> non-determinism.
>
> - hound developers who break the build
>
> Buildbot will send out a lot of email, and some people will simply ignore
> it, or mistrust it ("there's a bug in the compiler"). It's partly a
> cultural shift, as well as doing everything we can to make the build simple
> and reliable.
>
>
>
> On Mon, Jun 18, 2012 at 8:46 AM, Trask <trask at techsoft3d.com> wrote:
>
>> In the past 6 months my company has successfully used Buildbot to
>> implement continuous builds upon check-in.  It has been a boon to
>> turn-around time with build-breakages.  It is a Big Deal.  Responsible
>> authors to commits get an email if they break the build.  It didn't take
>> long for us fiddling with build steps or file uploads to cause a build
>> to fail without it actually being the engineers fault.  Soon the worry
>> was Buildbot turning into the "boy who cried wolf" and having engineers
>> ignore the email notifications.
>>
>> Ever since then I have been much more cautious about changes to
>> Buildbot; so cautious, in fact, that it hampers productivity.
>>
>> Has anyone set up parallel instances of their production Buildbot
>> instance in order to test changes before using them in production?  I'd
>> love to hear how people have done this
>>
>> My brain-storming with a co-worker lead me to this plan.  It seems a
>> little inelegant but should serve the purpose:
>>
>> My proposed steps to make a "test" instance of Buildbot:
>> - install a second slave on every slave that currently exists; increment
>> port numbers by 1.
>> - install a second master in the directory "master_test" beside the
>> existing "master" location.
>> - Modify master.cfg to include logic to determine if "isTestInstance."
>> Right now I just plan to look at the cwd of master.cfg and see if
>> "master_test" is where it lives.  If so, then it is the test instance.
>> Within the rest of master.cfg, occasionally the logic will differ
>> whether it is the "production" or "test" instance.  For example:
>> ... email notifications enabled in production; disabled in test
>> ... production: schedulers accept changeSources from git triggers; test:
>> force builds only
>> ... port numbers for slaves will differ
>> ... webstatus port must differ; etc
>>
>>
>> We control our Buildbot source files with git.  The "master_test"
>> directory would just be another clone of our Buildbot source (primarily
>> just "master.cfg" changes).  Pulling changes from the test branches to
>> the master branch would be clean.  This method is not ideal for testing
>> because the same config file will have different code-paths if it is on
>> the test instance versus the production instance, but it is definitely
>> better than testing everything in production!
>>
>> Does anyone have any suggestions before I implement this?  Has anyone
>> else done this already?
>>
>> ~Trask
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20120618/a41cda23/attachment.html>