[Buildbot-devel] Dev / test / production instances of Buildbot?

[Trask]

In the past 6 months my company has successfully used Buildbot to 
implement continuous builds upon check-in.  It has been a boon to 
turn-around time with build-breakages.  It is a Big Deal.  Responsible 
authors to commits get an email if they break the build.  It didn't take 
long for us fiddling with build steps or file uploads to cause a build 
to fail without it actually being the engineers fault.  Soon the worry 
was Buildbot turning into the "boy who cried wolf" and having engineers 
ignore the email notifications.

Ever since then I have been much more cautious about changes to 
Buildbot; so cautious, in fact, that it hampers productivity.

Has anyone set up parallel instances of their production Buildbot 
instance in order to test changes before using them in production?  I'd 
love to hear how people have done this

My brain-storming with a co-worker lead me to this plan.  It seems a 
little inelegant but should serve the purpose:

My proposed steps to make a "test" instance of Buildbot:
- install a second slave on every slave that currently exists; increment 
port numbers by 1.
- install a second master in the directory "master_test" beside the 
existing "master" location.
- Modify master.cfg to include logic to determine if "isTestInstance." 
Right now I just plan to look at the cwd of master.cfg and see if 
"master_test" is where it lives.  If so, then it is the test instance. 
Within the rest of master.cfg, occasionally the logic will differ 
whether it is the "production" or "test" instance.  For example:
... email notifications enabled in production; disabled in test
... production: schedulers accept changeSources from git triggers; test: 
force builds only
... port numbers for slaves will differ
... webstatus port must differ; etc


We control our Buildbot source files with git.  The "master_test" 
directory would just be another clone of our Buildbot source (primarily 
just "master.cfg" changes).  Pulling changes from the test branches to 
the master branch would be clean.  This method is not ideal for testing 
because the same config file will have different code-paths if it is on 
the test instance versus the production instance, but it is definitely 
better than testing everything in production!

Does anyone have any suggestions before I implement this?  Has anyone 
else done this already?

~Trask