[Buildbot-devel] Going beyond the tutorial; ideas for simplifying larger configurations

[Tom Prince]

"Dustin J. Mitchell" <dustin at v.igoro.us> writes:

> The danger in having one "preferred" method is that we pick it too
> soon.

I certainly agree that it is much too soon to merge anything like this
into buildbot.

> You pointed to three *very* different approaches to this
> problem, and there are lots of others.

Well, but they all have the same basic shape: take some configuration
data and run some code on it.

So: something like

class IConfigurator(Interface):
      def processConfig(BuildmasterConfig, data):
          pass

class MozBB: implements(IConfigurator) ...
class Metabot: implements(IConfigurator) ...
class TwistedBB: implements(IConfigurator) ...


config.yml:
configurator: MozBB
   ....

Except that isn't fined grained enough for what I am imagining.

metabot for example, has a list of slaves and their properties, and then
creates families of builders based on that. For example, some slaves
have a slave-specific builder that just use a simple configuration. Some
slaves have various dbs, and so get added to builders that test against
those specifc dbs. For each os that is present there is a builder create
that uses all the appropriate slaves.

Now, all those things are specific to buildbot, but they look like
- for all slaves that have this property, create a builder
- for all values of this property, create a builder that uses any of the
  slaves with it
- for all slaves with this property, create this family of builders that
  will run on any of the slaves
- etc...

I haven't looked in enough at mozilla's configuration, but I suspect
that at least in part all those patterns appear.

> At least initially, I'd like to have a number of frameworks to choose
> from -- just as there are a number of Python web frameworks out there.
> If one turns out to be the clear winner -- and there are a number of
> characteristics to judge on -- then we can at least make it
> "recommended", and perhaps include it in the main distribution too.

I think that there is (or at least could be), and that the common parts
should be moved into buildbot.

> I don't like using big words like "ecosystem", but I think that's what
> I'm looking for here.

I agree that this is a desirable thing.

Georges Racinet <gracinet at anybox.fr> writes:
> In my experience with other systems, configuration files for tools 
> intending to be generic tend to become bloated and much more complicated 
> than programmatic configuration. When it appears, everyone's happy, 
> because people tend to think code is more complicated than 
> configuration. Over the years, the configuration format has to 
> accommodate more and more use-cases and becomes quite intricated.

Certainly a unified config format that has all the power of buildbot
would be completely unwieldly (hence the programmatic
configuration). What I have noticed, though, is that there is still a
large amount of non-programmatic configuration, but that it isn't always
well factored from the programmatic parts. What I am advocating is
encouraging a clear seperation.

One benefit of this, is that if there is a clear seperation, and
buildbot is aware of the seperation, is that if one wants to reload
simply the delcarative part of the configuration, there is no issue with
reloading python code, and so the reconfig should be much more resilient.

> To summarize, I'd be all in favor of a series of standard, officially 
> endorsed configurators (with web UI) for clearly delimited cases, while 
> resisting the pression to bloat : new use-cases should either stay 
> programmatic, or become new configurators on a per-case basis.

Well, so given that we want to support multiple configurators, it seems
reasonable that buildbot also bee a framework for building configurators
for buildbot.

  Tom