[Buildbot-devel] Multiple change sources and scheduler issues

[Axel Hecht]

2008/7/23 Dustin J. Mitchell <dustin at zmanda.com>:
> On Wed, Jul 23, 2008 at 11:16 AM, Axel Hecht <l10n.moz at googlemail.com> wrote:
>> We can name the changesources something like "l10n-af", "l10n-ar",
>> "l10n-be", but doing
>> dict(forrest="l10n", repo="af") would be prettier. So all our en-US
>> scheduler would pay attention for dict(repo="mozilla-central"), while
>> the l10n scheduler would pay attention to dict(forrest="l10n") and
>> would then set build properties based on the "repo" color to trigger
>> repacks for only a particular language.
>
> Hmm, so this is even more complex than I thought.  I thought you were
> basically doing an intersection of the keys for the colordict of a
> Change and a Scheduler, and triggering the build if the intersection
> was nonempty.  But it looks like you're treating the dictionary on a
> Change as a set of attributes, and the dictionary on a Scheduler as a
> smaller set of attributes which must match -- basically a logical AND.
>  I'm not sure how you'd implement an OR -- I think by adding a new
> attribute on the Changes.
>
> I'm resistant to this idea because it differs from the other means of
> connecting one bbot component to the next, which is based on lists of
> names.  I'm big on consistency and minimal complexity.
>
> We, too, have a large configuration (in our case with lots of various
> types of builders), but our solution is to dynamically generate the
> names in master_cfg.py.  For example, we have:
>
> for arch in all_arches:
>    c['builders'].append({
>        'name': "archtest-%s" % arch,
>        'slavenames': [ buildslave.botname
>                for buildslave in buildslaves
>                if buildslave.arch == arch ],
>        'builddir': "archtest-%s" % arch,
>        'category' : "archtest",
>        'factory': archtest_factory,
>          })
>
> I think your situation could be solved similarly by creating compound
> names for each ChangeSource (something like "%(forrest)-%(repo)") and
> then dynamically creating the lists of names passed to the scheduler.
>
> This suits your need in a complex situation, and will suit others'
> complex situations without having to puzzle out how to map their
> situation into your dict-partition model.  But it leaves the basic
> functionality simple and easy to describe.

Mine is yet more complex. I want to change the set of localizations at
runtime. Here's how Benjamin Smedberg did the working prototype:

We have one master poller for l10n, which will poll a file on hg for a
list of localizations, and then kick off a poller for each of those.
Those will then see the changes, create the Change objects and set the
locale as property, just

change.locale = self.locale

They then send them back upstream into the master. That's basically in
order to not having to touch the master just because we added a new
localization. We don't need to the change the builders or the
schedulers.

We'd like to put something upstream that is less hacky than setting
change.locale.

Right now, hgpoller uses branch to denote the repository. I find that
awkward personally as there are still named branches in hg, but let's
ignore that for a moment. So for l10n vs. en-US, we're using 'branch'
right now. As I said I wouldn't be too unhappy to be able to change
that.

I need the borders in my stuff to be semi-permeable, too. The l10n
builds should see the en-US changes, but not vice-versa.

I saw you using categories in your example, how about re-using that
concept. Instead of using dicts with arbitrary values, we could use
list of strings. I confess, it'd be functionally equivalent, but less
complex to explain.

I'm worried about calling them categories, too, as that term is used
from the builder down to the stati, and the scheduler is free to break
the category boundaries quite heavily. Which I think is good. But
something like a mirrored feature of categories for the world from the
change sources up to the schedulers.

To map that into my example above, the changes would get an ["l10n",
"ar"] for l10n, and ["mozilla-central"] (or none, "mozilla-central"
actually appears as a branch in our setup, too), or just ["ar] for
l10n, assuming that the branches notation for different repos is fine.

Axel, who's having his head turn his back on him