[Buildbot-devel] modules, reconfig, lazy_reload & a better alternative?

[Kay Hayen]

Hello,


Then it would be interesting, should Buildbot provide some sort of standard
>> configuration format that a general factory reads and created Buildbot
>> objects from.  But I suppose there ain't such a thing.
>>
> This discussion comes back from time to time. Its actually pretty
> difficult to create a one size fits all design that would make everybody
> happy.
>
> Coming up with some kind of IPC mechanism might be feasible, but every
> step would have to support serialization across the wire (& probably a lot
> of other stuff too).
>

I am not talking about real IPC. I am talking about this made up: master.cfg

### actual use ####
c = BuildmasterConfig = {}
from buildbot.changes.gitpoller import GitPoller
c['change_source'].append(
     GitPoller(
         'git at nuitka.net:public/Nuitka.git',
         workdir      = 'gitpoller-public',
         branches     = ['develop', 'master'],
         project      = "public",
         pollinterval = 300
     )
 )
######################

now a potential configuration generator:
######################
c = BuildmasterConfig = {}
from buildbot.factory.changes.gitpoller import GitPoller
c['change_source'].append(
     GitPoller(
         'git at nuitka.net:public/Nuitka.git',
         workdir      = 'gitpoller-public',
         branches     = ['develop', 'master'],
         project      = "public",
         pollinterval = 300
     )
 )
######################
Notice, the only change is to the import statement.
It could output:
###########
GitPoller:
         reporul     = 'git at nuitka.net:public/Nuitka.git'
         workdir      = 'gitpoller-public'
         branches     = ['develop', 'master'],
         project      = "public"
         pollinterval = 300
###########
and then we would have a master.cfg

from buildbot.factory import configreader

configreader("configfile.txt") # yaml, pickle, whatever
#####

Where the "configreader" would create objects identical to what
"master.cfg" did, with the added ability to monitor the configfile for
changes, and to update the running buildbot with them. So adding a builder,
changing URLs, adding a slave, etc. would no longer need to restart the
master.

This would be more complex to do (read more work) than anything one of us
needs, but it would be general, and maximally powerful. And it would give
the benefits of a configuration file (easy to diff maybe, checking if your
code has the intended results), with the flexibility of code.

I have e.g. code like this:

for python in ("python2.6", "python2.7", "python3.2", "python3.3",
"python3.4"):
    for test_case in ("basics", "programs", "packages", "standalone",
"syntax",
                      "optimizations", "reflected", "mercurial"):

        # These versions have no "lxml" available typically on the Debians.
        if test_case == "optimizations" and python in ("python2.6",
"python3.2"):
            continue

        # Mercurial tests only on that version.
        if test_case == "mercurial" and python != "python2.7":
            continue

        name = test_case + "-" + encodePythonVersion(python)

        if test_case in("reflected", "standalone"):
            slaves = fast_linux
        elif test_case == "mercurial":
            slaves = ["cubox-linux-slave"]
        else:
            slaves = list(all_linux)

            if python == "python2.6":
                for slave in no_python26:
                    slaves.remove(slave)

        c['builders'].append(
            BuilderConfig(
                name       = name,
                slavenames = slaves,
                factory    = makeBuildFactory(
                    "/usr/bin/" + python,
                    ["tests/" + test_case + "/" + getRunner(test_case),
"search"],
                ),
            )
        )

        test_cases.append(name)

That's probably not too special.

There is no way I would consider wanting to write up all the detail
differences out in a configuration file. On the other hand, making manual
code reloads feels icky to me, so I for now just accept loosing ca. 10-12h
of CPU test time worst cases and restart.

Yours,
Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150125/55308a47/attachment.html>