[users at bb.net] Capability system

Fri Apr 1 16:34:39 UTC 2016

On 03/27/2016 05:35 PM, Georges Racinet wrote:
> I have now evidence that our capability system works with Nine (with
> actually almost no change).
> Now for our own sake, I will separate from our actual build
> definitions in a git repo of its own, and see if I can make nice
> examples based on meta buidbot and SimpleConfig use-cases.
So, here it is : https://github.com/anybox/anybox.buildbot.capability
(please, don't bother about the name itself, it's just an easy way for
me make sure it won't conflict with anything else).

As it's been said before, it's a system to declare "capabilities" on
workers, and spawn automatically builders according to requirements or
multiplication (build_for) directives. This is "static" because this
happens at buildmaster startup or reconfiguration time. There's a full
explanation at http://docs.anybox.fr/anybox.buildbot.capability/master

It's now in production on Anybox's buildbots for public projects (can't
show until I've found a safe way to re-enable dual anonymous & auth access).

Now, we could either (with increasing preference on my part)

- include it in contrib/
- leave it and publish on PyPI as-is
- publish it separately on PyPI as maybe buildbot_capability ?
- include it in the core (Pierre seemed to favor that, but how precisely ?)

I don't favor inclusion in contrib because if I understand it correctly,
its tests would not enter the general buildbot test suite (and it has
very good coverage).

Also, I've taken a look at contrib/SimpleConfig.py, but its examples
don't seem to be a good match to display the multiplication feature,
because the build variants are based on arbitrary tags, not on versions
(there's an example with workers having two Ubuntu versions, but no
build uses both of them if I got it right). Those examples could be
rewritten using the filtering requirements, but that actually makes it
less simple.
On the other hand, this looks closer to what meta-buildbot might need
(looked really too quick to be sure)

Regards,

>
> Pierre, I don't forget the dynamic variant, it's just simply farther
> away, given what we already have.
>
> Regards,
>
> On 03/23/2016 03:39 PM, Dan Kegel wrote:
>>
>> fwiw, I'm using a simple static capability system (the OS field in
>> https://github.com/buildbot/buildbot/blob/master/master/contrib/SimpleConfig.py
>> ), and am thinking of extending it.  So I'm interested in the
>> subject, too.
>>
>> On Mar 23, 2016 5:27 AM, "Georges Racinet" <gracinet at anybox.fr
>> <mailto:gracinet at anybox.fr>> wrote:
>>
>>     On 03/23/2016 01:07 PM, Pierre Tardy wrote:
>>>
>>>
>>>     Le mar. 22 mars 2016 à 22:07, Georges Racinet
>>>     <gracinet at anybox.fr <mailto:gracinet at anybox.fr>> a écrit :
>>>
>>>         Hi there,
>>>
>>>         After a long time during which we couldn't do much more than
>>>         maintaining
>>>         our instances, we (Anybox) are considering migrating our
>>>         buildbots to Nine.
>>>
>>>
>>>     This is good news!
>>>      
>>>
>>>
>>>         Among the many things that I've developed for our needs [1],
>>>         there is a
>>>         rather complete capability system [3], and hence I must
>>>         decide whereas
>>>         it's worth porting it or not [4]. I know it's a fairly
>>>         common need that many
>>>         development shops have been doing in private or almost, but
>>>         it's not
>>>         obvious to me what is freely available (for Nine) on this topic.
>>>
>>>         http://trac.buildbot.net/ticket/3120 does not get to a
>>>         conclusion, and I
>>>         couldn't find much on it in a quick search of recent
>>>         messages on this
>>>         list. Maybe I missed something else ?
>>>
>>>
>>>     Since that, I figured Workers have properties, which can be used
>>>     in NextSlave callback in order to match any capability.
>>>     I think it would be much better to have something in the core,
>>>     not depending on having the users writing a capability specific
>>>     nextSlave/nextBuild callbacks.
>>>
>>>     Actually, there are two aspect in the capability problem. The
>>>     static and the dynamic.
>>>
>>>     1\ Static problem is the easiest. It looks this is what you
>>>     implemented already (and metabuildbot has in a smaller extend):
>>>     - One part of your config describe the workers, and their
>>>     capabilities
>>>     - Another part is describing the builders, and their required
>>>     capability.
>>>     The system will then automatically configure which worker to
>>>     associate to which builders.
>>     Yes, that is exactly what I'm doing. It's a bit more actually,
>>     since our system also spawns builder variants (applying the same
>>     BuildFactory several times for different versions of some
>>     capabilities).
>>      
>>     The full capability description is stored as a dict-valued Worker
>>     property, something you may very well call a hack.
>>     Then at build time, a custom step extracts the needed properties
>>     for that build. This is good enough for us, but it is intrusive :
>>     the step must be introduced in the build sequence.
>>>
>>>     2\ The dynamic problem. Slave are chosen at the start build phase.
>>>     - One part of your config describe the workers, and their
>>>     capabilities
>>>     - Another part is describing the builders, and their required
>>>     capability.
>>>     - Each build requests can require an additional set of
>>>     capabilities, which can restrict more the set of slaves which
>>>     can run them.
>>>     In order to implement that efficiently, one will have to mess
>>>     around inside the buildrequest distributor code, which I can
>>>     understand looks scary.
>>     I just took a quick tour of that for an unrelated reason
>>     (http://trac.buildbot.net/ticket/3498), yes it's a bit convoluted.
>>
>>>
>>>     I am not sure which one you are needing/implementing.
>>>     I think it is fine that you only implement 1/ if this is what
>>>     you need, but I would like that the design opens the door to
>>>     implementing 2 without need to rewrite everything.
>>     Interesting. I think I didn't know about nextWorker at the time I
>>     started this. What concrete use-case do you have in mind for 2/  ?
>>     Also, if the idea is that users don't need to implement 
>>     nextWorker/nextBuild, then it means they may implement it for
>>     other purposes, and it follows that the dynamic dispatching must
>>     play nice with that, isn't it ?
>>
>>     Regards,
>>
>>     -- 
>>     Georges Racinet
>>     Anybox SAS, http://anybox.fr
>>     Téléphone: +33 6 51 32 07 27 <tel:%2B33%206%2051%2032%2007%2027>
>>     GPG: 0x33AB0A35, sur serveurs publics
>>
>>
>>     _______________________________________________
>>     users mailing list
>>     users at buildbot.net <mailto:users at buildbot.net>
>>     https://lists.buildbot.net/mailman/listinfo/users
>>
>
>
> -- 
> Georges Racinet
> Anybox SAS, http://anybox.fr
> Téléphone: +33 6 51 32 07 27
> GPG: 0x33AB0A35, sur serveurs publics

-- 
Georges Racinet
Anybox SAS, http://anybox.fr
Téléphone: +33 6 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160401/4a7e4848/attachment.html>