[Buildbot-devel] modules, reconfig, lazy_reload & a better alternative?

[Vitali Lovich]

> On Jan 23, 2015, at 10:44 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
>> 
>> On Jan 23, 2015, at 10:38 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>> 
>>> 
>>> On Jan 23, 2015, at 10:34 AM, Mikhail Sobolev <mss at mawhrin.net <mailto:mss at mawhrin.net>> wrote:
>>> 
>>> Hi Vitali,
>>> 
>>> On Fri, Jan 23, 2015 at 10:15:55AM -0800, Vitali Lovich wrote:
>>>> I was trying to find a way to keep my buildbot organized into modules like a
>>>> regular project but not lose the functionality of reconfig & whatnot like one
>>>> normally would.
>>>> 
>>>> I stumbled across  https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>
>>>> <https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>> & the lazy-reload links.  However, I
>>>> was unhappy with lazy reload as it seemed like I would have to annotate that
>>>> everywhere I load one of my modules, I need to lazy-reload it.  Additionally,
>>>> it’s unclear how that works well with complex dependencies (it seems like
>>>> modules would keep getting reloaded multiple times which would have its own
>>>> headaches & possibly bugs)
>>>> 
>>>> From my prototyping, this seems like a much better solution.  Are there any
>>>> problems?  I haven’t tried it out with buildbot yet.
>>>> 
>>>> import os
>>>> import sys
>>>> 
>>>> codebase_dir = os.path.dirname(os.path.abspath(__file__))
>>>> modules_to_delete = [name for name, module in sys.modules.iteritems() if hasattr(module, '__file__') and module.__file__.startswith(codebase_dir)]
>>>> for module in modules_to_delete:
>>>>   del sys.modules[module]
>>>> 
>>>> Basically, this is in the master.cfg before I load any modules.  The
>>>> master.cfg is at the top-level directory of all other python files I might
>>>> import.  So the master.cfg basically unloads any modules that are in my
>>>> project.
>>> Meta buildbot (http://buildbot.buildbot.net <http://buildbot.buildbot.net/>) uses this approach:
>>> 
>>>   https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3 <https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3>
>>> 
>>> Do you see any problems with it?  (Obivously, the "2nd" level modules would not
>>> get reloaded unless they are explicitly marked so.)
>> Isn’t there a concern that they get loaded twice on startup?
>> 
>> Also, I do have 2nd level modules.  Having to explicitly & manually reload everything is not ideal.
>> 
>> That being said, I’m not having much luck putting the above proto-type into practice so I may have to do this.
> More specifically, my concern is my code is organized as followed (reloads added where I think they would have to be)
> 
> — master.cfg
> import configuration
> import configuration.webstatus
> import projects
> reload(configuration)
> reload(configuration.webstatus)
> reload(projects)
> 
> — configuration/__init__.py
> import webstatus
> reload(webstatus)
> 
> — projects/__init__.py
> import configuration
> reload(configuration)
> 
> From this setup, configuration gets reloaded something like 3 times unless I’m mistaken.
> Is there a better way?
Actually, I think even metabotcfg has the same problem even though it’s simpler.

If you modify common.py, nothing picks that up, no?

If you add a reload of common into the master.cfg after builders reload, then builders won’t pick up the correct value since it doesn’t reload the modules it imports, right?

So it seems like there’s an ordering problem that you’ll run into with more complicated projects...

-Vitali

> -Vitali
> 
>>> -- 
>>> Misha
>>> 
>>> ------------------------------------------------------------------------------
>>> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
>>> GigeNET is offering a free month of service with a new server in Ashburn.
>>> Choose from 2 high performing configs, both with 100TB of bandwidth.
>>> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
>>> http://p.sf.net/sfu/gigenet <http://p.sf.net/sfu/gigenet>
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150123/0e6a554c/attachment.html>