[Buildbot-devel] modules, reconfig, lazy_reload & a better alternative?

Fri Jan 23 20:56:09 UTC 2015

Never mind.  Found a much better & robust way using lazy-reload that actually looks like it works.  In my master.cfg:

modules_to_reload = [module for name, module in sys.modules.iteritems() if hasattr(module, '__file__') and module.__file__.startswith(buildmaster_rootdir)]     
map(lazy_reload, modules_to_reload)                                                                                                                             

import master # master.py

That’s it.  No changes to buildbot.  Everything seems to reload correctly with all new changes.

-Vitali

> On Jan 23, 2015, at 12:18 PM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
> I think I got my approach working but I ran into a couple of exceptions being thrown in buildbot.
> I think the pull request prevents the exceptions, but I don’t know if that’s necessarily the complete fix.
> It’s just the spots I encountered:
> https://github.com/buildbot/buildbot/pull/1503 <https://github.com/buildbot/buildbot/pull/1503>
> 
> Would be nice if this was applied onto 0.8.x branch too - should apply cleanly.
> 
> Thanks,
> Vitali
> 
>> On Jan 23, 2015, at 10:47 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>> 
>>> 
>>> On Jan 23, 2015, at 10:44 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>>> 
>>>> 
>>>> On Jan 23, 2015, at 10:38 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>>>> 
>>>>> 
>>>>> On Jan 23, 2015, at 10:34 AM, Mikhail Sobolev <mss at mawhrin.net <mailto:mss at mawhrin.net>> wrote:
>>>>> 
>>>>> Hi Vitali,
>>>>> 
>>>>> On Fri, Jan 23, 2015 at 10:15:55AM -0800, Vitali Lovich wrote:
>>>>>> I was trying to find a way to keep my buildbot organized into modules like a
>>>>>> regular project but not lose the functionality of reconfig & whatnot like one
>>>>>> normally would.
>>>>>> 
>>>>>> I stumbled across  https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>
>>>>>> <https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>> & the lazy-reload links.  However, I
>>>>>> was unhappy with lazy reload as it seemed like I would have to annotate that
>>>>>> everywhere I load one of my modules, I need to lazy-reload it.  Additionally,
>>>>>> it’s unclear how that works well with complex dependencies (it seems like
>>>>>> modules would keep getting reloaded multiple times which would have its own
>>>>>> headaches & possibly bugs)
>>>>>> 
>>>>>> From my prototyping, this seems like a much better solution.  Are there any
>>>>>> problems?  I haven’t tried it out with buildbot yet.
>>>>>> 
>>>>>> import os
>>>>>> import sys
>>>>>> 
>>>>>> codebase_dir = os.path.dirname(os.path.abspath(__file__))
>>>>>> modules_to_delete = [name for name, module in sys.modules.iteritems() if hasattr(module, '__file__') and module.__file__.startswith(codebase_dir)]
>>>>>> for module in modules_to_delete:
>>>>>>   del sys.modules[module]
>>>>>> 
>>>>>> Basically, this is in the master.cfg before I load any modules.  The
>>>>>> master.cfg is at the top-level directory of all other python files I might
>>>>>> import.  So the master.cfg basically unloads any modules that are in my
>>>>>> project.
>>>>> Meta buildbot (http://buildbot.buildbot.net <http://buildbot.buildbot.net/>) uses this approach:
>>>>> 
>>>>>   https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3 <https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3>
>>>>> 
>>>>> Do you see any problems with it?  (Obivously, the "2nd" level modules would not
>>>>> get reloaded unless they are explicitly marked so.)
>>>> Isn’t there a concern that they get loaded twice on startup?
>>>> 
>>>> Also, I do have 2nd level modules.  Having to explicitly & manually reload everything is not ideal.
>>>> 
>>>> That being said, I’m not having much luck putting the above proto-type into practice so I may have to do this.
>>> More specifically, my concern is my code is organized as followed (reloads added where I think they would have to be)
>>> 
>>> — master.cfg
>>> import configuration
>>> import configuration.webstatus
>>> import projects
>>> reload(configuration)
>>> reload(configuration.webstatus)
>>> reload(projects)
>>> 
>>> — configuration/__init__.py
>>> import webstatus
>>> reload(webstatus)
>>> 
>>> — projects/__init__.py
>>> import configuration
>>> reload(configuration)
>>> 
>>> From this setup, configuration gets reloaded something like 3 times unless I’m mistaken.
>>> Is there a better way?
>> Actually, I think even metabotcfg has the same problem even though it’s simpler.
>> 
>> If you modify common.py, nothing picks that up, no?
>> 
>> If you add a reload of common into the master.cfg after builders reload, then builders won’t pick up the correct value since it doesn’t reload the modules it imports, right?
>> 
>> So it seems like there’s an ordering problem that you’ll run into with more complicated projects...
>> 
>> -Vitali
>> 
>>> -Vitali
>>> 
>>>>> -- 
>>>>> Misha
>>>>> 
>>>>> ------------------------------------------------------------------------------
>>>>> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
>>>>> GigeNET is offering a free month of service with a new server in Ashburn.
>>>>> Choose from 2 high performing configs, both with 100TB of bandwidth.
>>>>> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
>>>>> http://p.sf.net/sfu/gigenet <http://p.sf.net/sfu/gigenet>
>>>>> _______________________________________________
>>>>> Buildbot-devel mailing list
>>>>> Buildbot-devel at lists.sourceforge.net <mailto:Buildbot-devel at lists.sourceforge.net>
>>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel <https://lists.sourceforge.net/lists/listinfo/buildbot-devel>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150123/717c372d/attachment.html>