[Buildbot-devel] modules, reconfig, lazy_reload & a better alternative?

[Vitali Lovich]

I think I got my approach working but I ran into a couple of exceptions being thrown in buildbot.
I think the pull request prevents the exceptions, but I don’t know if that’s necessarily the complete fix.
It’s just the spots I encountered:
https://github.com/buildbot/buildbot/pull/1503 <https://github.com/buildbot/buildbot/pull/1503>

Would be nice if this was applied onto 0.8.x branch too - should apply cleanly.

Thanks,
Vitali

> On Jan 23, 2015, at 10:47 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
>> 
>> On Jan 23, 2015, at 10:44 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>> 
>>> 
>>> On Jan 23, 2015, at 10:38 AM, Vitali Lovich <vlovich at gmail.com <mailto:vlovich at gmail.com>> wrote:
>>> 
>>>> 
>>>> On Jan 23, 2015, at 10:34 AM, Mikhail Sobolev <mss at mawhrin.net <mailto:mss at mawhrin.net>> wrote:
>>>> 
>>>> Hi Vitali,
>>>> 
>>>> On Fri, Jan 23, 2015 at 10:15:55AM -0800, Vitali Lovich wrote:
>>>>> I was trying to find a way to keep my buildbot organized into modules like a
>>>>> regular project but not lose the functionality of reconfig & whatnot like one
>>>>> normally would.
>>>>> 
>>>>> I stumbled across  https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>
>>>>> <https://github.com/dabrahams/bbproto <https://github.com/dabrahams/bbproto>> & the lazy-reload links.  However, I
>>>>> was unhappy with lazy reload as it seemed like I would have to annotate that
>>>>> everywhere I load one of my modules, I need to lazy-reload it.  Additionally,
>>>>> it’s unclear how that works well with complex dependencies (it seems like
>>>>> modules would keep getting reloaded multiple times which would have its own
>>>>> headaches & possibly bugs)
>>>>> 
>>>>> From my prototyping, this seems like a much better solution.  Are there any
>>>>> problems?  I haven’t tried it out with buildbot yet.
>>>>> 
>>>>> import os
>>>>> import sys
>>>>> 
>>>>> codebase_dir = os.path.dirname(os.path.abspath(__file__))
>>>>> modules_to_delete = [name for name, module in sys.modules.iteritems() if hasattr(module, '__file__') and module.__file__.startswith(codebase_dir)]
>>>>> for module in modules_to_delete:
>>>>>   del sys.modules[module]
>>>>> 
>>>>> Basically, this is in the master.cfg before I load any modules.  The
>>>>> master.cfg is at the top-level directory of all other python files I might
>>>>> import.  So the master.cfg basically unloads any modules that are in my
>>>>> project.
>>>> Meta buildbot (http://buildbot.buildbot.net <http://buildbot.buildbot.net/>) uses this approach:
>>>> 
>>>>   https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3 <https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3>
>>>> 
>>>> Do you see any problems with it?  (Obivously, the "2nd" level modules would not
>>>> get reloaded unless they are explicitly marked so.)
>>> Isn’t there a concern that they get loaded twice on startup?
>>> 
>>> Also, I do have 2nd level modules.  Having to explicitly & manually reload everything is not ideal.
>>> 
>>> That being said, I’m not having much luck putting the above proto-type into practice so I may have to do this.
>> More specifically, my concern is my code is organized as followed (reloads added where I think they would have to be)
>> 
>> — master.cfg
>> import configuration
>> import configuration.webstatus
>> import projects
>> reload(configuration)
>> reload(configuration.webstatus)
>> reload(projects)
>> 
>> — configuration/__init__.py
>> import webstatus
>> reload(webstatus)
>> 
>> — projects/__init__.py
>> import configuration
>> reload(configuration)
>> 
>> From this setup, configuration gets reloaded something like 3 times unless I’m mistaken.
>> Is there a better way?
> Actually, I think even metabotcfg has the same problem even though it’s simpler.
> 
> If you modify common.py, nothing picks that up, no?
> 
> If you add a reload of common into the master.cfg after builders reload, then builders won’t pick up the correct value since it doesn’t reload the modules it imports, right?
> 
> So it seems like there’s an ordering problem that you’ll run into with more complicated projects...
> 
> -Vitali
> 
>> -Vitali
>> 
>>>> -- 
>>>> Misha
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
>>>> GigeNET is offering a free month of service with a new server in Ashburn.
>>>> Choose from 2 high performing configs, both with 100TB of bandwidth.
>>>> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
>>>> http://p.sf.net/sfu/gigenet <http://p.sf.net/sfu/gigenet>
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net <mailto:Buildbot-devel at lists.sourceforge.net>
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150123/b22baf24/attachment.html>