[Buildbot-devel] modules, reconfig, lazy_reload & a better alternative?

Pierre Tardy tardyp at gmail.com
Sat Jan 24 09:17:09 UTC 2015


Hi Vitali,
I have some experience in trying hard at reloading python modules, and this
has been very painful for me.
The main problem with reload it looks like your PoC does not take in
account is load order.

if a imports b, and a is reloaded before b, then a holds a reference to the
*old* version of b.
This ends up with lots of weird behavior and bugs.

So I would advice to give up on module reloading, and prefer to use
separate data only config files.
I recently published yamltypes which is the solution I use internally in my
masters.

https://pypi.python.org/pypi/yamltypes/1.0

Idea is that all things that you need to reconfigure are put in the yaml
files, not in python. reconfig will just load new version of yaml files,
and the problem is solved..
If you need a change to the python logic, then you'll need a restart.



Le Fri Jan 23 2015 at 21:58:01, Vitali Lovich <vlovich at gmail.com> a écrit :

> Never mind.  Found a much better & robust way using lazy-reload that
> actually looks like it works.  In my master.cfg:
>
> modules_to_reload = [module for name,
> module in sys.modules.iteritems() if hasattr(module, '__file__') and module.__file__.startswith(buildmaster_rootdir)]
>
> map(lazy_reload, modules_to_reload)
>
>
>
> import master # master.py
>
> That’s it.  No changes to buildbot.  Everything seems to reload correctly
> with all new changes.
>
> -Vitali
>
> On Jan 23, 2015, at 12:18 PM, Vitali Lovich <vlovich at gmail.com> wrote:
>
> I think I got my approach working but I ran into a couple of exceptions
> being thrown in buildbot.
> I think the pull request prevents the exceptions, but I don’t know if
> that’s necessarily the complete fix.
> It’s just the spots I encountered:
> https://github.com/buildbot/buildbot/pull/1503
>
> Would be nice if this was applied onto 0.8.x branch too - should apply
> cleanly.
>
> Thanks,
> Vitali
>
> On Jan 23, 2015, at 10:47 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>
>
> On Jan 23, 2015, at 10:44 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>
>
> On Jan 23, 2015, at 10:38 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>
>
> On Jan 23, 2015, at 10:34 AM, Mikhail Sobolev <mss at mawhrin.net> wrote:
>
> Hi Vitali,
>
> On Fri, Jan 23, 2015 at 10:15:55AM -0800, Vitali Lovich wrote:
>
> I was trying to find a way to keep my buildbot organized into modules like
> a
> regular project but not lose the functionality of reconfig & whatnot like
> one
> normally would.
>
> I stumbled across  https://github.com/dabrahams/bbproto
> <https://github.com/dabrahams/bbproto> & the lazy-reload links.  However,
> I
> was unhappy with lazy reload as it seemed like I would have to annotate
> that
> everywhere I load one of my modules, I need to lazy-reload it.
> Additionally,
> it’s unclear how that works well with complex dependencies (it seems like
> modules would keep getting reloaded multiple times which would have its own
> headaches & possibly bugs)
>
> From my prototyping, this seems like a much better solution.  Are there any
> problems?  I haven’t tried it out with buildbot yet.
>
> import os
> import sys
>
> codebase_dir = os.path.dirname(os.path.abspath(__file__))
> modules_to_delete = [name for name, module in sys.modules.iteritems() if
> hasattr(module, '__file__') and module.__file__.startswith(codebase_dir)]
> for module in modules_to_delete:
>   del sys.modules[module]
>
> Basically, this is in the master.cfg before I load any modules.  The
> master.cfg is at the top-level directory of all other python files I might
> import.  So the master.cfg basically unloads any modules that are in my
> project.
>
> Meta buildbot (http://buildbot.buildbot.net) uses this approach:
>
>   https://github.com/buildbot/metabbotcfg/blob/master/master.cfg#L3
>
> Do you see any problems with it?  (Obivously, the "2nd" level modules
> would not
> get reloaded unless they are explicitly marked so.)
>
> Isn’t there a concern that they get loaded twice on startup?
>
> Also, I do have 2nd level modules.  Having to explicitly & manually reload
> everything is not ideal.
>
> That being said, I’m not having much luck putting the above proto-type
> into practice so I may have to do this.
>
> More specifically, my concern is my code is organized as followed (reloads
> added where I think they would have to be)
>
> — master.cfg
> import configuration
> import configuration.webstatus
> import projects
> reload(configuration)
> reload(configuration.webstatus)
> reload(projects)
>
> — configuration/__init__.py
> import webstatus
> reload(webstatus)
>
> — projects/__init__.py
> import configuration
> reload(configuration)
>
> From this setup, configuration gets reloaded something like 3 times unless
> I’m mistaken.
> Is there a better way?
>
> Actually, I think even metabotcfg has the same problem even though it’s
> simpler.
>
> If you modify common.py, nothing picks that up, no?
>
> If you add a reload of common into the master.cfg after builders reload,
> then builders won’t pick up the correct value since it doesn’t reload the
> modules it imports, right?
>
> So it seems like there’s an ordering problem that you’ll run into with
> more complicated projects...
>
> -Vitali
>
> -Vitali
>
> --
> Misha
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
>
> ------------------------------------------------------------
> ------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet_______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150124/3508d6d6/attachment.html>


More information about the devel mailing list