[Buildbot-devel] Re: don't pickle me! (+ bug)

[Brian Warner]

Stephen Davis writes (about an exception):
>          cPickle.PicklingError: Can't pickle __builtin__.MyLookup: 
> attribute lookup __builtin__.MyLookup failed
> 
> The master shutdown process is trying to pickle my cvs username -> 
> email address lookup object.  Should it be doing that if the object 
> doesn't support IPersistable?  Is there some property I can set to say 
> "don't picke me" ???

Two separate things, either of which would fix the problem. One, any classes
which provide instances that might get serialized needs to be in a separate
file and 'import'ed into your master.cfg file. This gives the class a module
name, which gets saved in the pickle, which gets used to 'import' that module
in at reload-time. For example:

mysupport/mystuff.py:
 class MyLookup(): ...

master.cfg:
 from mystuff import MyLookup
 ...

PYTHONPATH=/path/to/mysupport twistd -f buildbot.tap

the 'import' command will search PYTHONPATH for 'mystuff.py', parse it, then
pull the 'MyLookup' class into the namespace. The object will be pickled with
'mystuff.MyLookup' as its fully-qualified classname. When the pickle is
reloaded, the unpickler will do the equivalent of 'from mystuff import
MyLookup' to get the class for the new object.

That said, the second thing is that starting with 0.6.0, you probably
shouldn't be saving the buildmaster's shutdown pickle at all. The build
history has been completely moved into external files, as well as the change
history. So it probably makes sense to run twistd with the '-o' or
'--no_save' flag to inhibit creation of the -shutdown.tap file.

The only things that are persisted in 0.6.0 are status objects (from
buildbot/status/builder.py) and Change objects. BuildFactories and Builds
(and BuildSteps) might be persisted in a future version, when we figure out
how to resume interrupted builds. We're also going to add some long-term
statistics, but those will probably just be pickled dictionaries rather than
actual instances.


(The more specific answer is that you can mark classes as unpicklable by
having them inherit from twisted.persisted.styles.Ephemeral, but that just
emits a warning at save/load time. You could also have their __getstate__
method raise an exception, which would be a bit more forceful sort of "do not
pickle me" mark. However, you usually want to find out who is referring to
the Ephemeral object and give them the responsibility for not including the
child, again in their __getstate__ method.)

> There's also a bug in changes/mail.py at line 294 of TOT.  Causes the 
> web server to choke if you click the "changes" link and you're using a 
> maildir-based source.
> 
>      def describe(self):
>          return "%s mailing list in maildir %s" % (self.name,
>                                                    self.maildir.where)
> 
> 	FIX: self.maildir.where ==> self.basedir

Fixed in CVS.. thanks.

 -Brian