[Buildbot-devel] Buildbot through stacktrace during SVNPoller

[William Deegan]

Dustin,
On Jun 15, 2011, at 10:19 PM, Dustin J. Mitchell wrote:

> On Tue, Jun 14, 2011 at 6:54 PM, William Deegan
> <bill at baddogconsulting.com> wrote:
>> Fulvio,
>> 
>> The machine is running buildbot + svn + a few nx hosts.
>> The startup in question doesn't have a lot of hardware and will likely be migrating everything but svn off of that machine in the near term.
>> It's centos 5.5 64 bit linux.
>> 
>> Buildbot's eating ~1GB
>> SVN+ Subversion edge (activestate) eating another 1GB
>> So that doesn't leave a whole lot for the rest.
>> 16GB should be sufficient.
> 
> At Mozilla, we've had some problems with Buildbot eating a lot of
> memory.  It's still not clear what portion of that is due to internal
> extensions to buildbot, rather than things that are in the Buildbot
> repository itself.  Running Python code (and Buildbot pulls in a lot
> of it - Twisted is not small) uses a lot of RAM, and on a 64-bit
> system just about everything in Python takes twice as much space, so
> 1GB doesn't surprise me that much.
> 
> That said, we've seen the same problem with ENOMEM from fork().  It
> turns out, fork() will not allow you to over-commit memory, so to
> fork() you need space free (between swap and RAM) equivalent to the
> virtual size of the forking process - which is a lot for the
> buildmaster.

Interesting, what about using vfork() or similar or is there no way to call vfork from python?
(It's a lazy copy fork)

-Bill