[users at bb.net] And even more multi-master anecdotes.

[Pierre Tardy]

Mmh.. sounds like you found a memory leak in the UI.

Several points to check that could be out of control for UI:
Message could be queuing up over hanged (or badly closed) websockets
REST queries data could keep some reference in some way, which would make
them just accumulate in memory.
sombody is doing huge REST apis queries. It is unfortunatly quite easy to
just hit /api/v2/builds, and dump the whole build db.
This wouldn't go all the way to 45GB IMO.

During my perf tests, I saw the logs system are quite memory hungry, and
its buildbot is trading memory for cpu if the master can't keepup recording
all the logs in the data (nothing fancy. the twisted queues will just be
growing waiting for callbacks to be called).
Due to several encoding and decoding, the memory usage in the log queue
could be 2-4 times the amount of actual log bytes processed. anyway, this
is probably not your problem.

You can look at https://github.com/tardyp/buildbot_profiler maybe improve
it to add a memory profiler..


Pierre


Le jeu. 10 août 2017 à 21:52, Neil Gilmore <ngilmore at grammatech.com> a
écrit :

> Hi all,
>
> A couple of times in the last couple weeks, the master running our UI
> has crashed/exited/ceased to run. This most recent time it was a
> MemoryError, backed up by the kernel log saying it was OOM. Current run
> shows it using 43.1G memory. Seems like a lot, doesn't it? The other 2
> larger masters (the ones actually running builds) are about 2G, and the
> remaining, smallest one is about 1.7G.
>
> What does the UI hold onto that makes it so large?
>
> I'm supposed to get some heap profiling thingy wedged into buildbot so I
> can try to figure it out. My current plan is to introduce a new build
> step (like our current custom ones) that triggers the profiling process,
> create a sort of dummy build using the step, and create a force
> scheduler to trigger that build. That'll keep it on our UI master
> anyway. Is there a better way to trigger such code in the master at
> arbitrary times?
>
> I'm also starting to get complaints (and I've noticed this myself), that
> sometimes queued builds take a long time to start after the previous
> build finishes. Sometimes on the order of a half-hour or more. Or at
> least it appears so. I'm going on what the UI tells me. I haven't tried
> delving into the logs of the master controlling the builds to match up
> times. Any ideas? I'm afraid I'm not up on exactly how/when builds are
> supposed to get started.
>
> And out database seems to be accumulating integrity errors again.
>
> Neil Gilmore
> raito at raito.com
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170810/e335a364/attachment.html>