[Buildbot-devel] Scaling buildbot

[Dan Kegel]

I'm facing long response times on our buildbot instance (a naive one
configured with
the default local database).  It can take a minute or more to refresh
the waterfall
or to wget a log (it just now took 78 seconds to fetch a 22 megabyte log file),
and its response time to webhooks is so slow that gitlab by default resends
them, causing hilarity.  And the git polling time has to be set high
(six minutes)
to avoid swamping the git server... and even then, buildbot master grinds to a
halt any time it's doing git polling (that's when the long response times hit).

Ideas so far, in order of easiest to hardest:
- stop running a slave on the same machine as the master (already
done, but didn't help much)
- stop using git polling and switch everything to webhooks (will have
to wait for our gitlab instance to come back from the dead first)
- switch to MySQL
- add a master (doc suggests splitting duties where one master handles
slaves, 2nd master handles everything else)

Am I missing anything?

I'm looking forward to buildbot getting better at reporting metrics on
response times.  (Or does it already do that?)