[users at bb.net] Convert buildbot 0.8 sqlite database to postgresql

[Ryan Schmidt]

> On Oct 17, 2016, at 3:19 AM, Pierre Tardy <tardyp at gmail.com> wrote:
> 
> Hello.
> On buildbot eight, the database only store information of the build requests before the build is started.
> 
> Switching to a new database you will only lose the buildrequests and not the builds which are stored in pickle files.

So, if I wait until all scheduled builds have been completed and the builders are all idle, I could then stop the system, switch the config to postgresql, start the system, and not have lost any information about prior builds? If so, that would be fine.


> There is no tool to migrate data from db to db except what you may find on the internet with pure SQL dumps. That should work as the database model is exactly the same.
> 
> For your scaling problems i fear there are few hopes that switching db will help a lot as the db is not used that much in eight. Multimaster is the usual solution to scaling. buildbot nine helps a lot to simplify the setup for multimaster config.

I haven't looked into multimaster, but the benefits listed here don't seem to apply in our situation:

http://docs.buildbot.net/current/manual/cfg-global.html#multi-master-mode

We only have a single worker handling each possible build on each OS version.


We use buildbot to build binaries of ports for MacPorts. When a new version of macOS is released and we set up a new buildbot worker and builder for it, we need to build all 20,000 ports on it.

With our old buildbot setup, we would force a single build that would build all 20,000 ports. This build would take several weeks to complete. If a problem was encountered that caused the build to abort, we had to start over. And when it did complete, the huge log that was generated caused an out of memory condition in buildbot, but not before making the web interface unresponsive for a few minutes.

With our new buildbot setup, the single build that gets forced in turn triggers individual builds for each port on another builder. We haven't tried doing this for all 20,000 ports yet, but I have tried doing around 3,000 ports at once, and the process of triggering the individual builds took several minutes, during which the web interface would become unresponsive. Also, looking at a builder page for a builder that has hundreds of scheduled builds can take a long time to display.