<div dir="ltr"><div>Thanks Neil. That's good to know that buildbot is supporting this much workers easily. Also, rjarry mentioned over IRC that he is using buildbot for 500+ builders in single-master config with a startup time of less than 10s. So, number of builders (~200) doesn't seems like an issue in my case.<br><br><div>Also, tardyp pointed out on IRC that <b>ROLLBACK is not the problem</b>, it's probably normal behavior for selects with sqlalchemy (<a href="https://stackoverflow.com/questions/7559570/make-sqlalchemy-commit-instead-of-rollback-after-a-select-query" target="_blank">link</a>). I confirmed this by creating a fresh new postgresql database, and I still saw ROLLBACKs.<br><br></div><div>I am still trying to debug the performance issues. I tried removing schedulers, but that didn't help in performance.<br></div>
<br></div><div>Also, I have ~25000 pending buildrequests (since I have test instance running for a while). Can this be the cause of performance issues (and large startup time i am seeing)?<br></div><div><br></div><div>Thanks<br></div>Aakash<br><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 7:33 AM, Neil Gilmore <span dir="ltr"><<a href="mailto:ngilmore@grammatech.com" target="_blank">ngilmore@grammatech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
While we are using a multi-master system, one of our masters has at
least that many builders. I think we have on the order of 178
workers on that master, and at least 2 builders on each worker (most
have 3 or more). We also use a postgresql database and I've never
seen that as a source of any of our problems.<br>
<br>
Starting that master doesn't take more than 30 seconds, and probably
less (I haven't timed it). Reconfigs take a lot longer.<br>
<br>
In fact we had a reconfig take more than 17K seconds over the
weekend. It never finished, because I killed and restarted it. Users
were getting annoyed that none of their builds were in the UI.<br>
<br>
Neil Gilmore<br>
<a href="http://grammatech.com" target="_blank">grammatech.com</a><div><div class="m_5063189424227819344gmail-h5"><br>
<br>
<div class="m_5063189424227819344gmail-m_-2380960802746973949moz-cite-prefix">On 6/27/2017 8:30 PM, Aakash Jain
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="m_5063189424227819344gmail-h5">
<div dir="ltr">
<div>Hi Everyone,<br>
<br>
</div>
<div>I am working on migrating our buildbot instance from 0.8.12
to 0.9.8. I did not have performance issue with 0.8.12.
However with buildbot 0.9 (with the same configuration), I am
seeing a lot of performance issues, mostly related to
database. I was expecting a performance improvement since
buildbot 0.9 moved from pickles to database (<a href="https://medium.com/@tardyp31/d0d41bba07e1" target="_blank">and designed for
scalability</a>), however I am noticing the reverse.<br>
</div>
<div><br>
</div>
<div>My buildbot configuration is:<br>
</div>
<div>- buildbot 0.9.8<br>
</div>
<div>- single master<br>
</div>
<div>- postgresql db<br>
</div>
<div>- 200+ builders (queues)<br>
</div>
<div>- only 2 workers connected currently for testing<br>
</div>
<div>- decent hardware on master (8 gb ram, 8 cpu)<br>
<br>
<br>
</div>
<div>"buildbot start" is taking ~8 minutes (the start time
increased gradually over last few weeks). Also loading various
buildbot webpages are slow (many times webpage loads, but the
build data load after ~30s). During this time there isn't too
much cpu/memory/IO load on master.<br>
<br>
I enabled postgresql logs and I see extremely large number of
SELECT statements in the logs. I also see a large number of
"ROLLBACK" in postgresql logs. I have few questions:<br>
<br>
</div>
<div>1) Why I am seeing so many ROLLBACK in the postgresql logs?
Any suggestions to fix/debug it?<br>
</div>
<div><br>
</div>
<div>2) From postgresql logs, I notice that buildbot is running
database queries in a loop fetching data for individual
builder/build/ (resulting in large number of queries).
Wouldn't it be more efficient to use a single database query
to fetch data for all the builders and then filter it in
python code?<br>
</div>
<div><br>
</div>
<div>3) has anyone tested buildbot 0.9 for 200+ builders (for
single-master)?<br>
<br>
</div>
<div><br>
postgresql logs: <span style="color:rgb(68,68,68);font-family:Roboto,Helvetica,Arial,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none"><a href="https://goo.gl/ZCd19o" target="_blank">https://goo.gl/ZCd19o</a><br>
</span></div>
<div><br>
<br>
Relevant logs for ROLLBACK (#1):
<blockquote><span style="font-family:monospace,monospace"><font size="2">2017-06-23 21:19:45.491 GMT 24705 0
localhost(56256) LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.494 GMT 24697 0 localhost(56242)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6<br>
2017-06-23 21:19:45.496 GMT 24717 0 localhost(56270)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6<br>
2017-06-23 21:19:45.496 GMT 24715 0 localhost(56266)
LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.498 GMT 24697 0 localhost(56242)
LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.499 GMT 24698 0 localhost(56244)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6</font><br>
<br>
</span></blockquote>
</div>
<div>Relevant logs for #2:<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace"></span>
<blockquote><span style="font-family:monospace,monospace"><span style="color:rgb(0,0,0)">2017-06-23 21:19:52.221 GMT
24738 localhost(56312) LOG: statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a>
<br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 218<br>
2017-06-23 21:19:52.223 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 219<br>
2017-06-23 21:19:52.226 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 220<br>
2017-06-23 21:19:52.235 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 221</span><br>
</span></blockquote>
<br>
</div>
<div>Thanks in Advance<br>
</div>
<div><br>
-Aakash<br>
</div>
</div>
<br>
<fieldset class="m_5063189424227819344gmail-m_-2380960802746973949mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
devel mailing list
<a class="m_5063189424227819344gmail-m_-2380960802746973949moz-txt-link-abbreviated" href="mailto:devel@buildbot.net" target="_blank">devel@buildbot.net</a>
<a class="m_5063189424227819344gmail-m_-2380960802746973949moz-txt-link-freetext" href="https://lists.buildbot.net/mailman/listinfo/devel" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/devel</a></pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
devel mailing list<br>
<a href="mailto:devel@buildbot.net" target="_blank">devel@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/devel" rel="noreferrer" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/devel</a><br></blockquote></div><br></div></div></div></div>