<div dir="ltr"><div><div>large pending buildrequests were indeed the issue. I cancelled most of the buildrequests and the buildbot startup time dropped from ~8m to ~10s.<br></div><div><br></div><div>I had the testing instance up and running for ~1 month without any workers, and continuously accumulating buildrequests. In production we should have workers most of the times, except for any unexpected maintenance in which case worker downtime shouldn't be more than few days.<br><br>However, I wanted to confirm if 25,000 is indeed very large number of pending buildrequests for buildbot? Has anyone ever tried such situation?<br></div><div><br></div>Thanks<br></div>Aakash<br><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 7:57 PM, Aakash Jain <span dir="ltr"><<a href="mailto:aj355@cornell.edu" target="_blank">aj355@cornell.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thanks Neil. That's good to know that buildbot is supporting this much workers easily. Also, rjarry mentioned over IRC that he is using buildbot for 500+ builders in single-master config with a startup time of less than 10s. So, number of builders (~200) doesn't seems like an issue in my case.<br><br><div>Also, tardyp pointed out on IRC that <b>ROLLBACK is not the problem</b>, it's probably normal behavior for selects with sqlalchemy (<a href="https://stackoverflow.com/questions/7559570/make-sqlalchemy-commit-instead-of-rollback-after-a-select-query" target="_blank">link</a>). I confirmed this by creating a fresh new postgresql database, and I still saw ROLLBACKs.<br><br></div><div>I am still trying to debug the performance issues. I tried removing schedulers, but that didn't help in performance.<br></div>
<br></div><div>Also, I have ~25000 pending buildrequests (since I have test instance running for a while). Can this be the cause of performance issues (and large startup time i am seeing)?<br></div><div><br></div><div>Thanks<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888">Aakash</font></span><div><div class="h5"><br><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 7:33 AM, Neil Gilmore <span dir="ltr"><<a href="mailto:ngilmore@grammatech.com" target="_blank">ngilmore@grammatech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
While we are using a multi-master system, one of our masters has at
least that many builders. I think we have on the order of 178
workers on that master, and at least 2 builders on each worker (most
have 3 or more). We also use a postgresql database and I've never
seen that as a source of any of our problems.<br>
<br>
Starting that master doesn't take more than 30 seconds, and probably
less (I haven't timed it). Reconfigs take a lot longer.<br>
<br>
In fact we had a reconfig take more than 17K seconds over the
weekend. It never finished, because I killed and restarted it. Users
were getting annoyed that none of their builds were in the UI.<br>
<br>
Neil Gilmore<br>
<a href="http://grammatech.com" target="_blank">grammatech.com</a><div><div class="m_-133502346078533705m_5063189424227819344gmail-h5"><br>
<br>
<div class="m_-133502346078533705m_5063189424227819344gmail-m_-2380960802746973949moz-cite-prefix">On 6/27/2017 8:30 PM, Aakash Jain
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="m_-133502346078533705m_5063189424227819344gmail-h5">
<div dir="ltr">
<div>Hi Everyone,<br>
<br>
</div>
<div>I am working on migrating our buildbot instance from 0.8.12
to 0.9.8. I did not have performance issue with 0.8.12.
However with buildbot 0.9 (with the same configuration), I am
seeing a lot of performance issues, mostly related to
database. I was expecting a performance improvement since
buildbot 0.9 moved from pickles to database (<a href="https://medium.com/@tardyp31/d0d41bba07e1" target="_blank">and designed for
scalability</a>), however I am noticing the reverse.<br>
</div>
<div><br>
</div>
<div>My buildbot configuration is:<br>
</div>
<div>- buildbot 0.9.8<br>
</div>
<div>- single master<br>
</div>
<div>- postgresql db<br>
</div>
<div>- 200+ builders (queues)<br>
</div>
<div>- only 2 workers connected currently for testing<br>
</div>
<div>- decent hardware on master (8 gb ram, 8 cpu)<br>
<br>
<br>
</div>
<div>"buildbot start" is taking ~8 minutes (the start time
increased gradually over last few weeks). Also loading various
buildbot webpages are slow (many times webpage loads, but the
build data load after ~30s). During this time there isn't too
much cpu/memory/IO load on master.<br>
<br>
I enabled postgresql logs and I see extremely large number of
SELECT statements in the logs. I also see a large number of
"ROLLBACK" in postgresql logs. I have few questions:<br>
<br>
</div>
<div>1) Why I am seeing so many ROLLBACK in the postgresql logs?
Any suggestions to fix/debug it?<br>
</div>
<div><br>
</div>
<div>2) From postgresql logs, I notice that buildbot is running
database queries in a loop fetching data for individual
builder/build/ (resulting in large number of queries).
Wouldn't it be more efficient to use a single database query
to fetch data for all the builders and then filter it in
python code?<br>
</div>
<div><br>
</div>
<div>3) has anyone tested buildbot 0.9 for 200+ builders (for
single-master)?<br>
<br>
</div>
<div><br>
postgresql logs: <span style="color:rgb(68,68,68);font-family:Roboto,Helvetica,Arial,sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none"><a href="https://goo.gl/ZCd19o" target="_blank">https://goo.gl/ZCd19o</a><br>
</span></div>
<div><br>
<br>
Relevant logs for ROLLBACK (#1):
<blockquote><span style="font-family:monospace,monospace"><font size="2">2017-06-23 21:19:45.491 GMT 24705 0
localhost(56256) LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.494 GMT 24697 0 localhost(56242)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6<br>
2017-06-23 21:19:45.496 GMT 24717 0 localhost(56270)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6<br>
2017-06-23 21:19:45.496 GMT 24715 0 localhost(56266)
LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.498 GMT 24697 0 localhost(56242)
LOG: statement: <b>ROLLBACK</b><br>
2017-06-23 21:19:45.499 GMT 24698 0 localhost(56244)
LOG: statement: SELECT changes.changeid,
changes.author, changes.comments, changes.branch,
changes.revision, changes.revlink,
changes.when_timestamp, changes.category,
changes.repository, changes.codebase, changes.project,
changes.sourcestampid, changes.parent_changeids <br>
FROM changes <br>
WHERE changes.sourcestampid = 6</font><br>
<br>
</span></blockquote>
</div>
<div>Relevant logs for #2:<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace"></span>
<blockquote><span style="font-family:monospace,monospace"><span style="color:rgb(0,0,0)">2017-06-23 21:19:52.221 GMT
24738 localhost(56312) LOG: statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a>
<br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 218<br>
2017-06-23 21:19:52.223 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 219<br>
2017-06-23 21:19:52.226 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 220<br>
2017-06-23 21:19:52.235 GMT 24738 localhost(56312) LOG:
statement: SELECT <a href="http://tags.name" target="_blank">tags.name</a> <br>
FROM tags JOIN builders_tags ON <a href="http://tags.id" target="_blank">tags.id</a>
= builders_tags.tagid <br>
WHERE builders_tags.builderid = 221</span><br>
</span></blockquote>
<br>
</div>
<div>Thanks in Advance<br>
</div>
<div><br>
-Aakash<br>
</div>
</div>
<br>
<fieldset class="m_-133502346078533705m_5063189424227819344gmail-m_-2380960802746973949mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
devel mailing list
<a class="m_-133502346078533705m_5063189424227819344gmail-m_-2380960802746973949moz-txt-link-abbreviated" href="mailto:devel@buildbot.net" target="_blank">devel@buildbot.net</a>
<a class="m_-133502346078533705m_5063189424227819344gmail-m_-2380960802746973949moz-txt-link-freetext" href="https://lists.buildbot.net/mailman/listinfo/devel" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/devel</a></pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
devel mailing list<br>
<a href="mailto:devel@buildbot.net" target="_blank">devel@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/devel" rel="noreferrer" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/devel</a><br></blockquote></div><br></div></div></div></div></div></div>
</blockquote></div><br></div></div></div></div>