[Buildbot-devel] schedulerdb progress update

[Brian Warner]

Buildbot-schedulerdb status report, 11/03/2009

= Code =

As before, latest code is published on GitHub:

 http://github.com/warner/buildbot/tree/schedulerdb-REBASES

= Progress Since Last Update =

 * end-to-end unit test (in test_db.py) implemented and working
 * end-to-end manual system tests work, both sqlite and mysql
 * moved all SQL into db.py
 * added generic key-value store, for e.g. polling ChangeSources
 * ported all Schedulers except Try
 * BuildRequest claims are updated every 10 minutes

Code is now ready for early system testing: time to shake out the
boneheaded bugs, and find out which Status interfaces were broken.
Install a new buildmaster ("buildbot create-master" and "buildbot
upgrade-master" will create the database), give it a buildslave, and see
what happens.

= Next Steps =

 * port Try
 * historydb
 * graceful master shutdown
 * update unit tests, make them run again (currently most tests fail)

= Remaining Work =
 * what history do we want to preserve? where should it be stored?
   separate tables?
 * change allocate-next-highest-idnumber technique
 * figure out status interfaces for scheduler pieces:
   BuildRequestStatus, (current) BuilderStatus. Any "statusdb" work is
   explicitly deferred, but the existing interfaces need to keep working
   even if their backing store has been moved from RAM to the DB
 * avoid race condition in claiming buildrequests (only update if the
   claimed_at/claimed_by is unchanged)
 * db loss/reconnection handling
 * retain compatibility with c['prioritizeBuilders']
 * figure out GC: keep DB from unbounded growth, remove old/unused
   SourceStamp/BuildRequest/Properties rows
 * construct buildmaster identity (for buildrequest claim):
   name+boottime
 * clean up mixed sync/async DB access: decide on what should not block,
   what can block, where the complexity of returning Deferreds is
   warranted


cheers,
 -Brian