[Buildbot-devel] schedulerdb progress report: week 1

[Brian Warner]

Just an update on what I've gotten done so far:

* http://github.com/warner/buildbot/tree/schedulerdb contains the new
  code, branched off of 'master'
* "buildbot master" will create a sqlite-based database in
  BASEDIR/state.sqlite , containing the new "changes" table (which
  replaces the old changes.pck file.
* "buildbot upgrade-master" will create the same, then copy all of your
  Changes out of changes.pck and store them in the database
* when ChangeSources hand changes to the buildmaster, they'll be stored
  in the database
* rearranged buildbot/changes/changes.py to slowly get rid of the old
  pickled classes: i.e. ChangeMaster is now a stub that exists solely
  for pickle migration purposes, and a new ChangeManager class replaces
  it. The Change class remains in place, since ChangeSources expect it
  (they instantiate a Change and pass it to self.parent.addChange) as
  well as for pickle migration.
* updated ChangeManager.getChangedNumbered and .eventGenerator to pull
  from the DB

The new database schema is in buildbot/db.py . I wrote a wrapper around
twisted.enterprise.adbapi to permit synchronous access as well as the
usual Deferred-returning runQuery/runInteraction. This will make the
transition process easier (in particular, ChangeSources don't expect
addChange() to return a Deferred, so things work better if addChange()
blocks until the db write has finished). The long-term goal is to only
use async access. The Scheduler/Builder code that I'm writing will use
async access from the start.

Unit tests are currently broken, but "upgrade-master" should not throw
out your old changes (unlike the commit from earlier in the week), and a
basic test install correctly displays changes on the Waterfall.

Next steps are:

* test against mysql too (not just sqlite), test database-reconnection
  code, think about how to provide database location to "buildbot
  master" and "upgrade-master".
* update unit tests to match new classnames
* define schema for scheduler-state table, create notification loop,
  start converting Schedulers to use the DB

BTW, the plan is for an 0.8.0 release to use the schedulerdb but be a
fairly painless upgrade from 0.7.11p3 . Only users who have written
custom Schedulers (and possibly, to a lesser extent, ChangeSources) will
need to modify their local code.

cheers,
 -Brian