[Buildbot-devel] schedulerdb progress report: week 2

[Brian Warner]

Another progress report. As usual, my latest code is in:

 http://github.com/warner/buildbot/tree/schedulerdb

* got MySQL environment running, fixed schema to work with both SQLite
  and MySQL (sqlite is amazingly lenient). Added argument-placeholder
  substitution code to handle sqlite3's "?" vs MySQLdb's "%s" (grr).
* added --db= argument to "buildbot create-master" and "buildbot
  upgrade-master". The database specification is stored in the
  buildbot.tac file, not master.cfg, which appears to simplify more than
  it complicates. "upgrade-master" will migrate changes.pck into the
  newly-created database, them move it aside so later upgrade calls
  won't do it a second time.
* added asynchronous accessors to ChangeManager. Scheduler code will use
  async calls, changesources (for now at least) and status code will use
  synchronous calls.
* implemented a single-file no-reentrancy notification-loop runner,
  attached c["schedulers"] to use it. For now this just runs once every
  30 seconds, but eventually this will only fire when the database has
  been changed and the Schedulers need to see if they have work to do.
* halfway converted the most basic tree-stable-timer one-branch
  Scheduler to the new database-checking format. It looks for
  unprocessed Changes, classifies each as important, unimportant, or
  irrelevant, then prints "YES BUILD" when the tree-stable-time has
  expired and there are important changes that need to be built

I'm on vacation next week. The next steps (to begin on 9/29) will be:

- define the SourceStamp, BuildSet, and BuildRequest tables
- finish converting the basic Scheduler: have it generate a SourceStamp
  for the changes it wants to build, then add a BuildSet row and one or
  more BuildRequest rows to actually request the build
- convert other Schedulers: Dependent will watch the BuildSet for triggers,
  Periodic and Nightly will add to the db but won't read from it at all.
- define the Build table and start working on conversions of the
  Builders, to read from the BuildRequest table, claim a build, and
  update the BuildRequest with the build's results when finished.
- lots of isolated tests: database reconnection, transaction rollback
  handling, schema rationalization

Some notes:
- SQL is painful, and the differences between even SQLite and MySQL are
  larger than I was expecting (and python's dbapi does less than I'd
  hoped). I'm still hoping to complete this project without resorting to
  an extra compatibility library like SQLAlchemy, but I may have to use
  it to provide cross-db compatibility that isn't completely broken,
  especially to work with a third DB like postgres. We'll see.


cheers,
 -Brian