[Buildbot-devel] schedulerdb progress report: week 2
warner at lothar.com
Fri Sep 18 01:21:40 UTC 2009
Another progress report. As usual, my latest code is in:
* got MySQL environment running, fixed schema to work with both SQLite
and MySQL (sqlite is amazingly lenient). Added argument-placeholder
substitution code to handle sqlite3's "?" vs MySQLdb's "%s" (grr).
* added --db= argument to "buildbot create-master" and "buildbot
upgrade-master". The database specification is stored in the
buildbot.tac file, not master.cfg, which appears to simplify more than
it complicates. "upgrade-master" will migrate changes.pck into the
newly-created database, them move it aside so later upgrade calls
won't do it a second time.
* added asynchronous accessors to ChangeManager. Scheduler code will use
async calls, changesources (for now at least) and status code will use
* implemented a single-file no-reentrancy notification-loop runner,
attached c["schedulers"] to use it. For now this just runs once every
30 seconds, but eventually this will only fire when the database has
been changed and the Schedulers need to see if they have work to do.
* halfway converted the most basic tree-stable-timer one-branch
Scheduler to the new database-checking format. It looks for
unprocessed Changes, classifies each as important, unimportant, or
irrelevant, then prints "YES BUILD" when the tree-stable-time has
expired and there are important changes that need to be built
I'm on vacation next week. The next steps (to begin on 9/29) will be:
- define the SourceStamp, BuildSet, and BuildRequest tables
- finish converting the basic Scheduler: have it generate a SourceStamp
for the changes it wants to build, then add a BuildSet row and one or
more BuildRequest rows to actually request the build
- convert other Schedulers: Dependent will watch the BuildSet for triggers,
Periodic and Nightly will add to the db but won't read from it at all.
- define the Build table and start working on conversions of the
Builders, to read from the BuildRequest table, claim a build, and
update the BuildRequest with the build's results when finished.
- lots of isolated tests: database reconnection, transaction rollback
handling, schema rationalization
- SQL is painful, and the differences between even SQLite and MySQL are
larger than I was expecting (and python's dbapi does less than I'd
hoped). I'm still hoping to complete this project without resorting to
an extra compatibility library like SQLAlchemy, but I may have to use
it to provide cross-db compatibility that isn't completely broken,
especially to work with a third DB like postgres. We'll see.
More information about the devel