[Buildbot-devel] Scheduler work coming along nicely

[Brian Warner]

Hi all. Sorry to be so slow in responding to the mailing list.. I think I'm
about five days behind. I'll respond to everything eventually.

Most of my buildbot time has been consumed by work on the new Scheduler code.
Just a few minutes ago I got the build-on-branch unit tests to pass. I got
Locks working yesterday, and scheduler.Dependent got finished last weekend,
so the code is looking pretty good. There are a few more issues I want to
address before I merge this (arch) branch into the CVS mainline, but it's
getting close. Mostly I want to make sure that it is good enough for
adventurous folk to use before I put it on the trunk. The *really*
adventurous can pull a copy from the Arch/Bazaar repository that's tracking
this branch: http://arch.lothar.com/warner@monolith.lothar.com--2005 (version
name is buildbot--usebranches--0).

Just to share what I'm thinking, here's my personal TODO list for the next
release. Items with '#' in the first column are finished.

 features:
# Dependencies
   a way to ask about the status of each Change: complete on which Schedulers,
   waiting for dependencies on which, ignored by which
# Locks (per-step? per-build? scoped to the slave or the whole buildmaster?)
  a scheme for Schedulers to subscribe to all ChangeSources, so it will be
   clear how to write others that only subscribe to some or none
  updating Status targets to match new stuff
   HTML:
    waterfall: N slaves (instead of 0 or 1), showing Scheduler status
    new page with one column per Scheduler, track BuildSets (still
     chronological)
    new page with one box per scheduler (not chronological)
    new front page with links to status pages
    make it easy to insert arbitrary URL objects
   Mail: something to watch Schedulers/BuildSets instead of Builders
   IRC: query Schedulers, submit BuildSets
   IStatus: IScheduler, IBuildSetStatus, IBuildRequestStatus?
    IBuilderStatus.getState: redefine
    IBuildRequestControl: make sure the Control objects are returned at the
     right times
  VC branches:
   svn url-to-branch function, #branch-to-url function
#  checkout-vs-update of arbitrary branches, .buildbot-branch stampfiles
  try
  buildmaster-watching Scheduler
  docs for it all

 tests:
# VC branches: need to create a new repository tarball with branches
  BuildSet/Scheduler: in test_buildreq
  SlaveBuilder:
   verify multiple slaves connected at the same time
   claiming a slave for a new build
   slaveping
   failing a slaveping moves the request to the front of the list
   slave in the slaveping state is not claimable by a new build
   Ping
  try

 design issues:
  sharing files (build products) between Builders
  'try': two command paths:
     1 (ssh): create a command queue maildir in the buildmaster's space,
              admin sets the ownership/permissions to only grant write access
              to trusted developers. 'buildbot try' formats an rfc822-style
              message (sourcestamp goes in a header, patch goes in the body)
              and adds it to the queuedir. The config file entries used by
              'buildbot try' either specify a local queuedir (for which write
              and mv are used) or a remote one (using scp and ssh). +: quite
              secure. -: requires fiddling outside the buildmaster config.
     2 (pb): each developer gets a username/passwd pair, known to the
             buildmaster. 'buildbot try' connects to the slaveport and
             identifies itself as that user, then sends a PB command to force
             the build (with the sourcestamp+diff as arguments). +: don't
             need a special group or filesystem fiddling. -: depending upon
             the Cred mechanics used, the password may be passed in plaintext
             or plaintext-equivalent.
     maybe offer both?
  use/require Nevow for HTML pages? how would this affect twisted1-vs-2?
   (nevow-0.4.1 uses z.i if available, nevow-0.5pre uses z.i unconditionally)
  framework for long-term build stats?

 compatibility:
  slave's vc-command args

Other items to include in the next release:

 darcs tree (tailor) for win32 buildslave
 publish metabuildbot, probably move slaves to a more capable machine
 base URL needs fixing
 that win32 command-not-really-running kill-old-command bug
 panic when saved build pickle already exists? or overwrite silently?
 the 100k logfile stall with twisted-1.3
 change Mail object to include logfiles even without python2.3's email module
 change Mail object to include URL for the specific Build
 remove all uses of forceBuild other than the deprecation test, consider
  removing it altogether (I'm not sure it was ever really exposed)
 verify that deferredResult is gone from unit tests under retrial

Things to do soon, but not in the 0.7.0 release:

 Monotone
 mercurial
 cogito tests / branches



anyway, that's the day's update.. now back to the code.

cheers,
 -Brian