[Buildbot-devel] sending files back to the master

[Brian Warner]

> the output is generated as a set of html files with relative links.
> What would be the easiest way to get them to the master and have them
> served from there ? Zip them up, transfer, unzip, add some hooks to show
> an arbitrary directory tree to the waterfall ?

Yeah. Hmm. I can think of a half-solution, but it's missing some important
pieces that will require more thought to work out. I'd start with something
like this:

 create buildbot.status.builder.HTMLLogTree, which will contain a zip
 file that has the whole set of HTML files (along with an attribute that
 points at the bundle's top-level page). Build it like the existing
 HTMLLogFile, but open the persistent file in __init__ and give it a .write
 method that streams the .zip-format data into the file. When .finish is
 called, close the file and enable reading on the object. Instead of
 implementing IStatusLog, it should implement some new interface like
 IStatusLogTree,which could accept methods like getRootContents() and
 getContents(filename). getContents() should use the standard zipfile module
 to pull out component files on demand.

  add a buildbot.status.builder.Build.addLogTree(name) method which creates
  one of these, adds it to a self.logTrees list, notifies subscribers, and
  returns the new log object.

 create a generic "transfer file from the slave to the master"
 RemoteCommand/slave.Command pair (note: not a BuildStep), that takes a
 file-like object (i.e., one with .write and .close methods) to which the
 file's contents are sent. Then make a BuildStep for your process that first
 does a RemoteShellCommand to zip up the html files, then creates an
 HTMLLogTree, then does the GetFile command to stream the zip contents into
 that HTMLLogTree object.

 then extend the HTML rendering code to deal with IStatusLogTree objects,
 probably with an adapter that uses orig.getRootContents() to render the main
 page and has a getChild() method which knows to ask the HTMLLogTree for the
 component files.

The missing parts in this is how to provide this same status information to
other StatusTargets. If the IStatusLogTree interface is done right, I guess
it shouldn't be too hard. It feels a bit clunky for the HTMLLogTree object to
have such asymmetric IO (you write zipped+aggregated data to it, but you read
uncompressed individual files from it), but it makes more sense to do the
aggregation on the slave side. I'm tempted to want a method to let you add
component files to the zipped-logtree, but perhaps that should wait until we
have a use case for it. I'm also tempted to make getContents() return a
(contentType, data) pair (so you could put text/plain in there instead of
text/html, and since the web code will need to be told the content-type
anyway, because there's no real filename for it to work with). Also, if these
files get large, I'd want a getContents() form that can return an open
filehandle, and for the HTML adapter to be able to properly stream the
contents without buffering everything in memory first.. there is probably
enough code in twisted.web.static.File to build this without too much work,
you just have to make it get the filehandle from the logtree instead of from
open().

Also, to support doing something with the log tree other than rendering it to
a web page, the HTMLLogTree object should probably have an
enumerate-all-files method (imagine an IRC interface that could send you
specific component files, or an email-sender that wanted to send you a
mixed/multipart collection of individual HTML files, or a status plugin that
wanted to extract some of the coverage data from one of the files and add it
to a long-term database).

Also, it might be cleaner to say that buildbot.status.builder.Build.logs is a
list of objects that implement either IStatusLog or IStatusLogTree, rather
than keeping the two kinds of objects in separate lists (specifically because
keeping two lists probably means two sets of notification methods and two
sets of retrieve-a-log methods). I'm not sure.. if we do that there is code
that needs to be updated to test the log object before doing something with
it.

anyway, hope that gets you started. This would be great stuff to merge into
the trunk once it's done.

thanks!
 -Brian