[Buildbot-commits] [Buildbot] #1699: Add support for wild-card matching for logfiles parameter in build steps

[Buildbot]

#1699: Add support for wild-card matching for logfiles parameter in build steps
------------------------+--------------------
Reporter:  lantictac    |       Owner:
    Type:  enhancement  |      Status:  new
Priority:  minor        |   Milestone:  1.0.+
 Version:  0.8.2        |  Resolution:
Keywords:               |
------------------------+--------------------

Comment (by lantictac):

 Sorry for the incredibly slow response (production deadlines got the
 better of me). As it happens I did manage to get a hacky implementation
 working, which we're using to dynamically pick up multiply log files
 without explicitly declaring them in the master. Unfortuantely I might be
 missing some key details but here's the basis for the implementation. Make
 of it (and take of it) what you will...

 On the slave side in runprocess.py...
 {{{
 # Used to watch for any number of files matching a given wild-card
 class WildcardLogFileWatcher:
     LIST_INTERVAL = 4

     def __init__(self, command, name, logfile, follow=False):
         self.command = command
         self.name = name
         self.logfile = logfile
         self.dirPath = os.path.split(logfile)[0]

         log.msg("WildcardLogFileWatcher created to watch %s dir %s" %
 (logfile, self.dirPath))

         self.started = False

         # follow the file, only sending back lines
         # added since we started watching
         self.follow = follow

         # every few seconds we check on the directory again
         self.dirPoller = task.LoopingCall(self.dirPoll)

         # maintain a list of child log file watches (initially empty)
         self.logFileWatchers = []

         # maintain a set of found log files to compare with the latest
 matches
         self.logFileSet = set()

     def start(self):
 self.dirPoller.start(self.LIST_INTERVAL).addErrback(self._cleanupListPoll)

     def _cleanupListPoll(self, err):
         log.err(err, msg="Polling error for file listing")
         self.dirPoller = None

     def stop(self):
         self.dirPoll()
         if self.dirPoller is not None:
             self.dirPoller.stop()
         for w in self.logFileWatchers:
             w.stop()

     def listDir(self):
         if os.path.exists(self.dirPath):
             return set(glob.glob(self.logfile))
         return set()

     def dirPoll(self):
         fileSet = self.listDir()

         if fileSet == self.logFileSet:
             return # not started yet

         # The fileSet is different so add the new files
         newFileSet = fileSet - self.logFileSet

         for filePath in newFileSet:
             # Generate a unique name based on format:
 "logname;realfilename"
             uniqueName = "%s;%s" % (self.name, os.path.split(filePath)[1])
             log.msg('WildcardLogFileWatcher found new file "%s" and
 assigned it the name "%s"' % (filePath, uniqueName))
             w = LogFileWatcher(self.command,
                     uniqueName,
                     filePath,
                     follow=self.follow)

             w.start()
             self.logFileWatchers.append(w)

         self.logFileSet = self.logFileSet | newFileSet
 }}}

 And later in RunProcess __init__()

 {{{
             w = None

             if filename.find('*') == -1:
                 w = LogFileWatcher(self, name,
                                    os.path.join(self.workdir, filename),
                                    follow=follow)
             else:
                 w = WildcardLogFileWatcher(self, name,
                                    os.path.join(self.workdir, filename),
                                    follow=follow)

             self.logFileWatchers.append(w)
 }}}

 While on the master side in buildstep.py:


 {{{
     def addToLog(self, logname, data):

         oldname = logname
         realname = logname
         isWildCardPath = False

         # Wild-card matches are returned encoded in the logname as
 "oldname;realname"
         # This allow a degree of backwards compatibilty and avoids
 altering the protocol
         if logname.find(';') != -1:
             splitname = logname.split(';')
             oldname = splitname[0]
             realname = splitname[1]
             assert oldname != realname
             isWildCardPath = True

         # Activate delayed logs on first data.
         if oldname in self.delayedLogs and not (realname in self.logs):

             (activateCallBack, closeWhenFinished) =
 self.delayedLogs[oldname]

             # Only wild-card based delayed logs should remain around to
 deal with additional matches
             if not isWildCardPath:
                 del self.delayedLogs[oldname]

             # Allow the build step to create a new local log instance for
 the status
             loog = activateCallBack(self, realname)
             self.logs[realname] = loog
             self._closeWhenFinished[realname] = closeWhenFinished

         if realname in self.logs:
             self.logs[realname].addStdout(data)
         else:
             log.msg("%s.addToLog: no such log %s" % (self, realname))
 }}}

 Pretty ugly I'm sure you'll agree but the best I could throw together at
 the time. It seems to work most of the time but I don't doubt there are
 issues with this as it stands. Feel free to use and abuse if it might be
 of use. Either way I'd love to see official support for this feature in
 the near future - it certainly made managing our master.cfg vastly easier.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1699#comment:3>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation