[Buildbot-devel] Transfering files inside a build step

[Daniel Eggert]

On Jan 15, 2010, at 3:34 PM, Daniel Eggert wrote:

> 
> On Jan 15, 2010, at 3:28 PM, Andrew Melo wrote:
> 
>> On Fri, Jan 15, 2010 at 5:26 PM, Daniel Eggert <deggert at apple.com> wrote:
>>> On Jan 15, 2010, at 3:17 PM, Daniel Eggert wrote:
>>> 
>>>> I need to transfer files inside a build step.
>>>> 
>>>> My build step might generate crash logs (on the slave). I parse the stdio log for the paths to those logs, but how do I dynamically move them to the master?
>>>> 
>>>> I have
>>>> 
>>>> def createSummary(self, log):
>>>>      ShellCommand.createSummary(self, log)
>>>>      crashIndex = 0
>>>>      for crashLogPath in self.crashLogs:
>>>>              text = None
>>>>              try:
>>>>                      with open(crashLogPath, 'r') as f:
>>>>                              text = f.read()
>>>>              except:
>>>>                      pass
>>>>              if text is not None:
>>>>                      crashIndex += 1
>>>>                      name = "Crash"
>>>>                      if (crashIndex > 1):
>>>>                              name = "Crash {0}".format(crashIndex)
>>>>                      self.addCompleteLog(name, text)
>>>> 
>>>> 
>>>> but the paths inside self.crashLogs are paths on the slave.
>>>> 
>>>> /Daniel
>>> 
>>> 
>>> It seems to me that I should use a RemoteShellCommand(), but I am clueless as to how to integrate that into my build step during createSummary().
>> 
>> you could set a property with the path to the log you want, and then
>> later use that property in the upload step
> 
> That is true, but messy since the log belongs to the original step, and this way I would need twice the number of steps. But I guess it would do as a workaround.
> 
> I was hoping someone could explain how to use RemoteShellCommand() within a steps createSummary().
> 
> /Daniel

I 'mostly' got it working by invoking a "cat" on the slave. Something like this:


    def finished(self, results):
        # Override this to pull over the log files
        crashIndex = 0
        self.crashLogsAddedCount = 0
        crashCount = len(self.crashLogs)
        if crashCount < 1:
            ShellCommand.finished(self, results)
        for crashLogPath in self.crashLogs:
            crashName = "crash log {0}".format(crashIndex)
            if crashIndex == 0:
                crashName = "crash log"
            crashIndex += 1
            log = self.addLog(crashName)
            cmd = RemoteShellCommand(workdir='.',
                                     command=["cat", crashLogPath],
                                     want_stderr=False,
                                     timeout=20)
            cmd.useLog(log, closeWhenFinished=True, logfileName='stdio')
            d = self.runCommand(cmd) # might raise ConnectionLost
            def _getLogDone(subResults):
                self.crashLogsAddedCount += 1
                if self.crashLogsAddedCount >= crashCount:
                    ShellCommand.finished(self, FAILURE)
            d.addCallback(_getLogDone).addErrback(self.failed)


Not super elegant. I'd like to actually be able to copy the file (e.g. if it's not text). But this will work for now.

/Daniel