[users at bb.net] Uploading files to the master
Pierre Tardy
tardyp at gmail.com
Wed Jan 10 20:58:49 UTC 2018
Hi Ivan,
looks like your approach of doing the analysis in a different step is what
was inefficient in the first place.
For this usecase, Buildbot has the concept of LogObserver.
You can find detailed documentation on them here:
http://docs.buildbot.net/latest/manual/customization.html#adding-logobservers
So with the logobserver, you can gather the list of files that will need to
be downloaded.
Then in order to retrieve them, there is the CompositeStepMixin. It looks
like this one is undocumented. I should really fix that because it is quite
useful.
You can find here an example on how to use it:
https://github.com/buildbot/buildbot_travis/blob/master/buildbot_travis/steps/base.py#L58
What you would have to do here is to call getFileContentFromWorker() and
addCompleteLog() in the end of the step (overriding evaluateCommand looks
like a good place in your case)
Pierre
On Tue, Jan 9, 2018 at 3:13 PM Ivan Muzzolini <ivan.muzzolini at gmail.com>
wrote:
> Dear All,
>
> My name is Ivan and this is the first time I write to this mailing list...
> ...so please understand me if sometimes I will be a little unprecise... :-)
>
> Premise: I am using Buildbot 0.9.15.
> I am working on a builder that does the following:
>
> 1. update the local source view
> 2. compile and test (the standard output of this step is attached to the
> Build object)
> 3. analysis: the standard output of step 2. is retrieved from the current
> Build object and it is parsed to produce a report containing a list of
> files; each of those files is then uploaded from the worker to the master
> and attached as a log of this analysis step
>
> The step 3. is a custom build step derived from
> "buildbot.process.buildstep.BuildStep"; I called that class "Analysis" and
> you can find its source code at the end of this email
> The step 3. is the one that does NOT work... The problem that I am facing
> is that I am NOT able to upload the contents of the files to the master.
>
> Could you please help me to find out what I am doing wrong? I illustrate
> what I have done by commenting the code of my class "Analysis".
>
> At line 65, the method "self.retrieve_execution_report" is invoked to
> create a report that contains a list of files.
> The logics for uploading the files to the master are in the range 71-94.
> As you can see at line 71, the main_report contains a list of
> failing_tests; these are the paths of the files that we want to upload.
> For each of these failing_tests, a "RemoteCommand" is created (line 85);
> upon completion of this command, the function "publish" is executed (line
> 87). The function "publish" is defined a few lines before (lines 76-83).
> The "RemoteCommand" is created by the method "setupRetrieveCmd" defined
> after the method "run".
>
> The issue that I see is that the buffer that I am adding as a "complete
> log" (line 82) is always empty.
>
> QUESTION: how can I setup the RemoteCommand so that the "publish" method
> is executed only when the buffer has been filled with the contents coming
> from the worker?
>
> Any help will be very appreciated!
> Thank you very much!
>
> Best Regards
> Ivan Muzzolini
>
>
> 40: class Analysis(buildstep.BuildStep, VISconsMixin):
> 41: def __init__(self, working_dir):
> 42: self.root_dir = self.compose_args(working_dir)['workdir']
> 43: self.main_report = None
> 44: kwargs = {'name' : 'analysis'}
> 45: kwargs['flunkOnFailure'] = True
> 46: buildstep.BuildStep.__init__(self, **kwargs)
> 47:
> ...
> ...
> 53:
> 54: @defer.inlineCallbacks
> 55: def run(self):
> ...
> 65: self.main_report =
> self.retrieve_execution_report(self.build.main_stdout)
> ...
> 71: for test in self.main_report.failing_tests:
> 72: log.msg('uploading ', test)
> 73: try:
> 74: log_path = './' +
> self.main_report.failing_tests[test].replace('\\', '/') if
> buildbot_development_instance() else self.main_report.failing_tests[test]
> 75:
> 76: def publish(remote_cmd):
> 77: import itertools
> 78: file_path = remote_cmd.args['slavesrc'] if
> self.workerVersionIsOlderThan('uploadFile', '3.0') else
> remote_cmd.args['workersrc']
> 79: path_items = [i.split('/') for i in
> file_path.split('\\')]
> 80: path_items_flat =
> list(itertools.chain(*path_items))
> 81: test_name = path_items_flat[-1]
> 82: self.addCompleteLog('TEST FAILED: ' +
> test_name, remote_cmd.args['writer'].buffer)
> 83: log.msg('uploading SUCCEEDED.')
> 84:
> 85: cmd = self.setupRetrieveCmd(log_path)
> 86: d = cmd.run(self, self.remote,
> self.build.builder.name)
> 87: d.addCallback(publish)
> 88: yield d
> 89: except Exception as e:
> 90: log.msg('uploading FAILED: ', str(e))
> 91: finally:
> 92: self.cmd = None
> 93:
> 94: defer.returnValue(FAILURE)
> ...
> 101:
> 102: def setupRetrieveCmd(self, file_path):
> 103: from buildbot.process import remotecommand
> 104: from buildbot.process import remotetransfer
> 105:
> 106: remote_cmd_name = 'uploadFile'
> 107:
> 108: self.checkWorkerHasCommand(remote_cmd_name)
> 109:
> 110: fileWriter = remotetransfer.StringFileWriter()
> 111: args = {
> 112: 'workdir': self.workdir,
> 113: 'writer': fileWriter,
> 114: 'maxsize': None,
> 115: 'blocksize': 32 * 1024,
> 116: }
> 117:
> 118: if self.workerVersionIsOlderThan(remote_cmd_name, '3.0'):
> 119: args['slavesrc'] = file_path
> 120: else:
> 121: args['workersrc'] = file_path
> 122:
> 123: cmd = remotecommand.RemoteCommand(remote_cmd_name, args)
> 124: if hasattr(self, "rc_log"):
> 125: cmd.useLog(self.rc_log, False)
> 126:
> 127: cmd.worker = self.worker
> 128: return cmd
>
>
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20180110/ebb109e6/attachment.html>
More information about the users
mailing list