[users at bb.net] Uploading files to the master

[Ivan Muzzolini]

Dear All,

My name is Ivan and this is the first time I write to this mailing list...
...so please understand me if sometimes I will be a little unprecise... :-)

Premise: I am using Buildbot 0.9.15.
I am working on a builder that does the following:

1. update the local source view
2. compile and test (the standard output of this step is attached to the
Build object)
3. analysis: the standard output of step 2. is retrieved from the current
Build object and it is parsed to produce a report containing a list of
files; each of those files is then uploaded from the worker to the master
and attached as a log of this analysis step

The step 3. is a custom build step derived from
"buildbot.process.buildstep.BuildStep";
I called that class "Analysis" and you can find its source code at the end
of this email
The step 3. is the one that does NOT work... The problem that I am facing
is that I am NOT able to upload the contents of the files to the master.

Could you please help me to find out what I am doing wrong? I illustrate
what I have done by commenting the code of my class "Analysis".

At line 65, the method "self.retrieve_execution_report" is invoked to
create a report that contains a list of files.
The logics for uploading the files to the master are in the range 71-94.
As you can see at line 71, the main_report contains a list of
failing_tests; these are the paths of the files that we want to upload.
For each of these failing_tests, a "RemoteCommand" is created (line 85);
upon completion of this command, the function "publish" is executed (line
87). The function "publish" is defined a few lines before (lines 76-83).
The "RemoteCommand" is created by the method "setupRetrieveCmd" defined
after the method "run".

The issue that I see is that the buffer that I am adding as a "complete
log" (line 82) is always empty.

QUESTION: how can I setup the RemoteCommand so that the "publish" method is
executed only when the buffer has been filled with the contents coming from
the worker?

Any help will be very appreciated!
Thank you very much!

Best Regards
Ivan Muzzolini


40: class Analysis(buildstep.BuildStep, VISconsMixin):
41:     def __init__(self, working_dir):
42:         self.root_dir = self.compose_args(working_dir)['workdir']
43:         self.main_report = None
44:         kwargs = {'name' : 'analysis'}
45:         kwargs['flunkOnFailure'] = True
46:         buildstep.BuildStep.__init__(self, **kwargs)
47:
...
...
53:
54:     @defer.inlineCallbacks
55:     def run(self):
...
65:             self.main_report = self.retrieve_execution_report
(self.build.main_stdout)
...
71:                 for test in self.main_report.failing_tests:
72:                     log.msg('uploading ', test)
73:                     try:
74:                         log_path = './' +
self.main_report.failing_tests[test].replace('\\',
'/') if buildbot_development_instance() else self.main_report.failing_tests
[test]
75:
76:                         def publish(remote_cmd):
77:                             import itertools
78:                             file_path = remote_cmd.args['slavesrc'] if
self.workerVersionIsOlderThan('uploadFile', '3.0') else
remote_cmd.args['workersrc']
79:                             path_items = [i.split('/') for i in
file_path.split('\\')]
80:                             path_items_flat =
list(itertools.chain(*path_items))
81:                             test_name = path_items_flat[-1]
82:                             self.addCompleteLog('TEST FAILED: ' +
test_name, remote_cmd.args['writer'].buffer)
83:                             log.msg('uploading SUCCEEDED.')
84:
85:                         cmd = self.setupRetrieveCmd(log_path)
86:                         d = cmd.run(self, self.remote,
self.build.builder.name)
87:                         d.addCallback(publish)
88:                         yield d
89:                     except Exception as e:
90:                         log.msg('uploading FAILED: ', str(e))
91:                     finally:
92:                         self.cmd = None
93:
94:                 defer.returnValue(FAILURE)
...
101:
102:     def setupRetrieveCmd(self, file_path):
103:         from buildbot.process import remotecommand
104:         from buildbot.process import remotetransfer
105:
106:         remote_cmd_name = 'uploadFile'
107:
108:         self.checkWorkerHasCommand(remote_cmd_name)
109:
110:         fileWriter = remotetransfer.StringFileWriter()
111:         args = {
112:             'workdir': self.workdir,
113:             'writer': fileWriter,
114:             'maxsize': None,
115:             'blocksize': 32 * 1024,
116:         }
117:
118:         if self.workerVersionIsOlderThan(remote_cmd_name, '3.0'):
119:             args['slavesrc'] = file_path
120:         else:
121:             args['workersrc'] = file_path
122:
123:         cmd = remotecommand.RemoteCommand(remote_cmd_name, args)
124:         if hasattr(self, "rc_log"):
125:             cmd.useLog(self.rc_log, False)
126:
127:         cmd.worker = self.worker
128:         return cmd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20180109/8c4849fb/attachment.html>