<div dir="ltr">Hi Ivan,<div><br></div><div>looks like your approach of doing the analysis in a different step is what was inefficient in the first place.</div><div>For this usecase, Buildbot has the concept of LogObserver.</div><div><br></div><div>You can find detailed documentation on them here:</div><div><a href="http://docs.buildbot.net/latest/manual/customization.html#adding-logobservers">http://docs.buildbot.net/latest/manual/customization.html#adding-logobservers</a><br></div><div><br></div><div>So with the logobserver, you can gather the list of files that will need to be downloaded.</div><div>Then in order to retrieve them, there is the CompositeStepMixin. It looks like this one is undocumented. I should really fix that because it is quite useful.</div><div>You can find here an example on how to use it:</div><div><a href="https://github.com/buildbot/buildbot_travis/blob/master/buildbot_travis/steps/base.py#L58">https://github.com/buildbot/buildbot_travis/blob/master/buildbot_travis/steps/base.py#L58</a><br></div><div><br></div><div>What you would have to do here is to call getFileContentFromWorker() and addCompleteLog() in the end of the step (overriding evaluateCommand looks like a good place in your case)<br></div><div><br></div><div>Pierre</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 9, 2018 at 3:13 PM Ivan Muzzolini <<a href="mailto:ivan.muzzolini@gmail.com">ivan.muzzolini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear All,<br><div><div><div><div><div><div><div><div><div><br></div><div>My name is Ivan and this is the first time I write to this mailing list... ...so please understand me if sometimes I will be a little unprecise... :-)</div><div><br></div><div>Premise: I am using Buildbot 0.9.15.<br></div><div>I am working on a builder that does the following:</div><br><div>1. update the local source view<br></div><div>2. compile and test (the standard output of this step is attached to the Build object)</div><div>3. analysis: the standard output of step 2. is retrieved from the current Build object and it is parsed to produce a report containing a list of files; each of those files is then uploaded from the worker to the master and attached as a log of this analysis step<br></div><div><br></div><div>The step 3. is a custom build step derived from "buildbot.process.buildstep.BuildStep"; I called that class "Analysis" and you can find its source code at the end of this email<br></div>The step 3. is the one that does NOT work... The problem that I am facing is that I am NOT able to upload the contents of the files to the master.<div><br></div><div>Could you please help me to find out what I am doing wrong? I illustrate what I have done by commenting the code of my class "Analysis".<br></div></div><div><br></div><div>At line 65, the method "self.retrieve_execution_report" is invoked to create a report that contains a list of files.<div>The logics for uploading the files to the master are in the range 71-94. </div>As
you can see at line 71, the main_report contains a list of
failing_tests; these are the paths of the files that we want to upload.<br><div>For
each of these failing_tests, a "RemoteCommand" is created (line 85); upon
completion of this command, the function "publish" is executed (line
87). The function "publish" is defined a few lines before (lines 76-83).<br></div>The "RemoteCommand" is created by the method "setupRetrieveCmd" defined after the method "run".<br><br>The issue that I see is that the buffer that I am adding as a "complete log" (line 82) is always empty.<br></div><div><br></div><div>QUESTION:
how can I setup the RemoteCommand so that the "publish" method is
executed only when the buffer has been filled with the contents coming from the worker?<br></div><div><br></div><div>Any help will be very appreciated!<br>Thank you very much!<br><br>Best Regards<br>Ivan Muzzolini<br><br><br>40: class Analysis(buildstep.BuildStep, VISconsMixin):<br>41: def __init__(self, working_dir):<br>42: self.root_dir = self.compose_args(working_dir)['workdir']<br>43: self.main_report = None<br>44: kwargs = {'name' : 'analysis'}<br>45: kwargs['flunkOnFailure'] = True<br>46: buildstep.BuildStep.__init__(self, **kwargs)<br>47: <br>...</div><div></div><div>...</div><div>53: <br>54: @defer.inlineCallbacks<br>55: def run(self):<br>...<br>65: self.main_report = self.retrieve_execution_report(self.build.main_stdout)<br>...</div><div>71: for test in self.main_report.failing_tests:<br>72: log.msg('uploading ', test)<br>73: try:<br>74: log_path = './' + self.main_report.failing_tests[test].replace('\\', '/') if buildbot_development_instance() else self.main_report.failing_tests[test]<br>75: <br>76: def publish(remote_cmd):<br>77: import itertools<br>78: file_path = remote_cmd.args['slavesrc'] if self.workerVersionIsOlderThan('uploadFile', '3.0') else remote_cmd.args['workersrc']<br>79: path_items = [i.split('/') for i in file_path.split('\\')]<br>80: path_items_flat = list(itertools.chain(*path_items))<br>81: test_name = path_items_flat[-1]<br>82: self.addCompleteLog('TEST FAILED: ' + test_name, remote_cmd.args['writer'].buffer)<br>83: log.msg('uploading SUCCEEDED.')<br>84: <br>85: cmd = self.setupRetrieveCmd(log_path)<br>86: d = cmd.run(self, self.remote, <a href="http://self.build.builder.name" target="_blank">self.build.builder.name</a>)<br>87: d.addCallback(publish)<br>88: yield d<br>89: except Exception as e:<br>90: log.msg('uploading FAILED: ', str(e))<br>91: finally:<br>92: self.cmd = None<br>93: <br>94: defer.returnValue(FAILURE)<br>...</div><div>101: <br>102: def setupRetrieveCmd(self, file_path):<br>103: from buildbot.process import remotecommand<br>104: from buildbot.process import remotetransfer<br>105: <br>106: remote_cmd_name = 'uploadFile'<br>107: <br>108: self.checkWorkerHasCommand(remote_cmd_name)<br>109: <br>110: fileWriter = remotetransfer.StringFileWriter()<br>111: args = {<br>112: 'workdir': self.workdir,<br>113: 'writer': fileWriter,<br>114: 'maxsize': None,<br>115: 'blocksize': 32 * 1024,<br>116: }<br>117: <br>118: if self.workerVersionIsOlderThan(remote_cmd_name, '3.0'):<br>119: args['slavesrc'] = file_path<br>120: else:<br>121: args['workersrc'] = file_path<br>122: <br>123: cmd = remotecommand.RemoteCommand(remote_cmd_name, args)<br>124: if hasattr(self, "rc_log"):<br>125: cmd.useLog(self.rc_log, False)<br>126: <br>127: cmd.worker = self.worker<br>128: return cmd<br><br></div><br><br><br></div></div></div></div></div></div></div></div>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote></div>