[Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter
Fulvio Cervone
fcervone at me.com
Tue Feb 8 01:20:37 UTC 2011
Thanks Philippe
I've actually found that I don't even have to use the WarningCountShellCommand, since I can override the regular method in ShellCommand called createSummary (as you suggested).
Once done that I can add multiple logs in the waterfall, so the first problem is solved! So I avoid to have multiple steps, and as you noticed, I must present warnings and errors in the same step.
This is the code that I use (maybe can be useful for someone else)
class customShellCommand(ShellCommand):
command=[ "xcodebuild", "-configuration", "Debug" ]
def createSummary(self, log):
errors = []
warnings = []
for line in log.readlines():
if "error:" in line:
errors.append(line)
elif "warning:" in line:
warnings.append(line)
self.addCompleteLog('errors', "".join(errors))
self.addCompleteLog('warnings', "".join(warnings))
f.addStep(customShellCommand)
I am surprised how easy is now to customize classes overriding certain methods; is just a matter of finding the right info about how to use or overload a specific class :)
Now is there anything in the docs that can show me how can i get these warnings and error logs and add them to the email formatter in the mailnotifier?
thanks!
On Feb 7, 2011, at 4:20 PM, Philippe McLean wrote:
> The vstudio.py (and others) present stdio, warnings, and error logs.
>
> You can add any number of named logs using text you have built up, in the createSummary() method.
>
> This should all be done in one Step, since (I'm guessing) you want to present summaries of the results of a one compile?
>
> xcode understands python files.
>
> I have found the best way to work with buildbot is to start with a very simple modification of an existing class, that executes a simple dummy compile, and make small changes iteratively.
>
> good luck!
>
> On Mon, Feb 7, 2011 at 3:12 PM, Fulvio Cervone <fcervone at me.com> wrote:
> Thanks Alexander and Philippe
>
> I've found (not without a lot of perils) the class called WarningCountingShellCommand;
>
> http://buildbot.net/buildbot/docs/0.8.3/reference/buildbot.steps.shell.WarningCountingShellCommand-class.html
>
> Once found the class, I tried to apply it instead of the ShellCommand, and actually I can get the warnings and errors (just have to modify the warning pattern)
>
> Now if i wanna get 2 logs(warning and errors) should i run 2 steps or is there a way to specify more than one log, with different name? (as now the Waterfall page shows "warnings (n)" independently from what i use as search pattern)
>
> BTW I use Xcode, is the vstudio.py still useful to get the gist of how this works?
>
>
>
>
> On Feb 7, 2011, at 3:01 PM, Philippe McLean wrote:
>
>> The visual studio compile steps in buildbot/steps/vstudio.py also have an example of doing this.
>>
>> On Mon, Feb 7, 2011 at 1:55 PM, Alexander O'Donovan-Jones <Alexander at ccpgames.com> wrote:
>> Have you looked into the WarningCountingShellCommand class? It does pretty much what you're after.
>> ________________________________________
>> From: Fulvio Cervone [fcervone at me.com]
>> Sent: 07 February 2011 21:16
>> To: buildbot-devel list
>> Subject: [Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter
>>
>> I am getting kinda swamped in this.....
>>
>> I've checked the manual and the mailing list, and seems that there is no quick way to do a grep on the stdio and just get the output and show it on the waterfall page, so from what i have seen, you have to use logfiles and LogObservers.
>>
>> What i want to achieve is to get the errors and warnings from the build process and put them in 2 files on the waterfall page, and in case of errors to get also the output of the file into my customized email formatter.
>>
>> I've found this example here on a post in the mailing list history:
>>
>> === 2
>>
>> class BetterCompile(Compile):
>> command = ["make", "all"]
>>
>> def createSummary(self, log):
>> # 'log' is the stdio LogFile, there might be others
>> text = StringIO(log.getText())
>> warnings = [line for line in text.readlines()
>> if line.startswith("warning:")]
>> self.addCompleteLog('warnings', "".join(warnings))
>>
>> f.addStep(BetterCompile)
>>
>> This would work for what i need (just have to do another parsing for the errors and i am done basically), but I am not clear how this works...
>>
>> 1) the class is created inheriting from Compile, I do not use compile but ShellCommand; may I do the same using ShellCommand? (changing of course the command)
>>
>> 2) when the def createSummary is done, self refers to the actual step (and the class of course), but log is coming from where? I was not able to find the declaration or assignment statement for logs.....it should come from the step itself right? In this case what kind of declaration should i do to get the object "log"?
>>
>> I have done this so far to get the errors:
>>
>> class BetterCompile(ShellCommand):
>> command = ["xcodebuild", "-configuration", "Debug"]
>>
>> def createSummary(self, log):
>> # 'log' is the stdio LogFile, there might be others
>> text = StringIO(log.getText())
>> warnings = [line for line in text.readlines()
>> if line.startswith("warning:")]
>> self.addCompleteLog('warnings', "".join(warnings))
>>
>> f.addStep(BetterCompile)
>>
>> and when i do the checkconfig i don't get any error, but when i run the step it gives me an exception
>>
>> 2011-02-07 13:04:39-0800 [-] BuildStep.failed, traceback follows
>> 2011-02-07 13:04:39-0800 [-] Unhandled Error
>> Traceback (most recent call last):
>> File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 186, in addCallbacks
>> self._runCallbacks()
>> File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 328, in _runCallbacks
>> self.result = callback(self.result, *args, **kw)
>> File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 243, in callback
>> self._startRunCallbacks(result)
>> File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 312, in _startRunCallbacks
>> self._runCallbacks()
>> --- <exception caught here> ---
>> File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 328, in _runCallbacks
>> self.result = callback(self.result, *args, **kw)
>> File "/Library/Python/2.6/site-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/process/buildstep.py", line 1093, in <lambda>
>> d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
>> File "/masterbot/master.cfg", line 140, in createSummary
>> text = StringIO(log.getText())
>> exceptions.NameError: global name 'StringIO' is not defined
>>
>>
>> Am I missing something or this example is not working? Is there any other way to just get the stdio file output and grep it for errors and warning, and then put the output in an object that can be accessed by the mail notifier and the waterfall page?
>>
>> Thanks!
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>> Pinpoint memory and threading errors before they happen.
>> Find and fix more than 250 security defects in the development cycle.
>> Locate bottlenecks in serial and parallel code that limit performance.
>> http://p.sf.net/sfu/intel-dev2devfeb
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20110207/21058e98/attachment.html>
More information about the devel
mailing list