[Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter

Benoît Allard benoit at aeteurope.nl
Tue Feb 8 08:57:43 UTC 2011


As a side-note, this trouble would'nt have been there if an XCode 
compile step would be provided by buildbot.

Would you care making your one generic //enough// and sending it for 
inclusion ? I think it should'nt be that different from the Visual 
Studio steps (without all the ENV stuff maybe.)

About your MailNotifier trouble, what about the addLogs parameter ?

Regards
Benoit

Fulvio Cervone wrote:
> Thanks Philippe
> 
> I've actually found that I don't even have to use the 
> WarningCountShellCommand, since I can override the regular method in 
> ShellCommand called createSummary (as you suggested).
> 
> Once done that I can add multiple logs in the waterfall, so the first 
> problem is solved! So I avoid to have multiple steps, and as you 
> noticed, I must present warnings and errors in the same step.
> 
> This is the code that I use (maybe can be useful for someone else)
> 
> class customShellCommand(ShellCommand):
> 
> command=[ "xcodebuild", "-configuration", "Debug" ]
> def createSummary(self, log):
> errors = []
> warnings = []
> for line in log.readlines():
> if "error:" in line:
> errors.append(line)
> elif "warning:" in line:
> warnings.append(line)
> self.addCompleteLog('errors', "".join(errors))
> self.addCompleteLog('warnings', "".join(warnings))
> 
> f.addStep(customShellCommand)
> 
> I am surprised how easy is now to customize classes overriding certain 
> methods; is just a matter of finding the right info about how to use or 
> overload a specific class :)
> 
> Now is there anything in the docs that can show me how can i get these 
> warnings and error logs and add them to the email formatter in the 
> mailnotifier?
> 
> thanks!
> 
> On Feb 7, 2011, at 4:20 PM, Philippe McLean wrote:
> 
>> The vstudio.py (and others) present stdio, warnings, and error logs. 
>>
>> You can add any number of named logs using text you have built up, in 
>> the createSummary() method.
>>
>> This should all be done in one Step, since (I'm guessing) you want to 
>> present summaries of the results of a one compile?
>>
>> xcode understands python files.
>>
>> I have found the best way to work with buildbot is to start with a 
>> very simple modification of an existing class, that executes a simple 
>> dummy compile, and make small changes iteratively.
>>
>> good luck!
>>
>> On Mon, Feb 7, 2011 at 3:12 PM, Fulvio Cervone <fcervone at me.com 
>> <mailto:fcervone at me.com>> wrote:
>>
>>     Thanks Alexander and Philippe
>>
>>     I've found (not without a lot of perils) the class
>>     called* *WarningCountingShellCommand;
>>
>>     http://buildbot.net/buildbot/docs/0.8.3/reference/buildbot.steps.shell.WarningCountingShellCommand-class.html
>>
>>     Once found the class, I tried to apply it instead of the
>>     ShellCommand, and actually I can get the warnings and errors (just
>>     have to modify the warning pattern)
>>
>>     Now if i wanna get 2 logs(warning and errors) should i run 2 steps
>>     or is there a way to specify more than one log, with different
>>     name? (as now the Waterfall page shows "warnings (n)"
>>     independently from what i use as search pattern)
>>
>>     BTW I use Xcode, is the vstudio.py still useful to get the gist of
>>     how this works?
>>
>>
>>
>>
>>     On Feb 7, 2011, at 3:01 PM, Philippe McLean wrote:
>>
>>>     The visual studio compile steps in buildbot/steps/vstudio.py also
>>>     have an example of doing this. 
>>>
>>>     On Mon, Feb 7, 2011 at 1:55 PM, Alexander O'Donovan-Jones
>>>     <Alexander at ccpgames.com <mailto:Alexander at ccpgames.com>> wrote:
>>>
>>>         Have you looked into the WarningCountingShellCommand class?
>>>         It does pretty much what you're after.
>>>         ________________________________________
>>>         From: Fulvio Cervone [fcervone at me.com <mailto:fcervone at me.com>]
>>>         Sent: 07 February 2011 21:16
>>>         To: buildbot-devel list
>>>         Subject: [Buildbot-devel] Parse the stdio file for errors and
>>>         warnings and add them to the waterfall page and on an email
>>>         formatter
>>>
>>>         I am getting kinda swamped in this.....
>>>
>>>         I've checked the manual and the mailing list, and seems that
>>>         there is no quick way to do a grep on the stdio and just get
>>>         the output and show it on the waterfall page, so from what i
>>>         have seen, you have to use logfiles and LogObservers.
>>>
>>>         What i want to achieve is to get the errors and warnings from
>>>         the build process and put them in 2 files on the waterfall
>>>         page, and in case of errors to get also the output of the
>>>         file into my customized email formatter.
>>>
>>>         I've found this example here on a post in the mailing list
>>>         history:
>>>
>>>         === 2
>>>
>>>         class BetterCompile(Compile):
>>>         command = ["make", "all"]
>>>
>>>         def createSummary(self, log):
>>>         # 'log' is the stdio LogFile, there might be others
>>>         text = StringIO(log.getText())
>>>         warnings = [line for line in text.readlines()
>>>         if line.startswith("warning:")]
>>>         self.addCompleteLog('warnings', "".join(warnings))
>>>
>>>         f.addStep(BetterCompile)
>>>
>>>         This would work for what i need (just have to do another
>>>         parsing for the errors and i am done basically), but I am not
>>>         clear how this works...
>>>
>>>         1) the class is created inheriting from Compile, I do not use
>>>         compile but ShellCommand; may I do the same using
>>>         ShellCommand? (changing of course the command)
>>>
>>>         2) when the def createSummary is done, self refers to the
>>>         actual step (and the class of course), but log is coming from
>>>         where? I was not able to find the declaration or assignment
>>>         statement for logs.....it should come from the step itself
>>>         right? In this case what kind of declaration should i do to
>>>         get the object "log"?
>>>
>>>         I have done this so far to get the errors:
>>>
>>>         class BetterCompile(ShellCommand):
>>>         command = ["xcodebuild", "-configuration", "Debug"]
>>>
>>>         def createSummary(self, log):
>>>         # 'log' is the stdio LogFile, there might be others
>>>         text = StringIO(log.getText())
>>>         warnings = [line for line in text.readlines()
>>>         if line.startswith("warning:")]
>>>         self.addCompleteLog('warnings', "".join(warnings))
>>>
>>>         f.addStep(BetterCompile)
>>>
>>>         and when i do the checkconfig i don't get any error, but when
>>>         i run the step it gives me an exception
>>>
>>>         2011-02-07 13:04:39-0800 [-] BuildStep.failed, traceback follows
>>>         2011-02-07 13:04:39-0800 [-] Unhandled Error
>>>         Traceback (most recent call last):
>>>          File
>>>         "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>         line 186, in addCallbacks
>>>           self._runCallbacks()
>>>          File
>>>         "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>         line 328, in _runCallbacks
>>>           self.result = callback(self.result, *args, **kw)
>>>          File
>>>         "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>         line 243, in callback
>>>           self._startRunCallbacks(result)
>>>          File
>>>         "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>         line 312, in _startRunCallbacks
>>>           self._runCallbacks()
>>>         --- <exception caught here> ---
>>>          File
>>>         "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>         line 328, in _runCallbacks
>>>           self.result = callback(self.result, *args, **kw)
>>>          File
>>>         "/Library/Python/2.6/site-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/process/buildstep.py",
>>>         line 1093, in <lambda>
>>>           d.addCallback(lambda res:
>>>         self.createSummary(cmd.logs['stdio']))
>>>          File "/masterbot/master.cfg", line 140, in createSummary
>>>           text = StringIO(log.getText())
>>>         exceptions.NameError: global name 'StringIO' is not defined
>>>
>>>
>>>         Am I missing something or this example is not working? Is
>>>         there any other way to just get the stdio file output and
>>>         grep it for errors and warning, and then put the output in an
>>>         object that can be accessed by the mail notifier and the
>>>         waterfall page?
>>>
>>>         Thanks!
>>>
>>>
>>>
>>>
>>>         ------------------------------------------------------------------------------
>>>         The ultimate all-in-one performance toolkit: Intel(R)
>>>         Parallel Studio XE:
>>>         Pinpoint memory and threading errors before they happen.
>>>         Find and fix more than 250 security defects in the
>>>         development cycle.
>>>         Locate bottlenecks in serial and parallel code that limit
>>>         performance.
>>>         http://p.sf.net/sfu/intel-dev2devfeb
>>>         _______________________________________________
>>>         Buildbot-devel mailing list
>>>         Buildbot-devel at lists.sourceforge.net
>>>         <mailto:Buildbot-devel at lists.sourceforge.net>
>>>         https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>
>>>
>>
>>
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6031 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://buildbot.net/pipermail/devel/attachments/20110208/f52d55d9/attachment.bin>


More information about the devel mailing list