[Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter
Benoît Allard
benoit at aeteurope.nl
Tue Feb 8 08:57:43 UTC 2011
As a side-note, this trouble would'nt have been there if an XCode
compile step would be provided by buildbot.
Would you care making your one generic //enough// and sending it for
inclusion ? I think it should'nt be that different from the Visual
Studio steps (without all the ENV stuff maybe.)
About your MailNotifier trouble, what about the addLogs parameter ?
Regards
Benoit
Fulvio Cervone wrote:
> Thanks Philippe
>
> I've actually found that I don't even have to use the
> WarningCountShellCommand, since I can override the regular method in
> ShellCommand called createSummary (as you suggested).
>
> Once done that I can add multiple logs in the waterfall, so the first
> problem is solved! So I avoid to have multiple steps, and as you
> noticed, I must present warnings and errors in the same step.
>
> This is the code that I use (maybe can be useful for someone else)
>
> class customShellCommand(ShellCommand):
>
> command=[ "xcodebuild", "-configuration", "Debug" ]
> def createSummary(self, log):
> errors = []
> warnings = []
> for line in log.readlines():
> if "error:" in line:
> errors.append(line)
> elif "warning:" in line:
> warnings.append(line)
> self.addCompleteLog('errors', "".join(errors))
> self.addCompleteLog('warnings', "".join(warnings))
>
> f.addStep(customShellCommand)
>
> I am surprised how easy is now to customize classes overriding certain
> methods; is just a matter of finding the right info about how to use or
> overload a specific class :)
>
> Now is there anything in the docs that can show me how can i get these
> warnings and error logs and add them to the email formatter in the
> mailnotifier?
>
> thanks!
>
> On Feb 7, 2011, at 4:20 PM, Philippe McLean wrote:
>
>> The vstudio.py (and others) present stdio, warnings, and error logs.
>>
>> You can add any number of named logs using text you have built up, in
>> the createSummary() method.
>>
>> This should all be done in one Step, since (I'm guessing) you want to
>> present summaries of the results of a one compile?
>>
>> xcode understands python files.
>>
>> I have found the best way to work with buildbot is to start with a
>> very simple modification of an existing class, that executes a simple
>> dummy compile, and make small changes iteratively.
>>
>> good luck!
>>
>> On Mon, Feb 7, 2011 at 3:12 PM, Fulvio Cervone <fcervone at me.com
>> <mailto:fcervone at me.com>> wrote:
>>
>> Thanks Alexander and Philippe
>>
>> I've found (not without a lot of perils) the class
>> called* *WarningCountingShellCommand;
>>
>> http://buildbot.net/buildbot/docs/0.8.3/reference/buildbot.steps.shell.WarningCountingShellCommand-class.html
>>
>> Once found the class, I tried to apply it instead of the
>> ShellCommand, and actually I can get the warnings and errors (just
>> have to modify the warning pattern)
>>
>> Now if i wanna get 2 logs(warning and errors) should i run 2 steps
>> or is there a way to specify more than one log, with different
>> name? (as now the Waterfall page shows "warnings (n)"
>> independently from what i use as search pattern)
>>
>> BTW I use Xcode, is the vstudio.py still useful to get the gist of
>> how this works?
>>
>>
>>
>>
>> On Feb 7, 2011, at 3:01 PM, Philippe McLean wrote:
>>
>>> The visual studio compile steps in buildbot/steps/vstudio.py also
>>> have an example of doing this.
>>>
>>> On Mon, Feb 7, 2011 at 1:55 PM, Alexander O'Donovan-Jones
>>> <Alexander at ccpgames.com <mailto:Alexander at ccpgames.com>> wrote:
>>>
>>> Have you looked into the WarningCountingShellCommand class?
>>> It does pretty much what you're after.
>>> ________________________________________
>>> From: Fulvio Cervone [fcervone at me.com <mailto:fcervone at me.com>]
>>> Sent: 07 February 2011 21:16
>>> To: buildbot-devel list
>>> Subject: [Buildbot-devel] Parse the stdio file for errors and
>>> warnings and add them to the waterfall page and on an email
>>> formatter
>>>
>>> I am getting kinda swamped in this.....
>>>
>>> I've checked the manual and the mailing list, and seems that
>>> there is no quick way to do a grep on the stdio and just get
>>> the output and show it on the waterfall page, so from what i
>>> have seen, you have to use logfiles and LogObservers.
>>>
>>> What i want to achieve is to get the errors and warnings from
>>> the build process and put them in 2 files on the waterfall
>>> page, and in case of errors to get also the output of the
>>> file into my customized email formatter.
>>>
>>> I've found this example here on a post in the mailing list
>>> history:
>>>
>>> === 2
>>>
>>> class BetterCompile(Compile):
>>> command = ["make", "all"]
>>>
>>> def createSummary(self, log):
>>> # 'log' is the stdio LogFile, there might be others
>>> text = StringIO(log.getText())
>>> warnings = [line for line in text.readlines()
>>> if line.startswith("warning:")]
>>> self.addCompleteLog('warnings', "".join(warnings))
>>>
>>> f.addStep(BetterCompile)
>>>
>>> This would work for what i need (just have to do another
>>> parsing for the errors and i am done basically), but I am not
>>> clear how this works...
>>>
>>> 1) the class is created inheriting from Compile, I do not use
>>> compile but ShellCommand; may I do the same using
>>> ShellCommand? (changing of course the command)
>>>
>>> 2) when the def createSummary is done, self refers to the
>>> actual step (and the class of course), but log is coming from
>>> where? I was not able to find the declaration or assignment
>>> statement for logs.....it should come from the step itself
>>> right? In this case what kind of declaration should i do to
>>> get the object "log"?
>>>
>>> I have done this so far to get the errors:
>>>
>>> class BetterCompile(ShellCommand):
>>> command = ["xcodebuild", "-configuration", "Debug"]
>>>
>>> def createSummary(self, log):
>>> # 'log' is the stdio LogFile, there might be others
>>> text = StringIO(log.getText())
>>> warnings = [line for line in text.readlines()
>>> if line.startswith("warning:")]
>>> self.addCompleteLog('warnings', "".join(warnings))
>>>
>>> f.addStep(BetterCompile)
>>>
>>> and when i do the checkconfig i don't get any error, but when
>>> i run the step it gives me an exception
>>>
>>> 2011-02-07 13:04:39-0800 [-] BuildStep.failed, traceback follows
>>> 2011-02-07 13:04:39-0800 [-] Unhandled Error
>>> Traceback (most recent call last):
>>> File
>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>> line 186, in addCallbacks
>>> self._runCallbacks()
>>> File
>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>> line 328, in _runCallbacks
>>> self.result = callback(self.result, *args, **kw)
>>> File
>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>> line 243, in callback
>>> self._startRunCallbacks(result)
>>> File
>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>> line 312, in _startRunCallbacks
>>> self._runCallbacks()
>>> --- <exception caught here> ---
>>> File
>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>> line 328, in _runCallbacks
>>> self.result = callback(self.result, *args, **kw)
>>> File
>>> "/Library/Python/2.6/site-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/process/buildstep.py",
>>> line 1093, in <lambda>
>>> d.addCallback(lambda res:
>>> self.createSummary(cmd.logs['stdio']))
>>> File "/masterbot/master.cfg", line 140, in createSummary
>>> text = StringIO(log.getText())
>>> exceptions.NameError: global name 'StringIO' is not defined
>>>
>>>
>>> Am I missing something or this example is not working? Is
>>> there any other way to just get the stdio file output and
>>> grep it for errors and warning, and then put the output in an
>>> object that can be accessed by the mail notifier and the
>>> waterfall page?
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> The ultimate all-in-one performance toolkit: Intel(R)
>>> Parallel Studio XE:
>>> Pinpoint memory and threading errors before they happen.
>>> Find and fix more than 250 security defects in the
>>> development cycle.
>>> Locate bottlenecks in serial and parallel code that limit
>>> performance.
>>> http://p.sf.net/sfu/intel-dev2devfeb
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> <mailto:Buildbot-devel at lists.sourceforge.net>
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>
>>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6031 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://buildbot.net/pipermail/devel/attachments/20110208/f52d55d9/attachment.bin>
More information about the devel
mailing list