[Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter
Fulvio Cervone
fcervone at me.com
Tue Feb 8 09:30:30 UTC 2011
Agree Benoit; but Buildbot is generic so it can be used with many different compilers and IDE; Xcode is not even close to VS so I guess that this is the reason why there is no specific step for the compile class.
I can try t make a generic one, but my python skills are beginner level :) I've started learning it with BB a couple of weeks ago.
The only thing that i have found is the addLog parameter in the buildstep.ShellCommand class, but from what i can tell it will just point to a log that you use to save ; I am using the only fully working example on the mail notifier doc is the one that prints the last 80 lines of stdio
Tried to do something like
if log.getName() == 'warnings': #warnings is the name of the log file that I have created with the addComplete statement)
in the example on the manual, and this will print the warnings, but if i need also the errors I am kinda lost; since the iterator can get one log file at time, so I should go in and load warnings, parse, send and append in the mail_formatter, then load the next file and do the same? Or is there a way to concatenate the 2 logs and just do a text.append for the whole content of both?
On Feb 8, 2011, at 12:57 AM, Benoît Allard wrote:
> As a side-note, this trouble would'nt have been there if an XCode compile step would be provided by buildbot.
>
> Would you care making your one generic //enough// and sending it for inclusion ? I think it should'nt be that different from the Visual Studio steps (without all the ENV stuff maybe.)
>
> About your MailNotifier trouble, what about the addLogs parameter ?
>
> Regards
> Benoit
>
> Fulvio Cervone wrote:
>> Thanks Philippe
>> I've actually found that I don't even have to use the WarningCountShellCommand, since I can override the regular method in ShellCommand called createSummary (as you suggested).
>> Once done that I can add multiple logs in the waterfall, so the first problem is solved! So I avoid to have multiple steps, and as you noticed, I must present warnings and errors in the same step.
>> This is the code that I use (maybe can be useful for someone else)
>> class customShellCommand(ShellCommand):
>> command=[ "xcodebuild", "-configuration", "Debug" ]
>> def createSummary(self, log):
>> errors = []
>> warnings = []
>> for line in log.readlines():
>> if "error:" in line:
>> errors.append(line)
>> elif "warning:" in line:
>> warnings.append(line)
>> self.addCompleteLog('errors', "".join(errors))
>> self.addCompleteLog('warnings', "".join(warnings))
>> f.addStep(customShellCommand)
>> I am surprised how easy is now to customize classes overriding certain methods; is just a matter of finding the right info about how to use or overload a specific class :)
>> Now is there anything in the docs that can show me how can i get these warnings and error logs and add them to the email formatter in the mailnotifier?
>> thanks!
>> On Feb 7, 2011, at 4:20 PM, Philippe McLean wrote:
>>> The vstudio.py (and others) present stdio, warnings, and error logs.
>>> You can add any number of named logs using text you have built up, in the createSummary() method.
>>>
>>> This should all be done in one Step, since (I'm guessing) you want to present summaries of the results of a one compile?
>>>
>>> xcode understands python files.
>>>
>>> I have found the best way to work with buildbot is to start with a very simple modification of an existing class, that executes a simple dummy compile, and make small changes iteratively.
>>>
>>> good luck!
>>>
>>> On Mon, Feb 7, 2011 at 3:12 PM, Fulvio Cervone <fcervone at me.com <mailto:fcervone at me.com>> wrote:
>>>
>>> Thanks Alexander and Philippe
>>>
>>> I've found (not without a lot of perils) the class
>>> called* *WarningCountingShellCommand;
>>>
>>> http://buildbot.net/buildbot/docs/0.8.3/reference/buildbot.steps.shell.WarningCountingShellCommand-class.html
>>>
>>> Once found the class, I tried to apply it instead of the
>>> ShellCommand, and actually I can get the warnings and errors (just
>>> have to modify the warning pattern)
>>>
>>> Now if i wanna get 2 logs(warning and errors) should i run 2 steps
>>> or is there a way to specify more than one log, with different
>>> name? (as now the Waterfall page shows "warnings (n)"
>>> independently from what i use as search pattern)
>>>
>>> BTW I use Xcode, is the vstudio.py still useful to get the gist of
>>> how this works?
>>>
>>>
>>>
>>>
>>> On Feb 7, 2011, at 3:01 PM, Philippe McLean wrote:
>>>
>>>> The visual studio compile steps in buildbot/steps/vstudio.py also
>>>> have an example of doing this.
>>>> On Mon, Feb 7, 2011 at 1:55 PM, Alexander O'Donovan-Jones
>>>> <Alexander at ccpgames.com <mailto:Alexander at ccpgames.com>> wrote:
>>>>
>>>> Have you looked into the WarningCountingShellCommand class?
>>>> It does pretty much what you're after.
>>>> ________________________________________
>>>> From: Fulvio Cervone [fcervone at me.com <mailto:fcervone at me.com>]
>>>> Sent: 07 February 2011 21:16
>>>> To: buildbot-devel list
>>>> Subject: [Buildbot-devel] Parse the stdio file for errors and
>>>> warnings and add them to the waterfall page and on an email
>>>> formatter
>>>>
>>>> I am getting kinda swamped in this.....
>>>>
>>>> I've checked the manual and the mailing list, and seems that
>>>> there is no quick way to do a grep on the stdio and just get
>>>> the output and show it on the waterfall page, so from what i
>>>> have seen, you have to use logfiles and LogObservers.
>>>>
>>>> What i want to achieve is to get the errors and warnings from
>>>> the build process and put them in 2 files on the waterfall
>>>> page, and in case of errors to get also the output of the
>>>> file into my customized email formatter.
>>>>
>>>> I've found this example here on a post in the mailing list
>>>> history:
>>>>
>>>> === 2
>>>>
>>>> class BetterCompile(Compile):
>>>> command = ["make", "all"]
>>>>
>>>> def createSummary(self, log):
>>>> # 'log' is the stdio LogFile, there might be others
>>>> text = StringIO(log.getText())
>>>> warnings = [line for line in text.readlines()
>>>> if line.startswith("warning:")]
>>>> self.addCompleteLog('warnings', "".join(warnings))
>>>>
>>>> f.addStep(BetterCompile)
>>>>
>>>> This would work for what i need (just have to do another
>>>> parsing for the errors and i am done basically), but I am not
>>>> clear how this works...
>>>>
>>>> 1) the class is created inheriting from Compile, I do not use
>>>> compile but ShellCommand; may I do the same using
>>>> ShellCommand? (changing of course the command)
>>>>
>>>> 2) when the def createSummary is done, self refers to the
>>>> actual step (and the class of course), but log is coming from
>>>> where? I was not able to find the declaration or assignment
>>>> statement for logs.....it should come from the step itself
>>>> right? In this case what kind of declaration should i do to
>>>> get the object "log"?
>>>>
>>>> I have done this so far to get the errors:
>>>>
>>>> class BetterCompile(ShellCommand):
>>>> command = ["xcodebuild", "-configuration", "Debug"]
>>>>
>>>> def createSummary(self, log):
>>>> # 'log' is the stdio LogFile, there might be others
>>>> text = StringIO(log.getText())
>>>> warnings = [line for line in text.readlines()
>>>> if line.startswith("warning:")]
>>>> self.addCompleteLog('warnings', "".join(warnings))
>>>>
>>>> f.addStep(BetterCompile)
>>>>
>>>> and when i do the checkconfig i don't get any error, but when
>>>> i run the step it gives me an exception
>>>>
>>>> 2011-02-07 13:04:39-0800 [-] BuildStep.failed, traceback follows
>>>> 2011-02-07 13:04:39-0800 [-] Unhandled Error
>>>> Traceback (most recent call last):
>>>> File
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>> line 186, in addCallbacks
>>>> self._runCallbacks()
>>>> File
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>> line 328, in _runCallbacks
>>>> self.result = callback(self.result, *args, **kw)
>>>> File
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>> line 243, in callback
>>>> self._startRunCallbacks(result)
>>>> File
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>> line 312, in _startRunCallbacks
>>>> self._runCallbacks()
>>>> --- <exception caught here> ---
>>>> File
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>> line 328, in _runCallbacks
>>>> self.result = callback(self.result, *args, **kw)
>>>> File
>>>> "/Library/Python/2.6/site-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/process/buildstep.py",
>>>> line 1093, in <lambda>
>>>> d.addCallback(lambda res:
>>>> self.createSummary(cmd.logs['stdio']))
>>>> File "/masterbot/master.cfg", line 140, in createSummary
>>>> text = StringIO(log.getText())
>>>> exceptions.NameError: global name 'StringIO' is not defined
>>>>
>>>>
>>>> Am I missing something or this example is not working? Is
>>>> there any other way to just get the stdio file output and
>>>> grep it for errors and warning, and then put the output in an
>>>> object that can be accessed by the mail notifier and the
>>>> waterfall page?
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> The ultimate all-in-one performance toolkit: Intel(R)
>>>> Parallel Studio XE:
>>>> Pinpoint memory and threading errors before they happen.
>>>> Find and fix more than 250 security defects in the
>>>> development cycle.
>>>> Locate bottlenecks in serial and parallel code that limit
>>>> performance.
>>>> http://p.sf.net/sfu/intel-dev2devfeb
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> <mailto:Buildbot-devel at lists.sourceforge.net>
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>
>>>>
>>>
>>>
>> ------------------------------------------------------------------------
>> ------------------------------------------------------------------------------
>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>> Pinpoint memory and threading errors before they happen.
>> Find and fix more than 250 security defects in the development cycle.
>> Locate bottlenecks in serial and parallel code that limit performance.
>> http://p.sf.net/sfu/intel-dev2devfeb
>> ------------------------------------------------------------------------
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
More information about the devel
mailing list