[Buildbot-devel] Parse the stdio file for errors and warnings and add them to the waterfall page and on an email formatter
Benoît Allard
benoit at aeteurope.nl
Tue Feb 8 10:14:58 UTC 2011
Fulvio Cervone wrote:
> Agree Benoit; but Buildbot is generic so it can be used with many different compilers and IDE; Xcode is not even close to VS so I guess that this is the reason why there is no specific step for the compile class.
>
> I can try t make a generic one, but my python skills are beginner level :) I've started learning it with BB a couple of weeks ago.
>
What I meant there is that they both do similar stuff: launch a
particular command (with optionally a particular environment), and parse
the logs. To just implement those functionnality, reproducing what the
visualstudio steps do should work fine.
It can be found there:
<https://github.com/buildbot/buildbot/blob/master/master/buildbot/steps/vstudio.py>
> The only thing that i have found is the addLog parameter in the buildstep.ShellCommand class, but from what i can tell it will just point to a log that you use to save ; I am using the only fully working example on the mail notifier doc is the one that prints the last 80 lines of stdio
>
I meant the addLogs from the MailNotifier
<http://buildbot.net/buildbot/docs/current/MailNotifier.html#MailNotifier>
and then scroll further down to "MailNotifier arguments"
> Tried to do something like
>
> if log.getName() == 'warnings': #warnings is the name of the log file that I have created with the addComplete statement)
>
> in the example on the manual, and this will print the warnings, but if i need also the errors I am kinda lost; since the iterator can get one log file at time, so I should go in and load warnings, parse, send and append in the mail_formatter, then load the next file and do the same? Or is there a way to concatenate the 2 logs and just do a text.append for the whole content of both?
>
>
> On Feb 8, 2011, at 12:57 AM, Benoît Allard wrote:
>
>> As a side-note, this trouble would'nt have been there if an XCode compile step would be provided by buildbot.
>>
>> Would you care making your one generic //enough// and sending it for inclusion ? I think it should'nt be that different from the Visual Studio steps (without all the ENV stuff maybe.)
>>
>> About your MailNotifier trouble, what about the addLogs parameter ?
>>
>> Regards
>> Benoit
>>
>> Fulvio Cervone wrote:
>>> Thanks Philippe
>>> I've actually found that I don't even have to use the WarningCountShellCommand, since I can override the regular method in ShellCommand called createSummary (as you suggested).
>>> Once done that I can add multiple logs in the waterfall, so the first problem is solved! So I avoid to have multiple steps, and as you noticed, I must present warnings and errors in the same step.
>>> This is the code that I use (maybe can be useful for someone else)
>>> class customShellCommand(ShellCommand):
>>> command=[ "xcodebuild", "-configuration", "Debug" ]
>>> def createSummary(self, log):
>>> errors = []
>>> warnings = []
>>> for line in log.readlines():
>>> if "error:" in line:
>>> errors.append(line)
>>> elif "warning:" in line:
>>> warnings.append(line)
>>> self.addCompleteLog('errors', "".join(errors))
>>> self.addCompleteLog('warnings', "".join(warnings))
>>> f.addStep(customShellCommand)
>>> I am surprised how easy is now to customize classes overriding certain methods; is just a matter of finding the right info about how to use or overload a specific class :)
>>> Now is there anything in the docs that can show me how can i get these warnings and error logs and add them to the email formatter in the mailnotifier?
>>> thanks!
>>> On Feb 7, 2011, at 4:20 PM, Philippe McLean wrote:
>>>> The vstudio.py (and others) present stdio, warnings, and error logs.
>>>> You can add any number of named logs using text you have built up, in the createSummary() method.
>>>>
>>>> This should all be done in one Step, since (I'm guessing) you want to present summaries of the results of a one compile?
>>>>
>>>> xcode understands python files.
>>>>
>>>> I have found the best way to work with buildbot is to start with a very simple modification of an existing class, that executes a simple dummy compile, and make small changes iteratively.
>>>>
>>>> good luck!
>>>>
>>>> On Mon, Feb 7, 2011 at 3:12 PM, Fulvio Cervone <fcervone at me.com <mailto:fcervone at me.com>> wrote:
>>>>
>>>> Thanks Alexander and Philippe
>>>>
>>>> I've found (not without a lot of perils) the class
>>>> called* *WarningCountingShellCommand;
>>>>
>>>> http://buildbot.net/buildbot/docs/0.8.3/reference/buildbot.steps.shell.WarningCountingShellCommand-class.html
>>>>
>>>> Once found the class, I tried to apply it instead of the
>>>> ShellCommand, and actually I can get the warnings and errors (just
>>>> have to modify the warning pattern)
>>>>
>>>> Now if i wanna get 2 logs(warning and errors) should i run 2 steps
>>>> or is there a way to specify more than one log, with different
>>>> name? (as now the Waterfall page shows "warnings (n)"
>>>> independently from what i use as search pattern)
>>>>
>>>> BTW I use Xcode, is the vstudio.py still useful to get the gist of
>>>> how this works?
>>>>
>>>>
>>>>
>>>>
>>>> On Feb 7, 2011, at 3:01 PM, Philippe McLean wrote:
>>>>
>>>>> The visual studio compile steps in buildbot/steps/vstudio.py also
>>>>> have an example of doing this.
>>>>> On Mon, Feb 7, 2011 at 1:55 PM, Alexander O'Donovan-Jones
>>>>> <Alexander at ccpgames.com <mailto:Alexander at ccpgames.com>> wrote:
>>>>>
>>>>> Have you looked into the WarningCountingShellCommand class?
>>>>> It does pretty much what you're after.
>>>>> ________________________________________
>>>>> From: Fulvio Cervone [fcervone at me.com <mailto:fcervone at me.com>]
>>>>> Sent: 07 February 2011 21:16
>>>>> To: buildbot-devel list
>>>>> Subject: [Buildbot-devel] Parse the stdio file for errors and
>>>>> warnings and add them to the waterfall page and on an email
>>>>> formatter
>>>>>
>>>>> I am getting kinda swamped in this.....
>>>>>
>>>>> I've checked the manual and the mailing list, and seems that
>>>>> there is no quick way to do a grep on the stdio and just get
>>>>> the output and show it on the waterfall page, so from what i
>>>>> have seen, you have to use logfiles and LogObservers.
>>>>>
>>>>> What i want to achieve is to get the errors and warnings from
>>>>> the build process and put them in 2 files on the waterfall
>>>>> page, and in case of errors to get also the output of the
>>>>> file into my customized email formatter.
>>>>>
>>>>> I've found this example here on a post in the mailing list
>>>>> history:
>>>>>
>>>>> === 2
>>>>>
>>>>> class BetterCompile(Compile):
>>>>> command = ["make", "all"]
>>>>>
>>>>> def createSummary(self, log):
>>>>> # 'log' is the stdio LogFile, there might be others
>>>>> text = StringIO(log.getText())
>>>>> warnings = [line for line in text.readlines()
>>>>> if line.startswith("warning:")]
>>>>> self.addCompleteLog('warnings', "".join(warnings))
>>>>>
>>>>> f.addStep(BetterCompile)
>>>>>
>>>>> This would work for what i need (just have to do another
>>>>> parsing for the errors and i am done basically), but I am not
>>>>> clear how this works...
>>>>>
>>>>> 1) the class is created inheriting from Compile, I do not use
>>>>> compile but ShellCommand; may I do the same using
>>>>> ShellCommand? (changing of course the command)
>>>>>
>>>>> 2) when the def createSummary is done, self refers to the
>>>>> actual step (and the class of course), but log is coming from
>>>>> where? I was not able to find the declaration or assignment
>>>>> statement for logs.....it should come from the step itself
>>>>> right? In this case what kind of declaration should i do to
>>>>> get the object "log"?
>>>>>
>>>>> I have done this so far to get the errors:
>>>>>
>>>>> class BetterCompile(ShellCommand):
>>>>> command = ["xcodebuild", "-configuration", "Debug"]
>>>>>
>>>>> def createSummary(self, log):
>>>>> # 'log' is the stdio LogFile, there might be others
>>>>> text = StringIO(log.getText())
>>>>> warnings = [line for line in text.readlines()
>>>>> if line.startswith("warning:")]
>>>>> self.addCompleteLog('warnings', "".join(warnings))
>>>>>
>>>>> f.addStep(BetterCompile)
>>>>>
>>>>> and when i do the checkconfig i don't get any error, but when
>>>>> i run the step it gives me an exception
>>>>>
>>>>> 2011-02-07 13:04:39-0800 [-] BuildStep.failed, traceback follows
>>>>> 2011-02-07 13:04:39-0800 [-] Unhandled Error
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>>> line 186, in addCallbacks
>>>>> self._runCallbacks()
>>>>> File
>>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>>> line 328, in _runCallbacks
>>>>> self.result = callback(self.result, *args, **kw)
>>>>> File
>>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>>> line 243, in callback
>>>>> self._startRunCallbacks(result)
>>>>> File
>>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>>> line 312, in _startRunCallbacks
>>>>> self._runCallbacks()
>>>>> --- <exception caught here> ---
>>>>> File
>>>>> "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py",
>>>>> line 328, in _runCallbacks
>>>>> self.result = callback(self.result, *args, **kw)
>>>>> File
>>>>> "/Library/Python/2.6/site-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/process/buildstep.py",
>>>>> line 1093, in <lambda>
>>>>> d.addCallback(lambda res:
>>>>> self.createSummary(cmd.logs['stdio']))
>>>>> File "/masterbot/master.cfg", line 140, in createSummary
>>>>> text = StringIO(log.getText())
>>>>> exceptions.NameError: global name 'StringIO' is not defined
>>>>>
>>>>>
>>>>> Am I missing something or this example is not working? Is
>>>>> there any other way to just get the stdio file output and
>>>>> grep it for errors and warning, and then put the output in an
>>>>> object that can be accessed by the mail notifier and the
>>>>> waterfall page?
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> The ultimate all-in-one performance toolkit: Intel(R)
>>>>> Parallel Studio XE:
>>>>> Pinpoint memory and threading errors before they happen.
>>>>> Find and fix more than 250 security defects in the
>>>>> development cycle.
>>>>> Locate bottlenecks in serial and parallel code that limit
>>>>> performance.
>>>>> http://p.sf.net/sfu/intel-dev2devfeb
>>>>> _______________________________________________
>>>>> Buildbot-devel mailing list
>>>>> Buildbot-devel at lists.sourceforge.net
>>>>> <mailto:Buildbot-devel at lists.sourceforge.net>
>>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>>
>>>>>
>>>>
>>> ------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------
>>> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>>> Pinpoint memory and threading errors before they happen.
>>> Find and fix more than 250 security defects in the development cycle.
>>> Locate bottlenecks in serial and parallel code that limit performance.
>>> http://p.sf.net/sfu/intel-dev2devfeb
>>> ------------------------------------------------------------------------
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6031 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://buildbot.net/pipermail/devel/attachments/20110208/89ce50e4/attachment.bin>
More information about the devel
mailing list