[Buildbot-devel] Marking a build step as failed unless the STDOUT contains a magic string.

[Matisse Enzer]

On Jan 22, 2008, at 9:29 AM, Scott Lawrence wrote:

>
> On Tue, 2008-01-22 at 18:17 +0100, Daniel Svensson wrote:
>> On Jan 22, 2008 5:10 PM, Scott Lawrence <slawrence at pingtel.com>  
>> wrote:
>>> It's not very robust to look for magic strings.  It's much better to
>>> just make sure that your makefile is written so that 'make' will  
>>> return
>>> a failure status unless all tests pass.
>>
>> How do you show how many tests passed/failed? What if you want to
>> extend the mechanism to keep track of testcases that doesn't meet the
>> required coverage goal?
>
> I didn't mean to imply that you shouldn't keep track of the numbers -
> just that looking for strings in the output is easy to break.
>
> With some work, you can get the makefile itself to enforce nearly any
> rule you can dream up to make the pass/fail decision, but it's easiest
> if you just get that decision as an exit status from 'make'.


The make target in question runs tests and then submits the results to  
a server (http://smoke.parrotcode.org/smoke/) - the target "succeeds"  
if the submission of data succeeds, regardless of the tests themselves  
passing or failing.

So unless I want to have a separate step that just runs 'make  
test' (which duplicates running the tests), looking for the magic  
string is the best I can do for now.


-------------------------------------------------------
Matisse Enzer <matisse at matisse.net>
http://www.matisse.net/  - http://www.eigenstate.net/