[users at bb.net] StopIteration Exceptions in LineConsumerLogObserver

[Yngve N. Pettersen]

Hi,

Minor correction i PoC below, doesn't change the result.

On Sun, 24 Jan 2021 02:30:03 +0100, Yngve N. Pettersen <yngve at vivaldi.com>  
wrote:

>
> Hi again,
>
> I just did a small bit of PoC testing, and based on that testing I  
> suspect that the problem is that LineConsumerLogObserver.finishReceived  
> is called too early from LogObserver.gotData because data is None,  
> before all data have actually been added to the log.
>
> The next line add then triggers a StopIteration exception.
>
> PoC:
>
> ================
> def foo():
>    while True:
>      try:
>        x = yield
>
>        print("X: %d" % x)
>      except StopIteration:

Should be "except GeneratorExit:"

>        print("Finished!")
>        return
>
> generator = foo()
>
> next(generator)
>
> feed = generator.send
>
> feed(1)
> feed(2)
> feed(3)
>
> generator.close()
>
> feed(4)
> ================
>
> feed(4) will fail with the exception StopIteration
>
> On Wed, 13 Jan 2021 17:52:51 +0100, Yngve N. Pettersen  
> <yngve at vivaldi.com> wrote:
>
>> Hi,
>>
>> We are using a class based on steps.ShellCommandNewStyle to process log  
>> output using LineConsumerLogObserver from a number of tests.
>>
>> After the v2.10 upgrade (from v2.7) I have noticed several cases of  
>> Exceptions being thrown and being registered as such in the Build  
>> status information. One such case, missing handling of GeneratorExit,  
>> has been fixed.
>>
>> But today a lot of these test runs started failing (next on todo list  
>> is finding out why), and was reported terminated due to an exception.
>>
>> The exception turns out to be due to a StopIteration from  
>> LineConsumerLogObserver.headerLineReceived (at least in the case I  
>> investigated). My guess is that the "problem line" is one of the two  
>> last lines in the log, which reports the exit status of the task.
>>
>> The log handler function is written like the logConsumer function in  
>> <http://docs.buildbot.net/current/developer/cls-logobserver.html#buildbot.process.logobserver.LineConsumerLogObserver>
>>
>> Based on what I've quickly skimmed about StopIteration, it should be  
>> handled by the caller of the logConsumer function. This suggests that  
>> LineConsumerLogObserver or its caller need to be updated.
>>
>>
>> BTW, this is not new in v2.10, this exception also happened in v2.7  
>> based on my grep of the log files. I guess the reason it is now showing  
>> up as an issue is that these exceptions are reported upwards to the  
>> step.
>>
>>
>
>


-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS