[devel at bb.net] deadlock upon errors on log.addRawLines

[Ion Alberdi]

Hello to all,
As the error might require more analysis, I copy/paste the issue mentioned
on irc.

>From my understanding,
when an error appears in the buildbot<->db conversation to add a log line:
https://github.com/buildbot/buildbot/blob/master/master/buildbot/process/log.py#L76

the step calling addStdout will not be able to finish, as it will wait
for the lock to be released, forever.

I see two solutions for now:
1. return the deferred in log.addStdout calls, and let the developper
    handle the issue if there is any. It has the drawback of changing
    the way steps are implemented:
    before:
        log.addStdout
        log.addStdout
    after:
        yield log.addStdout
        yield log.addStdout

2. implement a retry mechanism in addRawLines (with a random sleep between)
    and raise an error if the issue is not solved (that the developer will
handle or not).
    This aims at reducing the number of discarded logs so that they could
become tolerable.

gracinet (correct me if i'm wrong) prefers 2, inputs are welcome :)


P.S:
I'm currently testing solution 2 in a use case that stresses the logs long
enough
to make the db (mysql innodb) report the following issues:

- sqlalchemy.exc.OperationalError: (OperationalError) (1213, 'Deadlock
found when trying to get lock; try restarting transaction') None None
- sqlalchemy.exc.OperationalError: (OperationalError) (1213, 'Deadlock
found when trying to get lock; try restarting transaction') 'INSERT INTO
logchunks (logid, first_line, last_line, content, compressed) VALUES (%s,
%s, %s, %s, %s)' (924L, 315L, 315L,
'x\xdaKU\x80\x02\xbf|\x85\x92\xcc\xdcT\x85\xaa\xfc\xbcT\x85\xcc\xbc\xb4|
\xa1P\x92\x91Y\xac\x90\x92X\x92\xaa\xa0ad`h\xa6k`\xa2kd\xa1```\x05F\x9a:\n\xc5\xa9%%\x99y\xe9\n%\xf9\n\xa1!\xce\x00BM\x15\x96',
1)



-- 
Ion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/devel/attachments/20160428/fc555453/attachment.html>