[Buildbot-devel] retries being ignored when mode=copy for Git
sgarman at zenlinux.com
Wed Jun 9 00:17:58 UTC 2010
I'm regularly run into issues with the git servers I clone our sources
from. This results in the "fatal: The remote end hung up unexpectedly"
error that I'm sure you've all seen before.
I'm running Buildbot 0.8.0 and was pleased to discover the retry option,
which is intended to retry SCM checkouts when things like the above
happen. I'm using it as follows in my master.cfg:
timeout=10000, retry=(5, 3)))
...in order to retry the git clone up to three times with a 5-second
interval between retries.
However, in my build configurations that use mode="copy" (as above), the
retries are not attempted when a "remote end hung up unexpectedly" error
I've been looking through buildbot/slave/commands/vcs.py to understand
what's going on, and admit I'm a bit uncertain where the control flow is
happening due to the inheritance between the SourceBase and Git classes,
so bear with me.
It appears that the following is happening:
Git's _doFetch() is invoked, which sets up a git fetch command, that
gets passed along to Git's _dovccmd(). When the command fails, I get the
following in the buildslave's log:
2010-06-08 16:48:05-0700 [-] command finished with signal None, exit
code 128, elapsedTime: 0.189545
2010-06-08 16:48:05-0700 [-] _checkAbandoned [Failure instance:
Traceback: <class 'buildbot.slave.commands.base.AbandonChain'>: 128
--- <exception caught here> ---
It looks like within Git's _dovccmd(), the _abandonOnFailure callback
gets added, which I can comment out and get the retrying behavior to
work, but not until after a _didFetch() gets run with a missing revision
option (and so git will try to do a reset --hard FETCH_HEAD). That
behavior seems broken and so I'm reluctant to call that a workaround.
I'm hoping one of the more experienced developers can get to the bottom
of this issue more quickly given the above information.
Thanks for your time,
sgarman at zenlinux dot com
More information about the devel