[Buildbot-devel] Windows Slave Hanging

[Alan Jones]

Hi,

I am having an issue with a buildbot slave running under windows. As best I
can tell, the slave is receiving a command from the master, executing the
command, getting the result, but never sending the result code back to the
master.

Interestingly, eventually it will finally figure out something is wrong much
time later (at least 1200 s), causing the command that was running to
actually complete successfully, then the command folowing it to throw an
exception "twisted.internet.error.ConnectionLost".

I am running 0.7.5.
My master is a gentoo box with twistd 2.4.0, python 2.3.5.
The slave is windows 2000, twistd 2.4.0, python 2.4.3. Running as a service.


My slave logs show something like this:

2007/01/09 08:35 Mountain Standard Time [Broker,client]
ShellCommand._startCommand
2007/01/09 08:35 Mountain Standard Time [Broker,client]
C:\WINNT\system32\cmd.exe /c p4 ...args...
2007/01/09 08:35 Mountain Standard Time [Broker,client]   in dir
c:\buildbot\branchname (timeout 1200 secs)
2007/01/09 08:35 Mountain Standard Time [Broker,client]   watching logfiles
{}
2007/01/09 08:35 Mountain Standard Time [Broker,client]   --- cut ---
2007/01/09 08:35 Mountain Standard Time [-] sending app-level keepalive
2007/01/09 08:35 Mountain Standard Time [-] command finished with signal
None, exit code 0
2007/01/09 09:17 Mountain Standard Time [-] SlaveBuilder.commandComplete
<buildbot.slave.commands.P4 instance at 0x012DF710>
2007/01/09 09:17 Mountain Standard Time [-] sending app-level keepalive
2007/01/09 09:17 Mountain Standard Time [-] BotFactory.checkActivity:
nothing from master for 2503 secs
2007/01/09 09:17 Mountain Standard Time [Broker,client]
SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2007/01/09 09:17 Mountain Standard Time [Broker,client]
SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2007/01/09 09:17 Mountain Standard Time [Broker,client]
SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2007/01/09 09:17 Mountain Standard Time [Broker,client]
SlaveBuilder._ackFailed: sendComplete
2007/01/09 09:17 Mountain Standard Time [Broker,client]
BotFactory.keepaliveLost
2007/01/09 09:17 Mountain Standard Time [Broker,client] lost remote
2007/01/09 09:17 Mountain Standard Time [Broker,client]
<twisted.internet.tcp.Connector instance at 0x008DA440> will retry in 2
seconds
2007/01/09 09:17 Mountain Standard Time [Broker,client] Stopping factory
<buildbot.slave.bot.BotFactory instance at 0x010043F0>
2007/01/09 09:17 Mountain Standard Time [-] Starting factory
<buildbot.slave.bot.BotFactory instance at 0x010043F0>
2007/01/09 09:17 Mountain Standard Time [Broker,client] message from master:
attached



And the master is throwing this:

2007/01/09 08:31 MST [Broker,11,192.168.64.30] ping finished: success
2007/01/09 08:31 MST [Broker,11,192.168.64.30] <Build branchname>.startBuild
2007/01/09 08:31 MST [Broker,11,192.168.64.30] acquireLocks(step <Build
branchname>, locks [])
2007/01/09 08:31 MST [Broker,11,192.168.64.30] acquireLocks(step
<buildbot.steps.source.P4 instance at 0xb75ce2cc>, locks [])
2007/01/09 08:31 MST [Broker,11,192.168.64.30]
ShellCommand.startCommand(cmd=%s) (<RemoteCommand 'p4' at -1218991892>,)
2007/01/09 08:31 MST [Broker,11,192.168.64.30] <RemoteCommand 'p4' at
-1218991892>: RemoteCommand.run [17]
2007/01/09 08:31 MST [Broker,11,192.168.64.30] LoggedRemoteCommand.start
2007/01/09 09:17 MST [Broker,11,192.168.64.30] <RemoteCommand 'p4' at
-1218991892> rc=0
2007/01/09 09:17 MST [-] closing log <buildbot.status.builder.LogFile
instance at 0xb757a88c>
2007/01/09 09:17 MST [-] releaseLocks(<buildbot.steps.source.P4 instance at
0xb75ce2cc>): []
2007/01/09 09:17 MST [-]  step 'p4' complete: success
2007/01/09 09:17 MST [-] acquireLocks(step
<buildbot.steps.shell.ShellCommand instance at 0xb75ceeac>, locks [])
2007/01/09 09:17 MST [-] ShellCommand.startCommand(cmd=%s)
(<RemoteShellCommand '['bash', '-c', './vxmake TGT=v2 mk']'>,)
2007/01/09 09:17 MST [-] <RemoteShellCommand '['bash', '-c', './vxmake
TGT=v2 mk']'>: RemoteCommand.run [18]
2007/01/09 09:17 MST [-] command '['bash', '-c', './vxmake TGT=v2 mk']' in
dir 'build'
2007/01/09 09:17 MST [-] LoggedRemoteCommand.start
2007/01/09 09:17 MST [Broker,11,192.168.64.30] BuildStep.failed, traceback
follows
2007/01/09 09:17 MST [Broker,11,192.168.64.30] Traceback (most recent call
last):
        Failure: twisted.spread.pb.PBConnectionLost: [Failure instance:
Traceback (failure with no frames): twisted.internet.error.Connecti
onLost: Connection to the other side was lost in a non-clean fashion.
        ]


Any help is appriciated.

-Alan