[Buildbot-commits] [Buildbot] #845: shutil.copytree timeout for mode=copy

[Buildbot]

#845: shutil.copytree timeout for mode=copy
--------------------+--------------------
Reporter:  phil     |       Owner:
    Type:  defect   |      Status:  new
Priority:  major    |   Milestone:  0.8.+
 Version:  0.7.12   |  Resolution:
Keywords:  windows  |
--------------------+--------------------

Comment (by glaz):

 We have the same problem with mode=copy on our WindowsXP buildbot 0.8.1
 and the reason is shutil.copytree() that freezes buildslave until copying
 will be finished. There is a massage that buildslave sends to buildmaster
 that says:
 "Since we're on a non-POSIX platform, we're not going to try to execute cp
 in a subprocess, but instead use shutil.copytree(), which will block until
 it is complete."

 And buildslave is really blocked from this moment... If data size is
 enough large to block buildslave for at least 'keepalive' seconds, you
 will get the same error with any data source that inherits doCopy() method
 from SourceBase class.

 Buildmaster = BM

 Buildslave = BS

 In fact:
 1. BM sends task to buildslave to update source with mode=copy;
 2. BS receives task and starts downloading data from some server (BS sends
 keepalive packets on-time).
 3. BS recieved all data from some server and starts copying. (Note if it
 is Windows platform, shutil.copy will be used in this step).
 4. (on BS) shutil.copy will lock buildslave unlit copying will be
 finished.
 5. (on BS) shutil.copy finished copying but keepalive+ seconds left from
 last keepalive answer received.
 6. BS(in fact - twisted reactor) detects that keepalive packet should be
 sent and sends it.
 7. BS(in fact - twisted reactor) immedeatly after keepalive packet was
 send detects that no answer was reciewed and breaks process.
 8. BM answers for keepalive packet but BS already decided connection is
 lost so this answer does not help us.

 So that problem is running shutil.copy in buildslave's process. It is
 supposed to be launched in subprocess to avoid such problems.

 So simpliest way for now is to use mode=update (which will be launched as
 subprocess) and then use some 'manual' copying like this:
 create on slave machine python script that will recursively copy folder
 using shutil.copy and then call
 that script via 'ShellCommand'(that will also be launched as subprocess).

-- 
Ticket URL: <http://trac.buildbot.net/ticket/845#comment:8>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation