[Buildbot] #1061: Buildbot locks downloaded files

[Buildbot trac]

#1061: Buildbot locks downloaded files
------------------------------+--------------------
Reporter:  parshin            |       Owner:
    Type:  defect             |      Status:  new
Priority:  major              |   Milestone:  0.9.+
 Version:  0.8.2              |  Resolution:
Keywords:  windows, transfer  |
------------------------------+--------------------

Old description:

> OS: Windows XP SP2.
>
> Steps to reproduce bug:
>
>  1. Create two builds both with the following build steps:
>   a. Download file from the master (!FileDownload)
>   b. Run shell command, which uses this file
>  2. Run both builds simultaneously at the same slave.
>
> In my case downloaded file was an executable, shell command was batch
> file which simply starts the executable.
>
> With rather high probability (~50%) one of builds fails because of the
> executable was locked by another process.
>
> ----------
>
> Looks like the reason is the following. When files are downloaded at
> slave, slave process has two open handles. Next, if one build has
> finished download first, it starts separate process for shell command,
> which inherits all handles of the parent process (slave) including open
> handle for other downloading file (it is default behavior of Twisted
> spawnProcess() ). As a result, downloaded file is locked even after
> second build finishes download and close its handle.
>
> ----------
>
> I've tried to fix this problem by preventing handle inheritance by
> setting HANDLE_FLAG_INHERIT flag of handle to zero. Attached patch fixes
> that bug for me. Maybe, better solution is to download to temporary file
> and rename it after download.

New description:

 OS: Windows XP SP2.

 Steps to reproduce bug:

  1. Create two builds both with the following build steps:
   a. Download file from the master (!FileDownload)
   b. Run shell command, which uses this file
  2. Run both builds simultaneously at the same slave.

 In my case downloaded file was an executable, shell command was batch file
 which simply starts the executable.

 With rather high probability (~50%) one of builds fails because of the
 executable was locked by another process.

 ----------

 Looks like the reason is the following. When files are downloaded at
 slave, slave process has two open handles. Next, if one build has finished
 download first, it starts separate process for shell command, which
 inherits all handles of the parent process (slave) including open handle
 for other downloading file (it is default behavior of Twisted
 spawnProcess() ). As a result, downloaded file is locked even after second
 build finishes download and close its handle.

 ----------

 I've tried to fix this problem by preventing handle inheritance by setting
 HANDLE_FLAG_INHERIT flag of handle to zero. Attached patch fixes that bug
 for me. Maybe, better solution is to download to temporary file and rename
 it after download.

--

Comment (by sa2ajj):

 Jason, what version of Buildbot did you patch?

--
Ticket URL: <http://trac.buildbot.net/ticket/1061#comment:10>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation