[Buildbot-devel] FTP step (was: upcoming release plan: freeze 9/5, release 9/12)

[A.T.Hofkamp]

Didier Trosset wrote:
> A.T.Hofkamp wrote:
>> - Use FTP implementation from Twisted, you don't want to block the 
>> whole bb slave application while downloading an ISO DVD image from a 
>> slow server.
> 
> I didn't know about it. Will probably use it.

It may be less optional than you think.

The whole idea of Twisted is that a slave itself never does any work directly, 
it only supervises the work. Actual processing is done outside the bb 
application (eg by a child process) managed by Twisted usually. That is how 
concurrency between slaves in the same bb application is handled.

When you use a standard library to do actual work in a slave rather than just 
supervising it, you break the concurrency of the application.

>> - Create a way to avoid use of hard-coded tar command (ie maybe I want to
>>  use a non-standard tar, different options (gzip or bzip2 
>> decompression?), my tar doesn't know --strip, etc
> 
> Yes. I've already thought using the python tarfile library. I guess I will
> have to, in the end. Does Windows have the tar command, BTW? But this 
> brings
> a lot of manual handling because the tarfile library does not have the
> --strip option. So the extraction must be done by hand, the correct way, I
> mean avoiding the pitfals of directory access rights.

Use of tarfile lib has the same disadvantage as use of ftplib above, you break 
concurrency of the application.

If you want to do 'tar' yourself it should be done in Twisted, imho to prevent 
blocking of other slaves. Don't know whether Twisted currently has support for 
tar processing.



A reason for not hard connecting 'tar x' to the ftp step is the answer to the 
question:

How would I use your step to ftp an 'iso.gz', 'archive.zip', or 'archive.rar'? 
(or any other non-tar archive for that matter)



Last but not least, given that you are about to spend a lot of effort for 
making a proper 'tar x' step, some users may want to use that for untarring an 
archive obtained by by other means than an ftp step. If you split ftp and 
untarring, these users would also be happy.

(ie the answer to the question 'how do I use your untar without needing to do 
ftp first'?)

> 
> Does anyone know if the python tarfile library could be enhanced?

yes it can, Python is an open source project.
Add an issue to the tracker if it doesn't exist, and submit a patch.

>> - Allow for some paranoia checking, upper limit of downloaded file 
>> size (and since this step would usually be used to download 3rd-party 
>> code, I
>>  can imagine even an exact byte count)
> 
> Shouldn't be that difficult to implement.

great!

> Jean-Paul Calderone wrote:
>>
>> It would make more sense to have separate FTP and untar/gzip steps.
> 
> I wondered a long time on this one, yes. But in the end, I decided that I
> wanted this step to be a source step. Using this source step would 
> result in
> having a directory ready-to-be-compiled.

An ambitious goal, given the zillion different ways that people use to pack a 
set of files into a single archive, and the ease of inventing the 
(zillion+1)'th way.


Albert