[Buildbot-devel] FTP step

[Didier Trosset]

A.T.Hofkamp wrote:
> Didier Trosset wrote:
>> A.T.Hofkamp wrote:
>>> - Use FTP implementation from Twisted, you don't want to block the 
>>> whole bb slave application while downloading an ISO DVD image from a
>>>  slow server.
>> I didn't know about it. Will probably use it.
> It may be less optional than you think.

Yes, this is necessary.

> ---- 8< ----

> A reason for not hard connecting 'tar x' to the ftp step is the answer to
>  the question:
> 
> How would I use your step to ftp an 'iso.gz', 'archive.zip', or 
> 'archive.rar'? (or any other non-tar archive for that matter)

By using an FTP followed by another custom command.

> Last but not least, given that you are about to spend a lot of effort for
>  making a proper 'tar x' step, some users may want to use that for 
> untarring an archive obtained by by other means than an ftp step. If you
>  split ftp and untarring, these users would also be happy.

This is something I want to avoid: creating a too much proper Uncompress
step that can handle tar (gzip, bzip2 or plain), and zip, and rar, and ...
Far too wuch work for little gain I think. (And I cannot afford that.) Each
bb user can use the command he wants.

> (ie the answer to the question 'how do I use your untar without needing 
> to do ftp first'?)

This a lot easier, just use a ShellCommand ;-)

My goal here is really to make a source step, providing a tree ready to be
build. I did FtpTar first without implementing a new source step. It
required multiple (4) ShellCommand steps: clear build directory, then create
it, then FTP the file, then untar it. It looked ugly, both in master.cfg
configuration, and in the waterfall display.

But the point of getting other types of archives is pertinent. One solution
could be to have the FtpTar step being more customizable on the tar side,
using ShellCommand like arguments to do any un-compression the user wants.

Maybe this FtpTar step could be renamed FtpArchive, or FtpSource step. It
would download a file from an FTP server, plus run a custom ShellCommand to
uncompress the (supposedly compressed) downloaded file.

Then why don't make the FTP step separate? Because a Source step make one
thing very well but it does not show it: it manages the build directory. My
hand made FtpTar steps with succussive ShellCommands shows me that managing
this build directory looks like the key to this FtpTar step.

Didier

-- 
Didier Trosset-Moreau
SND / Data Converters
Agilent Technologies
Geneva, Switzerland