[Buildbot-devel] Buildbot-devel Digest, Vol 8, Issue 2

Thu Dec 14 23:55:06 UTC 2006

Reply to 4. Multiple builds on the same host (Ben Hearsum)

I noticed this same problem, which ever slave connects first is the one 
that all the builders attempt to use. Try using slave locks to observe 
the behaviour. This also causes a similar problem in that it uses the 
same slave as prior builds even though one is available. With respect to 
slave lock usage, I think the builders need to check which lock is 
available for it, grab it, and start the build. Instead I think it 
attempts to grab a slave, if the slave is busy and no free lock, it 
waits in it's queue.

Manny

buildbot-devel-request at lists.sourceforge.net wrote:
> Send Buildbot-devel mailing list submissions to
> 	buildbot-devel at lists.sourceforge.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> or, via email, send a message with subject or body 'help' to
> 	buildbot-devel-request at lists.sourceforge.net
>
> You can reach the person managing the list at
> 	buildbot-devel-owner at lists.sourceforge.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Buildbot-devel digest..."
>
>
> Today's Topics:
>
>    1. Re: update on the next release (and a suggestion) (Brian Warner)
>    2. Re: update on the next release (and a suggestion) (Niklaus Giger)
>    3. Bonsai Filter by E-Mail patch (Ben Hearsum)
>    4. Multiple builds on the same host (Ben Hearsum)
>    5. buildbot-0.7.5 released (Brian Warner)
>    6. Re: maximiz (Isidoro Chadburn)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 06 Dec 2006 23:58:53 -0800 (PST)
> From: Brian Warner <warner-buildbot at lothar.com>
> Subject: Re: [Buildbot-devel] update on the next release (and a
> 	suggestion)
> To: niklaus.giger at member.fsf.org
> Cc: buildbot-devel at lists.sourceforge.net
> Message-ID: <20061206.235853.59483782.warner at lothar.com>
> Content-Type: Text/Plain; charset=us-ascii
>
>   
>> I am very interested in a few of the new features (e.g. SVNPoller,
>> FileUp/Download and have tested them in my test environment. Also
>> the trial tests pass without any errors on my GNU/Debian machine.
>>
>> Therefore I would like to ask if you could give us a hint when the next
>> version will be out.
>>     
>
> Very soon now. The reconfiguring-builders fix was the last big issue, which I
> finally got done about a week ago. The only thing left on my list is to clean
> up file upload/download a little bit (specifically adding some helpful error
> messages if the slave is too old or if the target file cannot be found). My
> current goal is to cut the release this coming weekend.
>
>   
>> Another question is, that I would have found it nice to specify the
>> attributes for a downloaded file, as I would like to download some test
>> scripts which have to be executed on the slave.
>>     
>
> So something like FileDownload(source, dest, mode=0755) ? Sure, I'd accept a
> patch for that, after 0.7.5 is out. Don't write it just yet, as I'll probably
> be changing FileDownload a little bit before the release, but as soon as
> 0.7.5 is cut feel free to submit one.
>
>   
>> And I would prefer to specify in a Shell step "./my_bash.sh", instead of [
>> "/bin/bash", "my_bash.sh"].
>>     
>
> If my_bash.sh has the executable bit set, then command=["./my_bash.sh"] ought
> to work (assuming it has a #!/bin/bash line, of course).
>
>   
>> Would you accept patches for this? Or do you consider it a too big security
>> risk?
>>     
>
>
> I guess I should settle on how ShellCommands should be specified. I've
> certainly had cases myself where I want to use shell metacharacters, things
> like "scp ../*.deb host:~/public_html/" and similar. My hunch is that things
> like that belong in a Makefile rather than in the Buildbot, and making it
> difficult or annoying to use the full power of bash within ShellCommands is a
> sort of passive/aggressive way to accomplish that :).
>
> There's not necessarily a security risk in string-based commands. The risk
> comes when you are interpolating other strings into them, specifically
> strings that come from a hostile party, where you get injection attacks and
> games with passing shell metacharacters in through a string that was
> "supposed" to be something simple. If you can only use argv array-based
> ShellCommands, these threats are a lot less severe. But there aren't a whole
> lot of places where we interpolate things into a string-based command
> anyways, unless you're using WithProperties a lot.
>
>
> Would it make sense for this test script that you want to download and
> execute on the slave instead be checked in to your source tree? I'd like to
> understand the use case a bit more before coming to a decision about whether
> to encourage or discourage string-based commands.
>
> [note: by "string-base commands" I mean something like
> ShellCommand(command="ls *.foo >stuff.txt") . When you pass a non-array to
> command=, the buildslave pretends you had written command=["/bin/sh", "-c",
> "ls *.foo >stuff.txt"] instead.]
>
> cheers,
>  -Brian
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 7 Dec 2006 23:11:54 +0100
> From: Niklaus Giger <niklaus.giger at member.fsf.org>
> Subject: Re: [Buildbot-devel] update on the next release (and a
> 	suggestion)
> To: Brian Warner <warner-buildbot at lothar.com>
> Cc: buildbot-devel at lists.sourceforge.net
> Message-ID: <200612072311.55254.niklaus.giger at member.fsf.org>
> Content-Type: text/plain;  charset="iso-8859-1"
>
> Am Donnerstag, 7. Dezember 2006 08:58 schrieb Brian Warner:
> <...>
>   
>> So something like FileDownload(source, dest, mode=0755) ? Sure, I'd accept
>> a patch for that, after 0.7.5 is out. Don't write it just yet, as I'll
>> probably be changing FileDownload a little bit before the release, but as
>> soon as 0.7.5 is cut feel free to submit one.
>>     
> Yes, that is what I am looking for. And I have time to wait.
>
>   
>>> And I would prefer to specify in a Shell step "./my_bash.sh", instead of
>>> [ "/bin/bash", "my_bash.sh"].
>>>       
>> If my_bash.sh has the executable bit set, then command=["./my_bash.sh"]
>> ought to work (assuming it has a #!/bin/bash line, of course).
>>     
> Yes, that why I want to be able to specify the mode. The example above is just 
> my current work around.
> <..>
>   
>> Would it make sense for this test script that you want to download and
>> execute on the slave instead be checked in to your source tree? I'd like to
>> understand the use case a bit more before coming to a decision about
>> whether to encourage or discourage string-based commands.
>>     
> I do not have a need for string based commands. I just want to be able to 
> download scripts and configuration. If you want to study my setup have a look 
> at the (a little bit outdated) 
> http://ngiger.dyndns.org/xenomai-data/xeno_buildbot.pdf
> The buildbot itself may be reached at http://ngiger.dyndns.org:8010/
>
> The FileUp/Download commands are all what I need to be able to keep (backup in 
> Subversion repository) all my test scripts, kernel/.config etc in one place 
> (e.g. the master) and not distributed on several slaves. I have no write 
> privilege (never asked for it neither) in the main Xenomai (realtime 
> extension to the Linux kernel) repository.
>
> Also I like to have a separate test installation for my buildbot. And with 
> scripts/configs scattered over test/stable and different master/slaves I 
> sometimes just got a little bit lost, which config file /scripts was already 
> updated and which not.
>
> Realtime kernel often have very specific (non public) HW, e.g. I am working in 
> a industrial manufacturer for plastic injection machines, where machines 
> costing (100k - 1M US$) are controlled by custom PPC405 based controller 
> card. Upgrade cycles are quite long and we prefer to stick with known good 
> release (unfortunately still a proprietary OS) for years.
>
> The embedded people like to tweak their environment, as each project has 
> different needs (e.g. no harddisk, only 4 MB of memory)
> The whole setup till you can really test an embedded board involves a lot,
> e.g. 
> LinuxFrom Scratch (or a Debian debootstrag or ...)
> Linux kernel source +
> Xenomai patches +
> probably board specific patches
> a kernel config
> Xenomai application code
> optionally RTnet (another project using Xenomai for RealTime Networking), 
> pulled in from another Subversion repository
> Bash or BusyBox
> a RootFileSystem
> Ethernet and/or serial connection to the board
> cross compilation (for various architectures)
> Remotely actuated PowerOn switch
>
> It took me quite a few weeks to get the buildbot started. Keeping it up to 
> date with all the changes + trying to improve it + trying to cover more 
> corner cases (unit test, system test) is quite time consuming, but helps to 
> catch a few (but far from all) errors.
>
> In short a lot of things which just were better not integrated in the main 
> trunk. And till now my whole buildbot setup is not yet mature enough to be 
> shared. Once I get more people interested into running their own slave and we 
> are able to come up with common scripts to cover the different variants I 
> have no objection into getting my stuff into the Xenomai code base. 
>
> I think that the buildbot I am running is quite complex as it involves 
> (remote) system testing. But I found no better tool, and Buildbot served me 
> fine and I really appreciated how you integrated a lot of suggestions from 
> the mailing list. And someday I will attack also the problem how to trigger a 
> build when one of several code sources are needed for a single project. I 
> think in this respect CruiseControl has some advantages and a colleague of 
> mine is using successfully also. But it has neither distributed slaves nor 
> a "try" feature. The last two features made me choose buildbot.
>
> Best regards
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20061214/53f90342/attachment.html>