[Buildbot-devel] Buildbot-devel Digest, Vol 8, Issue 2

Fri Dec 15 00:12:31 UTC 2006

Yep. I had the same behaviour. I had one build from one builder going on
'linux1' and I tried to start a build on a different builder. It got queued on
'linux1' instead of starting on 'linux2'!

Minesh Patel wrote:
> Reply to 4. Multiple builds on the same host (Ben Hearsum)
> 
> I noticed this same problem, which ever slave connects first is the one
> that all the builders attempt to use. Try using slave locks to observe
> the behaviour.  This also causes a similar problem in that it uses the
> same slave as prior builds even though one is available. With respect to
> slave lock usage, I think the builders need to check which lock is
> available for it, grab it, and start the build. Instead I think it
> attempts to grab a slave, if the slave is busy and no free lock, it
> waits in it's queue.
> 
> Manny
> 
> 
> buildbot-devel-request at lists.sourceforge.net wrote:
>> Send Buildbot-devel mailing list submissions to
>> 	buildbot-devel at lists.sourceforge.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> or, via email, send a message with subject or body 'help' to
>> 	buildbot-devel-request at lists.sourceforge.net
>>
>> You can reach the person managing the list at
>> 	buildbot-devel-owner at lists.sourceforge.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Buildbot-devel digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: update on the next release (and a suggestion) (Brian Warner)
>>    2. Re: update on the next release (and a suggestion) (Niklaus Giger)
>>    3. Bonsai Filter by E-Mail patch (Ben Hearsum)
>>    4. Multiple builds on the same host (Ben Hearsum)
>>    5. buildbot-0.7.5 released (Brian Warner)
>>    6. Re: maximiz (Isidoro Chadburn)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 06 Dec 2006 23:58:53 -0800 (PST)
>> From: Brian Warner <warner-buildbot at lothar.com>
>> Subject: Re: [Buildbot-devel] update on the next release (and a
>> 	suggestion)
>> To: niklaus.giger at member.fsf.org
>> Cc: buildbot-devel at lists.sourceforge.net
>> Message-ID: <20061206.235853.59483782.warner at lothar.com>
>> Content-Type: Text/Plain; charset=us-ascii
>>
>>   
>>> I am very interested in a few of the new features (e.g. SVNPoller,
>>> FileUp/Download and have tested them in my test environment. Also
>>> the trial tests pass without any errors on my GNU/Debian machine.
>>>
>>> Therefore I would like to ask if you could give us a hint when the next
>>> version will be out.
>>>     
>>
>> Very soon now. The reconfiguring-builders fix was the last big issue, which I
>> finally got done about a week ago. The only thing left on my list is to clean
>> up file upload/download a little bit (specifically adding some helpful error
>> messages if the slave is too old or if the target file cannot be found). My
>> current goal is to cut the release this coming weekend.
>>
>>   
>>> Another question is, that I would have found it nice to specify the
>>> attributes for a downloaded file, as I would like to download some test
>>> scripts which have to be executed on the slave.
>>>     
>>
>> So something like FileDownload(source, dest, mode=0755) ? Sure, I'd accept a
>> patch for that, after 0.7.5 is out. Don't write it just yet, as I'll probably
>> be changing FileDownload a little bit before the release, but as soon as
>> 0.7.5 is cut feel free to submit one.
>>
>>   
>>> And I would prefer to specify in a Shell step "./my_bash.sh", instead of [
>>> "/bin/bash", "my_bash.sh"].
>>>     
>>
>> If my_bash.sh has the executable bit set, then command=["./my_bash.sh"] ought
>> to work (assuming it has a #!/bin/bash line, of course).
>>
>>   
>>> Would you accept patches for this? Or do you consider it a too big security
>>> risk?
>>>     
>>
>>
>> I guess I should settle on how ShellCommands should be specified. I've
>> certainly had cases myself where I want to use shell metacharacters, things
>> like "scp ../*.deb host:~/public_html/" and similar. My hunch is that things
>> like that belong in a Makefile rather than in the Buildbot, and making it
>> difficult or annoying to use the full power of bash within ShellCommands is a
>> sort of passive/aggressive way to accomplish that :).
>>
>> There's not necessarily a security risk in string-based commands. The risk
>> comes when you are interpolating other strings into them, specifically
>> strings that come from a hostile party, where you get injection attacks and
>> games with passing shell metacharacters in through a string that was
>> "supposed" to be something simple. If you can only use argv array-based
>> ShellCommands, these threats are a lot less severe. But there aren't a whole
>> lot of places where we interpolate things into a string-based command
>> anyways, unless you're using WithProperties a lot.
>>
>>
>> Would it make sense for this test script that you want to download and
>> execute on the slave instead be checked in to your source tree? I'd like to
>> understand the use case a bit more before coming to a decision about whether
>> to encourage or discourage string-based commands.
>>
>> [note: by "string-base commands" I mean something like
>> ShellCommand(command="ls *.foo >stuff.txt") . When you pass a non-array to
>> command=, the buildslave pretends you had written command=["/bin/sh", "-c",
>> "ls *.foo >stuff.txt"] instead.]
>>
>> cheers,
>>  -Brian
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 7 Dec 2006 23:11:54 +0100
>> From: Niklaus Giger <niklaus.giger at member.fsf.org>
>> Subject: Re: [Buildbot-devel] update on the next release (and a
>> 	suggestion)
>> To: Brian Warner <warner-buildbot at lothar.com>
>> Cc: buildbot-devel at lists.sourceforge.net
>> Message-ID: <200612072311.55254.niklaus.giger at member.fsf.org>
>> Content-Type: text/plain;  charset="iso-8859-1"
>>
>> Am Donnerstag, 7. Dezember 2006 08:58 schrieb Brian Warner:
>> <...>
>>   
>>> So something like FileDownload(source, dest, mode=0755) ? Sure, I'd accept
>>> a patch for that, after 0.7.5 is out. Don't write it just yet, as I'll
>>> probably be changing FileDownload a little bit before the release, but as
>>> soon as 0.7.5 is cut feel free to submit one.
>>>     
>> Yes, that is what I am looking for. And I have time to wait.
>>
>>   
>>>> And I would prefer to specify in a Shell step "./my_bash.sh", instead of
>>>> [ "/bin/bash", "my_bash.sh"].
>>>>       
>>> If my_bash.sh has the executable bit set, then command=["./my_bash.sh"]
>>> ought to work (assuming it has a #!/bin/bash line, of course).
>>>     
>> Yes, that why I want to be able to specify the mode. The example above is just 
>> my current work around.
>> <..>
>>   
>>> Would it make sense for this test script that you want to download and
>>> execute on the slave instead be checked in to your source tree? I'd like to
>>> understand the use case a bit more before coming to a decision about
>>> whether to encourage or discourage string-based commands.
>>>     
>> I do not have a need for string based commands. I just want to be able to 
>> download scripts and configuration. If you want to study my setup have a look 
>> at the (a little bit outdated) 
>> http://ngiger.dyndns.org/xenomai-data/xeno_buildbot.pdf
>> The buildbot itself may be reached at http://ngiger.dyndns.org:8010/
>>
>> The FileUp/Download commands are all what I need to be able to keep (backup in 
>> Subversion repository) all my test scripts, kernel/.config etc in one place 
>> (e.g. the master) and not distributed on several slaves. I have no write 
>> privilege (never asked for it neither) in the main Xenomai (realtime 
>> extension to the Linux kernel) repository.
>>
>> Also I like to have a separate test installation for my buildbot. And with 
>> scripts/configs scattered over test/stable and different master/slaves I 
>> sometimes just got a little bit lost, which config file /scripts was already 
>> updated and which not.
>>
>> Realtime kernel often have very specific (non public) HW, e.g. I am working in 
>> a industrial manufacturer for plastic injection machines, where machines 
>> costing (100k - 1M US$) are controlled by custom PPC405 based controller 
>> card. Upgrade cycles are quite long and we prefer to stick with known good 
>> release (unfortunately still a proprietary OS) for years.
>>
>> The embedded people like to tweak their environment, as each project has 
>> different needs (e.g. no harddisk, only 4 MB of memory)
>> The whole setup till you can really test an embedded board involves a lot,
>> e.g. 
>> LinuxFrom Scratch (or a Debian debootstrag or ...)
>> Linux kernel source +
>> Xenomai patches +
>> probably board specific patches
>> a kernel config
>> Xenomai application code
>> optionally RTnet (another project using Xenomai for RealTime Networking), 
>> pulled in from another Subversion repository
>> Bash or BusyBox
>> a RootFileSystem
>> Ethernet and/or serial connection to the board
>> cross compilation (for various architectures)
>> Remotely actuated PowerOn switch
>>
>> It took me quite a few weeks to get the buildbot started. Keeping it up to 
>> date with all the changes + trying to improve it + trying to cover more 
>> corner cases (unit test, system test) is quite time consuming, but helps to 
>> catch a few (but far from all) errors.
>>
>> In short a lot of things which just were better not integrated in the main 
>> trunk. And till now my whole buildbot setup is not yet mature enough to be 
>> shared. Once I get more people interested into running their own slave and we 
>> are able to come up with common scripts to cover the different variants I 
>> have no objection into getting my stuff into the Xenomai code base. 
>>
>> I think that the buildbot I am running is quite complex as it involves 
>> (remote) system testing. But I found no better tool, and Buildbot served me 
>> fine and I really appreciated how you integrated a lot of suggestions from 
>> the mailing list. And someday I will attack also the problem how to trigger a 
>> build when one of several code sources are needed for a single project. I 
>> think in this respect CruiseControl has some advantages and a colleague of 
>> mine is using successfully also. But it has neither distributed slaves nor 
>> a "try" feature. The last two features made me choose buildbot.
>>
>> Best regards
>>
>>   
> 
> !DSPAM:35,4581e4a4491186964617024!
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> 
> !DSPAM:35,4581e4a4491186964617024!
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> 
> 
> !DSPAM:35,4581e4a4491186964617024!