[Buildbot-devel] buildbot for testers

[Kregg Kemper]

James wrote:
Wouldn't this be easier if the tests were simply included in the same
branch as the code to which they apply?

  sometimes the testers have their own management tool to keep all the test 
scripts and what have not. For instance, I use HP's Quality Center (QC) that 
stores and manages all test scripts and test information in a separate 
repository from the code. I don't know the numbers, but a lot of test groups 
may store all there test automation outside of the code branch, so I would 
like buildbot to be able to interface with the testers environment and run 
and the desired tests and get the test result information. I'll start with 
adding support for QTP/QC since that's what I use.

James Wrote:
> a test plan determines what resource the tests will run on and which tests
> will be ran.
>
> resource could be a computer or a VM that on a server that shares limited
> resources.

Buildbot seems to do this reasonably well.

    What I don't think builbot does is start and stop VMs that are needed to 
run tests on and also manage the limited resources on a VM server. I'll give 
an example from my experience what this means. A test groups might be 
testing many code baselines within a company but have limited VM server(s) 
that can only handle a handful of images running at one time. Testers for 
these different baselines might have 5+ VMs that launch for each build they 
support. When a lot of builds come in (Friday evening) the testbot must 
store the tests on a queue and launch tests as a slot is available on the 
server. I use VMWare for virtualization,  but these capabilities should work 
on other technologies like Xen or Virtual PC.
  This is the kind of stuff I do and would like to ad the capability to 
build bot.


----- Original Message ----- 
From: "James Youngman" <jay at gnu.org>
To: "Kregg Kemper" <kreggk at comcast.net>
Cc: "buildbot-devel list" <buildbot-devel at lists.sourceforge.net>
Sent: Saturday, May 23, 2009 12:45 PM
Subject: Re: [Buildbot-devel] buildbot for testers


On Fri, May 22, 2009 at 6:24 AM, Kregg Kemper <kreggk at comcast.net> wrote:
> Hello,
>
> I would like to start getting involved with buildbot and do some
> development that might expand the capabilities to be usable by a test 
> group
> for running regression testing. So far have created a fork, read some
> documentation, and browsed the source code. That's about it. Not sure what
> to do next.
>
> I have developed a system that does some similar stuff as buildbot and
> would like to merge some of my ideas into buildbot. The one oh wish list
> mentioned something like a test bot that handled running automated tests
> that could come from a separate code base. This is what I would like to
> expand on and start developing. This bot would be used to run black box
> regression testing created by the test team in their environment. Some of
> the stuff I want to do:
>
> testbot receives notification that a build is complete and installer is
> ready. The bot see if it matches a pre-defined test plan for that
> build/baseline. The valid test plans are run.

Wouldn't this be easier if the tests were simply included in the same
branch as the code to which they apply?


> a test plan determines what resource the tests will run on and which tests
> will be ran.
>
> resource could be a computer or a VM that on a server that shares limited
> resources.

Buildbot seems to do this reasonably well.

> This is the stuff I do at work and want to incorporate into build bot,
> but I have no idea how interested other people or in this stuff. Any
> feedback on this would be cool.

My main interest is in getting Buildbot to buid and test a tarball, as
opposed to needing to check the source out in the slave.   I'm working
on it, but not devoting much time to it.

James.