[Buildbot-devel] [GSoC-14] New Idea: Dynamic scaling and instance metadata configuration for Latent slaves

[Dustin J. Mitchell]

I replied to the proposal in melange, but since I don't use latent
slaves with Buildbot, I'd be interested to hear feedback from users
who do.

Dustin

On Sun, Mar 16, 2014 at 6:59 AM, kuldeep porwal <2591kuldeep at gmail.com> wrote:
> Hello,
>               I am masters student from India. I have successfully completed
> GSoC-2013 with drizzle database. Here is the link of my work:
> http://www.google-melange.com/gsoc/project/google/gsoc2013/kuldeep2591/5997011295797248
>
> As I was involved with my thesis work I didn't pay attention to gsoc'14
> earlier but now as I am free I would like to contribute to buildbot , here
> is my work experience in continuous integration and its tools:
>
> 1. I have worked on jenkins as a part of my summer internship.
>     There I was supposed to design continuous integration (CI) for one of
> their project. And it involves:
>     1.1 merging build request: I have developed a protocol which merges
> multiple build requests for same ticket scheduled across multiple slaves
> . Buildbot already have this feature as mentioned in the
> documentation/system-architecture.
>
>     1.2  Workspace cleanup: Wrote my own cleanup plugin as desired for our
> CI.
>
>    1.3 Automatic launch of amazon aws slave:  On demand launching of slave
> helps in many ways:
>          i) saves money   ii) Load balancing
>         I have used amazon slave plugin of Jenkins to serve our purpose
> .They had bugs that time so I patched them specifically for the purpose as
> desired for  project that time.
>
> MyIdea :
>
> I have a short discussion with Dustin sir about it a day back. Here is the
> idea please suggest any changes :
>
> As buildbot already have a support for Amazon slaves and it starts and kills
> those slave as per demand (Latent Slaves).  But at the time of load spike if
> you want let say X # of slaves then you need to create X # BuildSlave
> instances and their bid price (if spot instances).
> ---> The change I would like to suggest is to make this Dynamic. BuildMaster
> will  first observe the load present based on user demand (can be considered
> as build request priority and total number of build request present at a
> time in schedulers). Then  based on that demand and load it automatically
> launches the Latent Slave.
> It will only launch the # slaves up till the maximum limit (user parameter ,
> let say 50).
>
> ---> The above launch of build slave will be governed by automatic
> configuration by instance metadata. As dustin sir suggested:
> Currently, the user must set up each instance image with a different
> Buildbot slave name and password, and configure the Buildbot slave to start
> on boot.  This could be made much easier for users, perhaps with a pre-built
> AMI that starts the Buildbot slave on boot and takes all of its
> configuration from the instance metadata.
>
> currently I am focusing on Amazon AWS but as the time permits I can look
> this for other virtualization  systems.
>
>
> --
> Regards,
> Kuldeep Porwal
> IIIT Hyderabad
> 09550605256
> http://web.iiit.ac.in/~kuldeep.porwal
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/13534_NeoTech
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>