[Buildbot-devel] [GSoC-14] New Idea: Dynamic scaling and instance metadata configuration for Latent slaves

[kuldeep porwal]

Hello,
              I am masters student from India. I
have successfully completed  GSoC-2013 with drizzle database. Here is the
link of my work:
http://www.google-melange.com/gsoc/project/google/gsoc2013/kuldeep2591/5997011295797248

As I was involved with my thesis work I didn't pay attention to gsoc'14
earlier but now as I am free I would like to contribute to buildbot , here
is my* work experience *in continuous integration and its tools:

1. I have worked on jenkins as a part of my summer internship.
    There I was supposed to design continuous integration (CI) for one of
their project. And it involves:
    1.1 *merging build request:* I have developed a protocol which merges
multiple build requests for same ticket scheduled across multiple slaves
 . Buildbot already have this feature as mentioned in the
documentation/system-architecture.

    1.2  *Workspace cleanup: *Wrote my own cleanup plugin as desired for
our CI.

   1.3 *Automatic launch of amazon aws slave*:  On demand launching of
slave helps in many ways:
         i) saves money   ii) Load balancing
        I have used amazon slave plugin of Jenkins to serve our purpose
.They had bugs that time so I patched them specifically for the purpose as
desired for  project that time.

*MyIdea **:*

I have a short discussion with Dustin sir about it a day back. Here is the
idea please suggest any changes :

As buildbot already have a support for Amazon slaves and it starts and
kills those slave as per demand (Latent Slaves).  But at the time of load
spike if you want let say X # of slaves then you need to create X #
BuildSlave instances and their bid price (if spot instances).
---> The change I would like to suggest is to make this *Dynamic*.
BuildMaster will  first observe the load present based on user demand (can
be considered as build request priority and total number of build request
present at a time in schedulers). Then  based on that demand and load it
automatically launches the Latent Slave.
It will only launch the # slaves up till the maximum limit (user parameter
, let say 50).

---> The above launch of build slave will be governed by automatic
configuration by instance metadata. As dustin sir suggested:
Currently, the user must set up each instance image with a different
Buildbot slave name and password, and configure the Buildbot slave to start
on boot.  This could be made much easier for users, perhaps with a
pre-built AMI that starts the Buildbot slave on boot and takes all of its
configuration from the instance metadata.

currently I am focusing on Amazon AWS but as the time permits I can look
this for other virtualization  systems.


-- 
Regards,
Kuldeep Porwal
IIIT Hyderabad
09550605256
http://web.iiit.ac.in/~kuldeep.porwal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140316/5cdc60a5/attachment.html>