[Buildbot-devel] GSoC Proposal (draft) : Consistent, Efficient Source Steps

Pradeepkumar Gayam in3xes at gmail.com
Sun Apr 3 07:57:14 UTC 2011

GSoC Proposal (draft) : Consistent, Efficient Source Steps

:author: Pradeepkumar Gayam

.. topic:: Abstract

    Consistent, Efficient Source Steps. Currently to make different kind of
    builds, for example complete or incremental build we set different modes.
    But these modes doesn't match the users requirement. For that modes should
    have the facility to specify what exactly user want to do.

Outline of this proposal

This section provides the outline of proposal, doesn't contain any
description. I listed the main part of the work which will be described in
detail in later parts. This proposal by no means is final, and plan is *will*
change as we advance through project.

    0. Deliverables
    1. Studying relevant material for the work
    2. Implantation steps
    3. Finalizing the all the modes should be implemented
    4. Summarising the common feature between the different Vase's
    5. Diving into buildbot's code
    6. Timeline

As mentioned before, the objective of the proposal is to implement source step
consistently and efficiently. Elaborating, the modes currently present are not
give enough option to specify for different kind of builds to end users. Target
of the project is to improve this feature.

After the summer, an user should be able to specify what kind of build he wants
to do and how. For example: User can specify a whether he want to remove all
the complied artifacts and build again[clean] or build only applying the
changes[incremental]. And user can also specify how to these kind of build with
the other arguments in the source step of build depending on the version
control system.

Study material resource

.. [#] http://buildbot.net/buildbot/tutorial/
.. [#] http://buildbot.net/buildbot/docs/current/full.html
.. [#] http://buildbot.net/buildbot/docs/current/reference/
.. [#] http://twistedmatrix.com/trac/wiki/Documentation
.. [#] http://buildbot.net/buildbot/docs/latest/Buildbot-Tests.html
.. [#] http://trac.buildbot.net/ticket/669

Implementation steps

 1. First step would be finding out what all the modes any user would like to
with VC steps.

 2. Second step would be figuring out the common behaviors across a number of
version control systems, and how to represent those behaviors to Buildbot's
users. This step involves reading about significant reading about different
VCSs. As a git and svn user and having worked on mercurial internal finding out
those behaviors should not be hard.

 3. Third step: Start writing code. As suggested it will be good to work on a
single VCS first and extended the feature to other VCSs. I planned to work first
on Mercurial then git and svn. I fairly figured which part of code lies where, I
will dig into the code in community bonding period.

Finalizing modes

As per the discussion on IRC and on the thread
http://trac.buildbot.net/ticket/669 The possible mode any user wants to have

 1. Incremental
 2. Clean
 3. Fresh
 It is same as the current update mode. Update the current repo with the changes
 and run the build keeping the all the build artifacts. Some other proposed. And
 in the case of mercurial other options can be update to the default branch, or
 build only if the revision is tagged.

Clean build is removing all the build artifacts, pull the changes and build
again.This may be similar to doing 'make clean'. In mercurial's philosophy is
equivalent to 'hg purge', but not 'hg purge --all'.

This should be pristine working directory, either completely fresh clone or
created by doing 'hg purge --all', but later one is not as fresh as the
first. Difference between this mode and previous mode should be defined
clearly. Or should this be replaced by clobber? Sometime users may want to make
a new clone even at the cost of network traffic. If not what is the difference
between this and clobber?

And I think mode=copy should be there as it is for users choice.

Summarising common features
This part requires significant reading about various version control systems. This can started after work mentioned in the above section is finished.

 References for this section

.. [#] http://en.wikipedia.org/wiki/Comparison_of_revision_control_software
.. [#] https://git.wiki.kernel.org/index.php/GitDocumentation
.. [#] http://mercurial.selenic.com/wiki/
.. [#] http://svnbook.red-bean.com/
.. [#] http://darcs.net/manual/
.. [#] http://doc.bazaar.canonical.com/en/
.. [#] http://ximbiot.com/cvs/manual/

Diving into code

As far as I understand from the manual and api, each builder has a build factory
which is responsible for creating build objects. Build objects are performs each
build. These builds execute a sequence of steps called *buildsteps*. Source step
is a process of fetching the source from which build is performed. Depending
upon the properties set to build object eg. type, mode slaves check out source.

Most of the work would be modifying buildbot.step.source and making corresponding
changes in the slave side builslave.command.*


Community bonding period
Reading API documentation and code, get familiar with code.

1. Discussion and decision about various mode and properties of
various source step                                                     [1 week] Till May 30
2. Reading about different version control systems and finding out
various common funtionalities                                    [1 - 1.5 weeks] Till Jun 9
3. Start writing code. Write source step for mercurial                  [1 week] Till Jun 16
4. Write source step for Git and SVN                                   [2 weeks] Till Jun 30
5. Code for CVS                                                         [1 week] Till July 7
6. Write tests and fix issues (if any)                           [Till midterms]
-- Midterm evaluations
6. Code for bzr                                                         [1 week] Till July 22
7. Code for Darcs                                                       [1 week] Till July 29
8. Code for P4                                                          [1 week] Till Aug 5Th
9. Write tests and fix issues (if any)                             [Till finals]
10. Wind up and start submitting code to google

Contact Info

Name : Pradeepkumar Gayam
Email: in3xes at gmail.com
IRC nick: in3xes on Freenode
Timezone: IST



I have significant experience in Python. I worked on mercurial internals and
submitted bunch of successful patches.Accepted patches can be found in mercurial
source tree. Besides that some academic projects and nearly 100 Project Euler
problems solved in Python.

Twisted Python

I understood basic concepts and working of twisted. I haven't done anything for
real. I learned only after knowing about buildbot.

HTTP-based Protocols

Implemented HTTP protocol in C just to learn about protocols and C. Source can
be found at https://github.com/in3xes/downloader I wrote it as
self-exercise. So, its not completely functional.

Web Development

I am member of core team of official college web development team[1], which
maintains all college data and websites. I developed couple of fully functional
applications in php. So, I think I can say I am good at it.

[1] http://www.iitr.ac.in/campus_life/pages/Groups_and_Societies+IMG+present_team.html


As part of the team mentioned above we should also maintain user database. I am
well versed with both MySQL and PostgreSQL.

Java Script frameworks

Some simple applications of jQuery as part of applications mentioned above. I am
not an expert in this area, but I can make things work.

Git and Github

Fairly good at Git. I use Github as my primary hosting service. I didn't do
collaborative development before.

Development for Windows 

None. (I don't think writing some very basic algorithm in visual C++ considered
 as development :) )

Note: This is draft proposal. I'll change the contents as per your feedback.


- Pradeepkumar Gayam

More information about the devel mailing list