[Buildbot-devel] GSoC 2012: JavaScript frontend

[Marc-Antoine Ruel]

Le 25 mars 2012 14:34, Muslim Chochlov <muslim.chochlov at gmail.com> a écrit :

> Hi,
>
> I'm a second year master student at Free University of Bolzano (Italy). I
> am mostly a Java/C programmer
> and have a fair experience in both languages. Though previously I've done
> some programming in Python and JS.
> One of my responsibilities in my previous work was to administer a Bamboo
> CI. Currently as a part of my research at the uni
> I'm setting up a multi-platform project build using Jenkins.
> The reason I've chosen this project is because I would like to develop CI
> myself and find out how it works internally.
> Another advantage to carry out from this summer is that I'll be able to
> improve my Python/JS coding skills.
>

Hi

That sounds great! The fact that you played with other CIs is definitely an
advantage as you can compare the way each CI displays its build status and
figure out the best ways to distill the information in an efficient way.

Note that registering to GSoC implies a significant contribution so this
could conflict with your research work. I just want you to be sure you plan
accordingly.


> Now I did some research on JS frameworks and found a nice comparison in
> wiki [1]. I was mostly choosing between
> Dojo, jQuery and Pyjamas. And I think that jQuery would be the best choice.
> - It has a testing framework - QUnit
> - it supports most widely used browsers (IE, Firefox, Safari, Chrome)
> - has built in functions to call Json services
> - supports Ajax
> - licensing is MIT and GPL (which I think should be fine, though I'm not a
> big license expert so correct me if I'm wrong)
>
> A few questions for you:
> 1. Which part of buildbot needs to be refactored? If I understand
> correctly web files are only in the master part of buildbot. If browsing
> the code
> there is "buildbot/master/buildbot/status/web" folder. Is it the only
> place that is going to be affected or there are hidden dependencies in the
> code.
>


> 2. In the idea's description it is said "The JS frontend will talk to the JSON
> REST service provided by the Buildmaster. Much of this service is already
> implemented, but it will need to be expanded and improved, which will
> require some Python programming and learning a bit about Twisted Python".
> What exactly has to be improved?
>

It is on an "as-needed basis" to implement API end points required to be
able to display whatever status page you want to implement in the most
efficient way. So it is possible you don't need to touch the python code at
all. If you find a hole in the /json API, the file you would need to touch
is
https://github.com/buildbot/buildbot/blob/master/master/buildbot/status/web/status_json.py
which
contains all the json API. Then it is possible you need to touch the
database requests if needed.

For example, if could be interesting to have an end point that keeps the
channel open and sends event as they are coming, a bit like HttpStatusPush,
but in a listen-wise access method with a "cursor".
https://github.com/buildbot/buildbot/blob/master/master/buildbot/status/status_push.py.
So the idea is that you would have a HttpStatusListen end point, added
under WebStatus
https://github.com/buildbot/buildbot/blob/master/master/buildbot/status/web/baseweb.py
that
would accept a "event id" that would add a cursor and *streams* events to
the listener as they happen. This idea is totally optional, I'm just saying
that as an idea of what would require python work. Note that I wrote the
initial version of both status_json.py and status_push.py.

Another example, let say in part of displaying the build status, you'd like
to get the build step 'test slave' for the last 5 builds. While you could
do something like:
http://buildbot.buildbot.net/json/builders/slave-minime/builds/?as_text=1&select=-1&select=-2&select=-3&select=-4&select=-5

but that request downloads several times too much information as it
downloads all the build steps for each of the last 5 builds. Sometime a
builder can have tens of build steps so it is not efficient. So it makes
sense to optimize the request to reduce the amount of information sent by
the server. In that specific case, it is already implemented as you can do
instead:
http://buildbot.buildbot.net/json/builders/slave-minime/builds?as_text=1&select=/-1/steps/test%20slave&select=/-2/steps/test%20slave&select=/-3/steps/test%20slave&select=/-4/steps/test%20slave&select=/-5/steps/test%20slave
.

Doing something like http://build.chromium.org/p/chromium/stats (it doesn't
seem to be enabled on buildbot.buildbot.net) would require a huge amount of
json request in the current format, so maybe it would require an API to do
light computation on the master. Having more statistics is gold.

---

You can play with it "live" with the buildbot's CI at
http://buildbot.buildbot.net/json/help. Since the /json API support CORS (
http://en.wikipedia.org/wiki/Cross-origin_resource_sharing), you can create
a local web page and experiment "live". Be sure to not do too many requests
to not kill Dustin's master. :)

While I haven't formally registered as a mentor yet, it's been a long time
I haven't contributed to buildbot and my javascript experience is totally
lacking so I couldn't do proper code reviews, I'm extremely excited in
getting this going forward.

In particular, I would personally judge the javascript code by 3 factors:

   1. The code is readable, maintainable and the code style is consistent.
   e.g. ";" after each lines, consistent brackets placement, sane level of
   comments, no variable aliasing, proper local variable scoping etc. Using
   something like the closure compiler isn't a bad idea either.
   https://developers.google.com/closure/compiler/
   2. It does the strict minimal requests with the minimal amount of data
   to display its status pages so it doesn't overload the master*. I'm totally
   open to doing it progressively so that optimization can be done later. I
   prefer something that *works *at all than something that is efficient
   and doesn't work. :)
   1. There is two goals here:
         1. Reduce the load on the master, as it doesn't need to render
         huge amounts of html from templates but only a smaller amount
of json data.
         2. The data is easily cacheable by a http proxy. For example, once
         a build is completed, its json representation will never change so its
         Cache-Control header value should be set accordingly in
status_json.py.
      2. This requires a minimal knowledge about how to profile XHR
      requests and the load it causes on the master. This should be
learned on an
      as-needed basis, it's not a huge priority here.
   3. It displays on a vast amount of browsers. In particular, it should
   work on the latest version* of each of: Google Chrome, Firefox, Safari
   mobile (for iphone/ipad) and Chrome on Android.
      1. * The latest version as current as next summer. So for Firefox,
      it's probably be around Firefox v13 or v14.
      2. If you can get work done faster by not supporting IE, I don't
      care. It's a nice to have. IMHO for IE, it'd be IE10 whenever once it's
      released.

I hope this sheds some light and looks like an interesting project.

One last thing, I followed the buildbot setup tutorials [2] and
> successfully installed buildbot on my machine and even run build for some
> project called pyflakes :)
> It is mentioned that you could help with hosting. I would appreciate if
> you do that.
>

Dustin could reply about that.

M-A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20120326/1b582754/attachment.html>