[users at bb.net] BuildBot versus Jenkins

[Alan Ray]

Jeremy,
I'm in the process of spinning up Buildbot for my current company as I've grown dissatisfied with Jenkins. Greg's assessment is spot-on in my opinion; I'll throw in a few more thoughts. 

A little background: At a previous company, we used Buildbot quite successfully. At my current job, Jenkins was chosen before I arrived, but I've been pretty active in maintaining, updating, adding projects, and reworking workflows. Fair warning: With Buildbot, while I've the read 0.9 documentation and started planning out the workflows, I haven't actually used Buildbot since an early 0.8 release so I may be slightly off on the current Buildbot functionality. 

My overall goals:   
   - A solid yet agile continuous deployment / continuous delivery pipeline model for multiple projects. These projects are complex (e.g. they need to build and test on multiple operating systems) as are the pipelines (i.e. unit tests, multiple functional tests that require different system fixtures, manual testing stages that aren't yet automated). At the same time, these projects use similar build systems with common tools (e.g. virtualenv, pylint). Consequently, I'd like to have a reusable or templated implementation of these tools.    

   - Change management is important as a project scales up. I want to know how the build process changed (e.g. version control), and be able to predictably update / roll it back. If unit tests are disabled, it should be intentional and trackable, not a forgotten side-effect of forgetting to add the '--test' flag to one of five jobs.   

   - Documentation, comprehensibility, and flexibility of the build system matter, especially for enabling different teams to customize their process.    


Regarding those goals, here's what led me to BuildBot:   
   - Almost nobody is handling pipelining well (Go: Continuous Delivery may be the exception [2]). In particular, I believe that complex pipelines are better written in code than a UI. Practically, Jenkins doesn't handle pipelining at all, although there is some alpha work toward that support...see my comments later about the fragmented nature of Jenkins addons. Buildbot is somewhat lacking here as well (especially in documentation / examples), although I believe all the basic tools for pipelining exist natively in Buildbot [1]. And since it is designed to be customized and extensible, it's much more natural to build the necessary missing pieces.    

   - The Jenkins plugin approach is very fragmented and poorly documented, which makes configuration very difficult. Plugin X pulls in Y and Z which pull in S, T, and R. If I upgrade T for one project, will that break other projects that depend on Y? It's really hard to tell. Additionally, many of components aren't clear how they work. We've had incredible difficulty getting matrix builds to cascade properly into a matrix functional test, for example, or getting the scheduler to drop long-running tests that are outdated because of branch advancements.   

   - The configuration issues boil down to the "primarily done in UI" vs. "primarily done in code". A programming language (text) has a flexibility that enable details like code reuse, object reuse, dynamically generated build names, etc. Jenkins really shines at a "I have a simple build I want to setup once" type problems. Buildbot solves the "I have a large number of similar but slightly different pipelines that I want to be able to quickly update" type problems.    

   - Because Buildbot configuration is code, all the standard software practices for change control (source control management, unit tests, deployment scripts, etc.) can be applied to it. One could probably even design a functional test for a buildbot configuration that made sure it could run all the builds. Jenkins has some methods of remote control (there's a job configuration API) and a plugin or two that try to generate jobs from templates. But it is all very rigid and I haven't been able to make it work in a "When I use this tool, does it feel natural or do I feel I am trying to do a handstand while riding a unicycle and eating a banana?" sense.    

   - We are a C/C++/Python/JavaScript/HTML shop. Conveniently, we use SQLAlchemy, Angular, and Jinja2 as well -- Buildbot was where I learned about Jinja2. So diving into Buildbot innards or writing a few lines of Python isn't strange turf for us. And I really like the Buildbot tool choices. In contrast, nobody works with Java or Groovy regularly where I work, so Jenkins is a foreign language. Buildbot is a better fit for our team skills.    


Overall, I think Buildbot is state-of-the-art when it comes to leveraging code (e.g. class/function reusability, dynamic configuration, version control) for the build automation. Even looking at commercial products, nobody seems to be advertising similar functionality. When it comes to a complex system, it's much easier to write or modify a for loop and some construction arguments than it is to painstakingly modify the UI elements on 15 different pages. And reusable objects make it clear which builds use the same object (and thus what will be impacted by a change). Yes, there are challenges, but software architecture is the realm of developers, so practices such as good naming conventions and modularity should be very natural. This leads to a very predictable, agile build configuration.
>From what I've seen, Buildbot is still behind the curve in the statistics / UI realm, although I expect with 0.9 release and Angular, it will be in a position to catch-up. But as the number of build permutations and pipeline stages increase, Jenkins simply forces too much uncontrolled code duplication, leaving various builds in "similar but not quite identical states". At the end of the day, I care about "Was this build built the way I expect?" more than I care about "Did my build make a pretty picture?" 
 [1] For example, there's very nice presentation from back in 2012 that describes almost exactly the sort of problems I'm looking at: http://releng.polymtl.ca/RELENG2013/html/slides/buildbot-talk/index.html. There, pipeline management talks about the idea of "flocks", but that ended up getting aborted in favor of existing objects (sourcesetstamps, see http://trac.buildbot.net/ticket/875). Reading between the lines in the documentation, I think I agree that DependencySchedulers, properly defined SourceStampSets, and maybe some additional customizations will work. But I haven't found a nice tutorial that takes the examples in the slide show and shows the corresponding Buildbot implementation. (Don't get me wrong -- I get that documentation is hard, it competes with other priorities, etc. I suspect that Buildbot can do a lot more than it advertises though, and I wish it showed that capability better. Maybe I'll write it up myself if I get a system I like running.)
[2] I really like what Go: Continuous Delivery has done with making concepts like pipelining, fan-in, and manual gating first class citizens. However, it doesn't have the community infrastructure yet, and it is a UI-based approach. I think my dream solution is combining the GoCD perspective on top the Buildbot capabilities. There's a nice GoCD vs Jenkins writeup here: https://highops.com/insights/continuous-delivery-pipelines-gocd-vs-jenkins/
    
I hope that helps,
Alan


Hi Jeremy,

I have experience with both. Jenkins is more about providing canned plugins that offer specific workflows whereas Buildbot is more of a toolkit to build exactly what you need. With Jenkins you'll spend a lot of time looking around for plugins that may or may not be a fit. If those plugins are buggy you'll have to wait for a fix from the community or learn java and dig in anyway. On the up side for Jenkins there are tons of plugins.

Jenkins is easier to use initially but ultimately less powerful; BB is more complicated to learn up front but very powerful. For example, BB nine's angular based UI plugin system is cutting edge and incredibly powerful, but you'll have to learn angular. On the up side angular is a very popular and well-designed system worth learning. There is a lack of UI plugins with BB, but I'm hoping to see more community contributed plugins in py-pi after nine releases.

Both are fine build systems, but in my opinion I think Jenkins is good for smaller projects that aren't too complex. Since you have a sophisticated system I would tell your boss it's not worth the effort to change. Jenkins is far from perfect. And buildbot is composed with industry standard technologies like python, nodejs, and angular. As well as solid libraries like twisted. It's also one of the most well designed projects I've ever come across.

-Greg

From: users [mailto:users-bounces at buildbot.net] On Behalf Of Jeremy Cornett
Sent: Sunday, January 31, 2016 2:00 PM
To: users at buildbot.net
Subject: [users at bb.net] BuildBot versus Jenkins

I've been an administrator of my company's automated build system for almost 4 years now, and for all of that time we've been using BuildBot. We have a rather sophisticated system, with a lot of added functionality, and I rather like BuildBot. My boss is wanting me to explore replacing our BuildBot instance with Jenkins, with the reasoning that Jenkins is more of an industry standard (more active development community) and it's easier to use from a developer/tester perspective. I really don't know if his reasoning is correct or not as I've never used Jenkins, but that aside, I was wondering if anyone else has faced this question, "BuildBot versus Jenkins," and how did you answer it?

[Description: http://www.venafi.com/img/venafi_email_signature_logo.gif]

| Jeremy Cornett | Configuration Management Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160206/8c69ce5e/attachment.html>