[Buildbot-devel] RFC: Crowd-sourcing CI using volunteer computing

[Stefan Seefeld]

Hi Dustin,

I'm happy to see you like the idea. Yes, this would open the door to a
large landscape of new applications. As you say, things get pretty
difficult if treated in full generality. Thus I think it might be best
to pick one or two "smaller" use-cases that will help explore in this
direction, without requiring all the features at once. Once those work,
we can assess what we have got and lay out next steps that get us closer
to the long-term goals. As I mentioned before, I'm a happy user of
incremental processes. :-)

A rather specific use-case I have in mind is this:

I'm working on a few projects to develop portable libraries, running on
a wide range of (sometimes exotic) hardware. At present we are using
buildbot to do CI, running on a build farm of a dozen (or so) machines.
As we add support for more hardware, it's becoming increasingly
difficult to grow the build farm to include machines to cover all the
platforms we want to support. So, I'm wondering whether users couldn't
contribute their own hardware to fill in for this need. In addition to a
few traditional build slaves that use one of the built-in schedulers,
I'd like to add a mechanism whereby users could "pull" builds whenever
their hardware is available, and report back test results, to allow us
to have full coverage without the need for all the hardware (or
platforms in general) in-house.


So, to get this set up I think we may make some simplifying assumptions:

1) we could make this work on "simple" build steps not involving
master-side logic, i.e. the equivalent of "configure; make; make check"
2) Contributors know the project and trust the build process to not
compromise their system, so we need neither a VM nor other security
mechanism

I believe this would be just the right set of initial goals to get this
off the ground, i.e. work on the basic design for spawning off-line
builds and defining the set of meta-data that needs to be exchanged for
this to become useful.

Once this works we can think about more complex (master-centric) build
logic, build encapsulation (via virtualization), security, and
packaging, as well as all the other use-cases you are alluding to in
your reply.

Regards,
        Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...