[devel at bb.net] GSOC 2018 & buildbot improvements

Fri Feb 16 13:02:32 UTC 2018

+dev-buildbot. If any of you guys are interested in mentoring GSoC, please
raise hands.

Hi Mojca,

Congrats for the GSOC grant!
We have experience with GSOC, with already a dozen student mentored.
We didn't manage to get a new grant when the gsoc management team changed,
and was a bit disappointed by the lack of explanation on why we weren't
chosen. So we gave up trying.

I liked the GSOC mentoring though, and I would be glad to do it again.

I acknowledge your pain points on Buildbot eight, and agree that nine will
probably not help that much with the current turnkey visualisation plugins.
Other people are also frustrated by the nine version of the waterfall,
including the webkit people https://github.com/buildbot/buildbot/issues/3884
I added lengthy explaination in this issue on why it is like that and why
it is not easy to improve. But help is appreciated.

My suggestion for webkit is also valid for macports. What you may need is a
new UI plugin, which is carefully crafted for your requirements.

I think this is a very good match for a GSOC project. This would be a
project to address your points 3, 4 and 5.
I think this can be done by recruiting a front-end student. Probably this
kind is a bit easier to recruit, and the problem are simple enough to
enunciate clearly for a student.
The project might need to develop some python code in order to speed-up the
queries (like the one to matches every builds associated to a change
https://github.com/buildbot/buildbot/issues/3927 ). This is something that
I could help with if the student is not confident to do the backend part by
him/herself.

For point 1. Buildbot nine now has several tools to mitigate security
issues related to testing PRs.
- SecretManager can restrict the access of secrets to builds happening
after patch is merged
- LatentWorkers can make sure a new VM is spawn for each build, but
probably the MacOs part is not as easy on your side.
In any case it is a tough problem that is not easy to resolve as basically
the goal of the CI is to give developers access to machines, a malicious
developer will easily take control of that machine.
A potential project for this problem would be to add some extra triggering
criterias for building a MR, like the approval of at least one trusted
member of the team.
It looks like a lone a bit short subject for GSoC, I would say.

For point 2, eight and nine both have ways to prioritize buildrequests
according to various criterias. The way did change a little bit, and nine
is a little bit better for that, but I don't think it is significant for
what I understand from your usecase.

There might be enough of a workload for a GSoC project if you combine point
1 and point 2, but this will be a little bit more difficult to mentor as it
is very specific problems which for me could use a bit more experienced
people to conduct.

Regards
Pierre

On Wed, Feb 14, 2018 at 9:55 AM Mojca Miklavec <mojca at macports.org> wrote:

> Dear Pierre,
>
> MacPorts got accepted for GSOC 2018.
>
> A while back we discussed some aspect of BuildBot that could be
> improved (I remember something about a GSOC student doing a poor job
> implementing waterfall for buildbot nine?).
>
> A short question: would anyone from your team be willing to (co)mentor
> a potential applicant working on buildbot?
>
> A long explanation follows.
>
> We are currently using buildbot eight. Reasons why:
> - We wrote the setup in 2016 when nine was not totally stable yet and
> we wanted to play a bit on a safer side. Nobody did any major rewrite
> since then, but we plan to have an in-person meeting for an extended
> weekend around March 10 in Slovenia and trying to migrate to version
> 1.0 is on agenda for that meeting as well. (If anyone from your team
> needs some hacky holidays on snow, just let me know.)
> - Back in 2016 the waterfall in nine was relatively inferior to eight,
> at least for our needs. On the one in version nine it was much more
> difficult to see anything.
>
> Now, let me admit that I didn't look too closely at buildbot
> development since 2016, I do feel sorry that we were are able to use
> some of the features existing in version nine, but just to explain
> that please excuse me if I'm not aware of all the development that
> must have happened since.
>
> Here's what we are currently missing:
>
> 1.) A safe setup. Someone wrote travis setup for testing pull requests
> last year. Travis has lots of drawbacks, the main one being that most
> of the PRs simply time out, the less severe one that we are not able
> to test on arbitrary setups (it would be super useful if we had the
> ability to occasionally test changes on Mac OS X 10.6 for example).
> I'm pretty sure that there's nothing buildbot itself can do about it,
> but we had no clue if there's a painless way to "make our own
> buildbot-based travis-like service" that would not be susceptible to
> malicious attempts.
>
> 2.) I believe this is a specific deficiency of buildbot eight, but we
> have troubles controlling the order of scheduled builds. (We have one
> builder that checks which ports need to be built and then sends ten
> jobs to another builder; those jobs then get executed in random order
> which is somewhat painful. I'm sure there's a workaround as well, but
> also pretty positive that 1.0 works better and can work in a
> completely different way anyway.)
>
> 3.) One of the biggest "pains" we are facing is that there's almost no
> way to check whether commit nr. 1234567 failed or succeeded on various
> builders. One can do some kind of "bisection" in either waterfall view
> or by manually changing numbers for a particular builder, but that's
> waaaaay too tedious and what we usually do is simply run the build
> again and check the results from that new build.
>
> 4.) An equally big pain is that we would like to have an overview of
> all the builds of package "foo". On "normal" buildbot setups one would
> have one builder for package "foo", one builder for package "bar" etc.
> We need to build all packages on the same builder (we have tens of
> thousand of packages, building one package potentially requires
> building 20 other ones first) and that obfuscates the results. The
> problem is that we cannot show any "expected time left" based on
> previous statistics, that we cannot do any port/package-based summary
> etc. I would like to be able to pick a package and show when it was
> built, at which version and on which builders it failed to build (and
> at which step: while installing a dependency or while building the
> package itself). It would also be helpful to be able to select a
> committer and display the history of all of his successful/failed
> builds.
>
> 5.) Waterfall on nine/1.0: Last time I checked (long time ago) there
> was no text, just green and red boxes. For us that's hardly useful, we
> need to see the name and phase of the ports being built. You said this
> should be configurable in web though.
>
> Some of these problems could be remedied by writing our own set of
> scripts: json parser for buildbot logs, database and our own website
> to display the results. But I'm pretty sure that some of these
> problems could be addressed from the buildbot core code in a way that
> could be beneficial for other projects as well.
>
> What I would potentially like to see done during a GSOC project would
> be a combination of buildbot codebase extension (with what makes sense
> for other projects) + fine-tuning master.cfg for our own specific
> setup & needs. But even if just the first part is done (implementing
> the required functionality on the buildbot side while leaving
> fine-tuning to us) would be fine by me. But this is only doable in a
> very close collaboration with a (co)mentor from buildbot.
>
> Please let me know what you think of this and if there's a way to
> merge our forces and interests together. Feel free to forward this (or
> perhaps reword it into "your own language" first) to the mailing list
> if you feel others have some valuable feedback or opinion. I don't
> have sufficient insight into what is currently possible and what would
> need to be implemented and how difficult that would be.
>
> Our current setup is here:
>     https://build.macports.org/waterfall
>
> Thank you,
>     Mojca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/devel/attachments/20180216/db262c71/attachment.html>