[Buildbot-devel] Discussion on source steps

[Alexander O'Donovan-Jones]

We're a games company, and a single branch currently contains 3 games. Unfortunately one of the games in development is based upon the Unreal 3 engine which requires the sdk's to be in a particular location.
Art and audio resources combined with extensive 3rd party sdk's causes a lot of bloat in the branch that we can't really avoid.

It does, however, nicely encapsulate your argument about custom source requirements though. I'm sure there are others out there with sufficiently non-mainstream requirements, who, for whatever reason find the default source classes insufficient.

From: Amber Yust [mailto:ayust at yelp.com]
Sent: Sunday, 12 June, 2011 2:25 AM
To: Alexander O'Donovan-Jones
Cc: Dustin J. Mitchell; Dmitry Nezhevenko; buildbot-devel at lists.sourceforge.net
Subject: Re: [Buildbot-devel] Discussion on source steps

I kind of stopped reading at "our branches are ~40 GB" - dear lord, what are you putting in your repository? :o

~Amber
On Sat, Jun 11, 2011 at 5:16 PM, Alexander O'Donovan-Jones <Alexander at ccpgames.com<mailto:Alexander at ccpgames.com>> wrote:
Hah! Our branches are ~40GB! We gave up trying to use the standard Buildbot Perforce VCS steps for some of the issues you outlined, for example, if you modified a client spec mapping the branch to disk (such as adding an extra folder that you had previously omitted for brevity) it would clean out and force sync the entire thing, and then usually timeout due to the huge deluge of stdio input it would receive, causing the process to hang.
________________________________________
From: Dustin J. Mitchell [dustin at v.igoro.us<mailto:dustin at v.igoro.us>]
Sent: 11 June 2011 19:46
To: Dmitry Nezhevenko
Cc: buildbot-devel at lists.sourceforge.net<mailto:buildbot-devel at lists.sourceforge.net>
Subject: Re: [Buildbot-devel] Discussion on source steps

On Mon, Jun 6, 2011 at 1:38 PM, Dmitry Nezhevenko <dion at dion.org.ua<mailto:dion at dion.org.ua>> wrote:
> This one looks very interesting for me. But maybe even better. Something like
> script-based VCS command that calls custom helper scripts for every
> particular action:
> - Cleanup
> - Update to specific branch and revision.

This would be super-cool, and give people a great way to implement
their own insane checkout procedures without having to write custom
code on the buildmaster.  I'm envisioning a factory that runs a
FileDownload step to put the latest and greatest version of the script
on the slave, and then runs the ScriptSource step to interface with
it.

> Another VCS-related issue I'm experiencing is some kind of timeout
> handling. Usually VCS operations depends on network or any other factors
> that may fail. In case of network-related issues (like temporary 30
> seconds downtime) it sounds reasonable to not fail source step and just
> wait a bit and retry. At the same time not every VCS error should cause
> retry. Like "unable to authenticate" error or "repository not found" has
> no changes to recover.
>
> Currently buildbot doesn't provide any way to handle this some "standard"
> way and it's up to slave command to handle everything. I'm doing
> everything on slave by reading CLI client exit code and sometimes parsing
> of it's stderr.
>
> So since you are trying to refactor VCS stuff, maybe it's good idea to
> think about this.

I agree - one of the most common problems with builders at Mozilla is
failure to download.  It doesn't help that we're cloning a 250MB
repository, and since we're using hg that's bringing the entire
history with it.  Let's all honor the sacrifice of hg.mozilla.org<http://hg.mozilla.org> (and
build some read-only mirrors for the buildfarm..).

Anyway, most (all?) VCSes support a retry parameter right now, which
specifies how many times to retry, and how long to wait between those
retries.  I'm using this on the metabuildbot because some of the
donated slaves have a hard time keeping a connection to github
running, but after 5 retries they almost always manage to get it
working.

Dustin

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel at lists.sourceforge.net<mailto:Buildbot-devel at lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/buildbot-devel


------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel at lists.sourceforge.net<mailto:Buildbot-devel at lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20110612/aa033864/attachment.html>