[Buildbot-devel] copying information between steps or schedulers

[Stefan Seefeld]

A.T.Hofkamp wrote:

>> svn itself can tell you anything you want to know about the current
>> working copy. Your build system should be able to call that to compute
>> package names.
> 
> 
> I thought of that, but it seems a bit stupid to me to ask the working 
> copy repeatedly for its revision, so I'd prefer a different solution.

'repeatedly' ? Why isn't once per update enough ?

>> buildbot can call 'svn update' with a specific revision number. While
>> I can see some advantages in sharing a single source tree among multiple
>> build trees, this is not strictly required.
>> So, you could work with a single scheduler and let each slave do a 
>> separate
>> update / build / package.
> 
> 
> Not quite useful here, 'build from the repository' needs a lot of 
> additional software, hence this step is restricted to one host.
> Restricting to one host also eliminates problems that may be caused by 
> different behavior of the additional software at different slaves.

I see, so you have generated a package in an upstream builder but don't
quite know its name in a downstream builder that wants to test it. Right ?
That sounds indeed very similar to a problem I tried to solve, where
my upstream builders build and test, generating test result files whose
names encode the platform they were generated on, and a downstream
builder that wants to collect all the results to build a test report.

>>> The names of the tar.gz files (or the repository revision alone would 
>>> be sufficient) need to be transported between the upstream scheduler 
>>> and the downstream schedulers/slaves. No idea how to do that.
>>
>>
>>
>> I agree about the need for a facility to let schedulers (and associated
>> builders) share data. Right now I'm working around that by letting
>> upstream builders put all the info I need into a file that the downstream
>> builders will read. Obviously, that requires some communication mechanism
>> outside the buildbot (such as NFS or scp).
>> It would be great if a solution for that could be offered inside the
>> buildbot framework.
> 
> 
> Ok, starting with a simple-minded approach, what about a dictionary with 
> such (small) values which is moved to and from the slave at each step in 
> a build ?
> Each step can use the values of the map, and also perform updates in it 
> as it sees fit.

Yes, that sounds good. Builders as well as Steps should all have such a
'context' dictionary attached to them which can be accessed and modified
during processing.

> A second step could be to forward the dictionary to a different 
> scheduler, although this is more risky as the latter may start running 
> steps at a different slave.

Yes, the context(s) from an upstream scheduler should be forwarded to
downstream schedulers. It should be understood that it makes little sense
to put filenames or similar into such a dictionary, at least not without
further info, as that isn't useful in a distributed environment.

But these are all concerns buildbot users have to face, not buildbot
developers themselves.

As I'm not a buildbot developer I'll let others answer the other
(interesting) questions.

Regards,
		Stefan