[Buildbot-devel] copying information between steps or schedulers

[A.T.Hofkamp]

Hello,

Stefan Seefeld wrote:
> A.T.Hofkamp wrote:
> 
>> Hello,
>>
>> I am busy convincing buildbot to build, install, and test our code and 
>> have a few challenges.....
>>
>> The basic sequence is
>> 1. From the SVN repository, build a number of tar.gz-files
>> 2. Install these tar.gz files at 3 different slaves
>> 3. Test the installed version
>>
>> And the challenges I currently see:
>>
>> ad 1: 'svn update' is easy, but the repository revision is used as 
>> suffix in the name of the .tar.gz files in the following builds. I 
>> dont' see a way to forward such information between build-steps 
>> currently.
> 
> 
> svn itself can tell you anything you want to know about the current
> working copy. Your build system should be able to call that to compute
> package names.

I thought of that, but it seems a bit stupid to me to ask the working copy 
repeatedly for its revision, so I'd prefer a different solution.

> 
>> ad 2: The best approach seems to have 1. as a seperate schedule, and 
>> have 3 downstream schedulers for 2., one for each slave.
> 
> 
> buildbot can call 'svn update' with a specific revision number. While
> I can see some advantages in sharing a single source tree among multiple
> build trees, this is not strictly required.
> So, you could work with a single scheduler and let each slave do a separate
> update / build / package.

Not quite useful here, 'build from the repository' needs a lot of additional 
software, hence this step is restricted to one host.
Restricting to one host also eliminates problems that may be caused by 
different behavior of the additional software at different slaves.

>> The names of the tar.gz files (or the repository revision alone would 
>> be sufficient) need to be transported between the upstream scheduler 
>> and the downstream schedulers/slaves. No idea how to do that.
> 
> 
> I agree about the need for a facility to let schedulers (and associated
> builders) share data. Right now I'm working around that by letting
> upstream builders put all the info I need into a file that the downstream
> builders will read. Obviously, that requires some communication mechanism
> outside the buildbot (such as NFS or scp).
> It would be great if a solution for that could be offered inside the
> buildbot framework.

Ok, starting with a simple-minded approach, what about a dictionary with such 
(small) values which is moved to and from the slave at each step in a build ?
Each step can use the values of the map, and also perform updates in it as it 
sees fit.

A second step could be to forward the dictionary to a different scheduler, 
although this is more risky as the latter may start running steps at a 
different slave.

A practical question in this context, if I want to experiment with these 
things, can I change the Python code at the master only, or would I need to 
keep the code base of buildbot between master and slave consistent?
(I think I am asking 'is Python (byte-)code transferred between master and slave?)
Also, where should I change things, in the master.cfg file or is there another 
recommended way of extending buildbot with user-defined code?


Albert