[Buildbot-devel] copying information between steps or schedulers

[A.T.Hofkamp]

Hello,

Stefan Seefeld wrote:
> A.T.Hofkamp wrote:
> 
>>> svn itself can tell you anything you want to know about the current
>>> working copy. Your build system should be able to call that to compute
>>> package names.
>>
>>
>>
>> I thought of that, but it seems a bit stupid to me to ask the working 
>> copy repeatedly for its revision, so I'd prefer a different solution.
> 
> 
> 'repeatedly' ? Why isn't once per update enough ?

repeatedly, since I currently build 7 different tar.gz files from the repository.
In the future, this number will grow.

This may not even be good enough. I'd really like to optimize this further and 
build only those tar.gz files that have actually changed between repository 
revisions. Some builds take a long time, and I do not want to do them if it 
can be avoided.
To optimize, I need knowledge of what tar.gz files are already installed at 
the various slaves, which kind of boils down to having to know which other 
tar.gz files exist (at the 3 slaves).

In other words, I would need history of builds done in the past which have not 
yet been deleted.

Not sure what role buildbot should play in such a situation, the best solution 
may be to inspect some file system for this information.

On the other hand, if the dictionary is persistent between invocations of the 
build steps, I could extract such information (probably by adding some 
additional schedule that deletes obsolete tar.gz files, but that program is 
needed anyway).
Just dreaming...... :-)

>> Not quite useful here, 'build from the repository' needs a lot of 
>> additional software, hence this step is restricted to one host.
>> Restricting to one host also eliminates problems that may be caused by 
>> different behavior of the additional software at different slaves.
> 
> 
> I see, so you have generated a package in an upstream builder but don't
> quite know its name in a downstream builder that wants to test it. Right ?

yep.

There is another reason why I hesitate to query the repository for its revision.
With a hook in SVN you can trigger buildbot into starting the build. At that 
point, there is revision information available, which, ideally, should be 
transferred to the steps so it is certain that you build what buildbot asks.

If instead I run 'svn update', 'svn info' the working copy of the build slave 
may have a different revision than the commit that originally triggered 
buildbot (additional revisions may have been added to the repository between 
the start of the build and the execution of 'svn update').

I do agree that this chance is almost zero though.


> That sounds indeed very similar to a problem I tried to solve, where
> my upstream builders build and test, generating test result files whose
> names encode the platform they were generated on, and a downstream
> builder that wants to collect all the results to build a test report.

Similar indeed.
The biggest difference seems the size of the information. I need to transport 
entire files, you only need to transport file names.
(obviously, 'entire files' are easily reduced to 'filenames' by adding some 
host+user information, together with a scp action)

Tnx for the information. I just printed a number of source files to study this 
evening.

Albert