[Buildbot-devel] copying information between steps or schedulers

[A.T.Hofkamp]

Hello,

Stefan Seefeld wrote:
> A.T.Hofkamp wrote:
> 
>> This may not even be good enough. I'd really like to optimize this 
>> further and build only those tar.gz files that have actually changed 
>> between repository revisions. Some builds take a long time, and I do 
>> not want to do them if it can be avoided.
> 
> 
> While I agree, I'm not sure I understand how this relates to the buildbot.
> Build systems track dependencies precisely so they can determine what
> needs to be redone after an update. And this is why buildbot supports
> 'update', not just 'checkout'.

Hmm, I was not clear enough, it seems.
I was referring to something else, which is quite a lot more complicated than 
I can explain here in a few sentences it seems.
Sorry for trying to bring it forward in the first place, it didn't help the 
discussion I am afraid.

I'd be happy to discuss it with you if you like, but not here. It has to do 
with the way we organize the source code, not with how to convince buildbot to 
build it.

> (Note that you keep referring to slaves, not builders, which I think is
> not quite accurate, as you could host multiple builders on a single slave.)

You are right indeed, I am thinking too much in the way I see buildbot operate 
at our site instead of the general case.

In the general case, I would have several slaves that can perform builds from 
the same builders. In other words, results from builds would become scattered 
over the slaves if the results are kept at the slave hosts, ie not good!!

That would lead to the conclusion that the only good place to store results is 
at some central point, eg the buildmaster host or some common directory 
accessible to all hosts involved.

>> In other words, I would need history of builds done in the past which 
>> have not yet been deleted.
> 
> 
> Right.

Generalising the above, I seem to need a form of (persistent) state common to 
some components of buildbot.
I don't yet understand fully what components yet; I tend to say 'builders', 
but more components may need it.

I just reaslised that the status output is also state; it shows which parts 
are good and which parts are bad (throughout the last X days). I have no idea 
how that information is stored (or even whether it is stored anywhere 
explicitly), maybe builders should have access to such information (or maybe 
other (user defined) forms of information....).

Hmm, looks dangerously like a fundamental shift, lots of thinking to do first.....


>> There is another reason why I hesitate to query the repository for its 
>> revision.
>> With a hook in SVN you can trigger buildbot into starting the build. 
>> At that point, there is revision information available, which, 
>> ideally, should be transferred to the steps so it is certain that you 
>> build what buildbot asks.
> 
> 
> but the revision *is* available to the builder: the SVN step uses it !
> See http://buildbot.sourceforge.net/manual-0.7.2.html#Attributes-of-Changes
> for related docs.

Ah, this is new information. I did read the docs, but didn't pick up that 
there is magic link from 'sources' to (SVN) steps (through 
schedulers/builders/builds?).
It would have been easier to understand this if the information would be 
passed on explicitly by eg a 'svnrevision' argument in the step.SVN object.





I thought about passing information from one step to another yesterday 
evening, and it seems there is another (bigger?) obstacle here, one step 
further down the road.

Suppose we have this dictionary with information which is passed on from step 
to step. How is information from this dictionary then forwarded to the shell 
script, python script, C binary, etc used to perform the actual step?
(ie how to go from { 'svnrevision' : '2938' } in a Step object to having 
'2938' in '$revision' in my shell script/python program/binary/etc?)

Also, to update the dictionary, we need to go back, ie if some program 
produced 'my-file-r2938.tar.gz' and we want to add that to the dictionary as 
'resultfilename', how does a Step object obtain the information that it should 
do that from the script?


At the input side of the script, I can see a solution with pattern 
substitutions, ie I would write a step as

   s(step.ShellCommand, command=['tar','xzf','myfile-r%(svnrevision)s.tar.gz'])

maybe we should optionally be able to specify which substitutions should take 
place here.

At the output side, things seem more difficult; how to state which changes 
should be made in the dictionary as a result of executing the command, and 
what values to use for the changes?
Lots of puzzles to solve I am afraid.... :-)


Albert