[Buildbot-devel] copying information between steps or schedulers

[A.T.Hofkamp]

Hello,

Stefan Seefeld wrote:
> Right now users specify steps in the master.cfg file by providing 
> string-based
> commands. Assuming steps to have some toplevel 'run' method (just to 
> illustrate
> my point), I could imagine users to be able to derive their own Step 
> classes
> inside master.cfg, where they can provide their own 'run' version that 
> accesses
> the shared context. In the end they would of course delegate down to 
> existing
> methods, but instead of setting up the 'command' during initialization they
> could do that at runtime, taking the specific state of the context 
> dictionary
> into account. I'm not sure whether that is viable, or whether there are 
> better

Hmm.
Not sure that your proposal is a good direction; having to write a derived 
class doesn't immediately strike me as easy-to-adopt (that is, I have a hard 
time imagining a busy sys-admin taking the time to learn enough Python to set 
up buildbot by writing a set of derived classes).
Also, assuming that buildmaster and buildslave are at two different machines, 
the derived class seems to be at the wrong machine for easily interfacing with 
the program it is supposed to control. It would be much easier (probably) if 
the new object would be at the buildslave.

While there seems the be room for improvement, your thought experiment was 
quite useful for me; it gave some direction for the solution. Also, your 
remark about initialization versus runtime is a very good one.



During the weekend I installed an experimental setup at my own machine at home 
and implemented the following as a proof of concept:

1. A build (at a slave) has a context dictionary at its disposal. Technically, 
it is a map from key-string to value-string.

2. Prior to execution at a slave, the command-line arguments are searched for 
strings of the form '%(key)', where key is a key-string in the context 
dictionary. When found, the string is replaced by the associated value-string.

3. A new common parameter for a step is 'savepatterns'. It is a list of 
tuples, where each tuple is a pair (key, pattern), where pattern is a RE 
expression with a group. This list is transferred to the slave.

4. At the slave, the text of stdout is collected and split into lines. Each 
line is matched against all patterns of the savepatterns list. When matched, 
the associated key of the savepattern tuple along with the first group of the 
pattern is inserted into the context dictionary.

5. Before the first step, the context dictionary is empty.

[ point 4 is new here, afaik the only output of a program is a) stdout, b) 
stderr, c) exit code. With a simple elimination, option a seemed the best 
solution for obtaining new values for the context dictionary ]


As an example, consider a build of two steps. The first step generates a file 
with a random filename and outputs a line of the form 'Outputname=<name of the 
file>', the second step processes this file further.
In my test setup you'd write the following build sequence:

steps = \
[ s( step.ShellCommand,
      command=['generate-file'],
      savepatterns=[ ('outname',r'Outputname=(\w+)$') ] ),
   s( step.ShellCommand,
      command=['process-file', '%(outname)' )
]

This seems to work in my test setup, at least for ShellCommand's.

Not sure whether the implementation is correct at a larger scale though, I 
didn't do much testing.

The current implementation only affects single builds at a slave, context 
information is not transferred over the network to allow it being forwarded 
from schedule to schedule.


Albert