[Buildbot-devel] How to create sub-columns for each buildslaves under the same builders in waterfall

[Marcus Lindblom]

On 2010-01-27 04:56, Charles Lepple wrote:
> On Jan 26, 2010, at 9:22 PM, Dustin J. Mitchell wrote:
>
>> I think your first question should be: is the waterfall the best view?
>>
>> For most purposes, either the console or one of the grids is the most
>> useful display - they show the things developers are interested in -
>> commits and builds - while omitting the things that developers don't
>> care for - relative timing of various steps.
>
> As someone who used to run Buildbot on a 30+ minute build, the
> waterfall is the most useful way to visualize *progress* on the build
> (rather than a simple pass/fail status). We used the individual step
> labels, as well as a custom status procedure that kept track of the
> last directory that the build process descended into.

I've been thinking to add some kind of progress to the grid or 
one_box_per_build result. Like two progress bar, one that works on time 
(estimation) and another that shows % of all steps completed.

I'm thinking something like Hudson's bars, in the boxes for each 
builder: http://wiki.hudson-ci.org//download/attachments/753667/1.png

(Hudson has a rather slick UI I must say...)

> The build
> process also kept running after errors ("make -k"; not my decision) so
> this was the easiest way to see if a large chunk of code was being
> skipped over due to failed dependencies.

This is what we do as well, since we want to isolate errors as far as 
possible, then it's a good idea to build as much as possible. I haven't 
added a custom step that flags the build as failed as soon as errors 
appear in the log though. That is probably a good thing to have when 
running with -k (or similar.)

> I would agree that the waterfall is not the best place to start
> hacking if you are new to Buildbot and Python, but I also hope that
> the waterfall doesn't start to fall by the wayside just because there
> are a few more high-level summary views.

Not at all. The waterfall has its place, especially when debugging locks 
or looking at detailed progress. But the code _is_ a big spaghetti pile, 
although it might be slightly easier to work with in 0.8. (F.ex, I 
didn't manage to extract all HTML from the python code there when 
converting to jinja. More work there is welcome, so that one is able to 
embellish that data.)

Also, as Dustin said, views that access a lot of Buildbot's stored data 
are a bit slow, and the waterfall is one example. I suppose adding more 
features is nice, but we might need to address the performance if we do 
even more stuff here.  (I'm seeing response times of 0.5-2 secs here 
when I have three builders with a few hundred builds each. This is not 
ok for public buildbots.) Some form of caching of generated html would 
be nice to have here, so that the we only regenerate the boxes contents 
if they actually changed. That'd keep the Waterfall from digging deep 
into build-history every time.

Actually, I'm not sure the grid & console views are good performers 
either...

/Marcus