[Buildbot-devel] Build Properties through the DB / data API

[Benoît Allard]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/18/2014 08:22 PM, Pierre Tardy wrote:
>> inspired myself from buildset_properties and change_properties,
>> both also do it that way. Then I thought, I *just* needed to wire
>> my table to the data API, and all calls from buildbot setting /
>> retrieving the properties would automatically be backed-up in the
>> DB, hence available to the web interface ...
> 
> Sounds like a good start. Gaetan did suggest that we fix the
> datamodel, so that value, source have their own column instead of
> of storing a JSON blob of the tuple.

As properties can be of compound value (dict, lust, whatever), we
can't get rid of JSON. As to if the source belong to the 'value' of
the properties, this disturbed me at first as well, but when I started
thinking about it, I did not really see a real interest in making it a
real column (as we can't get rid of JSON). I think we can continue go
on treating it as part of the 'value' (at the DB level, that is). Or
would you like to perform cross-requests based on the real value, or
the source ? Those are both free fields, so not very suited for that ...

> 
> Turned out to be quite a naive idea ...
>> 
> Sometimes naive works well.
> 
>> 
>> Back in eight's time, build properties used to be the properties
>> that are related to a specific build. I.e. all values accessible
>> through a `WithProperties` call. They include change properties,
>> buildset properties, as well as a dozen of other source for
>> properties. For instance, the build itself sets some, and some
>> steps also do. Some come from the config, other from the
>> schedulers, even the builders can set some. All properties
>> regardless from origin were gathered in the build object, saved
>> as pickle on disk, and the content of that pickle was displayed
>> in the web interface. Back in the time, everything seemed easy !
>> 
> Actually it did not change at all in nine. The build properties are
> still gathered at the beginning of the build. Code is there: 
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/process/build.py#L172
>
>  You can see here how the properties of a build are gathered
> together. Back in the eight time the properties was stored in the
> status object (the one that is pickled). In nine this is stored
> directly in the process.build object (or it will, I'm not sure that
> this task is done yet)
> 
> Anyway, when the build are started, everything is gathered together
> in memory, and I think this is fine that way. The only thing that
> needs to be done is to flush this information to the db from time
> to time so that the data is available to the web interface.

The biggest question was: Where in DB ? should we make a sister-table
for each table that can possibly have properties (like change /
change_properties, buildset / buildset_properties, ...), or is it fine
to just have a build_properties table ? Do slave properties also
belong there ? Scheduler properties ? Builder properties ? Other
properties ?

Which properties are allowed to change value during a build ? I.e.
should they be bound to the build itself, or to their 'origin' ?
That's what I meant with 'in eight was everything better'.

Either way (one big table / lots of sister tables) don't please me
very well, (hence this mail at first). The reason why I feared that I
missed a subtle thing somewhere that would help me fix my mental model.

> 
> What I would suggest is to do that at the end of each step.
> probably at _stepDone: 
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/process/build.py#L409
>
>  You could flush the whole set of properties to the db, or manage a
> cache of what has already been flushed, and only flush the
> difference.

Another trouble about flushing the value back to the DB from time to
time is that you need to keep track of the table the properties are
in. I.e. You can't just put them in a 'properties' variable
alltogether like it is now.

Beside, caches are a hell to manage ...

> 
> 
>> Now I'm quite confused.
>> 
> Hope this help
> 

Well ... as you can see at the amount of questions left in my answer,
I guess we are nearer to a common understanding of the current
situation. As to how far we are to a solution, I still have honestly
no idea ...

Thanks for answering !

Best Regards,
Ben.

- -- 
Benoît Allard (B30A05B0)|Greenbone Networks GmbH|http://greenbone.net
Neuer Graben 17, 49074 Osnabrück, Germany | AG Osnabrück, HR B 202460
Executive Directors: Lukas Grunwald, Dr. Jan-Oliver Wagner
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBAgAGBQJUbbMXAAoJEHZCfVOzCgWwY/AH/iDUXb5/HZ8WGcjnhzxlAk1E
QvJ+8lQmb8yWnFD30MWM746CbA28R3J2o6D+K4unE3gO3QFPq5FwGcryeq0AP27w
kuOA9WxwuCiGbKl6Ehbyo+9no56YIcqyTyOY4yAr6bV8NllqpZ4dxyQmVfURqPyC
uKZ5GACkEKjCh2JLZVWsHRD3lsQ+5sMipdaeAPBgRB8n63YXY9G/n4JSHG3P+BaG
vH0UYER+CLLkzdhmvVPsHpxWtGph0T5b6lHpGiphCHAiT4BWIVenZT8X+t0Zkf6I
7n/KBm/3R1cBtcrg94bGpgi16v0vlYAVXnQEDmfvaGkR0vPrVhjvp4qPrnXVoXw=
=yCLP
-----END PGP SIGNATURE-----