[Buildbot-devel] 0.8.9, step_status.urls persistence

Dustin J. Mitchell dustin at v.igoro.us
Sat Aug 30 22:36:44 UTC 2014 

Yuck, this is one of the messy bits of saving things in pickles!  And yes,
I expect this problem to exist in 0.8.8 as well, which is the release where
the trigger URLs were added, I think.

You can serialize again by calling buildstatus.saveYourself(), although
that has a substantial cost in terms of CPU and IO.  It won't be a problem
unless you've got a busy master, but it's something to be aware of.

Dustin


On Mon, Aug 25, 2014 at 7:32 AM, Thierry at spoludo <thierry at spoludo.com>
wrote:

> ‎I've added some traces to understand the issue.
> Here they are :
> 2014-08-25 13:40:56+0300 [-]  step 'trigger' complete: success
>
> 2014-08-25 13:40:56+0300 [-]  <Build zzz>: build finished
> 2014-08-25 13:40:56+0300 [-] SAVE build 29
> 2014-08-25 13:40:56+0300 [-] SAVE step 0 trigger,
> 2014-08-25 13:40:56+0300 [-]  setting expectations for next time
> 2014-08-25 13:40:56+0300 [-] new expectations:27.0128679276 seconds
> 2014-08-25 13:40:56+0300 [-] releaseLocks(<BuildSlave 'build_all_slave'>):
> []
> 2014-08-25 13:40:56+0300 [-] ADD URL xxx #29
> 2014-08-25 13:40:56+0300 [-] ADD URL yyy #29
>
>
> So I use a triggerable on a slave 1 (build_all_slave), that triggers 2
> builds on another slave. The URL I was talking of are these children builds
> url.
> It appears that the trigger step is saved before the url are added because
> the build is locked. But after the build is unlocked, the url are added
> ‎but it doesn't trigger the serialization of the updated build status.
> How to force the serialization after adding the urls? Or how to solve‎
> this in a cleaner way?
>
> ‎The traces are obtained with the following modifications:
> in master/buildbot/status/buildstep.py:
>
>     def addURL(self, name, url):
>         log.msg("ADD URL %s " % name)
>         self.urls[name] = url
>
>
>
> in master/buildbot/status/build.py:
>     def saveYourself(self):
>         filename = os.path.join(self.builder.basedir, "%d" % self.number)
>         if os.path.isdir(filename):
>             # leftover from 0.5.0, which stored builds in directories
>             shutil.rmtree(filename, ignore_errors=True)
>         tmpfilename = filename + ".tmp"
>         try:
>             with open(tmpfilename, "wb") as f:
>                 c = 0
>                 log.msg("SAVE build %d" % self.number)
>                 for s in self.steps:
>                     steps = 'urls '.join(s.urls)
>                     log.msg("SAVE step %d %s, %s" % (c,s.name, steps))
>                     c += 1
>                 dump(self, f, -1)‎
> I guess the same issue might span over several releases.
>
>
>
>   *From: *Vasily
> *Sent: *vendredi 22 août 2014 09:17
> *To: *Thierry at spoludo
> *Reply To: *vasslitvinov at pisem.net
> *Cc: *buildbot-devel list; Dustin J. Mitchell
> *Subject: *Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>
> I'm afraid I cannot help any further since I don't use 0.8.9 yet, thus I
> probably don't understand what those urls are.
> Let's hope someone else on the list could give you a hand.
>
> Thanks,
> Vasily
> 21 авг. 2014 г. 19:06 пользователь "Thierry at spoludo" <thierry at spoludo.com>
> написал:
>
>>  Thank you, I'm not so familiar with python. So I've then used in the end
>> pickle tools to see the fullcontent of the build file. I see that the urls
>> are not there.
>> There is a build step status for the triggered step, but its urls
>> property is an empty dict.
>> The persistence version is 4.
>>  Not sure when they disappear, because I think this is where they should
>> looking at the steps.Triggeted.create_summary code
>> ‎And looking at what the waterfall looks like when it is initially
>> created before getting lost before serializing with pickle
>>
>>    *From: *Vasily
>> *Sent: *jeudi 21 août 2014 15:43
>> *To: *Thierry at spoludo
>> *Reply To: *vasslitvinov at pisem.net
>> *Cc: *Dustin J. Mitchell; buildbot-devel list
>> *Subject: *Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>>
>> Hi Thierry,
>>
>> If you're familiar with Python, you could just unpickle those files and
>> poke at their contents using interactive Python prompt.
>>
>> Thanks,
>> Vasily
>> 21 авг. 2014 г. 14:18 пользователь "Thierry at spoludo" <thierry at spoludo.com>
>> написал:
>>
>>> Well, for now I'm only using the releases provided by the distributions,
>>> and it looks like I have to work on 0.8.9.
>>> I see that the builders in the master have files which are numbered
>>> accordingly to get build numbers.
>>> I suspect that the data used by the waterfall are serialized in it,
>>> probably using some twisted mechanism. Is there a way to dump them as text
>>> after deserializing? This could tell me if the urls were saved or not.
>>>
>>> Regards
>>>
>>>   Original Message
>>> From: Dustin J. Mitchell
>>> Sent: mercredi 20 août 2014 01:19
>>> To: Thierry at spoludo
>>> Cc: BuildBot Devel
>>> Subject: Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>>>
>>> This may be an issue with the old 0.8.x status interface. In the new
>>> AngularJS-based interface, we have something even better than links:
>>> you can "twirldown" to see the triggered builds nested within the
>>> triggering build.
>>>
>>> Dustin
>>>
>>> On Tue, Aug 19, 2014 at 10:08 AM, Thierry at spoludo <thierry at spoludo.com>
>>> wrote:
>>> >
>>> > I observe that a steps.Trigger creates some urls for the build names it
>>> > triggers, that are listed in the Waterfall StepBox‎.
>>> > However these URLS don't seem to persist well, because StepBoxes
>>> created
>>> > after a restart for example don't expose them.
>>> > I'm not familiar with twister persistence framework, so what could
>>> explain
>>> > that the BuildStepStatus.urls disappear?
>>> > I see that db.steps
>>> > StepConnectorComponent‎ shall store and retrieve them.
>>> > Is this a known issue?
>>> >
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> >
>>> > _______________________________________________
>>> > Buildbot-devel mailing list
>>> > Buildbot-devel at lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Slashdot TV.
>>> Video for Nerds.  Stuff that matters.
>>> http://tv.slashdot.org/
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>
>>
>>
>
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140830/c3529a75/attachment.html>

More information about the devel mailing list