[Buildbot-devel] 0.8.9, step_status.urls persistence

Dustin J. Mitchell dustin at v.igoro.us
Mon Sep 1 17:13:13 UTC 2014 

That's a good fix - nice work!  Do you want to make a pull request against
the `eight` branch?
  https://github.com/buildbot/buildbot/tree/eight

No need for a Trac bug in that case.

Dustin


On Mon, Sep 1, 2014 at 3:34 AM, Thierry at spoludo <thierry at spoludo.com> wrote:

> In the end, I've locally fixed it with the patch below. But I'm not sure
> this is the best solution.
> A second call to self.end(result) in the callback is useless as it already
> called self.end(result) while returning from self.start() given the
> current control flow, and a second call to end doesn't call anew finished.
> I haven't tried keeping the callback and returning just after creating it,
> but without calling self.end(result), in case it's worth returning early
> from start, because it looks like end shall be called on each exit path of
> the start method. ‎
> I wanted to add an entry in the bug tracker, but got registration issues,
> may retry later
>
> diff --git a/master/buildbot/steps/trigger.py
> b/master/buildbot/steps/trigger.py
> index 1cd4726..68da1d4 100644
> --- a/master/buildbot/steps/trigger.py
> +++ b/master/buildbot/steps/trigger.py
> @@ -208,11 +208,9 @@ class Trigger(LoggingBuildStep):
>                              url = master.status.getURLForBuild(bn, num)
>                              self.step_status.addURL("%s #%d" % (bn, num),
> url)
>
> -                return self.end(result)
> -
>              builddicts = [master.db.builds.getBuildsForRequest(br) for br
> in br
> -            dl = defer.DeferredList(builddicts, consumeErrors=1)
> -            dl.addCallback(add_links)
> +            res = yield defer.DeferredList(builddicts, consumeErrors=1)
> +            add_links(res)
>
>          self.end(result)
>          return
>
>     Original Message
>   *From: *Dustin J. Mitchell
> *Sent: *dimanche 31 août 2014 01:36
> *To: *Thierry at spoludo
> *Cc: *BuildBot Devel
> *Subject: *Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>
> Yuck, this is one of the messy bits of saving things in pickles!  And yes,
> I expect this problem to exist in 0.8.8 as well, which is the release where
> the trigger URLs were added, I think.
>
> You can serialize again by calling buildstatus.saveYourself(), although
> that has a substantial cost in terms of CPU and IO.  It won't be a problem
> unless you've got a busy master, but it's something to be aware of.
>
> Dustin
>
>
> On Mon, Aug 25, 2014 at 7:32 AM, Thierry at spoludo <thierry at spoludo.com>
> wrote:
>
>> ‎I've added some traces to understand the issue.
>> Here they are :
>> 2014-08-25 13:40:56+0300 [-]  step 'trigger' complete: success
>>
>> 2014-08-25 13:40:56+0300 [-]  <Build zzz>: build finished
>> 2014-08-25 13:40:56+0300 [-] SAVE build 29
>> 2014-08-25 13:40:56+0300 [-] SAVE step 0 trigger,
>> 2014-08-25 13:40:56+0300 [-]  setting expectations for next time
>> 2014-08-25 13:40:56+0300 [-] new expectations:27.0128679276 seconds
>> 2014-08-25 13:40:56+0300 [-] releaseLocks(<BuildSlave
>> 'build_all_slave'>): []
>> 2014-08-25 13:40:56+0300 [-] ADD URL xxx #29
>> 2014-08-25 13:40:56+0300 [-] ADD URL yyy #29
>>
>>
>> So I use a triggerable on a slave 1 (build_all_slave), that triggers 2
>> builds on another slave. The URL I was talking of are these children builds
>> url.
>> It appears that the trigger step is saved before the url are added
>> because the build is locked. But after the build is unlocked, the url are
>> added ‎but it doesn't trigger the serialization of the updated build status.
>> How to force the serialization after adding the urls? Or how to solve‎
>> this in a cleaner way?
>>
>>  ‎The traces are obtained with the following modifications:
>> in master/buildbot/status/buildstep.py:
>>
>>     def addURL(self, name, url):
>>         log.msg("ADD URL %s " % name)
>>         self.urls[name] = url
>>
>>
>>
>> in master/buildbot/status/build.py:
>>     def saveYourself(self):
>>         filename = os.path.join(self.builder.basedir, "%d" % self.number)
>>         if os.path.isdir(filename):
>>             # leftover from 0.5.0, which stored builds in directories
>>             shutil.rmtree(filename, ignore_errors=True)
>>         tmpfilename = filename + ".tmp"
>>         try:
>>             with open(tmpfilename, "wb") as f:
>>                 c = 0
>>                 log.msg("SAVE build %d" % self.number)
>>                 for s in self.steps:
>>                     steps = 'urls '.join(s.urls)
>>                     log.msg("SAVE step %d %s, %s" % (c,s.name, steps))
>>                     c += 1
>>                 dump(self, f, -1)‎
>> I guess the same issue might span over several releases.
>>
>>
>>
>>    *From: *Vasily
>> *Sent: *vendredi 22 août 2014 09:17
>> *To: *Thierry at spoludo
>> *Reply To: *vasslitvinov at pisem.net
>> *Cc: *buildbot-devel list; Dustin J. Mitchell
>> *Subject: *Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>>
>> I'm afraid I cannot help any further since I don't use 0.8.9 yet, thus I
>> probably don't understand what those urls are.
>> Let's hope someone else on the list could give you a hand.
>>
>> Thanks,
>> Vasily
>> 21 авг. 2014 г. 19:06 пользователь "Thierry at spoludo" <thierry at spoludo.com>
>> написал:
>>
>>>  Thank you, I'm not so familiar with python. So I've then used in the
>>> end pickle tools to see the fullcontent of the build file. I see that the
>>> urls are not there.
>>> There is a build step status for the triggered step, but its urls
>>> property is an empty dict.
>>> The persistence version is 4.
>>>  Not sure when they disappear, because I think this is where they should
>>> looking at the steps.Triggeted.create_summary code
>>> ‎And looking at what the waterfall looks like when it is initially
>>> created before getting lost before serializing with pickle
>>>
>>>    *From: *Vasily
>>> *Sent: *jeudi 21 août 2014 15:43
>>> *To: *Thierry at spoludo
>>> *Reply To: *vasslitvinov at pisem.net
>>> *Cc: *Dustin J. Mitchell; buildbot-devel list
>>> *Subject: *Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>>>
>>> Hi Thierry,
>>>
>>> If you're familiar with Python, you could just unpickle those files and
>>> poke at their contents using interactive Python prompt.
>>>
>>> Thanks,
>>> Vasily
>>> 21 авг. 2014 г. 14:18 пользователь "Thierry at spoludo" <
>>> thierry at spoludo.com> написал:
>>>
>>>> Well, for now I'm only using the releases provided by the
>>>> distributions, and it looks like I have to work on 0.8.9.
>>>> I see that the builders in the master have files which are numbered
>>>> accordingly to get build numbers.
>>>> I suspect that the data used by the waterfall are serialized in it,
>>>> probably using some twisted mechanism. Is there a way to dump them as text
>>>> after deserializing? This could tell me if the urls were saved or not.
>>>>
>>>> Regards
>>>>
>>>>   Original Message
>>>> From: Dustin J. Mitchell
>>>> Sent: mercredi 20 août 2014 01:19
>>>> To: Thierry at spoludo
>>>> Cc: BuildBot Devel
>>>> Subject: Re: [Buildbot-devel] 0.8.9, step_status.urls persistence
>>>>
>>>> This may be an issue with the old 0.8.x status interface. In the new
>>>> AngularJS-based interface, we have something even better than links:
>>>> you can "twirldown" to see the triggered builds nested within the
>>>> triggering build.
>>>>
>>>> Dustin
>>>>
>>>> On Tue, Aug 19, 2014 at 10:08 AM, Thierry at spoludo <thierry at spoludo.com>
>>>> wrote:
>>>> >
>>>> > I observe that a steps.Trigger creates some urls for the build names
>>>> it
>>>> > triggers, that are listed in the Waterfall StepBox‎.
>>>> > However these URLS don't seem to persist well, because StepBoxes
>>>> created
>>>> > after a restart for example don't expose them.
>>>> > I'm not familiar with twister persistence framework, so what could
>>>> explain
>>>> > that the BuildStepStatus.urls disappear?
>>>> > I see that db.steps
>>>> > StepConnectorComponent‎ shall store and retrieve them.
>>>> > Is this a known issue?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> >
>>>> > _______________________________________________
>>>> > Buildbot-devel mailing list
>>>> > Buildbot-devel at lists.sourceforge.net
>>>> > https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>> >
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Slashdot TV.
>>>> Video for Nerds.  Stuff that matters.
>>>> http://tv.slashdot.org/
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>
>>>
>>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Slashdot TV.
>> Video for Nerds.  Stuff that matters.
>> http://tv.slashdot.org/
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140901/1d013bcd/attachment.html>

More information about the devel mailing list