[Buildbot-commits] [Buildbot] #1777: Dependent scheduler forgets who to trigger

[Buildbot]

#1777: Dependent scheduler forgets who to trigger
-------------------------------+--------------------
Reporter:  armenzg             |       Owner:
    Type:  defect              |      Status:  new
Priority:  major               |   Milestone:  0.8.4
 Version:  0.8.3p1             |  Resolution:
Keywords:  database, reconfig  |
-------------------------------+--------------------
Changes (by dustin):

 * keywords:   => database, reconfig
 * milestone:  undecided => 0.8.4


Old description:

> We triggered a release on Friday and in between then and today 2
> reconfigurations happened.[[BR]]
> On Monday the "updates" builder [1] got triggered by an ftpPoller [2] and
> it was supposed to trigger the "update_verify" builders [3].
>
> The problem is that a reconfigure happened before that and it made the
> Dependent scheduler to forget who to trigger.
>
> We could switch to trigger steps but then it prevents us from doing a
> "force build" and '''not''' have any dependent jobs to be triggered.
>
> The release was triggered at 14:32 on Friday.[[BR]]
> The updates builder was triggered at 13:01 on Monday.[[BR]]
> 2 reconfigures happened in between.
>
> Could this be the place where the factory is considered to be changed and
> the memory loss happen?
>
> 2011-01-21 21:56:03-0800 [-] updating builder release-
> mozilla-1.9.2-updates: factory changed
>   nextSlave changed from <function _nextFastReservedSlave at 0x1953e64c>
> to <function _nextFastReservedSlave at 0x1d747b1c>[[BR]]
> 2011-01-21 21:56:03-0800 [-] consumeTheSoulOfYourPredecessor: <Builder
> ''release-mozilla-1.9.2-updates'' at 500572364> feeding upon <Builder
> ''release-mozilla-1.9.2-updates'' at 240793324>
>
> Reconfigs on masters listed chronologically (first two happened before
> "updates" job got triggered):[[BR]]
> twistd.log.228:2011-01-21 21:56:05-0800 [-] configuration update
> started[[BR]]
> twistd.log.228:2011-01-21 21:56:37-0800 [-] configuration update
> complete[[BR]]
> twistd.log.65:2011-01-24 12:48:34-0800 [-] configuration update
> started[[BR]]
> twistd.log.65:2011-01-24 12:48:55-0800 [-] configuration update
> complete[[BR]]
> twistd.log.52:2011-01-24 16:29:40-0800 [-] configuration update
> started[[BR]]
> twistd.log.52:2011-01-24 16:31:24-0800 [-] configuration update
> complete[[BR]]
>

> [1] Updates scheduler -
> http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l313
> [[BR]]
> [2] ftpPoller -
> http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l212
> [[BR]]
> [3] Update_verify builder -
> http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l327
> [[BR]]

New description:

 We triggered a release on Friday and in between then and today 2
 reconfigurations happened. On Monday the "updates" builder [1] got
 triggered by an ftpPoller [2] and it was supposed to trigger the
 "update_verify" builders [3].

 The problem is that a reconfigure happened before that and it made the
 Dependent scheduler to forget who to trigger.

 We could switch to trigger steps but then it prevents us from doing a
 "force build" and '''not''' have any dependent jobs to be triggered.

 The release was triggered at 14:32 on Friday.
 The updates builder was triggered at 13:01 on Monday.
 2 reconfigures happened in between.

 Could this be the place where the factory is considered to be changed and
 the memory loss happen?

 {{{
 2011-01-21 21:56:03-0800 [-] updating builder release-
 mozilla-1.9.2-updates: factory changed
   nextSlave changed from <function _nextFastReservedSlave at 0x1953e64c>
 to <function _nextFastReservedSlave at 0x1d747b1c>[[BR]]
 2011-01-21 21:56:03-0800 [-] consumeTheSoulOfYourPredecessor: <Builder
 ''release-mozilla-1.9.2-updates'' at 500572364> feeding upon <Builder
 ''release-mozilla-1.9.2-updates'' at 240793324>
 }}}

 Reconfigs on masters listed chronologically (first two happened before
 "updates" job got triggered):
 *twistd.log.228:2011-01-21 21:56:05-0800 [-] configuration update started
 *twistd.log.228:2011-01-21 21:56:37-0800 [-] configuration update complete
 *twistd.log.65:2011-01-24 12:48:34-0800 [-] configuration update started
 *twistd.log.65:2011-01-24 12:48:55-0800 [-] configuration update complete
 *twistd.log.52:2011-01-24 16:29:40-0800 [-] configuration update started
 *twistd.log.52:2011-01-24 16:31:24-0800 [-] configuration update complete


 [1] Updates scheduler -
 http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l313
 [[BR]]
 [2] ftpPoller -
 http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l212
 [[BR]]
 [3] Update_verify builder -
 http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l327
 [[BR]]

--

Comment:

 I don't think that particular log snippet is related, but in any case the
 schedulers are all being rewritten.  Once Dependent is rewritten, I'll
 double-check that this works in 0.8.4.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1777#comment:1>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation