[Buildbot] #3490: re-enable foreign keys for sqlite

[Buildbot trac]

#3490: re-enable foreign keys for sqlite
------------------------+------------------------
Reporter:  tardyp       |       Owner:
    Type:  enhancement  |      Status:  new
Priority:  major        |   Milestone:  undecided
 Version:  master       |  Resolution:
Keywords:               |
------------------------+------------------------

Comment (by delanne):

 = context =

  - DB backend: sqlite
  - SingleBranchScheduler with treestabletimer > 0
  - updated my buildbot master to the HEAD of the master branch (which was
 2d11233d9b87052355c8d228ab30de44cf8162f8).

 after doing a git bisect, 68b8542f2f86dcc0c10ae475fa24801083261369 was the
 changeset which introduce the issue.

 = symptoms =

  - No build scheduled

 = investigations =
  - in schedulers/basic.py
 {{{
     @util.deferredLocked('_stable_timers_lock')
     @defer.inlineCallbacks
     def stableTimerFired(self, timer_name):
         # delete this now-fired timer, if the service has already been
 stopped
         # then just bail out
         if not self._stable_timers.pop(timer_name, None):
             return

         classifications = \
             yield self.getChangeClassificationsForTimer(self.objectid,
                                                         timer_name)

         # just in case: databases do weird things sometimes!
         if not classifications:  # pragma: no cover
             return

         changeids = sorted(classifications.keys())
         yield self.addBuildsetForChanges(reason=self.reason,
                                          changeids=changeids)

         max_changeid = changeids[-1]  # (changeids are sorted)
         yield self.master.db.schedulers.flushChangeClassifications(
             self.objectid, less_than=max_changeid + 1)
 }}}
 the function self.getChangeClassificationsForTimer returns an empty dict,
 as a consequence the buildset is not created.

 = discussion on irc =

 Here's a copy/paste of the discussion we had on IRC about this issue:
 {{{
 <delanne> I think there's a regression in the master branch, since I
 updated my buildbot from 605ed7bdb8686a710a4b7e7a51e9d78f03e346c1 to HEAD,
 my builders aren't triggered ...

 <tardyp> delanne: weird. There is no change on the scheduler
 <tardyp> the only big thing is the slaveworker
 <delanne> well, I did a git bisect and the result is:
 <delanne> with 60e6ef4496090a649d8cd910b9719e54300d88f3, everything works
 as expected
 <delanne> and the commit after, my builders aren't triggered
 <delanne> so, 68b8542f2f86dcc0c10ae475fa24801083261369 seems to introduce
 the regression
 <delanne> I continue to investigate ...
 <tardyp> can you git bisect?
 <tardyp> ah sorry
 <tardyp> you did it :-}
 <tardyp> I read too fast
 <tardyp> 68b8542f2f86dcc0c10ae475fa24801083261369 is a merge commit
 <tardyp> I also experience some issue with schedulers
 <delanne> well,
 <tardyp> but it is not always. some works some dont (always the same
 project)
 <delanne> if it helps
 <delanne> schedulers/basic.py line 191
 <delanne> yield self.getChangeClassificationsForTimer(self.objectid,
 <delanne>
 timer_name)
 <delanne> returns an empty dict in my case
 <delanne> so :
 <delanne>         if not classifications:  # pragma: no cover
 <delanne>             return
 <delanne> as a consequence, the buildset is not added
 <tardyp> hmm
 <delanne> (I use a stableTimer in my config)
 <delanne> and I use  SingleBranchScheduler
 <tardyp> you are sqlite?
 <delanne> yes
 <tardyp> bug3472 is adding foreign key checks enforcements
 <delanne> for the SingleBranchScheduler, the function
 getChangeClassifications use the db layer directly, and not the data
 layer. is this expected ?
 <tardyp> I expect it to be a legacy issue
 <delanne> I deleted my sqlite file, and recreate it with upgrade-master
 <tardyp> you copied you sqlitefile!
 <tardyp> so that we could postmortem?
 <delanne> I first upgraded my sql file ... in order to see if that solves
 the issue, but no
 <delanne> so I deleted it, and recreated it from scratch
 <delanne> but, the issue is still there.
 <tardyp> ok
 <tardyp> in enginestrategy.py +118
 <tardyp> can you disable fk quick?
 <tardyp> and see if this fixes?
 <delanne> ok
 <tardyp> as nobody is using pg with bb nine, we may have some fk bugs
 <delanne> it fixes
 <delanne> thanks
 <delanne> to be clear, what I did is:
 <delanne> connection.execute('pragma foreign_keys=OFF')
 <tardyp> You can use this as a workaround. but the bug is probably
 elsewere
 <tardyp> either its a bug in sqlite foreigh key
 <tardyp> or its a bug in our code, and other people will see it with fk
 enabled db

 }}}

--
Ticket URL: <http://trac.buildbot.net/ticket/3490#comment:2>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation