[Buildbot-commits] [Buildbot] #2454: SiGHUP doesn't always work

[Buildbot trac]

#2454: SiGHUP doesn't always work
----------------------------+----------------------
Reporter:  virgilg          |       Owner:
    Type:  support-request  |      Status:  new
Priority:  major            |   Milestone:  ongoing
 Version:  0.8.7p1          |  Resolution:
Keywords:                   |
----------------------------+----------------------

Comment (by virgilg):

 I can repro this as follows:

 1) save the attached script as e.g. test.py
 2) comment and uncomment line 12: time.sleep(1)

 With time.sleep(1) uncommented, one cannot deliver a SIG<MUMBLE> to the
 process, since the MainThread is too busy inside the while True: loop and
 python is not really multithreaded.

 With time.sleep(1) commented out, handler() will get a chance to run.


 In our case, buildbot's MainThread is in this loop:
 file: buildbot/status/persistent_queue.py
 224   def _findNext(self, id):
         while True:
             path = os.path.join(self.path, str(id))
             if os.path.isfile(path):
                 return id
             id += 1
         return None

 I see it process a ton of "events" that never finish:
 events_company.com/11305023
 Where do these events come from? What generates them?
 events_company.com/state is not the droid I'm looking for, is it?

 The other 6 threads are all stuck in threading.py:

 > thread
 List of active threads known to the debugger:

     No    Tid  Name             State
 -----------------------------------------------
  >   0  139697960785664  MainThread       waiting at break point
      1  139697837831936  PoolThread-DBThreadPool-1  running
      2  139697149482752  PoolThread-DBThreadPool-2  running
      3  139697787475712  PoolThread-twisted.internet.reactor-2  running
      4  139697132697344  PoolThread-twisted.internet.reactor-1  running
      5  139697111598848  PoolThread-DBThreadPool-5  running
      6  139697119991552  PoolThread-DBThreadPool-4  running
      7  139697141090048  PoolThread-DBThreadPool-3  running

 threading.py
 242        try:    # restore state no matter what (e.g.,
 KeyboardInterrupt)
             if timeout is None:
                 waiter.acquire()

 So therefore, SIGHUP will never be intercepted in the MainThread because
 of how tight st = os.stat(path) is in the above while True: loop.
 If we "wait" sending the SIGHUP until the queue has been processed, then
 it will be caught and honored.

 I think deferring the queue processing on a separate thread and leaving
 MainThread free to catch SIG<MUMBLE> would be a good thing to do here.

 But the real question is still unanswered: what generates those events and
 why aren't they eventually "consumed"? At what point is the queue empty?

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2454#comment:12>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation