[Buildbot] #2902: EC2 spot instances fail on InvalidSpotInstanceRequestID.NotFound

[Buildbot trac]

#2902: EC2 spot instances fail on InvalidSpotInstanceRequestID.NotFound
---------------------+-----------------------
Reporter:  bgilbert  |      Owner:
    Type:  defect    |     Status:  new
Priority:  major     |  Milestone:  undecided
 Version:  0.8.9     |   Keywords:
---------------------+-----------------------
 After EC2LatentBuildSlave creates a spot request, it immediately queries
 it using `get_all_spot_instance_requests()`.  Since the EC2 API
 [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/query-api-
 troubleshooting.html#eventual-consistency uses eventual consistency], the
 query can fail (and does, fairly reliably) with
 `InvalidSpotInstanceRequestID.NotFound`.  EC2LatentBuildSlave treats this
 as a fatal error:

 {{{
 Traceback (most recent call last):
   File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
     self.__bootstrap_inner()
   File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
     self.run()
   File "/usr/lib/python2.7/threading.py", line 505, in run
     self.__target(*self.__args, **self.__kwargs)
 --- <exception caught here> ---
   File "/home/foo/env/local/lib/python2.7/site-
 packages/twisted/python/threadpool.py", line 196, in _worker
     result = context.call(ctx, function, *args, **kwargs)
   File "/home/foo/env/local/lib/python2.7/site-
 packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/home/foo/env/local/lib/python2.7/site-
 packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/home/foo/env/local/lib/python2.7/site-
 packages/buildbot/buildslave/ec2.py", line 362, in _request_spot_instance
     request = self._wait_for_request(reservations[0])
   File "/home/foo/env/local/lib/python2.7/site-
 packages/buildbot/buildslave/ec2.py", line 406, in _wait_for_request
     request_ids=[reservation.id])
   File "/home/foo/env/local/lib/python2.7/site-
 packages/boto/ec2/connection.py", line 1342, in
 get_all_spot_instance_requests
     [('item', SpotInstanceRequest)], verb='POST')
   File "/home/foo/env/local/lib/python2.7/site-
 packages/boto/connection.py", line 1166, in get_list
     raise self.ResponseError(response.status, response.reason, body)
 boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
 <?xml version="1.0" encoding="UTF-8"?>
 <Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The
 spot instance request ID 'sir-0231f26g' does not
 exist</Message></Error></Errors><RequestID>cade547a-bf90-482f-b44b-
 2a7f5b3f8c01</RequestID></Response>
 }}}

--
Ticket URL: <http://trac.buildbot.net/ticket/2902>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation