[Buildbot-devel] 504 timeouts & slow responses with nginx proxy

[Edward Armes]

The only differences I could think of the SSH connection makes is that 
the connection is already established so the timeout rules for nginx 
proxy don't apply (according to proxy timeout documentation). I would 
guess as well that also reduces the load on the BuildBot server as well 
as it doesn't have to establish anything below the application layer as 
it's using an already existing reliable transport, how this applies for 
data sending not a clue as it would depend on a huge number of factors

Edward

On 10/04/2015 00:21, Vitali Lovich wrote:
> Yup.  I know in theory that's the problem.  The challenge is that if I 
> tunnel the port over SSH, I'm able to access the server without any of 
> the issues I'm seeing (and it's just faster).  Of course it could be 
> just observation bias as I haven't yet come up with a way to benchmark.
>
> I have turned on twisted profiling so I'm going to take a look with 
> QCacheGrind at the data to see if there's any buildbot bottlenecks.
>
> I'll try increasing the proxy read/write timeout on the off chance 
> that it is the issue, but I usually only get the 504 around 30s after 
> the browser has tried the page which seems to line-up with the overall 
> client request timeout I have set in Nginx.
>
> On Apr 9, 2015, at 4:14 PM, Edward Armes <edward.armes at gmail.com 
> <mailto:edward.armes at gmail.com>> wrote:
>
>> Hi Vitali,
>>
>> If you are not aware a 504 means Gateway Timeout Error. This means 
>> that you nginx server is not getting a response in time from the 
>> BuildBot server itself.
>>
>> Why this happens however could be for multiple reasons including:-
>>
>>   * The buildbot server is taken too long to respond
>>   * The response/request got lost in the network
>>   * The nginx server didn't finish processing the proxy result before
>>     forwarding it on and dropped the request (unlikely).
>>
>> I would suspect it's the first one as I've noticed if the buildbot 
>> master is busy handling requests it can sometimes take a while to 
>> serve a page when it's loading a page with a lot of dynamic data 
>> (i.e. waterfall view).
>>
>> A simple fix might be for while the web status is building the 
>> response it might be an idea to send a 202 code back if the request 
>> takes a long time (to delay the 504 timeout), or (if this is 
>> possible) to increase the wait time for proxy directive from within 
>> nginx.
>>
> Can you expand on this?  How would I do it?
>>
>> Hope that helps
>>
>> Edward
>>
>> On 09/04/2015 23:16, Vitali Lovich wrote:
>>> Hi,
>>>
>>> I’m wondering if anyone has had any experience with nginx sitting as a proxy server in front of buildbot.
>>> I’m getting random 504 timeouts which I *think* are something to do with nginx as opposed to buildbot itself,
>>> but I’m not sure how to confirm/triage this issue.
>>>
>>> Any tips would be great.
>>>
>>> My nginx configuration just sets proxy_pass to the buildbot instance.
>>> proxy_buffering is off & gzip proxying is on.
>>>
>>> Thanks,
>>> Vitali
>>> ------------------------------------------------------------------------------
>>> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
>>> Develop your own process in accordance with the BPMN 2 standard
>>> Learn Process modeling best practices with Bonita BPM through live exercises
>>> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
>>> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150410/995e06c6/attachment.html>