[Buildbot-devel] 504 timeouts & slow responses with nginx proxy

Sat Apr 18 03:28:58 UTC 2015

OK - so I opened http://trac.buildbot.net/ticket/3249#ticket <http://trac.buildbot.net/ticket/3249#ticket> in the hopes that it helps other people.
Not sure what next steps to diagnose this would be.

It’s super-reproducible.  Just doing curl http://localhost:8010 <http://localhost:8010/> in a loop would sporadically hang.
No output in any buildbot logs (twistd.log nor http.log).

The workaround we found was to switch to unix domain sockets.

I’m happy to help diagnose this if there’s anything else I can do.

-Vitali

> On Apr 10, 2015, at 2:14 AM, Pierre Tardy <tardyp at gmail.com> wrote:
> 
> Not sure if you are talking about nine but there is a sample config for nginx in the doc.
> 
> http://buildbot.readthedocs.org/en/latest/manual/cfg-www.html#reverse-proxy-configuration <http://buildbot.readthedocs.org/en/latest/manual/cfg-www.html#reverse-proxy-configuration>
> Time out can happen with web sockets add they are long lasting
> 
> 
> Le ven. 10 avr. 2015 01:30, Edward Armes <edward.armes at gmail.com <mailto:edward.armes at gmail.com>> a écrit :
> The only differences I could think of the SSH connection makes is that the connection is already established so the timeout rules for nginx proxy don't apply (according to proxy timeout documentation). I would guess as well that also reduces the load on the BuildBot server as well as it doesn't have to establish anything below the application layer as it's using an already existing reliable transport, how this applies for data sending not a clue as it would depend on a huge number of factors
> 
> 
> Edward
> 
> 
> On 10/04/2015 00:21, Vitali Lovich wrote:
>> Yup.  I know in theory that's the problem.  The challenge is that if I tunnel the port over SSH, I'm able to access the server without any of the issues I'm seeing (and it's just faster).  Of course it could be just observation bias as I haven't yet come up with a way to benchmark.
>> 
>> I have turned on twisted profiling so I'm going to take a look with QCacheGrind at the data to see if there's any buildbot bottlenecks.
>> 
>> I'll try increasing the proxy read/write timeout on the off chance that it is the issue, but I usually only get the 504 around 30s after the browser has tried the page which seems to line-up with the overall client request timeout I have set in Nginx.
>> 
>> On Apr 9, 2015, at 4:14 PM, Edward Armes <edward.armes at gmail.com <mailto:edward.armes at gmail.com>> wrote:
>> 
>>> Hi Vitali,
>>> 
>>> If you are not aware a 504 means Gateway Timeout Error. This means that you nginx server is not getting a response in time from the BuildBot server itself.
>>> 
>>> Why this happens however could be for multiple reasons including:-
>>> The buildbot server is taken too long to respond
>>> The response/request got lost in the network
>>> The nginx server didn't finish processing the proxy result before forwarding it on and dropped the request (unlikely).
>>> I would suspect it's the first one as I've noticed if the buildbot master is busy handling requests it can sometimes take a while to serve a page when it's loading a page with a lot of dynamic data (i.e. waterfall view).
>>> 
>>> A simple fix might be for while the web status is building the response it might be an idea to send a 202 code back if the request takes a long time (to delay the 504 timeout), or (if this is possible) to increase the wait time for proxy directive from within nginx.
>> Can you expand on this?  How would I do it?
>>> 
>>> Hope that helps
>>> Edward
>>> On 09/04/2015 23:16, Vitali Lovich wrote:
>>>> Hi,
>>>> 
>>>> I’m wondering if anyone has had any experience with nginx sitting as a proxy server in front of buildbot.
>>>> I’m getting random 504 timeouts which I *think* are something to do with nginx as opposed to buildbot itself,
>>>> but I’m not sure how to confirm/triage this issue.
>>>> 
>>>> Any tips would be great.
>>>> 
>>>> My nginx configuration just sets proxy_pass to the buildbot instance.
>>>> proxy_buffering is off & gzip proxying is on.
>>>> 
>>>> Thanks,
>>>> Vitali
>>>> ------------------------------------------------------------------------------
>>>> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
>>>> Develop your own process in accordance with the BPMN 2 standard
>>>> Learn Process modeling best practices with Bonita BPM through live exercises
>>>> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual <http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual>- event?utm_
>>>> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net <mailto:Buildbot-devel at lists.sourceforge.net>
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel <https://lists.sourceforge.net/lists/listinfo/buildbot-devel>
>>> 
> 
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live exercises
> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- <http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-> event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF_______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net <mailto:Buildbot-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel <https://lists.sourceforge.net/lists/listinfo/buildbot-devel>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150417/50d8fa58/attachment.html>