[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

Wed Oct 8 13:46:04 UTC 2014

In a previous reply, Charles wrote:

In Apache, I dropped the proxy buffer from the default of 8 kB to 512 byte.

I am assuming you are talking about the ProxyReceiveBufferSize. Am I
assuming right?

On Tue, Oct 7, 2014 at 11:24 AM, Francesco Di Mizio <
francescodimizio at gmail.com> wrote:

> Just found out there's a proxy server (Apache with proxy_mod) on the same
> Linux server running the buildbot.
>
> ProxyPass /global/ http://localhost:19811/
> ProxyPassReverse /global/ http://localhost:19811/
>
> with master.cfg containing the following:
>
> HTTP_PORT = 'tcp:19811:interface=127.0.0.1'
> EXTERNAL_URL = 'http://buildbot/global/'
>
> I've tried to set ProxyReceiveBufferSize to multiple values (512, 1024,
> 2048) and restarting Apache every time but nothing has changed. Logs still
> get stuck.
>
> Francesco
>
> On Mon, Oct 6, 2014 at 5:41 PM, Francesco Di Mizio <
> francescodimizio at gmail.com> wrote:
>
>> Sadly I can see the log being dumped as it runs.
>>
>> Francesco
>>
>> On Mon, Oct 6, 2014 at 5:25 PM, Dustin J. Mitchell <dustin at v.igoro.us>
>> wrote:
>>
>>> It could be responsible, yes.  One way to test is to run your command
>>> by hand, redirected to a file, and then tail -f that file.  If you
>>> don't see anything in the file until the command is complete, then
>>> buffering is the problem.
>>>
>>> Dustin
>>>
>>> On Mon, Oct 6, 2014 at 11:06 AM, Francesco Di Mizio
>>> <francescodimizio at gmail.com> wrote:
>>> > I came across a line searching on the web:
>>> >
>>> > you might want to turn off output buffering for python so that you can
>>> see
>>> > the results of your build steps as they happen. This is of course only
>>> > required if your build steps are calling python scripts, like I am. If
>>> you
>>> > are calling something else like make or devenv, then you really dont
>>> have
>>> > any control over the output buffering. You can turn off output
>>> buffering in
>>> > python scripts by setting an environment variable
>>> PYTHONUNBUFFERRED=TRUE
>>> >
>>> > Does it make any sense to you guys? I mainly call py scripts that in
>>> turn
>>> > can call exec, batch/bash scripts.It seems to fit.
>>> >
>>> >
>>> >
>>> > On Mon, Oct 6, 2014 at 4:44 PM, Francesco Di Mizio
>>> > <francescodimizio at gmail.com> wrote:
>>> >>
>>> >> And indeed I was wrong. I am not using lazylogfiles
>>> >>
>>> >> I am getting the 'logs dont show up until step stops' thingy from any
>>> >> step, even when just trying to open a stdio or sterr, not only from
>>> custom
>>> >> logs (i.e. by using logfiles={"triallog": trial_temp/test.log"}).
>>> >> Furthermore there's no proxy between me (or any pc on the network)
>>> and the
>>> >> buildbot.
>>> >> Not sure what additional info may be usedul to provide to help sort
>>> this
>>> >> out.
>>> >>
>>> >> Cheers,
>>> >>  Francesco
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Sat, Oct 4, 2014 at 4:23 PM, Francesco Di Mizio
>>> >> <francescodimizio at gmail.com> wrote:
>>> >>>
>>> >>> Well I dont remeber changing them. I only know tthey used to stream
>>> and
>>> >>> now they don't.
>>> >>> I might be wrong though, Next week I'll check the history and update
>>> >>> here.
>>> >>>
>>> >>> Thanks
>>> >>>
>>> >>> On Sat, Oct 4, 2014 at 3:37 PM, Dustin J. Mitchell <
>>> dustin at v.igoro.us>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hang on, I just noticed the thread subject.. If these are, indeed,
>>> >>>> configured as lazy logs then they shouldn't even show up on the
>>> master
>>> >>>> until the step stops.  In other words, lazy logs are not streamed,
>>> >>>> they are downloaded at step completion.
>>> >>>>
>>> >>>> Dustin
>>> >>>>
>>> >>>> On Fri, Oct 3, 2014 at 5:56 PM, Thomas Middeldorp
>>> >>>> <thomas at ranzer.geek.nz> wrote:
>>> >>>> > I can confirm "proxy_buffering off;" is what fixed this for me in
>>> 0.8.
>>> >>>> > In my
>>> >>>> > setup I have /waterfall behind a proxy_cache (you could add other
>>> >>>> > locations if
>>> >>>> > needed) and then / is a straight proxy_pass with buffering turned
>>> off.
>>> >>>> >
>>> >>>> > - Thomas
>>> >>>> >
>>> >>>> >
>>> >>>> > On 4/10/2014 1:46 a.m., Dustin J. Mitchell wrote:
>>> >>>> >> My googling is failing to find previous threads about the proxy
>>> issue
>>> >>>> >> and the nginx fix for it -- there's a parameter of some sort to
>>> turn
>>> >>>> >> off buffering, similar to what Charles has suggested.
>>> >>>> >>
>>> >>>> >> In 0.9.0, we're no longer using the technique of dripping out an
>>> HTTP
>>> >>>> >> response slowly, so the header change isn't necessary.  However,
>>> the
>>> >>>> >> header or at least a note about Apache and Nginx configuration
>>> in the
>>> >>>> >> docs, would be a good addition to the 'eight' branch.
>>> >>>> >>
>>> >>>> >> Dustin
>>> >>>> >>
>>> >>>> >> On Thu, Oct 2, 2014 at 11:10 PM, Charles Lepple <
>>> clepple at gmail.com>
>>> >>>> >> wrote:
>>> >>>> >>> Definitely not a 1:1 mapping, but in Apache, I dropped the proxy
>>> >>>> >>> buffer from the default of 8 kB to 512 bytes, and logs stream
>>> fine. The
>>> >>>> >>> nginx parameters look a bit more complicated:
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
>>> >>>> >>>
>>> >>>> >>> Here's the Apache parameter I tweaked:
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxyiobuffersize
>>> >>>> >>>
>>> >>>> >>> One thing that Buildbot could potentially do on the logs is add
>>> >>>> >>> "X-Accel-Buffering: no" for nginx (I think; unless I got it
>>> backwards).
>>> >>>> >>>
>>> >>>> >>> --
>>> >>>> >>> - Charles Lepple
>>> >>>> >>> http://ghz.cc/charles/
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> ------------------------------------------------------------------------------
>>> >>>> >>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>>> >>>> >>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>>> >>>> >>> Reports
>>> >>>> >>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>>> paper
>>> >>>> >>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>>> Analyzer
>>> >>>> >>>
>>> >>>> >>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>> >>>> >>> _______________________________________________
>>> >>>> >>> Buildbot-devel mailing list
>>> >>>> >>> Buildbot-devel at lists.sourceforge.net
>>> >>>> >>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> >>>> >>
>>> >>>> >>
>>> ------------------------------------------------------------------------------
>>> >>>> >> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>>> >>>> >> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>>> >>>> >> Reports
>>> >>>> >> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>>> paper
>>> >>>> >> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>>> Analyzer
>>> >>>> >>
>>> >>>> >>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>> >>>> >> _______________________________________________
>>> >>>> >> Buildbot-devel mailing list
>>> >>>> >> Buildbot-devel at lists.sourceforge.net
>>> >>>> >> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> ------------------------------------------------------------------------------
>>> >>>> > Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>>> >>>> > Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>>> >>>> > Reports
>>> >>>> > Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>>> paper
>>> >>>> > Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>>> Analyzer
>>> >>>> >
>>> >>>> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>> >>>> > _______________________________________________
>>> >>>> > Buildbot-devel mailing list
>>> >>>> > Buildbot-devel at lists.sourceforge.net
>>> >>>> > https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> >>>>
>>> >>>>
>>> >>>>
>>> ------------------------------------------------------------------------------
>>> >>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>>> >>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>>> Reports
>>> >>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
>>> >>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
>>> >>>>
>>> >>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>> >>>> _______________________________________________
>>> >>>> Buildbot-devel mailing list
>>> >>>> Buildbot-devel at lists.sourceforge.net
>>> >>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20141008/d2c4f6a3/attachment.html>