[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

Tue Oct 7 09:24:30 UTC 2014

Just found out there's a proxy server (Apache with proxy_mod) on the same
Linux server running the buildbot.

ProxyPass /global/ http://localhost:19811/
ProxyPassReverse /global/ http://localhost:19811/

with master.cfg containing the following:

HTTP_PORT = 'tcp:19811:interface=127.0.0.1'
EXTERNAL_URL = 'http://buildbot/global/'

I've tried to set ProxyReceiveBufferSize to multiple values (512, 1024,
2048) and restarting Apache every time but nothing has changed. Logs still
get stuck.

Francesco

On Mon, Oct 6, 2014 at 5:41 PM, Francesco Di Mizio <
francescodimizio at gmail.com> wrote:

> Sadly I can see the log being dumped as it runs.
>
> Francesco
>
> On Mon, Oct 6, 2014 at 5:25 PM, Dustin J. Mitchell <dustin at v.igoro.us>
> wrote:
>
>> It could be responsible, yes.  One way to test is to run your command
>> by hand, redirected to a file, and then tail -f that file.  If you
>> don't see anything in the file until the command is complete, then
>> buffering is the problem.
>>
>> Dustin
>>
>> On Mon, Oct 6, 2014 at 11:06 AM, Francesco Di Mizio
>> <francescodimizio at gmail.com> wrote:
>> > I came across a line searching on the web:
>> >
>> > you might want to turn off output buffering for python so that you can
>> see
>> > the results of your build steps as they happen. This is of course only
>> > required if your build steps are calling python scripts, like I am. If
>> you
>> > are calling something else like make or devenv, then you really dont
>> have
>> > any control over the output buffering. You can turn off output
>> buffering in
>> > python scripts by setting an environment variable PYTHONUNBUFFERRED=TRUE
>> >
>> > Does it make any sense to you guys? I mainly call py scripts that in
>> turn
>> > can call exec, batch/bash scripts.It seems to fit.
>> >
>> >
>> >
>> > On Mon, Oct 6, 2014 at 4:44 PM, Francesco Di Mizio
>> > <francescodimizio at gmail.com> wrote:
>> >>
>> >> And indeed I was wrong. I am not using lazylogfiles
>> >>
>> >> I am getting the 'logs dont show up until step stops' thingy from any
>> >> step, even when just trying to open a stdio or sterr, not only from
>> custom
>> >> logs (i.e. by using logfiles={"triallog": trial_temp/test.log"}).
>> >> Furthermore there's no proxy between me (or any pc on the network) and
>> the
>> >> buildbot.
>> >> Not sure what additional info may be usedul to provide to help sort
>> this
>> >> out.
>> >>
>> >> Cheers,
>> >>  Francesco
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sat, Oct 4, 2014 at 4:23 PM, Francesco Di Mizio
>> >> <francescodimizio at gmail.com> wrote:
>> >>>
>> >>> Well I dont remeber changing them. I only know tthey used to stream
>> and
>> >>> now they don't.
>> >>> I might be wrong though, Next week I'll check the history and update
>> >>> here.
>> >>>
>> >>> Thanks
>> >>>
>> >>> On Sat, Oct 4, 2014 at 3:37 PM, Dustin J. Mitchell <dustin at v.igoro.us
>> >
>> >>> wrote:
>> >>>>
>> >>>> Hang on, I just noticed the thread subject.. If these are, indeed,
>> >>>> configured as lazy logs then they shouldn't even show up on the
>> master
>> >>>> until the step stops.  In other words, lazy logs are not streamed,
>> >>>> they are downloaded at step completion.
>> >>>>
>> >>>> Dustin
>> >>>>
>> >>>> On Fri, Oct 3, 2014 at 5:56 PM, Thomas Middeldorp
>> >>>> <thomas at ranzer.geek.nz> wrote:
>> >>>> > I can confirm "proxy_buffering off;" is what fixed this for me in
>> 0.8.
>> >>>> > In my
>> >>>> > setup I have /waterfall behind a proxy_cache (you could add other
>> >>>> > locations if
>> >>>> > needed) and then / is a straight proxy_pass with buffering turned
>> off.
>> >>>> >
>> >>>> > - Thomas
>> >>>> >
>> >>>> >
>> >>>> > On 4/10/2014 1:46 a.m., Dustin J. Mitchell wrote:
>> >>>> >> My googling is failing to find previous threads about the proxy
>> issue
>> >>>> >> and the nginx fix for it -- there's a parameter of some sort to
>> turn
>> >>>> >> off buffering, similar to what Charles has suggested.
>> >>>> >>
>> >>>> >> In 0.9.0, we're no longer using the technique of dripping out an
>> HTTP
>> >>>> >> response slowly, so the header change isn't necessary.  However,
>> the
>> >>>> >> header or at least a note about Apache and Nginx configuration in
>> the
>> >>>> >> docs, would be a good addition to the 'eight' branch.
>> >>>> >>
>> >>>> >> Dustin
>> >>>> >>
>> >>>> >> On Thu, Oct 2, 2014 at 11:10 PM, Charles Lepple <
>> clepple at gmail.com>
>> >>>> >> wrote:
>> >>>> >>> Definitely not a 1:1 mapping, but in Apache, I dropped the proxy
>> >>>> >>> buffer from the default of 8 kB to 512 bytes, and logs stream
>> fine. The
>> >>>> >>> nginx parameters look a bit more complicated:
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
>> >>>> >>>
>> >>>> >>> Here's the Apache parameter I tweaked:
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxyiobuffersize
>> >>>> >>>
>> >>>> >>> One thing that Buildbot could potentially do on the logs is add
>> >>>> >>> "X-Accel-Buffering: no" for nginx (I think; unless I got it
>> backwards).
>> >>>> >>>
>> >>>> >>> --
>> >>>> >>> - Charles Lepple
>> >>>> >>> http://ghz.cc/charles/
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> ------------------------------------------------------------------------------
>> >>>> >>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> >>>> >>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>> >>>> >>> Reports
>> >>>> >>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>> paper
>> >>>> >>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>> Analyzer
>> >>>> >>>
>> >>>> >>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>> >>>> >>> _______________________________________________
>> >>>> >>> Buildbot-devel mailing list
>> >>>> >>> Buildbot-devel at lists.sourceforge.net
>> >>>> >>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> >>>> >>
>> >>>> >>
>> ------------------------------------------------------------------------------
>> >>>> >> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> >>>> >> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>> >>>> >> Reports
>> >>>> >> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>> paper
>> >>>> >> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>> Analyzer
>> >>>> >>
>> >>>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>> >>>> >> _______________________________________________
>> >>>> >> Buildbot-devel mailing list
>> >>>> >> Buildbot-devel at lists.sourceforge.net
>> >>>> >> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> ------------------------------------------------------------------------------
>> >>>> > Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> >>>> > Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>> >>>> > Reports
>> >>>> > Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
>> paper
>> >>>> > Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>> Analyzer
>> >>>> >
>> >>>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>> >>>> > _______________________________________________
>> >>>> > Buildbot-devel mailing list
>> >>>> > Buildbot-devel at lists.sourceforge.net
>> >>>> > https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> >>>>
>> >>>>
>> >>>>
>> ------------------------------------------------------------------------------
>> >>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> >>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
>> Reports
>> >>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
>> >>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
>> >>>>
>> >>>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>> >>>> _______________________________________________
>> >>>> Buildbot-devel mailing list
>> >>>> Buildbot-devel at lists.sourceforge.net
>> >>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>> >>>
>> >>>
>> >>
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20141007/edbc8a39/attachment.html>