[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

Mon Oct 6 15:41:42 UTC 2014

Sadly I can see the log being dumped as it runs.

Francesco

On Mon, Oct 6, 2014 at 5:25 PM, Dustin J. Mitchell <dustin at v.igoro.us>
wrote:

> It could be responsible, yes.  One way to test is to run your command
> by hand, redirected to a file, and then tail -f that file.  If you
> don't see anything in the file until the command is complete, then
> buffering is the problem.
>
> Dustin
>
> On Mon, Oct 6, 2014 at 11:06 AM, Francesco Di Mizio
> <francescodimizio at gmail.com> wrote:
> > I came across a line searching on the web:
> >
> > you might want to turn off output buffering for python so that you can
> see
> > the results of your build steps as they happen. This is of course only
> > required if your build steps are calling python scripts, like I am. If
> you
> > are calling something else like make or devenv, then you really dont have
> > any control over the output buffering. You can turn off output buffering
> in
> > python scripts by setting an environment variable PYTHONUNBUFFERRED=TRUE
> >
> > Does it make any sense to you guys? I mainly call py scripts that in turn
> > can call exec, batch/bash scripts.It seems to fit.
> >
> >
> >
> > On Mon, Oct 6, 2014 at 4:44 PM, Francesco Di Mizio
> > <francescodimizio at gmail.com> wrote:
> >>
> >> And indeed I was wrong. I am not using lazylogfiles
> >>
> >> I am getting the 'logs dont show up until step stops' thingy from any
> >> step, even when just trying to open a stdio or sterr, not only from
> custom
> >> logs (i.e. by using logfiles={"triallog": trial_temp/test.log"}).
> >> Furthermore there's no proxy between me (or any pc on the network) and
> the
> >> buildbot.
> >> Not sure what additional info may be usedul to provide to help sort this
> >> out.
> >>
> >> Cheers,
> >>  Francesco
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 4, 2014 at 4:23 PM, Francesco Di Mizio
> >> <francescodimizio at gmail.com> wrote:
> >>>
> >>> Well I dont remeber changing them. I only know tthey used to stream and
> >>> now they don't.
> >>> I might be wrong though, Next week I'll check the history and update
> >>> here.
> >>>
> >>> Thanks
> >>>
> >>> On Sat, Oct 4, 2014 at 3:37 PM, Dustin J. Mitchell <dustin at v.igoro.us>
> >>> wrote:
> >>>>
> >>>> Hang on, I just noticed the thread subject.. If these are, indeed,
> >>>> configured as lazy logs then they shouldn't even show up on the master
> >>>> until the step stops.  In other words, lazy logs are not streamed,
> >>>> they are downloaded at step completion.
> >>>>
> >>>> Dustin
> >>>>
> >>>> On Fri, Oct 3, 2014 at 5:56 PM, Thomas Middeldorp
> >>>> <thomas at ranzer.geek.nz> wrote:
> >>>> > I can confirm "proxy_buffering off;" is what fixed this for me in
> 0.8.
> >>>> > In my
> >>>> > setup I have /waterfall behind a proxy_cache (you could add other
> >>>> > locations if
> >>>> > needed) and then / is a straight proxy_pass with buffering turned
> off.
> >>>> >
> >>>> > - Thomas
> >>>> >
> >>>> >
> >>>> > On 4/10/2014 1:46 a.m., Dustin J. Mitchell wrote:
> >>>> >> My googling is failing to find previous threads about the proxy
> issue
> >>>> >> and the nginx fix for it -- there's a parameter of some sort to
> turn
> >>>> >> off buffering, similar to what Charles has suggested.
> >>>> >>
> >>>> >> In 0.9.0, we're no longer using the technique of dripping out an
> HTTP
> >>>> >> response slowly, so the header change isn't necessary.  However,
> the
> >>>> >> header or at least a note about Apache and Nginx configuration in
> the
> >>>> >> docs, would be a good addition to the 'eight' branch.
> >>>> >>
> >>>> >> Dustin
> >>>> >>
> >>>> >> On Thu, Oct 2, 2014 at 11:10 PM, Charles Lepple <clepple at gmail.com
> >
> >>>> >> wrote:
> >>>> >>> Definitely not a 1:1 mapping, but in Apache, I dropped the proxy
> >>>> >>> buffer from the default of 8 kB to 512 bytes, and logs stream
> fine. The
> >>>> >>> nginx parameters look a bit more complicated:
> >>>> >>>
> >>>> >>>
> >>>> >>>
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
> >>>> >>>
> >>>> >>> Here's the Apache parameter I tweaked:
> >>>> >>>
> >>>> >>>
> >>>> >>>
> http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxyiobuffersize
> >>>> >>>
> >>>> >>> One thing that Buildbot could potentially do on the logs is add
> >>>> >>> "X-Accel-Buffering: no" for nginx (I think; unless I got it
> backwards).
> >>>> >>>
> >>>> >>> --
> >>>> >>> - Charles Lepple
> >>>> >>> http://ghz.cc/charles/
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> ------------------------------------------------------------------------------
> >>>> >>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> >>>> >>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
> >>>> >>> Reports
> >>>> >>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
> paper
> >>>> >>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
> Analyzer
> >>>> >>>
> >>>> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
> >>>> >>> _______________________________________________
> >>>> >>> Buildbot-devel mailing list
> >>>> >>> Buildbot-devel at lists.sourceforge.net
> >>>> >>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> >>>> >>
> >>>> >>
> ------------------------------------------------------------------------------
> >>>> >> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> >>>> >> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
> >>>> >> Reports
> >>>> >> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White
> paper
> >>>> >> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
> Analyzer
> >>>> >>
> >>>> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
> >>>> >> _______________________________________________
> >>>> >> Buildbot-devel mailing list
> >>>> >> Buildbot-devel at lists.sourceforge.net
> >>>> >> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> ------------------------------------------------------------------------------
> >>>> > Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> >>>> > Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
> >>>> > Reports
> >>>> > Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> >>>> > Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> >>>> >
> >>>> >
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
> >>>> > _______________________________________________
> >>>> > Buildbot-devel mailing list
> >>>> > Buildbot-devel at lists.sourceforge.net
> >>>> > https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> >>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS
> Reports
> >>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> >>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> >>>>
> >>>>
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
> >>>> _______________________________________________
> >>>> Buildbot-devel mailing list
> >>>> Buildbot-devel at lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
> >>>
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20141006/5c36489b/attachment.html>