[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

[Francesco Di Mizio]

Hey Dustin,

thanks for taking the time to repro this.

Did some more testing just to be 100% sure.
I do have different results. I never see any output until the step is over,
be it buffered and unbuffered. Regardless of how many bytes I am
outputting. Regardless of usePTY.

When I click to see the stdio of a running step, the browser says "waiting
for "master-ip"". It then remains in this limbo state untill the step is
done. As soon as that happens, stdio gets shown all in one go.

Something must be wrong with my network - quite hard to tell what.

Cheers,
 FDM




On Sun, Oct 19, 2014 at 12:04 AM, Dustin J. Mitchell <dustin at v.igoro.us>
wrote:

> Great, thanks for the reproduction.
>
> With this configuration and Buildbot-0.8.9, I see the
> "test_default.py" run for a long time with no visible output in the
> log display.  This is expected, as stdio buffering generally has a
> buffer of several kilobytes with no timeout, and test_default.py is
> averaging 2.5 bytes per second.  I didn't wait 2000 seconds for it to
> get past 4096 bytes.  Adding 'usePTY=True' to that command gets the
> output immediately (because when connected to a terminal, the C
> library flushes its buffers after every newline).  Alternately, if I
> edit test_default to output a lot more data (`print 'test\n' * 1024`),
> the output appears in the log display.
>
> If I switch the step so that "test_unbuffered.py" runs, then I see the
> expected output: new lines containing "test" appearing periodically.
> Buildbot has a bit of its own buffering, with a five second timeout,
> so instead of one "test" every 2 seconds, you see two or three "tests"
> every five seconds, but this is close enough and saves a lot of
> network traffic and CPU on the master.
>
> So this is all behaivng as expected.  Are you seeing something different?
>
> Dustin
>
> On Fri, Oct 17, 2014 at 5:34 AM, Francesco Di Mizio
> <francescodimizio at gmail.com> wrote:
> > Same thing with a Linux box as slave.
> >
> > Here is my master.cfg:
> >
> >
> > c = BuildmasterConfig = {}
> >
> >
> > from buildbot.buildslave import BuildSlave
> > c['slaves'] = [BuildSlave("example-slave", "pass"),
> > BuildSlave("linux-slave", "pass")
> > ]
> >
> > c['protocols'] = {'pb': {'port': 9989}}
> >
> >
> > ####### SCHEDULERS
> >
> > from buildbot.schedulers.basic import SingleBranchScheduler
> > from buildbot.schedulers.forcesched import ForceScheduler
> > from buildbot.schedulers import timed
> > from buildbot.changes import filter
> > c['schedulers'] = []
> > c['schedulers'].append(timed.Periodic(
> >                             name="periodic",
> >                             builderNames=["test", "test_linux"],
> > periodicBuildTimer=60))
> >
> > ####### BUILDERS
> >
> >
> > from buildbot.process.factory import BuildFactory
> > from buildbot.steps.source.git import Git
> > from buildbot.steps.shell import ShellCommand
> >
> >
> >
> > testFactory = BuildFactory()
> > # run the tests (note that this will require that 'trial' is installed)
> > testFactory.addStep(ShellCommand(command=['python',
> > "/home/manage/loop_default.py"]))
> > testFactory.addStep(ShellCommand(command=['python',
> > "/home/manage/loop_unbuffered.py"]))
> >
> >
> >
> > linuxFactory = BuildFactory()
> > # run the tests (note that this will require that 'trial' is installed)
> > linuxFactory.addStep(ShellCommand(command=['python',
> > "c:\\loop_default.py"]))
> > linuxFactory.addStep(ShellCommand(command=['python',
> > "c:\\loop_unbuffered.py"]))
> >
> > from buildbot.config import BuilderConfig
> >
> > c['builders'] = []
> >
> > c['builders'].append(
> >     BuilderConfig(name="test",
> >       slavenames=["example-slave"],
> >       factory=testFactory))
> > c['builders'].append(
> >     BuilderConfig(name="test_linux",
> >       slavenames=["linux-slave"],
> >       factory=testFactory))
> >
> > ####### STATUS TARGETS
> >
> >
> > c['status'] = []
> >
> > from buildbot.status import html
> > from buildbot.status.web import authz, auth
> >
> > authz_cfg=authz.Authz(
> >     # change any of these to True to enable; see the manual for more
> >     # options
> >     auth=auth.BasicAuth([("pyflakes","pyflakes")]),
> >     gracefulShutdown = False,
> >     forceBuild = 'auth', # use this to test your slave once it is set up
> >     forceAllBuilds = False,
> >     pingBuilder = False,
> >     stopBuild = False,
> >     stopAllBuilds = False,
> >     cancelPendingBuild = False,
> > )
> > c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
> >
> > ####### PROJECT IDENTITY
> >
> > c['title'] = "Pyflakes"
> > c['titleURL'] = "https://launchpad.net/pyflakes"
> > c['buildbotURL'] = "http://192.168.15.174:8010/"
> >
> > ####### DB URL
> >
> > c['db'] = {
> >     # This specifies what database buildbot uses to store its state.  You
> > can leave
> >     # this at its default for all but the largest installations.
> >     'db_url' : "mysql://build:build@buildbot-coordinator02
> /bbmaster_fdm",
> > }
> >
> >
> > Cheers,
> >  Francesco
> >
> > On Thu, Oct 16, 2014 at 8:17 PM, Dustin J. Mitchell <dustin at v.igoro.us>
> > wrote:
> >>
> >> Please try reproducing on a linux slave.  Can you also provide the
> >> master.cfg that you're using to test?
> >>
> >> Dustin
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20141020/d50a491d/attachment.html>