[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

[Dustin J. Mitchell]

Great, thanks for the reproduction.

With this configuration and Buildbot-0.8.9, I see the
"test_default.py" run for a long time with no visible output in the
log display.  This is expected, as stdio buffering generally has a
buffer of several kilobytes with no timeout, and test_default.py is
averaging 2.5 bytes per second.  I didn't wait 2000 seconds for it to
get past 4096 bytes.  Adding 'usePTY=True' to that command gets the
output immediately (because when connected to a terminal, the C
library flushes its buffers after every newline).  Alternately, if I
edit test_default to output a lot more data (`print 'test\n' * 1024`),
the output appears in the log display.

If I switch the step so that "test_unbuffered.py" runs, then I see the
expected output: new lines containing "test" appearing periodically.
Buildbot has a bit of its own buffering, with a five second timeout,
so instead of one "test" every 2 seconds, you see two or three "tests"
every five seconds, but this is close enough and saves a lot of
network traffic and CPU on the master.

So this is all behaivng as expected.  Are you seeing something different?

Dustin

On Fri, Oct 17, 2014 at 5:34 AM, Francesco Di Mizio
<francescodimizio at gmail.com> wrote:
> Same thing with a Linux box as slave.
>
> Here is my master.cfg:
>
>
> c = BuildmasterConfig = {}
>
>
> from buildbot.buildslave import BuildSlave
> c['slaves'] = [BuildSlave("example-slave", "pass"),
> BuildSlave("linux-slave", "pass")
> ]
>
> c['protocols'] = {'pb': {'port': 9989}}
>
>
> ####### SCHEDULERS
>
> from buildbot.schedulers.basic import SingleBranchScheduler
> from buildbot.schedulers.forcesched import ForceScheduler
> from buildbot.schedulers import timed
> from buildbot.changes import filter
> c['schedulers'] = []
> c['schedulers'].append(timed.Periodic(
>                             name="periodic",
>                             builderNames=["test", "test_linux"],
> periodicBuildTimer=60))
>
> ####### BUILDERS
>
>
> from buildbot.process.factory import BuildFactory
> from buildbot.steps.source.git import Git
> from buildbot.steps.shell import ShellCommand
>
>
>
> testFactory = BuildFactory()
> # run the tests (note that this will require that 'trial' is installed)
> testFactory.addStep(ShellCommand(command=['python',
> "/home/manage/loop_default.py"]))
> testFactory.addStep(ShellCommand(command=['python',
> "/home/manage/loop_unbuffered.py"]))
>
>
>
> linuxFactory = BuildFactory()
> # run the tests (note that this will require that 'trial' is installed)
> linuxFactory.addStep(ShellCommand(command=['python',
> "c:\\loop_default.py"]))
> linuxFactory.addStep(ShellCommand(command=['python',
> "c:\\loop_unbuffered.py"]))
>
> from buildbot.config import BuilderConfig
>
> c['builders'] = []
>
> c['builders'].append(
>     BuilderConfig(name="test",
>       slavenames=["example-slave"],
>       factory=testFactory))
> c['builders'].append(
>     BuilderConfig(name="test_linux",
>       slavenames=["linux-slave"],
>       factory=testFactory))
>
> ####### STATUS TARGETS
>
>
> c['status'] = []
>
> from buildbot.status import html
> from buildbot.status.web import authz, auth
>
> authz_cfg=authz.Authz(
>     # change any of these to True to enable; see the manual for more
>     # options
>     auth=auth.BasicAuth([("pyflakes","pyflakes")]),
>     gracefulShutdown = False,
>     forceBuild = 'auth', # use this to test your slave once it is set up
>     forceAllBuilds = False,
>     pingBuilder = False,
>     stopBuild = False,
>     stopAllBuilds = False,
>     cancelPendingBuild = False,
> )
> c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
>
> ####### PROJECT IDENTITY
>
> c['title'] = "Pyflakes"
> c['titleURL'] = "https://launchpad.net/pyflakes"
> c['buildbotURL'] = "http://192.168.15.174:8010/"
>
> ####### DB URL
>
> c['db'] = {
>     # This specifies what database buildbot uses to store its state.  You
> can leave
>     # this at its default for all but the largest installations.
>     'db_url' : "mysql://build:build@buildbot-coordinator02/bbmaster_fdm",
> }
>
>
> Cheers,
>  Francesco
>
> On Thu, Oct 16, 2014 at 8:17 PM, Dustin J. Mitchell <dustin at v.igoro.us>
> wrote:
>>
>> Please try reproducing on a linux slave.  Can you also provide the
>> master.cfg that you're using to test?
>>
>> Dustin
>
>