[Buildbot-devel] lazy logs a bit too lazy and que'ed requests not being processed

Francesco Di Mizio francescodimizio at gmail.com
Fri Oct 24 12:14:17 UTC 2014


And indeed upgrading did not help.

At this point I guess I am better off waiting for 9 to be out :)

Francesco

On Fri, Oct 24, 2014 at 3:20 AM, Dustin J. Mitchell <dustin at v.igoro.us>
wrote:

> Well, I suspect this is some kind of an abuse of the chunked transfer
> encoding, anyway -- an HTML document isn't well-defined until the
> </html>, and we're relying on browsers to interpret and render an
> incomplete (and thus undefined) HTML document.  It's worked for over
> ten years, but things change.  It's odd that it still works for me and
> not for you, though.
>
> Anyway, the new (nine) web UI is a totally different technology, so I
> guess the best I can say!
>
> Dustin
>
> On Thu, Oct 23, 2014 at 9:05 PM, Francesco Di Mizio
> <francescodimizio at gmail.com> wrote:
> > That's what I thought as well. A quick Google search showed only one
> chrome
> > issue with chunks but it was many years ago.
> >
> > I've also tried Firefox and ie with same results. I'll try updating them
> > tomorrow, I doubt it'll work though...
> >
> > On Oct 24, 2014 2:55 AM, "Dustin J. Mitchell" <dustin at v.igoro.us> wrote:
> >>
> >> Sounds like a browser problem, then.  The "a4f", "795", and "1ee" are
> >> chunk headers, and the browser should be rendering those as it
> >> receives them.
> >>
> >> That's kind of crazy.  It works fine for me in a recent Firefox.  I
> >> haven't tried Chrome in a while.
> >>
> >> Dustin
> >>
> >> On Thu, Oct 23, 2014 at 9:41 AM, Francesco Di Mizio
> >> <francescodimizio at gmail.com> wrote:
> >> > As a test I've sligthly changed the py script I am running to render
> its
> >> > output more readable:
> >> >
> >> > import os
> >> > import time
> >> > import sys
> >> >
> >> > class Unbuffered(object):
> >> >    def __init__(self, stream):
> >> >        self.stream = stream
> >> >    def write(self, data):
> >> >        self.stream.write(data)
> >> >        self.stream.flush()
> >> >    def __getattr__(self, attr):
> >> >        return getattr(self.stream, attr)
> >> >
> >> >
> >> > sys.stdout = Unbuffered(sys.stdout)
> >> >
> >> >
> >> > batch = 0
> >> > while True:
> >> >
> >> > time.sleep(10)
> >> > print 'test bach %d\n' %batch
> >> > batch  += 1
> >> >
> >> >
> >> > I let the script go for a few seconds, then opened the log.
> >> > From Wireshark, when I go "follow tcp stream" from the log request I
> get
> >> > (sorry for the wall of text) :
> >> >
> >> > GET /builders/test_linux/builds/485/steps/shell/logs/stdio HTTP/1.1
> >> > Host: 192.168.15.174:8010
> >> > Connection: keep-alive
> >> > Accept:
> >> >
> >> >
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> >> > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
> >> > (KHTML,
> >> > like Gecko) Chrome/38.0.2125.104 Safari/537.36
> >> > Referer: http://192.168.15.174:8010/builders/test_linux/builds/485
> >> > Accept-Encoding: gzip,deflate,sdch
> >> > Accept-Language: en-US,en;q=0.8,it;q=0.6
> >> > Cookie: BuildBotSession=a12212813f540e2949c37e523ba45faa3027f6ac
> >> >
> >> > HTTP/1.1 200 OK
> >> > Transfer-Encoding: chunked
> >> > Date: Thu, 23 Oct 2014 13:29:36 GMT
> >> > Cache-Control: no-cache
> >> > Content-Type: text/html; charset=utf-8
> >> > Server: TwistedWeb/13.2.0
> >> >
> >> > a4f
> >> > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
> >> >     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
> >> >   <html>
> >> >   <head><title>Log File contents</title>
> >> >   <link rel="stylesheet" href="../../../../../../../default.css"
> >> > type="text/css" />
> >> >   <script type="text/javascript">
> >> >     var active = true;
> >> >     window.onscroll = function() {
> >> >       var html = document.documentElement;
> >> >       active = (document.body.scrollTop + html.clientHeight >=
> >> > document.body.scrollHeight - 30);
> >> >     }
> >> >     window.setInterval(function() {
> >> >       if (active) {
> >> >         document.body.scrollTop = document.body.scrollHeight;
> >> >       }
> >> >     }, 300);
> >> >   </script>
> >> >   </head>
> >> >   <body class='log'>
> >> >     <a href="stdio/text">(view as text)</a><br/>
> >> >     <pre>
> >> > 795
> >> > <span class="header">python /home/manage/loop_unbuffered.py
> >> >  in dir /var/lib/buildslave/test-slave/test_linux/build (timeout 1200
> >> > secs)
> >> >  watching logfiles {}
> >> >  argv: ['python', '/home/manage/loop_unbuffered.py']
> >> >  environment:
> >> >   HOME=/home/manage
> >> >   LANG=en_US.UTF-8
> >> >   LOGNAME=root
> >> >
> >> >
> >> >
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
> >> >   MAIL=/var/mail/root
> >> >   PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> >> >   PWD=/var/lib/buildslave/test-slave/test_linux/build
> >> >   SHELL=/bin/bash
> >> >   SUDO_COMMAND=/usr/bin/buildslave start test-slave
> >> >   SUDO_GID=1000
> >> >   SUDO_UID=1000
> >> >   SUDO_USER=manage
> >> >   TERM=xterm
> >> >   USER=root
> >> >   USERNAME=root
> >> >  using PTY: False
> >> > </span>
> >> > 1ee
> >> > <span class="stdout">test bach 0
> >> >
> >> > test bach 1
> >> >
> >> > test bach 2
> >> >
> >> > test bach 3
> >> >
> >> > .... guess what? :)
> >> >
> >> > test bach 32
> >> >
> >> > test bach 33
> >> >
> >> > </span>
> >> > 2a
> >> > <span class="stdout">test bach 34
> >> >
> >> > </span>
> >> > 2a
> >> > <span class="stdout">test bach 35
> >> >
> >> > </span>
> >> > 2a
> >> > <span class="stdout">test bach 36
> >> >
> >> > </span>
> >> >
> >> >
> >> > It keeps getting bigger and bigger. Still what I see if the browser
> >> > saying
> >> > "waiting for 192.168.15.174"
> >> >
> >> > Cheers,
> >> >  FDM
> >> >
> >> > On Mon, Oct 20, 2014 at 9:57 PM, Dustin J. Mitchell <
> dustin at v.igoro.us>
> >> > wrote:
> >> >>
> >> >> Try *not* using Fiddler (it's just one more, unnecessary variable),
> >> >> and if it still doesn't work, try using Wireshark to capture traffic
> >> >> between your browser and Buildbot.
> >> >>
> >> >> Dustin
> >> >>
> >> >> On Mon, Oct 20, 2014 at 1:07 PM, Francesco Di Mizio
> >> >> <francescodimizio at gmail.com> wrote:
> >> >> > Some more info I got from Fiddler:
> >> >> >
> >> >> > SESSION STATE: ReadingResponse.
> >> >> > DOWNLOAD PROGRESS: 33,614 bytes.
> >> >> >
> >> >> >
> >> >> > == FLAGS ==================
> >> >> > BitFlags: [ClientPipeReused, ServerPipeReused] 0x18
> >> >> > X-CLIENTIP: 127.0.0.1
> >> >> > X-CLIENTPORT: 51030
> >> >> > X-EGRESSPORT: 51031
> >> >> > X-HOSTIP: 192.168.15.174
> >> >> > X-PROCESSINFO: chrome:10936
> >> >> > X-SERVERSOCKET: REUSE ServerPipe#3
> >> >> >
> >> >> > == TIMING INFO ============
> >> >> > ClientConnected: 18:53:10.904
> >> >> > ClientBeginRequest: 18:53:14.133
> >> >> > GotRequestHeaders: 18:53:14.133
> >> >> > ClientDoneRequest: 18:53:14.133
> >> >> > Determine Gateway: 0ms
> >> >> > DNS Lookup: 0ms
> >> >> > TCP/IP Connect: 0ms
> >> >> > HTTPS Handshake: 0ms
> >> >> > ServerConnected: 18:53:10.905
> >> >> > FiddlerBeginRequest: 18:53:14.133
> >> >> > ServerGotRequest: 18:53:14.133
> >> >> > ServerBeginResponse: 18:53:14.138
> >> >> > GotResponseHeaders: 18:53:14.138
> >> >> > ServerDoneResponse: 00:00:00.000
> >> >> > ClientBeginResponse: 00:00:00.000
> >> >> > ClientDoneResponse: 00:00:00.000
> >> >> >
> >> >> >
> >> >> > The response was buffered before delivery to the client.
> >> >> >
> >> >> > == WININET CACHE INFO ============
> >> >> > This URL is not present in the WinINET cache. [Code: 2]
> >> >> > * Note: Data above shows WinINET's current cache state, not the
> state
> >> >> > at
> >> >> > the
> >> >> > time of the request.
> >> >> > * Note: Data above shows WinINET's Medium Integrity (non-Protected
> >> >> > Mode)
> >> >> > cache only.
> >> >> >
> >> >> >
> >> >> > DOWNLOAD PROGRESS keeps increasing during the step's execution.
> Just
> >> >> > nothing
> >> >> > is being shown.
> >> >> > Also this sounds suspicious: The response was buffered before
> >> >> > delivery
> >> >> > to
> >> >> > the client.
> >> >> >
> >> >> >
> >> >> > On Mon, Oct 20, 2014 at 2:54 PM, Francesco Di Mizio
> >> >> > <francescodimizio at gmail.com> wrote:
> >> >> >>
> >> >> >> GET /builders/test_linux/builds/398/steps/shell/logs/stdio
> HTTP/1.1
> >> >> >> Host: 192.168.15.174:8010
> >> >> >> Accept:
> >> >> >>
> >> >> >>
> >> >> >>
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> >> >> >> Accept-Encoding: gzip,deflate,sdch
> >> >> >> Accept-Language: en-US,en;q=0.8,it;q=0.6
> >> >> >> Cookie: BuildBotSession=4101a104fb9b505021468f7b7ee745e0bdf7adac
> >> >> >> Referer:
> http://192.168.15.174:8010/builders/test_linux/builds/398
> >> >> >> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
> >> >> >> (KHTML,
> >> >> >> like Gecko) Chrome/37.0.2062.124 Safari/537.36
> >> >> >>
> >> >> >> HTTP/1.1 200 OK
> >> >> >> Cache-Control: max-age=604800
> >> >> >> Content-Type: text/html; charset=utf-8
> >> >> >> Date: Mon, 20 Oct 2014 12:51:28 GMT
> >> >> >> Server: TwistedWeb/13.2.0
> >> >> >> Transfer-Encoding: chunked
> >> >> >>
> >> >> >> Once again, I can't see the response header until the step is
> over.
> >> >> >>
> >> >> >> Cheers,
> >> >> >>  FDM
> >> >> >>
> >> >> >>
> >> >> >> On Mon, Oct 20, 2014 at 2:10 PM, Benoît Allard
> >> >> >> <benoit.allard at greenbone.net> wrote:
> >> >> >>>
> >> >> >>> -----BEGIN PGP SIGNED MESSAGE-----
> >> >> >>> Hash: SHA1
> >> >> >>>
> >> >> >>> On 10/20/2014 11:53 AM, Francesco Di Mizio wrote:
> >> >> >>> > My production setup is indeed accessed via proxy (apache + mod
> >> >> >>> > proxy). Initially we all assumed that was the problem.
> >> >> >>> >
> >> >> >>> > After messing around with some proxy settings and seeing not
> >> >> >>> > light
> >> >> >>> > at the end of the tunnel, I tried a fresh install using a
> simple
> >> >> >>> > config with no proxy. It did not help.
> >> >> >>> >
> >> >> >>>
> >> >> >>> In my experience mod_proxy did not had the trouble nginx had by
> >> >> >>> default.
> >> >> >>>
> >> >> >>> But there also exist transparent proxies ...
> >> >> >>>
> >> >> >>> Look at the HTTP headers of the pages served by buildbot
> (somewhere
> >> >> >>> inside your browser, depends on the browser), one of them should
> >> >> >>> indicate what's happening ...
> >> >> >>>
> >> >> >>> The technology used by buildbot there is called "Chunked transfer
> >> >> >>> encoding", and (as I said) is known to give some troubles with
> some
> >> >> >>> proxies that wait for the connection to be closed to 'proxy' it.
> If
> >> >> >>> you mention that to your network administrator, it might shed
> some
> >> >> >>> lights ...
> >> >> >>>
> >> >> >>> Good luck.
> >> >> >>>
> >> >> >>> - --
> >> >> >>> Benoît Allard (B30A05B0)|Greenbone Networks
> >> >> >>> GmbH|http://greenbone.net
> >> >> >>> Neuer Graben 17, 49074 Osnabrück, Germany | AG Osnabrück, HR B
> >> >> >>> 202460
> >> >> >>> Executive Directors: Lukas Grunwald, Dr. Jan-Oliver Wagner
> >> >> >>> -----BEGIN PGP SIGNATURE-----
> >> >> >>> Version: GnuPG v1.4.12 (GNU/Linux)
> >> >> >>>
> >> >> >>> iQEcBAEBAgAGBQJURPvGAAoJEHZCfVOzCgWwyaMH/0VEKFFM8tsdykUzxZbSPLzG
> >> >> >>> 54RFzL8//WVcMO4zflGdqGPmNAhuGIaHDz1+uVl9S+yvmDtL5xtZ3DOFDhyi1iAw
> >> >> >>> 8rW73OUQHJIokXtJwHJDf9JN960dSpR6E/L7KibNooc+TWEmUv0HIdmSDBpdSUdK
> >> >> >>> HmBa3VNlBHJ/43LfKn6YbbKkRfHEaLYB2BGp87GRiT/WqmMiSn/w8AGTKyyuog36
> >> >> >>> CHu0tWwqm2SR+xeU7wYt6YyYRxDD0jJaPgObHD0p2dsJ6FrGElMz9bs2+puydchG
> >> >> >>> GqfgdxiJwDmS6XKVCjutXAdW2LX5uYkfVKzbutF/Wp/b0VksV6fN9z6d+y5HlfQ=
> >> >> >>> =s7lZ
> >> >> >>> -----END PGP SIGNATURE-----
> >> >> >>
> >> >> >>
> >> >> >
> >> >
> >> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20141024/bdea12d1/attachment.html>


More information about the devel mailing list