[users at bb.net] buildbot retains memory of an earlier location of a buildslave?
Greg Bullock
greg at nwra.com
Sat Jun 18 00:11:06 UTC 2016
Just to close this thread: The problem seems to be resolved now.
My best guess is that sqlite retains a cache somewhere that persists
when one deletes the buildbot master and reboots the machine.
As a result, the entire log contents for a step (not just the workdir,
as I originally thought) displayed results actually from an earlier,
outdated build.
Regardless of the underlying explanation, to workaround the problem, I
deleted, then re-created the buildbot master specifying a different db URL:
buildbot create-master --db sqlite:///freshstate.sqlite -r master
This requires a corresponding change to the master.cfg file:
c['db'] = {
'db_url' : "sqlite:///freshstate.sqlite",
}
After that, the logs for the individual steps (including the SVN
checkout step) started showing the correct workdir.
If anyone knows a way to clean sqlite's cache for the buildbot database
(without having to re-create the entire buildbot master), I'd be most
interested to learn that.
-Greg
On 6/17/2016 10:03 AM, Greg Bullock wrote:
>
> The step page indeed shows the start time and end time the step, but
> clicking the associated link under the Logs section displays content
> that itself includes no time stamp.
>
> For my current purposes, I'd like that log content to also report the
> date & time.
>
> Some anomaly on my system causes the buildbot to retain prior memory
> and ultimately insert into the log a clue as to what it remembers.
> I've been thinking that its memory is confined to just the workdir --
> related to location of the buildslave. But so far I can't rule out
> that it's the entire log content getting pulled from outdated memory.
>
>
>
> On 6/16/2016 10:45 AM, Pierre Tardy wrote:
>>
>> There is a start time and end time for each step that you can see in
>> the step page
>>
>>
>> Le jeu. 16 juin 2016 19:19, Greg Bullock <greg at nwra.com> a écrit :
>>
>> It dawns on me that the problem may not be that the buildbot
>> somehow retains a memory of the earlier location of a
>> buildslave. It could instead be that the entire log that
>> displays is actually from an earlier (and failed) build attempt.
>>
>> Is there a way to have buildbot insert a datetime stamp into the
>> log for a given step?
>>
>>
>>
>> On 6/14/2016 3:01 PM, Greg Bullock wrote:
>>> I'm using version 0.8.12.
>>>
>>> I have indeed rebooted the master. I've also deleted the CentOS
>>> virtual machine (the slave) and created a new one from a fresh
>>> install of the system. No luck.
>>>
>>> I have not tried both of these measures in the same test, I
>>> wouldn't think that could make the difference.
>>>
>>> I will look into purging the sun cache, from the log, it seems
>>> that the buildslave is already reporting using the obsolete (and
>>> nonexistent) working directory before it calls svn.
>>>
>>> Greg Bullock
>>> NorthWest Research Associates
>>> 301 Webster St.
>>> Monterey, CA 93940
>>> (831) 582-4907 <tel:%28831%29%20582-4907>
>>> greg at nwra.com
>>>
>>> -------- Original message --------
>>> From: Pierre Tardy <tardyp at gmail.com> <mailto:tardyp at gmail.com>
>>> Date: 6/14/16 1:12 PM (GMT-08:00)
>>> To: Greg Bullock <greg at nwra.com> <mailto:greg at nwra.com>,
>>> users at buildbot.net <mailto:users at buildbot.net>
>>> Subject: Re: [users at bb.net <mailto:users at bb.net>] buildbot
>>> retains memory of an earlier location of a buildslave?
>>>
>>> Hi,
>>> Looks indeed like an interresting problem. Did you try to reboot
>>> the master?
>>>
>>> The master might indeed store the workdir, but I would expect it
>>> to be reset when the work is reconnecting.
>>>
>>> Which version of buildbot is that?
>>>
>>> Also thing to check is svn. could svn cache something?
>>>
>>> Regards
>>> Pierre
>>>
>>> Le mar. 14 juin 2016 à 21:04, Greg Bullock <greg at nwra.com
>>> <mailto:greg at nwra.com>> a écrit :
>>>
>>> Where might the buildbot retain its memory of an earlier
>>> location of a
>>> buildslave? I've moved a buildslave to a different drive,
>>> but the log
>>> from the new build tests shows the steps (e.g., the SVN,
>>> ShellCommand,
>>> and Compile steps) all running from the original drive, not
>>> the new drive.
>>>
>>> My buildbot runs on Windows 7 64-bit, and this particular
>>> buildslave
>>> (called "centos7-worker") runs on a virtual machine on
>>> CentOS 7 (linux)
>>> hosted on the same Windows machine with the buildbot master.
>>>
>>> Originally, the buildslave's files were on a shared drive
>>> (which is also
>>> a Windows share shared from the Windows host), in
>>> /media/sf_G_DRIVE/buildbot/centos7-worker, but the SVN
>>> checkout step
>>> would successfully checkout only about half the project,
>>> then give an error
>>>
>>> ------------------------------------------------------------
>>> ... (previous lines omitted)
>>> svn: E200030: disk I/O error, executing statement
>>> 'RELEASE s10246'
>>> svn: E200030: sqlite: disk I/O error
>>> svn: E200030: sqlite: disk I/O error
>>> svn: E200030: no such savepoint: s10247, executing statement
>>> 'RELEASE s10247'
>>> svn: E200030: no such savepoint: s10247, executing statement
>>> 'ROLLBACK TO s10247'
>>> program finished with exit code 1
>>> ------------------------------------------------------------
>>>
>>> After a Google search turned up several mentions of
>>> conflicts between
>>> Windows shared drive and sqlite (see, for example,
>>> http://stackoverflow.com/questions/18782941/tortoisesvn-checkout-sqlite-disk-i-o-error-s10),
>>> I moved the buildslave files to a local drive within the virtual
>>> machine, with the path /home/buildbot/Documents/centos7-worker.
>>>
>>> At this point, I expect the log to have references only to
>>> the new
>>> /home/buildbot location, with no more references to the older
>>> /media/sf_G_DRIVE location. But instead, the log still
>>> shows steps
>>> running "in dir
>>> /media/sf_G_DRIVE/buildbot/centos7-worker/...". The top
>>> of the log from the SVN checkout step shows an example:
>>>
>>> ------------------------------------------------------------
>>> svn --version
>>> in dir
>>> /media/sf_G_DRIVE/buildbot/centos7-worker/gfortran_centos7_build/build
>>> (timeout 1200 secs)
>>> watching logfiles {}
>>> argv: ['svn', '--version']
>>> environment:
>>> DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-ebvEqaI7D1,guid=b74561f6a34d14afe5191e515759b435
>>> DESKTOP_SESSION=gnome-classic
>>> DISPLAY=:0
>>> GDMSESSION=gnome-classic
>>> GDM_LANG=en_US.UTF-8
>>> GJS_DEBUG_OUTPUT=stderr
>>> GJS_DEBUG_TOPICS=JS ERROR;JS LOG
>>> GNOME_DESKTOP_SESSION_ID=this-is-deprecated
>>> GNOME_SHELL_SESSION_MODE=classic
>>> GPG_AGENT_INFO=/run/user/1000/keyring/gpg:0:1
>>> HISTCONTROL=ignoredups
>>> HISTSIZE=1000
>>> HOME=/home/buildbot
>>> HOSTNAME=localhost.localdomain
>>> IMSETTINGS_INTEGRATE_DESKTOP=yes
>>> IMSETTINGS_MODULE=none
>>> LANG=en_US.UTF-8
>>> LESSOPEN=||/usr/bin/lesspipe.sh %s
>>> LOGNAME=buildbot
>>> LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;
>>> 5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
>>> MAIL=/var/spool/mail/buildbot
>>> OLDPWD=/home/buildbot
>>> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/buildbot/.local/bin:/home/buildbot/bin
>>> PWD=/media/sf_G_DRIVE/buildbot/centos7-worker/gfortran_centos7_build/build
>>> QTDIR=/usr/lib64/qt-3.3
>>> QTINC=/usr/lib64/qt-3.3/include
>>> QTLIB=/usr/lib64/qt-3.3/lib
>>> QT_GRAPHICSSYSTEM_CHECKED=1
>>> QT_IM_MODULE=ibus
>>> SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2119,unix/unix:/tmp/.ICE-unix/2119
>>> <mailto:SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2119,unix/unix:/tmp/.ICE-unix/2119>
>>> SHELL=/bin/bash
>>> SHLVL=2
>>> SSH_AGENT_PID=2306
>>> SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
>>> TERM=xterm-256color
>>> USER=buildbot
>>> USERNAME=buildbot
>>> VTE_VERSION=3803
>>> WINDOWID=44049837
>>> WINDOWPATH=1
>>> XAUTHORITY=/run/gdm/auth-for-buildbot-xH6kQ3/database
>>> XDG_CURRENT_DESKTOP=GNOME-Classic:GNOME
>>> XDG_MENU_PREFIX=gnome-
>>> XDG_RUNTIME_DIR=/run/user/1000
>>> XDG_SEAT=seat0
>>> XDG_SESSION_DESKTOP=gnome-classic
>>> XDG_SESSION_ID=1
>>> XDG_VTNR=1
>>> XMODIFIERS=@im=ibus
>>> _=/usr/bin/buildslave
>>> using PTY: False
>>> svn, version 1.7.14 (r1542130)
>>> compiled Nov 20 2015, 19:25:09
>>> (subsequent lines omitted)...
>>> ------------------------------------------------------------
>>>
>>> I've tried recreating the buildbot master and both of its
>>> buildslaves
>>> (deleting the master folder and the buildslaves folders,
>>> creating the
>>> buildbot and buildslaves, then restoring the buildbot.tacs and
>>> master.cfg from backup). And I also unmounted the
>>> /media/sf_G_DRIVE
>>> altogether. All of this seemed to purge the log as
>>> expected, but the
>>> next build attempt still makes reference to the old location
>>> of the
>>> CentOS buildslave in /media/sf_G_DRIVE drive.
>>>
>>> Where might the reference to this obsolete location of the
>>> buildslave be
>>> stored? The centos7-worker's buildbot.tac file contains no
>>> explicit
>>> reference to the obsolete location:
>>> ------------------------------------------------------------
>>>
>>> import os
>>>
>>> from buildslave.bot import BuildSlave
>>> from twisted.application import service
>>>
>>> basedir = '.'
>>> rotateLength = 10000000
>>> maxRotatedFiles = 10
>>>
>>> # if this is a relocatable tac file, get the directory
>>> containing the TAC
>>> if basedir == '.':
>>> import os.path
>>> basedir = os.path.abspath(os.path.dirname(__file__))
>>>
>>> # note: this line is matched against to check that this
>>> is a buildslave
>>> # directory; do not edit it.
>>> application = service.Application('buildslave')
>>>
>>> try:
>>> from twisted.python.logfile import LogFile
>>> from twisted.python.log import ILogObserver,
>>> FileLogObserver
>>> logfile = LogFile.fromFullPath(os.path.join(basedir,
>>> "twistd.log"),
>>> rotateLength=rotateLength,
>>> maxRotatedFiles=maxRotatedFiles)
>>> application.setComponent(ILogObserver,
>>> FileLogObserver(logfile).emit)
>>> except ImportError:
>>> # probably not yet twisted 8.2.0 and beyond, can't set
>>> log yet
>>> pass
>>>
>>> buildmaster_host = 'kraken'
>>> port = 9989
>>> slavename = 'centos7-worker'
>>> passwd = 'pass'
>>> keepalive = 600
>>> usepty = 0
>>> umask = None
>>> maxdelay = 300
>>> allow_shutdown = None
>>>
>>> s = BuildSlave(buildmaster_host, port, slavename, passwd,
>>> basedir,
>>> keepalive, usepty, umask=umask,
>>> maxdelay=maxdelay,
>>> allow_shutdown=allow_shutdown)
>>> s.setServiceParent(application)
>>>
>>>
>>> Likewise, the master.cfg file creates a single SVN checkout
>>> step as
>>>
>>> checkout = SVN(repourl = 'svn://kraken/trunk/sf_code',
>>> username = "buildbot",
>>> password = "buildbot",
>>> haltOnFailure = True,
>>> )
>>> ------------------------------------------------------------
>>>
>>> and this step is used in all builders on both buildslaves (the
>>> "centos7-worker" and the "win64-worker"), so I don't see how
>>> it could
>>> convey the obsolete location to the buildslave.
>>>
>>>
>>> --
>>> Greg Bullock
>>> NorthWest Research Associates
>>> 301 Webster St.
>>> Monterey, CA 93940
>>> (831) 582-4907
>>> greg at nwra.com <mailto:greg at nwra.com>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users at buildbot.net <mailto:users at buildbot.net>
>>> https://lists.buildbot.net/mailman/listinfo/users
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users at buildbot.net <mailto:users at buildbot.net>
>>> https://lists.buildbot.net/mailman/listinfo/users
>>
>> --
>> Regards.
>> Greg Bullock
>> NorthWest Research Associates
>> 301 Webster St.
>> Monterey CA 93940
>> (831) 582-4907
>> greg at nwra.com <mailto:greg at nwra.com>
>>
>> _______________________________________________
>> users mailing list
>> users at buildbot.net <mailto:users at buildbot.net>
>> https://lists.buildbot.net/mailman/listinfo/users
>>
>
> --
> Greg Bullock
> NorthWest Research Associates
> 301 Webster St.
> Monterey, CA 93940
> (831) 582-4907
> greg at nwra.com
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
--
Greg Bullock
NorthWest Research Associates
301 Webster St.
Monterey, CA 93940
(831) 582-4907
greg at nwra.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160617/9127f623/attachment.html>
More information about the users
mailing list