[users at bb.net] Patch for GitPoller

Dustin J. Mitchell dustin at v.igoro.us
Mon Nov 23 14:35:28 UTC 2015


Thanks!  The first patch is already implemented in the master branch,
but I applied the moral equivalent of the second patch in commit ID
a6b9bd640277f30dc9774f2c4040ca904e67f35c

Dustin

On Wed, Nov 18, 2015 at 11:46 AM, Jeremy Cornett
<jeremy.cornett at venafi.com> wrote:
> Hi,
>
>
>
> Just wanted to pass along this bit of info to anyone using the GitPoller.
> Maybe the change is worthwhile enough to implement in the main product.
>
>
>
> I was faced with two problems with the GitPoller, as follows…
>
>
>
> 1.       BuildBot polls four different repositories for me, which turns out
> that’s 100+ branches it’s looking at, every two minutes. Given that there’s
> a log entry for each branch that it looks at and despite the fact that it’s
> looking at different branches for each one, the entries in the twistd.log
> file don’t differentiate. Basically, GitPoller was repeatedly spamming the
> log with messages like the one below.
>
> gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
> gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
> gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
>>
> 2.       The next problem was that I kept getting a depreciated warning from
> twisted in the log, repeatedly, over and over and over again, exactly like
> the one below. I had no idea what process in BuildBot was causing this. The
> message had no additional information.
>
> /usr/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/internet/utils.py:25:
> exceptions.DeprecationWarning: Argument strings and environment keys/values
> passed to reactor.spawnProcess should be str, not unicode.
>
> Between the two above issues, ALL TEN of the twistd.log files were being
> filled with log messages that spanned only a couple minutes, which was not
> very helpful for tracking down other issues that would have been logged. For
> those of you experiencing these problems, read on and I’ll tell you what I
> did to fix it.
>
>
>
> For the first problem, instead of cutting down on the number of times
> gitpoller wrote a line for the polling it does, I made the output more
> helpful.
>
> From 5a5533d0afeffa86258b20ed00d5663e76d9ddcf Mon Sep 17 00:00:00 2001
>
> From: Jeremy Cornett <jeremy.cornett at venafi.com>
>
> Date: Wed, 18 Nov 2015 09:22:43 -0700
>
> Subject: [PATCH] Add the ref name to the log output so you can see which was
>
> examined.
>
>
>
> ---
>
> master/buildbot/changes/gitpoller.py | 4 ++--
>
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
>
>
> diff --git a/master/buildbot/changes/gitpoller.py
> b/master/buildbot/changes/gitp
>
> oller.py
>
> index e245537..9f943f0 100644
>
> --- a/master/buildbot/changes/gitpoller.py
>
> +++ b/master/buildbot/changes/gitpoller.py
>
> @@ -275,8 +275,8 @@ class GitPoller(base.PollingChangeSource, StateMixin):
>
>          self.changeCount = len(revList)
>
>          self.lastRev[branch] = newRev
>
>
>
> -        log.msg('gitpoller: processing %d changes: %s from "%s"'
>
> -                % (self.changeCount, revList, self.repourl))
>
> +        log.msg('gitpoller: processing %d changes for "%s", "%s": %s'
>
> +                % (self.changeCount, self.repourl, branch, revList))
>
>
>
>          for rev in revList:
>
>              dl = defer.DeferredList([
>
> --
>
> 1.9.5.msysgit.0
>
>
>
> The above patch changes the output to look like this, which is a bit more
> informative…
>
> gitpoller: processing 0 changes for "ssh://git@host.com/reponame",
> "refs/heads/branchname": []
>
>
>
> The next problem I was able to unravel by using a patch already applied to
> the master branch for the 0.9 release
> (https://github.com/buildbot/buildbot/pull/1578, commit
> 1ff73110394a8fa23ad2065415ea655ee1b1c92e). The 1ff73110 fix allowed me to
> see a Git error which allowed me to see the command BuildBot was running to
> produce the Twistd depreciation warning. Apparently, I had a bad object that
> the GitPoller was encountering when it tried to run a “git log” command. The
> bad object is irrelevant to the fix I implemented (it was from a previous
> polling and stored in state.sqlite, I just had to delete the sqlite file and
> run update-master to recreate the file anew). Here is the new log message
> from the above fix I had cherry-picked.
>
>
> buildbot.changes.gitpoller.GitError: command log ['--format=%H',
> 'bed339b10d9638d6ac8de892e866fb026454f13f',
> u'^9bc6bce7d5cb6f920cc4c5232fb26dd8c09e4c9e',
> u'^88713a7d0607b3ec7562843cba414039e96e46a8', …
>
>
>
> If you notice in that command, after the first hash, all the rest are casted
> as Unicode strings. That’s oddly familiar. By examining the GitPoller file
> some more, I came to realize that whatever was storing the values in the
> dictionary lastRev, was storing them as Unicode. Probably just how they were
> being extracted from the SQL lite database. I cast the rev as an ASCII
> string, and all was happy! Twistd deprecation warning was no more.
>
>
>
> From 0f400eab9b275683882683e7828a4a1b32fcea0e Mon Sep 17 00:00:00 2001
>
> From: Jeremy Cornett <jeremy.cornett at venafi.com>
>
> Date: Tue, 17 Nov 2015 16:41:51 -0700
>
> Subject: [PATCH] This is a shot in the dark. Twisted keeps complaining about
>
> wanting strings instead of unicode when the gitpoller is running. It looks
>
> like the unicode strings are coming from that self.lastRev variable.
>
>
>
> ---
>
> master/buildbot/changes/gitpoller.py | 2 +-
>
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>
>
> diff --git a/master/buildbot/changes/gitpoller.py
> b/master/buildbot/changes/gitp
>
> oller.py
>
> index 936027d..e245537 100644
>
> --- a/master/buildbot/changes/gitpoller.py
>
> +++ b/master/buildbot/changes/gitpoller.py
>
> @@ -264,7 +264,7 @@ class GitPoller(base.PollingChangeSource, StateMixin):
>
>
>
>          # get the change list
>
>          revListArgs = ([r'--format=%H', r'%s' % newRev] +
>
> -                       [r'^%s' % rev for rev in self.lastRev.values()] +
>
> +                       [r'^%s' % rev.encode('ascii', 'ignore') for rev in
> self.lastRev.values()] +
>
>                         [r'--'])
>
>          self.changeCount = 0
>
>          results = yield self._dovccmd('log', revListArgs,
> path=self.workdir)
>
> --
>
> 1.9.5.msysgit.0
>
>
>
> It’s probably not the most elegant solution, but it got the job done.
> Perhaps someone more into BuildBot coding could use this to implement a
> better solution.
>
>
>
> Thanks,
>
> Jeremy Cornett
>
>
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users


More information about the users mailing list