[users at bb.net] Patch for GitPoller

Wed Nov 18 16:46:23 UTC 2015

Hi,

 

Just wanted to pass along this bit of info to anyone using the GitPoller.
Maybe the change is worthwhile enough to implement in the main product.

 

I was faced with two problems with the GitPoller, as follows.

 

1.       BuildBot polls four different repositories for me, which turns out
that's 100+ branches it's looking at, every two minutes. Given that there's
a log entry for each branch that it looks at and despite the fact that it's
looking at different branches for each one, the entries in the twistd.log
file don't differentiate. Basically, GitPoller was repeatedly spamming the
log with messages like the one below.

gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
gitpoller: processing 0 changes: [] from "ssh://git@host.com/reponame"
.



2.       The next problem was that I kept getting a depreciated warning from
twisted in the log, repeatedly, over and over and over again, exactly like
the one below. I had no idea what process in BuildBot was causing this. The
message had no additional information.

/usr/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/t
wisted/internet/utils.py:25: exceptions.DeprecationWarning: Argument strings
and environment keys/values passed to reactor.spawnProcess should be str,
not unicode.



Between the two above issues, ALL TEN of the twistd.log files were being
filled with log messages that spanned only a couple minutes, which was not
very helpful for tracking down other issues that would have been logged. For
those of you experiencing these problems, read on and I'll tell you what I
did to fix it.

 

For the first problem, instead of cutting down on the number of times
gitpoller wrote a line for the polling it does, I made the output more
helpful.



>From 5a5533d0afeffa86258b20ed00d5663e76d9ddcf Mon Sep 17 00:00:00 2001

From: Jeremy Cornett <jeremy.cornett at venafi.com>

Date: Wed, 18 Nov 2015 09:22:43 -0700

Subject: [PATCH] Add the ref name to the log output so you can see which was

examined.

 

---

master/buildbot/changes/gitpoller.py | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/master/buildbot/changes/gitpoller.py
b/master/buildbot/changes/gitp

oller.py

index e245537..9f943f0 100644

--- a/master/buildbot/changes/gitpoller.py

+++ b/master/buildbot/changes/gitpoller.py

@@ -275,8 +275,8 @@ class GitPoller(base.PollingChangeSource, StateMixin):

         self.changeCount = len(revList)

         self.lastRev[branch] = newRev

 

-        log.msg('gitpoller: processing %d changes: %s from "%s"'

-                % (self.changeCount, revList, self.repourl))

+        log.msg('gitpoller: processing %d changes for "%s", "%s": %s'

+                % (self.changeCount, self.repourl, branch, revList))

 

         for rev in revList:

             dl = defer.DeferredList([

--

1.9.5.msysgit.0

 

The above patch changes the output to look like this, which is a bit more
informative.



gitpoller: processing 0 changes for "ssh://git@host.com/reponame",
"refs/heads/branchname": []

 

The next problem I was able to unravel by using a patch already applied to
the master branch for the 0.9 release
(https://github.com/buildbot/buildbot/pull/1578, commit
1ff73110394a8fa23ad2065415ea655ee1b1c92e). The 1ff73110 fix allowed me to
see a Git error which allowed me to see the command BuildBot was running to
produce the Twistd depreciation warning. Apparently, I had a bad object that
the GitPoller was encountering when it tried to run a "git log" command. The
bad object is irrelevant to the fix I implemented (it was from a previous
polling and stored in state.sqlite, I just had to delete the sqlite file and
run update-master to recreate the file anew). Here is the new log message
from the above fix I had cherry-picked.


buildbot.changes.gitpoller.GitError: command log ['--format=%H',
'bed339b10d9638d6ac8de892e866fb026454f13f',
u'^9bc6bce7d5cb6f920cc4c5232fb26dd8c09e4c9e',
u'^88713a7d0607b3ec7562843cba414039e96e46a8', .

 

If you notice in that command, after the first hash, all the rest are casted
as Unicode strings. That's oddly familiar. By examining the GitPoller file
some more, I came to realize that whatever was storing the values in the
dictionary lastRev, was storing them as Unicode. Probably just how they were
being extracted from the SQL lite database. I cast the rev as an ASCII
string, and all was happy! Twistd deprecation warning was no more.

 

>From 0f400eab9b275683882683e7828a4a1b32fcea0e Mon Sep 17 00:00:00 2001

From: Jeremy Cornett <jeremy.cornett at venafi.com>

Date: Tue, 17 Nov 2015 16:41:51 -0700

Subject: [PATCH] This is a shot in the dark. Twisted keeps complaining about

wanting strings instead of unicode when the gitpoller is running. It looks

like the unicode strings are coming from that self.lastRev variable.

 

---

master/buildbot/changes/gitpoller.py | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

 

diff --git a/master/buildbot/changes/gitpoller.py
b/master/buildbot/changes/gitp

oller.py

index 936027d..e245537 100644

--- a/master/buildbot/changes/gitpoller.py

+++ b/master/buildbot/changes/gitpoller.py

@@ -264,7 +264,7 @@ class GitPoller(base.PollingChangeSource, StateMixin):

 

         # get the change list

         revListArgs = ([r'--format=%H', r'%s' % newRev] +

-                       [r'^%s' % rev for rev in self.lastRev.values()] +

+                       [r'^%s' % rev.encode('ascii', 'ignore') for rev in
self.lastRev.values()] +

                        [r'--'])

         self.changeCount = 0

         results = yield self._dovccmd('log', revListArgs,
path=self.workdir)

--

1.9.5.msysgit.0

 

It's probably not the most elegant solution, but it got the job done.
Perhaps someone more into BuildBot coding could use this to implement a
better solution.

 

Thanks,

Jeremy Cornett

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20151118/06be2178/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5955 bytes
Desc: not available
URL: <http://lists.buildbot.net/pipermail/users/attachments/20151118/06be2178/attachment.bin>
