[Buildbot] #3183: fix network "spottiness" on new infra

[Buildbot trac]

#3183: fix network "spottiness" on new infra
---------------------+-------------------------
Reporter:  dustin    |      Owner:
    Type:  defect    |     Status:  new
Priority:  critical  |  Milestone:  sys - other
 Version:            |   Keywords:
---------------------+-------------------------
 We're seeing a lot of this sort of thing:

 {{{
 ansible-pull run on on syslog.int.buildbot.net failed.

 Starting ansible-pull at 2015-02-05 08:27:29
 localhost | FAILED >> {
     "cmd": "/usr/local/bin/git ls-remote origin -h refs/heads/master",
     "failed": true,
     "msg": "fatal: Unable to look up github.com (port 9418) (hostname nor
 servname provided, or not known)",
     "rc": 128,
     "stderr": "fatal: Unable to look up github.com (port 9418) (hostname
 nor servname provided, or not known)\n",
     "stdout": ""
 }
 }}}
 {{{
 ansible-pull run on on mysql.int.buildbot.net failed.

 Starting ansible-pull at 2015-02-03 21:10:00
 localhost | FAILED >> {
     "failed": true,
     "msg": "Failed to download remote objects and refs"
 }
 }}}
 {{{
 ansible-pull run on on bslave1.int.buildbot.net failed.

 Starting ansible-pull at 2015-01-30 10:00:00
 Sleeping for 81 seconds...
 localhost | FAILED >> {
     "cmd": "/usr/local/bin/git ls-remote origin -h refs/heads/master",
     "failed": true,
     "msg": "fatal: Unable to look up github.com (port 9418) (hostname nor
 servname provided, or not known)",
     "rc": 128,
     "stderr": "fatal: Unable to look up github.com (port 9418) (hostname
 nor servname provided, or not known)\n",
     "stdout": ""
 }
 }}}

 Those are all fairly recent.  On January 22, we had a bunch of hosts (not
 all *.int.bb.net) have a similar problem, but maybe that was a fluke.

 I also see this quite often from Trac:

 > Warning: The change has been saved, but an error occurred while sending
 notifications: SMTP server connection error ([Errno 60] Operation timed
 out). Please modify [notification] smtp_server or [notification] smtp_port
 in your configuration.

 The latter means that we're probably losing some Trac notifications which
 is never good.

--
Ticket URL: <http://trac.buildbot.net/ticket/3183>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation