[Buildbot-commits] [Buildbot] #2369: Hard to deploy buildbot in virtualenv with pip without network connection

[Buildbot]

#2369: Hard to deploy buildbot in virtualenv with pip without network connection
-------------------+-----------------------
Reporter:  dank    |      Owner:
    Type:  defect  |     Status:  new
Priority:  minor   |  Milestone:  undecided
 Version:  master  |   Keywords:
-------------------+-----------------------
 I'm using pip -emaster to deploy from source (since I'm trying to
 help test before release), and this is complicated on isolated
 machines by buildbot-master's use of the "setup_requires" keyword arg in
 its setup.py.

 I installed python and virtualenv with
   sudo apt-get install python-dev python-virtualenv
 and am using the commands
     $ PIP_CACHE=$HOME/pip_cache
     $ virtualenv --no-site-packages sandbox
     $ cd sandbox
     $ . bin/activate
     $ pip install --no-install --use-mirrors -I --download=$PIP_CACHE
 -emaster
 to install buildbot from source while saving downloaded packages.
 Then, on an isolated host, after copying ~/pip_cache, I run
     $ PIP_CACHE=$HOME/pip_cache
     $ virtualenv --no-site-packages sandbox
     $ cd sandbox
     $ . bin/activate
     $ pip install --find-links=file://$PIP_CACHE --no-index --index-
 url=file:///dev/null -emaster
 in an attempt to get rid of the need for an internet connection.
 ( As recommended in the second answer in
 stackoverflow.com/questions/4806448/how-do-i-install-from-a-local-cache-
 with-pip )

 This does fill the cache directory with some tarballs,
 but sadly, installing on the isolated system fails with

   Running setup.py egg_info for package from file:///home/buildbot/master-
 state/sandbox/buildbot-git/master
     Download error on http://pypi.python.org/simple/setuptools_trial/:
 [Errno 101] Network is unreachable -- Some packages may not be found!
 ...
     distutils.errors.DistutilsError: Could not find suitable distribution
 for Requirement.parse('setuptools-trial')

 According to the virtualenv guys:
 -- snip --

 This problem is caused by buildbot-master's use of the "setup_requires"
 keyword arg in its setup.py. This kwarg causes setuptools to try to go
 download the listed dependency immediately from PyPI whenever setup.py
 is executed, without going through pip's package-finder at all. It's
 quite difficult for pip to intervene here, since pip has to execute
 setup.py in order to even obtain the most basic metadata about a package.

 More explanation here: github.com/pypa/pip/issues/410

 That issue also has a workaround that might help you - if a .egg file
 for the named setup_requires dependency is already present in the
 directory next to the setup.py that requires it, setuptools will use
 that and not go to PyPI. So you should be able to just add a
 setuptools-trial egg manually in master/ next to setup.py and avoid the
 network access. (To be honest, I would recommend to the authors of
 buildbot to simply check a setuptools-trial egg into git at that
 location and pre-empt this problem for all users). "

 -- snip --

 I haven't tried the workaround yet (not sure what a setuptools-trial
 egg is or where I'd get one :-).

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2369>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation