[Buildbot-commits] [Buildbot] #2369: Hard to deploy buildbot in virtualenv with pip without network connection

[Buildbot]

#2369: Hard to deploy buildbot in virtualenv with pip without network connection
-------------------+--------------------
Reporter:  dank    |       Owner:
    Type:  defect  |      Status:  new
Priority:  minor   |   Milestone:  0.8.7
 Version:  master  |  Resolution:
Keywords:          |
-------------------+--------------------
Changes (by tom.prince):

 * milestone:  undecided => 0.8.7


Old description:

> I'm using pip -emaster to deploy from source (since I'm trying to
> help test before release), and this is complicated on isolated
> machines by buildbot-master's use of the "setup_requires" keyword arg in
> its setup.py.
>
> I installed python and virtualenv with
>   sudo apt-get install python-dev python-virtualenv
> and am using the commands
>     $ PIP_CACHE=$HOME/pip_cache
>     $ virtualenv --no-site-packages sandbox
>     $ cd sandbox
>     $ . bin/activate
>     $ pip install --no-install --use-mirrors -I --download=$PIP_CACHE
> -emaster
> to install buildbot from source while saving downloaded packages.
> Then, on an isolated host, after copying ~/pip_cache, I run
>     $ PIP_CACHE=$HOME/pip_cache
>     $ virtualenv --no-site-packages sandbox
>     $ cd sandbox
>     $ . bin/activate
>     $ pip install --find-links=file://$PIP_CACHE --no-index --index-
> url=file:///dev/null -emaster
> in an attempt to get rid of the need for an internet connection.
> ( As recommended in the second answer in
> stackoverflow.com/questions/4806448/how-do-i-install-from-a-local-cache-
> with-pip )
>
> This does fill the cache directory with some tarballs,
> but sadly, installing on the isolated system fails with
>
>   Running setup.py egg_info for package from file:///home/buildbot
> /master-state/sandbox/buildbot-git/master
>     Download error on http://pypi.python.org/simple/setuptools_trial/:
> [Errno 101] Network is unreachable -- Some packages may not be found!
> ...
>     distutils.errors.DistutilsError: Could not find suitable distribution
> for Requirement.parse('setuptools-trial')
>
> According to the virtualenv guys:
> -- snip --
>
> This problem is caused by buildbot-master's use of the "setup_requires"
> keyword arg in its setup.py. This kwarg causes setuptools to try to go
> download the listed dependency immediately from PyPI whenever setup.py
> is executed, without going through pip's package-finder at all. It's
> quite difficult for pip to intervene here, since pip has to execute
> setup.py in order to even obtain the most basic metadata about a package.
>
> More explanation here: github.com/pypa/pip/issues/410
>
> That issue also has a workaround that might help you - if a .egg file
> for the named setup_requires dependency is already present in the
> directory next to the setup.py that requires it, setuptools will use
> that and not go to PyPI. So you should be able to just add a
> setuptools-trial egg manually in master/ next to setup.py and avoid the
> network access. (To be honest, I would recommend to the authors of
> buildbot to simply check a setuptools-trial egg into git at that
> location and pre-empt this problem for all users). "
>
> -- snip --
>
> I haven't tried the workaround yet (not sure what a setuptools-trial
> egg is or where I'd get one :-).

New description:

 I'm using pip -emaster to deploy from source (since I'm trying to
 help test before release), and this is complicated on isolated
 machines by buildbot-master's use of the "setup_requires" keyword arg in
 its setup.py.

 I installed python and virtualenv with
   sudo apt-get install python-dev python-virtualenv
 and am using the commands
 {{{
 $ PIP_CACHE=$HOME/pip_cache
 $ virtualenv --no-site-packages sandbox
 $ cd sandbox
 $ . bin/activate
 $ pip install --no-install --use-mirrors -I --download=$PIP_CACHE -emaster
 }}}
 to install buildbot from source while saving downloaded packages.
 Then, on an isolated host, after copying ~/pip_cache, I run
 {{{
 $ PIP_CACHE=$HOME/pip_cache
 $ virtualenv --no-site-packages sandbox
 $ cd sandbox
 $ . bin/activate
 $ pip install --find-links=file://$PIP_CACHE --no-index --index-
 url=file:///dev/null -emaster
 }}}
 in an attempt to get rid of the need for an internet connection.
 ( As recommended in the second answer in
 http://stackoverflow.com/questions/4806448/how-do-i-install-from-a-local-
 cache-with-pip )

 This does fill the cache directory with some tarballs,
 but sadly, installing on the isolated system fails with
 {{{
 Running setup.py egg_info for package from file:///home/buildbot/master-
 state/sandbox/buildbot-git/master
   Download error on http://pypi.python.org/simple/setuptools_trial/:
 [Errno 101] Network is unreachable -- Some packages may not be found!
 ...
   distutils.errors.DistutilsError: Could not find suitable distribution
 for Requirement.parse('setuptools-trial')
 }}}

 According to the virtualenv guys:
 -- snip --

 This problem is caused by buildbot-master's use of the "setup_requires"
 keyword arg in its setup.py. This kwarg causes setuptools to try to go
 download the listed dependency immediately from PyPI whenever setup.py
 is executed, without going through pip's package-finder at all. It's
 quite difficult for pip to intervene here, since pip has to execute
 setup.py in order to even obtain the most basic metadata about a package.

 More explanation here: github.com/pypa/pip/issues/410

 That issue also has a workaround that might help you - if a .egg file
 for the named setup_requires dependency is already present in the
 directory next to the setup.py that requires it, setuptools will use
 that and not go to PyPI. So you should be able to just add a
 setuptools-trial egg manually in master/ next to setup.py and avoid the
 network access. (To be honest, I would recommend to the authors of
 buildbot to simply check a setuptools-trial egg into git at that
 location and pre-empt this problem for all users). "

 -- snip --

 I haven't tried the workaround yet (not sure what a setuptools-trial
 egg is or where I'd get one :-).

--

Comment:

 This is caused by https://github.com/buildbot/buildbot/pull/408

 My inclination would be to revert, since there are already reasonable ways
 to run the tests.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2369#comment:1>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation