[devel at bb.net] Buildbot dependencies

[Povilas Kanapickas]

Hi,

On 2022-05-07 19:09, Denis Ovsienko wrote:
> Hello list.
> 
> Thank you for maintaining this project, which enables many open source
> projects to run CI without vendor lock-in.  In particular, you might be
> interested to know that it powers a significant portion of the
> multiple-OS and multiple-architecture CI for the software projects of
> the tcpdump group at https://ci.tcpdump.org/
> 
> Based on this experience I would like to provide some feedback about
> the pip packages of Buildbot.
> 
> First of all, the web site at buildbot.net used to say: "Buildbot is
> dropping support for Python 2.7 for the master.  Users are encouraged to
> migrate to Python 3 as soon as possible."
> 
> However, this notice seems to be out of date.  About a year ago I tried
> to setup a Buildbot worker on a Python 2.7 system via pip, and that
> turned out to be impossible even with much older package versions.  The
> matter was, one of the Buildbot worker key dependencies (twisted?) is
> not compatible with Python 2.7 anyway, so there is no support for
> Python 2.7 in the first place.  I guess the same likely applies to the
> master component.

We still test Python 2.7 on Buildbot CI, so it still works.

The following are the package versions used:
https://github.com/buildbot/buildbot/blob/master/requirements-ciworker.txt

> This peculiarity does not make much practical difference in my use case
> because all the current workers have Python 3, but perhaps you would
> like to rephrase the notice.
> 
> Another thing is, and it does have unwanted practical consequences, that
> the worker package starting with version 3.5.0 started to depend on the
> "cryptography" package, which is not just Python code, but a mix of
> Python and Rust, which requires a Rust compiler.

According to pydeptree the requirement is transitive and comes from
autobahn dependency. We started depending on autobahn to implement a
MessagePack-based worker-master communication protocol. This protocol
will make it possible to implement worker in other languages than python
which would make deployment much easier. E.g. a C++ based worker could
easily run on any ancient or weird operating system.

So we're essentially in the process of completely addressing this pain
point.

> The problem with the latter is that not every OS has a Rust compiler.
> Common and well-supported cases, such as Linux/AMD64, typically have
> Rust as a distribution package, so it can be installed with a single
> command in a couple minutes.  For other operating systems and/or
> architectures Rust is available only as a port, i.e. the Rust compiler
> itself must be compiled from its source first.  This is sometimes
> difficult.
> 
> As one example, on NetBSD/AArch64 it takes 3 days to compile a number
> of packages (including Clang and GCC) from scratch.  The same system
> spent about 7 days trying to compile Rust before I ran out of patience.
> That's even if the Rust port is readily available.  The additional Rust
> luggage is very unpracticable, especially for the functionality that is
> unused in the first place.
> 
> The matter is, my setup very well handles all cryptography completely
> outside of the Python space (both at the worker end and the master
> end), so until version 3.5.0 it was completely fine not to install the
> Python/Rust "cryptography".  Unfortunately, the additional new
> dependency in version 3.5.0 has broken the modus operandi.  Could
> anyone explain the reason for the additional dependency?  Would it be
> OK to remove the new dependency and to make the worker package like it
> used to be before?

Autobahn requires cryptography >=3.4.6, so you could install an older
package without a rust dependency. I haven't tried, if it doesn't work,
please let us know and we'll think about possible solutions.

Regards,
Povilas