[Buildbot-devel] GSoC: provide forward compatibility for Python 3

Fri Apr 10 15:53:52 UTC 2015

If by "framework" you mean something like asyncio or gevent, that's
not an option that's on the table.

If you mean using a different master/slave protocol, aside from PB,
then that is (sort of) on the table.  I wouldn't expect a student to
work on completing the implementation of a new master/slave protocol.
However, the following is a legitimate plan: port everything *except*
the master/slave protocol to Python3.  Then when we have a new
master/slave protocol available, users who wish to run in Python3 can
use that protocol; users who wish to user perspective broker will need
to use Python2.

I have a little more data to add to the conversation, based on a
hallway conversation with Glyph here at PyCon in Montreal.  There's
nothing particularly resistant to Python3 in Perspective Broker -- the
work just hasn't been done yet.  Likely issues will include
bytes/strings and some confusion between tuples and namedtuples.

However, the other major component of Twisted that this project would
depend on is the test runner, trial. Trial is not yet able to run on
Python3, although HawkOwl has recently completed some of the blocking
issues to porting Trial.  This means that running Buildbot's unit
tests in Python3 could be difficult, and may require porting Trial.

I don't think any of this means that the overall project can't be made
suitable for GSoC -- just that the project will need to be carefully
re-scoped with the mentor once some of these questions are answered.
If the project ends up accomplishing only some basic syntax fixes to
Buildbot, plus some patches to Twisted, but doesn't get to the point
where Buildbot runs on Python3, I would still consider that a wild
success (as would Twisted).

Dustin

On Wed, Apr 1, 2015 at 11:04 PM, Vasily Kuz'min <vasily at freedom-web.ru> wrote:
> I will speak plainly: this project was originally called "provide
> forward compatibility for Python 3", which meant rewriting the source
> code with the new syntax. Now it turns out that it is necessary to
> change the framework, which creates additional problems. As already
> remarked Pierre Tardy, you must divide the project into subprojects,
> and divided to different people, so that some have been porting
> Buildbot to another framework, while others rewriting code under
> Python 3.
>
> I would choose the latter...
>
> 2015-04-01 10:13 GMT+03:00 Jørn Lomax <northlomax at gmail.com>:
>>
>>
>> On 30.03.2015 18:19, Pierre Tardy wrote:
>>
>> Another point is that pb does not have a bright future in buildbot. It is
>> used for slave/master communication. We have a plan to switch to something
>> more modern for slave/master communication. I have a Poc using wamp and
>> crossbar.io, we could easily adapt PoC that to amqp, or amp.
>>
>>
>> If this is the case, then why not make this a part of GSoC? If buildbot can
>> be ported to use crossbar.io/AutoBahn which is python3 compatible, I think
>> that would be a better solution than spending time on twisted if it's going
>> to be dropped in the future anyway.
>>
>> So for the py3k work I think we can just drop the pb part our of the
>> problem. There are plenty of work apart from that.
>>
>> I would estimate that py3k support for twisted's pb is a full GSoC by
>> itself. So this might also be a solution to share work between you guys.
>>
>> There are currently four py3k proposals this year, several of them looks
>> good for me, but there will be only one person of the projet, this is a
>> clear rule from Google.
>>
>> The only way we can take several student on py3k is to split the project
>> into clearly separated subprojects (also, provided we have enough mentors,
>> and funds from Google)
>>
>> I would suggest for the students to take the opportunity of this thread to
>> agree on who does what
>>
>> Do you think you will get enough slots for this? I don't mean to sound
>> negative to the idea (I'm not), but I would imagine it would be better to
>> use the slots you get for other project ideas as well, and not just py3
>> porting.
>>
>>
>> Regards,
>> Pierre
>>
>> Le lun. 30 mars 2015 à 08:46, Jørn Lomax <northlomax at gmail.com> a écrit :
>>>
>>> On 30.03.2015 06:16, Vasily Kuz'min wrote:
>>> > In my opinion, at first it is necessary to complete 40% of the
>>> > Twisted, and only then to start the main objective. On an evaluation
>>> > of alternative options some time will be required
>>> >
>>> I think it would be more important to look at the parts of twisted that
>>> buildbot relies the most on, which is the Prerspective Broker proker
>>> (pb). There are some parts of twisted buildbot never uses, so I don't
>>> think it should be within the scope of the project to port *all* of
>>> twisted. I think that would be an entire twisted GSoC project in itself.
>>>
>>> Unfortunately, pb is not a part of twisted I have much experience with
>>> so it's hard to say how hard it would be to port it to python3. Looking
>>> briefly at the source code it looks achvievable to do as part of the
>>> project, but maybe someone with more experience with the pb module have
>>> a better idea exactly how much work it would take
>>>
>>> regards,
>>> Jørn Lomax
>>> > 2015-03-30 2:10 GMT+03:00 Bob Hood <bhood2 at comcast.net>:
>>> >> On 3/29/2015 3:29 PM, Vasily Kuz'min wrote:
>>> >>> Hi Dustin,
>>> >>>
>>> >>> thank you for your reply. I've uploaded my proposal as PDF to my
>>> >>> server[0]
>>> >>> so that people who want could have a look. I've also attached it as
>>> >>> text
>>> >>> file.
>>> >>>
>>> >>> I'm quite unsure about the timeline part, since I have to better
>>> >>> understand the
>>> >>> structure of the source code before I can give my time estimation. I
>>> >>> think
>>> >>> I'll
>>> >>> have to correct it once I'll understand source code structure.
>>> >>>
>>> >>> As for Twisted, it seems that they mention several issues on the
>>> >>> corresponding
>>> >>> web page[0]. I doubt that's all that we should care about: milestone
>>> >>> for
>>> >>> Py3K
>>> >>> shows that Twisted is only 60% ready[2]. I think it means that we may
>>> >>> face
>>> >>> other problems except those listed. I don't think that will be a real
>>> >>> problem
>>> >>> though; moreover, I tend to think that means I can help both buildbot
>>> >>> and
>>> >>> Twisted.
>>> >>>
>>> >>> Generally speaking, we have two ways: create various work-arounds for
>>> >>> buildbot
>>> >>> or correct these issues in Twisted directly. The first is faster but I
>>> >>> think
>>> >>> it is incorrect, because it'll introduce backward incompatibilities
>>> >>> later
>>> >>> when
>>> >>> Twisted will get updated its APIs. What I really think about is to
>>> >>> collaborate
>>> >>> with Twisted guys. I tend to think that the second way is better, but
>>> >>> of
>>> >>> course
>>> >>> we shall concentrate on things related to buildbot. However, I'd like
>>> >>> to
>>> >>> know
>>> >>> your opinions.
>>> >>>
>>> >>> [0] http://derzsky.freedom-web.ru/proposal.pdf
>>> >>> [1] https://twistedmatrix.com/documents/14.0.0/core/howto/python3.html
>>> >>> [2] https://twistedmatrix.com/trac/milestone/Python-3.x
>>> >>
>>> >> Hello, Vasily!  Thanks for your proposal.
>>> >>
>>> >> I'm a bit concerned about the Twisted framework. Being 60% done means
>>> >> it's
>>> >> 40% incomplete, and that is a large risk to your success.  As you've
>>> >> pointed
>>> >> out, working on Twisted yourself is certainly one option, but that then
>>> >> takes time away from your focus on Buildbot.  A completed Twisted
>>> >> framework
>>> >> for Python 3.0 is a dependency to successfully completing your
>>> >> proposal, and
>>> >> on that page you provide, it actually says "No date set" for
>>> >> completion.
>>> >>
>>> >> In light of all this (which would be setting off alarms and Big Red
>>> >> Flags to
>>> >> a project manager), have you given any thought to alternatives?  I
>>> >> recall
>>> >> reading about 'asyncio' being a viable alterntive to Twisted in Python
>>> >> v3.
>>> >> Do you think it would be wise to have a "plan b" available in case
>>> >> Twisted
>>> >> is not fully ported to Python v3 by the time your project needs to
>>> >> complete?
>>> >> I'm not saying Dustin would accept that ( ;) ), but it would show
>>> >> recognition of that dependency and its risk to successful completion.
>>> >>
>>> >> Also, the schedule you include in your proposal could probably benefit
>>> >> from
>>> >> some additional detail.  Having a four-month block that only says
>>> >> "Reimplementing the core functionality" is quite nebulous and doesn't
>>> >> really
>>> >> offer anything from which a metric of task success can be measured.  I
>>> >> understand that you are not really familiar with the Buildbot code
>>> >> base, but
>>> >> I'm sure there must be some higher-level tasks that could be listed
>>> >> with
>>> >> time estimates, e.g.: "Familiarize myself with Buildbot's design - 1
>>> >> week";
>>> >> "Evaluate 'asyncio' as an alternative to the Twisted framework - 1
>>> >> week";
>>> >> etc.  There needs to be goals and milestones so the health of your
>>> >> progress
>>> >> on the task can be monitored.
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub for
>>> all
>>> things parallel software development, from weekly thought leadership blogs
>>> to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>>
>>
>
>
>
> --
> С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН
>
> Телефон:      +7 (999) 988-00-85
> E-mail:           vasily at freedom-web.ru
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel