[Buildbot-devel] GSoC: provide forward compatibility for Python 3

Tue Apr 14 00:42:57 UTC 2015

GSoC's rules don't stipulate much about language syntax.

Because Twisted is also an eligible open-source project, Google will
have no problem with a student contributing to that project throughout
the summer.

Dustin

On Mon, Apr 13, 2015 at 4:11 PM, Vasily Kuz'min <vasily at freedom-web.ru> wrote:
> That is, the code can be written in theory in accordance with the
> syntax of the language, this corresponds to rules GSoC. The work
> itself Buildbot will we see only after porting dependency on Python 3
>
> 2015-04-10 18:53 GMT+03:00 Dustin J. Mitchell <dustin at v.igoro.us>:
>> If by "framework" you mean something like asyncio or gevent, that's
>> not an option that's on the table.
>>
>> If you mean using a different master/slave protocol, aside from PB,
>> then that is (sort of) on the table.  I wouldn't expect a student to
>> work on completing the implementation of a new master/slave protocol.
>> However, the following is a legitimate plan: port everything *except*
>> the master/slave protocol to Python3.  Then when we have a new
>> master/slave protocol available, users who wish to run in Python3 can
>> use that protocol; users who wish to user perspective broker will need
>> to use Python2.
>>
>> I have a little more data to add to the conversation, based on a
>> hallway conversation with Glyph here at PyCon in Montreal.  There's
>> nothing particularly resistant to Python3 in Perspective Broker -- the
>> work just hasn't been done yet.  Likely issues will include
>> bytes/strings and some confusion between tuples and namedtuples.
>>
>> However, the other major component of Twisted that this project would
>> depend on is the test runner, trial. Trial is not yet able to run on
>> Python3, although HawkOwl has recently completed some of the blocking
>> issues to porting Trial.  This means that running Buildbot's unit
>> tests in Python3 could be difficult, and may require porting Trial.
>>
>> I don't think any of this means that the overall project can't be made
>> suitable for GSoC -- just that the project will need to be carefully
>> re-scoped with the mentor once some of these questions are answered.
>> If the project ends up accomplishing only some basic syntax fixes to
>> Buildbot, plus some patches to Twisted, but doesn't get to the point
>> where Buildbot runs on Python3, I would still consider that a wild
>> success (as would Twisted).
>>
>> Dustin
>>
>> On Wed, Apr 1, 2015 at 11:04 PM, Vasily Kuz'min <vasily at freedom-web.ru> wrote:
>>> I will speak plainly: this project was originally called "provide
>>> forward compatibility for Python 3", which meant rewriting the source
>>> code with the new syntax. Now it turns out that it is necessary to
>>> change the framework, which creates additional problems. As already
>>> remarked Pierre Tardy, you must divide the project into subprojects,
>>> and divided to different people, so that some have been porting
>>> Buildbot to another framework, while others rewriting code under
>>> Python 3.
>>>
>>> I would choose the latter...
>>>
>>> 2015-04-01 10:13 GMT+03:00 Jørn Lomax <northlomax at gmail.com>:
>>>>
>>>>
>>>> On 30.03.2015 18:19, Pierre Tardy wrote:
>>>>
>>>> Another point is that pb does not have a bright future in buildbot. It is
>>>> used for slave/master communication. We have a plan to switch to something
>>>> more modern for slave/master communication. I have a Poc using wamp and
>>>> crossbar.io, we could easily adapt PoC that to amqp, or amp.
>>>>
>>>>
>>>> If this is the case, then why not make this a part of GSoC? If buildbot can
>>>> be ported to use crossbar.io/AutoBahn which is python3 compatible, I think
>>>> that would be a better solution than spending time on twisted if it's going
>>>> to be dropped in the future anyway.
>>>>
>>>> So for the py3k work I think we can just drop the pb part our of the
>>>> problem. There are plenty of work apart from that.
>>>>
>>>> I would estimate that py3k support for twisted's pb is a full GSoC by
>>>> itself. So this might also be a solution to share work between you guys.
>>>>
>>>> There are currently four py3k proposals this year, several of them looks
>>>> good for me, but there will be only one person of the projet, this is a
>>>> clear rule from Google.
>>>>
>>>> The only way we can take several student on py3k is to split the project
>>>> into clearly separated subprojects (also, provided we have enough mentors,
>>>> and funds from Google)
>>>>
>>>> I would suggest for the students to take the opportunity of this thread to
>>>> agree on who does what
>>>>
>>>> Do you think you will get enough slots for this? I don't mean to sound
>>>> negative to the idea (I'm not), but I would imagine it would be better to
>>>> use the slots you get for other project ideas as well, and not just py3
>>>> porting.
>>>>
>>>>
>>>> Regards,
>>>> Pierre
>>>>
>>>> Le lun. 30 mars 2015 à 08:46, Jørn Lomax <northlomax at gmail.com> a écrit :
>>>>>
>>>>> On 30.03.2015 06:16, Vasily Kuz'min wrote:
>>>>> > In my opinion, at first it is necessary to complete 40% of the
>>>>> > Twisted, and only then to start the main objective. On an evaluation
>>>>> > of alternative options some time will be required
>>>>> >
>>>>> I think it would be more important to look at the parts of twisted that
>>>>> buildbot relies the most on, which is the Prerspective Broker proker
>>>>> (pb). There are some parts of twisted buildbot never uses, so I don't
>>>>> think it should be within the scope of the project to port *all* of
>>>>> twisted. I think that would be an entire twisted GSoC project in itself.
>>>>>
>>>>> Unfortunately, pb is not a part of twisted I have much experience with
>>>>> so it's hard to say how hard it would be to port it to python3. Looking
>>>>> briefly at the source code it looks achvievable to do as part of the
>>>>> project, but maybe someone with more experience with the pb module have
>>>>> a better idea exactly how much work it would take
>>>>>
>>>>> regards,
>>>>> Jørn Lomax
>>>>> > 2015-03-30 2:10 GMT+03:00 Bob Hood <bhood2 at comcast.net>:
>>>>> >> On 3/29/2015 3:29 PM, Vasily Kuz'min wrote:
>>>>> >>> Hi Dustin,
>>>>> >>>
>>>>> >>> thank you for your reply. I've uploaded my proposal as PDF to my
>>>>> >>> server[0]
>>>>> >>> so that people who want could have a look. I've also attached it as
>>>>> >>> text
>>>>> >>> file.
>>>>> >>>
>>>>> >>> I'm quite unsure about the timeline part, since I have to better
>>>>> >>> understand the
>>>>> >>> structure of the source code before I can give my time estimation. I
>>>>> >>> think
>>>>> >>> I'll
>>>>> >>> have to correct it once I'll understand source code structure.
>>>>> >>>
>>>>> >>> As for Twisted, it seems that they mention several issues on the
>>>>> >>> corresponding
>>>>> >>> web page[0]. I doubt that's all that we should care about: milestone
>>>>> >>> for
>>>>> >>> Py3K
>>>>> >>> shows that Twisted is only 60% ready[2]. I think it means that we may
>>>>> >>> face
>>>>> >>> other problems except those listed. I don't think that will be a real
>>>>> >>> problem
>>>>> >>> though; moreover, I tend to think that means I can help both buildbot
>>>>> >>> and
>>>>> >>> Twisted.
>>>>> >>>
>>>>> >>> Generally speaking, we have two ways: create various work-arounds for
>>>>> >>> buildbot
>>>>> >>> or correct these issues in Twisted directly. The first is faster but I
>>>>> >>> think
>>>>> >>> it is incorrect, because it'll introduce backward incompatibilities
>>>>> >>> later
>>>>> >>> when
>>>>> >>> Twisted will get updated its APIs. What I really think about is to
>>>>> >>> collaborate
>>>>> >>> with Twisted guys. I tend to think that the second way is better, but
>>>>> >>> of
>>>>> >>> course
>>>>> >>> we shall concentrate on things related to buildbot. However, I'd like
>>>>> >>> to
>>>>> >>> know
>>>>> >>> your opinions.
>>>>> >>>
>>>>> >>> [0] http://derzsky.freedom-web.ru/proposal.pdf
>>>>> >>> [1] https://twistedmatrix.com/documents/14.0.0/core/howto/python3.html
>>>>> >>> [2] https://twistedmatrix.com/trac/milestone/Python-3.x
>>>>> >>
>>>>> >> Hello, Vasily!  Thanks for your proposal.
>>>>> >>
>>>>> >> I'm a bit concerned about the Twisted framework. Being 60% done means
>>>>> >> it's
>>>>> >> 40% incomplete, and that is a large risk to your success.  As you've
>>>>> >> pointed
>>>>> >> out, working on Twisted yourself is certainly one option, but that then
>>>>> >> takes time away from your focus on Buildbot.  A completed Twisted
>>>>> >> framework
>>>>> >> for Python 3.0 is a dependency to successfully completing your
>>>>> >> proposal, and
>>>>> >> on that page you provide, it actually says "No date set" for
>>>>> >> completion.
>>>>> >>
>>>>> >> In light of all this (which would be setting off alarms and Big Red
>>>>> >> Flags to
>>>>> >> a project manager), have you given any thought to alternatives?  I
>>>>> >> recall
>>>>> >> reading about 'asyncio' being a viable alterntive to Twisted in Python
>>>>> >> v3.
>>>>> >> Do you think it would be wise to have a "plan b" available in case
>>>>> >> Twisted
>>>>> >> is not fully ported to Python v3 by the time your project needs to
>>>>> >> complete?
>>>>> >> I'm not saying Dustin would accept that ( ;) ), but it would show
>>>>> >> recognition of that dependency and its risk to successful completion.
>>>>> >>
>>>>> >> Also, the schedule you include in your proposal could probably benefit
>>>>> >> from
>>>>> >> some additional detail.  Having a four-month block that only says
>>>>> >> "Reimplementing the core functionality" is quite nebulous and doesn't
>>>>> >> really
>>>>> >> offer anything from which a metric of task success can be measured.  I
>>>>> >> understand that you are not really familiar with the Buildbot code
>>>>> >> base, but
>>>>> >> I'm sure there must be some higher-level tasks that could be listed
>>>>> >> with
>>>>> >> time estimates, e.g.: "Familiarize myself with Buildbot's design - 1
>>>>> >> week";
>>>>> >> "Evaluate 'asyncio' as an alternative to the Twisted framework - 1
>>>>> >> week";
>>>>> >> etc.  There needs to be goals and milestones so the health of your
>>>>> >> progress
>>>>> >> on the task can be monitored.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>>>> sponsored
>>>>> by Intel and developed in partnership with Slashdot Media, is your hub for
>>>>> all
>>>>> things parallel software development, from weekly thought leadership blogs
>>>>> to
>>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>>> conversation now. http://goparallel.sourceforge.net/
>>>>> _______________________________________________
>>>>> Buildbot-devel mailing list
>>>>> Buildbot-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН
>>>
>>> Телефон:      +7 (999) 988-00-85
>>> E-mail:           vasily at freedom-web.ru
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>>> things parallel software development, from weekly thought leadership blogs to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
>
> --
> С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН
>
> Телефон:      +7 (999) 988-00-85
> E-mail:           vasily at freedom-web.ru