[Buildbot-devel] GSoC: provide forward compatibility for Python 3

Wed Apr 15 01:52:48 UTC 2015

Ok. I understood

2015-04-14 3:42 GMT+03:00 Dustin J. Mitchell <dustin at v.igoro.us>:
> GSoC's rules don't stipulate much about language syntax.
>
> Because Twisted is also an eligible open-source project, Google will
> have no problem with a student contributing to that project throughout
> the summer.
>
> Dustin
>
> On Mon, Apr 13, 2015 at 4:11 PM, Vasily Kuz'min <vasily at freedom-web.ru> wrote:
>> That is, the code can be written in theory in accordance with the
>> syntax of the language, this corresponds to rules GSoC. The work
>> itself Buildbot will we see only after porting dependency on Python 3
>>
>> 2015-04-10 18:53 GMT+03:00 Dustin J. Mitchell <dustin at v.igoro.us>:
>>> If by "framework" you mean something like asyncio or gevent, that's
>>> not an option that's on the table.
>>>
>>> If you mean using a different master/slave protocol, aside from PB,
>>> then that is (sort of) on the table.  I wouldn't expect a student to
>>> work on completing the implementation of a new master/slave protocol.
>>> However, the following is a legitimate plan: port everything *except*
>>> the master/slave protocol to Python3.  Then when we have a new
>>> master/slave protocol available, users who wish to run in Python3 can
>>> use that protocol; users who wish to user perspective broker will need
>>> to use Python2.
>>>
>>> I have a little more data to add to the conversation, based on a
>>> hallway conversation with Glyph here at PyCon in Montreal.  There's
>>> nothing particularly resistant to Python3 in Perspective Broker -- the
>>> work just hasn't been done yet.  Likely issues will include
>>> bytes/strings and some confusion between tuples and namedtuples.
>>>
>>> However, the other major component of Twisted that this project would
>>> depend on is the test runner, trial. Trial is not yet able to run on
>>> Python3, although HawkOwl has recently completed some of the blocking
>>> issues to porting Trial.  This means that running Buildbot's unit
>>> tests in Python3 could be difficult, and may require porting Trial.
>>>
>>> I don't think any of this means that the overall project can't be made
>>> suitable for GSoC -- just that the project will need to be carefully
>>> re-scoped with the mentor once some of these questions are answered.
>>> If the project ends up accomplishing only some basic syntax fixes to
>>> Buildbot, plus some patches to Twisted, but doesn't get to the point
>>> where Buildbot runs on Python3, I would still consider that a wild
>>> success (as would Twisted).
>>>
>>> Dustin
>>>
>>> On Wed, Apr 1, 2015 at 11:04 PM, Vasily Kuz'min <vasily at freedom-web.ru> wrote:
>>>> I will speak plainly: this project was originally called "provide
>>>> forward compatibility for Python 3", which meant rewriting the source
>>>> code with the new syntax. Now it turns out that it is necessary to
>>>> change the framework, which creates additional problems. As already
>>>> remarked Pierre Tardy, you must divide the project into subprojects,
>>>> and divided to different people, so that some have been porting
>>>> Buildbot to another framework, while others rewriting code under
>>>> Python 3.
>>>>
>>>> I would choose the latter...
>>>>
>>>> 2015-04-01 10:13 GMT+03:00 Jørn Lomax <northlomax at gmail.com>:
>>>>>
>>>>>
>>>>> On 30.03.2015 18:19, Pierre Tardy wrote:
>>>>>
>>>>> Another point is that pb does not have a bright future in buildbot. It is
>>>>> used for slave/master communication. We have a plan to switch to something
>>>>> more modern for slave/master communication. I have a Poc using wamp and
>>>>> crossbar.io, we could easily adapt PoC that to amqp, or amp.
>>>>>
>>>>>
>>>>> If this is the case, then why not make this a part of GSoC? If buildbot can
>>>>> be ported to use crossbar.io/AutoBahn which is python3 compatible, I think
>>>>> that would be a better solution than spending time on twisted if it's going
>>>>> to be dropped in the future anyway.
>>>>>
>>>>> So for the py3k work I think we can just drop the pb part our of the
>>>>> problem. There are plenty of work apart from that.
>>>>>
>>>>> I would estimate that py3k support for twisted's pb is a full GSoC by
>>>>> itself. So this might also be a solution to share work between you guys.
>>>>>
>>>>> There are currently four py3k proposals this year, several of them looks
>>>>> good for me, but there will be only one person of the projet, this is a
>>>>> clear rule from Google.
>>>>>
>>>>> The only way we can take several student on py3k is to split the project
>>>>> into clearly separated subprojects (also, provided we have enough mentors,
>>>>> and funds from Google)
>>>>>
>>>>> I would suggest for the students to take the opportunity of this thread to
>>>>> agree on who does what
>>>>>
>>>>> Do you think you will get enough slots for this? I don't mean to sound
>>>>> negative to the idea (I'm not), but I would imagine it would be better to
>>>>> use the slots you get for other project ideas as well, and not just py3
>>>>> porting.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Pierre
>>>>>
>>>>> Le lun. 30 mars 2015 à 08:46, Jørn Lomax <northlomax at gmail.com> a écrit :
>>>>>>
>>>>>> On 30.03.2015 06:16, Vasily Kuz'min wrote:
>>>>>> > In my opinion, at first it is necessary to complete 40% of the
>>>>>> > Twisted, and only then to start the main objective. On an evaluation
>>>>>> > of alternative options some time will be required
>>>>>> >
>>>>>> I think it would be more important to look at the parts of twisted that
>>>>>> buildbot relies the most on, which is the Prerspective Broker proker
>>>>>> (pb). There are some parts of twisted buildbot never uses, so I don't
>>>>>> think it should be within the scope of the project to port *all* of
>>>>>> twisted. I think that would be an entire twisted GSoC project in itself.
>>>>>>
>>>>>> Unfortunately, pb is not a part of twisted I have much experience with
>>>>>> so it's hard to say how hard it would be to port it to python3. Looking
>>>>>> briefly at the source code it looks achvievable to do as part of the
>>>>>> project, but maybe someone with more experience with the pb module have
>>>>>> a better idea exactly how much work it would take
>>>>>>
>>>>>> regards,
>>>>>> Jørn Lomax
>>>>>> > 2015-03-30 2:10 GMT+03:00 Bob Hood <bhood2 at comcast.net>:
>>>>>> >> On 3/29/2015 3:29 PM, Vasily Kuz'min wrote:
>>>>>> >>> Hi Dustin,
>>>>>> >>>
>>>>>> >>> thank you for your reply. I've uploaded my proposal as PDF to my
>>>>>> >>> server[0]
>>>>>> >>> so that people who want could have a look. I've also attached it as
>>>>>> >>> text
>>>>>> >>> file.
>>>>>> >>>
>>>>>> >>> I'm quite unsure about the timeline part, since I have to better
>>>>>> >>> understand the
>>>>>> >>> structure of the source code before I can give my time estimation. I
>>>>>> >>> think
>>>>>> >>> I'll
>>>>>> >>> have to correct it once I'll understand source code structure.
>>>>>> >>>
>>>>>> >>> As for Twisted, it seems that they mention several issues on the
>>>>>> >>> corresponding
>>>>>> >>> web page[0]. I doubt that's all that we should care about: milestone
>>>>>> >>> for
>>>>>> >>> Py3K
>>>>>> >>> shows that Twisted is only 60% ready[2]. I think it means that we may
>>>>>> >>> face
>>>>>> >>> other problems except those listed. I don't think that will be a real
>>>>>> >>> problem
>>>>>> >>> though; moreover, I tend to think that means I can help both buildbot
>>>>>> >>> and
>>>>>> >>> Twisted.
>>>>>> >>>
>>>>>> >>> Generally speaking, we have two ways: create various work-arounds for
>>>>>> >>> buildbot
>>>>>> >>> or correct these issues in Twisted directly. The first is faster but I
>>>>>> >>> think
>>>>>> >>> it is incorrect, because it'll introduce backward incompatibilities
>>>>>> >>> later
>>>>>> >>> when
>>>>>> >>> Twisted will get updated its APIs. What I really think about is to
>>>>>> >>> collaborate
>>>>>> >>> with Twisted guys. I tend to think that the second way is better, but
>>>>>> >>> of
>>>>>> >>> course
>>>>>> >>> we shall concentrate on things related to buildbot. However, I'd like
>>>>>> >>> to
>>>>>> >>> know
>>>>>> >>> your opinions.
>>>>>> >>>
>>>>>> >>> [0] http://derzsky.freedom-web.ru/proposal.pdf
>>>>>> >>> [1] https://twistedmatrix.com/documents/14.0.0/core/howto/python3.html
>>>>>> >>> [2] https://twistedmatrix.com/trac/milestone/Python-3.x
>>>>>> >>
>>>>>> >> Hello, Vasily!  Thanks for your proposal.
>>>>>> >>
>>>>>> >> I'm a bit concerned about the Twisted framework. Being 60% done means
>>>>>> >> it's
>>>>>> >> 40% incomplete, and that is a large risk to your success.  As you've
>>>>>> >> pointed
>>>>>> >> out, working on Twisted yourself is certainly one option, but that then
>>>>>> >> takes time away from your focus on Buildbot.  A completed Twisted
>>>>>> >> framework
>>>>>> >> for Python 3.0 is a dependency to successfully completing your
>>>>>> >> proposal, and
>>>>>> >> on that page you provide, it actually says "No date set" for
>>>>>> >> completion.
>>>>>> >>
>>>>>> >> In light of all this (which would be setting off alarms and Big Red
>>>>>> >> Flags to
>>>>>> >> a project manager), have you given any thought to alternatives?  I
>>>>>> >> recall
>>>>>> >> reading about 'asyncio' being a viable alterntive to Twisted in Python
>>>>>> >> v3.
>>>>>> >> Do you think it would be wise to have a "plan b" available in case
>>>>>> >> Twisted
>>>>>> >> is not fully ported to Python v3 by the time your project needs to
>>>>>> >> complete?
>>>>>> >> I'm not saying Dustin would accept that ( ;) ), but it would show
>>>>>> >> recognition of that dependency and its risk to successful completion.
>>>>>> >>
>>>>>> >> Also, the schedule you include in your proposal could probably benefit
>>>>>> >> from
>>>>>> >> some additional detail.  Having a four-month block that only says
>>>>>> >> "Reimplementing the core functionality" is quite nebulous and doesn't
>>>>>> >> really
>>>>>> >> offer anything from which a metric of task success can be measured.  I
>>>>>> >> understand that you are not really familiar with the Buildbot code
>>>>>> >> base, but
>>>>>> >> I'm sure there must be some higher-level tasks that could be listed
>>>>>> >> with
>>>>>> >> time estimates, e.g.: "Familiarize myself with Buildbot's design - 1
>>>>>> >> week";
>>>>>> >> "Evaluate 'asyncio' as an alternative to the Twisted framework - 1
>>>>>> >> week";
>>>>>> >> etc.  There needs to be goals and milestones so the health of your
>>>>>> >> progress
>>>>>> >> on the task can be monitored.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>>>>> sponsored
>>>>>> by Intel and developed in partnership with Slashdot Media, is your hub for
>>>>>> all
>>>>>> things parallel software development, from weekly thought leadership blogs
>>>>>> to
>>>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>>>> conversation now. http://goparallel.sourceforge.net/
>>>>>> _______________________________________________
>>>>>> Buildbot-devel mailing list
>>>>>> Buildbot-devel at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН
>>>>
>>>> Телефон:      +7 (999) 988-00-85
>>>> E-mail:           vasily at freedom-web.ru
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>>>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>>>> things parallel software development, from weekly thought leadership blogs to
>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>> conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>>
>>
>> --
>> С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН
>>
>> Телефон:      +7 (999) 988-00-85
>> E-mail:           vasily at freedom-web.ru

-- 
С УВАЖЕНИЕМ, ВАСИЛИЙ КУЗЬМИН

Телефон:      +7 (999) 988-00-85
E-mail:           vasily at freedom-web.ru