[users at bb.net] Accessing properties within an extract_fn

Wed Jan 4 06:35:41 UTC 2017

Thanks Pierre.  I would have used setProperty, but I find it rather opaque
which parameters can have a function passed instead of a value, and which
cannot, and furthermore if that function would be passed the set of
properties or not.

I suggest you use the actual working version of the python function
<https://github.com/staticfloat/julia-buildbot/blob/0bfe31a611004c794eb0e5c673309d5476f682c4/master/package.py#L45-L77>,
it took me a little while to figure out how to finagle the data to flow
properly, and I wouldn't want to lead anyone astray.
-E

On Tue, Jan 3, 2017 at 5:23 AM, Pierre Tardy <tardyp at gmail.com> wrote:

> Please see:
> https://github.com/buildbot/buildbot/pull/2554/files
>
> I hope you don't mind, I took your artifact name function as an concrete
> example.
>
> Regards,
> Pierre
>
> Le mar. 3 janv. 2017 à 11:42, Pierre Tardy <tardyp at gmail.com> a écrit :
>
>> Hi Elliot,
>>
>> getRenderingFor is indeed returning a string, but via deferred (even if
>> your actual usecase does not really need io, the generic renderable case
>> may).
>>
>> So if you insist in using Interpolate to build your string, you will need
>> to make you renderer function an inlineCallbacks, and use
>>
>> local_filename = yield local_filename.getRenderingFor(props)
>>
>>
>> That said, I would rather use string.format as you are only doing "prop"
>> interpolation e.g:
>>
>>             local_filename = "contrib/mac/app/Julia-{versio
>> n}-{shortcommit}.{os_pkg_ext}".format(props.getProperties())
>>
>> This way you dont need to bother about deferred
>>
>> I like you proposal of manipulateProperties with full renderable
>> implementation will give it a try soon.
>>
>> Pierre
>>
>> Le mar. 3 janv. 2017 à 11:21, Elliot Saba <staticfloat at gmail.com> a
>> écrit :
>>
>> I would love a manipulateProperties, so that I don't have to use a
>> workaround like this.  Something that would allow me to return a dictionary
>> of new property values (akin to an extract_fn) would be ideal.  Although
>> I should say, I'm having some troubles with the workaround right now
>> because I'm not entirely certain how to get util.Interpolate() and
>> setProperty() to play nicely together.  I have this snippet
>> <https://github.com/staticfloat/julia-buildbot/blob/57258667835271c27aba559033898d0000f623ad/master/package.py#L45-L75>
>> right now, but I'm getting errors like this:
>>
>> twisted.internet.defer.FirstError: FirstError[#14, [Failure instance:
>> Traceback: <type 'exceptions.TypeError'>: <DeferredList at 0x7f9f3d3d9638
>> current result: u'tar.gz'> is not JSON serializable
>> /usr/local/lib/python2.7/site-packages/twisted/internet/defer.py:1299:_inlineCallbacks
>> /usr/local/lib/python2.7/site-packages/buildbot/process/buildstep.py:508:startStep
>> /usr/local/lib/python2.7/site-packages/buildbot/process/properties.py:198:render
>> /usr/local/lib/python2.7/site-packages/buildbot/process/properties.py:157:render
>> --- <exception caught here> --- /usr/local/lib/python2.7/site-
>> packages/twisted/internet/defer.py:150:maybeDeferred
>> package.py:73:munge_artifact_filename /usr/local/lib/python2.7/site-
>> packages/buildbot/process/properties.py:142:setProperty
>> /usr/local/lib/python2.7/json/__init__.py:244:dumps
>> /usr/local/lib/python2.7/json/encoder.py:207:encode
>> /usr/local/lib/python2.7/json/encoder.py:270:iterencode
>> /usr/local/lib/python2.7/json/encoder.py:184:default ]]
>> twisted.internet.defer.FirstError: FirstError[#14, [Failure instance:
>> Traceback: <type 'exceptions.TypeError'>: <DeferredList at 0x7f9f3d3d9638
>> current result: u'tar.gz'> is not JSON serializable
>> /usr/local/lib/python2.7/site-packages/twisted/internet/defer.py:1299:_inlineCallbacks
>> /usr/local/lib/python2.7/site-packages/buildbot/process/buildstep.py:508:startStep
>> /usr/local/lib/python2.7/site-packages/buildbot/process/properties.py:198:render
>> /usr/local/lib/python2.7/site-packages/buildbot/process/properties.py:157:render
>> --- <exception caught here> --- /usr/local/lib/python2.7/site-
>> packages/twisted/internet/defer.py:150:maybeDeferred
>> package.py:73:munge_artifact_filename /usr/local/lib/python2.7/site-
>> packages/buildbot/process/properties.py:142:setProperty
>> /usr/local/lib/python2.7/json/__init__.py:244:dumps
>> /usr/local/lib/python2.7/json/encoder.py:207:encode
>> /usr/local/lib/python2.7/json/encoder.py:270:iterencode
>> /usr/local/lib/python2.7/json/encoder.py:184:default ]]
>>
>> It appears that the manual "rendering" step I'm attempting to perform in
>> order to coerce my util.Interpolate() results to a string are not
>> working; I'm getting a DeferredList instead of a string.  What do I need to
>> do to massage my interpolation results into a form that I can pass into
>> setProperty?
>>
>> On Tue, Jan 3, 2017 at 12:47 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>>
>> renderer looks good.
>> Maybe we need an extra step kind like manipulateProperties, which would
>> only take a renderer and run it, so that we dont need the extra shell
>> command on the worker.
>>
>> BTW, I like you master.cfg a lot. nice work!
>>
>> Would you like to make a blog post explaining it, like the nice folks at
>> i3 did?
>>
>> https://i3wm.org/docs/buildbot.html
>>
>> We can host it in buildbot's Medium publication:
>> https://medium.com/buildbot
>>
>> Regards
>> Pierre
>>
>>
>> Le mar. 3 janv. 2017 à 00:14, Elliot Saba <staticfloat at gmail.com> a
>> écrit :
>>
>> Thanks Pierre,
>>
>> I have worked around this in a different way, with a dummy
>> setPropertyFromCommand step (my workaround is here
>> <https://github.com/staticfloat/julia-buildbot/blob/level_up/master/package.py#L45-L77>)
>> that just runs `/bin/true`, but within the `command=` callback I manipulate
>> the properties as I desire.
>> -E
>>
>> On Thu, Dec 29, 2016 at 6:15 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>>
>> Hi Elliot,
>>
>> Looking at the source code, it is not possible to get the properties from
>> extract_fn.
>> https://github.com/buildbot/buildbot/blob/master/master/buil
>> dbot/steps/shell.py#L335
>> However, you could easily override commandComplete in order to make your
>> implementation.
>>
>>
>> class mySetPropertyFromCommand(steps.SetPropertyFromCommand):
>>     def commandComplete(self, cmd):
>>         stdout = self.observer.getStdout()
>>         stderr = self.observer.getStderr()
>>         rc = cmd.rc
>>         os = self.getProperty("os")
>>         ...
>>         self.setProperty("prop_name", "value", "mySetPropertyFromCommand")
>>
>>
>> Le mer. 28 déc. 2016 à 20:09, Elliot Saba <staticfloat at gmail.com> a
>> écrit :
>>
>> Hi Pierre,
>>
>> I mean the function that gets called to parse out the results of a
>> setPropertyFromCommand call.
>>
>> On Wed, Dec 28, 2016, 04:14 Pierre Tardy <tardyp at gmail.com> wrote:
>>
>> Hi Elliot
>> What do you mean an extraction function?
>>
>> Le mer. 28 déc. 2016 12:16, Elliot Saba <staticfloat at gmail.com> a écrit :
>>
>> I want to access the properties of my build within a buildstep.  To do
>> so, I need to use something like `step.getProperty('foo')`, but I don't
>> know how to get the current step as a variable into my extraction
>> function.  Is there an easy way to get this information?  The reason I need
>> to do this is that I need to customize my extraction function based upon
>> the operating system of the worker my step is executing on.
>>
>> Thanks in advance,
>> -E
>>
>> _______________________________________________
>> users mailing list
>> users at buildbot.net
>> https://lists.buildbot.net/mailman/listinfo/users
>>
>> --
>>
>> -E
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170103/1cc63608/attachment.html>