[users at bb.net] Save build logs in a standalone report

[Alvaro Erickson]

Hi Pierre,

Why I need to keep the logs long term is:

   1. It is a goal of the project long term I am working on
   2. If the logHorizon ticket# 3572 <http://trac.buildbot.net/ticket/3572>was
   fixed I would not need to archive the logs

Regards,

Alvaro

On Tue, Mar 28, 2017 at 4:00 AM, Pierre Tardy <tardyp at gmail.com> wrote:

> Hi Mike,
>
> Thanks for sharing this code. I think it is a very interesting approach.
>
> All,
> I'd like to better understand the reasons why you want to archive those
> build logs.
> What do you want to do with them in the long term?
> Why having them accessible from the buildbot UI/rest api is not enough?
> Do we need a build report archiver reporter included in buildbot?
> Pierre
>
> On Tue, Mar 28, 2017 at 12:49 PM Mike Alexeev <mike.42.alexeev at gmail.com>
> wrote:
>
>> Hello Pierre,
>>
>> Thank you for advice.
>> I will use reporter class instead of Build class customization.
>>
>> Alvaro, anyway here is an example of subclassing Build:
>> https://pastebin.com/gfq1wQLG
>>
>> Maybe it will be helpful in other cases
>>
>> 2017-03-27 21:49 GMT+03:00 Pierre Tardy <tardyp at gmail.com>:
>>
>> Hi all,
>> Usually, the way to do that is to setup a custom reporter class.
>> You can then use the data api to fetch as many information on the build
>> as you need to build your html report, and push it to S3.
>>
>> The details of what you can get with rest api is documented here:
>> https://docs.buildbot.net/0.9.5/developer/rest.html
>>
>> The best is to look at the example of reporters/utils https://
>> github.com/buildbot/buildbot/blob/master/master/buildbot/
>> reporters/utils.py
>>
>> If you are not confident enough with twisted and async, you can just put
>> a MasterShellCommand with a python script in the end of your builds and use
>> the REST api to fetch the data you need.
>> In order to get raw logs, you can use the raw log endpoint (no need for
>> logobservers!):
>>
>> e.g: https://nine.buildbot.net/api/v2/logs/357614/raw
>>
>> Note that buildbot store the stdout/stderr information, which is removed
>> when you fetch it via raw endpoint.
>>
>> Here is an example on how to process the stream logs that we use for
>> debugging our integration tests:
>> https://github.com/buildbot/buildbot/blob/master/master/
>> buildbot/test/util/integration.py#L246
>>
>> Regards
>> Pierre
>>
>>
>> Le lun. 27 mars 2017 à 18:41, Alvaro Erickson <alvaro.erickson at gmail.com>
>> a écrit :
>>
>> Hi Mike,
>>
>> I too would like to see what your solution for saving a log (into plain
>> text, not html) for all steps after a build is finished. Specifically, if
>> you could please share the subclassed Build class for log generation. Any
>> examples would be very helpful.
>>
>> Background: On my work project, one of the goals is they want all the
>> build logs to be archived.
>>
>> Regards,
>>
>> Alvaro
>>
>> On Mon, Mar 27, 2017 at 8:52 AM, Mike Alexeev <mike.42.alexeev at gmail.com>
>> wrote:
>>
>> Hello, Koutheir
>>
>>
>> I have also similar problem.
>> I need to save log from all steps after build finished.
>> No html just plain text.
>>
>> It seems that there is no ready solution.
>> For example there is no file storage in buildbot for files (reports, logs
>> and other build artifacts).
>> So my solution is not universal though it can be helpful.
>>
>>
>> For log generation I subclassed Build class. You can set your own custom
>> Build class
>> to build factory in buildbot config.
>>
>> I overrided buildFinished method and added there code to collect log from
>> all steps.
>> In build instance you have access to Data API and you can read logs from
>> DB.
>> I can share example if you want.
>>
>> For file serving I use nginx.
>> I added links to log in build page template and substitute path to log
>> based on build properties.
>>
>> So every build has link on its page to log.
>>
>> I think that saving log can also be implemented as step running on Local
>> Worker
>> (it runs on master).  But I used custom build class because it looks
>> cleaner (no odd steps in build).
>>
>>
>> By the way, can be there default implementation for file storage in
>> buildbot?
>> For example default solution with twisted file serving
>> and possibility to use server like nginx in bigger setups. Just like db
>> setup - sqlite has no external
>>  processes and easy to setup, but  also you can use postgre which much
>> more faster.
>>
>>
>> What about adding post-build hook in configuration? Log-saving method for
>> example could be
>>  set there
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users at buildbot.net
>> https://lists.buildbot.net/mailman/listinfo/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users at buildbot.net
>> https://lists.buildbot.net/mailman/listinfo/users
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170328/5b6a4d12/attachment.html>