[users at bb.net] Save build logs in a standalone report

[Pierre Tardy]

Hi Mike,

Thanks for sharing this code. I think it is a very interesting approach.

All,
I'd like to better understand the reasons why you want to archive those
build logs.
What do you want to do with them in the long term?
Why having them accessible from the buildbot UI/rest api is not enough?
Do we need a build report archiver reporter included in buildbot?
Pierre

On Tue, Mar 28, 2017 at 12:49 PM Mike Alexeev <mike.42.alexeev at gmail.com>
wrote:

> Hello Pierre,
>
> Thank you for advice.
> I will use reporter class instead of Build class customization.
>
> Alvaro, anyway here is an example of subclassing Build:
> https://pastebin.com/gfq1wQLG
>
> Maybe it will be helpful in other cases
>
> 2017-03-27 21:49 GMT+03:00 Pierre Tardy <tardyp at gmail.com>:
>
> Hi all,
> Usually, the way to do that is to setup a custom reporter class.
> You can then use the data api to fetch as many information on the build as
> you need to build your html report, and push it to S3.
>
> The details of what you can get with rest api is documented here:
> https://docs.buildbot.net/0.9.5/developer/rest.html
>
> The best is to look at the example of reporters/utils
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/reporters/utils.py
>
> If you are not confident enough with twisted and async, you can just put a
> MasterShellCommand with a python script in the end of your builds and use
> the REST api to fetch the data you need.
> In order to get raw logs, you can use the raw log endpoint (no need for
> logobservers!):
>
> e.g: https://nine.buildbot.net/api/v2/logs/357614/raw
>
> Note that buildbot store the stdout/stderr information, which is removed
> when you fetch it via raw endpoint.
>
> Here is an example on how to process the stream logs that we use for
> debugging our integration tests:
>
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/util/integration.py#L246
>
> Regards
> Pierre
>
>
> Le lun. 27 mars 2017 à 18:41, Alvaro Erickson <alvaro.erickson at gmail.com>
> a écrit :
>
> Hi Mike,
>
> I too would like to see what your solution for saving a log (into plain
> text, not html) for all steps after a build is finished. Specifically, if
> you could please share the subclassed Build class for log generation. Any
> examples would be very helpful.
>
> Background: On my work project, one of the goals is they want all the
> build logs to be archived.
>
> Regards,
>
> Alvaro
>
> On Mon, Mar 27, 2017 at 8:52 AM, Mike Alexeev <mike.42.alexeev at gmail.com>
> wrote:
>
> Hello, Koutheir
>
>
> I have also similar problem.
> I need to save log from all steps after build finished.
> No html just plain text.
>
> It seems that there is no ready solution.
> For example there is no file storage in buildbot for files (reports, logs
> and other build artifacts).
> So my solution is not universal though it can be helpful.
>
>
> For log generation I subclassed Build class. You can set your own custom
> Build class
> to build factory in buildbot config.
>
> I overrided buildFinished method and added there code to collect log from
> all steps.
> In build instance you have access to Data API and you can read logs from
> DB.
> I can share example if you want.
>
> For file serving I use nginx.
> I added links to log in build page template and substitute path to log
> based on build properties.
>
> So every build has link on its page to log.
>
> I think that saving log can also be implemented as step running on Local
> Worker
> (it runs on master).  But I used custom build class because it looks
> cleaner (no odd steps in build).
>
>
> By the way, can be there default implementation for file storage in
> buildbot?
> For example default solution with twisted file serving
> and possibility to use server like nginx in bigger setups. Just like db
> setup - sqlite has no external
>  processes and easy to setup, but  also you can use postgre which much
> more faster.
>
>
> What about adding post-build hook in configuration? Log-saving method for
> example could be
>  set there
>
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170328/b7dc678b/attachment.html>