[users at bb.net] Save build logs in a standalone report

Tue Mar 28 12:43:28 UTC 2017

Thanks Mike for the feedback, it helps a lot.

There is not only one log for steps. 'stdio' is the default name for base
log produced by commands. You can add custom logs, which will have
different names
https://github.com/buildbot/buildbot/issues/3085

Maybe a virtual log concatenating all logs for a step would be an help for
you as well?

Note that logs are stored compressed already in the db, so you won't gain
much space by tgz them.
Agree on the grouping, though.

Release build system is an important use case for buildbot as we intend to
be a build and test framework (not only CI).

Pierre

On Tue, Mar 28, 2017 at 2:23 PM Mike Alexeev <mike.42.alexeev at gmail.com>
wrote:

> Pierre,
>
> In my case I'm developing release build system (not CI) with
> some requirements.
> For example - reproducing builds, generating reports with lists of all
> sources and binaries, etc.
>
> Log file is one of build artifacts among other.
> It is more suitable to have one big log for whole component instead of
> logs for each step then grepping or searching in text editor.
>
> Logs for each step can be downloaded in UI. But filenames are not good -
> they all named 'stdio'.
> (I haven't thought about it yet, but maybe it would be better to name them
> like  {buildername}_{buildnumber}_{stepname}_{slug}.log  ? )
>
> Archiving logs as I think can save disc space and also help grouping logs.
> For example on my job I'm setuping projects with about 100 builders in
> each. All of builders run to build in the end one package set (rpm and
> deb).
> And it is convenient to download one .tgz with 100 logs.
>
> In my opinion log system in buildbot is good. It is nice that we can see
> log during buildtime and logs are updating.
> Other cases (like mine) are very specific and maybe not suitable  for
> every project especially for CI cases.
>
>
> 2017-03-28 14:00 GMT+03:00 Pierre Tardy <tardyp at gmail.com>:
>
> Hi Mike,
>
> Thanks for sharing this code. I think it is a very interesting approach.
>
> All,
> I'd like to better understand the reasons why you want to archive those
> build logs.
> What do you want to do with them in the long term?
> Why having them accessible from the buildbot UI/rest api is not enough?
> Do we need a build report archiver reporter included in buildbot?
> Pierre
>
> On Tue, Mar 28, 2017 at 12:49 PM Mike Alexeev <mike.42.alexeev at gmail.com>
> wrote:
>
> Hello Pierre,
>
> Thank you for advice.
> I will use reporter class instead of Build class customization.
>
> Alvaro, anyway here is an example of subclassing Build:
> https://pastebin.com/gfq1wQLG
>
> Maybe it will be helpful in other cases
>
> 2017-03-27 21:49 GMT+03:00 Pierre Tardy <tardyp at gmail.com>:
>
> Hi all,
> Usually, the way to do that is to setup a custom reporter class.
> You can then use the data api to fetch as many information on the build as
> you need to build your html report, and push it to S3.
>
> The details of what you can get with rest api is documented here:
> https://docs.buildbot.net/0.9.5/developer/rest.html
>
> The best is to look at the example of reporters/utils
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/reporters/utils.py
>
> If you are not confident enough with twisted and async, you can just put a
> MasterShellCommand with a python script in the end of your builds and use
> the REST api to fetch the data you need.
> In order to get raw logs, you can use the raw log endpoint (no need for
> logobservers!):
>
> e.g: https://nine.buildbot.net/api/v2/logs/357614/raw
>
> Note that buildbot store the stdout/stderr information, which is removed
> when you fetch it via raw endpoint.
>
> Here is an example on how to process the stream logs that we use for
> debugging our integration tests:
>
> https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/util/integration.py#L246
>
> Regards
> Pierre
>
>
> Le lun. 27 mars 2017 à 18:41, Alvaro Erickson <alvaro.erickson at gmail.com>
> a écrit :
>
> Hi Mike,
>
> I too would like to see what your solution for saving a log (into plain
> text, not html) for all steps after a build is finished. Specifically, if
> you could please share the subclassed Build class for log generation. Any
> examples would be very helpful.
>
> Background: On my work project, one of the goals is they want all the
> build logs to be archived.
>
> Regards,
>
> Alvaro
>
> On Mon, Mar 27, 2017 at 8:52 AM, Mike Alexeev <mike.42.alexeev at gmail.com>
> wrote:
>
> Hello, Koutheir
>
>
> I have also similar problem.
> I need to save log from all steps after build finished.
> No html just plain text.
>
> It seems that there is no ready solution.
> For example there is no file storage in buildbot for files (reports, logs
> and other build artifacts).
> So my solution is not universal though it can be helpful.
>
>
> For log generation I subclassed Build class. You can set your own custom
> Build class
> to build factory in buildbot config.
>
> I overrided buildFinished method and added there code to collect log from
> all steps.
> In build instance you have access to Data API and you can read logs from
> DB.
> I can share example if you want.
>
> For file serving I use nginx.
> I added links to log in build page template and substitute path to log
> based on build properties.
>
> So every build has link on its page to log.
>
> I think that saving log can also be implemented as step running on Local
> Worker
> (it runs on master).  But I used custom build class because it looks
> cleaner (no odd steps in build).
>
>
> By the way, can be there default implementation for file storage in
> buildbot?
> For example default solution with twisted file serving
> and possibility to use server like nginx in bigger setups. Just like db
> setup - sqlite has no external
>  processes and easy to setup, but  also you can use postgre which much
> more faster.
>
>
> What about adding post-build hook in configuration? Log-saving method for
> example could be
>  set there
>
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170328/a6592901/attachment.html>