[Buildbot-commits] [buildbot/buildbot] 934312: Introduce master log compression.

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/buildbot/buildbot
  Commit: 934312698945f0dee8d55748123e4394b28171f3
      https://github.com/buildbot/buildbot/commit/934312698945f0dee8d55748123e4394b28171f3
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-05-29 (Fri, 29 May 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py
    M master/docs/manual/cfg-global.rst

  Log Message:
  -----------
  Introduce master log compression.

Log in the db is stored as chunck of text.
The implementation expected a compress operation to be done at the end
of the step. But it will be costly in time/db operations with db
fragmentation risk.

This patch suggest to do live compression when chunck size to insert
is smaller once compressed.
The compress operation will "just" have then to optimize/regroup
all contiguous text chunk, compress them and replace them in the db.
Compression algorithm is configurable in the master cfg.

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: 734f2c8762a07482dab44b20d8cdfa118f88f1e4
      https://github.com/buildbot/buildbot/commit/734f2c8762a07482dab44b20d8cdfa118f88f1e4
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-05-29 (Fri, 29 May 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py

  Log Message:
  -----------
  Fix review comments.

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: 5ef75c748d2f7bc68e65fdf721bffe33217259ef
      https://github.com/buildbot/buildbot/commit/5ef75c748d2f7bc68e65fdf721bffe33217259ef
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-05-29 (Fri, 29 May 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py
    M master/buildbot/test/unit/test_config.py

  Log Message:
  -----------
  Fix review comments, fix unit test.

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: 532cbd9d5f524239b03f403d953ad84fe8ce37f8
      https://github.com/buildbot/buildbot/commit/532cbd9d5f524239b03f403d953ad84fe8ce37f8
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-05-29 (Fri, 29 May 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py

  Log Message:
  -----------
  Fix review comments.

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: 170660fdfea4b26026db520868d3dce608d4dbc7
      https://github.com/buildbot/buildbot/commit/170660fdfea4b26026db520868d3dce608d4dbc7
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-05-29 (Fri, 29 May 2015)

  Changed paths:
    M .travis.yml
    M master/buildbot/test/unit/test_db_logs.py

  Log Message:
  -----------
  Add unit tests.

Extend unit tests to cover:
- no chunk compression when not needed
- all compression formalism

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: d23ed4b22821fb8b6aa1a05bcbd473705c4a387e
      https://github.com/buildbot/buildbot/commit/d23ed4b22821fb8b6aa1a05bcbd473705c4a387e
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-06-01 (Mon, 01 Jun 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py
    M master/buildbot/test/unit/test_config.py
    M master/buildbot/test/unit/test_db_logs.py
    M master/docs/manual/cfg-global.rst

  Log Message:
  -----------
  Fix unit test and review comments

- skip lz4 ut if lz4 not available
- fix error message if lz4 not available
- set default compression to gz instead of bz2
- fix lint issues

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: b99d35a8988bfb61130118459d9d877fbcce51a2
      https://github.com/buildbot/buildbot/commit/b99d35a8988bfb61130118459d9d877fbcce51a2
  Author: Sebastien Fusilier <sebastien.fusilier at intel.com>
  Date:   2015-06-01 (Mon, 01 Jun 2015)

  Changed paths:
    M master/buildbot/config.py
    M master/buildbot/db/logs.py

  Log Message:
  -----------
  Clean appendLog

Import from https://github.com/buildbot/buildbot/pull/1698
appendLog refacto and pylint fix issue.

Signed-off-by: Sebastien Fusilier <sebastien.fusilier at intel.com>


  Commit: 6b8756ee88107ff65709e3681c166a802065b08b
      https://github.com/buildbot/buildbot/commit/6b8756ee88107ff65709e3681c166a802065b08b
  Author: Pierre Tardy <pierre.tardy at intel.com>
  Date:   2015-06-04 (Thu, 04 Jun 2015)

  Changed paths:
    M .travis.yml
    M master/buildbot/config.py
    M master/buildbot/db/logs.py
    M master/buildbot/test/unit/test_config.py
    M master/buildbot/test/unit/test_db_logs.py
    M master/docs/manual/cfg-global.rst

  Log Message:
  -----------
  Merge branch 'compress_log3' of https://github.com/bastien31/buildbot into pr1696


Compare: https://github.com/buildbot/buildbot/compare/f0c08e80a9f4...6b8756ee8810