[Buildbot] #3455: Run validate.sh in Travis and metabuildbot

[Buildbot trac]

#3455: Run validate.sh in Travis and metabuildbot
-------------------+--------------------
Reporter:  dustin  |      Owner:
    Type:  task    |     Status:  new
Priority:  major   |  Milestone:  0.9.+
 Version:  master  |   Keywords:  simple
-------------------+--------------------
 We currently have three ways of running the "tests" for Buildbot:

  * common/validate.sh
  * .travis.yml
  * [https://github.com/buildbot/metabbotcfg the metabuildbot]

 These all use slightly different versions of various tools, and check
 slightly different things.

 Part of the reason for that is that we want to break things up differently
 in the three places: common/validate.sh is meant to run quickly to look at
 a range of revisions (so it doesn't run lint on all files, just the files
 that have changed, for example).  Travis is meant to validate a whole
 commit (and can take longer).  Metabuildbot wants to break the validation
 process into separate builders.

 Recently, we switched from pyflakes to flake8.  We fixed this in Travis,
 then remembered that it had to be fixed in validate.sh, then just
 yesterday remembered to update the metabuildbot, too.  Ideally all of
 those would change together.

 What I'd suggest is to break out the set of things that get checked in
 validate.sh into different files, and build some simple entry points for
 Travis, metabuildbot, and validation.  Things like `common/validate.sh
 lint` to just run the lint checks or `common/validate.sh tests` to just
 run the unit tests`.

 Finally, travis.yml has a bunch of pip installs of specific versions of
 libraries and packages like pylint.  It would be great to put those in a
 central location, too -- perhaps common/test_requirements.txt -- so that
 all of these tools can use the same versions.

--
Ticket URL: <http://trac.buildbot.net/ticket/3455>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation