[Buildbot-commits] [buildbot/buildbot] a3fcc9: Sourcestamps refactor (WARNING!)

Sat Jan 12 22:01:14 UTC 2013

  Branch: refs/heads/nine
  Home:   https://github.com/buildbot/buildbot
  Commit: a3fcc91ee3982eac7bd9d78a5c5202c0a9602986
      https://github.com/buildbot/buildbot/commit/a3fcc91ee3982eac7bd9d78a5c5202c0a9602986
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2012-12-29 (Sat, 29 Dec 2012)

  Changed paths:
    M README.md
    M master/buildbot/clients/debug.py
    M master/buildbot/clients/tryclient.py
    M master/buildbot/data/buildsets.py
    M master/buildbot/data/changes.py
    M master/buildbot/data/connector.py
    A master/buildbot/data/sourcestamps.py
    M master/buildbot/db/base.py
    M master/buildbot/db/builders.py
    M master/buildbot/db/buildsets.py
    M master/buildbot/db/changes.py
    M master/buildbot/db/connector.py
    M master/buildbot/db/masters.py
    A master/buildbot/db/migrate/versions/026_sourcestamps_refactor.py
    M master/buildbot/db/model.py
    M master/buildbot/db/schedulers.py
    M master/buildbot/db/sourcestamps.py
    R master/buildbot/db/sourcestampsets.py
    M master/buildbot/db/state.py
    M master/buildbot/db/users.py
    M master/buildbot/interfaces.py
    M master/buildbot/process/builder.py
    M master/buildbot/process/buildrequest.py
    M master/buildbot/process/debug.py
    M master/buildbot/schedulers/base.py
    M master/buildbot/schedulers/dependent.py
    M master/buildbot/schedulers/forcesched.py
    M master/buildbot/schedulers/timed.py
    M master/buildbot/schedulers/triggerable.py
    M master/buildbot/schedulers/trysched.py
    R master/buildbot/sourcestamp.py
    M master/buildbot/status/build.py
    M master/buildbot/status/builder.py
    M master/buildbot/status/buildrequest.py
    M master/buildbot/status/web/grid.py
    M master/buildbot/status/words.py
    M master/buildbot/steps/trigger.py
    M master/buildbot/test/fake/fakedata.py
    M master/buildbot/test/fake/fakedb.py
    M master/buildbot/test/fake/fakemaster.py
    M master/buildbot/test/fake/fakemq.py
    M master/buildbot/test/integration/test_upgrade.py
    M master/buildbot/test/regressions/test_import_unicode_changes.py
    M master/buildbot/test/regressions/test_oldpaths.py
    M master/buildbot/test/regressions/test_shell_command_properties.py
    R master/buildbot/test/regressions/test_sourcestamp_revision.py
    M master/buildbot/test/scenarios/base.py
    M master/buildbot/test/unit/test_changes_changes.py
    M master/buildbot/test/unit/test_changes_hgpoller.py
    M master/buildbot/test/unit/test_data_builders.py
    M master/buildbot/test/unit/test_data_buildsets.py
    M master/buildbot/test/unit/test_data_changes.py
    M master/buildbot/test/unit/test_data_connector.py
    M master/buildbot/test/unit/test_data_masters.py
    A master/buildbot/test/unit/test_data_sourcestamps.py
    M master/buildbot/test/unit/test_db_base.py
    M master/buildbot/test/unit/test_db_builders.py
    M master/buildbot/test/unit/test_db_buildrequests.py
    M master/buildbot/test/unit/test_db_builds.py
    M master/buildbot/test/unit/test_db_buildsets.py
    M master/buildbot/test/unit/test_db_changes.py
    M master/buildbot/test/unit/test_db_masters.py
    A master/buildbot/test/unit/test_db_migrate_versions_026_sourcestamps_refactor.py
    M master/buildbot/test/unit/test_db_schedulers.py
    M master/buildbot/test/unit/test_db_sourcestamps.py
    R master/buildbot/test/unit/test_db_sourcestampsets.py
    M master/buildbot/test/unit/test_master.py
    M master/buildbot/test/unit/test_process_builder.py
    M master/buildbot/test/unit/test_process_buildrequest.py
    M master/buildbot/test/unit/test_process_users_manual.py
    M master/buildbot/test/unit/test_process_users_users.py
    M master/buildbot/test/unit/test_schedulers_base.py
    M master/buildbot/test/unit/test_schedulers_dependent.py
    M master/buildbot/test/unit/test_schedulers_forcesched.py
    M master/buildbot/test/unit/test_schedulers_timed_NightlyTriggerable.py
    M master/buildbot/test/unit/test_schedulers_triggerable.py
    M master/buildbot/test/unit/test_schedulers_trysched.py
    R master/buildbot/test/unit/test_sourcestamp.py
    M master/buildbot/test/unit/test_status_build.py
    M master/buildbot/test/unit/test_status_client.py
    M master/buildbot/test/unit/test_status_mail.py
    M master/buildbot/test/unit/test_status_master.py
    M master/buildbot/test/unit/test_steps_trigger.py
    A master/buildbot/test/unit/test_test_util_validation.py
    R master/buildbot/test/unit/test_test_util_verifier.py
    M master/buildbot/test/util/connector_component.py
    M master/buildbot/test/util/scheduler.py
    R master/buildbot/test/util/types/__init__.py
    R master/buildbot/test/util/types/builders.py
    R master/buildbot/test/util/types/buildsets.py
    R master/buildbot/test/util/types/changes.py
    R master/buildbot/test/util/types/masters.py
    R master/buildbot/test/util/types/schedulers.py
    A master/buildbot/test/util/validation.py
    R master/buildbot/test/util/verifier.py
    A master/buildbot/util/pickle_prereqs.py
    M master/docs/developer/data.rst
    M master/docs/developer/database.rst
    M master/docs/developer/rtype-buildset.rst
    M master/docs/developer/rtype-change.rst
    A master/docs/developer/rtype-sourcestamp.rst
    M master/docs/manual/cfg-builders.rst
    M master/docs/manual/cmdline.rst
    M master/docs/manual/customization.rst
    M master/docs/relnotes/index.rst

  Log Message:
  -----------
  Sourcestamps refactor (WARNING!)

WARNING: this breaks the existing status hierarchy; in particular,
   BuildStatus.getSourceStamps now returns nothing.  It's all
   downhill from here!

SourceStampSets are gone; instead, buildsets have a collection of
sourcestamps (roughly equivalent to the existing sourcestampsets, but
treated as a simple many:many relationship instead); each change has one
sourcestamp, and sourcestamps are uniquely identified.  This
necessitates significant changes to schedulers and the build process.

More importantly, SourceStamps are gone!  The class has moved to the
pickle_prereqs module for use when importing pickles, but is otherwise
gone from Buildbot.  This, too, has required significant changes to the
process classes.  Note that BuildRequests add "fake" SourceStamp objects
so that the status hierarchy can remain relatively un-altered.

All of this begins to break the status hierarchy, which will disappear
by the end of the nine process anyway.  In particular,
BuildStatus.getSourceStamps now always returns an empty set, meaning
most of the web UI no longer shows anything useful.

Also in this commit: (sorry for rolling all this together!)

* a new method of validating data types, allowing more flexible
  specification of nested types.

* generalized column hashing to work around MySQL's lack of support for
  unique indexes.

* remove support for triggering builds from the debug client

* adds a master attribute to the DB connector, requiring changes to
  FakeDBConnector as well

* fix a few identifiers with underscores to use interCaps instead

  Commit: 7b62e3368e40201bb99c3931cebfa9987b6e3918
      https://github.com/buildbot/buildbot/commit/7b62e3368e40201bb99c3931cebfa9987b6e3918
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2012-12-30 (Sun, 30 Dec 2012)

  Changed paths:
    M master/buildbot/test/util/validation.py
    M master/docs/developer/data.rst
    M master/docs/developer/rtype-sourcestamp.rst

  Log Message:
  -----------
  minor docs fixes

  Commit: 08993b55fbad1d9e1afa341fa53de513deeae8a7
      https://github.com/buildbot/buildbot/commit/08993b55fbad1d9e1afa341fa53de513deeae8a7
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2012-12-31 (Mon, 31 Dec 2012)

  Changed paths:
    M master/buildbot/data/sourcestamps.py

  Log Message:
  -----------
  fix resource type 'type' attribute

  Commit: 8b0507bf7e30fca97ec7f3553f2d05348f4c9a19
      https://github.com/buildbot/buildbot/commit/8b0507bf7e30fca97ec7f3553f2d05348f4c9a19
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-02 (Wed, 02 Jan 2013)

  Changed paths:
    M README.md
    M master/buildbot/db/changes.py

  Log Message:
  -----------
  minor fixes suggested by jaredgrubb

  Commit: b4ae532b4bb624e9e7693545c62ed76537094850
      https://github.com/buildbot/buildbot/commit/b4ae532b4bb624e9e7693545c62ed76537094850
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-04 (Fri, 04 Jan 2013)

  Changed paths:
    M master/buildbot/test/util/validation.py
    M master/docs/developer/database.rst
    M master/docs/developer/tests.rst

  Log Message:
  -----------
  validator fixes as suggested by tomprince

  Commit: ef5b9421700a19723a0afe63c8c3710d00dba9ba
      https://github.com/buildbot/buildbot/commit/ef5b9421700a19723a0afe63c8c3710d00dba9ba
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/test/util/validation.py

  Log Message:
  -----------
  remove tuple deconstruction, better factoring of buildset validators

  Commit: 2fd12642d2bac82bfb80ab29a56aa4c6309f3797
      https://github.com/buildbot/buildbot/commit/2fd12642d2bac82bfb80ab29a56aa4c6309f3797
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/status/build.py
    M master/buildbot/test/scenarios/base.py
    M master/buildbot/test/unit/test_db_changes.py
    M master/buildbot/test/unit/test_db_sourcestamps.py
    M master/buildbot/test/unit/test_schedulers_trysched.py
    M master/buildbot/test/util/scheduler.py

  Log Message:
  -----------
  minor fixes to tests

  Commit: 208426ea2932fa6007ecc159ef8ba1104d04e14d
      https://github.com/buildbot/buildbot/commit/208426ea2932fa6007ecc159ef8ba1104d04e14d
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/schedulers/timed.py
    M master/buildbot/test/unit/test_schedulers_timed_NightlyTriggerable.py
    M master/docs/relnotes/index.rst

  Log Message:
  -----------
  add support for old lastTrigger format to NightlyTriggerable

  Commit: cb1b5a9121056535515b22edcf8bc2e07192c0df
      https://github.com/buildbot/buildbot/commit/cb1b5a9121056535515b22edcf8bc2e07192c0df
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/schedulers/base.py

  Log Message:
  -----------
  remove unnecessary inlineCallbacks

  Commit: 04843b2ce5eefa1d2cf6349b4b8c9713ad8b4186
      https://github.com/buildbot/buildbot/commit/04843b2ce5eefa1d2cf6349b4b8c9713ad8b4186
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/process/buildrequest.py

  Log Message:
  -----------
  make the temporary SourceStamp more obvious

  Commit: 864abf346122312009cf560ed41b28b7d4e3612f
      https://github.com/buildbot/buildbot/commit/864abf346122312009cf560ed41b28b7d4e3612f
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/db/buildsets.py

  Log Message:
  -----------
  remove unnecessary yield of a returnValue

  Commit: 8500289eacae0d2bba0e64b14c4788a75e78cf4f
      https://github.com/buildbot/buildbot/commit/8500289eacae0d2bba0e64b14c4788a75e78cf4f
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/data/buildsets.py

  Log Message:
  -----------
  use copy instead of deepcopy

  Commit: 43008003a28f97214d192a1fae64d71528bbae76
      https://github.com/buildbot/buildbot/commit/43008003a28f97214d192a1fae64d71528bbae76
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-05 (Sat, 05 Jan 2013)

  Changed paths:
    M master/buildbot/changes/changes.py
    M master/buildbot/db/migrate/versions/001_initial.py
    M master/buildbot/db/model.py
    M master/buildbot/status/build.py
    M master/buildbot/status/builder.py
    M master/buildbot/status/master.py
    M master/buildbot/status/persistent_queue.py
    M master/buildbot/test/integration/test_upgrade.py
    M master/buildbot/test/regressions/test_oldpaths.py
    M master/buildbot/test/regressions/test_unpickling.py
    M master/buildbot/test/unit/test_status_logfile.py
    A master/buildbot/test/unit/test_util_pickle.py
    M master/buildbot/test/util/change_import.py
    A master/buildbot/util/pickle.py
    R master/buildbot/util/pickle_prereqs.py
    M master/contrib/fix_changes_pickle_encoding.py
    M master/docs/developer/utils.rst

  Log Message:
  -----------
  handle old, no-longer-desired classes in pickles using a find_global function

  Commit: 323127543a038039931c781d2b5f9a05c8a9003a
      https://github.com/buildbot/buildbot/commit/323127543a038039931c781d2b5f9a05c8a9003a
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-08 (Tue, 08 Jan 2013)

  Changed paths:
    M master/buildbot/schedulers/timed.py

  Log Message:
  -----------
  fix logging statement

  Commit: 9a8fcf86ba90972afa5fc3e19d564d45fb1fd05d
      https://github.com/buildbot/buildbot/commit/9a8fcf86ba90972afa5fc3e19d564d45fb1fd05d
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-12 (Sat, 12 Jan 2013)

  Changed paths:
    M master/buildbot/status/build.py

  Log Message:
  -----------
  fix comment typo

  Commit: e0f133bbe64f2cc3a2c4ec3b29a2893b150f0c61
      https://github.com/buildbot/buildbot/commit/e0f133bbe64f2cc3a2c4ec3b29a2893b150f0c61
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-12 (Sat, 12 Jan 2013)

  Changed paths:
    M master/buildbot/test/util/validation.py

  Log Message:
  -----------
  be extra-careful about validation

  Commit: 438b525d71ec9d84231c6d574ae94fc44bd2e759
      https://github.com/buildbot/buildbot/commit/438b525d71ec9d84231c6d574ae94fc44bd2e759
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-12 (Sat, 12 Jan 2013)

  Changed paths:
    M master/buildbot/schedulers/timed.py

  Log Message:
  -----------
  fix logging fix

  Commit: b70f2b66e81e6231b14525c97e3f162e26250a61
      https://github.com/buildbot/buildbot/commit/b70f2b66e81e6231b14525c97e3f162e26250a61
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-12 (Sat, 12 Jan 2013)

  Changed paths:
    M README.md
    M master/buildbot/changes/changes.py
    M master/buildbot/clients/debug.py
    M master/buildbot/clients/tryclient.py
    M master/buildbot/data/buildsets.py
    M master/buildbot/data/changes.py
    M master/buildbot/data/connector.py
    A master/buildbot/data/sourcestamps.py
    M master/buildbot/db/base.py
    M master/buildbot/db/builders.py
    M master/buildbot/db/buildsets.py
    M master/buildbot/db/changes.py
    M master/buildbot/db/connector.py
    M master/buildbot/db/masters.py
    M master/buildbot/db/migrate/versions/001_initial.py
    A master/buildbot/db/migrate/versions/026_sourcestamps_refactor.py
    M master/buildbot/db/model.py
    M master/buildbot/db/schedulers.py
    M master/buildbot/db/sourcestamps.py
    R master/buildbot/db/sourcestampsets.py
    M master/buildbot/db/state.py
    M master/buildbot/db/users.py
    M master/buildbot/interfaces.py
    M master/buildbot/process/builder.py
    M master/buildbot/process/buildrequest.py
    M master/buildbot/process/debug.py
    M master/buildbot/schedulers/base.py
    M master/buildbot/schedulers/dependent.py
    M master/buildbot/schedulers/forcesched.py
    M master/buildbot/schedulers/timed.py
    M master/buildbot/schedulers/triggerable.py
    M master/buildbot/schedulers/trysched.py
    R master/buildbot/sourcestamp.py
    M master/buildbot/status/build.py
    M master/buildbot/status/builder.py
    M master/buildbot/status/buildrequest.py
    M master/buildbot/status/master.py
    M master/buildbot/status/persistent_queue.py
    M master/buildbot/status/web/grid.py
    M master/buildbot/status/words.py
    M master/buildbot/steps/trigger.py
    M master/buildbot/test/fake/fakedata.py
    M master/buildbot/test/fake/fakedb.py
    M master/buildbot/test/fake/fakemaster.py
    M master/buildbot/test/fake/fakemq.py
    M master/buildbot/test/integration/test_upgrade.py
    M master/buildbot/test/regressions/test_import_unicode_changes.py
    M master/buildbot/test/regressions/test_oldpaths.py
    M master/buildbot/test/regressions/test_shell_command_properties.py
    R master/buildbot/test/regressions/test_sourcestamp_revision.py
    M master/buildbot/test/regressions/test_unpickling.py
    M master/buildbot/test/scenarios/base.py
    M master/buildbot/test/unit/test_changes_changes.py
    M master/buildbot/test/unit/test_changes_hgpoller.py
    M master/buildbot/test/unit/test_data_builders.py
    M master/buildbot/test/unit/test_data_buildsets.py
    M master/buildbot/test/unit/test_data_changes.py
    M master/buildbot/test/unit/test_data_connector.py
    M master/buildbot/test/unit/test_data_masters.py
    A master/buildbot/test/unit/test_data_sourcestamps.py
    M master/buildbot/test/unit/test_db_base.py
    M master/buildbot/test/unit/test_db_builders.py
    M master/buildbot/test/unit/test_db_buildrequests.py
    M master/buildbot/test/unit/test_db_builds.py
    M master/buildbot/test/unit/test_db_buildsets.py
    M master/buildbot/test/unit/test_db_changes.py
    M master/buildbot/test/unit/test_db_masters.py
    A master/buildbot/test/unit/test_db_migrate_versions_026_sourcestamps_refactor.py
    M master/buildbot/test/unit/test_db_schedulers.py
    M master/buildbot/test/unit/test_db_sourcestamps.py
    R master/buildbot/test/unit/test_db_sourcestampsets.py
    M master/buildbot/test/unit/test_master.py
    M master/buildbot/test/unit/test_process_builder.py
    M master/buildbot/test/unit/test_process_buildrequest.py
    M master/buildbot/test/unit/test_process_users_manual.py
    M master/buildbot/test/unit/test_process_users_users.py
    M master/buildbot/test/unit/test_schedulers_base.py
    M master/buildbot/test/unit/test_schedulers_dependent.py
    M master/buildbot/test/unit/test_schedulers_forcesched.py
    M master/buildbot/test/unit/test_schedulers_timed_NightlyTriggerable.py
    M master/buildbot/test/unit/test_schedulers_triggerable.py
    M master/buildbot/test/unit/test_schedulers_trysched.py
    R master/buildbot/test/unit/test_sourcestamp.py
    M master/buildbot/test/unit/test_status_build.py
    M master/buildbot/test/unit/test_status_client.py
    M master/buildbot/test/unit/test_status_logfile.py
    M master/buildbot/test/unit/test_status_mail.py
    M master/buildbot/test/unit/test_status_master.py
    M master/buildbot/test/unit/test_steps_trigger.py
    A master/buildbot/test/unit/test_test_util_validation.py
    R master/buildbot/test/unit/test_test_util_verifier.py
    A master/buildbot/test/unit/test_util_pickle.py
    M master/buildbot/test/util/change_import.py
    M master/buildbot/test/util/connector_component.py
    M master/buildbot/test/util/scheduler.py
    R master/buildbot/test/util/types/__init__.py
    R master/buildbot/test/util/types/builders.py
    R master/buildbot/test/util/types/buildsets.py
    R master/buildbot/test/util/types/changes.py
    R master/buildbot/test/util/types/masters.py
    R master/buildbot/test/util/types/schedulers.py
    A master/buildbot/test/util/validation.py
    R master/buildbot/test/util/verifier.py
    A master/buildbot/util/pickle.py
    M master/contrib/fix_changes_pickle_encoding.py
    M master/docs/developer/data.rst
    M master/docs/developer/db.rst
    M master/docs/developer/rtype-buildset.rst
    M master/docs/developer/rtype-change.rst
    A master/docs/developer/rtype-sourcestamp.rst
    M master/docs/developer/tests.rst
    M master/docs/developer/utils.rst
    M master/docs/manual/cfg-builders.rst
    M master/docs/manual/cmdline.rst
    M master/docs/manual/customization.rst
    M master/docs/relnotes/index.rst

  Log Message:
  -----------
  Merge branch '9/sourcestamps' into nine

Conflicts:
	README.md
	master/buildbot/db/sourcestampsets.py

  Commit: 94aa366497ad8fba5a3c95646cf68d2bc93ecc60
      https://github.com/buildbot/buildbot/commit/94aa366497ad8fba5a3c95646cf68d2bc93ecc60
  Author: Dustin J. Mitchell <dustin at mozilla.com>
  Date:   2013-01-12 (Sat, 12 Jan 2013)

  Changed paths:
    M master/docs/developer/utils.rst

  Log Message:
  -----------
  fix doc typo

Compare: https://github.com/buildbot/buildbot/compare/1bfd31f8a29f...94aa366497ad