[Buildbot-commits] [djmitche/buildbot] ee9c72: util.py: add DictOfSets (with tests), LRUCache, de...
noreply at github.com
noreply at github.com
Mon Feb 15 06:26:55 UTC 2010
Branch: refs/heads/master
Home: http://github.com/djmitche/buildbot
Commit: ee9c72f6bb3d63e5c0ba41222e9f04853a678b9d
http://github.com/djmitche/buildbot/commit/ee9c72f6bb3d63e5c0ba41222e9f04853a678b9d
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/test/unit/test_util.py
M buildbot/util.py
Log Message:
-----------
util.py: add DictOfSets (with tests), LRUCache, defaultdict (for py2.4)
Commit: 4e102fdab84aed4482f4438c441b4c32dbead5fe
http://github.com/djmitche/buildbot/commit/4e102fdab84aed4482f4438c441b4c32dbead5fe
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
A buildbot/loop.py
A buildbot/test/test_loop.py
Log Message:
-----------
loop.py: a notification-triggered processor-loop handling class
When somebody modifies the database and sends a message through the
notification server, this is the code that will invoke the Schedulers (or
Builders) one-at-a-time, non-reentrantly, and as many times as is necessary.
It is a MultiService, and cleans up in stopService (to make tests easier).
The code that it runs can return a wakeup time to get re-run at some point in
the future. A loop_done() hook is available too.
Commit: 3785c599e9bcaaaa9018aa38312a8c09389e3646
http://github.com/djmitche/buildbot/commit/3785c599e9bcaaaa9018aa38312a8c09389e3646
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
A buildbot/eventual.py
A buildbot/test/test_eventual.py
Log Message:
-----------
eventual.py: add eventual-send operator, copied (with tests) from Foolscap
Commit: d17ea1d84f0366e063be73e943eef8fe775b3ab7
http://github.com/djmitche/buildbot/commit/d17ea1d84f0366e063be73e943eef8fe775b3ab7
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
A buildbot/db.py
M buildbot/scripts/runner.py
A buildbot/test/test_db.py
Log Message:
-----------
schedulerdb 1-of-5: main database code, create/upgrade-DB in runner.py
All database access occurs in buildbot/db.py, as well as DB creation, initial
schema initialization, and upgrade. The database adapter is derived from
twisted.enterprise, and (in theory) can use anything with a python DB-API
interface, however SQL differences are likely to limit actual compatibility.
SQLite and MySQL are the initial targets.
runner.py contains code to create the database in both "buildbot
create-master" and "buildbot upgrade-master" (which also migrates Changes
from the now-obsolete changes.pck file). The database must exist before
"buildbot start" is used on the master. The database specification (a class
which controls which type of database to use, and the arguments to pass to
its constructor, like filename, or dbhost+username+password) is created in
the .tac file and passed as an argument to the BuildMaster() constructor.
Several objects which used to exist as separate (pickleable) classes are now
expressed as rows in the database: BuildSets, BuildRequests, Properties,
Changes, and SourceStamps. Some of these still have classes defined for the
sake of older code, even though their canonical representation is in the DB.
For immutable objects like Changes, we use an LRU cache and try to avoid
creating duplicate instances for the same notional object.
Several extra fields exist in the DB to support performance metrics (e.g. how
long a build spends waiting for a buildslave) and scheduler dashboard tools.
There is currently no garbage collection, but the rules under which rows may
be deleted are defined in a comment.
Thanks to Kristian Nielsen and Chris Atlee for invaluable DB work.
This commit is the first of five that contain the core functionality of the
schedulerdb project: they are not useful without the others, but are broken
up for ease of review. There are other patches before and after these which
contain more isolated changes. The whole batch (about a dozen) is a git
rewrite of roughly 85 older revisions that included a number of wrong turns
and dead ends (and subsumes an earlier rewrite of about 70 revisions). The
full changelog for the original sequence is included here for history buffs.
commit 99b042292ec32c2ac55c1318a5f15f0b0fd0c083
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 18:29:45 2010 -0800
test_webparts: fix a schedulerdb-related problem, but the overall test
still fails because of some non-schedulerdb issue
commit 1989a62b5c96ff02476569b5e1119482e830d2c1
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 18:28:25 2010 -0800
hush pyflakes warnings in about 40 files. Many of the remaining
warnings are real bugs.
commit d69ede21d6e764e7af5429caa7c6be30f8f46868
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:58:25 2010 -0800
fix test_db
commit 5476c81d98af454055952d3bef27670ee4851774
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:57:12 2010 -0800
fix test_sourcestamp: patch= is always (level, diff)
commit 555b573bc158349f3c3b672b6047971759174fc7
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:40:33 2010 -0800
fix test_config, test_scheduler
commit a327e64fab80fc0268a5a3c635207a661f1407f1
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:31:21 2010 -0800
fix test_web_status_json.py
commit 83973d05052c4cb2d402bab5b6b76b477e433686
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:23:25 2010 -0800
fix test_status, I think
commit 44070c53f43fcbaf2802ec8c47750899b077f58c
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:14:06 2010 -0800
fix test_runner, test_slaves
commit 88ec271143c0b360d18f36323b5cd0260eba6391
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 17:09:48 2010 -0800
fix test_mergerequests
commit 5b07ffedb24cfc990514ab923be86b39043510be
Author: Brian Warner <warner at lothar.com>
Date: Sun Feb 14 15:00:18 2010 -0800
remove dummy setup.py line
commit d0a343706ae97fb64d28b57fbd7e92162b4c0ae2
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 23:44:41 2010 -0800
test_vc: put each test in a separate directory
commit f02ec0510acb6ea38678d7d472dede8a728296f4
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 23:25:09 2010 -0800
fix test_run.Basedir, by letting runutils.py optionally specify the
slavebasedir
commit e730c1f7f865206becfe0f47caf7ddf4f68a90ce
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 23:17:43 2010 -0800
db.py: hush debug prints
commit 2995ef51781d4e33fc00813bb12ec2439179cd5c
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 23:17:17 2010 -0800
test_config: fix testIRC, by using a no-network fake IRC target
commit a9545bd7df38bcc598c087c585000be14a73227e
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 23:01:30 2010 -0800
runutils: go back to using mktemp() for slave dirs, since using
self.basedir broke test_slaves.LatentSlave.testSequence
cleanups to test_slaves
commit c2840e693f4603e17a15b1e11fc66df81d79202c
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 22:29:59 2010 -0800
test_slaves.Slave: cleanups, use MasterMixin and separate basedirs
commit 59b000c18fd9fd0697c17a2e1f973270bb63e853
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 22:19:15 2010 -0800
test_slaves: general cleanup, use requestBuild() and StallMixin, hush
pyflakes
commit 3bdc429272fb15293f39b1153ead7e8c8b1d610a
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 22:18:24 2010 -0800
fix test_slaves.BuildSlave, reduce number of stall()s
commit dd768ed733425cb15e3b2c67a4aa85a757e21e8b
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 22:15:40 2010 -0800
test_scheduler: hush pyflakes
commit aca62e611d1160bf89e5a28d1cb1524ef0fc2c63
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 21:23:49 2010 -0800
Fix doCheckConfig() and check_master_cfg() to not leave lingering
things
master.loadConfig now returns a Deferred because it does DB
operations (some of which use deferToThread). It really needs to have a
reactor running. The fix was to add a new check_synchronously_only=
argument to loadConfig, which lets it do the useful validation parts
but skips the actual setup phase.
This was causing test_runner (working on upgradeMaster) to flunk the
subsequent test with a dirty reactor.
commit 3661ce32025a108b0f9e07acf404bff64bf78aea
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 20:28:21 2010 -0800
test_scheduler: fix testTryUserpass
commit 8b73e3841835af4129c003629218f6c38004c67f
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 20:21:16 2010 -0800
test_scheduler: fix testTryJobdir
commit a340f6fae8d258078ff4d91861604b8c1df0456a
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 20:03:04 2010 -0800
test_scheduler: fix testCategories, both by fixing Scheduler to respect
categories and by fixing the race conditions in the previous version
commit 3e82a6a91ff98c39dcfe3ff5c375bc494501333f
Author: Brian Warner <warner at lothar.com>
Date: Sat Feb 13 11:44:51 2010 -0800
test_scheduler: update testCategory, it fails, I think I broke
categories in the basic Scheduler
commit 6d11870b1b7659bc0cfd652c32f0becbd960db51
Author: Brian Warner <warner at lothar.com>
Date: Fri Feb 12 17:43:20 2010 -0800
test_scheduler: started work on testCategory, not done yet
commit e4a069d133982b325142d90c30eeeecbe1a6e139
Author: Brian Warner <warner at lothar.com>
Date: Fri Feb 12 17:29:03 2010 -0800
delete test_scheduler testGetBuildSets: the implementation changed
drastically
commit c39849d24736d1336a7ca270975aac98c09d6011
Author: Brian Warner <warner at lothar.com>
Date: Fri Feb 12 17:28:33 2010 -0800
test_scheduler: fix up basedirs
commit 3aa127a2fd2d8b9328fca8e063600c71ccdafff1
Author: Chris AtLee <catlee at mozilla.com>
Date: Fri Feb 12 17:02:57 2010 -0500
Fix testPeriodic1 (with a stall)
commit 0dc85adb7720a71684385efc0ac486ecd0303bb5
Author: Chris AtLee <catlee at mozilla.com>
Date: Fri Feb 12 16:51:54 2010 -0500
Fixed AnyBranch, AnyBranch2
commit 2bee223bfca7b9e2b0a1becbb8390a824ae68fb0
Author: Chris AtLee <catlee at mozilla.com>
Date: Fri Feb 12 16:31:12 2010 -0500
Fixes for testBranch (renamed to testOffBranch and
testImportantChanges)
commit b3a313dae9f379dcfee8333bee52ab5562952504
Author: Chris AtLee <catlee at mozilla.com>
Date: Thu Feb 11 18:59:05 2010 -0500
Working on testBranch
commit 31a5daae6196a1245fa97a94e1898c687ec6d0eb
Author: Chris AtLee <catlee at mozilla.com>
Date: Thu Feb 11 18:10:17 2010 -0500
Fixed Nightly
commit 6dc73c354143895c67f2045911e9c5167f1f53d6
Author: Dustin J. Mitchell <dustin at zmanda.com>
Date: Mon Feb 8 18:34:55 2010 -0600
add simplejson prereq
commit e127a17411b1c9de411fe677914525ad561af492
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 11 10:44:10 2010 -0800
test_run: hush pyflakes, fix a missing Deferred return
commit b00db15801c8d448a8f1de09909c0b98253b3e7d
Author: Chris AtLee <catlee at mozilla.com>
Date: Wed Feb 10 17:29:41 2010 -0500
Fix TestFlag
commit cb3bec310e40a078c3c818f837937180e164825f
Author: Chris AtLee <catlee at mozilla.com>
Date: Wed Feb 10 15:42:23 2010 -0500
Fixed TestBuilder
commit 0c50400b8310c3454616a8be47e41ff4bae8e266
Author: Chris AtLee <catlee at mozilla.com>
Date: Wed Feb 10 15:16:41 2010 -0500
Removed SchedulerWatchers test
commit cd553f5dd83d3cd1d4b2ae8133d98a3b41b0a424
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 11 09:12:38 2010 -0800
fix test_run.BuildPrioritization, by Chris Atlee
commit e8515c5a71c5477744bb3ced9a4462bf5e25573f
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 11 09:12:17 2010 -0800
db.py: virtualize DBConnector's notion of time. By Chris Atlee.
commit e68b8b2520e973975093688ceb519944c1b9e4bb
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 11 09:04:27 2010 -0800
test_run: fix GracefulShutdown(Idle|Busy), fixes by Chris Atlee
commit fd3725fdf3b035cfe59a49dc431cb9d5d0678a17
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 11 08:55:46 2010 -0800
test_run: fix PropertyPropagation, improve Triggers
commit c61115502726ed5aed7e70f451433aaebf583d6c
Author: Brian Warner <warner at lothar.com>
Date: Tue Feb 9 16:22:05 2010 -0800
fix test_run.Triggers
commit 9b68826f39408adca18f0fc0f7dfcafc6c282430
Author: Brian Warner <warner at lothar.com>
Date: Tue Feb 9 14:32:07 2010 -0800
test_run: merge Disconnect2 into Disconnect.testSlaveTimeout, remove
test_null
commit 72148bc764063d1c947ce0709a9fa8be5809b3bf
Author: Brian Warner <warner at lothar.com>
Date: Tue Feb 9 14:20:19 2010 -0800
runutils.MasterMixin.tearDown: use wait_until_idle(), works better
commit 2b4f36b1ad33fadcf8243fb593fbc955c4251d65
Author: Brian Warner <warner at lothar.com>
Date: Tue Feb 9 13:58:55 2010 -0800
test_run.Disconnect: delete remaining fragile tests
commit c4b77a3a834e0ef77d87945adf3d40fcd83ffa36
Author: Brian Warner <warner at lothar.com>
Date: Tue Feb 9 13:37:58 2010 -0800
fix test_run.Disconnect.(testDisappear|testDuplicate)
commit e1199b0b8b3f7737c6cb57d5bab09bd2a0a16f4c
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 8 16:49:57 2010 -0800
make test_run.Disconnect.testBuild1 work
commit 341ba2949ac7da50bbe7cc296228636a0c1b4408
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 8 16:48:38 2010 -0800
eventual.py: tolerate flushEventualQueue inside an eventual send call
commit 372272bb1e7f0ff70e034cfe2334ab85d00ca31b
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 8 09:32:16 2010 -0800
test_run: make Disconnect.testIdle work, add is-db-idle() helper
commit d4dfe912f3df0f0738b67d965026b7dfe8e5c93d
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 4 22:05:57 2010 -0800
status/web/grid.py: remove unused definition of grid_css (pointed at
the wrong place anyways)
commit 0f64fba019b72dce1c62308d60a2bfe7d5a1f85c
Author: Brian Warner <warner at lothar.com>
Date: Thu Feb 4 22:02:24 2010 -0800
test_run: get 5 test classes working (out of 15): Run, CanStartBuild,
Ping, BuilderNames, Basedir
commit d72bdbf23923abc2b77031fce24422bb769b4d35
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 17:02:19 2010 -0800
test_web/test_webparts: remove "coding:", ASCIIfy the test files
commit 8294027bd6d91dfcf01151b3d04473b6883b808a
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 16:58:07 2010 -0800
test_vc: use e.g. \N{LATIN SMALL LETTER C WITH CEDILLA} instead of raw
utf-8.
This returns test_vc.py to being ASCII, which is much easier to work
with. My git frontend (magit) and 'patch' were unhappy with the binary
characters.
commit f489d6167e95fbecc21be0b50c665aac3fa173c3
Author: Chris AtLee <catlee at mozilla.com>
Date: Wed Feb 3 15:10:11 2010 -0500
Fix test_reconfig
commit cb619e77022f4ab2e3976f1d906181ceba595dce
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 11:40:16 2010 -0800
test_webparts: fix
commit bf87dbabb8d3ce9dc78bc36c43bdc3d1e39d04b8
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 09:23:07 2010 -0800
test_webparts: fix quotes, remove trailing whitespace. No functional
changes.
commit a8167b5e4628fa38d9f9e07dffd83fe21bb38390
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 09:21:24 2010 -0800
test_ordering: fix
commit ef9d62a2beae1c0a2ff4d1746c7db22ec7ec2e0c
Author: Brian Warner <warner at lothar.com>
Date: Wed Feb 3 09:20:17 2010 -0800
add code-coverage tools: use "make test-coverage" to start
commit 71afe49d3ce9662f4e06ffad518b25b23dc5b111
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 1 17:22:20 2010 -0800
test_ec2buildslave: fix
commit 39ecf8bb290714e3239ec7dd19adf5e78d7c41e3
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 1 16:45:05 2010 -0800
test_dependencies: fix
commit d41e77675085b584be165a197f55dfd3aee03cf2
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 1 16:03:56 2010 -0800
fix test_control
commit c96fb543cbdbf8d4364f3364c33ef91e6596447d
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 1 14:51:03 2010 -0800
fix test_console
commit ba50ab754d52ca98625b49230ba1525bfa65c93f
Author: Brian Warner <warner at lothar.com>
Date: Mon Feb 1 14:40:15 2010 -0800
fixes: test_buildreq, test_buildset, test_changemaster, test_locks
Split out MasterMixin from RunMixin, to allow consistent basedirs.
Change the 'buildsets' table to only mark buildsets as complete
after *all* buildrequests have been retired, not just the first failing
one ("complete" vs "successful"). Update schedulers to match.
commit cd18078b31ebae5ff37bf3f5109fc59b174b736a
Author: Brian Warner <warner at lothar.com>
Date: Mon Jan 25 18:04:57 2010 -0800
test_config: bring up-to-date with DB changes
commit 3c469373e3596d932729f8d5f5f4c36deba6eadc
Author: Brian Warner <warner at lothar.com>
Date: Sat Jan 23 13:18:26 2010 -0800
fix test_db: treeStableTimer=None means don't merge changes, so use
0.01 instead
commit ced5cb09f64217b47044295881112bde475f71ac
Author: Brian Warner <warner at lothar.com>
Date: Sat Jan 23 12:55:19 2010 -0800
sendchange: remove --revision_number argument, now that all revisions
are strings
fix test_changes to match, by removing --revision_number tests.
commit 55ce64c9fd7185c3bdc5d48a6d3c54a22ad70126
Author: Brian Warner <warner at lothar.com>
Date: Sat Jan 23 12:47:09 2010 -0800
p4poller: revisions are strings, not ints. Clean up Deferred errbacks,
log.err problems, fix tests to ignore logged errors when testing bad
input.
commit 251cb9343f7933d7d37704663d8dfb07c6afff82
Author: Brian Warner <warner at lothar.com>
Date: Sat Jan 23 12:24:55 2010 -0800
fix test_status: revisions are now strings, never ints.
Also changed status/words.py to use str() instead of repr(), to avoid
the extra quotes. This relies upon revisions being ascii bytestrings.
commit 850d470d8d70825888ddd947ca9865aaef1e71a8
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 22 17:48:47 2010 -0800
fix test_webparts by updating web/console.py to use ChangeManager's
real API
I'm not entirely sure this is correct.. we pull 25 changes from the
database, but then look for 40 or 80 revisions. This needs more
extensive manual testing.
commit 205cce8f74de5fccb4dffddf971703d508f846d9
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 22 17:32:41 2010 -0800
fix test_runner: "buildbot create-master" is no longer idempotent,
bails on retry
commit 1133e741e7a5214c905e71e796fba60d83ce9535
Author: Brian Warner <warner at lothar.com>
Date: Thu Jan 21 14:06:15 2010 -0800
ChangeManager: remove unused Queue class
commit d6bd4299155bc79e3f83423bb592e6ee599b42fd
Author: Brian Warner <warner at lothar.com>
Date: Thu Jan 21 13:47:00 2010 -0800
base64-encode patches in SourceStamps, since they are arbitrary
bytestrings
commit 65825e8d73a862bda5a44909d030766f860a2abc
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 02:49:23 2010 -0800
accept unicode buildernames
* update buildbot/config.py to accept type==unicode, not just str
* change __repr__ in process/builder.py to only return ASCII
* change test_vc.py to use real unicode buildernames, not 8-bit
bytestrings
commit c4186f6c86a88b7debbdd2951b7da7adbf7b6b6c
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:01:22 2010 -0800
retry builds interrupted due to slave disconnect
Also add RETRY to the list of css_classes, now that's a possible
result.
commit d7da67fc92be639de7d6c3a9ebd62509da820916
Author: Brian Warner <warner at lothar.com>
Date: Wed Dec 16 13:38:33 2009 -0800
Scheduler: refine meaning of treeStableTimer=None to submit a separate
BuildSet for each Change.
commit 3d8ad3f55a2203a6d78361045ae63fe3bdf143c6
Author: Brian Warner <warner at lothar.com>
Date: Wed Dec 16 13:20:57 2009 -0800
add ['mergeRequests']=True to the Builder setup dictionary, for simple
per-Builder control of merging
commit f9bd3871de76219a281879e6466b43e2ce6bad55
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 02:21:44 2010 -0800
schedulerdb tests: update about a third of the test suite
Lots of little interfaces changed, so most tests need revision.
BuildMasters need to be constructed with an existing database, so a lot
of setUp() methods must be changed to create the DB before creating the
BuildMaster. And loadConfig() can no longer be called synchronously,
requiring some Deferred-chain juggling.
commit 347ee639fdbd606580f12a9b73f099119b9b9cb5
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 02:18:42 2010 -0800
schedulerdb 5-of-5: Builders, everything else
This contains the rest of the main schedulerdb changes. Most of the
changes are in buildbot/process/builder.py, which asks the database for
unclaimed BuildRequests instead of maintaining a .buildable attribute:
this allows queued build requests to survive a master bounce. Once a
build is started, the Builder must claim the BuildRequests by marking
them in the database, and builds that complete are similarly updated.
Much of the IBuilderControl functionality changed, since now the
database is used to manipulate queued builds.
buildbot/master.py also changed extensively. The database manager
object is created in the constructor and made available to several
other services. Builders and Schedulers are now Service instances, so
they must be handled differently during config reloads. A separate
SchedulerManager class is the parent for Schedulers, and manages the
Loop which drives them. The Deferred that loadConfig() returns *must*
be waited upon: the actual config changes will not occur right away, so
many unit tests need to be updated. Some of the scheduling
functionality of the BotMaster was moved, as were things like
submitBuildSet. Builds are always driven with BuildSets now (previously
they were optional), so the IControl methods are different, requiring
updates in other places.
The Status implementation was changed slightly, since BuildSets,
SourceStamps, and Changes now live in the database instead of pickles.
Some methods were added to retrieve DB-specific information (like
unique ID numbers) for these objects, or to exchange an ID number for
row contents. The top-level Status object uses DB notification
callbacks to implement existing subscription interfaces.
commit cd42dfac529e629aa7f4cdd06d2cb181cd25c1f6
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 02:04:01 2010 -0800
schedulerdb 4-of-5: Schedulers
Schedulers are drastically different now. They store their internal
state in the database, rather than as instance attributes, to allow the
buildmaster to be bounced without losing scheduler state. The
Change-based schedulers read information about Changes out of the DB
instead of subscribing to hear about new changes via method calls. The
timer-based schedulers use the wake-me-later feature of Loop to arrange
to be run later. All Schedulers submit BuildSets via a database call
instead of calling a method on the BuildMaster.
Try schedulers and Dependent schedulers effectively watch the database
to find out about the builds they need to follow (Try needs to provide
status information via PB, and Dependent needs to know when the
upstream builds have succeeded). Some of this code is in
buildbot/status/builder.py . Triggerable schedulers had similar issues.
Finally, Schedulers have been split into a number of smaller separate
files (buildbot/schedulers/*.py), with the original
buildbot/scheduler.py remaining as an import stub for the benefit of
existing master.cfg files.
commit 3290479a6227a690a5c1b586e2334d36950f30e5
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:56:00 2010 -0800
schedulerdb 3-of-5: BuildRequest/BuildSet
The canonical storage of a build request is now in the DB, but a
separate BuildRequest class is defined (in buildbot/buildrequest.py,
split out from buildbot/process/base.py) to pass around to code that
needs it. Likewise the old BuildSet class has been removed and its
funcionality moved into the database. The scheduling methods on
BuildSet (like waitUntilSuccess and start) have been subsumed by other
changes to the Schedulers and Builders, so buildbot/buildset.py was
removed altogether.
commit 18aa43449351f01e8b0d6d1fede4f662786f953b
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:51:37 2010 -0800
schedulerdb 2-of-5: Changes
Changes are now stored as database rows, instead of as list elements
inside a pickled ChangeMaster instance. Most of ChangeMaster was
removed, except for a small stub to allow old pickles to be loaded for
migration purposes (during "buildbot upgrade-master").
The new ChangeManager class (moved out to changes/manager.py) is
responsible for reading from the database and returning instances to
code that still expects a Change instance.
Some old implicit restrictions are now made explicit: branch names must
be ASCII bytestrings, revisions must be ASCII bytestrings too (not
ints, so SVN code must expect and provide strings). Databases care more
about types than pickles did, and anything which is interpolated into a
shell command (like "checkout") must be a bytestring. The patches
inside SourceStamps must be bytestrings too, not unicode objects.
commit 79b1e36d1769561eaee6f57130b85ce9780a6415
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:42:21 2010 -0800
schedulerdb 1-of-5: main database code, create/upgrade-DB in runner.py
All database access occurs in buildbot/db.py, as well as DB creation,
initial schema initialization, and upgrade. The database adapter is
derived from twisted.enterprise, and (in theory) can use anything with
a python DB-API interface, however SQL differences are likely to limit
actual compatibility. SQLite and MySQL are the initial targets.
runner.py contains code to create the database in both "buildbot
create-master" and "buildbot upgrade-master" (which also migrates
Changes from the now-obsolete changes.pck file). The database must
exist before "buildbot start" is used on the master. The database
specification (a class which controls which type of database to use,
and the arguments to pass to its constructor, like filename, or
dbhost+username+password) is created in the .tac file and passed as an
argument to the BuildMaster() constructor.
Several objects which used to exist as separate (pickleable) classes
are now expressed as rows in the database: BuildSets, BuildRequests,
Properties, Changes, and SourceStamps. Some of these still have classes
defined for the sake of older code, even though their canonical
representation is in the DB. For immutable objects like Changes, we use
an LRU cache and try to avoid creating duplicate instances for the same
notional object.
Several extra fields exist in the DB to support performance
metrics (e.g. how long a build spends waiting for a buildslave) and
scheduler dashboard tools. There is currently no garbage collection,
but the rules under which rows may be deleted are defined in a comment.
Thanks to Kristian Nielsen and Chris Atlee for invaluable DB work.
This commit is the first of five that contain the core functionality of
the schedulerdb project: they are not useful without the others, but
are broken up for ease of review. There are other patches before and
after these which contain more isolated changes. The whole batch (about
a dozen) is a git rewrite of roughly 90 older revisions that included a
number of wrong turns and dead ends. The full changelog for the
original sequence is included here for history buffs.
commit efcd0328da590defe10227054b5441d58ae4f217
Author: Brian Warner <warner at lothar.com>
Date: Sat Dec 19 17:23:46 2009 -0800
db.py: add start_time/finish_time to [builds] table, for
performance dashboards
commit de7c571fa5a2b1edee0c682f715b2e5b571f795d
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 17 13:14:39 2009 -0800
db: str-ify branch=/revision= when creating Change instances, to
avoid tripping the assertions I just added
* test_vc.Sources.testSVN2: use string revisions, not ints
commit 2581c545587c9ef526b68e65a349f3fd3562f214
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 17 13:07:47 2009 -0800
changes.py: assert() more about bytestring branchnames and
revisions
commit ef871d8452721e2a590b2e088debf4b689aa9342
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 17 12:58:41 2009 -0800
sourcestamp.py: assert() more about bytestring branchnames and
revisions
commit f3f3a12404931bfc0b72d3e3be104d84ad80dd74
Author: Brian Warner <warner at lothar.com>
Date: Wed Dec 16 13:38:33 2009 -0800
Scheduler: refine meaning of treeStableTimer=None to submit a
separate BuildSet for each Change.
commit 0149db322141c3c667885569285e744ebb67165f
Author: Brian Warner <warner at lothar.com>
Date: Wed Dec 16 13:20:57 2009 -0800
add ['mergeRequests']=True to the Builder setup dictionary, for
simple per-Builder control of merging
commit 62b686815459a69b31a3082c49d7f16bfc2062a4
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 10 16:50:54 2009 -0800
fix db.start/stop, so real buildmasters and unit tests both work
commit 4bbaae6006ed6b4fd2188a54fd316a852f265af2
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 10 16:44:14 2009 -0800
make test_properties work
commit 065b965c2bda63a957eca48f1b106a33f82d03a3
Author: Brian Warner <warner at lothar.com>
Date: Mon Dec 7 21:09:48 2009 -0800
fix test_slaves
* builder.py: slaves which disconnect while we're doing
db.get_unclaimed_buildrequests are treated as if they
disconnected during the initial ping, to fix this race condition.
Remove .buildable, add getBuildable() to replace it.
* schedulers/manager.py: fix signature of trigger_modify_buildset()
* status/builder.py: fix signature of _db_buildsets_changed()
commit 0bd05395ebb975401a359a063a139bb403836a41
Author: Brian Warner <warner at lothar.com>
Date: Fri Dec 4 16:15:10 2009 -0800
test_status.py: make it work
commit 16bdb6d084fdc4fbc9c62e1ec20b3daf83b79ac5
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 3 23:41:06 2009 -0800
test_steps: make it work (with run_one_build), hush pyflakes
commit ea8b11f1ae9436108e756d84d30edd1f985c404e
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 3 23:38:09 2009 -0800
test_vc, runutils: factor out run_one_build()
commit 735429933774b1544002b9ca8fcc6729214cd434
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 3 12:02:07 2009 -0800
oops, fix BotMaster, allow reconfig to work
commit 9420a337e21bb9f7aeb17110c5cef587f377658c
Author: Brian Warner <warner at lothar.com>
Date: Thu Dec 3 12:01:04 2009 -0800
make test_vc.py work, fix patch handling in SourceStamps
commit 885f152c3b06178ef7aa2465d3fccaa37c96eef9
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 30 17:47:37 2009 -0500
make test_webparts. Also re-enable the pool.start() in
DBConnector.__init__ .
commit 54f478f5ee5a4f91101348d957434f3be163ebc4
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 30 17:22:41 2009 -0500
start making unit tests work: test_web now works
* replace callLater(0) with eventual-send, so tests can flush the
queue at tearDown
* have each Loop assert that it is running whenever they get
triggered
* rewrite test/runutils.py to build a fake DB
* BotMaster: tolerate non-BuildSlave children (like the Loop
instance) when replacing the builders
commit 596a62d59883d53b664d33e7e632c3eefc4016d0
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 30 17:17:54 2009 -0500
add eventual-send operator (copy eventual.py and tests over from
Foolscap)
commit 99b414b8d8610c4b1fe681ff2d9eab4381b008f1
Author: Brian Warner <warner at lothar.com>
Date: Wed Nov 25 11:34:49 2009 -0800
db/status.builder: fix boneheaded bugs
commit 2131e7b902e3a67417f82c93261ebdc069355054
Author: Brian Warner <warner at lothar.com>
Date: Wed Nov 25 11:33:50 2009 -0800
loop: make it a MultiService, stop timers in stopService
commit 79c264df79a73479f8c249de624f02c75db37745
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 18:14:38 2009 -0800
replace submitBuildSet/submitBuildRequest with new working form.
Bring back words/web forceBuild functions.
* create BuilderControl directly, not with adapter; give it a
Control reference
commit 382fecd9288688d40ebddbdfa54b528db0989ecf
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 18:10:08 2009 -0800
remove buildset.py, it's been subsumed by DB rows
commit 069d14c0a06cc17a1496a6606ed1412f47942de5
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 18:09:38 2009 -0800
schedulers: remove now-redundant Builder notification call, rely on
DB.notify
commit 97cd95770d943cfcfb2f5b2568998c1c0faf454b
Author: Brian Warner <warner at lothar.com>
Date: Thu Jan 7 15:11:53 2010 -0800
big changes: port Try schedulers, DB-based notification system,
'builds' table
also:
* move BuildRequest out of buildbot.process.base
* move most BuildRequest/BuildSet interaction into db
* implement BuildRequest.subscribe/BuildSet.waitUntilFinished with
db
* stop requiring db.retire_buildset() caller to wake schedulers,
use notify()
* extract BuildRequestStatus/BuildSetStatus from db
* temporarily disable web/irc "force build" button
- until new submitBuildRequest interface is created
* disable IBuildRequestControl methods
commit b314d4a4f41f6cf1c4292cba66f61cbda101ae44
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 14:46:13 2009 -0800
tryclient.py: fix long-standing bugs: PB code would connect twice,
build results checked incorrectly
commit 88074ffde42b08e008079519eddbcc92547d151b
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 14:42:19 2009 -0800
util.py: add DictOfSets
commit e1ac70caaad83f460ef5d06c39ce85f1b6d07346
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 23 14:11:50 2009 -0800
schedulers/timed.py: return absolute wakeup time, handle first
build correctly
commit 8f5241054068445464a7569e60e445fa99d62377
Author: Brian Warner <warner at lothar.com>
Date: Tue Nov 10 17:39:15 2009 -0800
BaseScheduler.create_buildset: allow builderNames= to be
overridden, for Try
commit fe8687ff5a73508a354a21512cd9bc82ec324187
Author: Brian Warner <warner at lothar.com>
Date: Thu Jan 7 15:49:58 2010 -0800
retry builds interrupted due to slave disconnect
Also add RETRY to the list of css_classes, now that's a possible
result.
commit 1143968e9f60226b298229cfed326f1401a458b3
Author: Brian Warner <warner at lothar.com>
Date: Thu Nov 5 17:30:22 2009 -0800
schedulers: factor out get_state/set_state into _Base
commit f1f60d02c436957434e69fd085da2dbd8030c4e2
Author: Brian Warner <warner at lothar.com>
Date: Thu Nov 5 17:12:15 2009 -0800
DB.get_unclaimed_buildrequests: oops, forgot the join, causing
builds to acquire more and more Changes over time
commit 0375d831b3e696ceb767551bd930ca79abf282c4
Author: Brian Warner <warner at lothar.com>
Date: Thu Nov 5 16:11:10 2009 -0800
db.py: drop the "nonpool" connection on error, to provoke a new
connection next query. This fixes e.g. Waterfall after the database
has been bounced.
commit 6c5394f9b62cda9559ece20b3a012aa1c3e953d7
Author: Brian Warner <warner at lothar.com>
Date: Thu Nov 5 15:07:35 2009 -0800
master: recognize+steal our earlier claims quickly, by looking for
a matching mastername but different incarnation
commit f298aade17654d69b2a349e1e48a86f0d0ab4a30
Author: Brian Warner <warner at lothar.com>
Date: Thu Nov 5 12:22:08 2009 -0800
db.py: record start/finish times in both buildsets and
buildrequests, for historical monitoring
commit b2fd7f4f3bce0b1b3bcce3b48d0fc5399d8e45fa
Author: Brian Warner <warner at lothar.com>
Date: Wed Nov 4 17:12:34 2009 -0800
scheduler.py: replace old non-DB schedulers with imports of
schedulers/*.py
commit 75c9b82a57bafcb206df3f248a1e6d661a301151
Author: Brian Warner <warner at lothar.com>
Date: Tue Nov 3 12:23:09 2009 -0800
BotMaster.stopService: oops, upcall to the right parent class, to
fix test_db
commit 393b4e6e9132709d8a21006667c7691c369e0a5d
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 2 17:47:07 2009 -0800
db.claim_buildrequests: don't reclaim an empty list
commit b7b3882c4b97a80060ad62c15d73ade03d9962ba
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 2 17:35:04 2009 -0800
Builder: update the claim on active builds every 10 minutes.
Made Builders into service children (of the BotMaster) to make it
easier to manage the timer. Changed db.claim_buildrequests to
create a transaction if necessary.
commit 03e17bc4ad0b846f006dee0effe7dae50d2dec4b
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 2 15:35:27 2009 -0800
Triggerable: port to DB, use buildset subscriptions to implement
waitForFinish
db.py: replace create_sourcestamp with get_sourcestampid, add
ssid=None to SourceStamp, to make it easier to start builds
with old SourceStamps
commit 6e1fa2184c23d196f3727bc7b0836287caa7e55c
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 28 16:34:04 2009 -0700
make Dependent schedulers work.
- always create BuildSet
- remove BuildRequest properties, use BuildSet props instead
- move sourcestampid and reason to BuildSet
- db.retire_buildrequests is responsible for retiring BuildSets
too, and triggering the Scheduler loop as necessary
- added DB table for downstream schedulers who've subscribed to a
buildset
- move defaultdict to buildbot.util
commit a11e3a20ce6b91d18f3503dc846f98507ae3921e
Author: Brian Warner <warner at lothar.com>
Date: Tue Oct 27 14:10:52 2009 -0700
fix SQL to work with MySQL, thanks to Chris Atlee for doing all the
work
commit dece30624d9ffeb66997c889190ffb73c3685436
Author: Brian Warner <warner at lothar.com>
Date: Tue Oct 27 10:26:06 2009 -0700
test_config: update about half of the tests to the new setup
process
commit 433710cb74d58ecdff07cd4fd70c1c7702d25efd
Author: Brian Warner <warner at lothar.com>
Date: Tue Oct 27 10:24:54 2009 -0700
runutils.py: add ShouldFailMixin
commit 196aa14e1cbf61db69de483c0bfed3ba531ad318
Author: Brian Warner <warner at lothar.com>
Date: Mon Oct 26 17:07:16 2009 -0700
port Triggerable, except SourceStamp handling is incomplete
commit 16abcad4d60409b117c37b78f08ab62abd007435
Author: Brian Warner <warner at lothar.com>
Date: Mon Oct 26 16:21:03 2009 -0700
port AnyBranchScheduler, Periodic, Nightly to new DB-based scheme.
Untested.
commit ccea6ceb18b2ab739eab259c57677ff794efaa6b
Author: Brian Warner <warner at lothar.com>
Date: Thu Jan 7 15:56:33 2010 -0800
split new schedulers out into separate schedulers/ files
add buildbot.schedulers to list of packages in setup.py
commit ab20201ba1fce8e183483a29efba1252d128dfbd
Author: Brian Warner <warner at lothar.com>
Date: Mon Oct 26 10:42:17 2009 -0700
test_change.py: make it work, allow revision=None in Changes (for
CVS)
commit 3317ccffba2d4e88b27dc9f331d499523dbb8210
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 16:54:52 2009 -0700
db.py: add a generic key-value store, for things like ChangeSources
to use
commit d75479863a02851143c7b5cf6f4615c852af64e8
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 16:35:30 2009 -0700
move Builder DB-using code into db.py
commit 9eccfb15a5ca24c97e4843d8b446536b0d166af5
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 16:27:21 2009 -0700
consolidate BuildRequestManager into db.py
commit fa41492ef1f74e2d0854102fe9cf31cb03618502
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 16:20:32 2009 -0700
move Scheduler DB calls into db.py
commit 9e071c45e3544cf5a26dff46844627a59a776039
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 15:33:54 2009 -0700
consolidate get_properties_from_db() into db.py
commit c69bb320946518703d4c657c38dd5e450e2d4320
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 15:24:56 2009 -0700
consolidate SourceStampManager into db.py
commit 94c660162f5e55ed08c1f075a7408e05779b6a2a
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 15:11:40 2009 -0700
consolidate ChangeManager DB-using code into db.py
commit 8fa122ee351657569852cc01f628ea22de820059
Author: Brian Warner <warner at lothar.com>
Date: Fri Oct 23 14:43:11 2009 -0700
test_changemaster: make it work, but changeHorizion= is disabled
for now
commit b11e68f35a7b396d30af6a0d892ba1a7d9ecb1b0
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 22 16:43:50 2009 -0700
ChangeManager: update getChangeNumberedNow, now waterfall doesn't
crash! yay!
commit 35dfca9be8df20bace9998c92a8fb3e725871682
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 22 16:22:21 2009 -0700
Scheduler: don't use ComparableMixin, since it interferes with
using Schedulers as dict/set keys inside a Loop. Trigger the
Builder loop at the end of the whole scheduler run, not in the
middle, since the DB transaction will not have finished yet.
This gets me working builds, however status displays are still
broken.
commit 2253c70cc84d286abe8fa6379aee10fbde95679d
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 22 16:20:24 2009 -0700
loop: add loop_done() hook
commit 2de4d8178023feb03743f09820567d271128157c
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 21 16:47:11 2009 -0700
scheduler: provide per-Scheduler state, rewrite in terms of
transactions
commit 525d1cdb23648f7d2c4426bfed319ec82b9a5de8
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 21 15:40:16 2009 -0700
scheduler: factor out a base class, to make porting other
Schedulers easier
commit 292cbadbe066ad3acf8fbda9f57e393d4f17e658
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 21 14:25:17 2009 -0700
update Builder to use DB, add basic unit test
commit 93d72a55340ec4465c3f10159b9395018c3fb272
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 21 14:16:17 2009 -0700
fix BuildRequest/SourceStamp processing
commit cd85afcbe5c2bde1f74112081ca04c2b186b00a4
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 15 14:36:51 2009 -0700
fix get_properties_from_db, make ChangeManager use it
commit 8ddd47ec318a06efab435c99010030e9f9453f85
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 15 14:35:11 2009 -0700
fix LRUCache and its users
commit 87269482fa70ed246152183e76559a014306607a
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 14 16:24:54 2009 -0700
Create "sourcestamp manager" to replace getSourceStampNumberedNow
function, create new buildbot.buildrequest.BuildRequest and
BuildRequestManager to fetch buildreqs, add
get_properties_from_db() utility
commit e2d94ae5cfe99b73591102b0fbbcb9d7299bff90
Author: Brian Warner <warner at lothar.com>
Date: Wed Oct 14 16:23:35 2009 -0700
add util.LRUCache, make ChangeManager use it, add t= to
ChangeManager.getChangeNumberedNow
commit 07d3069d5ef764f265626ac8f14aa8b16734d4b0
Author: Brian Warner <warner at lothar.com>
Date: Tue Oct 6 15:05:56 2009 -0700
wire the changes/scheduler/builder Loops together properly
Builders still aren't converted, so this will break when it
actually tries to start a build.
commit 21a42e09478608abf23e2bc8cc961113064ac425
Author: Brian Warner <warner at lothar.com>
Date: Tue Oct 6 11:37:28 2009 -0700
loop.py: add DelegateLoop, which takes a callable to get the list
of processes
commit 965459facad9ff4e1a81ddc91463fe818c69be0a
Author: Brian Warner <warner at lothar.com>
Date: Mon Oct 5 21:17:11 2009 -0700
db: add submitted_at to BuildRequest record
commit e11328e8d902fcb0ae785186df0102e0f0d965f5
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 1 16:08:37 2009 -0700
minor formatting changes, plan for scheduler to notify
slave-manager
commit 3c081750f7693c6e293bfb79cce298a983cc5bf6
Author: Brian Warner <warner at lothar.com>
Date: Thu Oct 1 15:56:43 2009 -0700
Finish basic Scheduler: now puts BuildRequests in the db.
Defined BuildSet/BuildRequest/SourceStamp tables, rewrote Scheduler
to use db.runInteraction. Added
sourcestamp.getSourceStampNumberedNow(). Fixed test_db to use
separate per-test workdirs, and to assert the scheduler actually
created BuildRequests in the db.
commit 9b95d587a5cd9559a507368a70a8196aaad7dd81
Author: Brian Warner <warner at lothar.com>
Date: Wed Sep 30 12:25:05 2009 -0700
Update new Scheduler to use wakeup API, add basic Scheduler tests.
The Scheduler doesn't yet add BuildRequests to the db, so much of
the tests are stubbed out.
commit 6c3104c1642309936b0e4336c4580e0cbbeec811
Author: Brian Warner <warner at lothar.com>
Date: Wed Sep 30 12:18:01 2009 -0700
Add wakeup timers to Loop, to help tree-stable-timers in
Schedulers.
commit dd42c5f867cacb6ba4fbdb386485d51e7eeaa803
Author: Brian Warner <warner at lothar.com>
Date: Thu Sep 17 18:01:55 2009 -0700
Start on new Scheduler: check for unexamined Changes, classify
each, note when tree-stable-timer is expired and there are
important changes to build. Attach schedulers to a Loop, fire
them (for now) every 30 seconds.
Other cleanups: add async ChangeManager.getChangeByNumber(),
getChangesByNumber, and getChangesGreaterThan. Add new scheduler
tables. Don't manually start the adapi pool to avoid sqlite
segfaults. Create MultiServiceLoop to manage schedulers and other
things that want to be driven by a Loop and be stored in a Service
hierarchy.
Everything is still quite broken. No code to actually trigger
builds. Lots of debug prints. I'm seeing sqlite3.OperationalError
"database is locked" exceptions with multiple parallel INSERTs,
which may either need auto-retry or more code to limit/avoid
concurrency.
commit 358744f98e7fbefe942f4e1e404ac68faa25b6c2
Author: Brian Warner <warner at lothar.com>
Date: Tue Sep 15 18:05:10 2009 -0700
Add notification-triggered processor-loop handling class. When
somebody modifies the database and sends a message through the
notification server, this is the code that will invoke the
Schedulers (or Builders) one-at-a-time, non-reentrantly, and as
many times as is necessary.
commit 7605a9163587d1b075fb455e0eb97ac10f43337b
Author: Brian Warner <warner at lothar.com>
Date: Tue Sep 15 16:50:17 2009 -0700
Work with MySQL as well as SQLite: this involves schema changes and
using a placeholder specific to be backend ("%s" for MySQL, "?" for
SQLite). Clean up the schema, merging "changeid" and "number"
together as suggested by Kristian Nielsen.
Also shut down the threadpool when we're done, to fix some
destructor exceptions, and make runner.py really exit, to fix a
hang.
commit 731344def10327b3bd7257ca0de2ebba1c9aa28a
Author: Brian Warner <warner at lothar.com>
Date: Mon Sep 14 17:21:57 2009 -0700
set DB type in buildbot.tac, rather than master.cfg . Add --db
argument to create-master and update-master, to specify the DB to
be created.
commit c60fde95d6cb5b543ccfb8809a5bd93709ba467f
Author: Brian Warner <warner at lothar.com>
Date: Mon Nov 2 17:44:02 2009 -0800
add DB-migration of changes.pck to "upgrade-master" tool, create DB
in "buildbot master".
* Create ChangeManager class, leave stub of old ChangeMaster in
place for pickle compatibility.
* Implement DB-based getChangeNumbered and eventGenerator methods
in ChangeManager.
* Add unit tests.
The new test_db works, but lots of others are currently broken.
However, this version shouldn't clobber changes.pck like the
previous one.
commit 917831317892337517ad4a81d1e3fe3781682b9d
Author: Brian Warner <warner at lothar.com>
Date: Wed Sep 9 18:48:25 2009 -0700
database-backed scheduler project: first steps. Create sqlite
database, store changes in it (in addition to the changes.pck
pickle and in RAM), nothing more.
Broken in this snapshot: reading of old changes.pck pickle, but it
still gets written, so using this version will clobber that file.
Still to do for this milestone: read changes.pck on demand, make
eventGenerator/getChangeNumbered async, stop writing to
changes.pck, move classes out of the way so a pickle-loading
migration tool can be written, write that migration tool.
commit 3e80bcb2e9dc2930e41ab615b4d2d15ccfc65982
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:20:46 2010 -0800
eventual.py: add eventual-send operator, copied (with tests) from
Foolscap
commit 21e12be626f835a3a7c040a0c319be96d13f68d7
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:19:13 2010 -0800
loop.py: a notification-triggered processor-loop handling class
When somebody modifies the database and sends a message through the
notification server, this is the code that will invoke the
Schedulers (or Builders) one-at-a-time, non-reentrantly, and as many
times as is necessary. It is a MultiService, and cleans up in
stopService (to make tests easier). The code that it runs can return a
wakeup time to get re-run at some point in the future. A loop_done()
hook is available too.
commit ee9c72f6bb3d63e5c0ba41222e9f04853a678b9d
Author: Brian Warner <warner at lothar.com>
Date: Fri Jan 8 01:15:03 2010 -0800
util.py: add DictOfSets (with tests), LRUCache, defaultdict (for py2.4)
Commit: 0e7f18a328e44d2c54b663fc46d5b4ab54706687
http://github.com/djmitche/buildbot/commit/0e7f18a328e44d2c54b663fc46d5b4ab54706687
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/changes/changes.py
A buildbot/changes/manager.py
M buildbot/changes/p4poller.py
Log Message:
-----------
schedulerdb 2-of-5: Changes
Changes are now stored as database rows, instead of as list elements inside a
pickled ChangeMaster instance. Most of ChangeMaster was removed, except for a
small stub to allow old pickles to be loaded for migration purposes (during
"buildbot upgrade-master").
The new ChangeManager class (moved out to changes/manager.py) is responsible
for reading from the database and returning instances to code that still
expects a Change instance.
Some old implicit restrictions are now made explicit: branch names must be
ASCII bytestrings, revisions must be ASCII bytestrings too (not ints, so SVN
code must expect and provide strings). Databases care more about types than
pickles did, and anything which is interpolated into a shell command (like
"checkout") must be a bytestring. The patches inside SourceStamps must be
bytestrings too, not unicode objects.
Commit: 6f9156de6cbe239bb0a32515c41ecdee9cc23b85
http://github.com/djmitche/buildbot/commit/6f9156de6cbe239bb0a32515c41ecdee9cc23b85
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
A buildbot/buildrequest.py
R buildbot/buildset.py
M buildbot/process/base.py
Log Message:
-----------
schedulerdb 3-of-5: BuildRequest/BuildSet
The canonical storage of a build request is now in the DB, but a separate
BuildRequest class is defined (in buildbot/buildrequest.py, split out from
buildbot/process/base.py) to pass around to code that needs it. Likewise the
old BuildSet class has been removed and its funcionality moved into the
database. The scheduling methods on BuildSet (like waitUntilSuccess and
start) have been subsumed by other changes to the Schedulers and Builders, so
buildbot/buildset.py was removed altogether.
Commit: b965842be93f1be07056e69931762a3ae3e3dd59
http://github.com/djmitche/buildbot/commit/b965842be93f1be07056e69931762a3ae3e3dd59
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/clients/tryclient.py
M buildbot/scheduler.py
A buildbot/schedulers/basic.py
A buildbot/schedulers/manager.py
A buildbot/schedulers/timed.py
A buildbot/schedulers/triggerable.py
A buildbot/schedulers/trysched.py
M buildbot/steps/trigger.py
M setup.py
Log Message:
-----------
schedulerdb 4-of-5: Schedulers
Schedulers are drastically different now. They store their internal state in
the database, rather than as instance attributes, to allow the buildmaster to
be bounced without losing scheduler state. The Change-based schedulers read
information about Changes out of the DB instead of subscribing to hear about
new changes via method calls. The timer-based schedulers use the
wake-me-later feature of Loop to arrange to be run later. All Schedulers
submit BuildSets via a database call instead of calling a method on the
BuildMaster.
Try schedulers and Dependent schedulers effectively watch the database to
find out about the builds they need to follow (Try needs to provide status
information via PB, and Dependent needs to know when the upstream builds have
succeeded). Some of this code is in buildbot/status/builder.py . Triggerable
schedulers had similar issues.
Finally, Schedulers have been split into a number of smaller separate files
(buildbot/schedulers/*.py), with the original buildbot/scheduler.py remaining
as an import stub for the benefit of existing master.cfg files.
Commit: 10c01a73b5fadcab04c9f63542b0da36d761977a
http://github.com/djmitche/buildbot/commit/10c01a73b5fadcab04c9f63542b0da36d761977a
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/buildslave.py
M buildbot/config.py
M buildbot/interfaces.py
M buildbot/master.py
M buildbot/process/builder.py
M buildbot/process/buildstep.py
M buildbot/scripts/checkconfig.py
M buildbot/slave/commands.py
M buildbot/sourcestamp.py
M buildbot/status/builder.py
M buildbot/status/client.py
M buildbot/status/web/base.py
M buildbot/status/web/builder.py
M buildbot/status/web/console.py
M buildbot/status/web/grid.py
M buildbot/status/web/waterfall.py
M buildbot/status/words.py
M docs/buildbot.texinfo
Log Message:
-----------
schedulerdb 5-of-5: Builders, everything else
This contains the rest of the main schedulerdb changes. Most of the changes
are in buildbot/process/builder.py, which asks the database for unclaimed
BuildRequests instead of maintaining a .buildable attribute: this allows
queued build requests to survive a master bounce. Once a build is started,
the Builder must claim the BuildRequests by marking them in the database, and
builds that complete are similarly updated. Much of the IBuilderControl
functionality changed, since now the database is used to manipulate queued
builds.
buildbot/master.py also changed extensively. The database manager object is
created in the constructor and made available to several other services.
Builders and Schedulers are now Service instances, so they must be handled
differently during config reloads. A separate SchedulerManager class is the
parent for Schedulers, and manages the Loop which drives them. The Deferred
that loadConfig() returns *must* be waited upon: the actual config changes
will not occur right away, so many unit tests need to be updated. Some of the
scheduling functionality of the BotMaster was moved, as were things like
submitBuildSet. Builds are always driven with BuildSets now (previously they
were optional), so the IControl methods are different, requiring updates in
other places.
The Status implementation was changed slightly, since BuildSets,
SourceStamps, and Changes now live in the database instead of pickles. Some
methods were added to retrieve DB-specific information (like unique ID
numbers) for these objects, or to exchange an ID number for row contents. The
top-level Status object uses DB notification callbacks to implement existing
subscription interfaces.
buildbot.texinfo: document new treeStableTimer=None, mergeRequests features
schedulerdb 5-of-5: Builders, everything else
This contains the rest of the main schedulerdb changes. Most of the changes
are in buildbot/process/builder.py, which asks the database for unclaimed
BuildRequests instead of maintaining a .buildable attribute: this allows
queued build requests to survive a master bounce. Once a build is started,
the Builder must claim the BuildRequests by marking them in the database, and
builds that complete are similarly updated. Much of the IBuilderControl
functionality changed, since now the database is used to manipulate queued
builds.
buildbot/master.py also changed extensively. The database manager object is
created in the constructor and made available to several other services.
Builders and Schedulers are now Service instances, so they must be handled
differently during config reloads. A separate SchedulerManager class is the
parent for Schedulers, and manages the Loop which drives them. The Deferred
that loadConfig() returns *must* be waited upon: the actual config changes
will not occur right away, so many unit tests need to be updated. Some of the
scheduling functionality of the BotMaster was moved, as were things like
submitBuildSet. Builds are always driven with BuildSets now (previously they
were optional), so the IControl methods are different, requiring updates in
other places.
The Status implementation was changed slightly, since BuildSets,
SourceStamps, and Changes now live in the database instead of pickles. Some
methods were added to retrieve DB-specific information (like unique ID
numbers) for these objects, or to exchange an ID number for row contents. The
top-level Status object uses DB notification callbacks to implement existing
subscription interfaces.
buildbot.texinfo: document new treeStableTimer=None, mergeRequests features
Commit: ff021f62da31dd755e685c997266f8a789608610
http://github.com/djmitche/buildbot/commit/ff021f62da31dd755e685c997266f8a789608610
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
A buildbot/test/pollmixin.py
M buildbot/test/runs/test_console.py
M buildbot/test/runs/test_dependencies.py
M buildbot/test/runs/test_ec2buildslave.py
M buildbot/test/runs/test_locks.py
M buildbot/test/runs/test_mergerequests.py
M buildbot/test/runs/test_ordering.py
M buildbot/test/runs/test_properties.py
M buildbot/test/runs/test_reconfig.py
M buildbot/test/runs/test_run.py
M buildbot/test/runs/test_runner.py
M buildbot/test/runs/test_slaves.py
M buildbot/test/runs/test_status.py
M buildbot/test/runs/test_steps.py
M buildbot/test/runs/test_web.py
M buildbot/test/runs/test_web_status_json.py
M buildbot/test/runs/test_webparts.py
M buildbot/test/runutils.py
A buildbot/test/test_buildset.py
M buildbot/test/unit/test_authz.py
M buildbot/test/unit/test_buildreq.py
M buildbot/test/unit/test_buildstep.py
M buildbot/test/unit/test_changemaster.py
M buildbot/test/unit/test_changes.py
M buildbot/test/unit/test_config.py
M buildbot/test/unit/test_control.py
M buildbot/test/unit/test_package_rpm.py
M buildbot/test/unit/test_scheduler.py
M buildbot/test/unit/test_sourcestamp.py
M buildbot/test/unit/test_transfer.py
M buildbot/test/vc/test_vc.py
Log Message:
-----------
update all tests to work with the new schedulerdb
Big patch. Much of the work is to create a master (with a db) before
starting the test. It is now mandatory to wait for master.loadConfig's
Deferred. Master shutdown must be waited on too. And some of the
BuildRequest submission paths changed.
This also fixes a few pyflakes warnings.
Commit: 58c6c6132117a1fd715b233009b3e07aedcac616
http://github.com/djmitche/buildbot/commit/58c6c6132117a1fd715b233009b3e07aedcac616
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/changes/p4poller.py
M buildbot/test/unit/test_p4poller.py
Log Message:
-----------
p4poller: Clean up Deferred errbacks, log.err problems, fix tests to ignore
logged errors when testing bad input.
Commit: e6cfbdafd877a8c894ad86bf83d0f3d0921d0e04
http://github.com/djmitche/buildbot/commit/e6cfbdafd877a8c894ad86bf83d0f3d0921d0e04
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M buildbot/changes/bonsaipoller.py
M buildbot/changes/freshcvs.py
M buildbot/changes/pb.py
M buildbot/changes/svnpoller.py
M buildbot/clients/base.py
M buildbot/status/web/baseweb.py
M buildbot/status/web/build.py
M buildbot/status/web/grid.py
M buildbot/status/web/olpb.py
M buildbot/status/web/slaves.py
M buildbot/status/web/status_json.py
M buildbot/status/web/waterfall.py
M buildbot/steps/master.py
M buildbot/steps/shell.py
M buildbot/steps/transfer.py
M buildbot/util.py
Log Message:
-----------
hush most non-trivial pyflakes warnings
Commit: 4401e133365a7c39bf1f1a08822d23ddf04ad253
http://github.com/djmitche/buildbot/commit/4401e133365a7c39bf1f1a08822d23ddf04ad253
Author: Brian Warner <warner at lothar.com>
Date: 2010-02-14 (Sun, 14 Feb 2010)
Changed paths:
M .gitignore
M Makefile
A buildbot/test/trial_coverage.py
A contrib/coverage2text.py
A twisted/plugins/buildbot_trial.py
Log Message:
-----------
add code-coverage tools: use "make test-coverage" to get started
More information about the Commits
mailing list