[Buildbot-devel] What should be the default functionality of multiple repositories.

[Harry]

Here a short introduction for everyone who is waiting for support of
multiple repositories in a build.

 

We are now developing a feature in buildbot that makes it possible to submit
changes from more than one repository and will result in one buildrequest.

You will need this if your product is build from, for example, 3
repositories and you made changes in 2 repositories. A build should take the
sources (including the changes) from the two repositories (with the correct
branch and revision) and the sources (head or specific revision/branch) from
the remaining repository.

 

A scheduler will know about a set of repositories that form a single
(sub)product. 

The first thought was to get (from the scheduler) a sourcestamp for each set
of changes that have the same repository .. and a sourcestamp for each
remaining repository that does not have changes.

A buildstep with knowledge about a repository can ask the build for a
sourcestamp for that specific repository. In the example above, 2 buildsteps
would get a sourcestamp with revisions that include the changed sources and
1 buildstep would get a sourcestamp without changes but with a repository,
revision etc.

 

This solution works as long as all changes are always submitted from the
same repository provider (SVN-, CVS-, hg-, git, git-hub server). But what
happens if a developer submits changes from a local development  repository
to buildbot where the changes affect 2 of the 3 repositories. The
repositories in the changes are not recognized as the scheduler only knows
about the central repositories.

 

Therefore we introduced a logical identifier for each known repository:
codebase. The sources above are stored in three repositories, the first
repository gets the codebase 'main', the second repository gets the name
'database', and the third codebase is 'network'. Now we can assign different
repositories to the same codebase (https://hg/hg/prod1/main,
d:\develop\prod1\main, etc.). 

 

What happens when the new multirepo feature is introduced in existing
buildbot installations (no config change)

The current buildbot 0.8.5 version collects changes from all repositories
into one sourcestamp, this can results in build errors as not all changes
are pulled from all the repositories.

The new implementation would do the same, the scheduler still only has a
'repuUrl' but no set of codebases that it can process. All changes are
collected into one sourcestamp. The buildsteps do not have codebases
configured, they get the one-and-only sourcestamp from the build.

 

What happens when existing buildbot installations change their config to use
multiple repositories

The configuration of the schedulers is changed to have a set of codebases
which will all be used as input for the build.

Example: codebases = {'main':{'repository':'https://hg/hg/prod1/main',
'branch':'default', 'revision':'tip'}, 'database':{repository ...}

Each buildsteps is configured with a codebase, so it can get the right
sourcestamp from the build.

 

The configfile of buildbot is responsible for determining the codebase, of
incoming changes. It may have a callable that returns a codebase that is
based on the passed change. 

 

What happens when the configfile does not determine the codebase

If the codebaseGenerator callable is not defined in the configfile then
there are two options to implement in buildbot:

Codebase is set to ''

This means that a scheduler must be configured with
codebases={'':{repository='...'}}

All changes will be collected under one sourcestamp. If a sourcestep is
configured with a codebase it won't get a sourcestamp back unless it has the
configured codebase=''.

If a sourcestep has no codebase configured the build will return the
one-and-only sourcestamp.

Codebase is set to 'its repository '

            This means that a scheduler must be configured with
codebases={'https://hg/hg/prod1/main':{repository='http://hg/hg/prod..'}}

All changes will be collected under the sourcestamp with the same
repository. If a sourcestamp is configured with a codebase equal to the
repository it will get the correct sourcestamp.

If a sourcestep has no codebase configured then the build.getSourceStamp
will return an error saying that if one ore more sourcestamps exist in the
build a codebase must be passed as argument.

 

I think, the option 'codebase is set to its repository' is the most
intuitive solutions so installations that do not have multiple locations for
their repositories do not need to worry about codebases, they can just use
the repository as codebase. If schedulers are configured to handle multiple
repositories then changes are at least grouped under their repository, in
contrast with the first option where changes get mixed.

Of course both options can be implemented by defining the callable that
returns '' or change['revision'] but my question is about the default
behaviour if the configfile has no callable to determine the codebase.

 

If you have a idea for your situation please respond.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20120210/8c4d007d/attachment.html>