[Buildbot-devel] issues with buildbot

[Chris Leishman]

Thanks for the great response Brian!  A few more comments inline...

On Tue, Aug 30, 2005 at 12:49:12PM -0700, Brian Warner wrote:
<snip>
> Apart from that, the only problem I see with a buildbot setup where you
> assume revisions are monotonically increasing is that you wouldn't be able to
> build the same source code twice, which is kind of a pity. The buildslaves
> are supposed to behave a lot like a human developer doing checkouts and
> compiles.. hopefully your human developers aren't limited to one compile per
> source tree too.

No - they're not.  I probably should describe the use case a bit more,
and you can tell me how what I'm doing is evil and wrong ;-)

Basically I want builds to be produced and made available for developers
and for QA, and also for eventual selection of a final release.  All
these builds come out of the build system, so there is full
accountability of the build.

My build steps do the following:

1. "Clobber & check out"

2. "Calculate next version".  Our code uses a versioning scheme of:

	major.minor-buildnumber

The major and minor represent the usual, and the build number is
incremented for every sucessfull build of the tree.  This version number
is stored in the source tree and included in the built product so that
it can be determined which exact build a system is running.

For builds done outside the build system, there is also a '-dev' postfix
added to the version included in the built product to indicate that it
did not come from the build system and, whilst it is based on similar
code to the official build with that version, may also contain
additional changes not under revision control.

So this step reads the current version file, calculates the next build
number, and rewrites the file.

3. "Build & test".  If this fails, the entire build is stopped at this
point.

4. "Commit the new version".  This does a subversion commit of the new
version file.

5. "Tag source". This does a subversion copy operation to create
a copy that exactly captures the code used in the build, and puts it
under "project/tags/builds/<version>".

6. "Copy to archive".  The build is copied to a build archive (which is
available via HTTP, ftp, etc) for use by all the different groups.
This includes the final product image and a changelog.  (It would be nice
to also include the build logs and the list of changes that caused the
build).

Most of these steps are implemented using ShellCommand objects to run
the various scripts required.

My problem, of course, stems from the fact that I modify external
resources (the repository and the build archive) as a side effect of the
build process.  Which causes two major issues: the system doesn't
support building 'out-of-order', as then it will be unable to modify and
commit the version file (since it's not the HEAD version), and it
cannot re-build the same version for the same reason and because the
build will already be tagged and exist in the archive.

<snip>
> Or, having just written all that, I think I may have misunderstood you. When
> you say "more than one directory", do you mean "foo/bar/baz" as opposed to
> just "foo" ? In that case, it's just a bug. If you could provide me with an
> example of a working and a non-working case, I'll write up a test case and
> fix it. Oops.

Haha.  You gave a very nice overview, but yeah - I meant a prefix like
"project/trunk".  I just did a look, and the exact issue is here:

changes/pb.py:

def perspective_addChange(self, changedict):
    ...
    if self.prefix:
        bits = path.split(self.sep)
        log.msg("bits[0]: "+bits[0]);
        if bits[0] == self.prefix:
            if bits[1:]:
                path = self.sep.join(bits[1:])
            else:
                path = ''
        else:
            break
    ....

So it's only checking the first part of the split path against the
prefix.  I can supply a patch if you want, but I'll have to go read up
on my python first.  I imaging you can either just do a string match to
see if the path begins with the prefix (before splitting) or split both
and iterate through the prefix bits, comparing each bit to the
equivalent path bit.

<snip>
> I think you can use s(step.ShellCommand, name="my description") to set it.
> Most of the useful attributes of a BuildStep subclass will be copied from the
> kwargs you pass to __init__ (see BuildStep.__init__ where it loops through
> the names listed in BuildStep.parms).

The name and the description seem to get used differently.  The one I
was interested in was the string that gets used in the waterfall display
to show what it's currently up to.  Having a step that read like
"install -d /opt/builds/blah && copy XYZ ..." wasn't as nice as "copying
to build archive" ;-)

> It's your lucky day! :). The next release will include the build-on-branch
> feature we've been talking about for the last few months. Take a look at the
> user's manual on the web site (the CVS HEAD one, not the 0.6.6 version) and
> see if the functionality described in there will meet your needs. If not, let
> me know, I need more use cases.

I'll go have a read, but so far it sounds great!

<snip>
> So, I'd recommend doing archiving in a StatusTarget, but if you really need
> access to the logs from the BuildStep or the Build (or even the Builder, but
> that would get kind of ugly), you can do it.

I'll have to have a bit of a think about how this all works before I can
comment.  But would the StatusTarget also have access to the files built
on the builder?

Thanks again for your great work!  I'll have to really start learning
python so I can be a bit more detailed in future.  I played a bit with
it a while ago, but wasn't impressed and just stuck to perl.  But
people keep telling me it's a lot better now.

I'll also do some more investigation on the buffering issue and get back
to you with more detail.

cheers,
chris