[Buildbot-devel] Discussion on source steps

[A.T.Hofkamp]

On 05/16/2011 07:28 PM, Dustin J. Mitchell wrote:
> So I think this gets to the crux of the matter.  There *are* two
> choices to make, although if I may I'd like to adjust their
> presentation slightly.

Sure, I just typed what I could think of, at the spot.


> The first choice, I think should be binary and should be the same for
> every VC.  The second choice should specify behaviors, and thus be
> somewhat VC-specific.
>
> On Mon, May 16, 2011 at 4:30 AM, A.T.Hofkamp<a.t.hofkamp at tue.nl>  wrote:
>> Maybe it is my lack of understanding your other modes, but in my view there are 2 choices to make.
>> The main one is:
>>
>> 1. Throw away everything each time, or
>> 2. Keep existing directory.
>
> How about
>   1. Build from a pristine directory with no leftovers from previous builds
>   2. Build from an already-used directory with some/all files left intact

I agree this is much better, as it says *what* you want rather than how to do it.


> Let's refer to 1 as "clean" and 2 as "incremental" (although every
> project seems to have its own names - Mozilla uses "clobber" and
> "objdir")

The Mozilla names mean absolutely nothing to me, but maybe that's because English is not my native 
language. Otherwise, I have no real preferences, as long as it is documented in the manual.

>> In 2, you then have a second choice, namely
>>
>> 2a. Keep all files
>> 2c. Throw away *all* files not in the current revision of the VCS, and revert any changed source.
>> (yep, 2b is missing, but it is on purpose, see below.)
>
> Actually, in both cases you have a number of choices in how to
> implement this behavior; those choices differ from VCS to VCS.  For
> example:
>
> 1a. Delete the directory and re-checkout (or re-clone, or whatever)
> 1b. Keep a pristine 'source' directory, delete the build directory,
> copy from source to build, and update
> 1c. Use '$vcs clean' with appropriately harsh (delete it all!) options
>
> These may differ in efficiency in terms of filesystem access speed,
> disk space, and network bandwidth, and as such should be
> user-selectable, but that selection is basically an optimization and
> not part of the high-level clean vs. incremental choice.  Note that by
> this analysis, your 2a falls under "clean", not "incremental"

If you mean 2c rather than 2a, I agree.

My 2a is "2a. Keep all files" which contradicts with your 'clean' idea: "1. Build from a pristine 
directory with no leftovers from previous builds", as far as I can see.

> For incremental, I'm sure there are subtle per-vc behaviors that may
> need to be adjusted.  For example,

It may be useful to aim for minimal vcs-specific behavior here. That is, like the first choice, 
express *what* you want (eg "remove unmanaged, non-ignored files") rather than how exactly.

For different vcs-es, the implementation may be different (eg with one vcs you can do this in one 
command, while with another vcs you need to ask unknown files, and remove them manually or so).

 From a user point of view, understanding the source-step becomes much easier, especially when he 
has to make a buildbot install for a vcs that he doesn't know.


> 2b. Remove non-ignored files (as Amber suggested is sometimes desirable)
> 2d. if the branch has changed, do $some_other_behavior (e.g., for
> broken svn switches)

2b is a valid use-case, and would be a good addition.


2d I am less sure about. I can also imagine this to be the the first decision to make.

In other words, for a source step, there are two cases:

A. Build another revision in the same branch, and
B. Build a revision in another branch.

For both cases, I need to specify a choice 1/2, and a sub-choice a..x .

In other words, I would need to specify:
- for 'A', I want 2c, and
- for 'B' I want 1a.

However, maybe I am making things too complicated here (at this time)?

We'll need an implementation of all 1a ... 2? choices for all vcs-es, independent whether or not 
switching a branch is relevant for the decision.
For the project I can imagine that getting the choices implemented is more interesting than 
pondering about what questions exist exactly to decide the choice.

>   others?

The only other thing I can imagine is a "make mrproper"-support, or in other words

2e. run a user-specifed command to clean

You could also consider this to be a choice 3, since an arbitrary user-command can do anything, 
including all 1/2 choices discussed above.


Greetings,
Albert