[Buildbot-devel] Discussion on source steps

[A.T.Hofkamp]

On 05/16/2011 10:03 AM, Pradeepkumar Gayam wrote:
> Hi,
>
> I am Pradeep working on GSoC project to improve source steps. I am listing all
> reasonable source step modes here. I am not familiar with users requirement,

Hello Pradeep,

> As we are cleaning up the modes I think we should make it possible to take
> advantage of features of the version control systems along with the common
> modes.  There were discussions on IRC and bug tracker[1] on required modes.
>
> Proposed modes:
> 1. Incremental
> 2. Clean
> 3. Fresh
>
> Incremental mode: This is same as current update. Update the source keeping
> old build artifacts around and pull the changes then build.
>
> Clean: As written in bug#669[1] this is equivalent to clobber with
> ignore_ignores=False, i.e., remove all files generated in build process that
> are not under version control. Don't remove ignored files. Equivalent to
> `git clean`
>
> Fresh: This is equivalent to clobber with ignore_ignores=True. That is
> remove all the build artifact and ignored files. Equivalent to `git clean
> -x`. This will create a pristine working directory to make fresh builds.

I don't understand 'Clean' and 'Fresh'. The current 'clobber' simply removes the directory, thus 
*everything* is gone. I don't see how you can switch between keeping or removing generated files 
after removing the whole working directory.
(Maybe 'git clean -x' explains it, but I am not a git user, so the command means nothing to me, sorry.)

> I guess this is pretty much all any user want to do. These three modes can
> be implemented across any VCS. Right now ignore_ignores is implemented only

Maybe it is my lack of understanding your other modes, but in my view there are 2 choices to make. 
The main one is:

1. Throw away everything each time, or
2. Keep existing directory.

In 2, you then have a second choice, namely

2a. Keep all files
2c. Throw away *all* files not in the current revision of the VCS, and revert any changed source.
(yep, 2b is missing, but it is on purpose, see below.)

Comments:
In 1, you need to do a full checkout every time. Very safe, but may be costly in bandwidth.
Option 2 tries to reduce bandwidth needed for updating to another revision or branch by using the 
intelligence of the VCS.
Option 2a is the simplest as it assumes the VCS can handle unmanaged files, and the build system 
will not get confused by files which are not part of the current version. (See also [1].)
Option 2c reconstructs the situation after a checkout, before using the VCS to update the revision 
or branch.


My ideas are slightly different from your proposal, you don't have 2a, and you have a option 
in-between them (let's call it option 2b), where you keep the ignored files only.

I don't understand why you have this mode.
Do you have a use case in mind when this mode is useful?

In my view, ignored files are the same as files that are not part of the VCS checkout.
The only difference is that they exist so often that output of eg 'svn status' gets unreadable, so 
the VCS authors added a feature to suppress that output.



I am very happy you will create 2c (if that is what you means by 'fresh'), I wanted that for a long 
time!

Greetings,
Albert


[1]: To understand the possible problems:
Think of a project that has current revision X. A file gets added, and you get revision Y.
- If you build Y for the first time, the file gets added in the working directory.
- If I then build revision X again, the file stays. The VCS will silently drop the new file from
   its administration, but not remove it (I am thinking worst case scenario here). The project must
   be smart enough to consider the new file as part of the project while building X.
- Then I build revision Y again. The VCS needs to create the new file again, but ho, a file with
   the given name already exists!