[Buildbot-devel] fyi: use the same ShellCommand steps for Linux and Cygwin buildslaves

[Tony Wallace]

This is just to share a technique that I thought turned out rather well. Hope it 
helps. 


This method allowed me to use the same complex buildsteps (ShellCommands) I 
originally developed for Linux buildslaves, on Windows buildslaves also- without 
having to change much, and without maintaining libraries of custom shell scripts 
or bat files on the buildslaves.  It depends on Cygwin on Windows, but you can 
still run something (a make tool, for example) in a "pure" Windows environment 
when it needs to.  

Example:

    bf.addStep(ShellCommand(
        name='build_' + variant,
        env=envd,
        command=[bash, '-ex', '-c', r"""
umask 022
:
: build "$variant"
:
pwd
ls -la
if test "$target_os" = windows
then
    # build on windows
    cmd /C set 'PATH=%PATH_SYSTEM%' '&' scons -k "OS=$target_os" 
"CPU=$target_cpu" "VARIANT=$variant" "MSVC_VERSION=$msvc_version" 

elif test "$target_os" = android
then
    # build on linux, for android
    scons -k OS="$target_os" CPU="$target_cpu" VARIANT="$variant" 
ANDROID_NDK="$android_ndk" ANDROID_SRC="$android_src"
else
    # build on linux, for linux
    scons -k OS="$target_os" CPU="$target_cpu" VARIANT="$variant"
fi
"""]))


The bash Python variable is the shell program for the ShellCommand build step.  
The envd Python variable is a dictionary that defines several environment 
variables for the ShellCommand build step.  These variables are mostly build 
parameters, like $target_os, $target_cpu, $variant, and so on.

For builds to run on Windows, 
    bash = r'C:\cygwin-1.7\bin\bash.exe' 
and 
    envd['PATH'] = 
r'/usr/local/bin:/usr/bin:/bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS'

For builds to run on Linux,
    bash = 'bash'  
and
    envd[ 'PATH' ] is not defined.

Specifying the full Windows path to bash.exe (including the ".exe"), allows 
ShellCommand to bypass cmd.exe and just run bash.exe directly. If your 
buildslave runs ShellCommand through cmd.exe, you cannot pass a multiple-line 
command string. 


Those multiple-line command strings can be much, much longer than the above 
example.  You can also wrap them in WithProperties(). 

When something on the Windows buildslave needs to run in a "pure" Windows 
environment (no Cygwin influence), I can still run it from Cygwin- for example, 
see the commandline starting with "cmd /C" in the above.  To restore the 
original Windows PATH, the Windows buildslave must be pre-configured 
with an environment variable like PATH_SYSTEM, which you set equal to the 

original Windows PATH on that box- or whatever PATH you need for that part of 
the build process.  


It turned out that only a few individual commands ever need the above "cmd /C" 
treatment on Windows.  


Finally, as of buildbot 0.8.2 you must hack one python file in the Windows 
buildslave code, as shown below.  Without this change, bash cannot get a 
Linux-style pwd- it gets a Windows path instead, and your scripts break down.  
See buildbot bug #456 for more info. 


Left file: runprocess.py.prev
Right file: runprocess.py
383,384c383,384
<         if not self.environ.get('MACHTYPE', None) == 'i686-pc-msys':
<             self.environ['PWD'] = os.path.abspath(self.workdir)
---
> ###     if not self.environ.get('MACHTYPE', None) == 'i686-pc-msys':
> ###         self.environ['PWD'] = os.path.abspath(self.workdir)