[Buildbot-devel] ShellCommand question

[Brian Warner]

Duncan Ferguson <duncan.ferguson at altinity.com> writes:

>    Hiya list,
>
>    I have a step that attempts to send a number of files to a remove
>    server using scp, such as
>
>    factoryd.addStep(
>
>            shell.ShellCommand,
>
>
>    command=["scp","-q","built_files.*.tar.gz","server3:/build_files/"],
>
>    )
>
>    but the "build_files.*.tar.gz" is not expanded through a shell glob.
>    What is the best way to achieve this?

You have a couple of options. As other folks suggested, you could put the
command in a script, and have the BuildStep run that script. My preference is
to put such commands in the Makefile that gets checked into the source tree
(perhaps with a server name as a variable), so that as much information as
possible stays with the source code instead of being hidden in the
buildmaster config. For example, here's a fragment from the Makefile in one
of my projects that is intended to upload generated code-coverage data (HTML
files that annotate which lines of code were exercised by unit tests and
which were not) to a server:

  # 'upload-figleaf' is meant to be run with an UPLOAD_TARGET=host:/dir setting
  ifdef UPLOAD_TARGET
  upload-figleaf:
          rsync -a coverage-html/ $(UPLOAD_TARGET)
  else
  upload-figleaf:
          echo "this target is meant to be run with UPLOAD_TARGET=host:/path/"
          /bin/false
  endif

And then the buildmaster uses something like:

 f.addStep(ShellCommand(command="make upload-figleaf UPLOAD_TARGET=server:/foo/"))

to do the upload. That way everything but the actual hostname is visible to a
developer looking at the source code and its Makefile.


Another option is to set your command= to be a single string
(space-separated) instead of a list of argv values, something like:

 command = "scp -q built_files.*.tar.gz server3:/build_files/"

That triggers the buildslave into running your command with an argv of
["/bin/sh", "-c", command], which should make the shell do an expansion pass
over your command and expand the glob. You could accomplish the same thing by
adding the /bin/sh yourself:

 command = ["/bin/sh", "-c", "scp -q built_files.*.tar.gz server3:/build_files/"]

Of course, using /bin/sh -c means that spaces in your command have special
significance, so files or directories with spaces in their names will cause
problems.

hope that helps,
 -Brian