[Buildbot-commits] buildbot/docs slave.xhtml,NONE,1.1

Mon Dec 6 03:09:27 UTC 2004

Update of /cvsroot/buildbot/buildbot/docs
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv22115/docs

Added Files:
	slave.xhtml 
Log Message:
provide a buildslave setup checklist

--- NEW FILE: slave.xhtml ---
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html 
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>Buildslave Configuration</title>
<link href="stylesheet-unprocessed.css" type="text/css" rel="stylesheet" />
</head>

<body>
<h1>Buildslave Configuration</h1>

<p>This document describes the basics of setting up a buildslave.</p>

Typically, you will be adding a buildslave to an existing buildmaster, to
provide additional architecture coverage. The buildbot administrator will
give you several pieces of information necessary to connect to the
buildmaster. You should also be somewhat familiar with the project being
tested, so you can troubleshoot build problems locally.

The buildbot exists to make sure that the project's stated <q>how to build
it</q> process actually works. To this end, the buildslave should run in an
environment just like that of your regular developers. Typically the project
build process is documented somewhere (<q>README</q>, <q>INSTALL</q>, etc),
in a document that should mention all library dependencies and contain a
basic set of build instructions. This document will be useful as you
configure the host and account in which the buildslave runs.

<h2>Checklist</h2>

<p>You will need the following pieces before starting your buildslave for the
first time:</p>

<ol>
 <li>Set up the account: It is recommended (although not mandatory)
 to set up a separate user account for the buildslave. This account is
 frequently named 'buildbot'. This serves to isolate your personal working
 environment from that of the slave's, and helps to minimize the security
 threat posed by letting possibly-unknown contributors run arbitrary code on
 your system. The account should have a minimum of fancy init scripts.</li>

<li>Set up the host: Make sure the host can actually reach the
 buildmaster. Usually the buildmaster is running a status webserver on the
 same machine, so simply point your web browser at it and see if you can get
 there. Install whatever additional packages or libraries the project's
 INSTALL document advises. (or not, if your buildslave is supposed to make
 sure that building without optional libraries still works, then don't
 install those libraries).</li>

<li>Test the build process: Follow the instructions in the INSTALL
 document, in the buildbot account. Perform a full CVS (or whatever)
 checkout, configure, make, run tests, etc. Confirm that the build works
 without manual fussing. If it doesn't work when you do it by hand, it will
 be unlikely to work when the buildbot attempts to do it in an automated
 fashion.</li>

<li>Choose a base directory: This should be somewhere in the
 buildbot account, typically named after the project which is being tested.
 The buildslave will not touch any file outside of this directory.</li>

<li>Create the hostinfo files: When it first connects, the
 buildslave will send a few files up to the buildmaster which describe the
 host that it is running on. These files are presented on the web status
 display so that developers have more information to reproduce any test
 failures that are witnessed by the buildbot. Create a directory named
 'info' (directly under the buildbot home directory would be a good place).
 Inside it, make a file named 'admin', and put your name/email in it. This
 is the <q>buildslave admin address</q>, and will be reachable from the
 build status page (so you may wish to munge it a bit if address-harvesting
 spambots are a concern). Create a second file named 'host' and fill it with
 a brief description of the host: OS, version, memory size, CPU speed,
 versions of relevant libraries installed, and finally the version of the
 buildbot code which is running the buildslave. You will create a symlink to
 this directory when you finally set up the buildslave.</li>

<li>Get the buildmaster host/port, botname, and password: The
 buildmaster admin will give you a hostname:portno pair which specifies the
 TCP port on which the buildmaster is expecting connections from the
 buildslaves. They will also assign you a name and password which your bot
 will use.</li>

<li>Install the buildbot code: Obtain the latest tarball from <a
 href="http://buildbot.sf.net/">buildbot.sf.net</a>, verify the signature,
 unpack it, then do the usual <code>python ./setup.py build; sudo python
 ./setup.py install</code> dance. If you do not have root on this host, you
 can install it into a different directory as long as you remember to add it
 to the PYTHONPATH environment variable at the right time. <code>python
 ./setup.py install --home=~</code> and then
 <code>PYTHONPATH=~/lib/python</code> is a common technique.</li>
</ol>

<h2>Configuring the buildslave</h2>

With all that setup ready, you are ready to create the buildslave. There
is a tool provided in the buildbot package named (simply enough)
<code>buildbot</code>, which usually gets installed to
<code>/usr/bin/buildbot</code>. This tool provides a front-end to a Twisted
program named <code>mktap</code>. When you use <code>buildbot</code> to
create the buildslave, it will create the base directory for you (and
complain if it already exists). Take the BASEDIR you've picked, the HOST:PORT
buildmaster location, and the BOTNAME and PASSWORD you've been assigned, and
run <code>buildbot</code> as follows:

<pre>
buildbot slave BASEDIR HOST:PORT BOTNAME PASSWORD
</pre>

That will create and populate BASEDIR with some setup files. The
<code>buildbot.tap</code> file contains a <q>freeze-dried</q> buildslave
object, ready to be run by Twisted's daemon-launching utility
<code>twistd</code> (pronounced twist-dee). That's it. In the future,
<code>buildbot slave</code> will probably do more setup.

<p>Now symlink your hostinfo directory into place:</p>

<pre>
cd BASEDIR
ln -s ~/hostinfo ./info
</pre>

<p>Your buildslave is now ready to run!</p>

<h2>Starting the buildslave</h2>

<p>To start the buildslave manually, just use the <code>buildbot</code> tool
again:</p>

<pre>
buildbot start BASEDIR
</pre>

This will start any <q>freeze-dried</q> application found in the given
directory. (The same command is used to start a buildmaster instance). Note
that <code>buildbot start</code> is really just a front end for
<code>twistd</code>.

As soon as the buildslave starts, you should find two new files in its
base directory. The first is named <code>twistd.pid</code>, and simply
contains the process ID of the buildslave's <code>twistd</code> process. You
can use <code>ps</code> to find it in your process table. The second is
<code>twistd.log</code>, and is the buildslave's log file. Everything the
buildslave does is recorded in this file. It is the first place to look for
error messages or exception traces.

Once the buildslave connects to the buildmaster, new directories will
start appearing in the base directory. The buildmaster tells the slave to
create a directory for each <q>builder</q> which will be using that slave.
Within these directories, CVS checkouts, compiles, and tests are
performed.

<h2>Making sure the buildslave starts at each reboot</h2>

Before you are done, you need to make sure the buildslave will keep
running even if the host reboots. The easiest way I'm found to do this is to
add a <code>@reboot</code> crontab entry. Most modern versions of cron
interpret a time specification of <code>@reboot</code> to indicate that the
given job should be run the first time cron is started after system boot.
Something like the following usually works:

<pre>
@reboot buildbot start BASEDIR
</pre>

It is important to remember that the environment provided to cron jobs can
be quite different that your normal runtime. There may be fewer environment
variables specified, and the PATH may be shorter than usual. It is a good
idea to test out this method of launching the buildslave by using a time in
the near future, with the same command, and then check twistd.log to make
sure the slave actually started correctly.

<h2>Shutting down the buildslave</h2>

<p>To stop the buildslave manually, use the <code>buildbot</code> tool
again:</p>

<pre>
buildbot stop BASEDIR
</pre>

<p>This simply looks for the <code>twistd.pid</code> file and kills whatever
process is identified within.</p>

At system shutdown, all processes are sent a SIGKILL. The buildslave will
respond to this by shutting down normally.

<p>Troubleshooting</p>

Cron jobs are typically run with a minimal shell (/bin/sh, not /bin/bash),
and tilde expansion is not always performed in such commands. You may want to
use explicit paths, because the PATH is usually quite short and doesn't
include anything set by your shell's startup scripts (.profile, .bashrc,
etc). If you've installed buildbot (or other python libraries) to an unusual
location, you may need to add a PYTHONPATH specification (note that python
will do tilde-expansion on PYTHONPATH elements by itself).

<pre>
@reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir
</pre>

Take the time to get the @reboot job set up. Otherwise, things will work
fine for a while, but the first power outage or system reboot you have will
stop the buildslave with nothing but the cries of sorrowful developers to
remind you that it has gone away.

From the buildmaster's main status web page, you can force a build to be
run on your build slave. Figure out which column is for a builder that runs
on your slave, click on that builder's name, and the page that comes up will
have a <q>Force Build</q> button. Fill in the form, hit the button, and a
moment later you should see your slave's <code>twistd.log</code> filling with
commands being run. Using <code>pstree</code> or <code>top</code> should also
reveal the cvs/make/gcc/etc processes being run by the buildslave. Note that
the same web page should also show the <q>admin</q> and <q>host</q>
information files that you configured earlier.

</body> </html>