Avoiding orphaned processes in CI builds – TeamCity

Reason

As outlined in my previous post, it’s simpler than ever to run integration test using a WebListener. Once you’ve got your tests running locally, the logical next step is to automate the task of running them using a build server.

One caveat I’ve run into on several occasions when spawning new processes on a build server is that you’ll end up with “orphaned processes” sooner or later. This can be for a variety of reasons – builds being aborted mid run, network outages, out of memory exceptions or similar force majeure.

Orphaned processes are quite a nuisance as they tend to cause unexpected build and test failures, not to mention hogging system resources.

Although it’s infeasible to solve all causes of orphaned processes, the following simple pattern can help avoid some of the more common causes, i.e. manually aborted and failed builds. I’m using TeamCity as the build server, but the concept should be applicable to most CI services.

  1. When spawning a new process in a build step, store the process ID in a build parameter.
  2. Do stuff (e.g. run tests).
  3. Stop the spawned process in a later step, using the build parameter value.

Step 1 and 3 mainly rely on TeamCity’s ability to set build parameters from within build scripts. This is done by emitting log messages that follow a prescribed format, as described in “Adding or Changing a Build Parameter“.

Note: The following article assumes the steps spawning new processes are already in place. An example of starting new processes using PowerShell can be found here.

Code

Start by defining a build parameter in TeamCity and enter anything as it’s value.

Define a build parameter.

Define a build parameter.

Now set up a PowerShell build step which is run even if the build was stopped manually or if previous build steps failed. The Stop-Process commandlet is used to kill the process of choice.

Create PowerShell build step.

Create PowerShell build step.

The following snippet shows an example of a specially formatted log message which is picked up by TeamCity, taken from a Gulp build file. TeamCity will update our build parameter based on the provided value (e.g. in this case the value of the variable testServerProcessId). Make sure that the “name” attribute value matches the name of the build parameter in TeamCity.

console.log("##teamcity[setParameter name='TestServerProcessID' value='" + testServerProcessId + "']");

Example

Once a build has run, the value picked up during the build can be found on the “Parameters” tab of the build details, in the section “Actual Parameters on Agent”.

Actual parameters on agent.

Actual parameters on agent.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s