Integrating Test Swarm with Hudson

Over the past few days I've been working on getting our JavaScript test results into Hudson to give us a truly integrated testing setup. This is how I did it.

Test swarm

Test Swarm is a PHP application for managing the distributed testing of JavaScript. It enables you to run your unit tests on any number of browsers simultaneously and collate the results centrally.

Hudson

Hudson if you're not familiar with it is a continuous integration server for building and testing software projects.

QUnit

We're using QUnit as our test runner but Test Swarm supports several test runners out of the box and it's likely very easy to adapt yours to report results back to Test Swarm.

Why?

Until now we ran our JavaScript tests manually (and infrequently) across each browser we give A grade support for. Test swarm saves us this time and enables us to run our tests whenever we want with the click of a button.

This is great but sits outside our continuous integration environment which runs our server side tests whenever someone pushes code. Ideally we want someway to trigger Test Swarm in parallel to our server side tests.

How I've done it

Prerequisites

Firstly you'll need to install my fork of Test Swarm (http://github.com/carlmw/testswarm) which includes a couple of scripts to generate xUnit test reports and feed them into Hudson. Instructions can be found in the README.
Hudson, If you're using something other than Hudson I'm sure that this technique can be adapted for your environment (This should all be identical for Jenkins though I won't know for sure until we migrate to it) .
Hudson xUnit plugin.
cURL

Tying Test Swarm into Hudson

Firstly you'll need to make sure your tests are available to all your Test Swarm clients when the code for your build is checked out. Publishing the directory in your build workspace containing your tests through apache would likely be the easiest way to achieve this.

Add a job to Test Swarm that runs your tests against your desired browser set. Make a note of the jobs ID from the address bar.

Clearing current test state

In the configure page for your Hudson build, add/modify a shell script build step and before the command that runs your server side tests but after the one that checks out the code for your build — Add the following cURL command.


curl --data "job_id=%JOB_ID%&state=wipejob&type=reset" http://swarm-server

Replace %JOB_ID% with the id of your Test Swarm job and change http://swarm-server to point to your own Test Swarm server.

That cURL command is important and will reset the results of the job prior to running your tests.

Capturing test results

Capturing the results of your tests and feeding them back into Hudson is pretty easy.

Add the following command after your server side test runner. Doing it this way ensures that your server and front end tests can run in parallel.


php /path/to/testswarm/scripts/get_xunit_results.php http://swarm-server/job/%JOB_ID%/ /results/path/

Replace /path/to/testswarm/scripts/get_xunit_results.php with the correct path to that script on your Hudson server, also replace http://swarm-server/job/%JOB_ID%/ with the URL of the Test Swarm job you want to capture results for. Finally replace /results/xml/ with the path to the directory in which you want to write the xUnit test results (this directory will be emptied automatically before the results are written).

Screenshot of the build step in Hudson — Build step

Now that you have your test results you'll need to tell Hudson where to find them; add the path (E.g. /path/to/results/*.xml) to your Test Swarm xUnit test results to the xUnit config in your build.

Screenshot of the test result report config in Hudson — Test results config

If all has gone well you should be able to run your build and results from Test Swarm will be fed into your Hudson build.

Next up i'll be generating code coverage reports and feeding them into Hudson!

Comments

1 year ago evandegr said

Hello,

So I recently started at a new job as a front end web developer and I would like to set up a system that runs automated testing in Qunit and also integrate Test Swarm and Hudson if possible.

Here's the thing, we are running eclipse, subclipse svn, and tomcat 6 server system.

So, I would like to do a demonstration for my supervisor to show how good these tools are.

My thinking is like this: I set up the Hudson/Test Swarm environment you have envisioned here and then somehow use PDE in eclipse to get an Ant build for the very complicated and somewhat fragile eclipse project build what we have, and then use that to tie into the Hudson. Is this even possible? I'm worried about editing the whole build thing to include the automated testing suite because that is not a fully integrated part of our production yet.

Good article, I really want to get this working but I'm just not really sure if it is going to work.

Thanks

1 year ago Drew Wells said

This is interesting. I've gotten it all setup, however I'm lost on how this is automated. Hudson connects to my testswarm. It polls a few times saying waiting for browser, then it gives up after a couple minutes. I connect manually, but the count in Hudson doesn't go down. Connecting manually sort of defeats the purpose. So how am I supposed to use this?

1 year ago Gordon McMullan said

Hi Carl,

I was just looking up a GitHub repo I haven't touched in months and noticed you'd found a use for my synchronous job running additions to TestSwarm. I wrote them for just the purpose you have described here, integrating TestSwarm with Hudson, I'm really pleased someone found a use for my work.

I never got around to documenting it properly though, it was only a demonstration.

I don't know whether you noticed but there are in fact a couple of new scripts that I added:

http://swarm-server/?state=runjob which takes the same query string paramaters as the standard TestSwarm ?state=addjob command with one exception the output paramater can be either 'dump' or 'xml' if you select 'xml' then the document returned will be an xunit style xml doc. Which is generated by repeatedly calling the other script I added:

http://swarm-server/?state=xunit&run_id=&client_id=

This returns an XUnit style xml document for an individual testrun on a particular client.

I only wrote a parser for QUnit tests stored in Testswarm, you'd need to write your own for other test frameworks I suspect.