A basic guide to unit testing, or how not to write unit tests

Inside the bunch of implicit responsibilities I have in some projects, including the Chamilo project, one is to review the code of my fellow colleagues (or should I say team members). The fact that we asked people with little experience in PHP programming to start writing automated tests was something of an experience, and it certainly proved... interesting. Most of these tests are simple. In fact they are down right basic. Most of these are also useful, even if basic, as they check some of the things that should never be broken (like the type of value a function returns). I have had the first opportunity recently to check those tests, mostly concentrating on failing tests (which, by the way, are totally acceptable in non-previously tested code, as we will see below), and I must say I've had my share of surprises, even pulling tears out of my eyes on occasions. In short, some of these tests (around 2000 in total) really show why people writing unit tests *shoul be* experienced developers to start with. First things first, let's get a few basic things right

What is unit testing used for?

The first thing that might come to mind is this question. In fact, most people will just consider at first that testing would be much easier to be done by hand. Even initiated to the idea that automating tests will generate time savings, many developers being, by nature, lazy, will seriously reconsider it when faced when the additional task of learning how to setup automated tests, and then setting it up slowly. Let's not discuss test driven development here. This will come after developers already understand testing well enough. But the fundamental interest of unit testing comes from regular control, when integrating it into continuous integration systems, for example. This ensures tests do not just tell you whether your current code runs right now, but rather if the latest version of your code broke something previously.

A few tips on how to write tests

Let's not consider test suites, mock-ups, set up and tear down for now. What you need to know if that every test should be as independent of other tests as possible. As such, just testing a function should mean writing one *or more* test functions. For example, one to test the type of return value, and one to test the value returned for each combination of parameters. For example, trying to pass parameters should be done by testing a few extreme values (like sending null, string, negative, positive and zero values for integer types). Let's say you have a simple function that computes a special value from one integer. For the examples below, we are using the SimpleTest unit testing framework for PHP.

function compute_value($i) {
  return $i*$i+($i*2);
}

A simple set of tests would be:

function test_compute_value_zero_when_param_zero() {
  $res = compute_value(0);
  $this->assertIdentical($res,0);
}

function test_compute_value_false_when_param_string() {
  $res = compute_value('abc');
  $this->assertIdentical($res,false);
}
function test_compute_value_false_when_param_null() {
  $res = compute_value(null);
  $this->assertFalse($res);
}

function test_compute_value_int_when_param_int() {
  $res = compute_value(1);
  $this->assertTrue(is_int($res));
}

Let's analyse these:

• we defined several test functions, with distinct names for each one, covering only one function. In general, each parameter and combination thereof should be declined in all its possible values to make sure your function is rock-solid. Due to the impossibility to do that (there is possibly an infinity of possible values for each parameter), you use representative values that will help you sort out if your function reacts well to certain ranges.
• we use asserts functions to check the return value (this comes with SimpleTest, you can find more about SimpleTest on LastCraft's website)
• some of the tests will probably fail (for example the null and string stuff), but that's alright. In fact, it is very good to have tests fail. This shows that while testing, you thought of stupid things that you never thought could happen to your code, and now that you wrote tests about it, you will have to fix your code or else the tests won't work

How to run the tests (using SimpleTest)

Now, very quickly because this isn't the main idea of the article, if you want to run these tests, you'll have to take the following steps. I'm assuming a series of things first:

1. you downloaded the SimpleTest framework and put it into a 'tests/simpletest/' directory inside your project
2. your function is inside a file called functions.lib.php under the directory 'code/'
3. your test code is inside a file called functions.lib.test.php under the directory 'tests/'

Take the following steps:

1. inside your functions.lib.test.php file, wrap your functions into a class that extends UnitTestCase (e.g.: class TestFunctions extends UnitTestCase { /*your functions here*/ })
2. create a new file called "test.suite.php" inside the 'tests/' directory, and edit it
3. require the SimpleTest libraries (something like require_once 'simpletest/unit_tester.php'; require_once 'simpletest/autorun.php';)
4. require the 'code/functions.lib.php' file (something like require_once '../code/functions.lib.php';)
5. declare a class, extension of the TestSuite class (something like:  class MySuite extends TestSuite {})
6. inside the class, declare a constructor (function __construct() { })
7. inside the constructor, load the test library ($this->addFile('functions.lib.test.php');)
8. outside the class, instantiate your test suite: $test = &new MySuite();

Now get out of all these files and load the test.suite.php script (either from your browser or from the command line)

What NOT to do

Now this is a short series of examples that you should try to avoid "commiting" inside any of your tests

if ($var == '') {
  $this->assertTrue(empty($var));
}

Obviously, given empty will return true when its parameter is an empty string, this assert will verify that the value is true... The thing is to never put a condition before an assert, that could alter the results. You are supposed to call your tested function in such a way that you know whether it should return an empty string or something else. If you are testing whether it returned an empty string, this means that you don't understand what your function should return...

ob_start();
$res = compute_value(3);
$this->assertTrue(is_int($res));
ob_end_clean();

ob_start() and ob_end_clean() are used when you want to test whether a function returns the right value or not, but you don't want your test to be showing all the strings that the function is outputting, so you hide it inside the output buffer and then you destroy it (you don't care what it is, you just want the return value). Now, if you put the $this->assertTrue() call inside these two "markers", the output of the $this->assertTrue() call will not appear on your screen, so you will know (from the global report) that there was an error, but you won't know where it happened. Note that displaying directly to the standard output from inside a function is a questionable technique in many cases.

// when $res is the result of a function returning
// either true or false, and we called it with
// a value that should return true...
$this->assertTrue(is_bool($res));

If you already expect your result to be true, test if it is true, not if it is a boolean!

if ($perm_dir != '0777') {
 $msg = "Error";
 $this->assertTrue(is_string($msg));
} else {
 //...
}

More obvious even... what you should check is the return value of the function checked. Assigning a constant value to a string and then testing whether it is a string is just... too useless to avoid laughing.

Conclusion

Although manual testing might be efficiently done by patient and observing people without development experience, unit testing is a whole different experience. You will probably need a full day training in order for your developers to understand how to do it. Don't expect them to learn by doing it alone. Testing is a very complex process. Don't assume testing is easier than writing the code. In fact, don't assume anything. Try it, test it and see what's happening.