[This was originally planned to be a presentation at CSSS BarCamp in February '09.  I got sick, though, and wussed out on presenting.  I figured it'd be a waste to have done this research without sharing.]

What is testing?

“Testing” is such a nebulous term, and comes in many (blurry) flavours: functional testing, acceptance testing, unit testing, regression testing, etc.  I think most software developers at least think about testing, but there seems to be a huge variation in the level of formalism involved.

There’s a single common trend across all of the flavours of testing though:

Testing is running the software with the intent of finding bugs.

If your tests aren’t finding bugs (or preventing bugs from slipping into your software), then you’re probably not getting much for the time you’re spending.

Automated Tests

In my previous post, I talked about the automated processes we use at VendAsta to develop software.  Arguably, the most important automated process is running the suite of tests we’ve built up.

Having a quick, automated set of tests (unit tests, probably, although functional tests work too) lets you develop quicker, with fewer defects.

Here’s a few of the immediate, tangible benefits:

  • Confidence that fixed bugs remain fixed (no regressions).  If a test is added for each defect, then fixed defects stay fixed.
  • Confidence that we can change (refactor) code without breaking things.  By running the test suite before and after refactoring, we can be assured that behaviour hasn’t changed.
  • If something does break, you know exactly what is was (because its test breaks).
  • Confidence that when you commit to version control, you’re not committing code that is broken (which would trip up the whole team)

How Shouldn’t I Write Tests?

Before explaining my testing process, I want to point out a few ways to write bad tests:

  • Test that your code is implemented the way it is implemented (why would you do that?).  This happens quite often when people are new to writing unit tests.  I’ll explain more in a bit.
  • Test that every function in the method-under-test has been called. This is bad because it makes it difficult to refactor the method-under-test without rewriting all of its unit tests.
  • Test a bunch of things in a single test.  If the test fails, finding out why turns into an adventure.
  • Write tests that involve retrieving things over the network or Internet. (Network connections go down — this shouldn’t stop you from writing code)
  • … many, many more.  I wouldn’t want to try to enumerate all of the bad ways to write tests :)

How Do I Design Tests?

Testing can basically be boiled down to three simple steps (that are notoriously difficult to achieve):

  1. Determine the initial state required to test what you want to test
  2. Run the code you want to test
  3. Determine whether the final state matches what you expect.

And voila!  You’ve got a good test!

“Wait a minute!  What does that mean?”

High-quality testing is all about isolation.  In the “bad tests” section, I briefly touched on this when I talked about network access. Most languages have a “mocking” framework of some sort that allows you to fake dependencies on external systems (e.g. database access, network access, etc).  I may write more about this later, but if you want more info, look up the Python “mocker” framework.

So.  Isolation.  The easiest way to achieve this is to write small, self-contained functions that have well-defined inputs and outputs — these are easy to test.

Often, this isn’t the easiest thing to achieve.  The Python unittest framework provides the setUp() method, which lets you share initialization code between a suite of tests.  Also, because of Python’s liberal data hiding rules, you can cheat when you need to and set up objects by writing right into their attributes.

Honestly, I wish I had a better explanation for how to write easily-testable code, but it’s really tricky.  As you write more, you start to get a feel for the rough spots, and figure out ways around them.

Maybe an example will help. (Coming next time)

Part 2 and Part 3