Friday, May 16, 2014

How Many Tests Should You Have in Your Python Program?

Overview:
Tests are one of the most crucial things in programming, but very few programmers write enough tests. One of the beginning obstacles to writing tests is how to write proper, fast tests.

Idea behind tests:
Tests should be extremely fast and should cover as much of your program as possible (100% is always preferred). Rule of thumb is if you have an if statement, you should have at least 2 tests.

Reason for Having 100% test coverge:
The more code you write, the higher chance of things to go wrong. If you write 10 lines of code, that's at least 10 spots where you can have an unexpected error in your code.

I think it would be a little excessive to have a test for each line of code in your program, but it's reasonable to have at least 2 tests for each if statement in your program. Why this number? Because the complexity of your program comes mainly from conditional logic (Or if statements). Take these lines of code:

if input == 'cat':
  a = 'meow'
else:
  a = 'woof'

Here, our program has 2 things that could happen: `a` can either be 'meow' or 'woof'. So we'd have 2 tests here: a test with input == 'cat', and a test where input != 'cat'. By having these 2 tests, we have 100% code coverage. By having this, it saves us from 2 potential fatal flaws in our program.

Lets look at a slightly more difficult example:

if input == 'cat':
  a = 'meow'
elif input == 'dog':
  a = 'woof'
else:
  a = "Hello kind sir"

Here, our program has 3 things that could happen: `a` can either be 'meow', 'woof', or 'Hello kind sir'. So, instead of 2 tests, we'd have 3 tests: a test with input == 'cat', a test with input == 'dog' and a test that doesn't equal 'cat' or 'dog'.

So lets go into a slightly harder example with if statements inside if statements:

if input == 'cat':
  if input2 == 'siemese':
    a = 'mawww'
  else:
    a = 'meow'
elif input == 'cat2':
  if input2 == 'siemese':
    a = 'mawww'
  else:
    a = 'meow'
else:
  a = "Hello kind sir"

Now the calculation of complexity is slightly harder. We need to take into account the if statement inside the first if statement, so the calculation will be about 7, because we have 5 possible different things that could be set, but also have to be sure the outer if statements (`if input == 'cat'` and `if input == 'cat2'`) are true ever. The important thing to note is that we have written the inner if statement logic twice, so we can simplify this to this:

def output_on_cat_type(cat_type):
  if input2 == 'siemese':
    return 'mawww'
  return 'meow'

if input == 'cat':
  a = output_on_cat_type(input2)
elif input == 'cat2':
  a = output_on_cat_type(input2)
else:
  a = "Hello kind sir"

Our new calculation is now 5 because we need to test the 3 if statements in the regular code, and test the 2 conditions in the `output_on_cat_type` function. This is one of the reasons why it's so important to refactor out if statements within if statements: It gets increasingly more difficult to have 100% test coverage with every if statement.