A code coverage analogy

One of the tasks I have on my plate is code coverage tracking.  I've written about this in the past and today I want to go over an analogy on the usefulness of this statistic.

First I can point out it is relatively easy to track.  We have plenty of tools to do this for us, get reports, etc…  But just because it is easy to track does not mean it is all that useful.  I can track the contents of my glove compartment each day, but I think we can all agree that this would not be a useful piece of data to have.  There is not much I would do differently on a day to day basis based on the contents of the glove compartment.

Code coverage, on the other hand, can be useful.  It tells me how much code I have to validate that is not covered by automated tests - it tells me a risk level.  By itself, it is not enough (look up Mars ClimateExplorer as a terrific case study).  It does help make a decision, though. 

Here's my analogy:  code coverage is to a test engineer as a patient's temperature is to a doctor.  It is one useful metric that can help isolate risk.  By itself, a 98.6F temperature does not tell a doctor that a patient is healthy.  It only can help eliminate some possible causes of illness - certain stages of the flu, for instance.  No doctor could reasonably base a diagnosis on one temperature measurement.  It is one statistic among many others such as blood pressure, heart rate, etc.. that can help lead to a more accurate diagnosis.

Likewise, just because I have a 98.6% rate of code coverage, I cannot pronounce a feature risk free and ready to ship.  I also have to look at scenario pass rates, international/localization status, accessibility, performance goals and so on. 

Too often we get trapped in a mindset of "this statistics by itself is useless so let's not bother tracking it."  While the first half of that sentence may be true (and is, for code coverage), the second half of that sentence does not follow.  If given a choice of having a patient's temperature included in a checkup or not having the temperature included, doctors will always want to track that statistic.  It helps give part of a bigger picture.

And that is exactly what code coverage does.

Questions, comments, concerns and criticisms always welcome,

John