Getting 100% code coverage doesn't eliminate bugs

TL;DR - Getting 100% coverage on a project doesn’t mean you have zero bugs. Here is an extreme example to prove it.

Introduction

I am still finding a lot of people who want to increase code coverage to unrealistic numbers or who think that getting 100% code coverage is the holy grail that will eliminate all bugs and produce the perfect software.

There are many articles already on the net explaining why this is a fallacy, but I recently discovered that sharing an actual code example goes a long way towards proving why 100% code coverage doesn’t mean zero bugs. These people have their “aha” moment when they look at real code, instead of recycling theoretical arguments over and over.

So instead of explaining the same example again and again, I am sharing it in this post.

An “application” with just 1 line of code

I am writing a new application that has exactly one line of code.

func calculateVelocity(angle int, direction int) int {
	return ((3 * (4*angle - direction)) * 3) / (7 * (direction - (2 * angle))) * -1
}

This is a super important method that is going to be used for software that launches rockets into space. It needs to be rock solid.

Since I am a perfectionist, I am writing not one, not two but 6 unit tests for this function. And after running the tests I see that I have achieved the holy grail - 100% code coverage!

I go to bed and the next morning I see the news about the rocket explosion. Apparently this function has a bug and if the direction is double the angle, the function throws a divide-by-zero error.

So if for example direction is 4 and angle is 2 the rocket explodes:

panic: runtime error: integer divide by zero 

So there you have it. Right in front on your eyes there is an application with 100% code coverage that also has a bug. And this application was just one line of code. In a real application with 50,000 LOC you will probably have 100, 1000, 10000 bugs EVEN with 100% code coverage.

Achieving 100% therefore says nothing about your software quality. And by induction assuming that going from 90% to 100% will magically solve all bugs is also not correct.

F.A.Q.

Q. This article is useless. Everybody knows that 100% code coverage doesn’t eliminate bugs.
A. Junior developers certainly don’t know this. Misguided team managers don’t know it as well. Several articles that I am not going to link, also advocate for 100% code coverage. I actually wrote this article, because now I can give everybody the link instead of showing the same example again and again.

Q. Where did you get the input numbers for your unit tests?
A. From my business analyst, or my QA engineer or my buddy Fred. Does it really matter? All it matters is that I had 100% code coverage with them.

Q. Of course it matters. As soon as I saw your method I thought about the error and I would write a unit test for that. Your QA engineer missed that case.
A. Sure you can do that for 1 line of code. But you can do it for 5? for 500? for 50,000? If yes, then you have a superhuman ability and a) I want to hire you b) no need to write unit tests any more. Just fix the code right away if you can spot all bugs of a system.

Q. So you are saying that unit tests are useless?
A. No, of course not. Tests are great for catching regressions or functioning as written specifications. All I am saying is that trying to increase code coverage in order to have less bugs is a misconception.

Q. So if code coverage is not a good metric for the quality of the project, what metrics should I look instead ?
A. I have proposed the useful but largely unknown metrics PDWT,PBCNT,PTVB,PTD in my testing guide.

Q. So what should I do if I want less bugs on my mission critical software?
A. You should look at other approaches such as formal verification and proof checkers. Check also Mutation Testing.

Q. Look, this is great, but I still need to find the perfect code coverage for my new project. If it is not 100%, what should I aim for?
A. Without more context on your project all I can say is that the minimum code coverage is 20%. This number is based on the Pareto Principle and assumes that 20% of your code is responsible for 80% of your bugs. See also my point about “critical code” in my testing guide.

Q. My organization/manager/customer insists on achieving 70%/80%/100% code coverage on all projects. What should I do?
A. Send them this article. If you cannot convince them, maybe try to find a different organization/company/customer to work for?

Go back to contents, contact me via email or find me at Twitter if you have a comment.