A % test coverage number doesn't really answer the question, "What kind of confidence does running my tests give me about the state of the system?" The only way you answer that question adequately is when you know what is actually tested. What are the test cases? How are they executed? If I've changed a particular function, do I have a safety net? Or does there happen to be a gaping hole where I'm working?
An unjustified high expectation in terms of coverage means that I'm overconfident... which leads to defects. An unjustified low expectation in terms of coverage means that I'm wasting time on the wrong things and not spending that time on the right things... which leads to defects.
Yahoo!7 Girl Geek Dinner Pictures & Screencast
-
Above are some pictures from the Girl Geek Dinner night at Yahoo!7 If you
have photos on flickr please add them to the pool too. Pool/Group ID
1173495@...
3 weeks ago
5 comments:
Absolutely right. TDD has been perverted into the idea that you have to have a test before you write code. This is partly true, but of course the test should be about your functionality not about the code you are going to write.
If you cover 100% of your use cases you can actually use the coverage analysis to see what production code is useless...
I want to agree with you... I really do. But I think that the mere fact you feel able to make that statement means you are dealing with a different (higher) class of issues than I have had to deal with at most of my workplaces.
Because when you are in an area of code with 40% or 20% or even 0% test coverage, that number does provide useful information. And it is able to answer the question "What kind of confidence does running my tests give me about the state of the system?" The answer is somewhere between "only a little" and "none at all".
Which is to say, I completely agree that high code coverage numbers are not a sufficient condition for confidence. And I also believe that there can be a point of diminishing returns where the ROI does not justify getting your coverage up to 100%. And I agree that we shouldn't blindly chase after coverage metrics to the detriment of having tests that actually mean something.
But, I do think that relatively high test coverage is a necessary condition for confidence. And if you are developing all (or most) of your code using TDD or BDD, your coverage will be high (thus the metric can also useful for spotting which code was obviously not written using TDD).
Hi Nick,
My point is what action should one take when faced with only a % coverage number. I'm suggesting that the correct thing to do to be able to reasonably act with confidence is to take a closer look at what is actually tested.
Ah, okay. Yes, I agree with that. :-)
Have to agree with Nick's first comment above. Test coverage doesn't tell everything about the quality of the tests of your system, but it is better than nothing. If I compare a system with 90% test coverage to a system with 10% test coverage, I can surely say that the first one is in better state than the second :)
But I also agree with you in your reply to him :)
Post a Comment