So I typically don't bother regularly generating test coverage reports. The way I see it, there's only two reasons why I want a coverage report:
- compensation for lack of testing discipline
- safety net for not noticing a drop in testing discipline
Yahoo!7 Girl Geek Dinner Pictures & Screencast
-
Above are some pictures from the Girl Geek Dinner night at Yahoo!7 If you
have photos on flickr please add them to the pool too. Pool/Group ID
1173495@...
3 weeks ago
4 comments:
I set the coverage threshold somewhat below where it is currently (usually in the mid-80s). This means that build failures due to lack of test coverage are rare, and only as a result of an extended lapse.
When analysing why it's lapsed, having the historical data can be useful. "Gee, on Tuesday it was okay, but we can see here that on Wednesday and Thursday it dived.".
As an external reviewer of a project, it can also be really interesting to see if a team is paying attention. For example, following a TDD course a while back, we noticed that the coverage levels climbed rapidly for a while, then settled down again.
Robert.
Sometimes testing code coverage makes as much sense to me as doing a spellcheck on every word in a book, but not actually reading the chapters or the book in its entirety. Yes we have coverage on every word, but does the book get across the idea it intended to? Furthermore, coverage reports can have a lot of spurious variables. How do we measure the kinds of oracles used, or the testing techniques? One kind of testing might be the wrong tool for the job, or maybe using what James Bach calls diverse half-measures would be more effective.
Usually the only reason I want a coverage report is to see if there are areas we can focus different testing techniques. For example, if a developer has a lot of xUnit testing on one method, but few on another, I might spend more time testing where there is less coverage to complement what they have done. This is useful when writing code that uses a 3rd party tool that we may not be able to write xUnit tests against. I can always test it manually, and exercise the code at the interfaces from our project to the API.
-Jonathan Kohl
If the team has testing discipline the coverage report loses value. However, if you can automate it in your continuous integration, why not do it? It cost me about 20 minutes to set up NCover in my build file and now I have the coverage report if I ever need it. Maybe I won't, but if I do, I have the most recent one already generated.
Re: As an external reviewer
I agree that an external observer can use the report to assess testing discipline. However, if I was external, I'd rather assess defect rate.
Post a Comment