% test coverage doesn't provide useful information

A % test coverage number doesn't really answer the question, "What kind of confidence does running my tests give me about the state of the system?" The only way you answer that question adequately is when you know what is actually tested. What are the test cases? How are they executed? If I've changed a particular function, do I have a safety net? Or does there happen to be a gaping hole where I'm working?

An unjustified high expectation in terms of coverage means that I'm overconfident... which leads to defects. An unjustified low expectation in terms of coverage means that I'm wasting time on the wrong things and not spending that time on the right things... which leads to defects.

When should you use Agile versus when should you use Waterfall?

The question of when to use Agile versus Waterfall comes up every so often for me.  Sort of an occupational hazard.

I don't really like the question as it's typically asked because it usually reflects sloppy thinking.  What do you mean when you say "Agile"?  What do you mean when you say "Waterfall"?

Usually the question that is intended is really...

"When is it okay for me to specify everything up front, code it all, test it all at the end, and deploy it all in one shot?"

I don't really like the question because when I say Agile, if I were to bother using the word, I'm not just talking about a process; I'm talking about one way of thinking vs another.  For example, a few fundamental beliefs:

• If change requests are expensive, customers will always ask up front for more than they actually need
• Team commitment comes from more team involvement
• It's better to plan based on observed reality rather than hope
• Quality is fundamentally tied to frequent feedback to confirm assumptions and decisions

If you know your history, you know that the first paper that introduced waterfall as a process model also warned that it doesn't actually work in practice, that iteration due to learning will always occur.

Normally I say you use Waterfall on a project when:

• You have no legitimate competition or don't intend to be competitive.
• You've created this solution many times before and for some reason you haven't extracted a framework or library.
• You have close to perfect knowledge about what to build and how to build it and therefore you see no opportunity to learn more.

If this is a matter of comparing mindsets, then the question can be translated to...

"When is it okay for us to not care about being competitive and not care about learning?"

If you only spend money on absolute certainty...

If you only spend money on absolute certainty, you will absolutely destroy any culture of improvement.   However, cheap and quick is better than expensive and delayed.

CITCON Melbourne 2008: Visualising quality

This is the third session I facilitated at CITCON Melbourne 2008...

----

How do we see quality? How do we convert a vague concern about software quality into something that is a little more tangible and explicit?

It's worth distinguishing between measurement versus visualisation. Measurement is about acquiring information, finding indicators, improving understanding. Visualisation is about representing information, communicating understanding. Both aspects are required to achieve the result we want.

Because visualisation is about communicating, it's important to select representations that target the audience. This is one of the issues with tools that automatically generate charts and graphs without the users actually thinking about what goal they're trying to achieve, what questions they're trying to answer, and therefore what measures and representations they should use.

There is also visualisation to communicate other things besides the quality of the system. For example, communicating the emotional state of team members. A version of this is known as a niko-niko calendar and is specifically about measuring and communicating morale/motivation. I've also blogged about a South Park avatar variant.

Are there any bug-tracking tools that allow users to tag bugs with arbitrary classifications which could then allow the potential discovery of bug clusters simply by looking at a tag cloud? Bug trackers typically allow a single categorisation usually based on the system components. What we want are tools with the ability to classify using mulitple facets.

CITCON Melbourne 2008: Managing test data for regression integration testing

This is the second session I facilitated for CITCON Melbourne 2008...

----
Why do we test against data extracted from production? Why bother? Shouldn't we just generate all our data instead?

The problem is that in most cases when the production data set is large and complex enough, our understanding of the characteristics of the data is often incomplete.

Therefore...

The purpose of testing with production data extracts is to find inconsistencies with our understanding of production data.

This leads to have two primary sets of test data:

1. Data to exercise system boundaries
2. Sample extract from production
   

With repeated automated integration testing, gathering and maintaining production extract data is non-trivial as the sample may not have the type of data we want, the extract is regularly updated which requires updating the tests, etc.

There are several tactics to deal with this:

Query and filter the data for the sample. The criteria required for the tests must be included in the query to create the sample. On a related note, any data considered irrelevant can be filtered out.

Even with querying, it's possible, perhaps likely, that the data is still incomplete for the purposes of our tests. Therefore, we need to have a way to automatically setup the extra test data we need. The determination of what that extra test data is can also be automatically done or at least the missing scenarios can be automatically detected.

Some people will create full-on applications to setup test data. If we measure the time people are spending to do this, this may easily pay for itself quickly.

If the purpose is to find inconsistencies with our assumptions about the nature of production data, it's worth continually validating our assumptions. Some teams run regular suites against production data testing that validate our assumptions. In many cases, data sources do not necessarily have the ability to enforce constraints or the constraint mechanisms are not used to accommodate for legacy data.

We have to be careful about the validity of our sample so it may be worth introducing variation to the test data to ensure we increase the probability that we discover any hidden variants that our system can't handle. Of course, we would probably need to maintain both known and varying test data sets.

Something that seems rare but most of use acknowledge is worth doing is a dedicated test data team. The responsibility of such a team would be to improve the organisation's understanding of the nature of the production data, advocate production data clean-up, help supply other teams with test data, and provide support and tools for using, creating, and manipulating test data.

CITCON Melbourne 2008: What else contributes to quality before testing?

The first session I facilitated at CITCON Melbourne 2008...

----

If we think about software development in very broad steps, it consists of the following:

Think -> Produce -> Inspect

Testing falls under the Inspect bucket. If we focus on the other buckets, Think and Produce, what else can we do to contribute to quality? If we've learned anything, we know that relying on end-of-cycle inspection is both an inefficient and ineffective way to provide quality.

If we're looking for quality, we should start with engineering production practices, not testing practices. How we build something has a significant impact on how many defects will appear. This goes beyond always asking the question, "How can this be tested?" but also to the question, "How can I design this such that this class of problem cannot occur?". This is known in Lean circles as poka yoke. Some people suggest that Design By Contract is a form of poka yoke.

If we're looking for quality, we should prefer technology that has readily available expertise and is amenable to good practices. The most significant source of technical quality problems during the production of software is a lack of knowledge of the technologies being used. Commodity technologies, that is technologies that are in wide use, are more likely to have well-established established good practices and many people with expertise. This can be a trade-off as some commodity technologies are not as amenable to good practices.

If we're looking for quality, we want everyone to have an emotional connection to the customer's context.

Quality is a customer determination, not an engineer's determination, not a marketing determination, or a general management determination. It is based upon the customer's actual experience with the product or service, measured against his/her requirements - stated or unstated, conscious or merely sensed - and always represents a moving target. -- Armand V. Feigenbaum

It doesn't really matter how well we build something if we build the wrong thing. If we really want quality, with the understanding that it's wholly a customer determination, then everyone involved in the development effort, not just "domain experts", needs to be in the customer's head-space.... and this is not just in the abstract, academic sense but an emotional, visceral connection to the customer's context. The more people on the team that have this, the more likely we produce something that has quality.