Here’s an extract from a blog post I recently read:
Consider the boiling of water. That’s straightforward, water boils at 100 °C, right?…
Put yourself in the shoes of someone at the start of the 1800’s, with only a crude, unmarked mercury thermometer, trying to figure the physics of temperature.
Go to your stove, put some water in a pot, start heating some water, and pay attention as it heats.
The first thing you’ll probably notice is a lot of small bubbles gathering on the surface of the pot. Is that boiling? The water’s not that hot yet; you can still even stick your finger in. Then the bubbles will appear faster and start rising, but they somehow seem ‘unboiling’. Then you’ll start to see little bubble storms in patches, and you start to hear a hissing noise. Is that Boiling? Sort of? It doesn’t really look like boiling. The bubble storms grow larger and start releasing bigger bubbles. Eventually the bubbles get big and the surface of the water grows turbulent as the bubbles begin to make it to the surface. Finally we seem to have reached real boiling. I guess this is the boiling point? That seems kind of weird, what were the things that happened earlier if not boiling.
To make matters worse, if you’d used a glass pot instead of a metal one, the water would boil at a higher temperature. If you cleaned the glass vessel with sulfuric acid, to remove any residue, you’d find that you can heat water substantially more before it boils and when it does boil it boils in little explosions of boiling and the temperature fluctuates unstably.
Worse still, if you trap a drop of water between two other liquids and heat it, you can raise the temperature to at least 300 °C with nothing happening. That kind of makes a mockery of the statement ‘water boils at 100 °C’.
It turns out that ‘boiling’ is a lot more complicated than you thought.
This surprising amount of detail is not limited to “human” or “complicated” domains, it is a near universal property of everything from space travel to sewing, to your internal experience of your own mind.
Suppose that (say, at school) you were asked to test the claim that water boils at 100 °C. I’d like to highlight two possible approaches you could take.
Approach 1: You get some water, a pot and a thermometer. You heat the water in the pot until the thermometer reads 100 °C. You (probably!) observe bubbles on the surface of the water and steam coming out of the pot. You conclude that water does boil at 100 °C, and there’s not much more to test.
Approach 2: You actually observe the water as you heat it. You notice some things that don’t really make sense to you (how is it that the water bubbles and yet is still cold, for example?). You consider some other factors, such as whether the vessel you heat the water in makes a difference. You consider the interaction between water and other residue substances it may come into contact in before you heat it. And so on. You conclude that there’s so much that you could learn if you continued to test this claim further!
Now take a moment to consider the following: it’s really, really easy to take approach 1, and to (erroneously) conclude that you understand how boiling works.
This is a very common failure mode, which happens in the following way:
- You think that you know the expected behaviour, and all the factors that are relevant (in approach 1, the assumption is that the only relevant factor is the temperature of the water)
- You run a test that confirms the expected behaviour, ignoring all the details that aren’t considered in your model (like the type of vessel you heat the water in)
I certainly didn’t know most of the facts presented above about water boiling, and before reading that article, I thought I had a pretty good understanding of how boiling worked – how wrong I was!
Also, it’s worth highlighting that some of these surprising details are extremely important in certain circumstances: “The possibility of superheating liquids means it’s important to use a packed bed when boiling liquids in industrial processes lest your process be highly inefficient and unpredictable.”
Now, suppose that you’ve been asked to test a feature of a piece of software. Given infinite time, how much testing COULD you do?
As a concrete example, consider the following random number generator:
It’s worth taking some time (say, 5 minutes) to think about just how much you could test here. It’s worth thinking about this in two ways:
- If you made the assumption that this piece of software was random, how would you confirm that?
- If you threw out the assumption that this piece of software was always random, and desperately tried to find ways in which this software wasn’t random, how would you go about doing that?
If you’ve gone through that exercise, you should notice that in the latter case, your list is vastly longer!
All the above has been building up to a key point that I want to make: given that reality has a surprising amount of detail, the set of things that you could test is always vast! I want to make this point because I occasionally hear people say things like “there wasn’t much to test” or “We’ve got complete coverage of this feature”. Let me be clear – these statements are simply incorrect. I think what they mean to say is “there’s nothing else worth testing”. This is a very different statement.
Now, an obvious objection to what I’ve been talking about above is “well, clearly we’re not going to have time to test everything we could test”. This is obviously true. However, I think the exercise of thinking about everything you could test if you had infinite time is a useful exercise, because you’re more likely to notice that certain areas that you wouldn’t otherwise have considered are indeed worth testing, or worth testing in more detail.
So, I’d like to leave you with a couple of things. The first one is a heuristic for generating test strategies/plans:
- Do a charter focused on generating as many ideas as possible for things you could test, if you had infinite time
- Do a charter focused on evaluating the costs and risks of doing/not doing each area of testing
- Communicate to the relevant stakeholders which testing you are and are not going to do based on the constraints you have, explaining clearly the risks of not doing further testing
- Incorporate the feedback that you’ve got from stakeholders and repeat steps 1-3 as appropriate
The second thing is some good questions to ask in test debriefs or when testing is reported more generally:
testing would you do if you had another day?
- Do not accept “None” as an answer – there is definitely testing they could do if they had more time
- Why are we doing that additional bit of testing? Is it worth an additional day?
- Why aren’t we doing that additional bit of testing? What’s the risk if we don’t do it?