What do you want?

I

Two quick questions:

  1. Suppose that you won $10,000,000 tomorrow. How happy would you be in a year’s time?
  2. Suppose that you became paralysed from the neck down tomorrow. How happy would you be in a year’s time?

We have a strong tendency to overestimate the impact that extreme events will have on our lives. For example, with the two questions above, a surprising result:

“In a very famous study published by researchers at Northwestern University in 1978 it was discovered that the happiness levels of paraplegics and lottery winners were essentially the same within a year after the event occurred. You read that correctly. One person won a life-changing sum of money and another person lost the use of their limbs and within one year the two people were equally happy.” (See here)

Once you know about this effect, you’ll see it crop up again and again and again. A few examples (Examples 3-6 are based on this):

  1. This blog post.
  2. This blog post.
  3. “College students overestimated how happy or unhappy they would be after being assigned to a desirable or undesirable dormitory.
  4. “People overestimated how unhappy they would be two months after the dissolution of a romantic relationship.”
  5. “Untenured college professors overestimated how unhappy they would be five years after being denied tenure.”
  6. “Women overestimated how unhappy they would be upon receiving unwanted results from a pregnancy test.”

So, what’s going on?

II

Section I was basically describing “The Impact Bias”. Harvard Professor Dan Gilbert has done a fair bit of research on this, and he suggests the following as one of the contributors to this bias:

“Focalism: People tend to overestimate how much they will think about the event in the future, and at the same time, they underestimate the degree to which other events will influence their thoughts and feelings. So people get too focused on a particular event and its emotional impact.”

If I were to sum the above up in my own words, one way of describing it is “Stop tunnelling! Don’t get obsessed with one event, but focus on the bigger picture, and work out what actually matters!”

III

A slight change of pace now. Imagine that you were developing some software for a customer (I’m assuming imagining this won’t be too much of a stretch!) Here are some examples of things that could go horribly, horribly wrong:

  1. With quite a tight deadline, you write code to implement 10 different features. Because of time constraints, the quality of the 10 features is not that great. However, when you deliver the product to the customer, it transpires that they only care about 2 of the features, and wanted them to be really good quality.
  2. This product is completely new, and has never been delivered to customers. As normal, the testing focuses on functionality, robustness, scalability, interoperability, performance, security, and you’ve gathered enough information that you’re reasonably confident in making the judgement that the product is of good quality for all these quality criteria. However, it turns out that the product is really hard to install, and the documentation is completely unhelpful when it comes to using the product. As a result, the customer stops using the product, because they can never get started using it.
  3. Your customers hate regressions, and you decide that you need many automated, regressible checks to help guard against this. A very large proportion of your testing effort involves writing scripts up front to carry out these checks, and after spending 2 weeks writing scripts, debugging them, and getting them working, you eventually find some fairly serious issues. You then realise that, if you’d used method X instead, you’d have found all these issues in a day, rather than 2 weeks.

The common theme that all these failures have is the fact that the team has succumbed to focalism! They’ve narrowed in on these sub-goals, but missed the bigger picture. Note the parallel to Impact Bias – we’ve succumbed to tunnelling in exactly the same way! Fortunately, both these biases have the same solution, which I’ll describe to you now.

IV

In the book “The Coaching Habit”, the author describes seven questions that allow you to have a greater impact. I personally think that all these questions are worth adding to a tester’s arsenal, but I’m going to focus on question four:

“What do you want?”

A key aim of asking this question is to make sure that you truly understand what the other person is trying to achieve, at the highest-level of abstraction possible. Useful questions to ask in addition to this question are the following: “why do you want that”; “what will that help you achieve”; “What problem are you trying to solve”. Once you know what they actually want, you can do stuff that you know will be actively useful. A few examples:

Example 1: Consider the following dialogue.

  • Person X: We’re going to make these 10 features, and I’ll need you to test them all.
  • Tester: What do these features do?
  • Person X: Feature 1 does this, Feature 2 does this, … and feature 10 does this.
  • Tester: What is the customer going to do with these features?
  • Person X: (thinks for a bit) Well, they definitely said that they wanted features 3 and 7 to do Y, but I think the other features will be useful to them as well.
  • Tester: Do we need to make all these 10 features now? If we do, we’ll have much less time to test them, and the quality will be less good.
  • Person X: Hmmm well I suppose we could just make features 3 and 7 for now, and if the customer asks for more features, we can make them later.

Example 2: Consider the following dialogue:

  • Tester 1: I’m blocked.
  • Tester 2: In what way are you blocked?
  • Tester 1: My test script is blocked on bug 896135 (which makes the system crash)!
  • Tester 2: What does the test script do?
  • Tester 1: Well, it configures a network, and then has 20 more steps. I’m hitting this bug on step 2.
  • Tester 2: Is it important which order you do the steps in?
  • Tester 1: (thinks for a bit) No, not really. The steps basically delete and re-configure stuff, checking that things are okay after each re-configuration step. So the steps are mainly independent.
  • Tester 2: So, what could you do to stop step 2 blocking you?
  • Tester 1: Hmmm, I could temporarily remove step 2, and try the other steps.

Example 3: Consider the following dialogue:

  • Person Z: I’ve just had an issue raised by a customer saying that process P takes too long. Once we’ve fixed the bug we’ve raised, we should do some testing to make sure that process P takes less than a second.
  • Tester: What does process P do?
  • Person Z: Well, process P is used when a link fails to switchover traffic to a working link. And it’s important that this switchover happens quickly!
  • Tester: What happens if the switchover doesn’t happen quickly?
  • Person Z: Well, then voice traffic is dropped for too long, and the person on the other end can’t understand what’s being said!
  • Tester: How long does traffic need to be dropped for before you can notice it?
  • Person Z: Hmmm, let me check. (Checks) Apparently 50ms.
  • Tester: So would you prefer us to do some testing to make sure that process P takes less than 50ms?
  • Person Z: Yes please.

A key takeaway both in work and in life – when people say “I want X”, don’t just blindly trust them! Do some digging, and make sure that you understand what they actually want!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s