Playing Devil’s Advocate

I.

Recently, I’ve come to believe the Earth is flat, not round. Seriously.

I know this opinion will be controversial to many of you, but hear me out. Firstly, intuition tells us that the earth is obviously flat – just look at it! There is empirical evidence to back up our intuition – one such famous experiment is the Bishop Experiment (and there are many others):

“””

California Monterey Bay is a relatively long bay that sits next to the Pacific Ocean. The distance between the extremes of the Monterey Bay, Lovers Point in Pacific Grove and Lighthouse State Beach in Santa Cruz, is just over 23 statute miles.

On a very clear and chilly day it is possible to see Lighthouse Beach from Lovers Point and vice versa. With a good telescope, laying down on the stomach at the edge of the shore on the Lovers Point beach 20 inches above the sea level it is possible to see people at the waters edge on the adjacent beach 23 miles away near the lighthouse. The entire beach is visible down to the water splashing upon the shore. Upon looking into the telescope I can see children running in and out of the water, splashing and playing. I can see people sun bathing at the shore and teenagers merrily throwing Frisbees to one another. I can see runners jogging along the water’s edge with their dogs. From my vantage point the entire beach is visible.

Correcting for the height of the observer of about 20 inches, when looking at the opposite beach over 23 miles away there should be a bulge of water obscuring objects up to 300 feet above the far beach. There isn’t. Even accounting for refraction, the amount hidden should be around 260 feet – seeing down to the shoreline should be impossible.

“””

Not only is there strong empirical evidence to back up the claim, there are robust arguments against attempts to falsify this belief.

  • Surely we’ve sent people to space, and they’ve seen that the Earth is round? Nope. As a concrete example, contrary to popular opinion, “There is no Flat Earth Conspiracy. NASA is not hiding the shape of the earth from anyone. The purpose of NASA is not to ‘hide the shape of the earth’ or ‘trick people into thinking it’s round’ or anything of the sort. There is a Space Travel Conspiracy. The purpose of NASA is to fake the concept of space travel to further America’s militaristic dominance of space. That was the purpose of NASA’s creation from the very start: To put ICBMs and other weapons into space (or at least appear to).”
  • How is circumnavigation possible? Here are some maps of what the world could look like; in particular, “The earth is surrounded on all sides by an ice wall that holds the oceans back. This ice wall is what explorers have named Antarctica. Beyond the ice wall is a topic of great interest to the Flat Earth Society. To our knowledge, no one has been very far past the ice wall and returned to tell of their journey. What we do know is that it encircles the earth and serves to hold in our oceans and helps protect us from whatever lies beyond.”
  • How does sunrise/sunset work? Here’s an explanation. Why are other planets round, but not the Earth? “The Earth is not a planet by definition, as it sits at the centre of our solar system above which the planets and the Sun revolve.” Why doesn’t gravity pull the Earth into a spherical shape? This is explained by the theory of Universal Acceleration.

Convinced?

II.

I’m assuming the vast majority of readers haven’t been convinced by the above If I asked you to attempt to try and refute the above, I assume that you’d put all your cognitive effort into doing things like:

  • Calling out questionable assumptions (like the implicit assumption that the our senses are always an accurate guide to understanding the world)
  • Being sceptical of certain claims (Are NASA really spending billions of dollars a year purely for propaganda purposes? Has nobody ever tried to go beyond the ice wall?)
  • Noticing that you’re confused by certain things (What about the phenomena that don’t seem to fit with the theory of Universal Acceleration?)

I’m further assuming that you wouldn’t find it difficult to do this, because you already are pretty much convinced that the Earth is round, so it’s not hard to attempt to try and falsify the claim that the Earth is flat.

But I now want you to consider the fact that even people that you hold in high esteem often hold beliefs that you think are wrong. Isaac Newton is generally regarded as being a pretty smart cookie, yet people often forget that he spent a significant amount of time dabbling in alchemy. There are some very, very smart people (see here and here) who genuinely believe in transhumanism. Elon Musk is Elon Musk and he’s got some views that are pretty out there.

More generally, you’ve seen friends and family, or other people you know who you consider generally to be very intelligent believe things that are wrong. As an example, I’ve seen friends do degrees or go into careers, where it was obvious at the time that they were going to hate it. And the scary thing is that they’d come up with plausible-sounding (but incorrect) explanations about why they were making the right decision (they’ve revised that opinion since!). You yourself know that you’ve made bad decisions in your life or held beliefs that were just completely wrong – the scary thing is that you were certainly convinced at the time that the decisions or beliefs were right.

The bottom line is this: a significant proportion of the claims that both you and people you hold in high esteem are making have major flaws, which they themselves cannot see.

III.

Suppose now that you’re testing something. I’d like to highlight two possible attitudes that you might have whilst testing it:

  1. I believe that this thing should fundamentally work. We should do some testing to check that this is the case, and I’m sure we’ll find some bugs. However, I think that in general, the thing is going to be okay.
  2. There is basically no chance that this thing will fundamentally work. I am going to make every effort possible to find evidence that this thing doesn’t work. Only after I’ve made the best effort possible to disprove that this works, if the evidence permits me to believe that this thing works, I may start to believe it.

When it comes to testing products, it’s not too hard to take attitude 2. We’ve all seen enough examples of faulty software to know that often products don’t behave in the way that we want them to behave.

What’s much more common is to take attitude 1 when it comes to:

  • Authoritative documents (such as requirements documents, or functional specifications)
  • Designs (at the level of whiteboard sessions, High-level Designs, System-level Designs etc)
  • Claims made by authoritative people (such as senior engineers, product managers etc)

However, by taking attitude 1, we are doing our colleagues a disservice! Just like everyone else, they are fallible. As testers, we spend our time (amongst other things) being professional sceptics, and we are not helping to create a better product if we don’t apply our critical thinking in these contexts as well.

To be clear, I’m suggesting doing the following: Take something like a functional specification or a requirement given by an authoritative figure; review it with the same level of scepticism that you might apply to a claim like “the Earth is flat”; feed it back to the relevant people. I think there are at least two valuable contexts where it’s worth doing this:

  • In meetings such as “whiteboard sessions”, “design meetings”, “requirements specification” etc – the tester’s role here is to play “Devil’s Advocate” in the sense described above
  • When reviewing authoritative documents (such as designs, requirements, functional specifications etc) – again the tester’s role here is to play “Devil’s Advocate”

I’ve been trying to come up with a catchy name for this type of testing – I’m going with “Playing Devil’s Advocate”, but if people have better suggestions, let me know.

IV.

So, how might a tester go about practicing doing this? As a potential option, I’d suggest doing something like the following:

  1. Look at the functional specification for a product that you’re going to test
  2. Explicitly make a note of all the claims and assumptions made in the specification
  3. For each of the claims and assumptions:
    1. Check whether there are any given reasons or justifications for the claims/assumptions made
    1. “Apply critical thinking” to evaluate whether these reasons or justifications are valid (as a good rule of thumb, if you sense that you’re confused by any of the reasons, or you don’t feel that you could confidently explain this reason to your manager, they’re probably not valid)
    1. Try and generate contexts in which these reasons/justifications would not be valid (that will help you to explain why they’re valid in the specific context it’s being applied to)
  4. Report your findings to the appropriate people. Ideally, it should be the people who own the document, but if you’re feeling nervous about this, report your findings to your manager to raise as appropriate.

I’m also suggesting that it should be general practice for testers to do charters to review authoritative documents, and to attend design meetings etc, and play the role that I’ve specified above.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s