What Do You Do?

This blog post has been inspired by the recent Test Leadership Congress. I would highly recommend the conference!

I.

What do you do?

It’s a question we’re asked all the time, at networking events, at parties, on dates (good or bad), the list goes on. Often, we’ll answer by giving our job title: “I’m a tester”; “I’m a senior developer”; “I’m a product owner”, and so on.

Now, I’m not saying job titles are a bad thing. They’re labels that give useful information about activities that we may or may not do at work. If your job title is “junior developer”, you probably spend some time writing code, and you probably don’t attend quarterly board meetings where you attempt to justify the company’s overall strategy for the next 12 months. If your job title is “tester”, you probably spend some time looking for bugs, and you probably don’t spend time trying to define the overall architecture for the new product your team is building. If your job title is “<something> manager”, you probably spend most of your time doing “management things”. Labels do give a sense of what you do.

However, labels take away, reduce, and diminish the complete picture of what you do and who you are.

One of the talks I heard at the conference (titled “More Than That”) was given by someone who called himself a tester (amongst other things). He told an anecdote about how he applied for a job that was basically being a business consultant for a real estate company. At the interview, the hiring manager asked him the following: “So, I’ve taken a look at your CV, and there’s something I’m concerned about. I see that you’re… just a tester. What makes you think you’ll be able to do this job?”

II.

Just a tester. Do you ever think of yourself as “just a tester”?

I’ve certainly had that feeling before. Occasionally, I’ll meet people in social situations that work in tech. When they ask me what I do, and I tell them I’m a tester, I’ll often get looks of surprise or confusion. They’ll often say things like:

  • “But isn’t being a developer more interesting?”
  • “So, do you actually enjoy spending your time writing test cases?”
  • “If you want to progress your career, you need to get out of testing”

I’ve gotten pretty good now at explaining to people why testing is a complex, valuable thing and why being a testing specialist is a useful, challenging profession. However, it does make self-doubt creep in sometimes.

However, I’ve definitely seen examples where viewing myself as a tester has stopped me from doing things that would have been valuable and useful:

  • Believing that I can’t write good code because I’m just a tester, and therefore:
    • Being reluctant to do code reviews of our test automation code
    • Being reluctant to refactor our test automation code
    • Being reluctant to suggest new tools or try to write tools to help testing
  • Believing that I can’t contribute usefully to architecture or design meetings because I’m just a tester, and therefore:
    • Being reluctant to get involved in design meetings
    • Being reluctant to speak up or provide input in design meetings
  • Believing that I can’t really change processes because I’m just a tester, and therefore:
    • Accepting processes as they are, even if they’re not working for me
    • Only suggesting minor tweaks to processes, rather than wholesale changes
  • Believing that I can’t really change the project parameters because I’m just a tester, and therefore:
    • Accepting time or resource constraints, even if I think they’re unreasonable
    • Accepting the output of other teams, even if I think it’s not to the required quality bar for the given project

In all cases, it was this label of being “just a tester” that was stopping me. So, how to solve this problem?

III.

The answer came from a talk called “Finding Power in Authenticity”. What do I mean by authenticity? To quote the speaker:

Authenticity is acknowledging who you are: Finding power by embracing your strengths and working on improving your weaknesses.

In her inspiring talk, she told the story of how she moved country and found herself in an unfamiliar role as a quality analyst. It was a solo role, where she was expected to work as a modern QA as part of a high-performing team. After going through a period of struggling initially, she asked herself: “what differentiates me that others value?” By listening to her authentic voice, she found plenty of ways to provide value:

  • Using her curious nature as a “question asker” to drive quality conversations to help align the team
  • Using her natural attention to detail and product focus to provide value when pairing with developers on infrastructure
  • Using her passion for monitoring and logging to allow the team to monitor previously hidden issues in the quality of the product

IV.

Inspired by these talks, I’ve started to listen to my authentic self to shape what I do. If somebody were to ask me “what do I do”, I’d respond by saying I’m a quality inspirer – the thing I love doing most is leading and coaching teams to a more mature quality culture. This has started to manifest in the work I do:

  • I love coaching people: I’ve started mentoring new testers and coaching developers on writing end-to-end automated testing scripts
  • I love optimising processes: I’ve started to make large tweaks in the ways we do testing (e.g. embracing a Session-based test management approach and doing more time-boxed exploratory test sessions)
  • I am a dreamer who thinks things can be different: I’ve done some work to restructure our test automation framework to make it easier to write and maintain tests

Ultimately, by doing this, I’m even happier at work! If you want to make your work fun, I’d encourage you to embrace your authenticity. How?

  • Embrace your talents and interests
  • Open yourself to the possibility of change
  • Learn by experimenting, and find out what works for you

I’ll leave you with three questions:

  • What do you do?
  • Does what you do harness your talents and interests?
  • How could you change what you do to harness your inner strengths?
Advertisements

Lack Of Time = Lack Of Priorities

The following is inspired by the recent UKSTAR testing conference. I’d highly recommend it to anyone.

I.

Three questions:

  • Do you feel that you are always busy, and can’t do everything you have to do?
  • Do you think that a lack of busyness equates to laziness?
  • Do you often find yourself thinking “If only I had more time” or “I have too much to do”?

If the answer to any of the above questions is yes, then the following advice might be useful:

A lack of time is a lack of priorities.

In more detail (quoting from this blog post):

“””

Sometimes the simplest ideas really stick out.

This was particularly true for me when I read Tim Ferriss’s The 4-Hour Workweek. I pulled a bunch of good ideas from his book, but one in particular that stands above the rest is this:

A lack of time is a lack of priorities.

I think this idea has improved my life over the last few years more than any other tidbit of productivity or life hacking advice.

Most often, a lack of time—time pressures, rushing, scrambling to finish things, busyness—is simply a lack of priorities. We can choose to spend our time differently, on the things that are more important, and ditch the things that aren’t, thereby freeing up our time and energy.

We choose to be busy or rushed, and we can choose otherwise.

“””

II.

A lack of time is a lack of priorities.

I concur that this idea has improved my life over the last 18 months more than almost any other piece of advice.

When I first started working after uni, I found myself overwhelmed by the demands of doing a full-time job, handling domestic chores, doing self-care, socialising, maintaining relationships, pursuing my hobbies, and a million other demands on my time.

I found myself committing to way too much, realising far too late that there was no way that I was going to be able to do everything I’d committed too, and upset a lot of people along the way.

Something had to change.

III.

A lack of time is a lack of priorities.

The first thing that helped was realising that there was almost nothing I had to do.

In more detail (quoting from this blog post):

“””

There is nothing in the world that we truly must do. We can choose to do or not do just about anything. Take wearing clothes in public. It sure feels like you have to do it—and if you’re like me you want to do it too—but you don’t actually have to.

I found this idea very hard to internalize at first.

Let’s say you feel that you have to complete an important assignment. If I say, “You don’t have to do it, you can choose to do it or not,” you’ll probably think, “Okay, that’s true… but it still feels like I have to do it!”

This can be so deeply engrained in your psyche that it will take a long time to fix. I’m not entirely sure what helped me move away from “must” thinking to “choice” thinking; it’s been such a gradual change over time.

Being aware of the distinction is probably the place to start. If you can constantly remind yourself that you choose to do the things that you do, you’ll slowly shift away from “must” thinking.

“””

You can always make more time for things if you choose to.

IV.

A lack of time is a lack of priorities.

The second thing that helped was actually thinking about what my priorities were.

In more detail (quoting from this blog post):

“””

Without a clear sense of priorities you’ll have difficulties deciding where to best spend your time and energy.

If you don’t know your priorities it’s probably worth your time to sit down and think about it. You’d be amazed what you can come up with in just five minutes of conscious effort.

“””

How many of you know actually have a clear sense of your priorities?

Lots of things sound “plausibly important”, and without having a sense of priorities, I found it really hard not to say that everything was important.

At the beginning of this year, I spent an afternoon thinking about exactly what my priorities were, and explicitly writing them down. Most productive afternoon I’ve spent this year. Since doing that, I’m basically never stressed about having too much to do. I can pick what’s most important (based on my priorities), and do just that.

V.

A lack of time is a lack of priorities.

The final thing that helped me was learning to use the thought “I have too much to do” as a trigger to reduce the amount I’m going to do.

In more detail (quoting from this blog post):

“””

Over the last few years I’ve slowly improved my ability to notice that feeling of “If only I had more time!” or “I have too much to do!”

Whenever I catch myself reasoning along these lines I think, “Wait! I don’t have to be busy. What’s important here? What’s not? What else could I be doing with my time?” etc.

In other words: Pause. Reassess. Decide. Relax.

This has come in handy more times than I can count.

“””

I endorse practicing this technique – whenever you find yourself thinking “I have too much to do”, practice using that as a trigger to reassess what you actually have to do.

VI.

To summarise (quoting from this blog post):

“””

You have all the time in the world to do what’s important.

Don’t have the time? Then drop something else to make the time.

Or, accept that it’s not important enough and move on.

Or, accept that you’ll voluntarily be busy for the next little while; that it’s a choice, not a burden.

“””

Against Perfectionism

Against Perfectionism

The following is a blog post inspired by the recent UKSTAR testing conference. I’d highly recommend it to anyone.

I.

From this blog post:

“””

Say you’re a college student, and you have a paper due. The quality of the paper will depend upon the amount of effort you put in. We’ll say that you know the project pretty well: you can get an A with only moderate effort, and with significant effort you could produce something much better than the usual A-grade paper.

The education environment implicitly attempts to convince students that their preferences point ever rightward along this line. Parents and teachers say things like “you should put in your best effort,” and they heap shame upon people who don’t strive to push ever rightward along the quality line.

People generally react to this coercion in one of two ways. The first group (the “slackers”) rejects the implication that quality=preferences. These are the people who don’t care about the class, who complain constantly about the useless pointless work they have to do, who half-ass the assignment and turn in something that either barely passes or fails entirely. Slackers tend to resent the authority forcing them to write the paper.

The second group (the “tryers”) are the ones who accept the premise that quality=preferences, and strive ever rightwards on the quality line. Tryers include people of all ability levels: some struggle as hard as they can just to get a C, others flaunt their ability to produce masterpieces. Some try to curry favour with the teacher, others are perfectionists who simply can’t allow themselves to turn in anything less than their best effort. Some of them are scrupulous people, who feel guilty even after getting an A, because they know they could have done better, and think they should have. Some are humble, some are show-offs, but all of them are pushing rightward.

Society has spent a lot of time conditioning us to think of the tryers as better than the slackers. Being a tryer is a virtue. Slackers are missing the point of education; why are they even there? The tryers are going to go places, the slackers will never amount to anything.

But in fact, both groups are doing it wrong.

“””

II.

At university, I saw the worst effects of this “tryer” culture.

The outrageous expectations placed on the students: “You should be able to read the entirety of Paradise Lost and write a 2,000 word essay on it within a week”; “Here’s an explanation of Gödel’s incompleteness theorems in 60 minutes – we expect you to have completely understood it by the next lecture in 2 days time”; “We expect you to maybe take a week off during the holidays, and then spend 40 hours a week revising the material the rest of the time (an actual quote from a lecturer!)”.

The oppressive competitiveness that permeated every student interaction: “I heard she didn’t get an internship this summer – how embarrassing”; “I heard he got a 2:2 – poor him”; “God, you never see them out, they’re just studying all the time, what a loser”.

The glorification and normalisation of sacrificing your mental health towards the pursuit of perfection at all costs: People literally spending 18 hours a day in the library in the two months leading up to exams; people casually talking about “week 5 blues”, as if getting downtrodden by an environment in a month is just normal; the expectation that if you’re not working yourself to exhaustion you’re not trying hard enough.

The devastating effects this wrought on people’s happiness! People dropping out because the expectation of “getting a first, being in a uni sports team, having a really good relationship with a really attractive partner, getting an internship at a prestigious company, being president of 17 societies and having all these amazing friendships” was (shockingly!) not something that they could live up to. People having breakdowns because the pressure of failing and of being a failure was too much to bear. The endemic and epidemic rates of mental health issues.

There is a different way.

III.

When I was at university, for the first time ever, I realised that I couldn’t do well academically without putting in an extraordinary effort. To get that first, I needed to forsake my social life and relaxation time to work as hard as I physically could, nothing else would have sufficed.

Fortunately, I remembered what I was fighting for: (quote again from this blog post)

“””

What is your goal in taking this class? Perhaps you’re doing it thanks to a combination of social pressure (your parents said to), social inertia (everybody else goes to college), and a vague belief that this is the path towards a good job and a comfortable life. Or perhaps you’re there because you want good grades so you can acquire lots of money and power which you will use to fight dragons. Or perhaps you’re there out of a genuine thirst for knowledge. But no matter why you’re there, your reason for being there will pick out a single target point on the quality line. Your goal, then, is to hit that quality target — no higher, no lower.

Your preferences are not “move rightward on the quality line.” Your preferences are to hit the quality target with minimum effort.

If you’re trying to pass the class, then pass it with minimum effort. Anything else is wasted motion.

If you’re trying to ace the class, then ace it with minimum effort. Anything else is wasted motion.

If you’re trying to learn the material to the fullest, then mine the assignment for all its knowledge, and don’t fret about your grade. Anything else is wasted motion.

If you’re trying to do achieve some combination of good grades (for signalling purposes), respect (for social reasons), and knowledge (for various effects), then pinpoint the minimum quality target that gets a good grade, impresses the teacher, and allows you to learn the material, and hit that as efficiently as you can. Anything more is wasted motion.

Your quality target may be significantly left of F — if, say, you’ve already passed the class, and this assignment doesn’t matter. Your quality target may be significantly to the right of A — if, say, you’re there to learn the material, and grade inflation means that it’s much easier to produce an A-grade paper than it is to complete the assignment in the maximally informative way. But no matter what, your goals will induce a quality target.

Both the slackers and the tryers are pursuing lost purposes. The slackers scoff at the tryers, who treat an artificial quality line like it’s their actual preferences and waste effort over-achieving. The tryers scoff at the slackers, who are taking classes but refusing to learn. And both sides are right! Because both sides are wasting motion.

The slackers fail to deploy their full strength because they realize that the quality line is not their preference curve. The tryers deploy their full strength at the wrong target, in attempts to go as far right as possible, wasting energy on a fight that is not theirs. So take the third path: remember what you’re fighting for. Always deploy your full strength, in order to hit your quality target as fast as possible.

Half-ass everything, with everything you’ve got.

“””

And that’s exactly what I did at uni. I realised that my main aims were picking enjoyable courses and getting a 2:1, so aimed for that, in the full knowledge that a first was never on the cards. It made my uni experience much more enjoyable.

IV.

One possible objection might be the following: “We want to be the best at what we do. Therefore, we need to push as far right as we can on the quality curve”.

Here’s 9 examples of us not doing that:

As a developer:

  • We choose not to make a design the best we could possibly muster, because we decide it’s not worth the time or effort to do it.
  • We choose not to refactor every bit of sub-optimal code that we see, because we decide our efforts are better spent elsewhere.
  • We choose not to write our tools to the same quality bar as our products, because we decide it’s not worth the investment.

As a tester:

  • We accept that certain bugs aren’t worth fixing because they’re not required to get us to the desired quality bar.
  • We decide not to test most areas that we could test, or don’t test as deeply as we could, because we decide that the information we’d uncover isn’t worth the effort.

As a customer support engineer:

  • We don’t immediately drop everything and fix an issue a customer has raised, and may not fix the issue at all, because we decide that they could live with the issue.

As a manager:

  • We choose not to do the majority of the things that would be beneficial to the project because there’s not enough time. Instead, we pick the highest value things, and do them as well as they need to be done.

As a product owner or product manager:

  • We choose not to implement the majority of requests that people make of the product, because most of them aren’t worth the investment.

As a senior manager or executive:

  • We choose not to do 99% of things because they’re not the things that provide maximum value.

What all these cases have in common are the same goal: hitting one’s quality target as fast as possible, with minimum effort.

V.

The modern testing mission statement is “Accelerate the achievement of shippable quality”. I think that thinking about testing in this way has some advantages over the context-driven school of thought.

Yes, the context-driven school of thought is a better guide when it comes to “how to test”, in terms of “evaluating a product by learning about it through exploration, experimentation, observation and inference”, and is useful in thinking about all aspects of quality in the right contexts (capability, reliability, usability, performance, scalability etc).

However, what I really like about the modern testing way of thinking about things is its emphasis on hitting the desired quality bar with minimal effort.

Take the word “shippable” for example. The implication here is that there are bugs that we don’t need to fix to have a shippable product. The other day, somebody mentioned that “bugs are bad, and should always be fixed”. NO! It’s not always worth it!

Take principles 2 and 3 for example:

  • We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
  • We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.

A modern tester could consider these principles, look at their organisation’s development practices, and realise that they can provide most value by driving the organisation towards a CI/CD system. Crucially, this is valuable because it makes it much easier for developers to get information about the quality of their code, and allows us to ship at any time.

Or take principles 4 and 7:

  • We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture
  • We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist

The minimal effort way to achieve the desired quality bar from day one is to make sure that the entire team is unified in their understanding of what “quality” means for a given project, and equipping them with the skills for them to assess the “quality” of the product themselves.

What all these cases have in common are the same goal: enabling the team to hit their quality target as fast as possible, with minimum effort.

VI.

So, to summarise:

Strive to remember what you’re fighting for.

Aim to hit the quality target with minimal effort.

As a tester, your goal is to enable the team to hit their quality target as fast as possible.

Half-ass everything with everything you’ve got.

Reject perfectionism.

How To Pursue Development Goals Strategically (Part Two)

(This is a follow-on from the previous blog post – there, I explain why you should pursue development goals strategically. This blog post will focus specifically on the “how”).

I.

Recently, I spoke to my friend about how they do personal development at their company:

“In my company, this is how we do personal development – at the beginning of each month, my manager and I set high-level personal development goals. Each week, I set smaller personal development goals, which build towards the high-level goals. At the end of each week, I score myself from one to three on each small goal and discuss this with my manager. At the end of each month, we do a proper review, where we analyse the progress I made towards the high-level goals. Then we repeat the process again.

If you’re not doing this, how do you know whether you’re developing or not?

I agree with my friend – this is the bar we should be aiming for when it comes to personal development.

  • You wouldn’t write product code without doing some thinking about design
  • You wouldn’t carry out testing for a project without doing some thinking about strategy
  • So why would you try and do “personal development” without doing some thinking about your approach?

II.

This is the approach that I personally take:

Step one: At the beginning of each month, decide what your high-level goals are.

My high-level goals for Jan 2019 were the following:

  • Learn how to give effective feedback
  • Identify and implement changes to our Dev/ST process
  • Learn how to effectively review functional specs/designs from an ST perspective

Note – these goals are quite vague. That’s okay, because the weekly goals will be much more specific.

Step two: Before the beginning of each week, decide what your weekly goals are. These goal should be linked to the high-level goals.

Here are some examples of weekly goals that I used:

  • [Learn how to give effective feedback] For people X and Y, identify one specific area where they would improve their capability. Give feedback to them about this. Get an explicit example of them using the feedback and getting measurably improved performance as a result. Justify with examples to manager in status on Friday
  • [Identify and implement changes to our Dev/ST process] Have a meeting with the Segment Routing team and discuss some of the Dev/ST process changes you’d like to make and come to an agreement about what process changes you’ll use on this project.
  • [Learn how to effectively review functional specs/designs from an ST perspective] For the EVPN Multicast project, consider the estimates and testability implications of the project plan. Either give suggested changes to these estimates/testability plans, or else come up with independent justifications for why they are sensible. Justify to manager in status on Friday

Step Three: As part of my status with my manager every week, explicitly set some time to talk about how the weekly development goals went.

As an example, with the feedback goal, my feedback wasn’t quite having the desired effect. We discussed this, and identified two issues:

  • I wasn’t explaining the goal/mission/context clearly enough
  • I wasn’t being specific enough in the feedback (for example, saying something is “good” without explaining what specifically was “good” about the thing)

Step Four: At the end of each month, I have a longer check-in meeting with my manager (30 minutes), where we discuss how the high-level development goals have gone, and what the next set of things to work on is.

As an example, with the Jan 2019 goals, we agreed that I’d made good progress on learning how to effectively review functional specs/designs from an ST perspective. Giving effective feedback went less well and digging into it suggested that the wider mentoring skills were the limiting factor. This helped me to shape my Feb 2019 goals.

III.

Some things to think about when setting goals (also see previous blog post):

For the person trying to develop themselves

Thing One: Personally developing yourself is your job. Nobody else is going to do it for you. Only you know what you actually want to achieve.

Thing Two: You can ask for help – and you can ask people who aren’t your manager, or even in your management line. A couple of examples:

  • I wanted to learn about how to build a team. Yanqing Cheng has done a lot of good thinking in this area, so I asked to chat to her for half an hour about it, which was really useful.
  • I wanted to learn how to teach contractors to do session-based testing effectively. Pete Whiting had done this for various projects he’s worked on, so I asked to chat to him for half an hour about it, which was also really useful.

Thing Three: For personal development to work, and for people to be able to help you, you must be honest with yourself:

  • Developing yourself is hard – are you willing to put the time and energy into doing it?
  • Developing yourself will be uncomfortable – are you willing to try new things, be vulnerable and sometimes fail?
  • Developing yourself requires commitment – are you willing to put in time every week for the next month? The next year? The next decade?

For the person coaching/mentoring others in personal development

DO:

  • Encourage people to think about what they actually want to achieve, and why
  • Challenge people to stretch themselves and push themselves outside their comfort zone (if they’re willing)
  • Explicitly give them the time to work on personal development (if Google Engineers can spend up to 20% of their time on personal development, we can find the time too)
  • Ask probing questions to understand their motivations and suggest other ways of doing things
  • Listen

DON’T:

  • Tell people what they should aim to develop – if it doesn’t come from them, it won’t work
  • Force people to push themselves outside their comfort zone if they’re not willing – it requires them to be bought into it
  • Sacrifice personal development time for the sake of achieving short-term project goals – otherwise you’ll be fire-fighting forever
  • Impose your way of doing things onto them – they’re different to you, so different things will work for them
  • Make evaluations or judgements without doing the active listening first

How To Pursue Development Goals Strategically (Part One)

I.

“I want to get better at coding using Rust”.

Suppose this was one of your personal development goals. How would you go about achieving this?

Here are some approaches that you could take:

All the above have a non-zero chance of achieving the goal “I want to get better at coding using Rust”. But:

  • How do you know that the approach you’re taking is any good?
  • How do you know that there isn’t a vastly better approach you could be taking?

II.

When it comes to personal development, most of the time we choose to “pursue our goals” through routes that are far less effective than the routes that we could find if we tried.

More specifically, most people do at least the following:

  • Tell ourselves and others stories of how we’re aiming for various “goals”
  • Search out modes of activity that fit in with the narrative of goal-seeking
  • Feel happy or sad when we do/don’t achieve our “goals”

However, there are heuristics that would be useful to goal-achievement that we do not automatically carry out by default. By default, we do not do the following:

  1. Ask ourselves what we’re trying to achieve:
    • Is there a specific project I want to work on that requires Rust?
    • Am I learning just for interest’s sake?
  2. Ask ourselves how we could tell if we achieved it and how we can track progress?
    • Does “success” look like creating a new tool using Rust?
    • Does “success” look like being able to solve the first 100 HackerRank problems using Rust?
  3. Find ourselves strongly, intrinsically curious about information that would help us achieve our goal:
    • Have I spent 10 minutes sitting down and thinking about what information I need to get better at coding in Rust?
  4. Gather that information:
    • Have I asked anybody else how I could get better at coding in Rust?
    • Have I spent 5 minutes googling to try and find out how to get better at coding in Rust?
    • Have I thought about which strategies have/haven’t worked in the past when trying to achieve similar goals, like learning a new programming language?
  5. Systematically test many different conjectures for how to achieve the goals, including methods that aren’t habitual for us, while tracking which ones do and don’t work?
    • Have I tried more than one approach towards learning Rust?
    • Have I tried any non-habitual approaches towards learning Rust?
    • Do I have any idea what strategies actually work for me when learning a new thing?
  6. Focus most of the energy that isn’t going into systematic exploration on the methods that work best:
    • How much time do I actually spend using methods I know to be effective for learning Rust?
  7. Make sure that our “goal” is really our goal, that we actually want it, and we’re not constrained by fears or by uncertainty as to whether it is worth the effort
    • Am I actually just saying that I want to learn Rust to keep my manager happy?
    • Do I have people I can talk to for support in case it proves much harder than expected to learn Rust?
  8. Use environmental cues and social contexts to bolster our motivation
    • Have I got time budgeted and agreed with my manager to spend time learning Rust?
    • Have I scheduled one evening a week to go through that Rust tutorial?

III.

Most of the time, when we’re trying to achieve our goals, we just do things. Maybe we act from habit or from impulse; maybe we act based on a prompt from our manager; maybe we act because our friend suggested that this would be a cool thing to try. However, we do not systematically choose the narrow sets of actions that would effectively optimize for our claimed goals, or for any other goals.

But we can do much better! The heuristics above go a long way to optimizing for our claimed goals, but they require work to make habitual.

Do you want to get better at achieving your goals? And are you willing to put the work in to train yourself to get better?

In the next article, I’m going to share a framework for pursuing development goals strategically, based on the above heuristics.

Playing Devil’s Advocate

I.

Recently, I’ve come to believe the Earth is flat, not round. Seriously.

I know this opinion will be controversial to many of you, but hear me out. Firstly, intuition tells us that the earth is obviously flat – just look at it! There is empirical evidence to back up our intuition – one such famous experiment is the Bishop Experiment (and there are many others):

“””

California Monterey Bay is a relatively long bay that sits next to the Pacific Ocean. The distance between the extremes of the Monterey Bay, Lovers Point in Pacific Grove and Lighthouse State Beach in Santa Cruz, is just over 23 statute miles.

On a very clear and chilly day it is possible to see Lighthouse Beach from Lovers Point and vice versa. With a good telescope, laying down on the stomach at the edge of the shore on the Lovers Point beach 20 inches above the sea level it is possible to see people at the waters edge on the adjacent beach 23 miles away near the lighthouse. The entire beach is visible down to the water splashing upon the shore. Upon looking into the telescope I can see children running in and out of the water, splashing and playing. I can see people sun bathing at the shore and teenagers merrily throwing Frisbees to one another. I can see runners jogging along the water’s edge with their dogs. From my vantage point the entire beach is visible.

Correcting for the height of the observer of about 20 inches, when looking at the opposite beach over 23 miles away there should be a bulge of water obscuring objects up to 300 feet above the far beach. There isn’t. Even accounting for refraction, the amount hidden should be around 260 feet – seeing down to the shoreline should be impossible.

“””

Not only is there strong empirical evidence to back up the claim, there are robust arguments against attempts to falsify this belief.

  • Surely we’ve sent people to space, and they’ve seen that the Earth is round? Nope. As a concrete example, contrary to popular opinion, “There is no Flat Earth Conspiracy. NASA is not hiding the shape of the earth from anyone. The purpose of NASA is not to ‘hide the shape of the earth’ or ‘trick people into thinking it’s round’ or anything of the sort. There is a Space Travel Conspiracy. The purpose of NASA is to fake the concept of space travel to further America’s militaristic dominance of space. That was the purpose of NASA’s creation from the very start: To put ICBMs and other weapons into space (or at least appear to).”
  • How is circumnavigation possible? Here are some maps of what the world could look like; in particular, “The earth is surrounded on all sides by an ice wall that holds the oceans back. This ice wall is what explorers have named Antarctica. Beyond the ice wall is a topic of great interest to the Flat Earth Society. To our knowledge, no one has been very far past the ice wall and returned to tell of their journey. What we do know is that it encircles the earth and serves to hold in our oceans and helps protect us from whatever lies beyond.”
  • How does sunrise/sunset work? Here’s an explanation. Why are other planets round, but not the Earth? “The Earth is not a planet by definition, as it sits at the centre of our solar system above which the planets and the Sun revolve.” Why doesn’t gravity pull the Earth into a spherical shape? This is explained by the theory of Universal Acceleration.

Convinced?

II.

I’m assuming the vast majority of readers haven’t been convinced by the above If I asked you to attempt to try and refute the above, I assume that you’d put all your cognitive effort into doing things like:

  • Calling out questionable assumptions (like the implicit assumption that the our senses are always an accurate guide to understanding the world)
  • Being sceptical of certain claims (Are NASA really spending billions of dollars a year purely for propaganda purposes? Has nobody ever tried to go beyond the ice wall?)
  • Noticing that you’re confused by certain things (What about the phenomena that don’t seem to fit with the theory of Universal Acceleration?)

I’m further assuming that you wouldn’t find it difficult to do this, because you already are pretty much convinced that the Earth is round, so it’s not hard to attempt to try and falsify the claim that the Earth is flat.

But I now want you to consider the fact that even people that you hold in high esteem often hold beliefs that you think are wrong. Isaac Newton is generally regarded as being a pretty smart cookie, yet people often forget that he spent a significant amount of time dabbling in alchemy. There are some very, very smart people (see here and here) who genuinely believe in transhumanism. Elon Musk is Elon Musk and he’s got some views that are pretty out there.

More generally, you’ve seen friends and family, or other people you know who you consider generally to be very intelligent believe things that are wrong. As an example, I’ve seen friends do degrees or go into careers, where it was obvious at the time that they were going to hate it. And the scary thing is that they’d come up with plausible-sounding (but incorrect) explanations about why they were making the right decision (they’ve revised that opinion since!). You yourself know that you’ve made bad decisions in your life or held beliefs that were just completely wrong – the scary thing is that you were certainly convinced at the time that the decisions or beliefs were right.

The bottom line is this: a significant proportion of the claims that both you and people you hold in high esteem are making have major flaws, which they themselves cannot see.

III.

Suppose now that you’re testing something. I’d like to highlight two possible attitudes that you might have whilst testing it:

  1. I believe that this thing should fundamentally work. We should do some testing to check that this is the case, and I’m sure we’ll find some bugs. However, I think that in general, the thing is going to be okay.
  2. There is basically no chance that this thing will fundamentally work. I am going to make every effort possible to find evidence that this thing doesn’t work. Only after I’ve made the best effort possible to disprove that this works, if the evidence permits me to believe that this thing works, I may start to believe it.

When it comes to testing products, it’s not too hard to take attitude 2. We’ve all seen enough examples of faulty software to know that often products don’t behave in the way that we want them to behave.

What’s much more common is to take attitude 1 when it comes to:

  • Authoritative documents (such as requirements documents, or functional specifications)
  • Designs (at the level of whiteboard sessions, High-level Designs, System-level Designs etc)
  • Claims made by authoritative people (such as senior engineers, product managers etc)

However, by taking attitude 1, we are doing our colleagues a disservice! Just like everyone else, they are fallible. As testers, we spend our time (amongst other things) being professional sceptics, and we are not helping to create a better product if we don’t apply our critical thinking in these contexts as well.

To be clear, I’m suggesting doing the following: Take something like a functional specification or a requirement given by an authoritative figure; review it with the same level of scepticism that you might apply to a claim like “the Earth is flat”; feed it back to the relevant people. I think there are at least two valuable contexts where it’s worth doing this:

  • In meetings such as “whiteboard sessions”, “design meetings”, “requirements specification” etc – the tester’s role here is to play “Devil’s Advocate” in the sense described above
  • When reviewing authoritative documents (such as designs, requirements, functional specifications etc) – again the tester’s role here is to play “Devil’s Advocate”

I’ve been trying to come up with a catchy name for this type of testing – I’m going with “Playing Devil’s Advocate”, but if people have better suggestions, let me know.

IV.

So, how might a tester go about practicing doing this? As a potential option, I’d suggest doing something like the following:

  1. Look at the functional specification for a product that you’re going to test
  2. Explicitly make a note of all the claims and assumptions made in the specification
  3. For each of the claims and assumptions:
    1. Check whether there are any given reasons or justifications for the claims/assumptions made
    1. “Apply critical thinking” to evaluate whether these reasons or justifications are valid (as a good rule of thumb, if you sense that you’re confused by any of the reasons, or you don’t feel that you could confidently explain this reason to your manager, they’re probably not valid)
    1. Try and generate contexts in which these reasons/justifications would not be valid (that will help you to explain why they’re valid in the specific context it’s being applied to)
  4. Report your findings to the appropriate people. Ideally, it should be the people who own the document, but if you’re feeling nervous about this, report your findings to your manager to raise as appropriate.

I’m also suggesting that it should be general practice for testers to do charters to review authoritative documents, and to attend design meetings etc, and play the role that I’ve specified above.

Reality Has A Surprising Amount of Detail

I.

Here’s an extract from a blog post I recently read:

“””

Consider the boiling of water. That’s straightforward, water boils at 100 °C, right?…

Put yourself in the shoes of someone at the start of the 1800’s, with only a crude, unmarked mercury thermometer, trying to figure the physics of temperature.

Go to your stove, put some water in a pot, start heating some water, and pay attention as it heats.

The first thing you’ll probably notice is a lot of small bubbles gathering on the surface of the pot. Is that boiling? The water’s not that hot yet; you can still even stick your finger in. Then the bubbles will appear faster and start rising, but they somehow seem ‘unboiling’. Then you’ll start to see little bubble storms in patches, and you start to hear a hissing noise. Is that Boiling? Sort of? It doesn’t really look like boiling. The bubble storms grow larger and start releasing bigger bubbles. Eventually the bubbles get big and the surface of the water grows turbulent as the bubbles begin to make it to the surface. Finally we seem to have reached real boiling. I guess this is the boiling point? That seems kind of weird, what were the things that happened earlier if not boiling.

To make matters worse, if you’d used a glass pot instead of a metal one, the water would boil at a higher temperature. If you cleaned the glass vessel with sulfuric acid, to remove any residue, you’d find that you can heat water substantially more before it boils and when it does boil it boils in little explosions of boiling and the temperature fluctuates unstably.

Worse still, if you trap a drop of water between two other liquids and heat it, you can raise the temperature to at least 300 °C with nothing happening. That kind of makes a mockery of the statement ‘water boils at 100 °C’.

It turns out that ‘boiling’ is a lot more complicated than you thought.

This surprising amount of detail is not limited to “human” or “complicated” domains, it is a near universal property of everything from space travel to sewing, to your internal experience of your own mind.

“””

II.

Suppose that (say, at school) you were asked to test the claim that water boils at 100 °C. I’d like to highlight two possible approaches you could take.

Approach 1: You get some water, a pot and a thermometer. You heat the water in the pot until the thermometer reads 100 °C. You (probably!) observe bubbles on the surface of the water and steam coming out of the pot. You conclude that water does boil at 100 °C, and there’s not much more to test.

Approach 2: You actually observe the water as you heat it. You notice some things that don’t really make sense to you (how is it that the water bubbles and yet is still cold, for example?). You consider some other factors, such as whether the vessel you heat the water in makes a difference. You consider the interaction between water and other residue substances it may come into contact in before you heat it. And so on. You conclude that there’s so much that you could learn if you continued to test this claim further!

Now take a moment to consider the following: it’s really, really easy to take approach 1, and to (erroneously) conclude that you understand how boiling works.

This is a very common failure mode, which happens in the following way:

  1. You think that you know the expected behaviour, and all the factors that are relevant (in approach 1, the assumption is that the only relevant factor is the temperature of the water)
  2. You run a test that confirms the expected behaviour, ignoring all the details that aren’t considered in your model (like the type of vessel you heat the water in)

I certainly didn’t know most of the facts presented above about water boiling, and before reading that article, I thought I had a pretty good understanding of how boiling worked – how wrong I was!

Also, it’s worth highlighting that some of these surprising details are extremely important in certain circumstances: “The possibility of superheating liquids means it’s important to use a packed bed when boiling liquids in industrial processes lest your process be highly inefficient and unpredictable.”

III.

Now, suppose that you’ve been asked to test a feature of a piece of software. Given infinite time, how much testing COULD you do?

As a concrete example, consider the following random number generator:

It’s worth taking some time (say, 5 minutes) to think about just how much you could test here. It’s worth thinking about this in two ways:

  1. If you made the assumption that this piece of software was random, how would you confirm that?
  2. If you threw out the assumption that this piece of software was always random, and desperately tried to find ways in which this software wasn’t random, how would you go about doing that?

If you’ve gone through that exercise, you should notice that in the latter case, your list is vastly longer!

All the above has been building up to a key point that I want to make: given that reality has a surprising amount of detail, the set of things that you could test is always vast! I want to make this point because I occasionally hear people say things like “there wasn’t much to test” or “We’ve got complete coverage of this feature”. Let me be clear – these statements are simply incorrect. I think what they mean to say is “there’s nothing else worth testing”. This is a very different statement.

IV.

Now, an obvious objection to what I’ve been talking about above is “well, clearly we’re not going to have time to test everything we could test”. This is obviously true. However, I think the exercise of thinking about everything you could test if you had infinite time is a useful exercise, because you’re more likely to notice that certain areas that you wouldn’t otherwise have considered are indeed worth testing, or worth testing in more detail.

So, I’d like to leave you with a couple of things. The first one is a heuristic for generating test strategies/plans:

  1. Do a charter focused on generating as many ideas as possible for things you could test, if you had infinite time
  2. Do a charter focused on evaluating the costs and risks of doing/not doing each area of testing
  3. Communicate to the relevant stakeholders which testing you are and are not going to do based on the constraints you have, explaining clearly the risks of not doing further testing
  4. Incorporate the feedback that you’ve got from stakeholders and repeat steps 1-3 as appropriate

The second thing is some good questions to ask in test debriefs or when testing is reported more generally:

  1. What testing would you do if you had another day?
    1. Do not accept “None” as an answer – there is definitely testing they could do if they had more time
  2. Why are we doing that additional bit of testing? Is it worth an additional day?
  3. Why aren’t we doing that additional bit of testing? What’s the risk if we don’t do it?