Eight totally bodacious usability testing tips expressed as 80s hit singles

1 April 2021 -

The 80s were the best of all the decades. I don’t actually remember them much, but if Stranger Things is anything to go by, it was all big hair, synthesisers, arcade games and deadly monsters from alternate dimensions (fun!). Technology had struck the perfect balance between being exciting but non-intrusive, and nobody got annoyed if you asked them to accept cookies. The future we imagined was one of flying cars and space travel, rather than of floods, reality tv presidents and hyper-targeted ads.

So, I thought I’d take a walk down memory lane, using some of my favourite 80s songs to explain the most important things I’ve learned in my time running usability tests.

So sit back, plug in your walkman, and enjoy these sweet beats.

1. Don’t You (Forget About Me)

Remember to be diverse in your recruiting

We all have unconscious biases. Even when we think we’re being fair and impartial, research tells us that we still subconsciously favour those that are similar to ourselves. I’ve never understood why people find this so surprising; humans love familiarity and predictability, and our monkey brains associate these things with people who are similar to us in culture, gender, ability or ethnicity. This isn’t something to be ashamed of but it is something that needs to be acknowledged and accounted for, now more than ever. And yet, when it comes to testing, we can often forget about this bias completely.

It doesn’t help that a lot of usability testing takes place in homogenous circumstances, usually due to a lack of time or resources. Guerilla testing, for example, is great — as long as you remember that you’re at the mercy of the place you’re testing. Going into a coffeeshop during working hours? Hate to say it, but the type of person hanging out in a cool coffeeshop in the middle of the day is a specific type of person. With even shorter time constraints, I’ve seen usability tests carried out on people within the organisation itself, who quite often have prior knowledge of the product and industry, and are more likely to be digitally adept.

How do you address this bias? Well, the easy answer is to remember to recruit users that are truly representative of your user base — including making sure you’re inclusive of minorities and the less abled — and to make sure you try to keep aware of your biases when interpreting the results of the research.

However, we don’t live in a perfect world (and much less a perfect industry) so testing is sometimes shoehorned into a project with minimal time and budget. You may not always have the luxury of being able to source a perfect mix of test participants. That’s okay. Just try to:

  1. Get as diverse of a mix as possible

  2. Flag the issue with your organisation so that they know that the research may be biased towards a certain group

  3. Compensate for this bias when interpreting results and designing

And remember, if it’s raining men, you’re probably doing it wrong.

2. The Magic Number

Five tests capture 85% of usability issues… or do they?

It’s one of the most famous UX rules of thumb: 5 usability tests captures 85% of usability issues. NN Group put forward this magic number decades ago to maximise return on time investment, and it’s been a fantastic ally of UX designers. Having such an achievable number to aim has removed the perceived difficulty of the process for designers everywhere and added much-needed agility.

But, of course, it’s never that easy. Debate has raged in recent years over the claim, lead by top dogs such as Jared Spool:

Now, I’ve followed Jared on Twitter for long enough to know better than getting in an argument with him. Thankfully, I don’t think that’s necessary here. What Jared (and others) seem to reject to is the 85% part of the adage, which they see not only as false, but more importantly as an excuse for designers to be lazy with their testing.

By all means, test five users, but realise that you may capture more usability issues if you test more users. The amount of tests you need will be very dependant on what you’re testing, as Ellie Martin sums up well in this article:

Any number of factors — the level of refinement of your website, the size of your user base, or likelihood of discovery for a problem — can change your magic number for usability testing drastically.

Also, remember that a lot has changed in the last few decades — as long as you’re using established interaction patterns, most users will be a lot more comfortable using what you’ve designed than they would have 20 years ago.

In my opinion, the ‘five users’ rule is still invaluable for getting buy-in for testing from stakeholders as well as setting ourselves realistic and achievable testing goals.

3. Atmosphere

Make sure your users are using the product in the way they normally would

Once upon a time, usability tests were hard. Users were given a phone with a horrendous spidery arm attached to a camera on it, or were sat in a room with a big two-way mirror, like an interrogation scene out of a movie. Then, they were told to relax, and to use the phone like they normally would.

I still remember reading Steve Krug’s Don’t Make Me Think and thinking what an absolute endeavour it sounded to set up a usability test — thousands of dollars worth of equipment and planning.

"Don’t worry, we’re not testing you."

Thankfully, modern tools have allowed for the democratisation of usability testing. Laptops and phones are portable, so testing can be conducted anywhere, not just in a usability lab. Screen, face and voice recording programs allow us to record tests without having to strap a restrictive apparatus to the phone.

Armed with these new superpowers, you’ve got more opportunity than ever to create an environment in which user is comfortable and can use the device in the way they normally would. I often notice that a user will often try to put the phone on a table and tap with their index finger, perhaps due to the test environment and the fact that they’re not using their own phone. Encourage them to use the phone as they normally would — depending on the user and context, this could be one handed, two handed using their thumbs, or held in one hand and touched with the index finger. If it’s a desktop test, remember to provide a mouse and a trackpad, depending on which one they’re used to. And if it’s a laptop, remember that they might even want to put it — would you believe it — on their laps!

Atmosphere also has to do with where you test: make sure you’re in a neutral zone; I’ve found great results when testing from a user’s place of work. The more you can make the user feel comfortable and unobserved, the better.

4. Bring the Pressure Down

Get your user relaxed, in any way possible

Try this experiment: sit next to someone using their phone or laptop, and watch them as they use it. Nod your head every now and then, and grunt disapprovingly once or twice. Note how they shift uncomfortably in their seat. Watch how them squirm.

Watching someone use their phone is akin to watching them whilst they’re getting changed. Why? Well, the majority of the time that we spend using our phones/tablets/laptops, we spend by ourselves. Our digital life is an intensely personal experience. Plus, many of us are self-conscious about how we use technology. Am I doing this the right way? Is there a shortcut I should be using?

And yet, when we test users, we expect them to use the device naturally, just as they would on their own, whilst we stare them down and furiously scribble notes. No amount of ‘we’re testing the product, not you’ can reduce the plain suckiness of someone watching you use a device over your shoulder.

There’s no simple solution for this problem, until we develop the technology to shrink ourselves down to the size of cockroaches and hide in users’ clothes (which will sadly bring up a whole other set of problems). However, a great weapon to fight this is to make sure the first 5–10 minutes of your test are spent building rapport with the user and trying to reduce the implicit pressure that comes with the testing environment. These crucial first minutes will dictate how the next 40 will go — if you can master them, you’ll see vastly better results.

If you can, I find throwing in a couple of bad jokes works wonders in getting your user comfortable. And, if you don’t get a laugh, you can iterate on them and try again with the next user. With any luck, by your final test you’ll be the design equivalent of Robin Williams.

A couple of other tips to stop users being under too much pressure:

  • Try to avoid note-taking in the room with the user, if you can, and just use the recording to take notes after.

  • I’ve noticed that it’s easy to forget to make eye contact with the user when talking to them, since you’re both focused on the device. Force yourself to!

Remember to get consent when recording the user — and keep their details confidential

Like most ethical matters, this one is pretty simple to understand, and yet somehow can still get forgotten. If you’re recording someone’s voice or face during a usability test, you’re gonna need to get written consent from them, which stipulates how you’re going to be using the recording, and that they have the right to withdraw their consent at any time.

Usability test forms are super easy to find online, so there’s really no excuse. I’ve never had a problem getting consent, regardless of whether the participants were being paid or not, but it’s always worth checking with users before they come in for the test, rather than putting pressure on them once they’ve made the effort to meet with you.

Another good practice is to decouple the user’s name and their test recording as soon as you can, assigning each one a user number (U1, U2, U3) and keeping their personal details in one centralised location that is only available to a restricted group of people. Then, you’ll be able to share the recording internally without worrying about sharing around people’s personal details.

Of all the tips listed herein, this is probably the most boring and also the most important. So make sure you know the rules and you stick to them. It’s a sin not to.

6. Bigmouth Strikes Again

Keep your big mouth — and your ego — out of it

I’m a huge believer in the idea that to be a truly great designer, you must be able to remove your ego from the process. Ego poisons everything: it clouds judgement, it reduces empathy, and it frames problems within the context of one’s own personal goals and motivations. And yet, design and ego are naturally connected. We feel proud of the work we do, we become emotionally invested in it, and want our designs to be reflections of how we see ourselves as designers.

I particularly like Stuart Butterfield’s quote on the subject:

This is the greatest software development methodology: Don’t think about what you’re doing, have no ego. There’s no speculation.

The connection between ego and design is no more pronounced than in a test scenario. If you’re testing designs that you’ve made, your natural tendency is going to be to seek positive feedback, and to avoid, ignore, and explain away negative feedback.

Here’s some ways to avoid your ego from entering the room:

  • One of the best ways is to simple keep your mouth shut as much as you can. The less you talk during the test, the harder it is to impose your own opinions on the test. If a user is confused, stay silent. If a user has misunderstood something, don’t explain it.

  • When you do have to speak, avoid leading questions. Your questions should be short, direct, and ideally consistent across tests.

  • Answer questions with questions. If a user you something, ask them what they think the answer to that question would be, rather than being tempted to tell them.

I always find it’s helpful to tell people before the test that you may not be able to answer questions during the test. This stops you looking like an asshole when they do ask questions and you refuse to answer.

7. Push It

Sometimes, you’ll need to challenge users

Challenging users seems a direct contradiction to the last tip, but the fact is that no two tests are the same, and depending on the user vastly different approaches are often needed. Differences in personality, cultural background, and perceived psychological safety are just some of the factors that can affect a test. For example, I found that users I tested with in Uganda needed a lot more encouragement to feel comfortable talking about the product than users in the US and UK, who are usually more happy to volunteer their thoughts on the interface.

As such, it’s important for you to remember that as the interviewer, you will sometimes need to push the user to provide more or better information. I’ve noticed three cases in which this especially important:

  • The silent: No matter how much you ask, this user will not think out loud or vocalise their feedback. You’ll need to gently step in and guide them, asking questions and prodding deeper into their actions.

  • The pleaser: This user gives positive feedback that seems almost too good to be true. Some personalities are simply less inclined to criticise than others. I recommend using the Five Whys here to push this user to explain exactly what they like about the experience, and hopefully, this surfaces some things they don’t like, too.

  • The stakeholder: Some users will approach a test as if they were a consultant or stakeholder themselves. Look for phrases like ‘I think people will really find this helpful’, and ‘You could change [element] to help people do [thing]’. In this case, you need to remind the user that you want to know what they feel about the product, not what they think other people will feel.

It may seem awkward, but knowing when to push (and when not to) can be the difference between a successful and failed test.

8. Wanna be Startin’ Somethin’

Get out there and start testing

Look, there’s only so much advice I can give you (and truthfully, only so many 80s songs I can tenuously link to said advice). However, one of the most important things that I’ve learnt over the years is when it comes to testing, no one size fits all. No matter how much you plan and prepare, testing involves people, and the beautiful thing about people is that they are all very different. Anyone who says that there is just one ‘right’ or ‘universal’ way to approach testing is probably lying to you.

The key is to learn to be adaptable in a testing environment, to be able to roll with the punches, and the only way to fully learn this is to be involved in as many tests as you can. Don’t get me wrong, reading is great, and I hope you’ve learnt lots in this article. But no amount of reading can substitute real-life experience.

So get out there! And as Michael Jackson (a well-known usability test lover) once said, you got to be starting something.

Thanks for reading! If you’re interested in UX, remember to follow me on Medium and Twitter (@jamchiller) for more content.