When a face-to-face chat isn’t best

Might remote user-testing actually be better?

It’s generally not a good sign if someone bursts into tears during a user test.

In fairness, it wasn’t the tester who was crying but their child, a three-year-old who had been playing in the background.

I was on a video chat with the boy’s mother, a teacher, who was in her kitchen at home while her toddler amused himself in front of the TV. When her boy began crying she expertly swept him up, calmed him down in seconds, and insisted on carrying on the user test. She was used to working one-handed like this on her laptop in the evenings, coping with the different domestic distractions, including the noise of CBeebies characters I could hear burbling in the background.

As user tests go, it was hardly in laboratory conditions. Indeed it directly broke some of the suggested rules for carrying out product user tests. Yet, as with every test I’ve done recently via video chat, I felt I got some better insights than I might have had with other approaches.

I’m a product director at Tes, known to educators internationally for hosting the world’s largest collection of lesson materials made by teachers, for teachers. We have always put a major emphasis on user testing, both the kind we carry out ourselves and by working with organisations such as YouGov.

Our product team still goes out to schools, rings up teachers, chats with them daily, and regularly invites education professionals into our offices. But since last autumn we have also been doing user tests with teachers in their homes or classrooms via video chat every Thursday.

In Ash Maurya’s excellent Running Lean, his recommendations on user testing include that interviewers should:

• Aim to meet the customer face-to-face
• Pick a neutral location
• Avoid recording interviews

But our video user tests fail on all three counts. We have not literally been face-to-face with our users, so have probably not been able to detect subtle shifts in their body language. Neither have our locations been neutral — the teachers have usually been in their homes or in their classrooms while we have been in our office. And we have absolutely recorded the interviews, with the user’s permission, to play back their comments and screen capture how they have used our prototypes.

In Jake Knapp’s Sprint (a book we’ve been referring to a lot here recently) he also stresses that while it is fine for other team members to watch a test remotely via video, “what does matter is that the Interviewer and the customer are sitting side by side”.

There are indeed benefits of being in the same room that the video experience can’t replicate. But I think there may be some benefits of wholly-remote testing that are underappreciated.

1. Broader geographical reach

Back when I used to work as a journalist I was deeply aware of the risk of writing articles that might be London-centric, so took care to ensure I rang contacts across the country. But if you are a business trying to arrange face-to-face user testing near your office you will invariably end up speaking to a disproportionate number of users who are in easy travelling distance.

Yes, businesses can arrange testing anywhere, and it is important for everyone to get out of the office and meet a broad range of users. But if testing is to happen every single week, and involve as many of the team as possible, a roadshow approach isn’t really practical or affordable. This is especially the case if your audience is increasingly global, as ours is, and you want to talk to teachers in rural Yorkshire, central Doha and suburban Phoenix in the same week.

2. Reliance on user’s own devices

When you’re showing a user a digital prototype in your office or a fancy testing lab, chances are it’ll be on a decent computer, with an up-to-date browser and a strong wi-fi connection. When you see them testing at home (or in their workplace) on their own device, you’ll come across problems you may not have anticipated. These can include predictable-but-easy-to-forget matters such as them using a different screen size to the one you and your designers have been using. And then there are more specific problems, which for us have included the vagaries of schools’ firewalls.

Conversely, users can find prototypes easier to explore if they are trying them on their own devices instead of a laptop they have never touched before. During a recent face-to-face user test the teacher and I realised that what was really tripping him up was the way I’d set up my Macbook’s trackpad. On his own PC with a mouse he would have been fine.

3. The enforced “hands off”

I don’t think I have ever committed the cardinal sin of grabbing the mouse off a tester to show them how they should use a system. But any temptation to leap in and take over is removed when you can only give guidance through your voice. Your helplessness to intervene directly can also make you even more aware of the problems when they get stuck using a prototype.

4. Removal of body language cues

Part of the reason for doing user testing face-to-face is that you get to pick up on subtle signals from the user. But they can also pick up on the signals from you. I stress to the user every session that they shouldn’t worry about offending anyone by being negative and that actually critical comments are often the most helpful (I also intentionally mislead them by implying I’m not connected to the team that built the product). However, in a recent face-to-face user test I noticed that a user seemed more eager to praise the project than those I’d been chatting to by video — and it was my fault. Sitting next to him, I’d failed to conceal my delight when he rocketed through the system, using it exactly as we’d hoped. In the video tests I had found it easier to appear detached.

5. Visualising your user’s setting

Yes, it’s nosy. But getting a direct peek into our users’ homes and school does make it easier to visualise where they will use our products — the desk in their classroom where they will do a bit of extra lesson planning in a free period (and still get interrupted by a student) or the sofa where they will be half-watching TV while browsing jobs and news. It is a variant on the old technique of waiting in the store for a user who will buy your product and agree to let you follow them home to see how they use it. But, hopefully, a tad less creepy.

My colleague Tina Akinmade, who does the hard work at Tes finding teachers for our user tests, adds that she has found it significantly easier to get willing participants when they could join from home instead of travelling to meet us.

So while many things in life are better done face-to-face — especially involving relationships — perhaps remote testing really is better.

Let me know other benefits I have missed, or reasons face-to-face is still better, in the comments below.