What happens when we can carry out user research and testing remotely? Do we need to change the way we work?
The answer to this question is: “No“.
I recently had a chance of carrying out a research and usability testing consulting for a keyword research service based here in London. This experience led me to many interesting insights that I decided to share with the user experience community at the European conference of the Usability Professional Association in Turin (coming soon next week, 6-8 December).
This talk is about remote user research and testing; my feeling is that this practice is still seen as a ‘poor cousin’ of other user centred design techniques.
Well, remote user research is probably going to be a big change in the user experience. In my view, we still have just started seeing this change happen.
The problem is well know: the difficulty in selecting representative users for user research or testing. This is especially true for those B2B applications that have a very specialised, small but high value target audiences. In this case, it is of very little value of doing research or testing with anybody else. For example, where are you going to find stock market traders or – as in my case – Search Engine Marketing (SEM) specialists?
You won’t find these people on the street, waiting for you to come out and ask them questions. You have to rely on your’s and your client’s network to find them.
Even when you find them, it won’t be easy to convince them coming to your office for a one-hour usability test. This would probably take them half-a-day, and they won’t have that luxury. Even when they have, you are still constrained to users in your area, unless you have time and money to travel to other regions, countries, continents. To circumvent this problem, some usability consultancies, have established solid networks to carry out international usability studies. These help where also differences in language and culture play a big part in the user experience.
But:
- What if in your case differences in language and culture don’t play a big part?
- What if you want to carry out this kind of study or test on a budget?
The case study
In my case, the firm I was consulting for had a strong customer base in US, more than 50%, about 30% in UK and the remainder in other EU countries. Language and culture in this case was not much of a barrier. Budget and distance were.
What we did was using a desktop sharing and conferencing tool facility to carry out remote user research.
Remote user research methods
‘Remote user research and testing’ is where the user and the facilitator are in different places. Remote user research can fall into two categories, ‘synchronous’ and ‘asynchronous’.
- In synchronous protocols, a facilitator interacts with a participant who is remote and leads the research activities in real time.
- In asynchronous protocols, observers do not have access to the participants in real time, and there is no facilitator interacting with them during data collection.
Synchronous methods are enabled by online conferencing tools. Online conferencing tools allow two or more users to share their desktop, open an audio channel, record the video of the sharing session, send files or links via a chat, and much more. Usability practitioners now have a wide choice of online conferencing tools. The most famous are Web-Ex, GoToMeeting and Yugma, but there are many tools offering similar features on the market.
The work plan
Our work plan was split in two stages:
- Remote user research.
In the first stage we carried out semi-structured interviews and user observations. The goal was to inform their design activities by (a) selecting a panel of existing and new customers and (b) discover in which way they currently performed Search Engine Marketing (SEM). - Remote user evaluation .
In the second stage, we carried out the user experience evaluation. The goal was to gather qualitative feedback and inform the design activities.
Agile development was carried out between step 1 and 2.
1. Remote user research
The first stage required setting up the protocol for the semi-structured user interviews and user observations. For remote testing, it is important to introduce the user observation with a semi-structured interview to build a sense of trust and confidence in the participant.
It took some time to define the interview prompts because SEM is a highly-specialised business with its own domain knowledge and language. We tested the interview prompts a few times to allow the facilitator to become confident in these topics.
The semi-structured interview protocol included:
- Standard demographic questions (age, job, work experience)
- Prompts to discover the keyword research work practices beyond the mere usage of the online tool (How important is keyword research to your business? How would you describe the phases to carry it out? And so on)
- Prompts to identify the keyword research tools employed and their frequency of use.
The user observation protocol prompts the participant to use the online tool as normal, talking out loud about their journey into the tool. When required, we prompted the user to verbalise why they took some actions, explain in their own words the results obtained and the technical jargon displayed on the screen. We concluded the observation with a few questions about the perceived quality of user experience and how they would improve the current tool.
2. Remote user evaluation
The user experience evaluation took place 4 weeks after the presentation of the user research findings. The team developed a static mock-up of the revised, personalised homepage and a high-fidelity prototype of the keyword research tool.
The evaluation protocol fulfils two different goals:
(a) gather perceptions and expectations from the homepage
(b) prompt the user to explore the high-fidelity prototype as if they had to carry out an actual keyword research.
Based on the results of the first stage, the aim of the new interface was to be flexible enough to satisfy the needs of the most sophisticated and naïve users (e.g. the SEM professionals vs. the small and independent online retailers).
As planned, the second stage took a total of two weeks. The evaluation findings were presented to the team, who subsequently carried out one additional design iteration.
Be prepared for remote synchronous research
Besides the definition of the protocol, there are other important preparatory activities.
- Invitation to participate.
Writing the invitation to participate and identifying the incentives is important. The invitation email must clearly state the goals of the Customer Experience Program, its benefits and the incentives to participate; customers must be able to choose between a free-subscription to the service and an Amazon online voucher. - Scheduling the session .
After the participant agreed to participate, we scheduled a time slot and forwarded to the participant a confirmation email with the exact timing (in their time zone) and a link to join the session. We found it very useful to include a link to a special ‘troubleshooting’ page on the online conferencing vendor website that allows the user to check in advance if the system configuration (broadband, computer speed, operative system and browser) is adequate for the session. - Troubleshooting .
As expected, we discovered that making the user share the desktop with the facilitator was not always so immediate. While, in most of the cases, setting up the desktop-sharing session generally required just clicking a unique session link in the invitation (and a minute wait for the machine to configure the program), in some cases technical trouble-shooting was required. In some cases it took up 15 minutes to address the technical issues. It was really useful to test the tool with a few people in advance to identify the most common issues and preparing a list of questions about things that could go wrong. For example, we had issues with the security settings at the user side. In most cases, it was sufficient to restart the browser, but in one case we had to abort the session. Carrying out remote user research involves planning at least an additional 10-20% sessions than the desired target figure to make sure there is enough data to obtain representative results. - Test your workstation.
If the research requires audio and video recording, it is important to check if the facilitator’s machine can handle all the tasks at the same time. It is worthwhile to simulate one or two user sessions to avoid losing precious data with the actual users. - Remote observers: pros and cons .
An additional challenge is presented by allowing remote observers. In the same way that a typical usability laboratory setting allows stakeholders to watch from behind the one-way-mirror, online conferencing tools can allow one or more observers. However, it is important to agree with the stakeholders that they will not be allowed to intervene during the session. If they strongly want to participate, you can advise them to use an Internet Chat to send the questions to the facilitator, who can decide how and when to ask the question. Research participants can also see when there are other people attending the online conference, and it is therefore important to introduce them and explain that they will not intervene in the session.
Benefits and drawbacks of remote synchronous research
Remote synchronous research protocols will never substitute contextual inquiry and usability testing in-situ. There is a wealth of information that it is not accessible to the remote observer, such as non-verbal and environmental cues. In addition to that, it is not always easy to manage the interpersonal dynamics of the evaluation situation that must be managed across cultural and linguistic barriers and may require different approaches in different countries [6].
However, in our view, remote user research also presents unique opportunities:
- It doesn’t require a research facility. Remote user research can be carried out with a PC and a broadband connection.
- It decreases travel costs and times, both for the users and the investigators.
- It can be carried out in the actual context of use: users do not have to travel to a research facility and they can use their own computer, browser and plug-ins instead.
- It can involve highly specific, time-poor audience segments that can hardly afford to participate to user research otherwise.
- It allows international user research on a budget. Where no language and cultural barriers come into play, remote user research effectively allows user research across time zones.
- It also complements the agile iterative development processes that require a quick turnaround of research findings into design input.
To a large extent, the methodologies used in remote user research do not differ from traditional ones. This is a big advantage for user experience professionals that can leverage on their previous experience.
