1122 BBEESSTT PPRRAACCTTIICCEESS FFOORR

Successful Remote Unmoderated Usability Testing

2  

Unmoderated Remote Usability Testing (URUT) is a research companion that caters to varied research needs.

3  

Remote Testing is also known for its economy with time and money in comparison to a lab-based study, hence its growth in

popularity and increased adoption in recent years.

4  

At UserZoom we have been conducting remote research for over 6 years. We have compiled 12 key tips that will help you make the most out of the URUT methodology, be efficient, avoid pitfalls and gather as much quality data as possible.

Here we go.

5  

UX & CX Researchers who do not set up their study goals and requirements

at the onset, tend to shuffle from one remote study type to another:

•  Define your study goals well to achieve a tighter study design.

•  Have a hypothesis you would like to test.

•  Each question should have a clear purpose.

•  Have a vision of what your final report will look like.

Determine your tactical & strategic goals first.

1. Define Your Study Goals

Well

6  

Vague and unclear goals lead to study designs that are not well thought out and consequently lead to less than actionable data.

Good To Know

7  

Knowing the goal of your study helps you choose the right study type. Each research approach supports a wide range of study goals:

o  Customer Experience Studies: such as VOC – Voice of the Customer and website feedback surveys.

o  User Experience Studies: such as task-based Usability Tests, Card Sorting, Tree Testing, Screenshot Click Testing and Timeout Testing.

Choose the right approach for the digital touch point you want to test.

2. Select Your Research Approach

Live Websites

Prototypes Mobile Sites

Mobile Apps

8  

If the goal is to restructure the site’s information architecture, then Card Sorting and Tree Testing along with some survey questions might be the best bet.

If you want to compare your site’s performance with competitors, then task-based Usability Testing is the appropriate study type to select.

Good to know

9  

o  JavaScript Code: Needs to be added to the web pages. You need to have access to

updating your site’s code.

•  Browser Add-On: Facilitates tracking participant behaviour + no need to add any code to

the website. It will track navigation events / changes in the URL.

Choose betweent behaviour tracking options.

3. Choose Your

Behaviour Tracking

10  

With Java Script Code, you will have a higher completion rate, but as a UX & CX researcher you rarely have access to the website’s code.

A Browser Add-On is simple to execute, but it may reduce the completion rate.

Good To Know

11  

Invite participants who will help you achieve your study goals.

4. Choose The

Right Recruitment

Method

•  PPaanneell:: A panel provider is a third party vendor that specialises in screening and recruitment of participants.

•  CCuussttoommeerr lliisstt:: If you have a list of customers who have opted in to participate in research or receive communications, these customers can be recruited by email invitations.

•  IInntteerrcceepptt ffrroomm yyoouurr lliivvee wweebbssiittee:: A percentage of visitors from your website will receive an intercept message and be invited to take part of the study.

12  

The biggest benefit of intercepting live users on your webiste is that you can invite actual visitors.

There are substantial savings since there are no recruitment costs and incentives may not be necessary.

The biggest drawback, however, is to be mindful of disturbing the visitor experience. Intercept studies are best kept simple and short.

About Intercept Studies

13  

Clearly communicate with your participants from the start.

5.

Set Clear Expectations

CCrreeaattee aa ppoossiittiivvee ppaarrttiicciippaanntt eexxppeerriieennccee:: Set expectations at the invitation level to prevent surprises & frustration.

Inform participants of:

o  Device requirements.

o  The time commitment to take the study.

o  If a browser add-on installation is required.

o  The number of study tasks.

o  Notify them about their progress through the study.

o  Have a few wrap-up questions before completing the study.

14  

Let your participants know what they need to do in the study.

6.

Create Clear Tasks

•  WWhhiicchh ttaasskk ddoo tthheeyy nneeeedd ttoo ppeerrffoorrmm?? Send participants on short missions with clear instructions on what they are required to accomplish.

•  BBrreeaakk aa llaarrggee ttaasskk ddoowwnn iinnttoo ssmmaalllleerr ccoonnccrreettee oonneess.. It will not only ensure that they participants perform each of the tasks well. You will also receive accurate feedback on follow-up questions.

•  FFoorr BBeenncchhmmaarrkk--SSttuuddiieess -- ccoommppaarree aapppplleess--ttoo--aapppplleess:: if participants are asked to find a phone number for online banking on two banking sites, but are asked to find the address on the third site because the phone number is not available, the parameters are slightly different and not directly comparable.

15  

Define when a participant has successfully completed a task.

7.

Determine The Task Success

•  VVaalliiddaattiioonn QQuueessttiioonn:: To test findability of information, it is best to confirm by a validation question. Participants are then asked to find or remember a relevant piece of information from the website.

•  VVaalliiddaattiioonn bbyy UURRLL:: When tracking behaviour and clicks, validation by URL may be the right option. Success is based on whether participants reach the designated URL(s).

•  SSeellff--rreeppoorrtteedd vvaalliiddaattiioonn:: Users indicate whether they felt they succeeded in completing a task. This method of validation is more subjective or is useful when participants are asked to gather information or review content.

16  

Ask participants to review the content on a

pharmaceutical website:

Does the site adequately explain side effect

information of a drug? Do participants feel like they

are able to gather enough product information to

make an informed purchase decisions?

Self-Reported Validation – An Example

17  

Be succinct and keep the study as fun & interesting as possible

8. Keep

Participants Engaged

•  KKeeeepp iitt sshhoorrtt aanndd ssiimmppllee:: If the task description and study expectation are boring and long-winded, then users are more likely to abandon the task or study before completing it.

•  HHiigghhlliigghhtt kkeeyy ppaarrttss ooff tthhee ttaasskk ddeessccrriippttiioonn oorr tthhee qquueessttiioonn:: the use of bolding, a different color, underlining, italics, bullet points or numbering enables participants to quickly see and understand a question.

•  IInntteerraacctt wwiitthh yyoouurr uusseerrss:: One nice way to keep users engaged is to communicate directly with them during the study and to acknowledge their effort.

18  

The average completion rate for a unmoderated Remote Usability Test is about 75%.

Using fun engaging language and making tasks and questions easy to read and easy to understand will increase the completion rate by at least another 10%.

About URUT Studies

19  

Use quantitative & qualitative question types

9.

Create a good mix

•  UUssee yyoouurr UUXX mmeettrriiccss:: Remote Testing tools capture effectiveness (task completion success) and efficiency metrics (time on task) automatically.

•  AAdddd aa hhuummaann eelleemmeenntt:: IInclude qualitative data such as users´ comments or session videos in your usability report. If it´s all only graphs and statistics it may be Spartan in its tone and tenor.

•  TTeellll aa ssttoorryy:: Ensure that you ask the right number of questions with a mix of quantitative and qualitative question-types. It will help you create a more compelling story in the report.

•  BBee pprreeppaarreedd:: Come up with a thorough question list while designing the study. It will make the analysis much easier.

20  

Have at least one positive and one negative open- ended question in each task to capture what is working well for participants and what is not.

A multiple response question with a list of potential issues encountered is also helpful in drilling down and understanding the problems participants are experiencing on the site.

Good To Know

21  

Be mindful of participant´s time, and yours

10.

Keep An Eye On The Time

•  BBee aawwaarree ooff tthhee ttiimmiinngg:: A Remote Study should not be more than 20 minutes. A 20-minute study usually entails about 5 tasks that take users ~ 4 minutes each to complete. You may also then add a final questionnaire with ~ 10 questions that will take users 2-5 minutes to complete.

•  SSttaayy ffooccuusseedd:: There is much a UX & CX researcher would like to learn for the amount of time and money invested in a project. Splitting up a study into two or more separate projects might be one option to an ambitious research project. Another option is to have participants perform a subset of tasks rather than all of them.

22  

Keep study times short in length to increase the

completion rates. You thereby ensure that the data is

collected in time.

Long studies can lead to participant fatigue

compromising the quality and integrity of the data

captured.

Good To Know

23  

Soft-launch your study to a small sample

11. Preview

Internally

PPrreeppaarree yyoouurr ssttuuddyy llaauunncchh:: •  Preview the study to eliminate problems with the spelling, grammar, and logic.

•  Have another person review the study to help spot grammatical problems and logic issues and to provide feedback on task difficulty and the study length.

•  Conduct a soft-launch or pilot with about 10-20 users, once the study has been thoroughly QAed and tested. This way you identify any problems that may have been missed during the QA and testing process.

•  Evaluate the responses that come in. If problems are spotted in the soft launch and the study needs to be fixed and re-launched, then only a small number of participant data is lost.

24  

Be fair in your incentives. 12. Reward

Participants

LLeett uusseerrss kknnooww yyoouu ccaarree:: •  Incentives are a vital part of usability testing projects and it is important

to incentivise users fairly for their time and effort.

•  One important advantage of Remote Testing is that it is cost-effective. While a lab-based study incents anywhere between 50€-100€ per user, a RUT project incents between 10€ and 20€.

•  Incentives do not necessarily have to be monetary in nature: discount codes, access to exclusive content or information, donation to a charity, or a drawing to win an item are some other possibilities to consider.

25  

The incentive amount is dependent on factors

like the complexity of the participant profile,

study and device type.

A mobile study usually has higher completion

rates when panel providers introduce higher

incentives. For a 15-minute mobile study, you

can expect to provide 12€ – 25€ as incentives.

Good To Know

www.linkedin.com/company/userzoom

@UserZoom_UK

www.userzoom.co.uk

Contact us:

Cheshire (UK) 50a Alderley Road

Wilmslow, Cheshire SK9 1NT Phone: +44 (0) 1625 525 650

Any question?

marketingeurope@userzoom.com