WHITEPAPER CROWDSOURCED USABILITY TESTINGgo.applause.com/.../Crowdsourced-Usability-Testing.pdf ·...

CROWDSOURCED USABILITY TESTING

WHITEPAPER

of 9 www.applause.com

Le Crowdsourcing Pour Les Tests D’utilisabilité

SIX KEY STEPS TO HELP YOU LAUNCH AN INTUITIVE APP WITH THE SUPPORT OF CROWDSOURCED USABILITY TESTING

INTRODUCTIONThe goal of usability testing, simply put, is to make sure that a user can complete the tasks they are expected to complete. Usability testing doesn’t test whether or not the functions of the application, website or connected device work correctly, but rather that a user intuitively understands how to perform these tasks — and how easy or dif f icult it was to do so.

With usability testing, “close enough” won’t cut it . A product may have a superior architecture, a great set of features, good performance, scalability and a number of other positive attributes. However, all of this effort is wasted if the user experience is inadequate. An application, website or connected device that is not user-friendly is just as bad as a buggy version and can lead to diminished revenue,

product abandonment or a total failure. An application with poor usability can also negatively af fect a brand’s reputation. With the short patience of today’s consumers, this is an especially dangerous risk to take – you might only get one chance to make a good impression.

THE OLD WAY IS FLAWEDDespite its importance, companies have historically neglected usability testing. The old way to test usability was expensive, time-consuming and dif f icult. Companies seeking to perform thorough usability testing had no choice but to send the responsibility to an outside user experience company or schedule time to bring an expert in. Because of this, usability testing often felt isolated and disrupted the QA process. On top of that, unnatural testing



environments could skew results – when people know they’re being watched they can get nervous or self-conscious, which might af fect the way they interact with a product.

To avoid these dif f iculties, companies often rely on in-house testing efforts for “usability testing.” This approach is f lawed from the start. Internal employees are too close to the product and already know how the application

works. They lack the ‘fresh sets of eyes’ that external testing provides. On the other hand, friends and family who are recruited as test participants may generate biased results because they don’t want to upset anybody or are concerned about “not giving the right answer.” Plus, neither of these options (internal team or friends and family testing) will necessarily mirror your target audience – an important component of usability testing.

HOW CROWDSOURCING CHANGES THE GAMEThe rise of crowdsourced testing has made usability testing much more accessible and f lexible. Today, an informative round of user testing can be carried out in as lit tle as 10 days, producing results that not only provide benchmarks, but also allow for immediate turnaround and implementation of suggested changes. The shortened time frame and less-disruptive process mean that companies can f it UX testing comfortably into their overall testing efforts. Since users complete the tests in their own environments and are often professional testers who are used to honestly reporting their f indings, the results are more accurate. Best of all, crowdsourcing means usability testing no longer breaks the budget of startups and small businesses – sectors that had historically been priced out of these services.

But not all crowdsourced UX testing services are created equal. For crowdsourced usability testing to be successful, you need to keep a few key things in mind.

If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to check whether the product does what was intended, not whether what was intended was useful. Customers do not care about the distinction.

BRIAN MARICK, SOFTWARE TESTING SPECIALIST

“ “



KEYS TO CROWDSOURCED UX TESTING SUCCESSIf you want to have a successful crowdsourced usability testing experience, take some time to consider these factors. Knowing what you want from the process itself and from the testing results will help you have a successful crowdsourced experience.

PICK THE RIGHT CROWDSOURCING COMPANYThere are several dif ferent crowdsourced testing options that offer dif ferent levels of service. If you pick one blindly you could end up with a disappointing experience and lit tle usable feedback.

Some online crowdsourced UX testing solutions make usability testing seem simple and easy, but in reality rely on people with no testing experience. In these cases you can end up with lit tle more than noise – unhelpful feedback like “ugly,” “frustrating” or “useless.” What exactly is ugly, frustrating or useless? The whole product or just certain features? Why do they feel this way and how can you make it better? You might never know because you might not have follow-up access to these users.

It’s also important to look at the features and services of the crowdsourced company to make sure they are offering everything you want or need. Important features to pay attention to are: bug-tracking system integration options, how results are presented, access to the right demographic of testers, and a level of involvement from your crowdsourcing company that matches internal needs and expectations. If you have someone in-house that can set up, manage and analyze a usability test then you might use a more “hands-off” company. If you want the soup-to-nuts experience, choose a company that offers more comprehensive services.

Both options are available in the market today.

If possible, look into the service from the tester side. How big is the community? Do people seem to like testing for this service and do they keep coming back? You want to make sure you choose a crowdsourcing option that will give you access to good participants. Looking at how a company interacts with their crowd is a good indicator – the more engaged a company is with its crowd, the more likely these users will stay engaged with the company.

Another important factor to selecting a crowdsourced company for usability testing is continued access to the same testers. Some companies simply let testers claim projects and don’t give you much control over the process other than setting demographic parameters. This leaves you in a lurch if you have any questions following a test or would like the same users to test a later version of the application. If you plan on integrating usability testing into your overall QA plan it will be helpful to have more direct access to specif ic participants.

PICK THE RIGHT PARTICIPANTS FOR YOUR TESTAs technology becomes more common place, it becomes harder to test. Gone are the days of a handful of devices, operating systems and versions – we’re now in a world where almost everyone has a device and the hardware/software combinations are almost countless. This means companies have to make sure their applications, websites and connected devices perform correctly and look good on a host of dif ferent devices. Signif icant quality improvement is achieved when testing is done across a wide set of platforms, locations, languages, etc. Crowdsourced testing is the easiest way to cover this terrain. Testing participants can be recruited for each OS, browser, plug-in, mobile



.

Supposing is good, but finding out is better.”

MARK TWAIN

“

handset model, language, location and any other factor that mimics your end user base.

Selecting participants that mirror target user demographics is more crucial to usability testing than to any other testing. If your application is designed for new mothers, a man without children isn’t the ideal tester. You want participants who will use the application like your end users – they’re the ones who will f ind the UX “bugs” that your end users are most likely to notice or be bothered by.

If your product is intended for more general consumption, be sure to include participants that represent all types of consumers. A general sports website should be usable by people who know lit tle about sports and by dedicated fans who can handle more specif ic, nuanced information. If the site is too simple, the hardcore fans will turn elsewhere. Likewise, a site that is complicated and f illed exclusively with sports-jargon will confuse and ultimately drive away casual fans. You’ll only know you’ve struck the right balance if your usability test includes both sets of users – and people in between.

On a hardware level, pay attention to the most popular makes, models, platforms and versions among your target users. If users are more likely to be Chrome or Firefox fans, testing on Internet Explorer is less important. Does a certain mobile phone manufacturer dominate your target location? Employ testers who have models by that maker.

By the time your application reaches the hands of usability testers you should already have your target market identif ied. Use this information to help guide testing.

KNOW WHAT TO LOOK FORA usability test gathers feedback on any aspect of the user’s experience that is troublesome or could use improvement. A complete user experience covers many aspects, including:

• Functionality: A product or application must present the user with the appropriate functionality. If the functionality of



an application is perceived as incomplete or inadequate, customers may be lost.

• Layout & Design: Good layout and design allow a user to easily complete tasks. If a button is placed in the perceived ‘wrong’ place, users will get frustrated. If text color is too hard to read, the application will be unusable.

• Interaction: The f low of an application must be natural and allow the user to easily complete tasks. If a user believes they have been sent to the wrong page and have to manually backtrack in order to f ind the page for which they were looking, they can become frustrated and abandon the application.

• Language: The language used on all features of the application should be clear and understandable. Overly complicated language will confuse users and leave them feeling frustrated. Make sure all translations are correct and make sense to native speakers.

More often than not, technical or inappropriate labels and terminology make it into a consumer application. Usability testers will tell you that ‘Discounts’ sounds better than ‘Adjustments’ or they might let you know that your information section is too verbose or not clear enough.

If you expect to roll out an application, website or connected device to non-native speakers, be sure to test using native or f luent speakers. Translations are not always accurate; even if they are accurate they might not make sense colloquially. These are errors your users will notice instantly (and likely be very turned off by), but a non-native tester will likely miss. If you plan to roll out the product in multiple languages, use testers to provide extra edits and insight on the translations.

Even an hour of usability testing will likely not cover

every single possible action on your application. Start by testing those areas of the application that will be used the most or are the most important to your customers. Remember, you can run as many usability tests as you’d like (with crowdsourced testing you can even run multiple tests at once with dif ferent groups), so don’t try to cram everything into one test, it will muddy the waters and result in a less successful usability test.

KNOW WHAT KIND OF FEEDBACK YOU WANTWhile you can typically turn around a usability test quickly with crowdsourcing, you’ll get better results if you take time to understand what you’re looking for and create a good task plan to get those results. Decide if you want yes/no answers or more insightful feedback from open-ended questions. Consider which aspects of usability you want covered in a particular test and write the task plan accordingly. When creating a task plan, be thorough, but be careful not to inf luence the way testers will interact with or feel about the application.

Let the kind of feedback you want to receive inf luence how large the test is. Inviting too many participants to an open-question survey will give you a f lood of information to process. If you’re looking for input from a lot of people, stick with yes or no questions. If you want more detailed feedback, keep the testing group smaller but give them the opportunity to really explain what they’re experiencing and how they feel.

If you don’t know what type of feedback would be most useful, are concerned about pinpointing actionable data in the test results or are unsure about how best to write a comprehensive task plan, choose a crowdsourcing company that also gives you access to a professional Usability Expert.



DON’T FORSAKE EXPERTSJust because you’re using crowdsourcing doesn’t mean you have to sacrif ice the knowledge of an expert.

A UX expert is a user experience professional; someone who defines the usability testing tasks and questions. They can also help select the right cross section of testers. Once all participants have completed their tasks and answered all questions, the UX expert analyzes the data and writes a results report. The expert also investigates the interface themself for any usability issues based on best practices and industry standards.

If your company doesn’t have a resident UX professional, the expert’s background and experience in user experience and writing of tasks and questions can be extremely beneficial. Even companies that do have an in-house expert or team can benefit from the help of an outside expert who can look at the interface in an unbiased way.

Usability testing is far more subjective than functional or security testing. Rather than something working or not working, usability testing is about a user’s understanding and feelings about an application. Results based on longform, writ ten

feedback can be hard to process, understand and evaluate. Even more dif f icult, there’s not always a clear cut f ix to the “problem.” A UX expert can help separate the fringe opinions from the feedback that really matters and help you understand how to address any issues that come up.

If you want the help of a UX expert, be sure you choose a crowdsourced company that offers this service – a more “self-service” company will leave you to do everything yourself.

DON’T RUSH ITMany people think about running a quick test cycle before releasing or launching their application, website or connected device. Successful companies recognize that usability is a long game that needs continued attention. After all, a user isn’t a new user forever.

Crowdsourced usability testing gives you the chance to run longer tests without taking time away from internal resources. Longer tests are helpful because they allow you to see how new users change the way they interact with an application over time. There could be features or aspects of an application



Systems and software should work for us, not make us feel confused, stupid or frustrated.”

JONATHAN KOHL, SOFTWARE TESTING CONSULTANT

“

that only recurring or long-time users will need. Running a longer test will help you accurately gauge the usability of these more advanced features and see how easily users f igure them out.

Usability test participants aren’t sit ting in a lab anymore, they’re working on their own time in their own environments – there’s no rush! With the right crowdsourced company you can also have the same users come back to test future versions or updates – which will give you insight into how your real customers might feel about the upcoming changes.

Crowdsourced usability testing gives you the opportunity to run as many UX tests simulating as many situations over as long a period as you want. Your real users will be interacting with the application for more than two weeks – your usability testing shouldn’t stop there either.

CONCLUSIONYou only get one chance to make a first impression and a good first impression of your product is crucial to keeping customers engaged and happy. If users can’t f igure out your application, website or connected device quickly, they’ll leave quickly – and likely won’t be back. According to New Relic, 1 in 4 mobile apps are abandoned after the initial use. When asked how to best keep users engaged, Robert Gary, Vice President and General Manager of Mobile Care at Nuance, said, “The app must simply be easy to use and avoid frustrating customers.” It all comes down to usability.

Take advantage of crowdsourced usability testing and work out the kinks (with feedback from a group of people who mimic your real user base) before your application hits the hands of any actual users. Then continue testing to be sure your product will continue to delight them.

In the end, an application, website or connected device has to be usable by your users. It doesn’t matter if you, your developers and your mom find it usable, if the people in your target market are confused, don’t like it or find it frustrating or useless your product is a failure. Crowdsourced usability testing is uniquely positioned to help you avoid that.

NORTH AMERICA 100 Pennsylvania Avenue Framingham, MA 01701 1.844.300.2777

EUROPE Obentrautstr. 72 10963 Berlin, Germany +49.30.57700400

ISRAEL 10 HaMenofim Street Herzliya, Israel 4672561 +972.74.757.1300

ABOUT APPLAUSE

Applause is the worldwide leader in crowd-sourced digital quality testing. Software is at the heart of how all brands engage users, and digital experiences must work flawlessly everywhere. With 300,000+ testers available on-demand around the globe, Applause provides brands with a full suite of testing and feedback capabilities. This approach drastically improves testing coverage, eliminates the limitations of offshoring and traditional QA labs, and speeds time-to-market for websites, mobile apps, IoT, and in-store experiences.

Thousands of companies of every size – including Google, FOX, Nike, BMW, PayPal and Slack – rely on Applause to dramatically decrease decrease both the time and the costs of delivering great digital experiences for their customers.

Learn more at www.applause.com


WHITEPAPER CROWDSOURCED USABILITY TESTINGgo.applause.com/.../Crowdsourced-Usability-Testing.pdf ·...

Documents

Transcript of WHITEPAPER CROWDSOURCED USABILITY TESTINGgo.applause.com/.../Crowdsourced-Usability-Testing.pdf ·...