Involved Parties

Personal Details

Name: Martijn VerschuurenEducation: ICT & Media DesignUniversity: Fontys University of Applied ScienceCity: Eindhoven, The NetherlandsStudent Number: 2055808

Company Details

Name: Betterplace.org Address: Schlesische Straße 26 Postal Code: D-10997 BerlinPhone: +49 30 7676 4488 44Fax: +49 30 7676 4488 40Email: change@betterplace.orgInternet: www.betterplace.org

Contact person Betterplace: Niklas SumSkype: tehniklasE-Mail: Nsu@betterplace.org

Graduation Teacher Dhr. L. Maassen

Second Graduation Teacher Dhr. F. van Alebeek

Foreword

This research has been written to graduate at the studies ICT & Media Design at the Fon-tys University of Applied Science in Eindhoven. For this research I have done a twenty week graduation internship at Betterplace.org Berlin, Germany.

During this research I searched for tools to monitor and improve the User Experience for the Betterplace website. Next to my research, I improved parts of the website, added new content and worked out concepts to keep the visitors satisfied.

I would like to thank the people at Betterplace for the opportunity to graduate. In particular I would like to thank Niklas Sum for guiding me during this internship and my colleagues from development, with whom I worked together on daily bases.

Martijn Verschuuren

3

Explaining words list 5 Problem analysis 6 Aim of research 6 Research questions 7 Research methodology 8

Chapter 1: About User Experience What is User Experience? 10 UX Honeycomb 14 Why is UX important? 15 Example of bad UX 16 Why create a UX process? 18

Chapter 2: Tools & Techniques A/B split testing 21 Multivariate testing 27 Heat maps 33 Personas 36 Heuristic Evaluation 38 Low Budget Testing 39 Other Companies 44 Conclusion 51

Chapter 3: Advice & Recommendations Advice & Recommendations 54

Chapter 4: Appendix

Index

4

Explaining Words List

Bounce Rate Visitors who enter the site leave rather than continue viewing other pages within the same site

Conversion Rate Goals divided by number of visitors

Core Content Content that is impractical for competitors to replicate

Moderated Testing UX testing with live view of the users screen with the possibility to interact

MVT Multivariate Test(ing)

Pros & Cons Arguments for (pro) or against (con) a particular issue

Remote Based Testing Testing where users complete pre-defined tasks from their home

Response Time The time a user takes to react to a given input

UX User Experience

Unmoderated testing UX testing without possibility of interaction with users

URL Uniform Resource Locator - Link to webpage of file

WYSIWYG What You See Is What You Get

5

The ever changing expectations from users of the world wide web ask for continuous im-provement of websites today. Betterplace.org, as one of the biggest donation websites in Germany, is under continuous development and is always evolving to meet the changing demands of their visitors.

To keep up with the competition, Betterplace is constantly improving the usability and user experience of the website. This is achieved by incremental improvements in real-time. This way, the website is never finished and maintains up to date with the latest Internet trends.

The problem is that there is no process concerning user experience involved in the crea-tion or improvement of web pages. The product management and development teams have an eye for usability and user experience, but there’s no process if a page shows a low conversion rate.

Problem Analysis

Aim of Research

Betteplace.org is depending on the donations of visitors. In order to keep these visitors satisfied, it’s critical to give them the best user experience while visiting the website. Therefore the objective of this research is to investigate which tools Betterplace can use to improve the user experience and thus improve the bond with their visitors.

6

Research Questions

The main research question in this report is:

Which User Experience tools and techniques are relevant to improve the User Ex-perience process at Betterplace.org?

My research will exist out of three parts:

Part 1: User Experience Introduction

What is User Experience?Why is User Experience important to a company?Why have a UX Process?

Part 2: Overview of tools and techniques

Which tools and techniques are popular? Which tools and techniques are other companies using?Which tools and techniques are relevant for Betterplace?

Part 3: Advice & Recommendations

7

The following research methods have been chosen for the particular research which needs to be done. The mixture of the research methods will lead to the answers of the sub questions as well as the main question: Which User Experience tools and techniques are relevant to improve the User Experience process at Betterplace.org?

Desk research

In order to answer tvhe questions concerning, ”what is User Experience?”, “why is User Experience important for a company?”, and “why create a UX process?” I will first and foremost use secondary data.

Qualitative research

Qualitative research will take place in the form of meetings with my supervisor and the company’s development department. During these meetings I will collect data concerning Betterplace’ mission statement and vision and get an insight of the tools and techniques valuable to the company.

Quantitative research

The quantitative research in this document will be the testing of tools and techniques. During this part of my research I will compare price, compatibility, features and usability. The outcome of this research will give me an answer to the question “Which tools and techniques are relevant for Betterplace?”.

Research Methodology

8

It’s pretty hard to get a good definition of User Experience. User Experience is basically a sum of series of interactions that deal with the way a person feels about using a product, system or service. It represents the perception left in someone’s mind following a series of interactions between people, devices, and events.

In this day and age, User Experience has become a big deal in our daily lives, but only few people recognize the benefits of good UX. People only tend to see bad UX and consider it as unavoidable. Everyone has had encounters with bad UX: an MP3 player that doesn’t play music in a way that makes sense to you, an online form that’s confusing and takes more than one try to fill out or even a mobile phone that has so many features, making a phone call becomes a chore. These issues could have been avoided by testing the UX, instead of creating a product the way the designer would like it. Users notice these issues, but seldom take action to make a change. In their opinion, every device, system or ser-vice is bound to have a few flaws and you’ll just have to live with it. They will be left with a bad experience, which leads to a negative association with the company from which the product came. Ultimately, if these bad experiences keep up, they will stop using one company’s products and join the competition.

Exemplary user experience requires to meet the exact needs of the customer, without fuss or bother. Next comes simplicity and elegance which make a product of service a joy to use and own. True user experience goes beyond giving the costumers what they say they want, or providing a checklist of features. In order to achieve the ultimate User Experience in a company’s offerings, there must be a seamless merging of the services of multiple disciplines, including engineering, marketing, graphical and industrial design and interface design.

User Experience

What is User Experience (UX)?

10

Outdated design process

11

User Experience isn’t part of a process. It is a process itself which needs to be integrated in everything you do. In order to create a good experience for your users, you can’t just design something people would like to use. Instead, it is an ongoing effort, a process of continually learning about users, responding to their behaviours, and evolving the product or service. User Experience is like fashion, in the way that it’s never done. It’s dynamic. As time goes by, circumstances change and you will have to iterate. It doesn’t have to be a rigid process, but it does need to exist.

The term User Experience was originally used in reference to human-computer interac-tions and is still largely associated with those disciplines. But nowadays the term is used to refer to any specific human-design interaction, ranging from a digital device, to a sales process, to an entire conference. Perhaps due to its organic development and lack of formalization, User Experience may be defined by very different departments from or-ganization to organization. In some organizations, it is owned by marketing, in others it falls under IT. Then, from a solutions perspective, some organizations base their User Experiences around the research and academic-based approaches of human-computer interaction, others treat interface and product design as the source for User Experience, while still others let marketing or IT drive it.

A lot of companies still think they can do without User Experience Design and see good experience design as an extra, not a requirement. The biggest misconception these com-panies make is thinking they have a choice to invest in their user’s experience. They might be doing well now, but as time goes by, they won’t be able to survive.

User Experience

12

13

UX Honeycomb

Many people confuse User Experience with Usability. Although usability is an important part, UX is so much more. Peter Morville’s UX Honeycomb explains this best.

UX Honeycomb

Useful

As practitioners, we can’t be content to paint within the lines drawn by managers. We must have the courage and creativity to ask whether our products and systems are useful, and to apply our deep knowl-edge of craft and medium to define inno-vative solutions that are more useful.

Usable

Ease of use remains vital, and yet the interface-centered methods and perspec-tives of human-computer interaction do not address all dimensions of web design. In short, usability is necessary but not suf-ficient.

Desirable

Our quest for efficiency must be tempered by an appreciation for the power and val-ue of image, identity, brand, and other el-ements of emotional design.

Findable

We must strive to design navigable web sites and locatable objects, so users can find what they need.

Accessible

Just as our buildings have elevators and ramps, our web sites should be accessible to people with disabilities (more than 10% of the population). Today, it’s good busi-ness and the ethical thing to do. Eventu-ally, it will become the law.

Valuable

Our sites must deliver value to our spon-sors. For non-profits, the user experience must advance the mission. With for-prof-its, it must contribute to the bottom line and improve customer satisfaction.

Credible

Thanks to the Web Credibility Project, we’re beginning to understand the design elements that influence whether users trust and believe what we tell them.

14

Over the last decades, a product’s effectiveness, intuitiveness, and ease-of-use have become tremendously important competitive differentiators. Modern customers have be-come more likely to switch to the competition as more companies focus on user experi-ence and as it is becoming easier to find and switch to the competition’s products. There-fore, a modern business will find it ever more difficult to compete without optimizing the user experience of its products or services.

User experience practitioners can help to give products a competitive edge by incorporat-ing features that will most effectively meet customers’ needs and by making them easy and enjoyable to use. For certain products such as web sites, user experience designers can formulate a strategy and design that will increase key user interactions such as online purchases, viral action, and page views.

Like product managers, user experience professionals generate product concepts and strategy, screen product ideas, develop and design the products, and perform beta and market testing. However, user experience professionals perform all these tasks from a usability engineering and user-centered-design approach while product managers look at the same tasks from a business and marketing angle. That is why user experience de-signers should work closely with product management to help define, design, and refine the final product.

Why is UX important to a company?

15

A while ago, I was in a store to buy some new headphones. I found a set of Philips headphones that where in my price range and the cool thing was that you could change the print on the ear caps, or even design them yourself. The set included 4 pieces of pre printed cardboard designs, so people could change the look of the caps whenever they wanted. It also offered the option to design your own ear caps, and as a designer I couldn’t wait to print my first caps.

As I opened the box, I was looking for the measurements for custom made prints. Philips forgot to supply any information about printing your own headphone caps, so I went to the Philips website to look for a template or something similar. The website recommend I’d register my product first for maximum service. So I did.

The first thing I needed was a product registration number, which, according to the web-site, was on the back of the box. After examining the back of the box three times, I gave up and chose to look for the product manually. I later found out the code was on top of the box instead of the back.

I clicked ‘browse for product’ in the registration menu and the next page showed me sev-eral different categories. Now I had to guess under which category my headphones were stored. I first checked ‘Sound and vision’, then ‘Other products’ and finally ‘Accessories’. Here, I found over 350 different headphones, but I couldn’t find my model.

Bad UX

Example of bad UX

16

After some setbacks I managed to find my model and proceeded to the customer service, only to find out they had no template or whatsoever for my caps. I ended up scanning my default caps and editing them into my own “cap templates”. After some online research I found out I wasn’t the only one who had this problem.

For me this was a bad user experience with a Philips product (headset) and customer support (website). This could have been avoided if someone from Philips would have thought ahead. When your product says ‘Design your own headset!’ on the cover of the box, you might as well make sure you supply your customers with some information. A small book containing information or a link to a website would have provided.

This doesn’t mean I won’t buy any more products from Philips, but it did damage the brand for me. In my opinion, this was a onetime incident, but it could have easily been avoided. If these kind of flaws keep coming in the future, I’ll eventually stop buying products from Philips and buy them from the competition.

17

In the past, designers mainly focused on building products that fulfilled the functional requirements of the user. User experience was often considered late in the development process. But today the customer demands more than just a working product. Providing the right features is still the prerequisite for a good product, but to turn it into something extraordinary you need to provide a good user experience.

Providing a rich user experience is not a thing of fortune. It needs to be planed, designed and integrated into the development of a product. Designing a rich user experience is not only about make up your user interface by some graphics and gradients - it’s a much broader concept. It’s about creating an emotional connection between the user and your product. It makes the user feel good and so he likes to continue using the products.

This is where a UX process comes in to play. Involving a UX process from the beginning of a project helps build products people want and need. You’ll create a product that’s a good fit for the people who end up using it, instead of for the developer who built it or the CEO who envisioned it.This is particularly important if your users also spend their hard earned dollars to buy your product.

Adding a UX process to the production process might seem like a time and money con-suming business, but it will pay out in the long run. Your team will save valuable time and resources by getting it right, or mostly right, the first time. And they’ll be faster doing it.

UX Process

Why create a UX process?

18

In this chapter, I’ll be testing tools and techniques used to measure and improve UX. For this research, I’ve tested a lot of different tools, but I will only display the top tools that are valuable to Betterplace.org. During my search for tools, I’ve focused on unmoderated testing tools, because betterplace.org doesn’t have the time, nor the budget for moder-ated testing.

Starting my search for the best UX tools, I’ve split up my selection of tools into two groups: quantity- and quality based tools. Quantity based tools are designed to collect data from a large number of participants, with a focus on UX metrics such as task success, task completion times, click paths, abandonment rate, etc. These are the tools you need to run a benchmark usability study or compare subtle design treatments.

The other group consists of Quality based tools. These tools are essentially a substitute for lab or remote-based usability testing. These tools emphasize collecting feedback from a small number of end users in a quick and dirty fashion. You may or may not be able to actually derive any metrics, but at the very least you will gain some insight into the most significant usability issues and hopefully make the right design decisions.

Note: The tools are changing everyday in terms of functionality and pricing structure.

Tools & Techniques

Market leading tools and techniques

20

Also known as Split testing, A/B/n testing, bucket testing

The basics behind A/B testing are very simple: you have an A and a B version of a website or element and test them against each other to find out which one is more successful. You split up a group of users and choose a metric which defines the success of the experi-ment. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.

What to test is entirely up to you. It depends on the goals you set for your experiment. If your goal is to test a form, you might test the length of the form, experiment with taxonomy or the display of privacy. The goal in this case is to figure out what prevents visitors from finishing the form. Is the form to long? Is the placement of the fields confusing? Are people concerned about their privacy? By testing different elements of your page, you will get an insight of the elements that prevent your visitors from finishing the form.

The duration of the test is also entirely up to you. You can set a period of time or a number of visitors, but if you make some radical changes to your page, you might also get some radical results from your tests. If you set your test to 1000 unique visitors and after 400 visitors you see great differences between your A and B site, you might reconsider the setup of your test.

A/B Split Testing

21

There are two ways to set up an A/B test:

- Replace the element to be tested before the page loads: If you are testing a single element on a Web page, you’ll need to create variations of that element in your testing tool. When the test is live, the A/B tool will randomly replace the original element on the page with one of the variations before displaying the page to the visitor.

- Redirect to another page: If you want to A/B test an entire page, you’ll need to cre-ate and upload a new page on your website. When the test runs, your tool will redirect some visitors to one of your alternate URLs.

Every A/B test is unique and has a different goal, but there are some elements that are frequently tested:

Headline or product description Images on landing and products pages Call to action buttons Layout and style of website Form length and types of field Product Pricing and promotional offers Amount of text on the page

Once you’ve decided what to test, the next step is to select a tool for the job. I have tested a lot of different A/B split tools during my research. The results of this research can be found in the appendix of this research.

A/B Testing

22

Example of an A/B test

Version B is better

23

- Always test both (or more) versions simultaneously. Split traffic equally between these versions and test them for the same amount of time. If you test one version one month and the other version the next, your results may be false.

- Don’t confuse your regular visitors. If you are testing a basic page of your website, only include the new visitors in the test. If your new version turns out to be better, you can in-clude your old visitors afterwards.

- Know how long to run a test. If you give up too early, your results may not be reliable. If you give up too late, you will have a poorly running version, which can cost you conver-sions. To know how long to run a test, see page 32 of this report.

- Show repeat visitors the same variations. If you change your website with every visit, your visitors will be confused and are less likely to stay. Most tools have a mechanism to show one version to each unique visitor.

- Make sure you use your tests across the whole website. If you’re testing a sign-up but-ton that appears in multiple locations, make sure your visitors see the same button eve-rywhere.

- Don’t stop after one or two bad results. Chances are your first tests will return negative results. The key to optimizing conversion rates is to do lots of A/B tests, so that all positive results add up to a huge boost to your goals.

Tips

Tips

24

Bronze $ 19,-/month (2000 viewers/month)Silver $ 79,-/month (20000 viewers/month)Gold $ 399,-/month (200000 viewers/month)

Popular A/B test tools

Optimizely.com is a simple, fast and powerful AB-testing tool. It’s easy to use interface and WYSIWYG editor are great to test differ-ent lay-outs of your webpages.

Free Trial 1 monthSmall Business $ 49,-/month (10000 viewers/month)Small Agencies $ 129,-/month (30000 viewers/month)Large Agencies $ 249,-/month (100000 viewers/month)Enterprises $ 729,-/month (300000 viewers/month)

Although Google Web Optimizer (GWO) is free and widely used, I wouldn’t recommend it. For starters, it doesn’t offer a WYSIWYG- editor, which isn’t necessary, but it will save you time in the long run and can be used by colleagues with minimal programming experience. Also, the coding used in GWO has to be removed afterwards, while the other programs only use one simple code snippet to be added to your website.

Visual Website Obtimizer is much like Optimizely, but the WYSI-WYG-editor doesn’t work well with the Betterplace website.

25

During my research, I compared a lot of different A/B testing tools. I compared price, fea-tures, speed and usability. The results from this research can be found in the appendix of this document. The tools on the left scored the highest results.

Optimizely seems to be the best A/B split tool for Betterplace.org. The WYSIWYG-editor works great with the lay-out and the price fits well within our budget. It’s also easy to im-plement: just copy one line of javascript and paste it on top of your header and your ready to start testing.

The user interface is very easy to use and contains the many features Optimizely has to offer. As said before, you won’t have to edit any code while experimenting, so people with-out any HTML or PHP knowledge can still create experiments. For those who do know their programming, Optimizely offers some advanced options.

Optimizely also offers multivariate test (MVT) options, which I will cover later during my research.

Free Account

After I concluded Optimizely is the best A/B tools for Betterplace.org, I send an email to the Optimizely developers. I informed them about our company and asked them for a discount, due to our low budget. They offered us a free account. Therefore I would like to thank the developers and everyone involved at Optimizely.

Popular Tools

Best A/B tool

26

Also known as Multi-variable testing, MVT

Over the years our attention span for websites has decreased dramatically. According to a research from the BBC, the window to grab a visitor’s attention is about 9 seconds. So, how can you gain the interest of your visitors? What headline will you use? How much text should I use on the landing page? For these questions and more, you have Multivariate testing.

Taking a first glance at Multivariate testing, it looks a lot like A/B testing, but there are some crucial differences. In a Multivariate test, a website can be seen as a combination of elements that can affect a conversion rate. These elements can be headlines, images, buttons, texts, etcetera. During a test, you create different variations for these elements and test combinations of them.

Every test has a goal against which the performance of different variations is measured. Types of goals may be sign-ups, purchases, clicks, leads, page views, or bounce rate. It is important to define the goal which is closest to your business objectives. For example, an eCommerce store optimizing for sales shouldn’t define clicking on the “Add to Basket” button as a goal. Rather, it should define the goal as a visit to the “Thank you” page after a purchase is completed.

Multivariate Testing

27

Here’s an example of a MVT. Let’s assume you make the following variations:

Two Headlines Two texts Two images

You will be able to make the following combinations:

1. Headline 1 + Text 1 + Image 1 5. Headline 2 + Text 1 + Image 1 2. Headline 1 + Text 1 + Image 2 6. Headline 2 + Text 1 + Image 2 3. Headline 1 + Text 2 + Image 1 7. Headline 2 + Text 2 + Image 1 4. Headline 1 + Text 2 + Image 2 8. Headline 2 + Text 2 + Image 2

Just like in A/B testing, you split traffic equally between these different versions of the page and see which combination produces the highest conversion rate. Where an A/B test usually has a couple of variations, MVT can produce as many different variations as you want. The example above has eight different variation (2*2*2), but if you, for example, add two different buttons, you’ll have sixteen different variations (2*2*2*2). t

This way the number of variations can go up pretty quick. Therefore you’ll need more traf-fic to get meaningful results, which your website might not have. A good rule is to limit the total number of combinations to twenty five or fewer.

Multivariate Testing

28

Multivariate Testing

29

Full factorial testing

This type is the kind people generally refer to when they talk about MVT. The traffic is dis-tributed equally among all combinations. If there are sixteen variations, each one will re-ceive one-sixteenth of the website traffic. This way the test provides all of the data needed to determine which variation performed the best. A test might reveal a certain image had no effect on the conversion rate, while a specific headline was very influential. Full facto-rial method makes no assumptions with regard to statistics or the mathematics of testing.

Partial or fractional factorial testing

As the name suggests, partial testing only tests a fraction of variations on your website. For example, if you have sixteen variations, website traffic is only split among eight of those. For the other eight, we don’t get any conversion data. Instead, partial testing uses complex mathematics to determine the outcome of those pages. This way, fewer traffic is required, but I wouldn’t recommend it. No matter how advanced the mathematics are, hard data is always better than inference.

Taguchi testing

Taguchi testing is another way to get results with fewer traffic. It was originally used in the manufacturing industry, where specific assumptions were made in order to decrease the number of combinations needing to be tested for quality assurance and other experi-ments. In my opinion, these assumptions are not applicable to online testing. Therefore, Taguchi testing is not a theoretically sound method.

Types of MVT

Types of MVT

30

A lot of people ask; should I run A/B test or Multivariate test? The short answer is yes. You should use both methods to test your website, but you should know how to use both of them together.

Setting up new page

Whenever you test a new webpage, start with A/B testing of the major elements. Run these tests early on in the design process. This way you can implement some radical changes and you can follow their impact. Use A/B tests to determine the best layout con-figuration of elements.

After you have tested the basics of the page, start multivariate testing. Multivariate testing usually takes a lot more time, due to the higher number of variations. Use MVT to tweak the page and optimize it to get the highest conversion rate.

Experts usually run eight to ten A/B tests before they run a MVT.

Improving existing page

If you have an improved version of an existing webpage, always test your new version versus the old. Chances may be you new page is worse than the old one and you’ll be doing even worse. If your new page checks out to be an improvement, you should repeat the process above.

A/B or Multivariate test?

31

It’s hard to determine how long to run a test. As tester, you probably know the situation better than anyone and you can make a pretty good estimation, but here are some thumb rules to consider while testing.

Time

Run a test for at least ten days and include two weekends to level out any day-of-week seasonality. Fluctuations in conversions can occur depending on the day of the week. It’s also recommended to re-run a test a different time, to understand any long term seasonal-ity effects.

Number of visitors

Even if you have a clear winner after twenty conversions, wait until you have at least fifty conversions to make any decisions. Chances may be you’ve been lucky and the rest of your visitors may not react the same.

Statistical difference

Run experiments until a statistical difference at the 95% confidence level is achieved. Sta-tistical difference can be measured online or with a excel sheet which can be downloaded at tinyurl.com/statdif.

A/B or MVT

How long should I test?

32

Heatmaps are graphical overlays of your website which point out what content is hot and what not. This is mainly done by tracking the mouse clicks of the visitor. Heatmaps pro-vide you an insight for sensitive areas on your website which you can use to provide better placement to core content or promote product or information you are willing to highlight on your website to increase online sales, etc.

For example, the heatmap of a webpage shows you an advertisement is easily over-looked and therefore less clicked. There can be a number of reasons why it’s not getting the attention you want it to have. By changing the advertisement or giving it a different location, you can improve the number of clicks and find out what was wrong with the origi-nal. Heatmaps give you the opportunity to look through the eyes of your visitors as they navigate on your website.

Heatmaps help designers and marketers discover which areas of pages are more suc-cessful than others. Consumers tend to follow the same pattern when they view pages, so correctly analyzing where these areas are will increase the potential success of the page. Analyzing heat maps allows you to increase your conversion rates.

Heatmaps

33

The ever growing offer of heatmap software makes it hard to choose the best program for the job. New software seems to make its way to the web daily, all with different price range and extra features. The software that popped out for me was Mouseflow.

First of all, Mouseflow is very cheap. Next to the free version (100 visitors) it also provides a basic account (1000 visitors) for 10 euro a month. Compared to the other advanced software, this is very cheap.

Mouseflow has many features. Not only does it offer a detailed mouse click heatmap, it also a mouse movement and scroll heatmap. This is backed up with a powerful web ana-lytics package that give you lots of statistics and features to play around with. It records the pageviews, total visit length, interaction time, bounce rates, loading time, response time and the scroll reach.

On top of that Mouseflow offers livestreaming, a very nice feature with which you can follow your users live. It also records whole visitor sessions, which you can playback in your browser or simply download to your hard drive to watch it whenever you want. These recordings include the mouse movements, clicks, scroll events and key strokes.

Finally, the software is easy to implement. Just insert one line of Javascript and you’re ready to start testing.

Heatmaps

Heatmap Software

34

There’s no such thing as an

AVERAGE USER

35

There is no such thing as the average web user. Every user is unique and has his or her own interpretation of surfing the web. But we still want to make our website accessible for every user. This is where personas are very useful.

A persona is a fictional profile of a random user, which exists from relevant contextual information, based on a product or service. Personas are used to represent the different user types within a targeted demographic, attitude and/or behaviour set that might use a site, brand or product in a similar way. They can loosely be based on customers, friends or family members. This way a persona becomes more realistic and easier to implement. It can also be based on a character from a movie or a play.

In most cases, personas are synthesized from data collected from interviews with users. They are captured in short descriptions that include behaviour patterns, goals, skills, at-titudes, and environment, with a few fictional personal details to make the persona a real-istic character. Most personas include a portrait to make the character as 3D as possible. For each product, more than one persona is usually created, but one persona should always be the primary focus for the design.

Personas help prevent “self referential design” when the designer or developer may un-consciously project their own mental models on the product design which may be very different from that of the target user population. Personas also provide a reality check by helping designers keep the focus of the design on cases that are most likely to be en-countered for the target users and not on edge cases which usually won’t happen for the target population.

Personas

Personas

36

Observing people using a design will always trump an inspection

37

In 1990, Jakob Nielsen and Rolf Molich published the heuristic evaluation: ten basic prin-ciples (heuristics) that make sure a user interface is usable for its users. When multiple evaluators apply them to any UI, it should reveal gaps in the design of the UI that could cause problems for users. Someone who had never seen the target UI before, and who was not necessarily knowledgeable about the domain, should be able to determine wheth-er any UI complied with these ten commandments of usable UI design. When a group of people do this separately, they come up with a list of problems. They then compare these problems and hand a list off to some engineers to go fix it.

A heuristic evaluation provides a designer with fast, cheap and easy results. It can be done early in the development cycle and you only need a couple of testers, which can (partly) be novice users. Also, it doesn’t require any advanced planning.

Although the method is still used, it receives a lot of criticism. The method seems to be an inspection, rather then an evaluation. It is not about the user experience. The reviewer judges the UI against some established rules of thumb instead of own personal experi-ence/opinions on interface design. The evaluator does not perform any task or evaluates any flow as such, rather he just apply usability principles to find out usability issues.

Heuristic evaluation may help a team know whether their UI complies with someone else’s guidelines. But observing people using a design will always trump an inspection or audit for getting solid evidence to determine a design direction.

Heuristic Evaluation

Heuristic evaluation

38

As a designer you won’t always have a budget. Even larger projects can often lack a dedi-cated budget for user experience testing. Maybe the client doesn’t think it’s necessary, maybe the budget is really tight, or maybe there are other reasons. So you will have to get creative to test your work.

It doesn’t really matter how much you know about user experience, after working on a project for a couple of weeks you’ve become blind to many of the problems. Sure, experi-ence will help you avoid making many design mistakes in the first place, but something will slip through. Every time. That’s why doing even a very basic usability test will improve your site.

Here’s an overview of do it yourself techniques you can use, which are absolutely free. A quick warning: these techniques might work in theory, but they won’t always work in practice. That’s why I’ve added my experiences and pro’s and con’s.

Low Budget Testing

39

Location: Betterplace OfficeDuration: 30 minute setup

For this test I’ve set up a wall at the coffee machine in our office. Here, I hung up a big sheet of A0 paper and put up 3 mock ups and a couple of markers. I then send an email around the office, asking them if they could look at and comment on my mock ups, while waiting for their coffee.

Results

The office wall test was a great way to get a lot of information and feed back without doing a lot of work. Setting up the experiment only cost me about half an hour and the next day I got a wall full of suggestions from people who know the website very well. This might be the only downside. The people in the office have seen the website over and over. Some elements that are very obvious to them might be confusing to people who visit the website for the first time.

Low Budget Testing

Office Wall

40

Pros- Lots of great feedback in short period of time- People are willing to give their honest opinion- People benefit by giving useful feed back

Cons- People might know the site too well

Location: Free University of BerlinDuration: 5-10 minutes per survey

The downside of online surveys is that you can’t interact with people in any way. Experi-ence shows that most of the people will only give half answers and you will not have a chance to evaluate their survey. That’s why I set up a short real life survey, which con-tained 3 short assignments. The first and second assignment were about the Betterplace.org landing page and website and the third one was a short scenario in which they had to choose their favourite mock up. By fitting all 3 assignments on 1 peace of A4 paper, people wouldn’t be scared off by the length of the survey.

Results

I chose the Free university of Berlin, because I thought people there would respond posi-tive to a short survey, but it turned out I was wrong. The students at the university weren’t eager to spent 5 to 10 minutes of their time on a survey, even if I told them it was for charity and it would help me a lot for my research. Others we’re busy, got mad or simply ignored me. Also, the interviews could only be done with the use of a laptop. This meant I had to look for people with a laptop, who weren’t busy studying, but even students who weren’t studying weren’t interested. I was aiming for 25 surveys, but I only got to do 7. If you’re planning on doing field surveys in the future, choose a location that you know well with people you know are open to help you.

Surveys

Pros- Quick results - Option to ask extra questions

41

Cons- Hard to find people willing to spent some of their time

Location: Schlesische Strasse BerlinDuration: 1 minute

We had a deadline coming up for the donations page and we had to make a decision about the lay-out, so time was of the essence. Therefore, I decided to go out on the street and ask people for their opinion. I printed a page showing a charity raising money for the Japan tsunami incident. Then I asked people to pretend a friend sent them a link to that page and asked them to donate money. The attendant would agree and “clicks” the donate button. After that I showed them three mock ups and asked which one was the clearest and the most logical to them.

Results

It’s not possible to ask people for a lot of information. I told people my survey would take a minute tops, but still only one in six people were willing to spend this time. Others were in a hurry or not in the mood. Some people even walked out during the survey. The other problem with this sort of research is results aren’t always accurate. During my research I tended to ask people why they chose that mock up and what they liked and hated about the others. Someone could chose mock up 3 because it looks nice, without considering the taxonomy of the information given on the page. By asking people for comments, I came up with an advice that didn’t fit the number of votes.

Low Budget Testing

Street Research

Pros- Quick way to get a lot of opinions

Cons- Number of votes not always accurate

42

Tools & Techniques used by market leading companies

43

Many companies use the designed experiment approach to making marketing decisions. It is an increasingly common practice as the tools and expertise grows in this area. There are many UX testing case studies which show that the practice of testing is increasingly becoming popular with small and medium businesses as well.

Market leading companies known for UX testing include:

Amazon.com BBC Google Microsoft Playdom (Disney Interactive) Twitter Zynga eBay Wikipedia

While these companies are known for using behind the scenes tests to maximize profits, the practice occasionally makes it into the spotlight.

The next pages include UX tests done by market leading companies who published the test setups, and sometimes results. This will give you an insight of how these companies are using tests to improve their product, service or system.

Other Companies

Techniques used by big companies

44

I can’t fault Google for this reliance on data. And I can’t exactly point to financial failure or a shrinking number of users to prove it has done anything wrong.

- Douglas Bowman45

Google’s 41 shades experiment is one of the most known split test stories found on the internet. The story goes as follows: a team of designers at Google had to pick a shade of blue, so they picked a shade they all liked. But then a product manager tested a bunch of different colors and found out users were more likely to click another shade. And for Google, more clicks mean more money.

So Marissa Mayer, Google’s vice president of search products and user experience, de-cided to test 41 different shades of blue. Incoming users were randomly assigned to pag-es with different coloured links, from blue-with-green to blue-with-blue. It turned out that people were more eager to click a blue-with-blue shaded link then a blue-with-green one.

The result: the moment this story was posted on the internet, people were debating the necessity of testing 41 different shades of blue. Douglas Bowman, a very talented visual designer and manager at Google, even quit his job. He couldn’t live with Google’s design philosophy that strictly revolves around data.

At the other hand: Google is one of the biggest companies in the world, with millions or users and billions of shareholder dollars at stake. It’s no easy task to please all of them. In this case it might be better to rely on data then the opinion of a group of designers. As trivial as color choices might seem, clicks are a key part of Google’s revenue stream, and anything that enhances clicks means more money.

Google

Google’s 41 Shades Experiment

46

“Thanks for the brain message“

$ 582.21

“Admit it - without Wikipedia, you never could have finished that report. Click here to keep Wikipedia free for future students

$ 2,901.08

“You depend on Wikipedia for information. Now it depends on you“

$ 2,992.91

“Please read: A personal appeal from Wikipedia founder Jimmy Wales“

$ 24,274.21

Wikipedia Annual FundraiserBanner Results

x 1000 $

47

$ 582.21

Wikipedia Annual FundraiserBanner Results

Wikipedia is an online encyclopaedia which is created by people from all over the world. To keep Wikipedia running, they need money in the form of donations. That’s why Wikipe-dia organizes fundraisers to collect money. In 2010, the company came up with this great idea to have users and contributors make up banners and test which ones were most ef-fective. Wikipedia decided to publish their results online.

The banner above, showing Wikipedia founder Jimmy Wales, won by a landslide which is not that big of a surprise. Seeing the founder appeal for a donation creates a very per-sonal feel.

User Quote:Funny enough Jimmy’s ad got me to give them money. Two factors came into play, the personal touch from the founder and the fact that Wikipedia is a free service. Seeing the actual founder “appeal” directly to you felt a lot more personal, almost creating a sense of guilt for the avid Wiki-pedia user. The other two ads were from the Wikipedia organization instead of a human being. In the words of Seth Godin: “Humans like humans. They hate organizations.” This type of ads works very well for a free service like Wikipedia, but the same ad would not have performed as well for a for-profit corporations, since people do not relate to CEOs.

Wikipedia

Wikipedia Fundraiser

48

Over the years, there’s been a lot of research for the best text to put on a call to action buttons. For example: a button with ‘sign up’ can have a lower conversion rate than ‘sign up for free’. When Twitter started, they used five totally different texts.

Twitter Sign Up Tests

Give it a Try: Emphasizes a low commitment barrier.Sign-Up: Descriptive, uses common term so to trigger the action. Also consistent with the term “sign-out” which Twitter also uses.Let Me In: Emphasizes being an outsider, possibly to try and trigger feelings of being left out so as to motivate to take the action of getting “in”. Get started now: “now” emphasizes immediacy, while “get started” suggests there is a larger process at hand. Join Today: Also emphasizes group activity, that by signing up you are “joining” the Twit-ter community.

While Twitter was running this test, the other parts of their pages remained the same. This gives them cleaner test results. If you have fewer variables on your pages, you will have more reliable results.

Twitter

49

During the campaign for the 2008 presidential election in the United States of America, Barack Obama and his team used split testing to get people to sign up in their mailing list. The set up was simple: the page contained a media area, for either a picture or a video and a call to action button, in this case, enter your email address. The team used 3 pictures, 3 video’s and 4 different buttons. This means they had 3*3*4 = 24 different combinations.

Obama

Presidential MVT

These tests kept running, until every combination was seen by 13.000 people. The team then analyzed the results and came up with the perfect combination of picture/video and button. This combination was used for the rest of the campaign.

Dan Siroker, as part of the Obama team and CEO of Optimizely, claimed that the unoptimized page had a sign up rate of 8.26%. After they ran the tests and optimized their page, the sign up rate was 11.6%, and improvement of 40.6%. This im-provement led to an additional $60 million in donations for the Obama campaign.

Crazy as it sounds, the 2012 election could be decided by who makes the best use of website optimization.

50

There are a lot of new and innovative tools and techniques on the market that can have a big influence on your website. New tools are developed every day en existing ones have a growing amounts of extra features.

A/B testing

A/B testing is used to gain insight into visitor behaviour and to increase conversion rate and has been growing for the last couple of years with no end in sight. The simple idea of testing version A versus version B boosted conversion rates for dozens of websites. Even market leading companies use it to increase their traffic. A/B testing software nowadays is available for everyone and is going to play an important role in web design and marketing in the future.

Multivariate testing

In a multivariate test, a webpage is treated as a combination of elements that affect the conversion rate. By testing different combinations of elements you can tweak your web-page into the perfect combination. In combination with A/B testing, this can be a great technique to improve conversion rates at Betterplace.org

Conclusion

51

Heatmaps

Heat maps help designers and marketers discover which areas of pages are more suc-cessful than others. Properly analyzing heat maps allows you to increase your conversion rates. Heatmaps are a good way to get an insight of visitor behaviour and can be of great value to Betterplace.

Personas

By creating and using personas at an early stage in the process, you can get rid of flaws in your design. Personas are cheap and very useful and can be a great addition to Bet-terplace.

Heuristic Evaluation

I’ve included the Heuristic evaluation to this research, because the term is widely used by User Experience researchers and company executives. While testing this method, I real-ized it’s pretty much outdated. The evaluation has been around since the beginning of the 90’s and although it’s been updated multiple times since, it doesn’t apply today.

Low Budget Testing

Especially the office wall test showed to be very helpful during the design of new or im-provement of existing pages for the Betterplace website. As a designer, you’ll never know for sure how visitors will react to changes. By involving the experience from the other of-fice workers in the process, you can get rid of flaws in an early stage.

Conclusion

52

Now I know which tools and techniques are valuable for Betterplace and which aren’t, I can add them in the design process. Before I started my research, the design process at Betterplace looked like this:

As you can see, there is no process concerning the visitors. If a page is rebuild, it is de-ployed and the only way to check if it’s better or worse than the old one is to wait for the statistics.

With the tools and techniques from my research I can improve the design phase and add a testing phase in the process. This way we can measure if a new page is actually better than the old one.

Advice

Advice and Recommendations

54

At this moment the design phase is a one man job with minimal feedback. The designer makes the decisions the way he thinks is best. Therefore I recommend to include other people into this process without keeping them from their jobs.

Office Wall Test

The office wall test is a great way to share your design with the rest of the office and get quick feedback, without bothering your colleagues. There are a lot of people in the of-fice who have a very strong opinion about the Betterplace website. Not only in product management and development, but in every department there are colleagues with lots of experience. The office wall test costs nothing and is really easy to set up. After one day, you will have a wall full of suggestions and feedback from the people who have known and worked with the website for years.

Personas

Another good way to find flaws early in the design process is by using personas. These can be created or found online. By imagining another person performing a task on your new design, you can find confusing parts and change them.

Design Phase

55

To really measure the impact of a new page, I advice to implement a testing phase, which includes A/B testing and Multivariate testing.

A/B Testing

Even if you think your design is perfect, there will still be some parts that made you doubt. Just test these parts with an A/B test. Changes may well be that you’re wrong and visitors won’t react the way you expected them to.

With A/B testing you’ll have the ability to isolate and assess the performance of virtually every element of a web site. From page layout, to headline text, to complete color palette variations, A/B can be highly effective in identifying what works and what doesn’t from a user’s perspective. It can be easily scaled and deployed with no regard to internal technol-ogy infrastructure.

The reaction of your visitors is something unpredictable. Even the most professional in-teraction designers have it wrong sometimes. By testing as much as possible you can be sure of your case.

Advice

Test Phase

56

Always test new versus old

Whenever you improve a page, you should always test the new version versus the old version. Even if you have the statistics for the old page, they may be influenced by ele-ments beyond your control. Your new page also might not be as good as you think and you’ll end up with less then you started. Make sure only new visitors will see the new page. This way, your returning visitors won’t be confused by changing parts of your web-site. Until your new page turns out to be an improvement, you can introduce it to your returning visitors.

Multivariate Test

After you have tested your basics with A/B tests, it’s time to tweak your page with Multi-variate tests. This means headlines, pictures, texts, etcetera. By testing every combina-tion, you will see which works and which doesn’t. Even the smallest change in your page can lead to an increase of your conversion rate.

But be warned, with every extra combination you make, you need extra visitors. Better-place has an average low six digit number of visitors per month, depending on the time of the year, so you should be able to make an estimation for testing.

Try to include other departments in the process. Al lot of colleagues have a very strong opinion about the website, especially in marketing. A simple email with suggestions can help a lot. By analyzing the results, you can optimize pages and improve them.

57

Heatmaps

Heatmaps visualize the stream of visitors on your website, showing hot and cold click zones. They provide an overall view of the activity and helps you to see every mouse movement and every click. If used in the right order, it will improve web usability, conver-sions and revenue.

A heatmap can be very useful in designing your page, and in how to arrange the contents of each individual web page. It can provide you with an understanding of how people travel through web pages, and what parts of each page they tend to look at first. If you know the areas of your web page that visitors tend to focus on when they first arrive, you can design that page to suit. It is also useful to understand the areas that they tend not to look at.

Heatmaps can be used in combination with A/B testing and Multivariate testing. If you know where your visitors tend to go first, you might want to experiment with that part of the page. If you see which parts of the page aren’t used at all, you might want to improve that part of the page and see the results.

Advice

58

BooksBowles C., Box J. - (2011) Undercover User Experience Design - Berkeley, New Riders

Cagan M. - (2008) Inspired: How To Create Products Customers Love - California SVPG Press

Corbone G. - (2011) Simple and Usable, Web, Mobile and Interaction Design - Berkeley, New Riders

Krug S. - (2010) Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems - Berkeley, New Riders

Wroblewski L. - (2008) Web Form Design: Filling the gaps - New York, Rosenfeld Media

Recources

Recources

60

Articles

Managing UI Complexity by Brandon WalkinLean UX: Getting Out Of The Deliverables Business by Jeff GothelfHow to Shop for Unmoderated Usability Testing Tools by Bill AlbertForgotten principles of usability testing by David TravisPioneering a User Experience (UX) Process by Amy Hillman10 Usability Tips Based on Research Studies by Cameron ChapmanUser Experience Design by Peter Morville’s10 Most Common Misconceptions About User Experience Design by Whitney HessMultivariate Testing 101: A Scientific Method Of Optimizing Design by Paras ChopraMultivariate Testing in Action: Five Simple Steps to Increase Conver-sion Rates by Paras ChopraThe Ultimate Guide To A/B Testing by Paras Chopra by Jacob GubeA/B Split Testing vs. Multivariate: Pros & Cons by Chris GowardCase Studies & Success Stories by Virtual Website OptimizerMy big list of 24 Web Site Usability Testing Tools by Craig TomlinBuilding a Data-Backed Persona by Andrea WigginsWeb Analytics Tools: Where to Begin? by LynneUsability Expert Reviews: Beyond Heuristic Evaluation by David Travis

Recources

61

User experience design: Persona by Michel Kuik Picking the right tool for remote user testing by Matt Milosavljevic Putting a Bolder Face on Google by New York TimesVisualize This: Winning Wikipedia fundraiser banners by Flowing DataObama’s site is performing a test on you by Daryl Lang Twitter A/B testing sign-up button by Joshua Porter

Websites

www.smashingmagazine.comwww.johnnyholland.orgwww.uxbooth.comwww.boxesandarrows.comwww.wikipedia.comwww.visualwebsiteoptimizer.comwww.uxmag.comwww.conversionvoodoo.com/blog/

62

Note: The tools are changing everyday in terms of functionality and pricing structure.

A/B Tools Test Results

63

Heatmap Test Results

64

This is a list of tools and techniques I also tested, but didn’t make the re-search, because they turned out to be too expensive, time consuming or just not fit for Betterplace.

Video Tools Expert Reviews Online Surveys Eye Trackment Neuro Marketing Full Service Tools Card Sorting

Other tested software

65