What Can Be Learned From A One-Minute Online Survey

What Can Be Learned From a

One-Minute Online Survey

How people really feel about Amazon.com (and what does this mean)?

An Illustrative Case Study Using

Foreword: Web teams need constant feedback

By Gerry McGovern

Living systems get constant feedback from their external environment. To truly

succeed, web teams need constant feedback from their customers.

You're a manager in a restaurant. It's raining. A customer walks in and almost slips

on the mat in front of the door. You're very busy at this stage, but you make a

mental note: "I must change that mat." About 15 minutes later another customer

comes in. She, too, almost slips on the mat. You rush up to her, apologize profusely

and then change the mat.

People are slipping on our websites right now but, because we don't see them slip,

we don't change the mat. I'm one of the biggest offenders. Over the years I have

left content and applications on my websites that had problems that I was vaguely

aware of, but they just didn't seem important enough to warrant any action. Even

when I became clearly aware of the issue I didn't react with enough urgency.

Why was that? Why was I so complacent? I would like to think that if I was running

a restaurant I would have apologized to the customer and changed the mat. Why

don't I do that when it comes down to managing a website? I think a core part of the

problem is the lack of real feedback.

I'm not actually seeing the customer slip. I don't actually see real people use my

websites.

Customers are hugely impatient on the Web. When they slip, their first impulse is to

hit the Back button. Jared Spool wrote an excellent article in 2009 called the "The

$300 Million Button." In it he explained how the removal of a registration button

from a particular step in a purchase process resulted in a dramatic improvement in

sales.

The Web team had created the registration button so as to make it easier and faster

for regular customers to buy. But people absolutely hate registration. New

customers felt they would be spammed if they registered. One potential customer

summed up their feelings as follows: "I'm not here to enter into a relationship. I just

want to buy something."

What Can Be Learned From a One-Minute Online Survey

3

The regular customers didn't feel much happier. "45% of all customers had multiple

registrations in the system, some as many as 10," Jared wrote. "We also analyzed

how many people requested passwords, to find out it reached about 160,000 per

day."

The Web is so important today. And yet many of the web teams I deal with are way

down the management hierarchy. Intranet teams, in particular, tend to get

negligible resources. That needs to change because the reality is that the Web is

central to the present and future success of most organizations.

One of the ways we make that change happen is that we start developing much

better feedback mechanisms for our websites. At a most basic level, we must find

ways to regularly (weekly at minimum) observe our customers carry out top tasks

on our websites. That's how Jared Spool discovered there was a problem: by

watching customers trying to buy.

According to Wikipedia, "Living things are systems that tend to respond to changes

in their environment." Let us embrace our customer environment. Let us observe

and evolve. The rewards are very substantial.

Gerry is the founder and CEO of Customer Carewords. He is widely regarded as the

worldwide authority on increasing web satisfaction by managing customer tasks.

Gerry has spoken, written and consulted extensively on web content management

issues since 1994. His new book, The Stranger's Long Neck: How to Deliver What

Your Customers Really Want Online, was published in July 2010.

Read the first chapter of the book at http://www.gerrymcgovern.com/sln-ch1.htm

http://www.gerrymcgovern.com/sln-ch1.htm


4

Executive summary

In order to demonstrate what can be learned from a carefully-designed survey that

is very quick to complete, we asked a combination of consumers and Web

professionals to rate Amazon.com (chosen as it is very familiar to many people). As

expected, the site was rated very positively yet there was still much to learn. The

most compelling characteristic of the site for most visitors was the ratings and

reviews feature. But while ratings and reviews are a strength now, they might also

be under threat. The growth of social media has diminished the advantage of

ratings and reviews by strangers and Amazon is arguably no longer at the leading

edge in this respect. Other findings demonstrate that female visitors like to ‗get

things done quickly‘ and are more troubled by the difficulty of ‗contacting a person‘

while men value ‗complete information‘ a little more.

Perhaps surprisingly, Web professionals complained more than consumers that

Amazon.com had a cluttered layout, hinting that expert opinions cannot be relied on

exclusively without the input of regular Web users.


5

Table of contents

Introduction ....................................................................................... 6

Key findings ........................................................................................ 7

Results ............................................................................................... 8

Analysis .............................................................................................. 9

Ratings, reviews and recommendations are popular but under threat ... 9

People like Amazon‘s search but there is no room for complacency .... 10

Clutter and layout were problems, but not for all .............................. 11

Some people find it hard to contact a person ................................... 12

Visual appearance not of overriding importance ............................... 14

Methodology ..................................................................................... 16


6

Introduction

Consumers have opinions on websites; they express them to friends and family, not

in the technical and analytical terms that professionals might use, but in quick

summary statements that capture the essence of the sites.

Typically, Web surveys ask consumers to rate sites in technical terms, but this is

expecting consumers to think the way we experts do, to tell us what and where site

issues exist.

. We need to simulate being present at hundreds of social gatherings and

eavesdropping on conversations about websites. The approach used in this Amazon

review, tried and trusted over many years, puts the onus on the analyst to interpret

consumer sentiment, and makes it easier for them to express their feelings. It was

our goal to run through a illustrative example from start to finish to show you the

level of actionable information can be gleamed from this simple and comparatively

inexpensive method. You can be the judge of the success of that objective.

This technique does not ask questions in company-centric language (e.g. ‗Please

rate the options that are available for you to navigate this website?‘), but instead

simply asks people to express what they felt strongly about. This ensures that their

instinctive reactions are captured. It also helps that the survey was very quick to

complete — it genuinely took less than a minute and nobody abandoned it.

The success of the survey depends of course on the results. You have your opinion

of Amazon, so how well do the findings match your expectations, but also surprise

and provide ‗food for thought‘? Ultimately, what value does this survey bring to

website analysis?

This study was conducted by The Customer Respect Group using the Customer

Carewords methodology. It is far from perfect. We would have liked larger

participant numbers and we did invite web professionals with their different

preconceptions to get involved, which may have skewed overall results a little. But

a lot of data and analysis emerged from what was, superficially, a very simple

survey.

We instead need to

collect what people feel

strongly about; what

they would say over the

barbecue or at the water

cooler


7

Key findings

A very clear trend emerged from the results – the top five attributes accounted for

over half of the overall impressions. The strong trend showed that the survey

successfully identified the aspects of Amazon.com that are most compelling. The

positive sentiments were dominant and the site recorded an extremely high rating

of 86 out of 100. The general rating for websites tested with this methodology is

typically in the 50-60 range.

The top five attributes were:

1. Site has ratings, reviews and recommendations

2. Helpful search results

3. Fast to do things

4. Simple layout/easy to read

5. Accurate information

An important consequence of the survey is that it highlights site strengths just as

much as weaknesses. So while ratings, reviews and recommendations are a

much-loved feature this strength can become a threat should competitors neutralize

their value.

Although the negative sentiments were in the minority, a significant minority is not

happy with some aspects of Amazon.com.

Even a successful site like Amazon can be improved. Given the volume of traffic, a

small improvement could have a significant effect. The three main areas of concern

were:

1. Cluttered layout

2. Hard to contact a person

3. The site was visually unattractive


8

Results

Participants came from a pool of demographically evenly-spread consumers

(approx 60% of the total) and web professionals. The graph below reflects the

importance of each of the 26 attributes in the survey. A red bar indicates a negative

sentiment.

0.0%

0.2%

0.2%

0.2%

0.2%

0.4%

0.4%

1.1%

1.2%

1.4%

2.1%

2.2%

2.2%

2.3%

3.0%

4.1%

4.2%

4.9%

5.0%

5.2%

6.1%

6.2%

8.4%

9.3%

13.4%

16.3%

Inaccurate information

Misleading / not transparent

Has no ratings, reviews, recommendations

Full of jargon / corporate speak

Out-of-date information

Hard to participate / give feedback

Incomplete information

Poor search results

Confusing menus and links

Slow to do things

Easy to contact a person

Looks unattractive / unappealing

Plain language

Hard to contact a person

Gives me the facts / transparent

Up-to-date information

Cluttered layout / hard to read

Looks attractive / appealing

Clear menus and links

Easy to participate / give feedback

Complete information

Accurate information

Simple layout / easy to read

Fast to do things

Helpful search results

Has ratings, reviews, recommendations


9

Analysis

The data is a signpost and tells us where to look; the interpretation brings the data

to life and leads to actionable tasks and strategies to pursue. With websites so

complex, it is critical to garner consumer sentiment continuously to reflect changing

expectations as much as changes in functionality. When so many companies focus

on customer retention and growth, sentiment towards websites is a critical metric

that up to now has been difficult to harness.

Ratings, reviews and recommendations are popular but under threat The presence of ratings, reviews and recommendations was the top-rated attribute.

It was once what differentiated Amazon.com but the landscape has changed. Most

retailers have adopted, and even improved, reviews and for very good commercial

reasons:

78% of Internet users rate recommendations as most credible form of

advertising (Neilsen)

84% of US customers prefer the opinion of other customers versus

experts (Marketing Sherpa)

96% of online retailers rank customer ratings and reviews as an

effective or very effective tactic for driving conversion (Forrester)

65% trust ‗friends‘ recommendations while 33% trust

company-supplied recommendations (emarketer)

69% of consumers who read reviews then go on to share them with

family and friends (Deloitte)

Products that have reviews show a 35% increase in conversion rates

(Bazaarvoice)

One weakness of the Amazon implementation is that it does not allow for

recommendations by friends. We can read that hundreds of people rated this

product as 5-star, but it‘s not immediately clear if they share our tastes or

preferences. A clear finding of various studies is that consumers want to hear from

―someone like me‖.

Best Buy‘s Facebook page, which has almost 1.2 million fans (compared to

Amazon‘s 90,000) includes a ‗Shop + Share‘ tab on which consumers can browse

products and traditional reviews. A key enhancement, however, is the ‗Ask Friends‘

button which allows fans to collect input on their decision from friends‘. This makes

it very easy to get the opinions of trusted peers. Now, friends can offer an opinion

before we buy, in a manner similar to bringing a friend to a bricks-and-mortar store.

84% of US customers prefer

the opinion of other

customers versus experts.

—Marketing Sherpa


10

There are many other examples of how ratings and reviews have evolved and

developed. Once considered a unique feature of Amazon, this could be a

game-changing feature and Amazon can‘t afford to fall behind the innovation curve

in an area so ingrained in its users‘ experience. If a key strength is perceived as

being inferior to systems in use elsewhere, there is a danger that Amazon might lose

some of its popularity.

Best Buy‘s Facebook page brings reviews and ratings to a new level by allowing people to ask

their friends for advice on selected products

People like Amazon’s search but there is no room for complacency Good search results was considered to be the next most important attribute. Search

is sometimes considered to be ‗Plan B‘ when navigating through menus and links

has not resulted in success, but in this case it seems unlikely. Very few people chose

‗confusing menus and links‘, so it seems that Amazon‘s search is actually particularly

good – good enough to be a Plan A when arriving on the site. Customers know that

searching for that book or DVD will produce really accurate and fast results.

Best Buy has recruited 15

times more ‗fans‘ on

Facebook than Amazon

and more likely to benefit

from recommendations

from friends


11

Amazon search has customer-oriented features that can help them find what they

need very quickly. Sub-categories on the left menu allow customers to filter results

if the default results are not quite right, ratings are visible, and related searches are

suggested.

Amazon cannot afford to neglect improving its search facility. There are many web

shoppers that are not wedded to an Amazon-like one-stop shopping experience.

According to Hitwise, 34.5% of traffic to top retail sites in July 2010 came directly

from search engines.

Google‘s new instant search introduces the concept of consumers fine-tuning

search terms dynamically. If this proves popular, Amazon will need to respond.

There are also examples of reviews being included in search results (Bazaarvoice is

supplying reviews to Google that includes content from sites like BestBuy and

Macys).

As Facebook becomes more and more a destination site for consumers, its search

feature might drive traffic to rival sites, particularly if their Facebook presences are

more advanced or feature-rich.

Negative sentiments were in the minority but can’t be ignored The important thing about negative sentiments in a one-minute survey is that they

help companies isolate general issues and provide the roadmap of where to start

‗digging‘. Furthermore, we have found that Web teams welcome some negative

findings, which are often seen as a validation of their own opinions. A more typical

ratio of positive to negative is 65/35 or 70/30, and in these cases the discussion of

how to rectify problems would dominate the analysis.

Clutter and layout were problems, but not for all The most significant negative aspect of Amazon.com was ‗cluttered layout, hard to

read‘. Clearly, page layout matters - it has direct impact on the ability to complete

tasks. Too many long blocks of text, too many links, clutter, not enough white

space, lack of a coherent structure; these can all slow people down. According to a

consumer for whom this is the most important facet of Amazon.com:

I‘ve used the site many times and have found it confusing to use. It has taken me

too long in the past to get to the information I need or to find out that I had to

do something else to get where I wanted to be. Just a hassle!

- Female consumer, 25-34 years old, quite familiar with the site

Amazon cannot

afford to neglect

improving its search

facility... 34.5% of

traffic to top retail

sites in July 2010

came directly from

[external] search


12

The bigger takeaway story on this factor was that this was much more important for

professionals than consumers. It was the only factor where professionals differed

from consumers to any degree of significance. It may indicate that web

professionals tend to judge layout more than consumers do, years of experience

having trained them to look at websites differently. Very few Web professionals

would not criticize Craigslist yet it remains a very popular site. According to Jim

Buckmaster, CEO of Craigslist:

―I hear this all the time, ‗you guys are so primitive, you are like cavemen. Don‘t

you have any sense of aesthetic? ‗ But the people I hear it from are invariably

working for firms that want the job of redoing the site. In all the complaints and

requests we get from users, this is never one of them. Time spent on the site, the

number of people who post—we‘re the leader. It could be we‘re doing one or two

things right.‖

To reinforce this view, far more people chose ‗simple layout, easy to read‘, with

consumers more satisfied with the layout than the professionals.

Although subjectivity is also a factor in this attribute, the difference of opinion

shows the importance of getting the opinions of non-expert consumers, who vastly

outnumber the Web professionals in the online space. Without intending to, we

were able to show that web professionals may not always be the best predictors of

consumer sentiment.

Some people find it hard to contact a person At one time, it was extremely difficult to get in touch with Amazon support staff.

Things have improved, but it still is not that easy to engage with a representative.

The issue is the lack of real-time escalation – being able to contact an agent by

phone or chat without leaving the page. Many competitive sites offer telephone

numbers, live chat, instant callback and click-to-call to ensure that consumers don‘t

lose momentum by having to navigate to another page to resolve an issue.

According to Art Technology Group, 58% of consumers now look for live help if

missing information while online, while 53% will do so if there are problems

checking out. It was interesting that men looked for help less than women,

reinforcing the general perception of men having an aversion to asking for

directions.

... ‗you guys are so

primitive, you are

like cavemen.

Don‘t you have any

sense of

aesthetic?‘...

In all the

complaints and

requests we get

from users, this is

never one of them.

— Jim Buckmaster,

CEO of Craigslist

Sears offers chat, call

back and email on

every page to assist

the consumer

complete the buying

process — this is a

standard of

availability of help

that sets

expectations for

visitors to sites like

Amazon‘s.


13

Sears, a retailer that has announced major plans to expand its online offering with

the adoption of an ‗Amazon-like‘ marketplace that offers products from third parties,

provides prominent options for consumers to reach out. Service agents can view the

contents of the consumer‘s shopping cart and answer any questions. This helps to

reduce cart abandonment, a major issue for online retailers. Sears offers chat,

call-back and email on every page to assist consumers complete the buying process

— this is a standard of availability of help that sets expectations for visitors to sites

like Amazon‘s.

Live chat with an agent from product pages allows Sears customers to escalate issues or get

answers without interrupting their tasks

It would be very revealing to drill down with the participants that expressed concern

about contact options to find out, for example:

Why is this an issue – what was the reason they needed to contact

Amazon when they found it difficult?

Was the issue they tried to escalate to do with a site issue, or perhaps

an offline issue like late delivery or a billing discrepancy?

Did the issue cause them to abandon a purchase and go elsewhere?

What were their expectations of Amazon – a telephone number on

every page, or just an easier path to an email contact form?

Chatters that engage

via proactive

invitation are 6.3X

more likely to convert

than visitors who

don‘t chat.

— Bold Software Live

Chat Performance

Benchmarks October

2009


14

Visual appearance not of overriding importance Visual appearance is of course another subjective characteristic, but what is striking

is that the ‗look‘ of the site was not important to most people, whether they liked it

or not. (There was only a slight difference here between consumers and

professionals.)

Functionality seems to be what matters — with the appearance of Amazon not

getting in the way of people browsing products and buying. This indicates that once

the appearance of a website is not strikingly bad (and Amazon‘s certainly isn‘t), it

does not really register prominently with visitors. Once customers can do what they

need to do, they‘ll not be too concerned about colors, fonts, imagery or other design

features. That‘s not to say that any design will do – it still needs a professional to

design an interface that is visually coherent and serves the brand well. But once

people can get where they need to go quickly, and complete their tasks, the visual

appearance is very much secondary to functionality. Consumers are probably too

short on time to notice the visual design of site that is designed to get things done.

This finding also raises further questions:

What type of site visitor finds Amazon.com unattractive?

Are new visitors more critical of the appearance, perhaps it becomes

less important as they find out that the site is functionally excellent?

What exactly about the appearance is causing an issue – is it related to

clutter, or is it a purely subjective attribute like the choice of colors and

fonts?

Why do they feel strongly enough about appearance to pick this

attribute rather than one of the other 24 available?

… the ‗look‘ of the site was

not important to most

people, whether they liked

it or not


15

Report Summary

Consumers want choice and they want low prices. That‘s a big reason for the

success of Amazon.com. Alongside its own merchandise, Amazon also offers goods

from other retailers and the revenue from this ‗marketplace‘ in the first half of the

year was roughly $4.2 billion.

Major online competitors including Sears and Wal-Mart are aggressively building

their own marketplaces and industry experts say several more large retailers will

launch similar efforts in the coming months.

So, does our study help understand Amazon‘s strengths, weaknesses, opportunities

and threats?

Clearly, ratings and reviews are a critical attraction for Amazon but they

need a significant upgrade to exploit new channels of social media,

especially Facebook. Sears has already developed a sophisticated online

community which looks to be more advanced than Amazon‘s. This is

critical to consumers and in a world of diminishing loyalty may serve to

weaken Amazon‘s dominant position

Customers increasingly see online retail as a multi-channel experience

and it is likely that Amazon will need to provide better options for

speaking to a service person in real time. The study indicated a small

(but what is likely to be a growing) dissatisfaction in this area. A

continuous review process would be able to monitor that. Both

Wal-Mart and Sears already offer better capabilities as well as store

pick-up options and are experimenting with same-day delivery in major

cities

Search is highlighted as critical and while Amazon‘s is good, there are

sites which already surpass its capabilities. There is the added threat of

Facebook and Google search as the most popular starting points for

consumers.

In summary, Amazon received a very positive rating so on the surface it has little to

put right. But as we have shown significant challenges are emerging.


16

Conclusions

We produced this report to step through a complete process; we used Amazon

simply because many people are familiar with the site and have impressions. Many

readers of the report will have provided input so you know that it did take the 1

minute we say. You all have regular visitors to your sites; what are their

impressions, what effect does that have on viral recommendations or on repetitive

use.

The benefit of the methodology is the combination of a unique survey technique in

combination with the expert interpretation from a range of partners that have

extensive online experience.

Already the methodology has been applied to sites across multiple industries and

countries, it provides great value for a comparative inexpensive commitment.


17

Methodology

We created an online poll and asked participants to tell us which three of 26

statements most closely described how they felt about Amazon.com, and allocate ‗3‘

to the strongest sentiment, ‗2‘ to the next, and ‗1‘ to the third (mirroring Gerry

McGovern‘s widely-used Customer Centric Index).

The statements were displayed in a randomly-ordered list. Although there were 26

statements, they were in fact positive and negative sentiments of 13 core attributes.

Visual/Architectural factors

Clear menus and links Confusing menus and links

Helpful search results Poor search results

Simple layout / easy to read Cluttered layout / hard to read

Looks attractive / appealing Looks unattractive / unappealing

Fast to do things Slow to do things

Content factors

Accurate information Inaccurate information

Complete information Incomplete information

Up-to-date information Out-of-date information

Plain language Full of jargon / corporate speak

Social factors

Gives me the facts / transparent Misleading / not transparent

Easy to contact a person Hard to contact a person

Easy to participate / give feedback Hard to participate / give feedback

Has ratings, reviews,

recommendations

Has no ratings, reviews,

recommendations


18

The statements as they were presented to participants (randomized for each)

19

USA

The Customer Respect Group 978.412.4047

http://www.customerrespect.com

Net Reflector 206.462.4240 http://www.netreflector.com/

Bob Johnson Consulting

248.766.6425 http://www.bobjohnsonconsulting.com

Canada

Neo Insight, Inc 613.271.3001

www.neoinsight.com

Europe

Ireland

Customer Carewords, LLC www.customercarewords.com

+353 87 238.6136

United Kingdom

Barry Hagan

+44 77 1470.0066 www.customercarewords.com

Brian Lamb

+44 79 8070.0075 www.customercarewords.com

Sweden

Webbrådgivaren Sverige AB +46 040.134.200

http://www.wpr.se

Netherlands

Sabel Communicatie +31 088.227.2240

http://www.sabelcommunicatie.nl

Norway

NetLife Research

+47 9240.3165

http://www.netliferesearch.no

Customer Carewords is a methodology

that helps you to truly understand what

the top tasks of your customers are

when they come to your website and

the critical impressions they take away.

By continuously improving the per-

formance of your customers' top tasks,

you will maximize the performance of

your website.

We have been developing customer

centric strategies for websites since

1994, identifying the top tasks of

thousands of customers and employees

for organizations such as Microsoft,

Cisco, Tetra Pak, Thrivent Financial, US

Internal Revenue Service, NHS Choices,

Rolls-Royce, BBC, Innovation Norway,

etc. We have partners in the UK, Hol-

land, Sweden, Norway, Canada and

United States.

What Can Be Learned From A One-Minute Online Survey

Business

Transcript of What Can Be Learned From A One-Minute Online Survey