PG 1

PG 2

Improving Speech Applications with Usability Surveys

How does Nortel measure the ‘Usability Pulse’ of Self

Service?

Judith SherwoodSales Engineer

Nortel Self Service Solutions

PG 3

Many Usability Test Methods

Data from Each Participant

Num

ber

of

Par

ticip

ants Live Pilot

EmployeeTest Calls

Follow-Up(call-back)Surveys

Focus Groups

Usability Survey

PG 4

What is a Usability Survey?

• Usability testing is an evaluation of a customer touch-point from a user perspective

• Typically conducted using small focus groups (12-20 subjects) in a controlled studio environment

PG 5

What is a Usability Survey?

• Methodology: Usability Survey using 200 to 500+ panelists to reveal problems affecting fewer callers o Chosen from a large pool of over

80,000 panelists with known demographics

PG 6

• Traditional Usability Testing Methods have sample size limitations

• If a problem affects only 5% of the users:

o 10-call sample has a 40% chance of finding it

o 100-call sample has a 99% chance of finding it

• If a problem affects only 1% of the users:

o 100-call sample has a 63% chance of finding it

o 500-call sample has a 99% chance of finding it

Size Matters !

PG 7

Methodology

Panelist places call, completes a task, and then fills out a questionnaire on an internet website

Entire call conversation is recorded for analysis

Survey ties the individual caller experience and questionnaire response to the call recording

Call-in Platform

Target Service

Post-Questionnaire

Panelist Pool

Design Tasks &Questionnaire

Analysis &Recommen- dationsReport

PG 8

Methodology

More efficient, less expensive, faster execution, broader feedback Panelists recruitment

and call-in campaign is outsourced

Analysis provides recommendations for service improvement Listen to problem calls

to suggest ways to fix

Call-in Platform

Target Service

Post-Questionnaire

Panelist Pool

Design Tasks &Questionnaire

Analysis &Recommen- dationsReport

PG 9

• Percentile-based letter grading system to compare against other speech applications

• Raw Scores are based on:

o Caller Satisfaction (% very satisfied - % dissatisfied - % very dissatisfied)

o Task Completion (% who finish task in one call)

o Consistency (variability in call length)

Usability Survey Grading Process

PG 10

Lessons Learned: A Case Study

• “Acme”: A regional Managed-Care Health Insurance Company. Customer Service available for Members and Healthcare Providers

• Available Tasks in Self-Service:o Members:

• Check Co-Pay amount and Physician Name• Order a replacement ID card

o Providers: • Check Claim Status• Verify Member Status and Co-Pay

• Initial Usability Survey, then Tuning, followed by 2nd Survey

PG 11

Initial Results1.0 How easy was it for you to accomplish your objective in this call?

Very easy 16% (85 Panelists) Easy 27% (146 Panelists)

Neither easy nor difficult 15% (79 Panelists) Difficult 21% (110 Panelists)

Very difficult 4% (22 Panelists) I did not accomplish my

objective in this call. 17% (93 Panelists)

2.0 How satisfied were you with your overall experience?

Very satisfied 12% (66 Panelists) Satisfied 33% (174 Panelists)

Neither satisfied nor dissatisfied

14% (73 Panelists)

Dissatisfied 29% (155 Panelists) Very dissatisfied 12% (66 Panelists)

No Response 0.2% (1 Panelist)

• Satisfaction Grade = D• Satisfaction Score= %VS – %D – %VD• Satisfaction Score= +12 – 29 – 12 = – 29%

• Call Completion Grade = D• Call Completion Score= % accomplished objective in one-call only• Call Completion Score= (100–17) x (0.83) = 69%

PG 12

After Tuning

1. How easy was it for you to accomplish your objective in this call?

Very easy 52% (285 Panelists) Easy 25% (136 Panelists)

Neither easy nor difficult 8% (44 Panelists) Difficult 6% (35 Panelists)

Very difficult 1% (6 Panelists) I did not accomplish my

objective in this call. 7% (37 Panelists)

No Response 1% (3 Panelists)

2. How satisfied were you with your overall experience?

Very satisfied 50% (275 Panelists) Satisfied 30% (165 Panelists)

Neither satisfied nor dissatisfied

8% (46 Panelists)

Dissatisfied 8% (46 Panelists) Very dissatisfied 2% (13 Panelists)

No Response 0.2% (1 Panelist)

Call Completion Grade = CCall Completion Score= (100–7)x(0.86) = 80%

Satisfaction Grade = ASatisfaction Score= +50 – 8 – 2 = +40%

PG 13

UI Overall Improvement After Tuning

Very satisfied 12% (66 Panelists) Satisfied 33% (174 Panelists)

Neither satisfied nor dissatisfied

14% (73 Panelists)

Dissatisfied 29% (155 Panelists) Very dissatisfied 12% (66 Panelists)

No Response 0.2% (1 Panelist)

Very easy 16% (85 Panelists) Easy 27% (146 Panelists)

Neither easy nor difficult

15% (79 Panelists)

Difficult 21% (110 Panelists) Very difficult 4% (22 Panelists)

I did not accomplish my objective in this call.

17% (93 Panelists)

Very easy 52% (285 Panelists) Easy 25% (136 Panelists)

Neither easy nor difficult

8% (44 Panelists)

Difficult 6% (35 Panelists) Very difficult 1% (6 Panelists)

I did not accomplish my objective in this call.

7% (37 Panelists)

No Response 1% (3 Panelists)

• How easy was it for you to accomplish your objective in this call?

• How satisfied were you with your overall experience?

Very satisfied 50% (275 Panelists) Satisfied 30% (165 Panelists)

Neither satisfied nor dissatisfied

8% (46 Panelists)

Dissatisfied 8% (46 Panelists) Very dissatisfied 2% (13 Panelists)

No Response 0.2% (1 Panelist)

Initial Results

After Tuning80% Call Completion

69% Call Completion

+ 40% Satis-faction Score

– 29% Satis-faction Score

PG 14

• Look for the Red Flags of Voice Recognitiono Low satisfaction and call-completion scoreso Low voice-recognition rating scoreso Check Complaints in free responseso Observe if “can’t hear” problems for low-

score panelists

• Work with Developer and IT supporto Is speech level strong enough in the IVR?

Check switch gain levelso Then check speech detector parameterso Then check recognition confidence

thresholds and grammars

Best Practice 1:Catch Recognition Problems Quickly

PG 15

Voice Recognition Issues• How well or poorly did the system recognize your responses

when you spoke the answer to questions?

• Fix: Increased digital gain from host switch

Very well 47% (257 Panelists) Well 31% (167 Panelists)

Neither well nor poorly 8% (46 Panelists) Poorly 10% (57 Panelists)

Very poorly 3% (16 Panelists) No Response 1% (3 Panelists)

Very well 11% (59 Panelists) Well 21% (110 Panelists)

Neither well nor poorly 12% (66 Panelists) Poorly 34% (183 Panelists)

Very poorly 22% (116 Panelists) No Response 0.2% (1 Panelist)

Initial Results

After Tuning

• How quickly or slowly did the system respond to your spoken answers?

Very quickly 46% (249 Panelists) Quickly 41% (225 Panelists)

Neither quickly nor slowly

8% (46 Panelists)

Slowly 3% (16 Panelists) Very slowly 1% (8 Panelists)

No Response 0.4% (2 Panelists)

Very quickly 16% (88 Panelists) Quickly 44% (238 Panelists)

Neither quickly nor slowly

21% (113 Panelists)

Slowly 13% (72 Panelists) Very slowly 4% (23 Panelists)

No Response 0.2% (1 Panelist)

PG 16

• Look for the Red Flags of Prompt Confusiono Low satisfaction and call-completion scoreso Low ‘What-to-Speak’ scoreso Listen for caller hesitations for low-score

panelists

• Work with Dialog Designero Let callers know ahead of time that they can

speako Reword prompts; Callers appreciate clear

choiceso Give Touch-Tone options when repromptingo Coach your voice actor

Best Practice 2:Spot Prompt Clarity Confusions

PG 17

Voice-Prompt Issues

• Was it clear what you needed to select or say at each step of the call?

Clear for all steps 58% (315 Panelists) Clear for almost all

steps 29% (160 Panelists)

Clear for some steps 8% (45 Panelists) Clear for only a few

steps 4% (21 Panelists)

Clear for no steps 1% (4 Panelists) No Response 0.2% (1 Panelist)

Clear for all steps 27% (142 Panelists) Clear for almost all

steps 27% (147 Panelists)

Clear for some steps 24% (129 Panelists) Clear for only a few

steps 16% (88 Panelists)

Clear for no steps 5% (28 Panelists) No Response 0.2% (1 Panelist)

• How appropriate or inappropriate was the speaking style and voice of this service?

Very appropriate 58% (319 Panelists) Appropriate 36% (198 Panelists)

Neither appropriate nor inappropriate

4% (20 Panelists)

Inappropriate 1% (5 Panelists) Very Inappropriate 0.2% (1 Panelist)

No Response 1% (3 Panelists)

Very appropriate 32% (172 Panelists) Appropriate 52% (276 Panelists)

Neither appropriate nor inappropriate

11% (61 Panelists)

Inappropriate 3% (17 Panelists) Very Inappropriate 1% (7 Panelists)

No Response 0.4% (2 Panelists)

Initial Results

After Tuning

PG 18

Voice-Prompt Issues

• Fix: Clarify prompt choice wordingo Initial: “Are you a member, provider, or a

non-member looking for information?”o After Tuning: “First, tell me who you are:

a member, a provider, or a non-member. ”

• Fix: It’s not just what you say, but how you say it.o Coach your voice talent for proper

inflections.

PG 19

• Look for the Red Flags of Call-Flow Frustrationo Low satisfaction and call-completion scoreso Low Enough-Choices scoreso Check complaints about inflexible systemso Listen for caller frustrations for low-score panelists

• Work with Dialog Designero Look for easy ways to complete repetitive taskso Provide an easy exit strategyo Leverage the spoken language instinct

Best Practice 3: Spot Call Flow Frustrations

PG 20

Call-Flow Issues

• When you were given menu choices, were you given too many, just enough, or too few?

Too many choices 10% (55 Panelists) Just enough choices 85% (466 Panelists)

Too few choices 4% (22 Panelists) No Response 1% (3 Panelists)

Too many choices 11% (61 Panelists) Just enough choices 74% (395 Panelists)

Too few choices 15% (78 Panelists) No Response 0.2% (1 Panelist)

After Tuning

Initial Results

PG 21

Call-Flow Issues

• Fix: To clarify options, Provide Anchor, Split Main Menu choices, and Add Grammar Synonymso Initial: o “Alright, I’m going to tell you the things I can

help you with. When you hear the right one, just say it… Verify member status and office co-pays. Get status of a claim. Get member ID cards. Change PCP. Or, order forms and literature.

o After Tuning: o “Alright, Main Menu. Please say one of the

following options at any time … Verify member status. Check PCP co-pay. Get claim status. Order ID cards. Change PCP. Order forms. Or, order literature …”

PG 22

• Look for the Red Flags of Task Differenceso Low satisfaction and call-completion

scores for Specific Taskso Check complaints about complicated taskso Listen for caller confusion in low-scoring

tasks

• Work with Dialog Designero Look for easy ways to complete repetitive

taskso Make sure task instructions are clearo Provide an easy operator exit strategy

Best Practice 4: Spot Task Differences

PG 23

Task Differences

• Relative Task Satisfaction can change after tuningo Can’t Hear problem can swamp

other UI task issueso Claim Status Sat. low for business

reasons (no shortcut for multiple claims)

PG 24

Task Differences• Fixes

o Improve recognition to shift all scores up and reveal other UI issues

o Future: Give clearer exit menu optionso Future: For Claim Status, offer stream-

lined repeat-claim function

-75 -25 25 75

Office CoPay

ID Cards

Verify Status

Claim Status

Task

Satisfaction Score

Satisfaction AFTER

Satisfaction BEFORE

PG 25

• The Usability Survey method is very effective for tuning applications prior to pilot production

o Collect hundreds of calls and analyze results efficiently

• Recognition/Prompt/Call-Flow issues are revealed quickly in Usability Survey

o Listen to calls for which various ratings are low

• Task differences show the effect of Task Consistency, Complexity, and Business Rules on Usability Quality

o Longer tasks require more call-flow efficiency and friendly hand-holding.

o Transfer to Agent for business reasons can sometimes lower satisfaction if callers must first spend a long time in Self-Serve, or if transfer takes place with no explanation.

Conclusions

PG 26

Thank You!