Clasen Sample Usability Test

LISCI310‐Clasen‐IndividualProject Page1of15

UW‐MilwaukeeL&ISCI310

CommonIndustryFormatforUsabilityTestReportv2.0iPhone3Gver.4.1forElderlyUsers

TestLeadandReportPreparedbyWilliamClasenTestConducted:April4‐6,2011ReportPrepared:April7‐8,2011

Contents PageExecutiveSummary 2

Introduction 3

FullProductDescription 3

TestObjectives 3

Methodology 4‐9

ParticipantProfile 4

ContextofProductUseintheTest 4

TaskList 5

TestFacility 6

DeviceEnvironment 6

Design,Procedure,andInstructions 7

UsabilityMetrics 8

Results 9‐12

DataAnalysis 9

Task1:AccessWeb 9

Task2:SendTextMessage 10

Task3:SendEmail 12

Recommendations/Discussion 13‐14

Bibliography 15

Contact:WilliamClasen(608)921‐8324

[email protected]

Janesville,WI53546


ExecutiveSummary

Manyservicesreadilyutilizedbyseniorsaremigratingonline.

Marketinganalystshaveidentifiedthreecommongoalswithrespectto

usersofsmartphonesoverfifty‐yearsofage:textmessaging,email,and

webconnectivity.AusabilitytestwasconductedtargetingtheiPhone

3Gver.4.1(pictured)ontheAT&Tcellularnetworkforusebyelderlyclientele.Thisdocument

containstestmethodology,efficiency/effectivenessratings,taskcompletionrates,timeontask,

errors,participantfeedback,anddiscussionforimprovementsbasedonapplicabletheory.

Fourparticipantsovertheageoffifty‐fivewereselectedtoparticipateinthreetasksthat

targetedthefunctionsrequiringevaluation.Halfoftheparticipantshadprevioussmartphone

experiencewithadifferentdevice.Allparticipantsweredailyusersoftheweb,textmessaging,and

email.Overall,theparticipantsappearedveryreceptivetosmartphone

technology.Onlyahandfulofmodificationsarerequiredtomitigatesensoryand

cognitivechallengesexperiencedbythetargetusergroup:

Change Justification Severity1. Emphasizezoomandmagnifyin

marketing,sales,andhelp2. Add'magnify'buttonin

supportingapplicationstoincreasefontsizeall‐at‐once

Mitigationofsensorychallengesthatresultfromaging.

Allfourparticipantsfailedtoacknowledgethezoomandmagnifyfeaturestoreduceeyestrain.

25%ofparticipantsfailedtocompletewebbrowsingtaskasaresult

High

3. Auto‐correctiondefaultstatusto'off'orprovidelinkdirectlytothissettingonthehomescreen

Mitigationofcognitivestabilitychallengesthatresultfromaging.

50%ofparticipantsindicatedthisfeaturewas"irritating"or"distracting",butdidnotknowhowtoturnitoff

Moderate

4. Addcontextuallabelto'createnew'icon

5. ModifycontextuallabelforSafarito"WebBrowser"

6. Change'plus'iconto'contact'iconforconsistencyacrossallplatformapplications

Mitigationofcognitivesimplicityandmemorabilitychallengesthatresultfromaging.

75%ofusersfailedtorecognizeSafariasawebbrowserapplication

100%ofusersfailedtorecognizethe'plus'icontoaccessthecontactlist

75%failedtorecognizethe'plus'iconinsubsequenttasksafterassistance

High


Introduction

ProductDescription

TheiPhone3GiOSver.4.1(picturedbelow)asmanufacturedbyAppleandoperatedonthe

AT&Tcellularnetworkisbeingmarketedasasmartphoneformainstreamusers.Functionssuchas

textmessaging,email,andwebconnectivityofferedbysuchdevicesareofcurrentinterestto

smartphonemarketinganalystswithregardtoseniorcitizens.(Greengard,p21)

Manycommerce,healthcare,andgovernmentservicesreadilyutilizedbyseniorsare

migratingonline.Greengardmaintainsthatelderlyclientsareentirelyreceptivetosmartphone

technologysolongasitusefulandeasytooperate(Greengard,p20).Inordertoachieveusefulness

andeaseofoperation,thedesignmustmitigatechallengespresentedbydegradationofvision,

tactileperception,andcognitionastheageofclienteleadvancesbeyondfifty‐fiveyearsofage.

(Greengard,p20‐1).

TestObjectives

Thisreportassessesusabilityanddiscussesimprovementforthisdevicewithrespectto

seniorcitizensandtheircommongoalsforfunctionality.Inparticular,findingsregardingweb

navigation,sendingtextmessages,andsendingemailsarereported.Audio,asitisunrelatedto

theseparticularfunctions,isnotevaluated.


Methodology

ParticipantProfile

Allfourparticipantswerechosenasarepresentativesampleofthetargeteduser

population.Criteriaincludedageandprevioussmartphoneexperience.Allparticipantsareolder

thanfifty‐fiveyearsofage.Participantsweresegmentedbasedonprevioussmartphoneexperience.

TwoparticipantseachhadtwoyearsofexperienceusingwebenabledBlackberrydevices.Two

participantshadnopreviousexperiencewithsmartphones.

Priortoevaluation,eachparticipantwasaskedabouttheiraveragedailyinvolvementwith

thefollowingfunctions:hoursspentaccessingtheweb,numberoftextmessagessent,andnumber

ofemailssent.Thequestionsdidnotspecifycellulardevices.Thegoalwassimply

toascertainoverallawarenessandexperiencewiththesefunctionsviaany

availabletechnologythattheymayhavehad.Itwasfurtherdeterminedthatnone

oftheparticipantshadanyexperiencewithAppledevicesorsoftware.

Key Name Age Gender SmartphoneExperience(Years)

DailyWebAccess(Hours)

DailyTextMessagesSent

DailyEmailsSent

P1 JerryClasen 66 Male 2 2 12 8P2 CarolClasen 68 Female 2 4 10 15P3 JamesWuetrich 70 Male None 3 0 4P4 DorothyWuetrich 68 Female None 2 3 12

ContextofProductUseinTest

Thereisonesignificantdifferencebetweentheevaluatedcontextandtheexpectedcontext

ofuse.Theaudiocapabilitiesofthissmartphonedevicearenotcriticaltoevaluatingwebaccess,

textmessaging,andemailfunctionsofthetargetusergroup.Audiocapabilities,asseenlargelya

componentoftraditionalcellularphonesandmultimediausesassociatedwiththesmartphone,are

notevaluatedinthisreport.

Usebythetargetgroupwillbepersonalratherthanprofessionalandevaluatedassuch.


TaskList

Taskswereselectedtoevaluatethemostcommongoalsassociatedwiththedevicerelative

tothetargetaudienceasrepresentedbytheparticipants.

Task1Task Viewapple.comwebsiteandnavigateto[support]State HomescreenSSC Participant

1. openssafariapplication(33%)2. entersapple.cominaddressbar(33%)3. activateslinktosupporttablocatedonapple.comhomepage(33%)

Benchmark 0.5minute.TaskFailureat2.5minutes.Script Pleaseopentheinternetbrowserandgo toapple.com.Oncetheapple.comwebpage

hasloaded,findthesupportlinkonthatpageandclickit.

Task2Task Sendtextmessagetocontact JohnDoeState HomescreenSSC Participant

1. opensmessageapplication(25%)2. selectscontact(25%)3. typesmessage(25%)4. sendsmessage(25%)

Benchmark 0.5minute.TaskFailureat2.5minutes.Script JohnDoe'scontactinformationhasbeenstoredinthephone.Pleasesendthe

followingtextmessagetoJohnDoe:Hello

Task3Task Send30wordemailtocontact John DoeState Homescreen.Participant isgivenhardcopyof50wordemailtoduplicateandsend.SSC Participant

1. opensemailapplication(25%)2. selectscontact(25%)3. typesemailcontent(25%)4. sends(25%)

Benchmark 1.5minutes.TaskFailureat7.5minutes.Script JohnDoe'scontactinformationhasbeenstoredinthephone.Pleasesendthe

followingemailtoJohnDoe:DearJohn,YouarereceivingthefirstemailthatIamsendingfromanIPhone.IamhelpingtoconductausabilitystudyforAT&T.Ihopealliswell.


TestFacility

Participantswereobservedperformingthetaskssetforthbythisevaluationattheir

respectivehomesinordertoreplicatetypicaluserenvironment.

DeviceEnvironment

Allparticipantsusedthesamedevice(Serial#87926MD8Y7H)inordertonegatehardware

specificvariables.

AllparticipantswereevaluatedintheirhomesinthesamegenerallocationwithintheAT&T

cellularnetworkinordertonegatecellularsignalstrengthvariables.

Thefollowingtechnicalspecificationsrepresentaspectsofthedeviceenvironmentthatare

subjecttothisreport:

1. Sizeandweight

a. Height: 4.5inches(115.5mm)

b. Width: 2.4inches(62.1mm)

c. Depth: 0.48inch(12.3mm)

d. Weight: 4.7ounces(133grams)

2. Capacity

a. 8GBflashdrive

3. Cellularandwireless

a. AT&T3Gcellularnetwork

b. GSM/EDGE(850,900,1800,1900MHz)

4. Display

a. 3.5‐inch(diagonal)widescreenMulti‐Touchdisplay

b. 480‐by‐320‐pixelresolutionat163ppi

c. 16M(full)color

d. Supportfordisplayofmultiplelanguagesandcharacterssimultaneously

iPhone3Gver.4.1onAT&Tcellularnetwork


Task Evaluation

Section 1

Experience

Section 2

Orientation

Section 3

Tests

Section 4

Debriefing

TestMonitorRole

Thetestadministratorgreetedparticipants,explainedeachtaskandprovidesupporting

materialaseachtaskisintroduced,andrespondedtorequestsforassistanceonlyifaskeddirectly

bytheparticipant.AssistanceistobelimitedtothecurrentSSCsegmentforthetaskonhand.

Design,Procedure,andInstructions

Thetaskevaluationwillconsistofa

performancetestmadeoffoursections:

1. Participantgreetingandbackground

Eachparticipantwillbepersonally

greetedbythetestmonitorandmade

tofeelcomfortableandrelaxed.The

testmonitorwillrecordparticipant

responsestothefollowquestions:

a. Whatisyourname?

b. Whatisyourage?

c. Haveyoueverusedasmartphone?Ifso,forhowlong?

d. Howmuchtimeperdaywouldyouestimatethatyouaveragebrowsingthewebon

cellphones,computers,TVbrowsers,etc.?

e. Howmanyinstantortextmessagesperdaywouldyouestimatethatyousendon

averageviacellphones,computers,TVbrowsers,etc.?

f. Howmanyemailsperdaywouldyouestimatethatyousendonaverageviacell

phones,computers,TVbrowsers,etc.?

FourSectionTaskEvaluationModel


2. Orientation

Participantswilleachreceiveascripted,verbalintroductionandorientationwhichwill

explainthepurposeandgoalsofthetest.Theywillalsobeassuredthatthedeviceiswhat

isbeingtested,notthemselves.

3. PerformanceTest

Theperformancetestconsistsofaseriesoftasksthattheparticipantswillcarryoutwhile

beingobserved.Thesetaskswillbeconductedinthefollowingmanner:

a. Participantwillbehandedascript,whichthetestmonitorwillreadoutloud,

detailingthetasktobeaccomplished.Theywillthenattemptthattask.

b. Duringtheperformancetestthemonitorwillmakenotesonelapsedtimeand

participanterrors.Thetestmonitorwillrespondonlytodirectrequestsfor

assistancebyverballyofferingproceduralhelp.Thetestmonitorwilldocument

numberofrequestsandatwhatstageduringthetasktheyoccurred.Themonitor

willalsomakenoteofanyunusualcircumstances,suchasincorrectkeystrokes,that

maynotbepresentedasdirectmeasuresoftheSSC.

c. Afterconcludingthetasks,theparticipantwillbedebriefedbythetestmonitor.

4. ParticipantDebriefing

Eachparticipantwillbedebriefedbythetestmonitor.Eachsessionwillbedocumentedby

thetestmonitor.Thequestionisasfollows:

a. DoyouhaveanyparticularfeedbackorrecommendationsfortheiPhonebasedon

thetaskyoujustperformed?

UsabilityMetrics

1. Measures of effectiveness include percent of task

completion, quantity of errors, and frequency of assists.

2. Measures of efficiency includes time taken to achieve task.

Usabilitymetricstodeterminesatisfaction


Results

DataAnalysis

Taskcompletionissegmentedasapercentageinthetasklistforthepurposeofquantifying

results.Assistsrepresenteachdirectrequestfromtheusertothetestmonitorforassistance.

Tasktimebenchmarks,aspresentedinthetasklist,aremeasuresoftimerepresentingthe

durationofthetask,roundeduptothenearest0.5minuteinterval,ascompletedbythetest

monitor,WilliamClasen.WilliamClasenhasbeenadailymainstreamuserofthedevicefortwo

years.

Tasktimefailurearebenchmarkstimesmultipliedbyfive.Taskfailuretimesrepresent

time‐outduetofrustration.

Keystrokeerrorsrepresenteverykeystrokeonthekeypadthatdidnotleaddirectlytoward

taskcompletion.

Task1AccessWebSite

Key UnassistedTaskEffectiveness

(%TaskComplete)

AssistedTaskEffectiveness

(%TaskComplete)

Assists TaskTime(min)

KeystrokeErrors

P1 0 100 2 1.25 6P2 66 100 1 1.00 0P3 0 66 0 2.50 8P4 0 100 2 1.75 2

P1,P3,andP4failedtorecognizethedefaultwebbrowser

application,Safari.Asaresult,twooutoffourparticipantsasked,"How

doIgettotheweb?"Oneparticipantasked,"WhereisExplorer?"

Allparticipantsfailedtoimplementpinch‐zoomormagnifyas

toolstoexpandtextontheapple.comhomepage.Thesefeatureswere

necessarytorenderthesupportlinklegible.Activatingthesupportlink

representedthefinal33%oftheSSC.AllparticipantsthataskedforTask 1 ‐ Web Access


assistanceregardingthisfeatureasked,"HowdoIzoom?"P3didnotseekassistancewiththezoom

featureandtimed‐outwhilerandomlytappingthescreeninhopesoffindingthecorrectlink.All

otherkeystrokeerrorsresultedfromsinglecharactertypographicalerrorsandthesubsequent

deletionoftheerrantcharacter.

Figure 1 ‐ Task 1 Time and Assists

Allparticipantsexceededthebenchmarktimeof0.5minutesforexperiencedmainstream

users.P1andP2,bothexperiencedsmartphoneusers,askedmoreimmediatequestionswhile

requestingassistanceandcompletedthetaskfasterthanP3andP4,non‐experiencedsmartphone

users.Duringthetaskdebriefing,P2andP3expressedfrustrationwiththedurationofthetask.P1

andP2suggestedthatthezoomfeaturewasnotveryintuitive.

Task2SendTextMessage


(%TaskComplete)


(%TaskComplete)


KeystrokeErrors

P1 25 100 1 1.00 4P2 25 100 1 0.75 2P3 0 100 2 2.00 10P4 0 100 2 1.50 8

0

0.5

1

1.5

2

2.5

Benchmark P1 P2 P3 P4

Time (min)

Assists


P3andP4didnotknowhowtoaccessthetextmessaging

applicationonthedevice.Thetestmonitornoticedthatthesetwo

participantsheldthephonecloseasifstrainedtoreadthetextlabelsthat

accompaniedicons.P3,after25secondsand4randomkeystrokes,and

P4,after15secondsand2randomkeystrokes,askedthetestmonitor

howtoaccesstheapplication.P1andP2accessedtheapplicationdirectly

fromthehomepageratherthanbrowsingthroughthecontacts.

Noneoftheparticipantsrecognizedthe'plus'iconasthewayto

accessthecontactlist.Eachparticipantaveraged15secondssearchingthescreenbeforeaskingthe

testmonitorhowtoaccessthecontactlist.Similarobservationsweremaderegardingthe'create

newicon'.Allfourparticipantsexpressedfrustrationregardingtheiconasunintuitive.P2

suggestedthatthecontactsicononthehomepagewasmorerecognizable.

Keystrokeerrors,otherthanthosereportedabove,wereduetotypographicalerrorsand

subsequentcorrection.P2statedthekeypadresponsiveness,"seemedslow".P1andP3statedthat

theautocorrectfeature,"wasannoying".P4offeredanunsolicitedstatementthattheautocorrect

feature,"washandy",thoughitappearedtotakeP4moretimetoevaluatetheoutput.



users.P1andP2,bothexperiencedsmartphoneusers,completedthetaskfasterthanP3andP4,

0

0.5

1

1.5

2

2.5


Time (min)

Assists

Task 2 ‐ Text Message


non‐experiencedsmartphoneusers.Allparticipantscompletedthetaskwellbeforethetaskfailure

timeof2.5minutes.

Duringthedebriefing,P1andP2feltthattheprocessoverallwas"simple".P3offeredthat

thetaskmadehimfeel,"empowered".P4suggested,"Icouldgetusedtothis."

Task3SendEmail


(%TaskComplete)


(%TaskComplete)


KeystrokeErrors

P1 100 100 0 3.00 8P2 100 100 0 2.00 4P3 25 100 2 5.50 20P4 25 100 1 4.50 12

P1andP2,presumablyfromhavinghadpreviousexperience

emailingfromsmartphonedevices,completedthetaskwithout

assistance.P3andP4bothrecognizedthemailapplication,butfailedto

recallthatthe'plus'iconprovidedaccesstothecontactlist.P3,after10

secondsand4randomkeystrokes,andP4,after15secondsand6

randomkeystrokes,askedthetestmonitorhowtoaccessthecontactlist.

Theremainingkeystrokeerrorswereduetotypographicalerrorsand

subsequentcorrection.Again,P1andP3expressedfrustrationregarding

theautocorrectfeature.


0

1

2

3

4

5

6

7


Time (min)

Assists

Task 3 ‐ Email



users.P1andP2,bothexperiencedsmartphoneusers,againcompletedthetaskfasterthanP3and

P4,non‐experiencedsmartphoneusers.Allparticipantscompletedthetaskwellbeforethetask

failuretimeof7.5minutes.Duringthedebriefing,allparticipantscommentedthattheaccesstothe

contactlistinthemailprogramwasnotintuitive.P1andP2suggestedthatthecontacticononthe

homepagebeutilizedinsteadofthe'plus'icon.

Recommendations/Discussion

Asidefromsomesensoryandcognitivechallenges,theparticipantsexpressedeagerness

regardingthedevice.25%oftheparticipantsfelt"empowered"bythetasks.Thefollowing

guidelineswilloffersupportforassistingelderlyclientswithcellularweb‐basedinteractionbased

ontheaforementionedobservations.

Sensorydesignprinciplesrevolvearoundvisibilityoffonts,

pointers,andiconsinconsiderationoftheelderlymarket.Thefont

sizeusedonboththescreenandkeypadshouldsupportvisual

limitationsthatresultfromaging(Nielsen,2002).Nielsenfurther

stipulatesthataminimumtwelvepointfontshouldbeutilized.The

iPhone3Gprovidesuserswithwaystomitigatesensoryrestraints

andincreaseoverallusability.Asaresultofthistest,itappearsthattheexistenceofthesefeatures

isnotapparent.Itisrecommendedthatmarketingtothetargetgroupemphasizesaccesstothe

pinch‐zoomandmagnifyfeatures.Asafurthersuggestion,aniconrepresentinga

magnifyingglass(pictured)couldbeincludedinpredominantlytextdriven

applicationsasawaytocallattentiontothemagnifyfeature.Alternatively,theicon

couldservetoincreaseonscreenfontsizeallatonce.

Inordertomaximizeusability,thecognitiveloadrequiredtocarryouttasksmustbe

reducedtoaminimumwithregardtoelderlyclients(Greengard,p21).Cognitiveloadrefersto

Recommendmagnifyicon

Marketing and help should emphasize zoom feature


stability,simplicity,memorability,andpredictabilityindeterminingtheusefulnessofdesign

principles(Heim,pp198‐208).

Intermsofstability,theelderlyrequireinterfacesthatareforgivingtothepointofallowing

multipleformatsfordataentrysuchastelephoneandcreditcardnumbers(Nielsen,2002).During

thedebriefingfortasks3and4,50%ofparticipantsindicatedthattheinherentauto‐correct

featurethatprovidessuggestionswhiletypingwas"irritating"or

"distracting".Whenmarketingtoelderlyusers,specialattention

shouldbegiventowardasolutionthatwillmakethelocationofthesettingtoturnoffthisfeature

(pictured)morereadilyaccessible.Alternatively,thedefaultsettingcouldbestetto'off'.

Mainstreamuserswillbemoreadeptatfindingthesettingtoturnitbackonifdesired.Forgiving

interfacesrequirethaterrormessagesbeeliminated,ifpossible,duetoobscurityandimprecision

thatmayresultincognitivedisconnect(Nielsen,2002).

Intermsofsimplicity,windowsshouldutilizeasingledocument

interface(SDI)andshortcontextualmenusifastaticinterfaceis

unattainable(Heim,pp368‐69,376‐86).TheiPhone3Gpresentsanfairly

straightforwardSDI.Iconspresentedcontextualdilemmasforallofthe

elderlyparticipants,however.Intask2,allparticipantsfailedto

immediatelyrecognizethe'createnew'icon.Thisfunction(pictured)is

representedbyarelativelysmallsquarecontainingahashmark.The

usersexpressedfrustrationregardingthemeaningoftheicon.Inorder

toincreaseusabilityforseniors,itisrecommendedthatthisiconbegivena

contextuallabel.Alongsimilaranalysis,threeoutoffourusersdidnotrecognize

theSafariapplicationicon(pictured)asaccesstoaninternetbrowser.Moreissues

regardingiconrecognitionaredescribedbelow.

Browsericonnotrecognizeable

'CreateNew'iconnotidentifiable


Memorabilityrequireseventstobepronounced(Nielsen,2002).Asthetest

resultsindicate,fiftypercentofuserscannotbeexpectedtorecallicon

identificationevenafterthefunctionoftheiconisillustrated.25%ofparticipants

suggestedthatthecontactsiconvisibleonthehomescreenwasmorerecognizable.

Inthesamevein,predictabilityrequirestheinterfacetobefamiliar(Greengard,

p20).Inordertoovercometheobstaclesapparenttoaccessingthecontactlist,itisrecommended

thatthe'plus'iconbereplacedwiththemorecommonlyrecognizedcontactlisticonaspictured.

Inordertomaintainutility(Heim,p198)anysmartphonedeviceofferedtotheelderly

shouldincorporatebroadbandinternetaccess.Inordertosatisfythedemandsforwebaccess,text

messaging,andemailbyusersoverfifty‐fiveyearsofage,theinterfacemustmitigatesensoryand

cognitiveabilitiesasageprogresses.(Greengard,p21)Shouldphoneandmulti‐mediamarketing

analysisbedesired,audiousabilitytestingmayberequired.

Bibliography

Nielsen,J.2002.UsabilityforSeniorCitizens[online].Available:http://www.useit.com/alertbox/seniors.html(accessedMarch12,2011).

Greengard,Samuel."FacinganAge‐OldProblem"CommunicationsoftheACM.Sep.2009:20‐22.

Heim,Steven.TheResonantInterface.NewYork:PearsonEducation,Inc.2008.Print.

Contact icons

Clasen Sample Usability Test

Documents

Transcript of Clasen Sample Usability Test