Smart and adaptive interfaces for INCLUSIVE work environment · 2.2.1 Expert usability evaluation...
Transcript of Smart and adaptive interfaces for INCLUSIVE work environment · 2.2.1 Expert usability evaluation...
SmartandadaptiveinterfacesforINCLUSIVEworkenvironment
GrantAgreementN°723373
_______________________________________________________________________________________
Deliverable6.1–Reportonmethodsandtoolstomeasureworkersatisfaction
Contractualdeliverydate:2018-09-30
Actualdeliverydate:2018-09-30
ResponsiblePartner:CIOP-PIB
Author(s):DorotaŻołnierczyk-Zreda,ZofiaMockałło,KamilaNowak
WPn°:6
WPleader:CIOP-PIB
ProjectCoordinator:Prof.CesareFantuzzi
ProjectCoordinatorOrganisation:UNIMORE
Disseminationlevel:Public
TheresearchleadingtotheseresultshasreceivedfundingfromtheEuropeanUnion’sHorizon2020researchandinnovationprogrammeundergrantagreementN723377.
The author(s) is/are solely responsible for its content, it does not represent the opinion of theEuropeanCommissionandtheCommissionisnotresponsibleforanyusethatmightbemadeofdataappearingtherein.
ExecutiveSummary
Thisdeliverablereportsonmethodsandtoolstomeasureworkersatisfaction.Specifically,aliteraturereviewonmethodsandtoolstomeasureworkersatisfactionwithHMIinparticularandworkingeneralhasbeenconducted.OnthebasisofthisreviewaswellaspreviousDeliverables,amodelofworkersatisfactionhasbeen proposed and a questionnaire tomeasureworker satisfactionwith the adaptive HMI andworkingconditions has been developed. The questionnaire consists of two main parts: I. Working conditions(psychosocialandphysical);andII.SatisfactionwiththeadaptiveHumanMachineInterface(HMI),includingsafety issues. It has also been proposed to use physiological datameasurement inWP7 as an objectivemethodtomonitorworkersatisfaction.
TableofContent
1. Introduction _______________________________________________________________________ 1
2. Literaturereviewonmethodsandtoolstomeasureworkersatisfaction _______________________ 2
2.1. Usabilityconcepts_______________________________________________________________ 2
2.2. Methodsandtoolstomeasureusabilityandusersatisfaction____________________________ 3
2.2.1Expertusabilityevaluation _________________________________________________________ 3
2.2.2End-usersatisfactionevaluation_____________________________________________________ 6
2.2.3Objectiveusabilitymeasurement____________________________________________________ 8
2.3Satisfactionwithworkingeneral______________________________________________________ 9
2.4ConclusionsformeasuringworkersatisfactionintheINCLUSIVEproject _____________________ 10
3. TheusersatisfactionwiththeadaptiveHMIandworkingconditionsmodel____________________ 11
4. Thequestionnaire__________________________________________________________________ 12
4.1.Questionnaire’spreamble__________________________________________________________ 12
4.2.Occupationalanddemographicdata _________________________________________________ 12
4.3WorkingConditions _______________________________________________________________ 12
4.3.1PhysicalWorkingConditions_______________________________________________________ 12
4.3.2PsychosocialWorkingConditions___________________________________________________ 12
4.4SatisfactionwiththeadaptiveHMI ___________________________________________________ 13
4.4.1Safety_________________________________________________________________________ 13
4.4.2Satisfactionwiththedesign/visibilityoftheinterface___________________________________ 13
4.4.3Satisfactionwithease ____________________________________________________________ 14
4.4.4Satisfactionwithefficiency________________________________________________________ 14
4.4.5SatisfactionwiththeAdaptmodule _________________________________________________ 14
4.4.6SatisfactionwiththeMeasuremodule_______________________________________________ 14
4.4.7SatisfactionwiththeTeachmodule _________________________________________________ 14
4.4.8OverallSatisfactionwiththeadaptiveHMI____________________________________________ 14
5. Objectivemeasurementofworkersatisfaction___________________________________________ 16
6. Conclusion________________________________________________________________________ 17
7.References _________________________________________________________________________ 18
Appendix1–‘SatisfactionwiththeadaptiveHMIandworkingconditions’questionnaire
1
1. Introduction
TheINCLUSIVEprojectaimsatdevelopingsmarthuman-machineinterfaces(HMIs)thatadapttohumanskillsandcapabilities,andprovidetrainingandsupport.
TheWP6of theproject focusesondevelopingamethodologyofmeasurementandevaluationofworkersatisfaction in the industrial environment,with examples taken from the three industrial use cases. Themethodologytomeasureworkersatisfactionwillbebasedonsubjectiveandobjectivemeasures.TheresultofthisWPisamethodologytoassesstheadaptiveworkingenvironmentfromthehumanpointofview.
In this regard, this deliverable reports on outcomes of literature review conducted in order to analysemethodsand tools tomeasureworker satisfaction.Worker satisfaction isunderstood in thisprojectasaspecific case of user satisfaction, which refers to working environments, and work-related products orservices,includingHMIinterfacesandotherICT-basedsolutionsimplementedincontrolsystemsofindustrialmachinery andautomatedmanufacturing systems. Therefore, the literature reviewhasbeen focusedonmeasuresandtoolstoevaluatetheusabilityandsatisfactionwithHMIandwithworkingeneral.ResultsofthereviewarepresentedintheParagraph2.
Next,amodelofworkersatisfactionhasbeenproposed.Itincludesthreemainfactors:usersatisfactionwithHMI(includinghealth&safetyissues),physicalworkingconditions,andpsychosocialworkingconditionsðical aspects, taking into consideration individual differences, such as age, experience, capabilities,perception,cognitionormotorskills.ThemodelofsatisfactionwiththeadaptiveHMIandworkingconditionsispresentedintheParagraph3.
On thebasisof the literature reviewand theproposedworksatisfactionmodel,aquestionnaireentitled‘Satisfaction with the adaptive HMI andworking conditions’ has been developed. It is presented in theParagraph4.Thequestionnairecontainstwomainparts:I.Workingconditions;andII.SatisfactionwiththeadaptiveHumanMachineInterface(HMI),surveyingalsoskillsandcapabilitiesoftheusers.Inordertoensureworkers’satisfactionandparticipation, thequestionnairecontainsquestionsonproposedchangestotheHMI.ThequestionnairewillbepilottestedinTask6.2andthenadjustedforimplementationintheWP7.
As the worker satisfaction evaluation should be conducted by means of both subjective and objectivemeasurement, the ideaofphysiologicaldatameasurement intheWP7aspartof theMeasureModule isbrieflydescribedintheParagraph5.
Finally,theParagraph6followswithsomeconcludingremarks.
2
2. Literaturereviewonmethodsandtoolstomeasureworkersatisfaction
Theusersatisfactionderivesfromtheusabilityconcept.Ahighsystemusabilityisagoalofusers,employers,developersandresearchers.Itfacilitatessafe,productiveandenjoyableworkintheworkingenvironment(Ovaska,1991).Hence,accountingfortheHMIusabilityisoneofthekeyprojectandWP6goals.Inordertomeasure theHMI user satisfaction,we need to first understandwhat the usability principles are. In theliteraturereviewwepresentusabilityconceptsandmeasurementmethodsbasedonanexpertevaluationandend-user (worker) satisfactionmeasurement,bothsubjectiveandobjective. Itwasagroundwork forchoosingappropriatemetricstobeincludedintheworkersatisfactionmeasurement.
2.1. UsabilityconceptsUsabilityisabroadtermwithvariousdefinitions,dependingontheconceptorcontext.
Other,morebroadusabilitydefinitionsare:
• Ensuringthat interactiveproductsareeasyto learn,effectivetouseandenjoyablefromtheuserperspective(Rogersetal.,2011)
• Multidimensionalcharacteristicinthecontextofusersperformingtaskswithaproductinaspecificenvironment(Bevan,Kirakowski&Maissel,1991)
• The extent to which a product can be used by specified user to achieve specified goal witheffectiveness,efficiencyandsatisfactioninacontextofuse(ISO9241)
AccordingtoISO9241,effectivenessmeanstheaccuracyandcompletenesswithwhichspecifieduserscanachievespecifiedgoalsinaparticularenvironment.Efficiencymeanstheresourcesexpendedinrelationstothe accuracy and completeness of goals achieved. Satisfaction is defined as user comfort and useracceptabilityoftheworksystemandinrelationtootherpeopleaffectedbyitsuse.
In attempt of creating most comprehensive approach,Wang & Huang (2015) combined three differentusabilityprinciples:Nielsen’s(1993),Norman’s(2003)andYeh’s(2010).
Nielsen (1993) defined the following usability principles: memorability, errors, learnability, efficiency,satisfaction.
Norman’s(2003)conceptionofusabilityincludedsuchprinciplesas:visibility,agoodconceptualmodel,goodmappings,feedback.
Yeh’s(2010)usabilitydefinitionincludedfollowingprinciples:ease,enjoymentandeffectiveness.
On thebasisof theseconceptsandcomponents,Wang&Huang (2015)proposed the followingusabilityprinciples(fig.1):
• Visibility:clearinstructionsandinformation• Ease:easytolearnandfamiliarise,timeoflearningisminimal• Efficiency:oncelearneditiseasytousethefunctionsofthesystematfullcapacity• Enjoyment:usersaresatisfieduponcompletingataskwhenusingthesystem
3
Fig.1:Usabilityprinciples.Source:Wang&Huang,2015
InthecontextofHuman-Computer Interaction(HCI),Shackel (1991)definedusabilityas ‘thecapability inhuman functional terms tobeusedeasilyandeffectivelyby the specifiedrangeofusers, given specifiedtrainingandusersupport,tofulfillthespecifiedrangeoftasks,withinthespecifiedrangeofenvironmentalscenarios’.
Preece(1993)stated:`ThegoalsofHCIaretodevelopandimprovesystemsthatincludecomputerssothatuserscancarryouttheir tasks:safely,effectively,efficientlyandenjoyably.Theseaspectsarecollectivelyknownasusability’.
2.2. MethodsandtoolstomeasureusabilityandusersatisfactionTheusabilitycanbeevaluatedbymeansofasubjectiveandobjectivemeasurement,wherebythesubjectiveevaluationofusabilityincludesscalesandquestionnaires,whilstobjectivemeasurescomprisebehaviouralorphysiologicalmeasures.Theusabilityisalsomeasuredbyorwithhelpofexpertevaluators,andbyend-usersintermsofusersatisfaction.
Themost commonusability evaluation techniques are inspectionmethods and testmethods. Inspectionmethods identify usability problems and possibilities for improvement by checking it against establishedstandards. Usability test methods include testing usability with end users, in the presence of expertevaluators or by means of subjective user satisfaction questionnaires (Holzinger, 2005). Although it iscommon to understand the expert usability evaluation as an objectivemeasurement,we describe threegroupsofusabilitymeasurementmethods:expertusabilityevaluation,end-usersatisfactionandobjectivemeasurement,understoodasmonitoringbehaviouralorphysiologicaldata.
2.2.1ExpertusabilityevaluationSeveralapproachescanbeusedtoevaluatetheinterfaceusability(Lin,Choong&Salvendy,1997),e.g.:
• Thinking-aloud approach – end-users think aloudwhile they are using the system that is beingevaluated.Althoughthismethodincludesend-users,thereisanexpertevaluatortointerpretandassessuserreactionstothesystem.Itisaformofatestmethod.
• Guidelines/checklists–interfaceguidelinestobefollowedbydesigners.Itisaformofausabilityinspectionmethod.
4
• Heuristicevaluation-istheapplication(byevaluators)ofasetofheuristicstojudgetheadequacyoftheprototypedesign.Itisaformofausabilityinspectionmethod.
Ovaska(1991,p.51)noticedthatusabilityinspectionmethodssuchasguidanceandchecklistaredesignedto be used as engineering tools. Usually, they involve testing the interface in laboratory settings usingexample tasks. Themaindisadvantageof suchmeasures is that theyevaluate inadequateaspectsof thesystem,becausesomeaspectsofthesystemparticularlyimportanttouserscouldbeoverlooked(Ovaska,1991,p.51).
Guidance/checklistsareanexampleofameasureusedbythedevelopmentteamtogetherwiththeusersorexternalevaluators.Theevaluationisconductedonthebasisofanexemplarytaskdesignedbythesystemdevelopers after the user completes the task. An example of such amethod is the Evaluation ChecklistdevelopedbyRavdenandJohnson(1989).Thischecklistincludesthefollowingusabilityaspects:
• Visualclarity• Consistency• Compatibility• Informativefeedback• Explicitness• Appropriatefunctionality• Flexibilityandcontrol• Errorpreventionandcorrection• Userguidanceandsupport
The tool proposed by Ravden and Johnson (1989) consists of 9 sections, each from 13 to 17 questions.Examplequestionsare:‘Doesthesystemvalidateuserinputsbeforeprocessing,whereverpossible?’,‘Whereinterfacemetaphorsareusedaretheyrelevanttothetaskscarriedoutwiththesystem?’.AsOvaskanoticed,suchachecklistcanbedifficultfortheuserstofillinbecauseofdevelopers’terminologyorthelengthofthetool.Italsodoesnotincludeuserfeedbackandsuggestionsoncorrectiveactions.
Mostpopularusabilityinspectionmethodsareheuristicevaluation,togetherwithcognitivewalkthroughandheuristicwalkthrough(Cocktonetal.,2008).
• Cognitivewalkthroughprovidesevaluatorswithstep-by-stepinstructionsontheactionsequencetobe performed by users, where evaluators should determine whether the user would follow thecorrectaction(Lucas&Babaian,2015,p.844).Questionstobeansweredare:
- Willtheuserbetryingtoachievetherighteffect?- Willtheusernoticethatthecorrectactionisavailable?- Willtheuserassociatethecorrectactionwiththedesiredeffect?- Ifthecorrectactionisperformed,willtheuserseethatprogressisbeingmade?
Althoughthismethodprovedtobeabletoidentifyuniqueproblems,notidentifiedusingothermethods,italso has a tendency to overlook problems related to missing functionalities or error recovery (Lucas &Babaian,2015).
• Heuristicevaluation(Nielsen,1994)isamethodwhereanevaluatorassessesapartoftheinterfaceaccordingtothefollowingheuristics:
- Visibilityofsystemstatus- Matchbetweensystemandtherealworld- Usercontrolandfreedom- Consistencyandstandards- Errorprevention
5
- Recognitionratherthanrecall- Flexibilityandefficiencyofuse- Aestheticandminimalistdesign- Helpsusersrecognise,diagnoseandrecoverfromerrors- Helpanddocumentation
AsLucasandBabaianstated(2015),evaluatorsusingthismethodaregivenlittleguidanceconcerningtheusersorthetaskstoperform.
• Heuristicwalkthrough(Sears,1997)isatwo-passprocess,whereevaluatorsareprovidedwithalistofheuristicsandquestions,butalsowithaprioritisedlistofusertasks.Thismethodincludesafree-formandtask-specificevaluation.Thismethod,comparingtothecognitivewalkthroughandheuristicevaluation,wasproventoidentifymoreintermediate,minorandsevereusabilityproblemsthanthecognitivewalkthrough.Theheuristicwalkthroughandcognitivewalkthroughmethodstendedtoyieldlessfalsepositivesthantheheuristic evaluation, but this is explainedby the fact that theheuristic evaluationmethoddoesnot testspecifictasks,unliketwoothermethods(Lucas&Babaian,2015).
Anothermethodforexpertevaluationoftheuserinterfaceusabilityisthecollaborativecritique,designedfortestingHuman–ComputerCollaboration(Lucas&Babaian,2015).Theaimofthismethodistolowermentalloadofusers.Accordingtothisconcept,thesystemshouldserveitsuserstomeettheirgoals.Hence,thefocusisonthesystemusabilityinthecontextofthecurrenttask,userandinteraction,andnotonusercapabilities touse thesystemefficiently.Collaborativecritiquemethodconducts theassessment stepbystep. On every stage, experts-evaluators answer questions assessing the quality and possibility of thecollaborationandthesystem’sreactionstoerrors.Evaluatorsareaskedtofamiliarisethemselveswithagiventaskandemployee’sprofile.Everytaskisdividedintosub-taskswhicharethestagestobeevaluated.Theevaluators should be supported by experts in the field in the process of usability assessment. Thecollaborativecritiquemethodisconductedusingthefollowingquestions:
• Willtheuserfindtheoptionsforwhatshewantstodointhecurrentscreen?• Fortheusertofigureoutwhattodonow,howmuchexplorationisinvolved?• Fortheusertofigureoutwhattodonow,howmuchconfusionisinvolved?• Isthesystemusingknowledgeofthetaskingeneral,thecurrentuser,andthecontextofthecurrent
actiontothefullestextentinordertoa)appropriatelyguidetheuser?;b)reducetheeffortinvolvedinuserinput?
• Afterexecutionofthecurrentaction,willtheuserunderstand:a)whatprogresshasbeenmadesofartowardcompletingtheoveralltask?;b)whatdoesremaintobedoneinordertocompletetheoveralltask?
• Doesthesystemdisplayinformationthatclearlyexplainstheproblemtotheuser?• Doesthesystempresentstepstheusercantakeforpossiblecorrectiveactions?• Doesthesystempresentaneasywaytotakecorrectiveactions?
Althoughthismethodseemstobecomprehensiveinthecontextofthesystem’sabilitytocollaboratewiththe user, it does not contain themain usability components and amore subjective dimension, e.g. usersatisfaction.Moreover,theevaluationhastobeconductedbyusabilityexpertsandexpertsinthefield,notbyemployees.
The usability testing conducted by external expert evaluators usually in laboratory settings is a popularmethodtodesignusablesystemsorproducts.However,itischaracterisedbysomedisadvantagessuchaslackoftheend-userinvolvementintheprocessorlackofarealindustrialenvironmentandafailuretotesttheusability in everydayworkingprocesses. Including theuser assessmentof usability seems tobeof aparticularlyhighimportance.
6
2.2.2End-usersatisfactionevaluationOther approach includes measuring end-user satisfaction with the Human-Machine Interaction/Human-Machine Interfacewhich isalsoa formofusabilitymeasurement (Ovaska,1991).Theuser satisfaction isconsideredasthemostprevalentmeasureofaninformationsystem’ssuccessthankstoitsapplicabilityandeaseofuse,anditisdirectlyrelatedtothesystem’ssuccess(Zviran&Erlich,2003).Althoughthesubjectiveevaluation of usability tends to be neglected in favour of objective performancemeasures, Lund (2001)stated that the subjective measurement is most closely related to user behaviour. Other authors alsosuggestedthatlesstangiblefactorsofusability(e.g.enjoyment)arebecomingofahigherimportanceandshouldbeincorporatedinthesystemusabilitytesting(Linetal.,1991).
Themostpopularandwidely-usedtoolsare:theComputerUserSatisfactionscale(CUS),theQuestionnaireforUserInteractionSatisfaction(QUIS),theSystemUsabilityScale(SUS),andtheUSEQuestionnaire,whicharecharacterisedbelow.
TheComputerUserSatisfaction(CUS)questionnaire(Bailey&Pearson,1983)isa39-factorscale,witheachfactormeasuredwithfiveratings.Amongthe39factors,thereare:confidenceinthesystems,securityofdata,outputformat,convenienceofaccess,personaljobeffectsresultingfromthecomputer-basedsupport,precisionofinformationoutputandsystemflexibility,andothers.
Thefirstfourresponseitemsareforqualityratingsandthefifthisanimportancerating.Authorsfoundthatmostimportantfactorswere:accuracy,reliability,timeliness,relevancyandconfidenceinthesystem.Thefactorsofleastimportancewere:feelingsofcontrol,volumeofoutput,vendorsupport,degreeoftraining,andorganisationalpositionofelectronicdataprocessing.
Eachitemhastobeansweredusinga7-pointscaleandthelengthofthequestionnairecouldresultinerrorsofattrition(Ives,OlsonandBaroudi(1983).
QuestionnaireforUserInterfaceSatisfaction(QUIS;Chinetal.,1988)containsfiveareasofusersatisfaction:
• Overallreactiontothesoftware• Screen-Readingcharactersonthescreen,organisationofinformationandsequenceofscreens• Terminologyandsysteminformation–useof termsthroughoutsystem,terminology,positionon
messagesonscreen,promptsforinput,errormessages• Learning–Learningtooperatethesystem,performingtasks• Systemcapabilities–systemspeed,systemreliability,noiselevel,designforalllevelsofusers
The answers are pair-wise/dichotomous and focusedmainly on interface’s consistency, helpfulness andclearness.
TheQUISquestionnaireconsistsoffourfactors:screen,terminologyandsysteminformation,learning,andsystemcapabilities.however,factoranalysisfailedtoconfirmedtheproper loadingofdefinedfactors.AsLewisnoticed(1995,p.61),“thelackofcorrespondencebetweenitemgroupsandunderlyingfactorsandthepaucityofinformationregardingsubscalereliabilityareproblemsthatlimittheusefulnessoftheQUIS”.
Computer System Usability Questionnaire (CSUQ; Lewis, 1995) contains 19 questions measuring suchcomputersystemcharacteristicsas:
• Easeofuse• Easeoflearning• Simplicity• Effectiveness• Information• Userinterface
7
Exampleitemsare:“Overall,Iamsatisfiedwithhoweasyistousethissystem”,“Ifeelcomfortableusingthissystem”,“Theinterfaceofthissystemispleasant”.Thescaleisreliable,Cronbach’salpha=0.95forthewholescaleandhighCronbach’salphacoefficient for thesubscales: systemusefulness, informationqualityandinterfacequality. thequestionnaireproved tobe reliableandusefulalso innon-laboratorysettingand isopentousebyresearchers.
System Usability Scale (SUS) is a ten-item scale developed by Brooke (1986). It is a validated tool formeasuring theusabilityof awide varietyofproducts and services, includinghardware, software,mobiledevices,websites,applications.Itwasdesignedtomeettheneedofashort,simpletoolthatcouldbeusedinindustrialsettings.SUShasbeenmadefreelyavailableforuseinusabilityassessment.Theexemplaryitemsare:“IthinkthatIwouldliketousethissystemfrequently”,“Ifoundthesystemunnecessarilycomplex”,“Ifeltveryconfidentusing thesystem”.However, there isno informationon theusability/usersatisfactionprinciplesthatwoulddeterminefactorsincludedinthetool.ResultsfromtheanalysisofBangor,KortumandMiller(2008)showthattheSUSisahighlyrobustandversatiletoolforusabilityprofessionals.
USE Questionnaire is a public-access tool developed by Lund (2001), dedicated to measure the mostimportant dimensions of usability for users, across domains. It can measure the usability of interface,software,hardware,services,usersupportmaterials.Thequestionnairecontains30items(17itemsintheshort-formversion)onfoursubscales,correspondingtomostimportantusabilitydimensions:
• Usefulness• EaseofUse• EaseofLearning• Satisfaction
Example itemsare:“ItmakesthethingsIwanttoaccomplisheasiertogetdone”,“ItrequiresthefeweststepspossibletoaccomplishwhatIwanttodowithit”,“Ilearnedtouseitquickly”,“Itiswonderful”.
Lesscommonquestionnairesare:ASQ,CUSI,UISandPUTQ.
TheAfter-ScenarioQuestionnaire(ASQ) isanothertooldevelopedasapartoftheIBMquestionnaireset(Lewis,1995).Itcontainsonlythreequestionscorrespondingtothefollowingsystemusabilitysatisfactionfactors:easeoftaskcompletion,timetocompleteatask,adequacyofsupportinformation.Theexemplaryitemis:“Overall,Iamsatisfiedwiththeamountoftimeittooktocompletethetaskinthisscenario”.Thelengthofthequestionnaire isanadvantagebut it isaquestionnairedesignedfortheusabilityexpertstoanswerusingpredefined scenarios and tasks.However, thequestionnairehasbeenalso validated in theoffice-applicationstudies.IthasbeenshownthatthreeASQitemsshouldbecondensedintoasinglescale.Thatiswhyitcannotbespecified,whichusabilitydimensionneedsanimprovement.
ComputerSatisfactionInventory(CUSI)isascaledevelopedbyKirakowskiandDillon(1988;in:Lewis,1995).Itisa22-itemquestionnairecontainingtwosubscales:affectandcompetence.competencefactoraddressesusers’feelingofmasteryoverthecomputersystem.affectfactoraddressesusers’feelingoffearorpleasure.theoverallinternalconsistencyis0.94,with0.91foraffectand0.89forcompetence.itisdesignedtobeusedinlaboratoryenvironmentwithtasksdevelopedbyresearchers.UnliketheQUIS,researcherswhowishtousetheCUSImustpurchaseitfromtheauthors,whichlimitsitsgeneralavailabilityandusefulness(Lewis,1995,p.61).Also,Ovaskastated(1991,p.56)thatalthoughthismeasureisvalidated,“usabilityismuchmorethanuserfriendlinessorusercompetenceandaffect”.
UserInformationSatisfaction(UIS)isashortformoftheCUSquestionnaire,developedbyIvesetal.(1983).Thisquestionnairecontains13 factorswith tworatingspereach factor (insteadof five)withmaintainingsatisfactorypsychometriccharacteristics.Unfortunately,authorshavenotpublishedthefullversionofthequestionnairebutitisavailableuponrequest.
8
PurdueUsabilityTestingQuestionnaire(PUTQ)isaquestionnairedevelopedbyLinetal.(1997).Themainadvantageofthistool is that itwasdedicatedfortheendusers.Thequestionnaire isbasedonfollowingusabilityprinciples:
• Compatibility• Consistency• Flexibility• Learnability• Minimalaction• Minimalmemoryload• Perceptuallimitation• Userguidance
Althoughtheseprincipleswerededicatedtoman-machinesystems’usabilityingeneral,thequestionnairedevelopedonthebasisoftheseprinciples(PUTQ)wasdesignedspecificallyforaconventionalgraphicaluserinterfacesoftwarewithavisualdisplay,keyboardandmouse,thatiswhyitcanbeusedinspecificusabilityevaluationenvironments.
Zviran and Erlich (2003) suggested that future studies on system usability should includemodernworldchallenges,e.g.securityofthesystem,aswellasotherdimensionsreceivinglessattention,e.g.organisationalsupport.Otherfactorsthatshouldbeconsideredintheindustrialenvironmentarehealth&safetyissuesorthe rapid changes in technology, allowing for adapting the interface to the users’ capabilities and skills.QuestionnairedevelopedintheTask6.1wasanattempttoincludethesefactors.
2.2.3ObjectiveusabilitymeasurementPhysiologicaldata
Themonitoringofphysiologicaldata,suchaseye-tracking(Poole&Ball,2006),stresslevel(Mullins&Treu,1991),humanmovementanalysis(Belda-Loisetal.,2010),HeartRateVariability-HRV,ElectrodermalActivity-EDA,Electroencephalograph–EEG,Electromyography–EMG,CriticalFlickerFrequency -CFF (Hercegfi,2011)isconsideredasausabilitytestingmethodintheHuman-ComputerInteractionenvironment,althoughthereisstilllackofresearchinthisarea.
Behaviouraldata
This group of objective measurement factors can include primary-task performance, secondary-taskperformance, interaction times, time to execute task, number of mistakes, usage time, number ofreports/queriesissuedoveraspecificperiod,numberoffileupdates(Zviran&Erlich,2003;Kampetal.,2001,Harvey, 2009). This sortofmeasurement is regarded tobemoreaccuratebut in somecases it couldbedifficulttoapplyortomeasure,dependingonthefield/environment.TherearefewreasonsconsideredinZviranandErlich(2003)review:suchmethodsareperceivedbybusinessownersasexpensivebecausetheyoftenrequirefinancialinvestment(e.g.installingdedicatedsoftware).Moreover,usersareawareofbeingmonitoredandtheymayaltertheirbehaviourforthemeasurementperiod.Finally,accordingtoZviranandErlich(2003)thesemethodsdonotalwaysreflectthesuccessorusabilityofthesystem.
The subjective andobjectivemethodsofmeasuring systemusability have got their ownadvantages anddisadvantages.However,asKissel(1995)hasdemonstrated,subjectiveratingsofusabilityoftendifferfromobjective performancemeasures. Recognising the value of both objective and subjectivemeasurement,Harvey(2009)proposedtocombinedifferentapproacheswhenmeasuringtheusability.
9
2.3SatisfactionwithworkingeneralThejobsatisfactionisoftendefinedas‘anemotionalreactiontothejob’(e.g.Spector,1997;in:Rafferty&Griffin,2009)oras‘anindividual’sevaluationofthejob,beliefsaboutthejob,andaffectiveexperiencesonthejob’(Weiss,2002;in:Rafferty&Griffin,2009).
Theworkersatisfactionisafactorthatconstituteshealthyandproductivecompanies.Poorjobsatisfactionis relatedwithmental-health problems and increases the risk of sickness absence anddisability pension(Andersenetal.,2017).
Themostcommonmethodofmeasuringworkersatisfactionisaquestionnairemeasurement,usuallybeingapartofworkingconditionssurveys.
TheEuropeanWorkingConditionSurvey (EWCS) includesasingle-itemscaleofsatisfactionwithworkingconditions,whichisconsideredasaprerequisiteforworkermotivation.Answersaregivenona5-pointLikertscale,rangingfrom‘notatallsatisfied’to‘verysatisfied’.Thesurveyhasshownthatfactorspositivelyrelatedto satisfactionwithworkingconditionsare:goodqualityofmanagement,goodwork-lifebalance,havingcareerprospects.Supervisoryroleisalsorelatedtohighersatisfactionwithworkingconditions,whichshouldremindaboutgivingspecialattentiontoworkersonthebottomoftheorganisationalladder.Factorsthatarelikely to leadto lowsatisfactionwithworkingconditionsare:adversesocialbehaviour, feelingthatone’shealthisatriskbecauseofwork,holdingatemporarycontractandhavingexperiencedrestructuringinthecompany.
Intermsofassociationwithjobqualityindices,satisfactionwithworkingconditionsismoststronglyrelatedto social environment, prospects, and skills and discretion. The last factor – skills and discretion – is afoundationofworker’sautonomy/control,whichisoneofmostcrucialaspectsofmaintainingemployees’wellbeing(e.g.Karasek,1979).
Anothersingle-itemscaleisalsowidelyusedinworkingconditionsurveys.Thisquestionis:‘Howsatisfiedareyouwithyourjobingeneral–allthingsconsidered?’(RaffertyandGriffin,2009)
TheCopenhagenPsychosocialQuestionnaire(COPSOQ;Peterjsenetal.,2011)measuresworksatisfactionwithfouritems,askingaboutsatisfactionwithworkprospects,physicalworkingconditions,thewayone’sabilitiesareused,andjobasawhole,everythingtakenintoconsideration.
However,accordingtoRafferty’sandGriffin’sreviewofstudiesmeasuringjobsatisfaction,themostpopularjobsatisfactionmeasureshavebeenMichiganOrganizationalAssessmentQuestionnaire(Cammannetal.,1983;in:Rafferty&Griffin,2009)andJobDescriptiveIndex(Smithetal.,1969;in:Rafferty&Griffin,2009).
TheMichiganOrganizationalAssessmentQuestionnaire(Cammanetal.,1983;in:Rafferty&Griffin,2009)isatoolmeasuringanoveralljobsatisfactionwiththreeitems,e.g.‘Allinall,Iamsatisfiedwithmyjob’.Thisscaledemonstratedasufficientreliabilityanditisapartofaninstrumentmeasuringworkingconditions.
TheJobDescriptiveIndex(JDI;Smithetal.,1969;in:Rafferty&Griffin,2009)measuresjobsatisfactionwith72 items, assessing five dimensions of job satisfaction: satisfaction with work itself, pay, promotionopportunities,supervision,andco-workers.
Theabove-mentionedexamplesofmethodsandtoolstomeasurejobsatisfactioningeneralvarybetween1-itemto72-itemscales.Alloftheseinstrumentsarewidelyusedandvalidated.Thechoiceoftheinstrumentshould be dictated by the research goals, but the length of the scale is an important factor when thequestionnairecontainsotherscalesandshouldberespondent-friendly.
Theworkersatisfactioncanbealsomeasuredbymeansofaphysiologicalparametersmonitoring.Theuseoftheobjectivemeasurementisbasedonscientificevidence,statingthatworkers’wellbeing,includingworksatisfaction,isrelatedtophysiologicalreactiononthelevelofcardiovascularsystem,endocrinesystemor
10
immunesystem(Kuykendall,2015)thatareinvolvedintheorganismstressresponse.Thestimulationofthenervoussystemisreflected,amongothers,inanincreasedhearrate,stimulationofsweatglands.Therefore,thechangesinheartrate,galvanicskinresponse,skintemperaturecanbeusedtoevaluatestressand,inturn,satisfactionlevel.
2.4ConclusionsformeasuringworkersatisfactionintheINCLUSIVEprojectIntheliteraturereview,wepresentedcurrentapproachestothemeasurementofusability/usersatisfactionandsatisfactionwithworkingeneral.Thisreviewhasshownthatusersatisfactionwiththesystem(e.g.HMI)isa formofusability.As themaingroupof interest in the INCLUSIVEprojectareend-users–employeesworkingwiththeadaptiveHMI–wethereforedecidedtoputanemphasisontheirassessmentoftheHMI,insteadofconductingexternalexpertusabilityevaluation.However,thequestionnaireneedstobetailoredto the realworkenvironmentsand thenew,adaptive interface, reflecting threeHMIModules:Measure,Adapt,andTeach(describedinDeliverableD1.1).
Moreover,traditionally,thesatisfactionwithworkiscommonlymeasuredasaffectivereactionstoone’sjob.However, in INCLUSIVEprojectworker satisfaction is understoodas a specific caseofusers’ satisfaction,whichreferstoworkingenvironments,andwork-relatedproductsorservices,includingHMIinterfacesandother ICT-based solutions implemented in industrial machinery control systems and automatedmanufacturing systems. It is a more broad understanding than the classic approach presented in theliteraturereview(althoughsomeofthereviewedtoolsconsideredthesatisfactionwithworkingconditions).Accordingly,we are proposing a user satisfactionwith the adaptiveHMI andworking conditionsmodel,understood as a worker satisfactionmodel. Consequently, we needed to develop a new questionnaire,measuring user satisfaction with HMI, including satisfaction with the three INCLUSIVE modules, andsatisfactionwithpsychosocialandphysicalworkingconditions,takingintoaccountindividualvariables,suchascapabilitiesandskills.
ItisalsoforeseentoanalysetheobjectivedataobtainedintheWP2andWP7testsassatisfactionindicators,inadditiontothesubjectivemeasurement.
11
3. The user satisfaction with the adaptive HMI and working conditionsmodel
WhendevelopingthemodelofHMIusersatisfactionwithwork,weassumedthatsomeotherfactorspresentat the workplace can influence its assessment or are particularly important in the context of inclusiveindustrialenvironment.Thesearephysicalfactors,suchasnoise,temperature,dustorposture,aswellaspsychosocialworkingconditions,suchasautonomy,participation,justiceorsocialsupport.Equally,asthesystemwillprocesssensitivepersonaldata,whichdisclosesbarriersofhumancapabilities,differentethicalandlegalrequirementstoprotecttheuseragainstharmanddisadvantageshavealsobetakenintoaccount.BasedontheELSImodelofethical,socialandlegalaspects(ELSI)whicharedescribedintheDeliverableD1.2–‘Summaryofallsafety,healthandethicsrecommendationsfortheworkingenvironments’wehavealsoincludedsomeethicalandsocialaspectsinthe‘SatisfactionwiththeadaptiveHMIandworkingconditions’model.
Itsmaincomponent,namelysatisfactionwhileworkingwiththeHMI,itselfaccountsforhealthandsafetyrequirements which were identified and specified in the WP1 and described in the abovementionedDeliverableD1.2.
Fig.2:ProposedmodelofworkersatisfactionintheINCLUSIVEprojectframework.
Basedon thismodelwe thendeveloped thequestionnaire thatwould cover allmentioned sections andaspects.
12
4. Thequestionnaire
Before starting to develop the questionnaire, we had reviewed some other existing questionnaires (seeParagraph2).WhenthefirstdraftofoursurveyhadbeenreadywesentittotheConsortiumPartners,sotheycouldcommenton itoradd somemissing issues.After receiving thesecomments,we included themajorityofthemintotheattachedversionofthequestionnaire(seeApendix1).
4.1.Questionnaire’spreambleThisquestionnaire startswithapreambleexplaining thequestionnaire’saim,which is theassessmentofsatisfaction with the implementation of the adaptive HMI, as well as its general structure. In thequestionnaire preamble its potential respondents are assured that survey is anonymous, and that theindividualdatawillnotbedisclosedtoanyoftheirsupervisors.
4.2.OccupationalanddemographicdataThefirstdatatobecollectedinthequestionnaireisthedatarelatedtothecompany,approximatenumberofemployeesandjobpositionstructure.
Astheinteractivesystemisbeingdesignedforageingworkersorworkerswithlanguagefluencydifficulties,thenextsetofitemsrelatestosuchdatafactorsas:age,levelofeducation,nationality, leveloflanguageofficiallyusedintheworkenvironment.
Wearealso interested inusers’workexperience,sothequestionsaboutthe lengthofprofessionalworkexperience,includinganytrainingtoperformcurrenttasks,arealsoposedinthissection.
Fromthepointofviewofsystemadaptability,someaspectsrelatedtousers’healthandcapabilitiesarealsoimportant.Hence,usersareaskedquestionsonpotentialproblemswithvision,hearing,handmovements,precisemovements,memorisinginformation,concentrationorunderstandingandfollowinginstructions.
Attheendofthissection,theuserisaskedtoassesshis/hergeneralhealth,andhowoftenhe/sheisbeingstressedduringworktime.Inordertogetusersfamiliarwithwhatitmeans,thenotionofstresshasbeenexplained.
4.3WorkingConditions
4.3.1PhysicalWorkingConditionsInthissection,weaskusershowfrequentlytheyareexposedtodifficultworkingenvironmentconditionssuchas:excessivenoise,extremetemperatures,dust,toobright/toodarklight.Wealsoaskwhetheruser’sworkrequiretomaintainawkwardbodyposition,lifting,bendingorhandsup.
Theanswersformatisa4-pointLikertscalewhere4=mostofthetime,and1=never.
4.3.2PsychosocialWorkingConditionsInordertocoverallpossiblepsychosocialconditionspresentattheworkplace,weincludedastandardisedandwellvalidatedscale-TheCopenhagenPsychosocialQuestionnaire(COPSOQ)developedbyKristensenetal.(2005)initsrevisedversion(Pejtersenetal.,2010).Conceptually,itincludesthemaindimensionsofthe most influential psychosocial theories at work, including the Job-Strain, Demand-Control-Support(Karasek&Theorell,1990)andEffort-Reward-Imbalance(Siegriest,1996)models,butalsoothertheoriesandaspectsignoredinprevioustools,forinstanceemotionaldemandsorroleclarity.ThismakesCOPSOQusefulinanyworkplaceeitherintheindustrialorintheservicesbranch.AmongpsychosocialriskassessmenttoolsCOPSOQisunique,becauseitincludespopulation-basedreferencevaluestoassesstheneedforactionand to help the decisionmaking process on preventivemeasures at the workplace level. A strength of
13
COPSOQisthatithasbeentestedinmanycountriesallovertheworld.Thesevalidationstudiesshowthequestionnaire‘scapacityandusabilityinthelocalcontext.
Tryingtokeepthequestionnaireasshortaspossible,wedidnotincludewholescalesfromCOPSOQ,onlychosenquestions,mostappropriatetotheworkingconditionspresentintheinclusiveHMIenvironment.
Weusedquestionsassessingpsychologicalworkdemands,suchas:‘Doyouhaveenoughtimeforyourworktasks?’or‘Doyouhavetokeepyoureyesonalotofthingswhileyouwork?’.Theaspectoflearningnewthingswasalsoincludedusingsuchquestionsas:‘Doesyourworkdemandthatyouaregoodatcomingupwithnewideas?’and‘Doyouhavethepossibilityoflearningnewthingsthroughyourwork?’.
Theworkers’influenceonhowhe/sheperformshis/hertasksisassessedusingthequestionslike:‘Canyouinfluencetheamountofworkassignedtoyou?’and‘Doyouhavealargedegreeofinfluenceconcerningyourwork?’.Socialsupportisanotherimportantaspectofpsychosocialworkingconditions,inthequestionnaireitisassessedusingquestionslike:‘Howoftendoyougethelpandsupportfromyourcolleaguesifyouneedit?’,‘Howoftencanyougethelpandsupportfromyournearestsuperiorifyouneedit?’.
Themeaningofworkwillbeevaluatedusingthequestion:‘Doyoufeelthattheworkyoudoisimportant?’,andmanagementstyleintheorganisationwiththequestions,suchas‘Doyoufeelmotivatedandinvolvedinyourwork?’or‘Isyourworkrecognisedandappreciatedbythemanagement?’.
An importantfactorofworker’ssatisfaction isalsoorganisational justice,whichwillbeassessedwiththequestions, such as: ‘Are you treated fairly at yourworkplace?’ or ‘Is thework distributed fairly?’.Workinsecurity,specificallyrelatedtothemoderntechnologiescouldalsobeasignificantpredictorofworker’sfearandworkdissatisfaction,inthequestionnaireitwouldbemeasuredwiththequestions:‘Areyouworriedaboutnewtechnologymakingyouredundant?’and‘Areyouworriedaboutbeingtransferredtoanotherjobagainstyourwill?’.
Apotentialdiscriminationcouldalsosignificantlyinfluencetheworker’ssatisfaction,wearegoingtocheckitusing the followingquestions: ‘Is therespace foremployeesofadifferent raceandreligion?’, ‘Is therespaceforbothmenandwomen?’,‘Istherespaceforelderlyemployees?’or‘Istherespaceforemployeeswithvariousillnessesordisabilities?’.
4.4SatisfactionwiththeadaptiveHMI
4.4.1SafetyAsmentionedaboveintheParagraph3andbasedonDeliverableD1.2–‘Summaryofallsafety,healthandethicsrecommendationsfortheworkingenvironments’,wearegoingtoaskwhetherthesafetyfunctions(Emergencystop,Guardlockingfunctions,Indicationsandalarms)andthecontrolbuttons(ManualReset,Mode selection/muting, Hold-To-Run, Enabling Device Function, Two-hand control function, Locking –unlockingofthepanel)are:clearlyidentifiable,clearlyvisible,readilyaccessible.Similarly,errormessagesand warningmessages are very important from the point of view of safety, so the question ‘Are errormessagesandwarningmessagesclear,informative/sufficientlydetailed,unambiguous?’isalsoincludedinthissection.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
4.4.2Satisfactionwiththedesign/visibilityoftheinterfaceIn this section, questions on characters, visibility and clarity of texts/messages, signs/symbols on theinterface,buttonswereincluded.Wealsoaskusersaboutthesequenceofscreens,positionofmessagesonthescreen,thecoloursusedintheHMI,theHMIlayout.Exampleitemsonthisscaleare: ‘Ingeneral,theorganisationofinformationisclear’,‘Thesoundsdistractand/orannoyme’.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
14
4.4.3SatisfactionwitheaseInthissection,auserisaskedabouttheeaseofthesystem–oneoftheusabilityprinciplesdefinedintheParagraph2.1. Itmeans that thesystem iseasy to learnandbecomefamiliarisedwithaswellas timeoflearningisminimal.Theitemsrelatedtotheprocessofbecomingfamiliarwiththesystem’sfunctionsandoperations,memorisingthesystem’sfunctionsandoperations,findingtheinformationneeded,performingtasks, etc. Example items are: ‘I can easily withdraw an accidental command/action’, ‘Use of termsthroughoutthesystemisconsistentandunderstandable’.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
4.4.4SatisfactionwithefficiencySatisfactionwithefficiencyisanothersectiondirectlyrelatedtousabilityprinciples.Inthissectiontheuseris asked about the system efficiency. The items cover the amount of information presented, number ofoperations to perform the task, having sense of control over the system, cooperation with themachine/robot.Exampleitemsare‘Ifeelfullyincontrolofthemachine’and‘Ingeneral,theHMIhelpsmetobemoreproductiveinmywork’.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
4.4.5SatisfactionwiththeAdaptmoduleThis section covers satisfaction with one of the three core INCLUSIVE modules: Adapt Module (seeDeliverable D1.1). Users are here asked about their reactions to the newly developed system and itsadaptation to their capabilities and skills. The items verify whether the HMI has been adapted tocapabilities/mental statesanduserguidance.Example itemsare ‘I canget startedeasilyon the system’snewlyaddedfunctions’,‘IfeelImakelessmistakes/errorsusingtheadaptiveHMI’,‘IfeelIcancompletethetasksevenifIamtired’.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
4.4.6SatisfactionwiththeMeasuremoduleAccordingly, this section covers satisfaction with the Measure Module (see Deliverable D1.1) – usersreactionstomonitoringtheirphysiologicalparameterswithaneye-tracker,wristbandorspeechdetector.Itemsareprecededbyashortexplanationoftheaimofthemeasurementmodule,i.e.enablingthesystemtodetecthigherstresslevelsandtoreact(adapt).Exampleitemsare:‘Ifeelthatmonitoringmystraincanbeadvantageousforme’,‘Ifeelitcanchallengemyphysicalcomfort’.
Theanswerformatwasa4-pointLikertscale,where0–Never/HardlyEver,1–Seldom,2–Sometimes,3–Often,4–Always.
4.4.7SatisfactionwiththeTeachmoduleThissectioncoversuserreactionstotheTeachModule(seeDeliverableD1.1).Asthismoduleincludesbothan on-line and off-line teaching system, items have also been developed to measure satisfaction withdifferent teaching techniques. Example items are ‘The chosen way of assistance (AR-based assistance,Speech-basedassistance,Supportassistance)intheon-lineteachingwasappropriate’,‘Theoff-lineteachingwashelpfultomastertheHMI’.
Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.
4.4.8OverallSatisfactionwiththeadaptiveHMIThefinalsectioncoverstheoverallsatisfactionwiththeadaptiveHMIanditismeasuredwiththequestion:‘Regarding the adaptive HMI in general. How pleased are youwith it as awhole, everything taken intoconsideration?’.
15
Theanswerformatisa5-pointLikertscale,where1=Verysatisfied,and5=Veryunsatisfied.
Inorder toensure theengagementofworkers in thedesignandadaptationof theirworkplaces to theirneeds, which is the prerequisite to ensure the attractiveness of these novel solutions for workers, thequestionnaireswillincludecollectedusers’opinionsand/orproposalsonpreferredmodificationsoftheHMIdesignanditsoperatingmode.Thereforeweasktwoopenquestions:‘Ifyouaresatisfied,whatdoyoulikethemostintheadaptiveHMI?’and‘WhatshouldbeimprovedintheadaptiveHMI?’.
16
5. ObjectivemeasurementofworkersatisfactionAsmentionedearlier, the aimof theWP6 is tomeasureworker satisfaction, both subjectively (with thequestionnairedeveloped)andobjectively.Inordertoconductanobjectiveassessment,thedataobtainedfromtheobjectiveindicatorsformeasuringworkers’capabilityinthedemonstrators(WP7)willbeanalysed.The results of real-time measurements of indicators selected in WP2 (e.g. eye-tracking, galvanic skinresponse)will be correlatedwith the results of questionnaire-based subjectivemeasurementsofworkersatisfaction.
17
6. ConclusionTheaimofthisDeliverablewastoproposemethodsandtoolstomeasureworkersatisfactioninframeworkoftheINCLUSIVEproject.Theliteraturereviewconductedaspartofthisworkpackagetaskhasdescribedthecommonlyusedmethodsforassessingsystem’susability,usersatisfactionandsatisfactionwithwork.BasedonthisreviewaswellasonpreviousDeliverables(D1.2),wehaveselectedfactorstobeconsideredintheworkersatisfactionmodelandthequestionnairetomeasureworkersatisfactionwiththeadaptiveHMIandworkingconditions.Thequestionnairewillbetranslated,pilottestedandadjustedintheTask6.2.
18
7.ReferencesAndersen,L.L.,Fishwick,D.,Robinson,E.,Wiezer,N.M.,Mockałło,Z.,Grosjean,V.(2017).Jobsatisfactionis
morethanafruitbasket,healthchecksandfreeexercise:Cross-sectionalstudyamong10,000wageearners.ScandJPublicHealth,45(5),476-484.
Bailey, J.E.& Pearson, S.W. (1983). Development of a Tool forMeasuring andAnalyzing ComputerUserSatisfaction.ManagementScience,29,530-545.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale.InternationalJournalofHuman-ComputerInteraction,24(6),574-594.
Belda-Lois,J.-M.,deRosario,H.,Pons,R.,Poveda,R.,Morón,A.,Porcar,R.,...Gómez,A.(2010).Canhumanmovementanalysiscontributetousabilityunderstanding?HumanMovementScience,29(4),529-541.
Bevan,N.,Kirakowski,J.andMaissel,J.(1991).Whatisusability?In:H.J.Bullinger(Ed.):Proceedingsofthe4thInternationalConferenceonHumanComputerInteraction,Stuttgart,September1991.Elsevier.
Brooke,J.(1996).SUS:Aquickanddirtyusabilityscale.UsabilityEvaluationinIndustry,189(194),4–10.
Chin,J.P.,Diehl,V.A.,Norman,K.L.(1988).DevelopmentofanInstrumentMeasuringUserSatisfactionoftheHuman-ComputerInterface.ACMCHI'88Proceedings,213-218.
Cockton,G.,Woolrych,A.,&Lavery,D.(2008).Inspection-basedevaluations.InJ.A.Jacko&A.Sears(Eds.),The human–computer interaction handbook: Fundamentals, evolving technologies and emergingapplications(2nded.),pp.1171–1190.Hillsdale,NJ:Erlbaum
Eurofound(2016).SixthEuropeanWorkingConditionsSurvey–Overviewreport.PublicationsOfficeoftheEuropeanUnion,Luxembourg.
Harvey,C.(2009).DevelopmentofaMethodforEvaluatingtheUsabilityofIn-VehicleInformationSystems(IVISs).Human-ComputerInteraction-INTERACT2009,5727,856-859.
Hercegfi,K.(2011).HeartRateVariabilityMonitoringduringHuman-ComputerInteraction.ActaPolytechnicaHungarica,8,205-224.
Holzinger,A.(2005).Usabilityengineeringmethodsforsoftwaredevelopers.CommunicationsoftheACH,48,71-74.
Ives, B., Olson, M.H. & Baroudi, J.J. (1983). The Measurement of User Information Satisfaction.CommunicationoftheACM,26,785-793.
Kamp,J.F.,Marin-Lamellet,C.,Forzy,J.F.,Causeur,D.(2001).HMIaspectsoftheusabilityofinternetserviceswithanin-carterminalonadrivingsimulator.IATSSRESEARCH,25,29-39.
Karasek,R. (1979).JobDemands,JobDecisionLatitude,andMentalStrain: ImplicationsforJobRedesign.AdministrativeScienceQuarterly,24,285-308.
Karasek,R.A.,&Theorell,T.(1990).Healthywork:stress,productivityandthereconstructionofworkinglife.NewYork:BasicBooks.
Kissel, G.V. (1995). The effect of computer experience on subjective and objective software usabilitymeasures.CHI’95ConferenceCompaniononHumanFactorsinComputingSystemsProceeding,284-285.
19
Kristensen,T.S.,Hannerz,H.,Høgh,A.,andBorg,V.(2005).CopenhagenPsychosocialQuestionnaire–atoolfortheassessmentandimprovementofthepsychosocialworkenvironment.ScandJWorkEnvironHealth,31(6),438-449.
Lewis, J.R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation andInstructionsforUse.InternationalJournalofHumanComputerInteractions,7,57-78.
Lin,H.X.,Choong,Y.-Y.&Salvendy,G.(1997).Aproposedindexofusability:Amethodforcomparingtherelativeusabilityofdifferentsoftwaresystems.Behaviour&InformationTechnology,16,267-277.
Lucas,W.andBabaian,T.(2015).TheCollaborativeCritique:AnInspectionMethodforExpertEvaluationofUserInterfaces.InternationalJournalofHuman–ComputerInteraction,31,843-859.
Lund,A.M.(2001).MeasuringusabilitywiththeUSEquestionnaire.UsabilityInterface,8(2),3-6.
Mullins, P.M. & Treu, S. (1991). Measurement of stress to gauge user satisfaction with features of thecomputerinterface.Behaviour&InformationTechnology,10(4),325-343.
Nielsen,J.(1993).Usabilityengineering.Boston:AcademicPress.
Nielsen,J.(1994).Heuristicevaluation.InJ.Nielsen&R.L.Mack(Eds.),Usabilityinspectionmethodspp.25–62.NewYork,NY:Wiley&Sons.
Norman,D.A.(2002).ThedesignofeverydaythingsNewYork:BasicBooks.
Ovaska, S. (1991). Usability as a Goal for the Design of Computer Systems. Scandinavian Journal ofInformationSystems,3,47-62.
Pejtersen, J.H., Kristensen, T.S., Borg,V. andBjorner, J.B. (2010). The secondversionof theCopenhagenPsychosocialQuestionnaire.ScandinavianJournalofPublicHealth,38,8–24.
Poole,A.&Ball,L.J.(2006).EyeTrackinginHCIandUsabilityResearch.EncyclopediaofHumanComputerInteraction,211-219.
Preece,J.(1993).AGuidetoUsability:HumanFactorsinComputing,Addison-Wesley,Wokingham
Rafferty,A.&Griffin,M. (2009). Job satisfaction inorganizational research. In:D.Buchanan&A.Brynan(Eds.),TheSageHandbookofOrganizationalResearchMethods.196-212.
Ravden, S. & Johnson, G. (1989). Evaluating Usability of Human-Computer Interfaces, Chichester: Ellis-Horwood
Rogers,Y.,Sharp,H.&Preece,J.(2011).Interactiondesign:Beyondhuman-computerinteraction,3rded.USA:JohnWiley&Sons.
Sears,A. (1997).Heuristicwalkthroughs:Findingtheproblemswithoutthenoise. International JournalofHuman–ComputerInteraction,9,213–234.
Shackel,B.(1991).Usability-context,framework,designandevaluation.InB.Shackel&S.Richardson(Eds.),Humanfactorsforinformaticsusability,pp.21-38.Cambridge:CambridgeUniversityPress
Siegrist, J. (1996). Adverse health effects of high-effort/low-reward conditions. Journal of OccupationalHealthPsychology,1(1),27-41.
Wang,Ch.-M.&Huang,Ch.-H.(2015).AStudyofUsabilityPrinciplesandInterfaceDesignforMobilee-Books.Ergonomics,58,1-19.
20
Yeh,C.-J.(2010).Theprinciplesofinteractiondesigninthepost-digitalage.Taipei:ARTIST-MAGAZINE
Zviran,M.&Erlich,Z.(2003).MeasuringISUserSatisfaction:ReviewandImplications.CommunicationsoftheAssociationforInformationSystems,12,81-103.
21
Appendix1– ‘SatisfactionwiththeadaptiveHMIandworkingconditions’questionnaire
22
SatisfactionwiththeadaptiveHMIandworkingconditions
DearParticipant!
ThesurveyisconductedwithintheInclusiveProjectthatyourcompanyisinvolvedin.
Thisquestionnaireisaimedatassessmentofsatisfactionwiththeimplementationoftheadaptivehuman
machineinterface(HMI).Itconsistsoftwomainparts:satisfactionwithworkingconditionsandsatisfaction
withthenewadaptiveinterface.Thesurveyisanonymous.Theindividualdatawillnotbeavailabletoany
ofyoursupervisors.Wearereallyinterestedinwhatyouthinkabouttheadaptiveinterface.Weencourage
youtogiveyourhonestopinion!
DemographicDATA
1. Age__________________
2. Gender:qMan qWoman
3. Levelofeducation:
qPrimaryschool qSecondaryschool qUniversity(Bachelor)qUniversity(Master)
4. Forhowlongdoyouworkinthecompany?___________
5. Howlonghaveyoubeeninvolvedinthepresentworktasks?___________
6. Didyoureceiveanytrainingtoperformyourcurrenttask?qYESqNO
7. Nationality?………………………………………
8. Isyournationalitythesameasthemajorityofyourcolleagues?qYESqNO
9. Yourleveloflanguageofficiallyusedinyourworkenvironmentis:
qBasicqCommunicativeqAdvancedqNativespeaker
Company:
Code:
Approximatenumberofemployees:
JobPosition:
23
HEALTHANDCAPABILITIES
ERSONA
1. Ihaveproblemswithvision:qYES(checkbelow)qNO
IfYES:qNearqFarqIamcolorblindqOther(pleasedescribe)………………………………….
2. Ifananswertothequestion1isyes,myvisionwithcorrectionglasses/lensesis:
qExcellentqGoodqModerateqPoor
3. Ihaveproblemwithhearing:qYESqNO
4. Ifananswertothequestion3isyes,myhearingwithhearingaidis:
qExcellentqGoodqModerateqPoor
5. Ihaveproblemswithmovingmyhands:
qNotatallqToasmallextentqSomewhatqToalargeextent
6. Ihaveproblemswithprecisemovements(e.g.manipulatingsmallobjects):
qNotatallqToasmallextentqSomewhatqToalargeextent
7. Ihavedifficultiestorememberthings:
qNotatallqToasmallextentqSomewhatqToalargeextent
8. IngeneralIhaveproblemswithconcentration:
qNotatallqToasmallextentqSomewhatqToalargeextent
9. Ihaveproblemswithunderstandinginstructionsandfollowinginstructions:
qNotatallqToasmallextentqSomewhatqToalargeextent
10. Ingeneral,Iwouldsaymyhealthis:
qExcellentqVerygoodqGoodqFairqPoor
11. Howoftenhaveyoubeenstressed*duringthelast4weeks:
qEverydayqMostoftheweekqOnceaweekqNever/HardlyEver
*Stressmeansthesituationwhenapersonfeelstense,restless,nervous,oranxious,orisunabletosleepatnightbecausehismindistroubledallthetime.
I.WorkingConditionsPHYSICALWORKINGCONDITIONS
1. Duringworkareyoufrequentlyexposedtodifficultconditionssuchas:
a) ExcessiveNoise qMostofthetimeqSometimesqRarelyqNever
b) ExtremeTemperatures qMostofthetimeqSometimesqRarelyqNever
c) Dust qMostofthetimeqSometimesqRarelyqNever
d) Toobright/toodarklight qMostofthetimeqSometimesqRarelyqNever
24
2. Doesyourworkrequiretomaintain:
a) Awkwardbodyposition qMostofthetimeqSometimesqRarelyqNever
b) Lifting qMostofthetimeqSometimesqRarelyqNever
c) Bending qMostofthetimeqSometimesqRarelyqNever
d) Handsup qMostofthetimeqSometimesqRarelyqNever
e) Other
(Short description)………………………………………………..............………………………………………………………………………
………………………………………………………………………………………………………………………………………
PSYCHOSOCIALWORKINGCONDITIONS
0–Never/HardlyEver1–Seldom2-Sometimes3–Often4–Always
Never/H
ardlyEver
Seldom
Sometim
es
Often
Always
1. Doyouhaveenoughtimeforyourworktasks? 0 1 2 3 4
2. Doyouhavetokeepyoureyesonalotsofthingswhileyouwork? 0 1 2 3 4
3. Doesyourworkrequirethatyourememberalotofthings? 0 1 2 3 4
4. Doesyourworkdemandthatyouaregoodatcomingupwithnew
ideas?0 1 2 3 4
5. Doesyourworkrequireyoutomakedifficultdecisions? 0 1 2 3 4
6. Canyouinfluencetheamountofworkassignedtoyou? 0 1 2 3 4
7. Doyouhavealargedegreeofinfluenceconcerningyourwork? 0 1 2 3 4
8. Howoftendoyougethelpandsupportfromyourcolleaguesifyou
needit?0 1 2 3 4
9. How often can you get help and support from your nearest
superiorifyouneedit?0 1 2 3 4
10. Doyoufeelpartofacommunityatyourplaceofwork? 0 1 2 3 4
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent
Toaverysm
all
extent
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylarge
extent
25
11. Isitnecessarytokeepworkingatahighpace? 0 1 2 3 4
12. Doyouhavethepossibilityof learningnewthings throughyour
work?0 1 2 3 4
13. Doesyourworkrequireyoutotaketheinitiative? 0 1 2 3 4
14. Canyouuseyourskillsorexpertiseinyourwork? 0 1 2 3 4
15. Doesyourworkgiveyoutheopportunitytodevelopyourskills?
0 1 2 3 4
16. Doyoufeelthattheworkyoudoisimportant? 0 1 2 3 4
17. Doyoufeelmotivatedandinvolvedinyourwork? 0 1 2 3 4
18. Isyourworkrecognizedandappreciatedbythemanagement? 0 1 2 3 4
19. Areyoutreatedfairlyatyourworkplace? 0 1 2 3 4
20. Areyouworriedaboutnewtechnologymakingyouredundant? 0 1 2 3 4
21. Are youworriedaboutbeing transferred toanother jobagainst
yourwill?0 1 2 3 4
22. Areconflictsresolvedinafairway? 0 1 2 3 4
23. Areemployeesappreciatedwhentheyhavedoneagoodjob? 0 1 2 3 4
24. Istheworkdistributedfairly? 0 1 2 3 4
25. Istherespaceforemployeesofadifferentraceandreligion? 0 1 2 3 4
26. Istherespaceforbothmenandwomen? 0 1 2 3 4
27. Istherespaceforelderlyemployees? 0 1 2 3 4
28. Istherespaceforemployeeswithvariousillnessesordisabilities? 0 1 2 3 4
OverallSatisfactionwithworkingconditionsRegarding your work in general. How pleased are you with your job as a whole, everything taken intoconsideration?qVerysatisfiedqSatisfiedqNeithersatisfied,nordissatisfiedqUnsatisfiedqVeryunsatisfied
26
II.SatisfactionwiththeadaptiveHumanMachineInterface(HMI)SAFETY
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent
4–Toaverylargeextent5-NotapplicableAccordingtoyouropinion:
Toaverysm
allexten
t
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. Safety functions (Emergency stop, guard locking functions,
indicationsandalarms)are:
a) Clearlyidentifiable 0 1 2 3 4 5
b) Clearlyvisible 0 1 2 3 4 5
c) Readilyaccessible 0 1 2 3 4 5
2. The control buttons (Manual Reset, Mode selection/muting,Hold-To-Run, Enabling Device Function, Two-hand controlfunction,Locking–unlockingofthepanel)are:
d) clearlyidentifiable 0 1 2 3 4 5
e) Clearlyvisible 0 1 2 3 4 5
f) Readilyaccessible 0 1 2 3 4 5
3. Errormessagesandwarningmessagesare:
g) Clear 0 1 2 3 4 5
h) Informative/Sufficientlydetailed 0 1 2 3 4 5
i) Unambiguous 0 1 2 3 4 5
4. Overall,theadaptiveHMIfulfillsallthesafetyfunctions 0 1 2 3 4 5
27
SATISFACTIONWITHTHEDESIGN/VISIBILITYOFTHEINTERFACE
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent5-Notapplicable
Accordingtoyouropinion:
Toaverysm
all
extent
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. Charactersareeasytoread. 0 1 2 3 4 52. Thetexts/messagesontheinterfacearevisible 0 1 2 3 4 53. Thesigns/symbolsontheinterfaceareunambiguous/clear 0 1 2 3 4 54. Theinterfacebuttons(options)arevisibleonthescreen 0 1 2 3 4 55. Thesizeofthebuttons(options)isoptimal(nottoolarge,nottoo
small)0 1 2 3 4 5
6. Sequenceofscreensisappropriate 0 1 2 3 4 57. Positionofmessagesonthescreenisconsistent 0 1 2 3 4 58. ThecoloursusedintheHMIhelptobetterperceivetheinformation
onthescreen0 1 2 3 4 5
9. ThecoloursusedintheHMIdonotdistractme 0 1 2 3 4 510. TheHMIlayoutisaesthetic 0 1 2 3 4 511. TheHMIlayoutisminimalistic 0 1 2 3 4 512. Ingeneral,theorganizationofinformationisclear 0 1 2 3 4 513. Ingeneral,thelayoutoftheadaptiveHMIisappropriate 0 1 2 3 4 514. ThesoundsoftheHMIhelpmetobetteroperatetheHMI. 0 1 2 3 4 515. Thesoundsdistractand/orannoyme. 0 1 2 3 4 5
16. Thechanginginterfacedistractsme 0 1 2 3 4 5
17. IcaneasilyoperatetheadaptiveHMIusingmyhands 0 1 2 3 4 5
28
SATISFACTIONWITHEASE
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent
5-NotapplicableAccordingtoyouropinion:
Toaverysm
all
extent
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. Thesystemiseasytolearn. 0 1 2 3 4 5
2. I quickly familiarized myself with the system's functions andoperations.
0 1 2 3 4 5
3. Itwaseasytomemorizethesystem'sfunctionsandoperations. 0 1 2 3 4 5
4. Useoftermsthroughoutsystemisconsistentandunderstandable. 0 1 2 3 4 5
5. Thesigns/symbolsontheinterfacehelpmetonavigatethroughtheHMI
0 1 2 3 4 5
6. IcaneasilyfindalltheinformationIneed. 0 1 2 3 4 5
7. Icaneasilywithdrawunaccidentalcommand/action. 0 1 2 3 4 5
8. Icaneasilyreturntotheearliersteps. 0 1 2 3 4 5
9. Performingtasksisstraightforward. 0 1 2 3 4 5
SATISFACTIONWITHEFFICCIENCY
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent
4–Toaverylargeextent5-NotapplicableAccordingtoyouropinion:
Toaverysm
allexten
t
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. TheHMIprovidesinsufficientamountofinformation. 0 1 2 3 4 5
2. TheHMIprovidesexcessiveamountofinformation. 0 1 2 3 4 5
3. Thenumberofoperationstoperformatask/toachieveagoal/tosetupaprocessisoptimal.
0 1 2 3 4 5
4. Ifeelfullyincontrolofthemachine. 0 1 2 3 4 5
5. The HMI helps me to more efficiently cooperate with themachine/robot.
0 1 2 3 4 5
6. Ingeneral,theHMIhelpsmetobemoreproductiveinmywork. 0 1 2 3 4 5
29
SATISFACTIONWITHADAPTMODULE
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent
4–Toaverylargeextent5-NotapplicableAccordingtoyouropinion:
Toaverysm
allexten
t
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. Icangetstartedeasilyonthesystem’snewlyaddedfunctions 0 1 2 3 4 5
2. IfeeltheadaptiveHMIadjuststomyactualcapabilities/mentalstates 0 1 2 3 4 5
3. IfeelIcancompletethetasksevenifIamtired 0 1 2 3 4 5
4. IfeelIcanbeeasilyguidedwhenIgetlost/commitanerror 0 1 2 3 4 5
5. IfeellessstressedusingtheadaptiveHMI 0 1 2 3 4 5
6. Itrustthatthesystemallowsmetoperformmytaskssmoothly 0 1 2 3 4 5
7. IfeelmoreconfidentusingtheadaptiveHMI 0 1 2 3 4 5
8. IfeelImakelessmistakes/errorsusingtheadaptiveHMI 0 1 2 3 4 5
9. IfeellessexhaustedaftermyshiftwhenusingtheadaptiveHMI 0 1 2 3 4 5
SATISFACTIONWITHMEASUREMODULE
Pleaseanswerthesubsequentquestionsconsideringthefollowingscenario:Theworkingmachines
areequippedwithsensorsthatareabletotrackstrainofaworkingpersonbyreal-timemeasurementof
his/herphysiologicalparameters,e.g.heartrate,bloodpressure,etc. Ifthemeasuredstrainindicatorsare
toohigh,thehuman-machine-interfaceadaptstothesituationresultinginalowerstresslevel.
Howdoyoufeelaboutmonitoringyourphysiologicalparameters(e.g.usingawristband,eyetracker,etc.)?
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent
4–Toaverylargeextent5-Notapplicable
Toaverysm
all
extent
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylarge
extent
Notapp
licab
le
1. Ifeelitcanchallengemyphysicalcomfort 0 1 2 3 4 5
2. Itrustthesystemandthatmypersonaldatawillnotbeabused 0 1 2 3 4 5
3. I feel uncomfortable with not knowing what happens with mypersonaldata
0 1 2 3 4 5
4. Ifeelthatmonitoringmystraincanbebenfecialforme 0 1 2 3 4 5
30
SATISFACTIONWITHTEACHINGMODULE
0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent
4–Toaverylargeextent5-NotapplicableWhatdoyouthinkabouttheon-lineandoff-linetraining?
Toaverysm
allexten
t
Toasm
allexten
t
Somew
hat
Toalargeextent
Toaverylargeextent
Notapp
licab
le
1. Thechosenwayofassistance(AR-basedassistance,Speech-basedassistance,Supportassistance)intheon-linetrainingwasappropriate
0 1 2 3 4 5
2. The way the on-line training system presented theinformationwasadaptedtomycurrentworktask.
0 1 2 3 4 5
3. Theinformationintheon-linetrainingsystemwaseasytoreadandperceive.
0 1 2 3 4 5
4. Iwouldhaveneededmoredetailedinstructionstocompletemytasks.
0 1 2 3 4 5
5. Theon-linetrainingsystemwasadequateinrelationtomyskillsandcapabilities.
0 1 2 3 4 5
6. The interaction with the on-line training system wasintuitive.
0 1 2 3 4 5
7. Ienjoyedusingtheon-linetrainingsystem. 0 1 2 3 4 5
8. Theon-linetrainingsystemmakesworkeasier. 0 1 2 3 4 5
9. Theon-linetrainingsystemwashelpfultomastertheHMI. 0 1 2 3 4 5
10. Theoff-linetrainingsystemshouldhavebeenmorerealistictopreparemefortheinteractionswiththeHMI.
0 1 2 3 4 5
11. Iwould have neededmore detailed instructions from theoff-linetrainingsystemstolearnthetasksuccessfully.
0 1 2 3 4 5
12. Theoff-linetrainingsystemcouldreplaceteaching-inbyatrainerforthisprocedure.
0 1 2 3 4 5
13. The interaction with the off-line training system wasintuitive.
0 1 2 3 4 5
14. Ienjoyedusingtheoff-linetrainingsystem. 0 1 2 3 4 5
15. Theoff-linetrainingsystemwasadequateinrelationtomyskillsandcapabilities.
0 1 2 3 4 5
16. Theoff-linetrainingsystemwastoocomplex. 0 1 2 3 4 5
17. Theoff-linetrainingsystemwashelpfultomastertheHMI. 0 1 2 3 4 5
18. Thecontentoftheoff-linetrainingwasappropriate 0 1 2 3 4 5
31
OverallSatisfactionwiththeadaptiveHMIRegarding the adaptiveHMI in general. How pleased are youwith it as awhole, everything taken intoconsideration?qVerysatisfiedqSatisfiedqNeithersatisfied,norunsatisfiedqUnsatisfiedqVeryunsatisfiedIfyouaresatisfied,whatdoyoulikethemostintheadaptiveHMI?
________________________________________________________________________________
________________________________________________________________________________
WhatshouldbeimprovedintheadaptiveHMI?
________________________________________________________________________________
________________________________________________________________________________
Thankyouforyouranswers!!!