Teaching Team-effectiveness in Large Classes · Teaching Team-effectiveness in Large Classes ......

TeachingTeam-effectivenessinLargeClasses

PatriciaK.Sheridan,GregJ.Evans,DougReeve,UniversityofToronto

Publishedby

TheHigherEducationQualityCouncilofOntario

1YongeStreet,Suite2402Toronto,ONCanada,M5E1E5Phone: (416)212-3893Fax: (416)212-3899Web: www.heqco.caE-mail: [email protected]

Citethispublicationinthefollowingformat:Sheridan,P.K.,Evans,G.J.,&Reeve,D.(2015).TeachingTeam-effectivenessinLargeClasses. Toronto:HigherEducationQualityCouncilofOntario.

TheopinionsexpressedinthisresearchdocumentarethoseoftheauthorsanddonotnecessarilyrepresenttheviewsorofficialpoliciesoftheHigherEducationQualityCouncilofOntarioorotheragenciesororganizationsthatmayhaveprovidedsupport,financialorotherwise,forthisproject.©Queen’sPrinterforOntario,2016


HigherEducationQualityCouncilofOntario3

ExecutiveSummary Instructionofteamskillsisquicklyemergingasanimportantandmissingdimensionofengineeringeducation.Thisprojectevaluatedanewframeworkforguidingstudentsinprovidingself-andpeerassessmentsoftheireffectivenessinteamwork.Thisframeworkisthefoundationforanewweb-basedtoolthatoffersstudentsstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Specifically,thestudydocumentedinthisreportinvestigatedwhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,increasedstudents’abilitiestolearnaboutandimprovetheirteam-effectivenessinexecutingdesignprojects.Theframeworkconsistedof27competenciesacrossthreeaspectsofteam-effectiveness:organizational,relationalandcommunicationcompetencies.Theframeworkwastestedinarandomizedcontrolledexperimentinafirst-yearengineeringdesignclassof280studentsagainstanunstructuredfeedbackprompt.Studentswereaskedtoprovidefeedbackatthemid-pointofthecourseandtoprovidetheirthoughtsontheutilityofthefeedbacktheyreceivedinanend-of-termsurvey.Studentassessmentswerealsocomparedtoteachingassistantassessments.Studentsusingtheframeworkfoundthefeedbacktheyreceivedtobemoreactionablethanunstructuredfeedback,andfoundthatitmotivatedthemmoretoimprovetheirperformancethandidthestudentsreceivingunstructuredfeedback.Studentsintheunstructuredgroupreceivedfeedbackonfewerandlessdiverseteam-effectivenesscompetenciesthanthoseintheframeworkgroup.Studentsintheunstructuredgroupreceivedfeedbackonapproximately10competenciesonaverage,andapproximatelyhalfofthatfeedbackwasonorganizationalcompetencies;studentsintheunstructuredgroupreceivedlittlefeedbackonrelationalcompetencies.Theunstructuredfeedbackalsoprimarilyidentifiedthestudents’strengths,whichwerephrasedaspraiseoftheirperformance,withminimaldiscussionofthestudents’weaknesses.Studentsintheframeworkgroupwereabletopeer-assesstheirteammembersaccuratelywhenusingtheframeworkbutwerelessabletoself-assessaccurately.Whencomparedtotheassessmentsoftheirteachingassistants,students’peerassessmentscorrelatedsignificantlywithteachingassistants’assessmentsacrossallthreeaspectsofteam-effectiveness.However,onlystudents’self-assessmentsalongtheorganizationalaspectcorrelatedsignificantlywithteachingassistants’assessments.Whilestudentsintheunstructuredgroupdidnotreceivethismorecomprehensivefeedback,theydidcommentthatthetextualfeedbackmadethemfeelmorecommittedtotheirteam,asitdemonstratedthattheirteammembershadaninterestinthemandintheteamasawhole.Basedonthisbenefitandonstudentsinthefeedbackgroup’srequestsforexamplesandcommentsintheirfeedback,webelievethatahybridofthefeedbackframeworkalongwithsometextualfeedbackwouldbethebestmethodforprovidingfeedbackmovingforward.



TableofContents

1. Introduction...................................................................................................................................................9

2. BackgroundandMotivation..........................................................................................................................9

3. FeedbackFramework..................................................................................................................................11

4. StudyDesignandResearchMethods..........................................................................................................12

4.1. CourseContext ..................................................................................................................................... 13

4.2. StudyDesign ......................................................................................................................................... 14

4.3. StudentParticipation ........................................................................................................................... 14

4.4. TeachingAssistantParticipation .......................................................................................................... 16

4.5. ParticipationRates ............................................................................................................................... 17

4.6. ResearchMethods ............................................................................................................................... 18

5. DifferencesintheQuantity,BreadthandAccuracyofStudentFeedback..................................................19

5.1. QuantityofTeamMemberEffectivenessContent .............................................................................. 19

5.2. BreadthofTeamMemberEffectivenessContent ................................................................................ 21

5.3. OtherTypesofInformationintheFeedback ....................................................................................... 23

5.3.1.UnrelatedFeedback .................................................................................................................... 23

5.3.2.Team-levelfeedback ................................................................................................................... 23

5.4. Accuracy–AgreementbetweenSelf-andPeerAssessments ............................................................. 23

5.4.1.UnconsideredAssessments ........................................................................................................ 24

5.4.2.AgreementintheUnstructuredFeedback .................................................................................. 24

5.4.3.AgreementintheFrameworkFeedback ..................................................................................... 25

5.4.4.PerceivedAccuracy ..................................................................................................................... 26

5.5. Discussion ............................................................................................................................................ 27

5.5.1.LimitedQuantityofUnstructuredFeedback ............................................................................... 27

5.5.2.StudentPrivilegingofOrganizationalCompetencies .................................................................. 28

5.5.3.LimitedUtilityofSelf-assessmentsintheUnstructuredGroup .................................................. 28

5.6. ConclusionandResponsetoResearchQuestion ................................................................................. 29

6. DifferencesinStudentPerceptionsoftheUsefulnessoftheirFeedback...................................................29



6.1. IdentificationofaStudent’sStrengthsandWeaknesses ..................................................................... 29

6.1.1.UnstructuredFeedback ............................................................................................................... 29

6.1.2.FrameworkFeedback .................................................................................................................. 30

6.1.3.StudentPerceptionoftheStrengthsandWeaknessesIdentified .............................................. 31

6.2. ImprovementbasedonFeedback ........................................................................................................ 32

6.3. Discussion ............................................................................................................................................ 33


7. DifferencebetweenStudentandTAAssessments......................................................................................35

7.1. SimilarityinIdentificationofaStudent’sStrengthsandWeaknesses ................................................. 35

7.2. CorrelationbetweenTAandStudentFrameworkAssessments .......................................................... 36

7.3. Discussion ............................................................................................................................................ 37

7.3.1.TeachingAssistantHesitancyaroundtheAssessmentContext .................................................. 37

7.3.2.LimitationsontheAgreementforUnstructuredAssessments ................................................... 38

7.3.3.Aspect-levelDiscrepanciesinAssessment .................................................................................. 38


8. StudentandTAFeedbackontheUsabilityoftheFramework....................................................................39

8.1. HolisticAssessmentofFrameworkAccessibility .................................................................................. 39

8.2. AccessibilityoftheFramework’sCompetencies .................................................................................. 40

8.3 AccessibilityoftheCompetencies’BehaviourallyAnchoredRatingScales ......................................... 40

8.3.1.BARSDescriptions ....................................................................................................................... 40

8.3.2.SequencingoftheBARS .............................................................................................................. 41

8.3.3.NumberofPointsalongtheBARS ............................................................................................... 41

8.4. PresentationofFeedback .................................................................................................................... 42

8.5. MissingContent ................................................................................................................................... 42

8.6. Discussion ............................................................................................................................................ 42


9. ConclusionsandFutureWork......................................................................................................................43

References........................................................................................................................................................45



ListofFiguresFigure4-1:Overviewofthestudydesigndemonstratingwhichparticipantscompletewhichassessments,surveysandattendfocusgroups......................................................................................................................14

Figure4-2:Trainingsequenceinwhichtheteachingassistantsparticipatedpriortoprovidinganyassessmentsofstudentperformance...............................................................................................................17

Figure5-1:Numberofframeworkcompetenciesdiscussedinfeedbackreceivedbystudentsintheunstructuredfeedbackgroupinresponsetotheprompt“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject”.......................................20

Figure5-2:Percentofrelationalcontentinanunstructuredgroup’sstudent’sfeedbackbasedonthetotalamountoffeedbackreceived...........................................................................................................................21

Figure5-3:Distributionofresponsestothestatement“Ireceivedfeedbackon:abroadrangeoftopics”bygroup.................................................................................................................................................................22

Figure5-4:Distributionofperformanceagreementinstudentfeedbackforcompetenciesthatwerediscussedincommonbypeerassessors...........................................................................................................25

Figure6-1:Percentofpeer-feedbackreceivedbystudentsintheunstructuredfeedbackgroupthatdiscussedtheirstrengths..................................................................................................................................30

Figure6-2:Distributionofresponsestothestatement“thefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare”bygroup.......................................................................................32

Figure6-3:Numberofweaknessesidentifiedinstudents’end-of-termsurveysthattheycommentedasimprovingonbasedontheirfeedback,bygroup.............................................................................................34



ListofTablesTable3-1:The27competenciesofthefeedbackframeworkdividedintothreeaspectsofteammembereffectiveness.Thecompetencynumbersprecedingeachcompetencywillbeusedtoreferencethecompetenciesintheanalysis............................................................................................................................11

Table4-1:Demographicinformationoftheteachingassistants(TAs)whoparticipatedinthestudy,theirbackgroundsandtheirrespectivepreviousaffiliationswiththecourse..........................................................16

Table5-1:Contingencytableanalysiscomparinggroupagreementonreceivingabroadrangeoftopicsintheirfeedback...................................................................................................................................................22

Table5-2:DistributionofagreementbetweenpeerassessmentsofastudentasmeasuredusingICCsinatwo-wayrandomeffectsmodelforconsistency..............................................................................................26

Table5-3:Spearman’srankcorrelationbetweenstudents’self-andpeerassessmentsforeachaspectoftheframework........................................................................................................................................................26

Table6-1:Contingencytableanalysiscomparinggroupagreementonreceivingidentifiedweaknessesintheirfeedback...................................................................................................................................................31

Table6-2:Contingencytableanalysiscomparingthetoneofthefeedbackreceivedbythetwogroups.......33

Table7-1:FrequencyofagreementbetweenstudentandTAidentifiedthesamestrengthsandweaknessesintheunstructuredandframeworkfeedbackgroups......................................................................................36

Table7-2:Spearman’srankcorrelationbetweenstudents’peerassessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework..........................................................................................36

Table7-3:Spearman’srankcorrelationbetweenstudents’self-assessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework..........................................................................................37



AcknowledgmentsTheauthorswouldliketothankJasonFosterandRobIrishforallowingthisresearchtooccurintheircourse,JenniePhillipsforherassistanceinfacilitatingandtranscribingtheparticipantfocusgroups,LobnaElGammalforherassistancewiththedataanalysis,EstelleOliva-Fisherforherassistancewithquestiondevelopment,CindyRottmanforherguidancewiththeresearchmethodology,JasonFosterforhistechnicalassistancewiththesurveyingsystem,aswellasSusanMcCahan,JasonBazylakandPeterWeissfortheirconstructiveinputinthedesignandexecutionofthestudy.Theauthorswouldliketoespeciallythankallofthestudentsandteachingassistantswhomadethisstudypossible.



1. IntroductionIntentionalinstructionofteamskillsisquicklyemergingasanimportantandmissingdimensionofengineeringeducation.Thisprojectinvestigatedandevaluatedanewinterventionthatsupportspersonalizedlearningofteameffectivenessinlargeundergraduatecourses.Specifically,weevaluatedtheeffectivenessofanewweb-basedtooltoprovidestudentswithstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Thisinterventionaimedtoprovidestudentswithasafe,virtualenvironmentinwhichtheycould:i)learnabouttheirteam-effectivenessandteamissues,andii)identifymethodstoimprovetheirareasofweaknessbeforetryingthemwiththeirteammembers.Afundamentalcomponentofthisinterventionisthefeedbackframeworkthatformsthebasisofthetool.Thefeedbackframeworkgroundstheassessmentsstudentsprovideandthefeedbacktheyreceivealongspecificcompetencies,guidingstudentstorecognizeandreflectonnecessaryteamworkskills.Thisstudyaimedtoinvestigatewhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,wouldincreaseastudent’sabilitytolearnaboutandimprovetheirteam-effectivenessintheirprojectteams.

2. BackgroundandMotivationEngineeringisateam-basedprofessionthatrequiresstudentstobebothtechnicallyproficientandeffectiveatteamwork.TheCanadianEngineeringAccreditationBoardmandatesthatallstudentswhograduatefromengineeringprogramsbecapableofworkingeffectivelyaloneaswellasinteams(CanadianEngineeringAccreditationBoard,2010).Upongraduation,whenstudentstransitiontoworkinginindustry,theywillworkonlarge-scale,complexproblemsthatrequiremultipleindividualsworkingeffectivelyinteamstodesignappropriatesolutions.Inendeavouringtopreparestudentsforthisenvironment,mostundergraduateengineeringprogramshavestudentsworkonteam-basedprojectsthatmodelwhatthestudentswillexperienceinindustry.Theprojectsendeavourtoinstillinstudentstheskillsnecessarytoworkeffectivelyintheseteamsaswellasintheirfuturecareers.However,studentshavetraditionallybeenexpectedtodevelopteamworkskillsimplicitlysimplybyparticipatingintheteamproject.AttheUniversityofToronto,engineeringstudentsareintroducedtoteamworkintheirfirst-yearengineeringdesigncourses.First-yearclasses,inwhichstudentslearnthefoundationalskillstheyneedfortheremainderoftheirdegreesandfortheircareers,aretraditionallythelargest.Forexample,classsizeintheengineeringprogramattheUniversityofTorontorangesfrom100to1,000students,withthetwofirst-yearengineeringdesignclassesbeingthelargestatapproximately280and1,000students.Inthesecourses,studentshavebeengroupedintoteamstosimulateindustryworkingconditions.However,thesizesoftheseteams(rangingfromthreetosevenstudents)aremainlyconstrainedbyfacilitiesandinstructor/teachingassistanttime,especiallygradingtime.Intheselargeclasses,providingfeedbackonstudentperformanceinatimelymannerisalwaysachallenge.Asaresult,thesestudentshavetraditionallyreceivedfeedbackonthequalityoftheirteam’sdeliverables(e.g.,designreports,presentationsandprototypes)fromtheirteachingassistants,butrarelyreceivefeedbackonhowtheyasindividualsareperformingasteam



members.Courseshavetraditionallyinformedstudentsthattheteachingassistantsareavailabletosupportbutnotassesstheirteam-effectiveness.Theteam-basedprojectenvironmentinfirst-yearengineeringdesigncoursesisahigh-stakesenvironment.Theprojectsaretoocomplexforonepersontoaddressontheirownandrequiresomelevelofcollaborationbetweenteammembers.Thestudentsdonotalwaysworkeffectivelytogetherbutarestillgradedtogether–thewholeteamreceiveswhatevergradethedeliverablereceives,regardlessofhoweffectivelytheyworkedtogether.Whenstudentsintheprojectteamsdonotpulltheirweight,theremainderofthestudentsintheteamlearnnottoliketeamworkandnottotrusttheirteammembers.Theyseeteamworkasanundesirableexperiencebeingforcedonthemthatjeopardizesthethingtheyvalue:theirgrades.Ourbeliefisthatmanyoftheseissuescouldbemitigatedbyprovidingstudentswithpersonalizedsupportinthedevelopmentoftheirteamworkskills.Thefirstpartofthissupportisprovidingstudentswithanawarenessofwhatishappeningintheirteams.Insmallclasses,aninstructorcandothisthankstothepersonalnatureofteacher-studentinteractionsresultingfromasignificantlydecreasedassessmentload.However,evenhere,theinstructormayonlyseepartofwhatisoccurringwithinateam.Thus,duetothewayinwhichstudentteamswork,webelievethatthebestpeopletoprovidethisfeedback,bothinlargeandsmallclasses,arethestudentsthemselves.Theyseetheirteammembersfortheentiredurationoftheproject,inparticularinteammeetingsoutsideofclassortutorialtimewheninstructorsarenotpresent.Ourobjectiveistouseselfandpeerassessmentwithinthestudentprojectteamstoprovidepersonalizedfeedbacktoeachoftheteammembersontheireffectivenessasateammemberduringtheproject.Approachestoconductingselfandpeerassessmentsofteamworkskillsinengineershavebeenapopularfocusofresearchoverthelastfiveto10years.Web-basedtoolshavebeendevelopedintheUS,mostnotablytheComprehensiveAssessmentofTeam-memberEffectiveness(Loughry,Ohland&Moore,2007),andintheUK,theWebPA(LoughboroughUniversity,2009).Thesetoolsoriginatedtoensurethatinstructorshavesufficientinformationtoknowiftheyneedtomodifyindividualstudents’gradesbasedontheirrelativecontributionstoateamdeliverable.Asthesetoolshaveexpandedtobecomeweb-basedresourcestofacilitatethelearningofteam-effectiveness,theyhaveapproachedteamworkfromastrongtask-focusedorientation.Studentsinengineeringarealreadytraditionallymoreorientedtowardsthetasksideofteamworkthantothepeopleside,sothereexistsagreaterneedtofacilitatetheirdevelopmentoftherelationalandcommunicationskillsofteamwork.Asaresult,ratherthanadoptingoneoftheexistingframeworks,wedevelopedourownthatpushesstudentstothinkaboutrelationalskillsinadditiontotaskskillswhenprovidingandreceivingfeedback.Wesoughtwiththisstudytoassesstheeffectivenessofthisselfandpeerassessmentfeedbackframework.Thisreportdocumentstheassessmentofourteammembereffectivenessfeedbackframeworkasafoundationtofacilitatestudentdevelopmentoftheircompetencyasteammembers.Feedbackoriginatingfromtheuseofthefeedbackframeworkwascomparedtofeedbackdevelopedfromanunstructuredfeedbackprompttoassesstheusefulnessandeffectivenessofthefeedbackframework.ThisreportfirstpresentstheframeworkinSection3,outlinesthestudydesignandresearchquestionsinSection4,



respondstotheseresearchquestionsindividuallyinSections5-8,andfinallyconcludeswithourfindingsaroundtheutilityofthefeedbackframework.AdiscussionofplannedfutureworkisoutlinedinSection9.

3. FeedbackFrameworkTheteam-effectivenessfeedbackframeworkthatwasusedinthisinterventionispresentedinTable 3-1. Table3-1:The27competenciesofthefeedbackframeworkdividedintothreeaspectsofteammembereffectiveness.Thecompetencynumbersprecedingeachcompetencywillbeusedtoreferencethecompetenciesintheanalysis.

OrganizationalAspects RelationalAspects CommunicationAspects

O1.Supportteamrules R11.BuildthetrustofteammatesC20.Exchangeinformationinatimelymanner

O2.Attendteammeetingsprepared R12.Motivateothersontheteamtodotheirbest

C21.Introducenewideas

O3.Contributetomakingmeetingseffective

R13.Raisecontentiousissuesinaconstructiveway

C22.Openlyexpressopinions

O4.Dotheirfairshareofthework R14.Solicitinputbeforeproceeding C23.Promoteconstructivebrainstorming

O5.DelivertheirworkontimeR15.Adoptsuggestionsfromothermembers C24.Activelylistentoteammates

O6.ProducehighqualityworkR16.Acceptfeedbackaboutstrengthsandweaknesses C25.Provideconstructivefeedback

O7.Helptoplan,setgoals,andorganizework

R17.Showrespectforotherteammates

C26.Makesurethatteammatesunderstandimportantinformationandinstructions

O8.Trackteamprogressvs.yourtimeline

R18.Demonstrateaccountability C27.Helptheteambuildconsensus

O9.Encourageprogresstomeetgoalsanddeadlines R19.Collaborateeffectively

O10.Displaydedicationanddetermination

Note:Thisframeworkhasbeenanalyzedinotherstudiesaswell.SeeSheridanetal.(2013)and(2014).Thisframeworkwasdevelopedfromfourexistinginventoriesanddesignedtoincludeagreaterfocusonnon-task-relatedcompetencies(Bushe&Coetzer,1995;Lingard,2010;Moore,Diefes-Dux&Imbrie,2006;Maxwell,2011).Asynthesisoftheseinventorieswasdevelopedandredundancybetweenbehaviourswaseliminated.Competencieswerecategorizedaccordingtothreeaspectsofteammembereffectiveness(Table 3-1):organizationalcompetencies,whichfocusontheprojectmanagementandtaskcompletionaspectofeffectiveteamwork;relationalcompetencies,whichfocusonhowstudentsbuildeffectivemeansofcollaboratingtogetherandtrustineachother;andcommunicationcompetencies,whichfocusonhowstudentsleveragetheirinteractionstoexchangeinformationandpromoteproductivediscussions.



Foreachcompetency,thereisabehaviorallyanchoredratingscalethatdescribesastudent’slevelofengagementinutilizingthatcompetencytoimproveteamperformance.Studentsassessthemselvesandtheirteammatesaccordingtoa7-pointdescriptiveLikertscale,whereanassessmentof1-2representsanunengagedteammember,3-5aself-focusedteammember,and6-7ateam-focusedteammember.Toobtainateam-focusedassessmentalongeachcompetency,thestudenthastodemonstrateanabilitytoexhibitandpromotethebehavioursofthecompetencyintheirteammembers.ThebehaviourallyanchoredratingscaleforallcompetenciesisavailableinAppendixA.Bystructuringtheassessmentsusingabehaviourallyanchoredratingscaleforeachcompetency,studentsshould:i)beabletoassessusingacommonscale,increasingtheconsistencyinassessmentsreceivedbyastudent,andii)beabletoprovidefeedbackthatisspecificenoughthatateammatecanidentifytheirperformancelevelandhowtoimproveit.

4. StudyDesignandResearchMethodsTheobjectiveofthestudywastounderstandhowstudentsperceive,interpretanduseourfeedbackframeworktopromotelearningaboutteammembereffectivenessbehavioursintheirprojectteamsthroughtheuseofselfandpeerassessment.Ourgoalwastodesignaneffectiveframeworktoguidestudentstoprovidefeedbackinordertofacilitatethelearningofteamworkwithinstudentprojectteamsinlargeclasses,withminimalneedforadditionalcontact-hourresources.Ourhypothesiswasthat:

Studentscanbeguidedtoprovideusefulpersonalizedfeedbackonteam-effectivenesstotheirteammatesusingourteam-effectivenessframework.

AstudytoassesstheutilityoftheframeworkinfacilitatingfeedbackwastestedintheWinter2012termofPraxisII,a280-studentcornerstonedesigncourseinfirst-yearengineeringattheUniversityofToronto.Thiscoursepreviouslyhadstudentsprovidesomeunstructuredcommentsabouttheirteammemberstotheteachingteamasameansofexpressingconcernincaseswhereteammembersdidnotcompleteequitablesharesoftheprojectwork.Asprovidingunstructuredcommentsabouteachteammemberwasalreadyamodeofreflectionaboutteam-effectivenessinthecourse,wewantedtocompareourframeworktothepre-existingunstructuredfeedbackmethodtodeterminewhetherourframeworkcouldbemoreusefulinguidingstudentstoimprovetheirteam-effectivenessbehaviours.Toassessthis,weusedthefollowingfourresearchquestionstoguideourinquiry:

1. Doestheframeworkguidestudentstoprovideagreaterbreadth,quantityoraccuracyoffeedback?2. Dostudentsperceivethepeerfeedbackfromtheframeworktobemoreusefulthanunstructured

feedback?3. Canstudentsprovidefeedbacksimilartothatofatrainedobserver(courseteachingassistant)when

usingtheframework?4. Istheframeworkaccessible(e.g.,jargon,descriptions,levelsofcompetency)tostudentsandtrained

observers?



Thetheoreticalorientationtakentolearninginthisstudyisthatofsocialconstructivism.Thisisbecausetheteamactsasalearningcommunityinwhichtodevelopagreaterunderstandingofone’sownteammembereffectivenessbehaviours,aswellasagreaterunderstandingofteammembereffectivenessingeneral.Selfandpeerassessmentsofteammembereffectivenessactasscaffoldingtosupportstudentsinobservingandthereby developing a better understanding of team member effectiveness behaviours. Improvement inbehaviour isbelievedtobemotivatedboth intrinsicallyandextrinsically throughthedesiretobeabetterteammemberandtocreateabetterteam,respectively.4.1. CourseContextPraxisIIisafirst-yearengineeringdesigncourseinthefirst-yearEngineeringScienceprogramattheUniversityofToronto.PraxisIIisa280-studentcoursethattakesplaceinthesecondtermoftheschoolyearandbuildsontheintroductorydesign,communicationandteamworkprinciplescoveredintheirfirst-termcourse,PraxisI.ThecourseisbuiltonthepedagogicalmodelsofKolb,PerryandVygotskyinthedesignandexecutionofitscourselectures,tutorialsandassignments.Specificallyrelatedtothisstudy,theclass(studentsandteachingassistants)attendedthreehalf-hourlecturesonteam-effectiveness,wheretheywereintroducedtoTuckman’s(1965)andLencioni’s(2002)modelsofeffectiveteamwork,theaspectsandcompetenciesofourfeedbackframework(Table 3-1),andhowthesebehavioursmanifestinhighlyeffective,high-performanceteams.Thecourseisdesignedasaservice-,project-andteam-basedlearningcourseinwhichstudentsspendthefirsthalfofthe13-weektermengagingwithcommunitiesaroundtheCityofTorontotoidentifyengineeringdesignopportunitiestheycanaddress,andthesecondhalfofthetermaddressingasubsetofopportunitiesselectedbytheteachingteamandbasedonthoseidentifiedinthefirsthalfoftheterm.Thestudentsworkinself-selectedteamsofthreetofourstudentsfortheentire13weeksofthecourse.Studentscanselecttheirteamfromthe22to27studentsintheirtutorialsection;studentsmaynotbeabletoselecttoworkwiththeirfriendsiftheyarenotinthesametutorialsection.Asaresultofworkingwiththesameteamforthe13weeksofthecourse,studentshavesufficienttimetodeterminehowtheyandtheirteammembersworkinteams,aswellasimprovetheirteamworkskillsandbehavioursbasedonfeedbackduringthecourse.Thecoursehasateachingteamofninemembers:twocourseinstructors,oneofwhomisadesignexpertandtheotheracommunicationexpert,andseventeachingassistants,whoseareasofexpertisestraddleengineering,theartsandthehumanities.Studentsaregroupedintotutorialsof22to27studentsandarepairedwithtwoteachingassistantsofcomplementarybackgrounds.Theroleoftheteachingassistantsistochallengetheideasofthestudents,instillgooddesignandcommunicationpractices,modelconstructiveandcollegialdisagreement,anddemonstrateeffectivecollaborationintheirteachingstyle.



4.2. StudyDesignThestudylookedattheeffectivenessofthefeedbackframeworkbycomparingittounstructuredfeedbackinarandomizedcontrolledexperimentwithinthesameclass.AnoverviewofthestudydemonstratingwhichparticipantscompletedwhichcomponentsisprovidedinFigure4-1.Figure4-1:Overviewofthestudydesigndemonstratingwhichparticipantscompletewhichassessments,surveysandattendfocusgroups

4.3. StudentParticipationTheentireclass’teamsweredividedrandomlyandapproximatelyequallyintotheunstructuredandframeworkfeedbackgroups.Individualstudentswerethenabletoconsenttolettingususetheirfeedbackforresearchpurposes.Studentteamswereusedtoseparatestudentsintotheexperimentgroupsinordertoensurethatallstudentswithinateamwereusingonetypeoffeedbackmechanism,andthatstudentsonlyreceivedfeedbackofonetype.Immediatelyafterthestudentteamssubmittedtheirfirstmajordeliverableinweek7,studentswereaskedtoprovideself-andpeerassessmentsoftheireffectivenessasteammembers.Studentsprovidedpeerassessmentswithintheirprojectteamsonlyfortheirteammembers.Forty-eightpercentoftheclasswasgroupedintotheframeworkgroupandcompletedthisassessmentusingourfeedbackframework(AppendixA).Thesestudentscompletedtheirassessmentsbyselectingthedescriptorthatmatchedtheirteammember’sbehaviourforeachquestion.Theremaining52%weregroupedintotheunstructuredgroupandcompletedthisassessmentbyprovidingunstructuredfeedbackthatrespondedtotheprompt:“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject.”



Studentsinbothgroupswereaskedtoprovidefeedbackfortheirteammembersonlineonsequentialpagesofasurvey.StudentscompletedthesesurveysonLimeSurvey,anopen-sourcesurveyingsoftwarethatwasimplementedonasecureserveroncampus.Studentswereprovidedalinktocompletetheirsurvey,anddependingonwhethertheywereintheframeworkorunstructuredfeedbackgroup,theysawandcompletedadifferentsurvey.Oneweekaftercompletingtheassessment,studentsreceivedtheirselfandpeerfeedbackonlineforreview.Studentsintheframeworkgroupreceivednumericalfeedbackfromtheirteammembersthatcorrespondedtothedescriptorsthestudentsusedtoprovidetheirassessments(AppendixA).Feedbackintheframeworkgroupidentifiedforeachstudenttheirlowest-rankedcompetenciesasweaknesses(byhighlightingtheirfeedbackalongthesecompetenciesinred)andtheirhighest-rankedcompetenciesastheirstrengths(byhighlightingtheirfeedbackalongthesecompetenciesingreen).Therankingofthesecompetencieswasdeterminedaccordingtoasortingalgorithmthattookintoaccountthevaluesoftheself-assessment,peerassessments,andthedifferencebetweenthetwo.Thealgorithmworkedtoidentifythreestrengthsandthreeweaknessesforthestudents.However,inthecaseofcompetencieswithtiedrankings,theminimumnumberofranksthatresultedinthreeormorecompetenciesbeinghighlightedwerecolouredasstrengthsorweaknesses,respectively.Forexample,iffourcompetenciesallhadthesamerankingthatwashigherthanallothercompetencies,onlythatrank(thosefourcompetencies)wouldbehighlighted.Includingallstudentswhohadtiedrankings,themaximumnumberofcompetenciesthatwaseverhighlightedasastudent’sstrengthsorweaknessesbasedonthesortingalgorithmwasfive.Studentsintheunstructuredgroupreceiveduneditedtextualfeedback,withnostrengthsandweaknessesintentionallyhighlighted.Studentsinbothgroupsalsoreceivedalistoftoolsandtechniquesthattheycouldusetoenhancetheireffectivenessforeachofthecompetenciesoutlinedintheframework.AnexampleofframeworkfeedbackisprovidedinAppendixB,andanexampleofunstructuredfeedbackisprovidedinAppendixC.Attheendofthecourse,studentsinbothgroupscompletedthesameend-of-termsurveyontheusefulnessofthefeedbacktheyreceived(AppendixD).Questionsinthesurveyanalyzedthequantity,depthandbreadthofthefeedback,aswellasitsabilitytomotivatestudentstoimprovetheireffectivenessinteamsbasedonthefeedback.Focusgroupswithstudentswhousedtheframeworkwereconductedaftertheend-of-termsurveytoassesstheframework’sutility,capturedataonhowparticipantsperceivedandusedit,andtheirperceptionsoftheutilityofthefeedbackitgenerated.ThefocusgroupswereconductedbyanexperiencedfocusgroupfacilitatorfromoutsideofengineeringandaskedthequestionsoutlinedinAppendixE.



4.4. TeachingAssistantParticipationTeachingassistants(TAs)participatedinthestudybyusingtheframeworktoprovideassessmentsforallstudentsintheirtutorialsections.Teachingassistantswereusedastrainedobserverstoassesstheconcurrentvalidityoftheframework.TAsaretypicallytheindividualswhohaveidentifiedandaddressedteamdysfunctionsituationsandwhostudentsinthecourseapproachwhentheyhavequestionsorissueswiththeirteamwork.Additionally,withaclassof280students,theseweretheonlyindividualswhohadsufficientone-on-onecontactwiththestudentsintheirteams,andwerealreadyscheduledtobesupporting,andthereforeobserving,themintutorials.ThedemographicinformationofthesevenTAsisshowninTable4-1.TheseTAssupportedninetutorialsectionsinpairs,witheachTAstaffingatleasttwotutorialsandsomestaffingthree.ThepairingofTAsintutorialsprovidedafortuitousopportunitytoacquiretwoobservationsofeachstudentwithwhichtocomparethestudents’self-andpeerassessments.ThisallowedustoassesswhetherstudentsprovidedsimilarfeedbacktotheirtutorialTAs,aswellastheabilityoftheTAsastrainedobservers.TeachingassistantsweretrainedbeforeprovidingfeedbackabouttheirstudentsaccordingtotheprocessshowninFigure4-2.Afterattendingthesamelecturesonteam-effectivenessasthestudents,theTAsmetwiththedesignersoftheframeworkfor1.5hourstodiscusshowtheywouldbeusingtheframeworktoassesstheirstudents.AsalmostallTAshadpreviouslybeenTAsinthecourse,thegroupelectedtousethepreviousyear’steamsthattheyandtheframeworkdesignerwerefamiliarwithasthecasestudiestocalibratetheirassessments.TheseexemplarteamsweredescribedtothetwonewTAsinfrontoftheframeworkdesignerbytheotherfiveTAs,sothattheycouldallassesstheteamsbasedontheprovideddescriptions.Usingthesecasestudies,theTAsnegotiatedwhathighandlowperformancelookedlikeforeachframeworkcompetencyuntiltheirassessmentsofperformancewereinagreement.Finally,theTAsaskedtheframeworkdesigneranyquestionsofclarificationaboutwhatdifferentbehavioursintheframeworkcouldlooklikeoutsideoftheteamsdiscussed.Table4-1:Demographicinformationoftheteachingassistants(TAs)whoparticipatedinthestudy,theirbackgroundsandtheirrespectivepreviousaffiliationswiththecourse

TA1 TA2 TA3 TA4 TA5 TA6 TA7

Background Eng Eng Eng Eng Eng Arts Humanities

Gender Female Male Male Male Female Female Female

Previousexperiencewiththecourse

Student Student StudentandTA

StudentandTA

TA TA TA

NativeEnglishSpeaker Yes Yes Yes Yes Yes Yes Yes

Note:Forbackground,Engrepresentsengineering



Figure4-2:Trainingsequenceinwhichtheteachingassistantsparticipatedpriortoprovidinganyassessmentsofstudentperformance

TAswereaskedtocompletetheirassessmentsdifferentlythanwerethestudentsintworespects.First,teachingassistantscompletedtheassessmentstwoweeksafterthestudents.Asstudentsprovidedtheirfeedbackimmediatelyaftersubmittingtheirfirstmajorteamdeliverable,TAsweregradingatthattime.Approximatelytwoweeksafterthestudentsprovidedfeedback,theTAswereaskedtoindependentlyprovidefeedbackforeverystudentintheirtutorialsectionsusingtheframework.AsthismeantthateachTAhadbetween54and81studentstoassess,theyweregivenmoretimetocompletetheframeworkthanthestudentteamswhoonlyhadtoassessthreeorfourteammembers.TheTAsweregiventwoweekstocompletetheseassessments,meaningthatsomeTAassessmentswerecompleteduptoamonthfromthetimewhenthestudentscompletedthem.Second,atthetrainingsessionTAsrequesteda‘donotknow’optionforeachofthecompetencies.TheTAscommentedthattheywouldprefernottobeforcedtoprovideanincorrectassessmentforastudentwhentheyfeltthattheywerenotabletoassessthestudentalongthatcompetency.AfocusgroupwiththeTAswasconductedaftertheendoftermtoassesstheframework’sutility,capturedataonhowtheyperceivedandusedit,andtheirperceptionsoftheutilityofthefeedbackitgenerated.Thefocusgroupwasconductedbyanexperiencedfocusgroupfacilitatorfromoutsideofengineeringandaskedthesamequestionsasthestudentfocusgroup,outlinedinAppendixE.4.5. ParticipationRatesTherateofstudents’consenttoparticipateintheresearchstudywashighat77.5%,totalling218studentsfromtheclass–112fromtheunstructuredgroupand106fromtheframeworkgroup.Sincethestudentshadtocompletetheassessmentsoftheirteammembersforcoursepurposes,aspartofa2%completed/notcompleteddeliverable,therewasnoadditionaleffortonthepartofthestudentstoparticipateinthestudy.Studentswereincentivizedtoallowustousetheirdatainthestudythroughtheopportunitytowinone$100giftcard.Ofthe112studentsintheunstructuredgroupwhoagreedtoparticipateinthestudy,thedatafrom105studentswereusedinthisanalysis.Oneteamofthreestudentswascompletelydysfunctionalandasaresult,theyusedtheirunstructuredfeedbackasaplacetoventtheirangerratherthanprovidefeedback.Thesedatawereremoved,aswefeltthattheyhadthepotentialtosignificantlyskewtheresultsofthedepthandbreadthoftheunstructuredfeedbackcontent.Fourotherstudentswereremoved,aseachofthemhadonlyoneotherteammemberwhoconsentedtoparticipateinthestudy,thereforeeliminatingthe

Aqendcourselectureonteam-effecsvenesswithstudents

Negosatewhathighandlowperformancealongeachframeworkcompetencylookslike

Asktheframeworkdesignersquessonsofclarificasonandprocess



abilitytocompleteanyfeedbackcomparisonsbetweenpeers.All105studentsprovidedfeedbackandrespondedtotheend-of-termsurvey.Ofthe106studentsintheframeworkgroupwhoagreedtoparticipateinthestudy,thedatafrom105studentswereused.Onestudent’sfeedbackwasremoved,astheyweretheonlypersonintheirteamtoconsent,providingnopeerfeedbackforthisindividual.Only95ofthesestudentsrespondedtotheend-of-termsurvey.Allteachingassistantsinthecourseagreedtoprovidefeedbackontheirstudents,aswellasparticipateinafocusgroupontheiruseofthefeedbackframework.Asthiswasadditionalworkontopoftheteachingassistants’normalcourseduties,teachingassistantswerecompensatedattheuniversity’steachingassistantpayratefortheadditionalhoursrequiredtocompletetheframeworkfortheirstudents.Whiledatawereobtainedfromtheteachingassistantsfor100%ofthestudents,onlythedataforthe77.5%whoconsentedtoparticipatewereusedfortheanalysis.Eitherduetothelackofgrade-basedincentivesortothetimingafterexams,studentparticipationinthefocusgroupswassignificantlylowerthanexpectedatthreestudents,makingallfindingsfromthestudentfocusgroupinformativebutnotsignificant.4.6. ResearchMethodsThestudywasconductedasacontrol-conditionexperimentwithinthesameclass,thusallowingustocomparetheeffectsofbothmodesofprovidingfeedbackwithinthesamecontext.ThestudywasapprovedbytheUniversityofToronto’sResearchEthicsBoardforSocialSciences,Humanities,andEducation.Thisstudyfollowedamixedmethodsapproachthatusedbothqualitativeandquantitativeinquiry.AllqualitativeanalysiswascompletedinNVivo,withquantitativeanalysescompletedinSPSSandMicrosoftExcel.Feedbackthatstudentsreceivedinresponsetotheunstructuredfeedbackpromptwascodedaccordingtothefeedbackframework(Table3-1)toidentifywhichcompetencieswerediscussedincommonbetweentheunstructuredfeedbackandtheframeworkfeedback.Codesweresetupforthe27competencies,aswellasforfourothercompetencythemesthatemergedintheunstructuredfeedbackbutdidnotmapdirectlyontotheframework’scompetencies.AdiscussionofthesethemesisincludedinAppendixF.Thenumberofoccurrencesofeachofthecompetencycodeswasthencalculatedforcomparisonwiththeframeworkfeedbackandoverlapsbetweencodesinvestigated.Additionally,thetypeoffeedbackprovidedtoastudent(individualorteam-level)aswellasthecontentofthefeedback(designrelated,teamworkrelated,etc.)wasalsocoded.Feedbackthatstudentsreceivedfromtheframeworkwasanalyzedusingintra-classcorrelationstounderstandhowstudentswereusingtheframeworkforself-andpeerassessments.Trendsinassessmentpatternsweredeterminedandcomparedtothoseoftheunstructuredfeedback.AsTAsalsousedtheframework,TAassessmentswereanalyzedsimilarly.



Theend-of-termsurveycontainedacombinationofbothqualitativeandquantitativequestionsandwascompletedbystudentsinboththeunstructuredandframeworkfeedbackgroups.Qualitativequestionswerecodedthematicallyandacomparisonofcommentsacrossthetwogroupswasperformed.Codingthemesfocusedonmotivation(ifstudentstriedtoimprove,whydidtheydoit),action(whatstudentsdidtoimprove)andimpact(whatresponsedidthefeedbackevokeinthestudentsintermsofbelieforcommitment).Quantitativequestionsmeasuredstudents’levelofagreementwithdifferentstatementsregardingthefeedbackandwerecomparedtoidentifystatisticallysignificantdifferencesbetweenthetwogroups’perceptionsoftheirfeedback.Thefocusgroupswereaudiorecordedandtranscribed.ThetranscriptionswerethencodedthematicallytoidentifysimilaritiesanddifferencesinstudentandTAresponses,andwereanalyzedtodeterminesimilaritiesanddifferences.

5. DifferencesintheQuantity,BreadthandAccuracyofStudentFeedback

Thissectionrespondstothefirstresearchquestion–doestheframeworkguidestudentstoprovideagreaterquantity,breadthoraccuracyoffeedbackthantheunstructuredprompt–bycomparingthetypesoffeedbackreceivedbystudentsinthetwogroups.5.1. QuantityofTeamMemberEffectivenessContentQuantitywasmeasuredbythenumberofdifferentteammembereffectivenesscompetenciesastudentreceivedintheirfeedbackthatmatchedthoseinourfeedbackframework.Allstudentsintheunstructuredgroupreceivedfeedbackonatleastthreecompetencies,withafewstudentsreceivingfeedbackonupto20competencies(Figure5-1)discussedinourframework.



Figure5-1:Numberofframeworkcompetenciesdiscussedinfeedbackreceivedbystudentsintheunstructuredfeedbackgroupinresponsetotheprompt“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject”

Onaverage,studentsintheunstructuredgroupreceivedfeedbackonnineto10competencies.Allstudentsreceivedfeedbackonatleastoneorganizationalcompetency,97%ofstudentsreceivedfeedbackonacommunicationcompetency,butonly88%ofstudentsreceivedfeedbackonarelationalcompetency.Fourstudentsintheunstructuredgroup,includingthreefromthesameteam,chosetostructuretheirfeedbackbasedontheframework’scompetenciespresentedtostudentsduringthethreehalf-hourlecturesonteam-effectiveness.Eventhoughthesestudentschosetousetheframework’scompetency-languagetoprovidefeedback,theydidnotprovidefeedbackonall27competenciesbutprimarilyusedtheterminologyofthecompetenciestoframetheirfeedback.Bycomparison,allstudentsintheframeworkgroupreceivedfeedbackon27competencies,asstudentswererequiredtoprovidefeedbackonallcompetenciesincludedintheframework.Therefore,studentsintheframeworkgroupreceivedagreaterquantityofteammembereffectivenessfeedback.

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8 9 10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

Num

bero

fFramew

orkCo

mpe

tenciesd

iscussedin

Feed

back

Students

OrganizasonalItems RelasonalItems CommunicasonItems



5.2. BreadthofTeamMemberEffectivenessContentBreadthwasevaluatedbasedonwhetherstudentsreceivedfeedbackcoveringarangeofcompetenciesorwhetherthecommentsprovidedfeedbackonfewsimilarcompetencies.Competenciesdiscussedinthefeedbackweregroupedintothethreeaspectsofteammembereffectivenessintheframeworktoassessthebreadthofthefeedbackastudentreceived.Studentsintheunstructuredgroupgenerallyreceivedsignificantlymorefeedbackontheirorganizationalcompetenciesthanontheothertwoaspects’competencies.Studentsintheunstructuredgrouponaveragereceivedfeedbackonfiveorganizationalbehaviours(SD=1.7),tworelationalbehaviours(SD=1.6)andthreecommunicationbehaviours(SD=1.4).Thiscorrespondstoapproximately50%ofastudent’sfeedbackbeingonorganizationalcompetencies,22%onrelational,and28%oncommunicationcompetencies.Whiletheamountoforganizationalandcommunicationcontentbypercentagewasfairlystableregardlessofthenumberofcompetenciesonwhichastudentreceivedfeedback,studentswhoreceivedfeedbackonfewcompetenciesreceivedlittletonofeedbackonrelationalcompetencies(seeFigure5-2).Competenciesthatwerediscussedthemostinfeedback(over60%ofstudents)includedproduceshighqualitywork,doestheirfairshareofthework,deliverstheirworkontimeandintroducesnewideas.Competenciesthatwerediscussedtheleastinfeedback(lessthan10%ofstudents)includedsupportteamrulesandbuildthetrustofteammates.Figure5-2:Percentofrelationalcontentinanunstructuredgroup’sstudent’sfeedbackbasedonthetotalamountoffeedbackreceived

Studentsintheframeworkgroupreceivedfeedbackon10organizationalbehaviours,ninerelationalbehavioursandeightcommunicationbehaviours.Onaverage,studentsintheunstructuredgroupreceivedfeedbackonlessthanhalfasmanycompetenciesasstudentsintheframeworkgroup.

0%

5%

10%

15%

20%

25%

30%

35%

<5 5-9 10-14 15+

AveragePe

rcen

tofFeedb

ackon

Re

laso

nalCom

petencies

TotalNumberofCompetenciesonwhichFeedbackwasReceived



Atwo-waycontingencytableanalysiswasconductedtoevaluatewhetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthebreadthoftheirfeedbackinresponsetothesamequestionontheend-of-termsurvey,“Ireceivedfeedbackon:abroadrangeoftopics”(Table5-1).Thetwovariablesusedineachoftheseanalysesweretheexperimentalconditionwithtwolevels(unstructured,framework),andstudentresponsetothequestiononaLikertscale(stronglydisagree,disagree,slightlydisagree,neutral,slightlyagree,agreeandstronglyagree).Table5-1:Contingencytableanalysiscomparinggroupagreementonreceivingabroadrangeoftopicsintheirfeedback

Question Pearson’sχ2 Cramer’sV

Ireceivedfeedbackon:abroadrangeoftopics 20.4 .32**

Note:**p≤.01AscanbeseenfromTable5-1,therewasasignificantdifferenceinstudents’perceptionsofthebreadthoftheirfeedbackacrossthetwogroups.Lookingatthegraphofthestudentresponsestothisquestion,itcanbeseenthatstudentsintheunstructuredfeedbackgroupperceivedthattheydidnotreceivefeedbackonasbroadarangeoftopicsastheframeworkgroup(Figure5-3).Figure5-3:Distributionofresponsestothestatement“Ireceivedfeedbackon:abroadrangeoftopics”bygroup



5.3. OtherTypesofInformationintheFeedback Inadditiontothefeedbackdiscussedaboveonstudents’teammembereffectiveness,anumberofotheritemswerepresentedintheunstructuredfeedbackthatdidnotsupportstudentawarenessoftheirteammembereffectiveness.Twotypesoffeedbackfitthisdescription:unrelatedfeedbackandteam-levelfeedback. 5.3.1. UnrelatedFeedback Somestudentsintheunstructuredgroupreceivedfeedbackontheircourse-relatedknowledge/skills(17students)orwereprovidedwithadescriptionofwhatworktheirteammatesperceivedtheycontributedtotheproject(34students,ofwhich11receivednoteammembereffectiveness-relatedfeedback).Non-feedbackstatements,genericstatements(e.g.,“goodjob”)thatprovidednoinformationaboutthestudents’team-effectivenesscompetenciesorcourseskills,werereceivedbyninestudentsandprovidedthemwithnoinformationthattheycoulduseaseitherapointofreferenceorguidanceabouttheirperformance.Nostudentsintheframeworkgroupreceivedunrelatedfeedbackduetothenatureofthefeedbackframework. 5.3.2. Team-levelfeedbackSelf-assessmentsinparticularwerenotusedeffectivelybystudentsintheunstructuredgroup.Whenstudentswereaskedtoprovidefeedbackonthemselves,theyoftenreflectedontheirwholeteam’seffectivenessratherthanontheirownindividualeffectiveness.Twenty-fourpercent(24%)ofstudentsinthegroupdidthis,providingnospecificreflectionontheirownperformanceasateammember.Nostudentsprovidedteam-levelassessmentsaspeerfeedbackwithoutdiscussinghowthereceivercontributedtotheteam’seffectiveness–akeycomponentthatwasmissingfromtheself-assessments.Nostudentsintheframeworkgroupreceivedteam-levelfeedbackduetothenatureofthefeedbackframework.Asaresult,nostudentsintheframeworkgroupmissedoutonthisopportunitytocomparetheirself-andpeerassessments.5.4. Accuracy–AgreementbetweenSelf-andPeerAssessmentsAccuracywasmeasuredbyexaminingtheamountofunconsideredassessmentsprovidedbyeachgroup,theagreementbetweenself-andpeerassessments,aswellasthroughstudentperceptionsoftheaccuracyoftheirfeedbackasreportedintheirend-of-termsurveys.Assessmentsthatdemonstratednoconsideredreflectiononthepartoftheproviderwereunlikelytoprovideaccurateassessmentsofthereceiver’sperformance.Agreementintheunstructuredgroupwasmeasuredintermsofthenumberofcompetenciesdiscussedincommonacrossmultipleteammembers.Agreementintheframeworkgrouplookedat



correlationbetweenself-andpeerassessmentsusingintra-classcorrelationandSpearman’srankcorrelation.5.4.1. UnconsideredAssessmentsUnconsideredassessmentsfortheunstructuredgroupconsistedofnon-teamwork-relatedcomments,non-feedbackstatementssuchasgenericperformancestatements,anddescriptions(ratherthanassessments)ofastudent’scontributiontoteamdeliverables.Forty-sevenindividualassessmentswerejudgedtobeunconsidered,ofwhichthemajoritywereprovidedbyeightstudentstoeachoftheirteammembers.Thisresultedin45%ofstudentsintheunstructuredgroupreceivingatleastoneunconsideredassessment,outofatotalofthreeorfourassessmentsfromtheirwholeteam.Unconsideredassessmentsintheframeworkgroupwerethoseinwhichastudentgavethemselvesand/ortheirpeersthesamenumericassessmentacrossall27competencies.Twelveassessmentsmetthesecriteria,withsevenofthemcomingfromtwoindividuals.Onestudentprovidedthesameassessmentforallteammembersacrossallcompetencies,andasecondstudentprovidedtheirpeerswiththesameassessmentacrossallcompetenciesbutdemonstratedconsiderationintheirownassessment.Itwasunclearwhethertheotherfiveassessmentswereconsidered,astheirassessmentvaluedidnotdiffersubstantiallyfromtheconsideredassessmentsoftheirotherteammembers.Thus,sevenoutof94individualsintheframeworkgroupreceivedunconsideredassessments(approximately7.5%),substantiallylessthanthe45%intheunstructuredgroup.Theunconsideredassessments–12fromtheunstructuredgroupandsevenfromtheframeworkgroup–wereremovedbeforecompletingtheanalysesdiscussedbelow.5.4.2. AgreementintheUnstructuredFeedbackStudentsintheunstructuredgroupreceivedlimitedfeedbackthatdemonstratedagreementordisagreementontheirperformance,astheirfeedbackcontainedfewcompetenciesthatwerediscussedincommonbymorethanoneassessor.Thus,theabilitytoassessagreementislimitedtothecasesinwhicheitherthestudent’speersdiscussedatleastonecompetencyincommon,orinwhichastudentandatleastoneoftheirpeersdiscussedatleastonecompetencyincommon.Usingthe27competenciesthatwerecodedbasedonthefeedbackframework,14%ofallcompetency-codedfeedbackintheunstructuredgroupaddressedcompetenciesthatwerediscussedincommonbytwoormorepeers.Thisresultedin83studentsreceivingfeedbackthatdiscussedatleastonecompetencyincommon,withanagreementonthestudent’sperformanceonthatcompetency.Onaverage,24%ofastudent’sfeedbackdiscussedcompetenciesincommon.Themaximumpercentageoffeedbackdiscussingcompetenciesincommonwas63%.Competenciesdiscussedincommonweregroupedaccordingtowhetherthepeersagreedontheperformancelevelofthestudent,whetherthepeersdisagreedontheperformancelevel,orwhetheritwas



unclearwhethertherewasagreementordisagreementbetweenthepeerassessments.Thedistributionsoftheseperformance-levelagreementsonaperstudentbasiscanbeseenbelowinFigure5-4.Acrossthe83studentswhoreceivedfeedbackincommon,60studentsreceivedfeedbackthat,onaverage,agreedontheperformancelevelofthestudent,4receivedfeedbackthatdisagreedontheperformancelevelofthestudent,and6studentsreceivedfeedbackthatagreedontheperformancelevelofatleastonecompetencyanddisagreedonatleastoneother.Fortheremaining13students’competenciesdiscussedincommon,itwasunclearwhethertherewasagreementordisagreementbetweenassessments.Figure5-4:Distributionofperformanceagreementinstudentfeedbackforcompetenciesthatwerediscussedincommonbypeerassessors

5.4.3. AgreementintheFrameworkFeedbackToassesstheagreementofthefeedbackastudentreceivedintheframeworkgroup,theinter-raterreliabilitybetweenpeerassessorsandthecorrelationbetweenstudents’self-andpeerassessmentsweredetermined.Inter-raterreliabilitywasdeterminedusingtheintra-classcorrelationcoefficients(ICCs)forpeerassessments,usingatwo-wayrandomeffectsmodelforconsistency.Ninety-oneofthe95studentshadatleasttwopeerassessorsandwereconsideredinthisanalysis.PeerICCswerethengroupedaccordingtotheirlevelofagreementbetweenthepeerassessors(Table5-2).Thirty-onepercentofthefeedbackshowednoagreementbetweenthepeerassessors,and8.8%showedsubstantialagreementbetweenthepeerassessors.Themajorityofassessmentsshowedlimitedagreement,rangingfromslighttomoderateagreement.Therefore,itislikelythatthefeedbackshoweddifferinglevelsofagreementalongthedifferentaspectsoftheframework.



Asaresultofthelimitedagreementamongpeers,thelevelofagreementbetweenself-andpeerassessmentswasdetermined.Lookingspecificallyintoourhypothesisthatthelimitedagreementmaybeaspect-specific,accuracywasdeterminedbycomparingtheaverageself-assessmentstothepeerassessmentsacrossthethreeaspectsoftheframework.UsingaSpearman’srankcorrelation(Table5-3),weseeasignificantcorrelationbetweenthestudents’self-andpeerassessmentsintheorganizationalaspect.However,thereisnosignificantcorrelationbetweenstudents’self-andpeerassessmentsintherelationalandcommunicationaspects.Thisstrongcorrelationinself-andpeerassessmentsalongonlytheorganizationalaspectmayexplainthevarianceinagreementseeninthepeeragreementdiscussedabove.Table5-2:DistributionofagreementbetweenpeerassessmentsofastudentasmeasuredusingICCsinatwo-wayrandomeffectsmodelforconsistency

LevelofAgreement NumberofStudents ICCvaluerange

Noagreement 28 <0

Slightagreement 18 0-0.2

Fairagreement 20 0.21-0.4

Moderateagreement 17 0.41-0.6

Substantialagreement 8 0.61-0.8

Perfectagreement 0 0.81-1

Table5-3:Spearman’srankcorrelationbetweenstudents’self-andpeerassessmentsforeachaspectoftheframework

Aspect Correlation

Organizational .45**

Relational .19

Communication .16

Note:Students’self-assessments(n=numberofcompetencies)andpeerassessments(n=numberofcompetenciesxnumberofpeers)wereaveragedseparatelytodetermineeachstudent’sself-assessedandpeer-assessedlevelofcompetencyfortheaspect.Theseaveragesforeachaspectwerethencorrelatedoverallthestudents.**-p≤.01.5.4.4. PerceivedAccuracyUsingatwo-waycontingencytableanalysis,therewasnosignificantdifferenceacrossthetwogroupsintheiragreementwiththestatement“ThefeedbackIreceiveddescribedmeexactlyhowIperceivedmyself”fromtheirend-of-termsurveys.



Thissimilarityintermsofhowstudentsperceivedtheaccuracyofthefeedbackextendedintostudentsresponsestotheopen-endedquestionsontheend-of-termsurvey.Fourstudentsfromtheunstructuredfeedbackgroupandfivestudentsfromtheframeworkfeedbackgroupcommentedthattheyfelttheirfeedbackwasfakeandthatitwasnotaccurate.Asthisaccountedforthesamepercentageofstudentsineithergroup(5%),neithergrouponaverageperceivedthefeedbacktheyreceivedtobemoreorlessfakethantheothergroup.Additionally,anequalnumberofstudents(8)fromeachgroupchallengedtheirfeedbackfromtheirpeers,statingthattheyfeltthatitdidnotreflecthowtheybehavedintheirteam,orthattheirassessmentswerebasedononeincidentandnottheiraveragebehaviour.Onedifferencebetweenthegroups,however,wasintermsoftheperceivedconsistencyofthefeedback.Sevenstudentsintheframeworkgroupclaimedthattheirfeedbackpresentedtwodifferingperspectivesontheirperformanceandthatitthusbecamehardtoknowhowtheywereactuallyperformingintheirteam.Nosuchcommentswereprovidedbystudentsintheunstructuredfeedbackgroup.5.5. DiscussionInadditiontothedifferencesinfeedbackaroundquantity,breadthandaccuracyofteammembereffectivenesscompetencies,threeothernotabletrendsemergedintheanalysis.First,students,primarilythoseintheunstructuredfeedbackgroup,didnotalwaysreceiveenoughfeedbacktoprovidethemwithaclearassessmentoftheireffectivenessasateammember,limitingtheusefulnessofthefeedbacktheyreceived.Second,studentsinbothgroupsstronglyprivilegedorganizationalcompetenciesintheirfeedback.Third,studentsintheunstructuredfeedbackgroupdidnotalwaysusetheself-assessmentpromptcorrectly.5.5.1. LimitedQuantityofUnstructuredFeedbackWithintheunstructuredfeedbackgroup,students’limitedabilitytofindagreementamongtheirpeerassessmentspreventedthemfromgettingaclearpictureofhowtheywereperformingasateammember.Fewstudentshadcompetenciesdiscussedincommon,whichwaslimitedfurtherbytheamountofunrelatedfeedbackstudentsgeneratedduetotheunstructurednatureoftheprompt.Thislimitedquantityaffectedtheabilityofthefeedbacktoprovidethereceiverwithaconsistentmessageabouttheirperformance.Whiletherewasafairamountofagreementwithinthepeersaboutstudents’performancewhentheydiscussedcompetenciesincommon(66%ofallfeedbackdiscussedincommon),therewasnotsufficienttotalfeedbackincommon(14%oftotalfeedback)toprovidestudentswithasenseofconsistencyinpeerfeedback.Thisresultedinonly9.24%oftheunstructuredfeedbackdiscussingcompetenciesincommonwithagreementaboutperformance.Essentially,anunstructuredprompt(andhowstudentschoosetointerpretit)willmostlikelyproducepeerfeedbackthatraisesfewaspectsincommon.Studentsmaythusperceivethisfeedbackasrandombitsofinformationwithoutaconsistentmessage,makingiteasiertodismissasnotaccuratelydescribingthem.



5.5.2. StudentPrivilegingofOrganizationalCompetenciesStudentfeedbackinboththeunstructuredandframeworkgroupsdemonstratedastrongidentificationwithandprivilegingoforganizationalcompetencies.Thisisnotsurprising,asanecdotallystudentsinengineeringhaveatendencytoprivilegework-contribution-relatedcompetencies.Thisprivilegingwasevidentinthedistributionofcompetenciesdiscussedintheunstructuredgroup’sfeedbackthroughthestrongprevalenceoforganizationalitemsintheunstructuredfeedback(approx.50%ofallfeedbackwasorganizational).Additionally,studentswhoreceivedlittlefeedbackreceivedalmostalloftheirfeedbackonorganizationalcompetencies.Asaresult,unstructuredfeedbackmayperpetuatestudentprivilegingoforganizationalcompetenciesandlimittheirabilitytogrowaseffectiveteammembersbynotbroadeningstudentunderstandingofteammembereffectiveness.Whenlookingatwhichitemswerediscussedintheunstructuredfeedback,wefindthesameorganizationalpredominance,withtheexceptionoftwocompetencies.Thetopfivemostdiscussedcompetenciesintheunstructuredfeedbackwerepredominantlyorganizational,withoneexception.Giventheprivilegingbyengineeringstudentsofwork-contribution-relatedcompetencies,thehighlevelofresponserelatingtotheintroducenewideascompetency(thirdmostdiscussed)islikelyaproductoftheteam’sprojectwork.Studentswhocameupwiththeideathatwasselectedtoaddressthedesignprojectwerelaudedextensivelyfordoingsointheirfeedback.Oneorganizationalcompetencythatwasveryminimallydiscussedwassupportteamrules.Thisisnotalarmingasinthecoursecontext,teamruleswerediscussedasnormsofpracticebutwerenotcodified.Asaresult,studentsinthecoursewouldnotbepredisposedtoreflectingonastudent’srolewithrespecttoteamrules.Thisprivilegingcanalsobeseenintheframeworkfeedbackinthecorrelationbetweenstudentself-andpeerassessments.Theonlysignificantlycorrelatedassessmentswerethosealongtheorganizationalaspect.Whilethisdoesnotaffectthefeedbackastudentreceives,itdoesimplythatstudentsmayperceivethefeedbackprovidedalongtheorganizationalaspecttobemoreaccuratethanthefeedbackalongtheothertwoaspectsastheyagreemorecloselywiththeirself-assessments.5.5.3. LimitedUtilityofSelf-assessmentsintheUnstructuredGroupStudentself-assessmentsintheunstructuredgroupwerenotusedeffectively,asmanystudentsuseditasaspacetoreflectontheirteam’seffectivenessratherthantheireffectivenessasateammember.Theself-assessmentwasmeanttoactasapointofcomparisonforthestudentwhenreceivingfeedbacktoseehowdifferentlythey,ascomparedtotheirpeerteammembers,perceivedtheireffectiveness.Withoutanindividual-levelself-assessment,thiscomparisonislesseffectiveandthefeedbackthereforelessuseful.Additionally,itdoesnotprovidethereceiverwithanyguidanceonhowtoimprovetheirbehaviour,aswhatisneededtoimprovetheteamasawholemightbedifferentfromwhattheyneedtoimproveasateammember.Thus,nothavinganyself-assessmentofteammembereffectivenesslimitedtheusefulnessofthefeedbackincreatingagreaterawarenessofhowthestudentswereperformingforthis23%oftheunstructuredgroup.



5.6. ConclusionandResponsetoResearchQuestionComparingholisticallythefeedbackreceivedbetweenthetwogroups,studentsintheunstructuredgroupreceivedfeedbackonsubstantiallyfewerteamworkcompetenciesthandidthoseintheframeworkgroup,andreceivedtheirfeedbackprimarilyonorganizationalcompetencies.Studentsintheunstructuredfeedbackgroupreceivedsignificantlylessfeedbackonrelationalcompetenciesthandidthoseintheframeworkgroup–anareaknowntobeunderprivilegedinengineeringstudents.Studentsinthisgroupalsoreceivedasubstantialamountoftheirfeedbackonnon-teamwork-relatedissues/skills.Intermsoftheaccuracyofthefeedbackreceivedbystudentsineithergroup,theframeworkgroupreceivedlessfeedbackthatappearedtohavenoconsideredreflectiononthepartoftheproviderthandidtheunstructuredgroup.Consistencyinthefeedbackprovidedbystudentswasgreaterintheframeworkgroup,inpartduetothesetcompetenciesonwhichallstudentshadtoprovidefeedback,butitdidnotappeartolimittheabilitytoreceiveadiversityoffeedback,asseeninstudents’commentsonreceivingdifferentassessmentsfromtheirpeers.Therefore,inresponsetotheresearchquestion,theframeworkprovidesgreaterbreadth,quantityandaccuracyoffeedbackthantheunstructuredfeedbackprompt.

6. DifferencesinStudentPerceptionsoftheUsefulnessoftheirFeedback

Thissectionrespondstothesecondresearchquestion–dostudentsperceivethefeedbackfromtheframeworktobemoreusefulthanunstructuredfeedback?–bycomparingstudents’receivedfeedbackandtheircommentsontheend-of-termsurveys.Asstudentsinbothgroupsansweredthesamequestionsontheend-of-termsurvey,comparativeanalysesarepossible.Threenotabledifferencessurfacedintheanalysis:i)identificationofstrengthsandweaknessesinthefeedback,ii)perceivedamountofactionablefeedback,andiii)improvementbasedonfeedback.6.1. IdentificationofaStudent’sStrengthsandWeaknessesFeedbackwithclearstrengthsandweaknessesismoreusefultostudents,asitidentifieswhattheyaregoodatandwhattheyneedtoimprove.Thisallowsstudentstofocustheireffortsonimprovingcertainkeyareas.Feedbackfrombothgroupswascomparedtodeterminewhichmorereadilyidentifiedstrengthsandweaknessesinfeedback.6.1.1. UnstructuredFeedbackThefeedbackfromtheunstructuredgroupwassignificantlylessgranularintermsofperformancethantheframeworkfeedback.Studentswereeitherlaudedonwhattheydidwell,critiquedonwhattheydidpoorly,orinformedofwhattheydidordidnotdo.Asaresulttheunstructuredfeedback,whichhadalreadybeen



codedtoidentifyteammembereffectivenesscompetencies,wascodedagainaccordingtothesethreetypesoffeedback.Theunstructuredfeedbackreceivedbystudentsdiscussedthestrengthsofthestudentsmorecommonlythanitdidweaknesses.Twenty-fivestudents(24%)intheunstructuredgroupdidnotreceiveanyfeedbackonweaknessesorcritiqueoftheirbehaviour,andsixstudents(6%)receivednopraiseoridentifiedstrengthsintheirevaluations.Studentswithinbothofthesegroupsalsoreceivedneutralfeedback,whichsimplystatedwhattheycontributedtotheproject.Todeterminetheprevalenceofstrengthsinthefeedbackreceivedbystudents,thefeedbackwasbinnedintofivecategoriesaccordingtothepercentageofthefeedbackthatdiscussedstrengths.AscanbeseeninFigure6-1below,75%ofthestudentsintheunstructuredgroupreceivedfeedbackthatprimarilydiscussedtheirstrengthsasopposedtotheirweaknesses(opportunitiesforimprovement).Figure6-1:Percentofpeer-feedbackreceivedbystudentsintheunstructuredfeedbackgroupthatdiscussedtheirstrengths.

6.1.2. FrameworkFeedbackAllstudentsintheframeworkgrouphadasimilarnumberofidentifiedstrengthsandweaknesses.Thiswasaresultofthesortingalgorithmrankingstudentframeworkfeedback,suchthatstudents’strengthsandweaknessesweredeterminedandhighlightedforthem.Allstudentsintheframeworkgrouphadthreetofivecompetenciesidentifiedintheirfeedbackasstrengths,andthreetofivecompetenciesidentifiedas

0%

5%

10%

15%

20%

25%

30%

35%

40%

0-24% 25%-49% 50%-74% 75%-99% 100%Percen

tofStude

ntsintheUnstructuredGrou

p

PercentofFeedbackthatDiscussedStrengths



areasforimprovement.Thiscorrespondedtostudentshavingatleast11%oftheirfeedbackidentifyingstrengthsandatleastanother11%identifyingtheirareasforimprovement.Toensurethatthestrengthsandweaknesseswerenotjustarbitrarilyidentifiedbutactuallyflaggedcompetencieswithwhichthereceiverhadadistinctdifferenceintheirlevelofperformance,aRelated-SamplesWilcoxonsignedranktestwasconductedonthedifferencebetweentheaverageratingsofthecompetenciesidentifiedasstrengthsandtheaverageratingsofthecompetenciesidentifiedasweaknesses.Therewasasignificantdifference(p=.01)intheaveragesofthestrengthsandweaknessesidentifiedforthestudents,meaningthatonaveragestudentswerereceivingfeedbackthatdemonstrateddistinctdifferencesintheirperformancelevelacrossthecompetencies.6.1.3. StudentPerceptionoftheStrengthsandWeaknessesIdentifiedTherewasasignificantdifferenceintheperceptionoftheidentificationofstrengthsandweaknessesacrosstheunstructuredandfeedbackgroups.Whiletherewasnosignificantdifferencebetweenthegroupsastotheirperceptionoftheirstrengths,therewasasignificantdifferenceacrossthetwogroupsintermsoftheidentificationoftheirweaknesses.Atwo-waycontingencytableanalysiswasconductedtoevaluatewhetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthecontentoftheirfeedbackinresponsetothesamequestionontheirend-of-termsurvey(Table6-1).Thetwovariablesusedineachoftheseanalysesweretheexperimentalconditionwithtwolevels(unstructured,framework),andstudentresponsetothequestiononaLikertscale(stronglydisagree,disagree,slightlydisagree,neutral,slightlyagree,agreeandstronglyagree).Table6-1:Contingencytableanalysiscomparinggroupagreementonreceivingidentifiedweaknessesintheirfeedback


ThefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare

12.4 .25*

Note:*p≤.05AscanbeseenfromtheresponsedistributionsinFigure6-2,theframeworkgroupdemonstratedmoreoverallagreementwiththestatementthantheunstructuredgroup,indicatingthatstudentsintheunstructuredgroupnoticedthedistinctlackofweaknessesprovidedintheirfeedback.Thestudentsintheunstructuredgroupremarkedonthislackofcritiqueoftheirbehaviourintheopen-endedquestionsontheirend-of-termsurveys.Thestrongesttheme(greatestnumberofstudentscommentingonit)amongresponsesfromstudentswhoreceivedunstructuredfeedbackwasthattheydidnotreceivefeedbackonareasinwhichtheycouldimprove.Thirty-onestudents(30%)intheunstructuredgroupcommentedthattheywantedmorecritiqueintheirfeedback,comparedwithsevenstudents(7%)fromtheframeworkgroup.Thestrongestthemeamongthestudentswhoreceivedframeworkfeedbackwasthattheywantedmorejustification,commentsorexamplesfortheassessmentstheyreceived(23%).



Therefore,studentsintheframeworkgroupreceivedandperceivedthattheyreceivedmoreidentifiableweaknessesonwhichtoimprove,whilebothgroupsperceivedasimilarnumberofidentifiedstrengths.Figure6-2:Distributionofresponsestothestatement“thefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare”bygroup

6.2. ImprovementbasedonFeedbackImprovementbasedonfeedbackwasdifferentacrossthetwogroups.Motivationtoimproveperformanceappearedstrongerfortheunstructuredfeedbackgroup,duetothetoneoftheirfeedback.However,self-reportedimprovementalongspecificweaknessesidentifiedinthefeedbackwashigherfortheframeworkfeedbackgroup.Thetoneofthefeedbackappearedtohavealargeinfluenceonstudents’motivationtoimproveanddevelopasenseofteamcohesion.Atwo-waycontingencytableanalysiswasconductedtoevaluate



whetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthetoneoftheirfeedbackinresponsetothesamequestionontheirend-of-termsurvey(Table6-2).Thetwovariablesusedintheanalysisweretheexperimentalcondition(unstructured,framework)andstudentresponsetothequestiononathree-pointscale(Positive,Neutral,Negative).Table6-2:Contingencytableanalysiscomparingthetoneofthefeedbackreceivedbythetwogroups


Feedbacktomewasphrasedina{positive,neutral,negative}tone 8.5 .21**

Note:**p≤.01Therewasasignificantdifferenceintheperceptionofthetoneofthefeedback.Studentsintheunstructuredgroupfeltthattheirfeedbackwasphrasedmorepositively,whereasstudentsintheframeworkgroupweremostlysplitastowhethertheirfeedbackwaspresentedinaneutralorapositivetone.Twenty-threestudentsintheunstructuredgroup,comparedto12studentsintheframeworkgroup,commentedthatthefeedbacktheyreceivedincreasedtheircommitmenttotheirteamormadethemfeelthattheirteammembersvaluedtheircontributions.Commentsfromstudentsintheframeworkgroupseemedtobelookingforthispersonalconnectionintheirfeedback.Studentsintheframeworkgroupcommentedthattheywantedmorejustification,commentsorexamplesfortheassessmentstheyreceived(23students)(i.e.,sometextualfeedbackfromtheirpeers).Onedifference,however,betweenthetwogroupswasinthebehaviourstheyindicatedthattheyworkedonimprovingduringtheperiodbetweenreceivingtheirinitialfeedbackandcompletingtheend-of-termsurvey.Studentsfromtheframeworkgroupcitedmoreweaknessesthatwereidentifiedintheirfeedbackthandidstudentsintheunstructuredfeedbackgroup(Figure6-3).Bothgroupscitedworkingonotherimprovementsthatwerenotindicatedintheirfeedbackbutthattheythoughtwereequallyimportant.6.3. DiscussionWhilethetoneofthetextualfeedbackprovidedbystudentsintheunstructuredpromptincreasedstudentcommitmenttoimproving,thelackofidentifiedweaknesseslimitedtheusefulnessofthefeedbackasameanstopromoteimprovementandlearning.Sincetheunstructuredfeedbackgroupreceivedfeedbackthatwasphrasedinthewordsoftheirteammembersandmostlyfocusedonstrengthsinsteadofweaknesses,itmakessensethatstudentswouldperceivethisfeedbackasmorepositivethantheframeworkfeedbackreceivedbythatgroup.However,inafewcaseswherestudentswereprovidedconstructivecriticism,thecontextforthefeedbackwasprovidedtoexplaintheassessment.Thisaspectoftheunstructuredfeedbackstrengthenedtheutilityofthefeedbackforthesefewstudents.Thislackofidentifiedweaknessesmayalsoexplainwhythenumberofweaknessesonwhichstudentsidentifiedimprovingwasthesameacrossbothgroups,butthefeedback-identifiedweaknessesonwhich



studentsimprovedwaslessfortheunstructuredgroup.Havingfeweridentifiedweaknessescouldinhibitstudents’abilitytorecalland/orconsciouslyimprove.Additionally,sincetheamountofcompetency-specificfeedbackwasminimal(14%)fortheunstructuredfeedbackgroup,itislikelythatevenwhenaweaknesswasidentified,itwaswrittenoffasananomalyifitwasnotalsodiscussedbyotherpeers.Figure6-3: Numberofweaknessesidentifiedinstudents’end-of-termsurveysthattheycommentedasimprovingonbasedontheirfeedback,bygroup

6.4. ConclusionandResponsetoResearchQuestionInresponsetotheresearchquestion–dostudentsperceivethepeerfeedbackfromtheframeworktobemoreusefulthanunstructuredfeedback?–thefindingsaboveindicatethatstudentsdofindtheframeworkfeedbacktobemoreuseful.Fromtheseanalyses,itwasclearthattheframeworkguidedstudentstoidentifystrengthsandweaknessesmoreclearlythantheunstructuredfeedbackprompt,asstudentshadaprescribedsetofcompetenciestoassessintheframework.Studentsintheunstructuredgrouphadsignificantlylessagreementwiththestatementthattheirfeedbackidentifiedtheirweaknesses,andalsovocalisedthisissueintheiropen-endedresponsesintheend-of-termsurvey.Asaresult,studentsreceivingfeedbackfromtheframeworkreceiveagreateramountofcritiqueandthereforewereabletoidentifytheirweaknessessothattheycouldimproveonthem.Thiswasconfirmedwhenstudentsintheframeworkgroupcitedthattheyworkedonimproving



aspectsoftheirteam-effectivenessthatwereidentifiedasweaknessesintheirfeedbackmorefrequentlythandidstudentsintheunstructuredgroup.Therefore,becausetheframeworkfeedbackindicatesthepotentialtocreatemoreofthedesiredbehaviouralchangearticulatedbyteammembersintheirpeerfeedback,itmakestheframeworkfeedbackmoreuseful.However,basedontheimpactoftheunstructuredfeedback’stoneonteamcohesion,ahybridoftheframeworkandunstructuredfeedbackshouldbeinvestigatedinthefuturetoallowfortheinclusionoftextualcommentsintheframework.

7. DifferencebetweenStudentandTAAssessmentsThissectionrespondstothethirdresearchquestion–canstudentsprovidefeedbacksimilartothatofatrainedobserver(courseteachingassistant)whenusingtheframework?–bycomparingtheassessmentsprovidedbytheteachingassistants(usingtheframework)withthestudentsassessmentsinboththeunstructuredandframeworkgroups.Teachingassistantswereusedastrainedobserversbecauseofthequantityandqualityoftheirinteractionwiththestudentsinweeklytutorialsessions.Inendeavouringtoassesstheconcurrentvalidityoftheframework,itwasnotenoughforstudentstobeabletoprovidesimilarratingstotheirpeersusingtheframework.Inamoreabsolutesense,theyneededtoprovideratingsthatwereinfactarepresentationoftheassessedstudent’steammembereffectiveness.Aninvestigationofteachingassistants’abilitytoprovidefeedbackusingtheframeworkwasprovidedbySheridanetal.(2014).ThekeyfindingsaroundhowTAsusedtheframeworkwere:

1. TAsprovidedassessmentsthatwereonaveragelowerthanthoseofthestudents.2. TheaveragenumberofcompetenciesonwhichaTAprovidedfeedbackwas16(minimum2,

maximum27),duetotheiruseofthe‘donotknow’option.3. TAsprovidedasimilaramountoffeedbackacrossthethreeaspects.

Twonotabledifferencessurfacedinthiscomparisonofteachingassistanttostudentassessments:i)similarityofstrengthsandweaknessesidentification,andii)correlationbetweenteachingassistant,andself-andpeerassessments.7.1. SimilarityinIdentificationofaStudent’sStrengthsandWeaknessesThesamealgorithmusedonstudents’frameworkassessmentswasappliedtotheTAs’assessmentstosortoutthethreetofivetoprankedstrengthsandthreetofivebottomrankedweaknessesforeachstudent.TheseTA-identifiedstrengthsandweaknesseswerethencomparedtothestrengthsandweaknessesidentifiedbystudentsintheframeworkfeedbackgroup,andtothecompetency-codedstrengthsandweaknessesdiscussedinfeedbackfortheunstructuredfeedbackgroup.Whenlookingatthepercentageofstudentsineachgroup,theagreementbetweenstudentandTAfeedbackintermsofidentifyingatleastonestrengthorweaknessincommonwashigherfortheframework



group(86%)thanfortheunstructuredgroup(69%)(Table7-1).Additionally,whileTAsandstudentsrarelyagreedonbothastrengthandaweaknessforanygivenstudentinbothgroups,thisoccurredmorefrequentlyintheframeworkgroup(18%)thanintheunstructuredgroup(7%).Table7-1:FrequencyofagreementbetweenstudentandTAidentifiedthesamestrengthsandweaknessesintheunstructuredandframeworkfeedbackgroups

PercentofStudentsinFeedbackGroup

TypeofAgreement Unstructured Framework

1strengthOR1weakness 69% 86%

1strengthAND1weakness 7% 18%

7.2. CorrelationbetweenTAandStudentFrameworkAssessmentsTodeterminewhetherthestudentsprovidedsimilarfeedbacktotheTAswhengiventheframework,thecorrelationsbetweenstudentself-andpeerassessmentsandTAassessmentswerecomputed.Asthenumberofcompetencieswassubstantial,andgiventhatanumberofthewithin-aspectcompetenciesarehighlycorrelated,correlationbetweenstudentandTAassessmentstodeterminethesimilarityoftheirassessmentswasdoneattheaspect-leveloftheframework.Correlationswerecomputedacrossstudents’andTAs’averageorganizational,relationalandcommunicationassessments.TheseselfaveragesandpeeraveragesforeachaspectwerethencomparedtotheTAs’averageassessmentofthestudent’scompetencybycorrelatingoverallthestudents.ASpearman’srankcorrelationwasconductedbetweentheaspect-levelTAandpeerassessments(Table7-2)andTAandself-assessments(Table7-3)forallstudentsintheframeworkgroup.Whenusingtheframework,thepeerassessmentsandTAassessmentsofindividualstudentsweresignificantlycorrelatedacrossallaspects.However,theself-assessmentswereonlysignificantlycorrelatedwiththeTAassessmentsfortheorganizationalaspect–theaspectthatstudentsinboththeframeworkandunstructuredgroupweremorecomfortableassessing.ThisisthesametrendthatwesawinTable5-3,wherestudentself-andpeerassessmentswereonlysignificantlycorrelatedalongtheorganizationalaspectaswell.Table7-2:Spearman’srankcorrelationbetweenstudents’peerassessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework

Aspect Spearman’sRankCorrelation


Relational .27**

Communication .24*

Note:Students’peerassessments(n=numberofcompetenciesxnumberofpeers)wereaveragedseparatelytodetermineeachstudent’speer-assessedlevelofcompetencyfortheaspect.*p≤.05,**p≤.01



Table7-3:Spearman’srankcorrelationbetweenstudents’self-assessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework

Aspect Spearman’sRankCorrelation


Relational .00

Communication .16

Note:Students’self-assessments(n=numberofcompetencies)wereaveragedseparatelytodetermineeachstudent’sself-assessedlevelofcompetencyfortheaspect.**p≤.017.3. DiscussionInadditiontothedifferencesinteachingassistantandstudentassessments,threeothernotabletrendsemergedintheanalysis.First,teachingassistantswerehesitantabouttheassessmentcontext.Second,thequantityofunstructuredfeedbacklimitstheapplicabilityofthefindingsaroundagreement.Third,studentself-assessmentsbyaspectdemonstrateadifferentprivilegingthandoespeerassessments.7.3.1. TeachingAssistantHesitancyaroundtheAssessmentContextTheobjectiveofthispartofthestudywasinitiallytouseTAsasthegoldstandardagainstwhichtocomparestudents’feedbacktodetermineaccuracy.WhiletheresultsoftheassessmentsprovidedcorroborationthatstudentscanprovidepeerassessmentsthataresimilartothoseoftheirTAs,theteachingassistantswerehesitanttobeconsideredagoldstandard.TAsarethefront-linesupportforteamissuesandguidance,whichrequiresthemtobeattunedtohowtheteamsarefunctioning.However,theycommentedthattheyweremoretunedintotheteamsthantotheindividualswithintheteamsthemselves.Asaresult,theyweremorecomfortableandmoreconfidentassessingtheteamsthantheindividualstudents.PartoftheTAs’hesitationmayberelatedtothesizeofthefeedbackframework(27competencies),butbasedonTAcommentsfromthefocusgroupswehypothesizethatmostofthediscomfortisduetothecognitiveloadontheteachingassistantsduringtutorials.SincetheTAsneedtoprovidedifferentscaffoldingtoeachoftheteamstosupportthedevelopmentoftheirdesignandcommunicationskills,theirprimarymodeofobservationisduringtheseinteractions.Intheseconversations,theTAsneedtoassessthecurrentstateoftheteam’sdesignworkandwhatscaffolding(intheformofadvice,resourcesorcoaching)isnecessarytomovetheteam’sdesignworkforward.Asaresult,thissubstantiallylimitstheirabilitysimplytoobservethestudentsastheyworkintheirteamsandbiasestheirassessmentsagainstthosestudentswithwhomtheydonotfrequentlyinteractintheteams.However,giventhesignificantcorrelationsbetweenpeerandTAassessmentsattheaspectlevel,reducingtheTAfeedbackframeworktosubstantiallyfewercompetenciesortoanaspect-levelframeworkmaystrengthenthesecorrelations,asTAswouldthenbeabletoprovidemoreconfidentandhopefullymoreaccurateassessmentsoftheirstudents.



7.3.2. LimitationsontheAgreementforUnstructuredAssessmentsAsdiscussedinTable7-1,strengthsandweaknessesidentifiedbyTAsagreedstronglywithstudentsassessmentsfromtheframeworkgroup.However,itmustbenotedthatthisfindingislimitedbythequantityofunstructuredteammembereffectivenessfeedback.Asthequantityofunstructuredfeedbackwassubstantiallylessthanthatoftheframeworkfeedback,therewasalowerlikelihoodofastudenthavingchosentodiscussastrengthorweaknessidentifiedbytheTA.Sinceallstudentsintheframeworkgroupcommentedonallframeworkcompetencies,therewasagreaterlikelihoodofagreementbetweenthisgroupandtheTAs.Therefore,whiletheagreementoftheTAs’assessmentswiththeframeworkgroupisgreater,therearenotsufficientdatafromtheunstructuredgrouptoclaimanydifferenceinagreementbetweenthetwogroups.7.3.3. Aspect-levelDiscrepanciesinAssessmentAsdiscussedinTable7-2,TA-peerassessmentsweresignificantlycorrelatedalongallaspectsoftheframework,whileself-assessmentswereonlycorrelatedwithTAassessmentsalongtheorganizationalaspect.ThisfindingsupportsthediscussioninSection0thatstudentsinthisclassprivilegeorganizationalcompetenciesandareabletoself-assessthemmoreaccuratelythanthoseintheotheraspects.TA-self-assessmentcorrelationfortherelationalaspectwas.00,whichdemonstratednocorrelationbetweenTAandself-assessmentsatall.Thismaybeaproductoftheteachingassistants’lackofcomfortwiththerelationalcompetenciesaswell.Theteachingassistantscommentedintheirfocusgroupthattherelationalaspectwasthehardesttoassess.However,therewerenocleartrendsacrossthethreeaspectswithrespecttothenumberofcompetenciesonwhichtheyprovidedfeedback(duetothe‘donotknow’option).Thismayalsobeaproductofthepreviouslydiscussedprivileging,asstudentsinengineeringarenottraditionallycomfortablewithrelationalcompetencies.Thiswasseenalsointheunstructuredfeedback,inwhichlessthan20%ofthecompetenciesdiscussedwererelational.Therefore,whiletheframeworkdoesprovideasubstantialbasisforstudentstogivefeedbacktotheirteammembersthatissimilartothatoftheirTAs,studentshaddifficultyself-assessingtheirabilitiesinregardstorelationalandcommunicationcompetencies.Thisinabilitytoself-assessaccuratelyspeaksfurthertothevalueofusingpeerfeedbackasateachingtooltohelpstudentsdeveloptheirself-awarenessofandcompetencyinteammembereffectiveness.7.4. ConclusionandResponsetoResearchQuestionComparingthefeedbackreceivedbetweenthetwogroupsandtheteachingassistants,wecanclaimthatstudentpeerassessorscanbeusedforintra-teamevaluationsmoreeffectivelythanteachingassistants.Agreementbetweenteachingassistantandstudentfeedbackintermsoftheusefulnessofthefeedback(identifiedstrengthsandweaknesses)isgreaterwhenstudentsusetheframeworkthanwhentheyuseanunstructuredprompt.Additionally,studentpeerassessmentsaresignificantlycorrelatedwithteachingassistantassessmentsacrossallthreeaspectsoftheframework.Studentpeerscanprovidefeedbacksimilar



tothatoftrainedTAobservers.Thesamediscrepancybetweenself-andpeerassessmentswasseenwithself-andTAassessments,furthercorroboratingthatpeerassessmentsprovidesimilarinformationwithwhichstudentscanbroadentheirawarenessoftheirteamworkskills.Therefore,inresponsetotheresearchquestion,theframeworkguidesstudents(aspeers)toprovidefeedbacksimilartothatoftrainedobservers(teachingassistants).

8. StudentandTAFeedbackontheUsabilityoftheFrameworkThissectionrespondstothefourthresearchquestion–istheframeworkaccessible(e.g.,jargon,descriptions,levelsofcompetency)tostudentsandtrainedobservers?–bydiscussingfourprominentthemesaroundtheusabilityoftheframework,asdiscussedbystudentsandTAsintheirend-of-termfocusgroups.Thesefourthemesare:i)accessibilityofthecompetencies,ii)accessibilityoftheratingscale,iii)presentationofthefeedback,andiv)missingcontent.8.1. HolisticAssessmentofFrameworkAccessibilityThelanguageofthefeedbackframeworkreceivednocomplaintsorcommentsfromeitherthestudentsortheteachingassistants.Studentswereparticularlyhappythattherewasno“engineeringjargon”intheframework,whichtheyfeltenhanceditsaccessibility.Studentsinthefocusgroupthoughtthattheframeworkwas“thorough.”Theydescribedtheframeworkas“pertinent”and“relevant,”andsaidthatit“touchedonmoredetailedelementsofteamsthan[surveys]usuallytouchon.”Studentsalsocommentedthattheframeworkwasaneffectivelearningtoolinandofitself,andremarkedthatcompletingtheframeworkatthemid-pointoftheirteamprojectsallowedthemtoreflectonwheretheywereorwerenotdoingwellandremindedthemofwhatcompetencieseffectiveteamworkrequired.BothstudentsandTAslikedthattheassessmentsweregroupedintothethreeaspectsofteam-effectiveness.Bothgroupscommentedthattheyfoundthishelpedthemgetintoacertainheadspaceandevaluateaspecificaspectoftheexperience.Thissimplifiedthinkingandhelpedtoclarifyanysourcesofconfusioninthequestions.Onestudentcommentedthattheywouldhavepreferredthecompetenciestohavebeenpresentedasquestionsratherthanstatements,asitwould“allow[thestudent]tofocusmoreonwhat[theyare]grading[theirteammember]on.”Theydidnotelaborateonwhytheythoughtthiswouldbethecase.Studentscommentedthatinsteadofratingeachteammemberalongthecompetenciessequentially,theywantedtoprovidefeedbackforeachteammemberalongthecompetenciessimultaneously–thestudentswantedtocompleterelativerankingsoftheirteammembers.Thestudentsfeltthatiftheywereratingalltheirteammembersatonce,theywouldbeabletoprovidemoreaccurateassessments,andasaresultofhavingfewer(perceived)questionstocompletetheywouldprovidemoreaccurateassessments.Studentscommentedthatcompletingtheentireframeworkforeachteammembersequentiallyresultedinfatiguebytheendofthesurvey,suchthattheywerenotfullyreadingthecompetencyorratingscaleandmaynothavebeenasaccurateforthelastteammemberassessed.



8.2. AccessibilityoftheFramework’sCompetenciesStudentsinthefocusgroupdidnotcommentspecificallyonthelistofcompetencies,buttheTAsdidfeelthatsomecompetenciescouldnotbeassessedatanindividuallevel.Forcompetencieslikeproducehighqualitywork,theTAswouldonlybefamiliarwithteamperformanceandcouldnotrateteammembersindividually.TAsalsofeltthatthecompetenciesdisplaydedicationanddetermination,trackteamprogressvs.projecttimeline,demonstrateaccountability,exchangeinformationinatimelymanner,raisecontentiousissuesinaconstructivewayandsolicitinputbeforeproceedingwereallpoorlyworded,astheywerenotabletoassessthesecompetenciesthroughobservationduringthetutorials.TheTAssawthesecompetenciesastheproductofaseriesorsequenceofbehavioursthatcombinetogether.Giventhattheywerenotparticipatingintheteamsorobservingthemallthetime,theywereonlyabletoseesomeofthebehaviours,notallofthesequence,thatcombinetocreatethecompetency.Theycommentedthattheywouldhavepreferredtomarkoffspecificindividualbehaviourstheysawratherthantryingtodeducethestudents’performancealongthecompetency.8.3 AccessibilityoftheCompetencies’BehaviourallyAnchoredRatingScalesCommentsregardingtheaccessibilityofthebehaviourallyanchoredratingscales(BARS)foreachcompetencycentredaroundthreedifferentaspectsoftheBARS:theirdescriptions,theirsequencingandthenumberofratingpoints.8.3.1. BARSDescriptionsStudentsfoundtheBARSdescriptionslong.Giventhattheyhadtoprovidefeedbacktoeachteammemberindividually,theymentionedthattheywouldstopreadingthemafterawhile.Thestudentsproposedhavingalongerdescriptionofthecompetencyandafewwordsbrieflydescribingeachofthedifferentratingpoints.However,thiswouldnotprovideenoughinformationtodistinguishclearlythedifferentlevelsofperformanceforthestudents.TAsfeltthatthedescriptionsfortheBARSdidnotoutlinebehavioursthattheycouldclearlyobserveintutorialswithoutbeingamemberoftheteam.TAsfeltthatintheirassessmentcontext,theBARSrequireddeductionratherthanobservation,whichmadethemuncomfortablewhenanswering.Inparticular,theyfeltthatthedescriptionswerenotrepresentativeofthepersonalitytypesoftheirstudents.TAsperceivedthatthedescriptionsprivilegedextroversiontothedetrimentofintrovertedstudents,especiallywithrespecttothecommunicationaspect,astheonlywaytheycouldassessstudentsintroducingnewideasoropenlyexpressingopinionsintutorialswasverbally.Additionally,TAsfeltthatwhentheycompletedtheframeworkforteamstheyperceivedasstrong,theyfoundthemonlytobeperformingsatisfactorilybasedontheBARSdescriptions.TAsfoundinthesecasesthattheBARSdescriptionsdidnotmaptotheirperceptionsoftheteam’sfunctioning.



8.3.2. SequencingoftheBARSStudentsdidnotexpressanyconcernsaroundthesequencingofthedescriptionsalongtheBARS,buttheTAswereunsureofthecontinuumsalongwhichtheBARSweredevelopedforeachcompetency.ForcompetencyR12–“motivateothersontheteamtodotheirbest”–TAsfoundthattheexamplesthatwereprovidedinthedescriptionsoftheBARSinplaceoftheword“motivate”sometimescausedtheretoappearasthoughthereweretwocontinuumsalongwhichthecompetencywasbeingassessed.TAsfoundthisdualcontinuumtooccurmostprominentlyintherelationalaspect,wheretheywouldagreewithonehalfofadescriptionbutnottheotherhalf.Inthissituation,TAswereunsureofhowtoratethestudentandwouldmostfrequentlypickthelowerratingoption.Fortherelationalaspect,theTAscommentedthattheyfoundthein-betweenoptionstobelargelyirrelevant,asthescalewasmorenominalthanordinal.TAsfeltthatthelowendoftheBARSwasnotsequencedappropriately,astheyfeltthatitwasbettertodosomethingwrongthantoavoiddoinganythingatall.Inparticular,withrespecttocompetencyR13–“raisecontentiousissuesinaconstructiveway”–theyfeltthatthedescriptionsforlevels1and3onthescaleshouldbeflipped,sothat“avoidedcontentiousissues”wouldbethedescriptionforlevel1and“raisedcontentiousissuesinadestructivemanner”wouldbethedescriptionforlevel3,asthisorganizationwouldshowaprogressionofengagementwiththeteamthatbettermapstotheconceptualframeworkoftheBARS.Onthelowendofthescaleaswell,theTAscommentedthattheyfeltthatthelanguageusedoftenimpliedthatthestudentswouldbeactingmaliciouslyordemonstratinga“self-interestinthedestructionoftheteam,”whichdidnotmapontotheirunderstandingofthestudents.Thiswasnottheintentionoftheframeworkdesigners,whomeanttocommunicatelackofengagementonthelowestendofthescaleandself-centrednessatthemid-rangeofthescale.TAsfoundthatthelanguagearounddoingsomethingto“serveone’sownpurposes”appearedmalicious,whereastheframeworkdesignersdidnotintenditthatway.StudentsfoundthehighendoftheBARStohavevagueandover-archingcompetencywording,insteadofthecollaborativeorientationintendedbytheframeworkdesigners.ThestudentscommentedthatlanguagesuchasthatinR19–“Collaboratedwithothersinamannerwhichpromotedopennessandunderstandingamongteammembers”–wasopentointerpretationandthereforewouldbeassesseddifferentlybyeveryone.Thestudentsinthefocusgroupcommentedthatadescriptionthat“narrowstheanswermore”andspecifiesamannerofpromotingopennesswouldensurethattheframeworkhadasingleinterpretation.8.3.3. NumberofPointsalongtheBARSTwostudentsinthefocusgroupandtheTAsbelievedthatthereweretoomanyratingoptionsforeachcompetency.Havingdescriptionsononlyfourofthesevenpointsonthescalecreatedadditionalconfusion,astheywerenotsurehowtointerpretthemiddleoptions.Themajorityoffocusgroupparticipantsrecommendedeliminatingtheseundescribedratingpointsasawayofreducingthenumberofpointsalongthescaleandmakingtheratingscaleclearer.



8.4. PresentationofFeedbackStudentsinthefocusgroupwereaskedaboutthepresentationofthefeedback(seeexampleprovidedinAppendixB)todetermineiftherewerewaystomakethefeedbackeasiertoreadandunderstand.Nomarkedlynegativecommentswereprovidedaboutthepresentationofthefeedback,noranycommentssuggestingthatstudentswereunabletointerpretthefeedbackoridentifytheirstrengthsandweaknessesfromit.Studentsinthefocusgroupcommentedthattheylikedthecolourcoding,whichallowedthemtoknow“oninstantglance”whattheirstrengthsandweaknesseswere.However,whilehavingthestrengthsandweaknessesvisuallyseparablefromtheothercompetencieswasaplus,adifferentsetofcolourswasrecommendedbyacolour-blindstudent,whocommentedthatitwasdifficulttodistinguishbetweenthegreenandyellowthatdemarcatedstrengthsfromaveragecompetencies.8.5. MissingContent Finally,toensurethattheframeworkaddressedstudents’feedbackneeds,studentswereaskediftherewasanythingtheyperceivedasmissingfromtheirfeedback.Commentsfromthefocusgroupparticipantswerecombinedwithstudents’responsestoquestion9oftheend-of-termsurvey.Thestrongestthemeamongtheresponseswasthatofadesireforcommentsorexamplesfromthefeedbackproviders–somethingthatcouldguidethereceivertoknowexactlyhoworwhytheyreceivedthenumericassessmentsontheframeworkandhowtheirteammemberswishthemtoimprovetheirperformance.Studentsinthefocusgroupsawthisasawaytojustifyhighorlowassessments,whichwasalsoathemediscussedbytheTAs.TheTAsweremoreinclinedtohaveanoptionaljustificationboxaftereachquestioninordertojustifytheirassessmentand/ortheirconfidenceintheirassessmentwhentheyfeltitwasnecessary.Students,ontheotherhand,simplycommentedthattheywantedsomejustificationbutindicatednopreferencebetweenitbeingpercompetencyorperstudent.Thisideawassupportedbyallstudentsatthefocusgroupaswellasby23%ofthestudentsintheframeworkgroup,whocommentedonthisintheirend-of-termsurvey.Theywantedtoknowwheretheyexcelled,withspecificexamples,andtobeabletolearnbyknowingwhytheywereassessedthewaytheywere.8.6. DiscussionThemostcommonthemethatsurfacedunintentionallyinthefocusgroupswashowthefeedbackcouldbeleverageddifferentlyinfutureclassestobetterfacilitatestudentimprovement.Studentssuggestedsomeotherwaystorepresentthefeedback,includingasanaverageofteammembers’assessments,byshowingtheirperformancerelativetotheirteam’sortheirclass’averageperformance,orbyintegratingtheperformanceleveldescriptionsintothefeedbackinsteadofprovidingitasaseparatehandoutwiththefeedback.Givenengineeringstudents’competitivenature,thismightappealtotheirdesiretobethebestintheclass.Bycomparison,TAsprovidedmorefeedbackabouttheaccessibilityoftheframeworkasanassessmentinstrument.Thisislikelyduetotheirfamiliaritywithgradingrubricsasaresultoftheirteachingwork.



Onerecommendationforimprovingtheeffectivenessofthefeedbackfromthefocusgroups,andsomethingthatwascommentedonbysomestudentsontheirend-of-termsurveys,wastoaddadebriefonthefeedbackduringtutorialtimewiththeteamasawhole.Thisstudentrecommendedtheinclusionofastructureddiscussionwiththeteamabouttheirweaknessestoencourageeveryonetoworkmoreeffectivelytogether.Thiscouldbedesignedasawalk-throughorscriptedexercisethatthestudentsrunthemselves,withtheTAsavailableforsupportasneeded.Thiswouldalloweachteamtoreceiveitsownpersonalizedfeedbackwithinthestructureofthelargeclass.Thisactivitycouldguidethestudentsthroughspecificexercisesoractivitiesfortheteamtoexploreareaswheretheyallareweak,ortolookatnewwaysinwhichtheycanleveragethestrengthsoftheteammembers.Havingthedifferentresourceslistedseparatelyfromthefeedbackonlinedidnotappeartobeadetriment,thoughitwasnotcommentedonpositivelyinthesurveyorfocusgroupseither.Amoreintentionalwaytointegratethefeedbackandmethodsofimprovement–eitherinpersonoronline–needstobeconceived.8.7. ConclusionandResponsetoResearchQuestionCombiningstudentfeedbackabouttheaccessibilityoftheframeworkfromtheirend-of-termsurvey,andstudentandTAfeedbackfromthefocusgroupsattheendoftheterm,weconcludethattheframeworkisaccessibleasafeedbackguideforintra-teamuse.FortheTAs,theframeworkistoolonganddifficulttocompletegiventheirinteractionswiththestudentsandthecognitiveloadoftheirteachingresponsibilities.Bothgroupsidentifiedanumberofwaysinwhichtheframeworkcouldbemademorerelevantandintuitive.

9. ConclusionsandFutureWorkThisprojectevaluatedtheeffectivenessofafeedbackframeworkasafoundationforanewweb-basedtoolthatprovidesstudentswithstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Specifically,thestudyinvestigatedwhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,wouldincreaseastudent’sabilitytolearnaboutandimprovetheirteam-effectiveness.Whenfeedbackfromtheframeworkiscomparedtofeedbackgatheredusinganunstructuredprompt,wefindthatfeedbackfromtheframeworkhasagreaterabilitytoincreasestudentself-awarenessandprovidethemwithinformationthatcanguidethemindevelopingteam-effectivenesscompetencies.Studentswhousedtheframeworkhadsignificantlydifferentstrengthsandweaknessesreliablyidentifiedforthem,andtheyreceivedconsistentassessmentsoftheirperformancefromtheirteammembers.Studentsintheunstructuredfeedbackgrouphadlessconsistentfeedbackthatmoreoftendiscussedstrengthsalonethanbothstrengthsandweaknesses.Additionally,asstudentsintheunstructuredfeedbackgroupdiscussedwhichevercompetenciestheyfeltwereimportant,thefeedbackislesstargetedandfocusesonhowtheteamperceivedthestudent’sperformance.Studentsinthefeedbackgrouphadsignificantlygreateragreementwiththestatementsthattheyreceivedfeedbackonabroaderrangeoftopicsandontheirweaknessesthandidstudentsintheunstructuredgroup.



Whilestudentsintheunstructuredgroupdidnothavethesesamebenefits,theydidcommentthatthetextualfeedbackmadethemfeelmorecommittedtotheirteam,asitdemonstratedthattheirteammembershadaninterestintheirdevelopmentandinthatoftheteamasawhole.Basedonthisbenefitandonrequestsfromstudentsinthefeedbackgroupforexamplesandcommentsintheirfeedback,webelievethatahybridcombiningthefeedbackframeworkandsometextualfeedbackwouldbeideal.Thisdocumentaddressesanimportantcomponentofalarger,ongoingprojectlookingattheeffectivenessofourfeedbackframeworktofacilitatefeedbackthatguidesstudentstolearnaboutanddeveloptheircompetenceaseffectiveteammembers.Wehavealreadyincorporatedmuchofthelearningachievedthroughthisstudyintoourweb-basedtool.Forexample,thenumberofcompetenciesbeingaddressedhasbeensubstantiallyreducedandanoptionforholisticfreeformfeedbackincorporatedintotheframework.Further,asstudentsoftenworkindiverseteams,weareintheprocessofinvestigatinghowdiversityofpersonalitycharacteristicsinteamsaffectsthefeedbackastudentreceivesfromtheirteam.Thisinvestigationiscurrentlyinanalysisandwillbepublishedatalaterdate.



References

Bushe,G.R.,&Coetzer,G.(1995).AppreciativeInquiryasaTeam-DevelopmentIntervention:AControlledExperiment.JournalofAppliedBehavioralScience,31(1),13-30.

CanadianEngineeringAccreditationBoard(2010).AccreditationCriteriaandProcedures.Retrievedfromhttp://engineerscanada.ca/e/files/Accreditation_Criteria_Procedures_2010.pdf

Lencioni,P.M.(2002).TheFiveDysfunctionsofaTeam:ALeadershipFable.SanFrancisco,CA:Jossey-Bass.

Lingard,R.W.(2010).TeachingandAssessingTeamworkSkillsinEngineeringandComputerScience.JournalofSystemics,CyberneticsandInformatics,8(1),34-37.

LoughboroughUniversity(2009).Retrievedfromhttp://www.webpa.ac.uk/

Loughry,M.L.,Ohland,M.W.,&Moore,D.D.(2007).DevelopmentofaTheory-basedAssessmentofTeammemberEffectiveness.EducationalandPsychologicalMeasurement,67(3),505-524.

Maxwell,J.(2011,July).TeamEffectivenessQuestionaire.Retrievedfromhttp://www.firststepstraining.com/resources/teq/index.htm

Moore,T.J.,Diefes-Dux,H.,&Imbrie,P.K.(2006).AssessmentofTeamEffectivenessDuringComplexMathematicalModelingTasks.36thASEE/IEEEFrontiersinEducationConference.SanDiego,CA.

Sheridan,P.K.,Phillips,J.,ElGammal,L.,Evans,G.,&Reeve,D.A.(2013).TeamEffectivenessInventoryforGuidedReflectionandFeedback.AmericanSocietyofEngineeringEducationAnnualConferenceandExposition.Atlanta,Georgia.

Sheridan,P.K.,Reeve,D.,&Evans,G.(2014).UnderstandingTeachingAssistants'AssessmentofIndividualTeamworkPerformance.AmericanSocietyforEngineeringEducationAnnualConferenceandExhibition.Indianapolis,IN.

Tuckman,B.W.(1965).Developmentalsequenceinsmallgroups.PsychologicalBulletin,63(6),384-399.

Teaching Team-effectiveness in Large Classes · Teaching Team-effectiveness in Large Classes ......

Documents

Transcript of Teaching Team-effectiveness in Large Classes · Teaching Team-effectiveness in Large Classes ......