Teaching Team-effectiveness in Large Classes
Transcript of Teaching Team-effectiveness in Large Classes
TeachingTeam-effectivenessinLargeClasses
PatriciaK.Sheridan,GregJ.Evans,DougReeve,UniversityofToronto
Publishedby
TheHigherEducationQualityCouncilofOntario
1YongeStreet,Suite2402Toronto,ONCanada,M5E1E5Phone: (416)212-3893Fax: (416)212-3899Web: www.heqco.caE-mail: [email protected]
Citethispublicationinthefollowingformat:Sheridan,P.K.,Evans,G.J.,&Reeve,D.(2015).TeachingTeam-effectivenessinLargeClasses. Toronto:HigherEducationQualityCouncilofOntario.
TheopinionsexpressedinthisresearchdocumentarethoseoftheauthorsanddonotnecessarilyrepresenttheviewsorofficialpoliciesoftheHigherEducationQualityCouncilofOntarioorotheragenciesororganizationsthatmayhaveprovidedsupport,financialorotherwise,forthisproject.©Queen’sPrinterforOntario,2016
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario3
ExecutiveSummary Instructionofteamskillsisquicklyemergingasanimportantandmissingdimensionofengineeringeducation.Thisprojectevaluatedanewframeworkforguidingstudentsinprovidingself-andpeerassessmentsoftheireffectivenessinteamwork.Thisframeworkisthefoundationforanewweb-basedtoolthatoffersstudentsstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Specifically,thestudydocumentedinthisreportinvestigatedwhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,increasedstudents’abilitiestolearnaboutandimprovetheirteam-effectivenessinexecutingdesignprojects.Theframeworkconsistedof27competenciesacrossthreeaspectsofteam-effectiveness:organizational,relationalandcommunicationcompetencies.Theframeworkwastestedinarandomizedcontrolledexperimentinafirst-yearengineeringdesignclassof280studentsagainstanunstructuredfeedbackprompt.Studentswereaskedtoprovidefeedbackatthemid-pointofthecourseandtoprovidetheirthoughtsontheutilityofthefeedbacktheyreceivedinanend-of-termsurvey.Studentassessmentswerealsocomparedtoteachingassistantassessments.Studentsusingtheframeworkfoundthefeedbacktheyreceivedtobemoreactionablethanunstructuredfeedback,andfoundthatitmotivatedthemmoretoimprovetheirperformancethandidthestudentsreceivingunstructuredfeedback.Studentsintheunstructuredgroupreceivedfeedbackonfewerandlessdiverseteam-effectivenesscompetenciesthanthoseintheframeworkgroup.Studentsintheunstructuredgroupreceivedfeedbackonapproximately10competenciesonaverage,andapproximatelyhalfofthatfeedbackwasonorganizationalcompetencies;studentsintheunstructuredgroupreceivedlittlefeedbackonrelationalcompetencies.Theunstructuredfeedbackalsoprimarilyidentifiedthestudents’strengths,whichwerephrasedaspraiseoftheirperformance,withminimaldiscussionofthestudents’weaknesses.Studentsintheframeworkgroupwereabletopeer-assesstheirteammembersaccuratelywhenusingtheframeworkbutwerelessabletoself-assessaccurately.Whencomparedtotheassessmentsoftheirteachingassistants,students’peerassessmentscorrelatedsignificantlywithteachingassistants’assessmentsacrossallthreeaspectsofteam-effectiveness.However,onlystudents’self-assessmentsalongtheorganizationalaspectcorrelatedsignificantlywithteachingassistants’assessments.Whilestudentsintheunstructuredgroupdidnotreceivethismorecomprehensivefeedback,theydidcommentthatthetextualfeedbackmadethemfeelmorecommittedtotheirteam,asitdemonstratedthattheirteammembershadaninterestinthemandintheteamasawhole.Basedonthisbenefitandonstudentsinthefeedbackgroup’srequestsforexamplesandcommentsintheirfeedback,webelievethatahybridofthefeedbackframeworkalongwithsometextualfeedbackwouldbethebestmethodforprovidingfeedbackmovingforward.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario4
TableofContents
1. Introduction...................................................................................................................................................9
2. BackgroundandMotivation..........................................................................................................................9
3. FeedbackFramework..................................................................................................................................11
4. StudyDesignandResearchMethods..........................................................................................................12
4.1. CourseContext ..................................................................................................................................... 13
4.2. StudyDesign ......................................................................................................................................... 14
4.3. StudentParticipation ........................................................................................................................... 14
4.4. TeachingAssistantParticipation .......................................................................................................... 16
4.5. ParticipationRates ............................................................................................................................... 17
4.6. ResearchMethods ............................................................................................................................... 18
5. DifferencesintheQuantity,BreadthandAccuracyofStudentFeedback..................................................19
5.1. QuantityofTeamMemberEffectivenessContent .............................................................................. 19
5.2. BreadthofTeamMemberEffectivenessContent ................................................................................ 21
5.3. OtherTypesofInformationintheFeedback ....................................................................................... 23
5.3.1.UnrelatedFeedback .................................................................................................................... 23
5.3.2.Team-levelfeedback ................................................................................................................... 23
5.4. Accuracy–AgreementbetweenSelf-andPeerAssessments ............................................................. 23
5.4.1.UnconsideredAssessments ........................................................................................................ 24
5.4.2.AgreementintheUnstructuredFeedback .................................................................................. 24
5.4.3.AgreementintheFrameworkFeedback ..................................................................................... 25
5.4.4.PerceivedAccuracy ..................................................................................................................... 26
5.5. Discussion ............................................................................................................................................ 27
5.5.1.LimitedQuantityofUnstructuredFeedback ............................................................................... 27
5.5.2.StudentPrivilegingofOrganizationalCompetencies .................................................................. 28
5.5.3.LimitedUtilityofSelf-assessmentsintheUnstructuredGroup .................................................. 28
5.6. ConclusionandResponsetoResearchQuestion ................................................................................. 29
6. DifferencesinStudentPerceptionsoftheUsefulnessoftheirFeedback...................................................29
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario5
6.1. IdentificationofaStudent’sStrengthsandWeaknesses ..................................................................... 29
6.1.1.UnstructuredFeedback ............................................................................................................... 29
6.1.2.FrameworkFeedback .................................................................................................................. 30
6.1.3.StudentPerceptionoftheStrengthsandWeaknessesIdentified .............................................. 31
6.2. ImprovementbasedonFeedback ........................................................................................................ 32
6.3. Discussion ............................................................................................................................................ 33
6.4. ConclusionandResponsetoResearchQuestion ................................................................................. 34
7. DifferencebetweenStudentandTAAssessments......................................................................................35
7.1. SimilarityinIdentificationofaStudent’sStrengthsandWeaknesses ................................................. 35
7.2. CorrelationbetweenTAandStudentFrameworkAssessments .......................................................... 36
7.3. Discussion ............................................................................................................................................ 37
7.3.1.TeachingAssistantHesitancyaroundtheAssessmentContext .................................................. 37
7.3.2.LimitationsontheAgreementforUnstructuredAssessments ................................................... 38
7.3.3.Aspect-levelDiscrepanciesinAssessment .................................................................................. 38
7.4. ConclusionandResponsetoResearchQuestion ................................................................................. 38
8. StudentandTAFeedbackontheUsabilityoftheFramework....................................................................39
8.1. HolisticAssessmentofFrameworkAccessibility .................................................................................. 39
8.2. AccessibilityoftheFramework’sCompetencies .................................................................................. 40
8.3 AccessibilityoftheCompetencies’BehaviourallyAnchoredRatingScales ......................................... 40
8.3.1.BARSDescriptions ....................................................................................................................... 40
8.3.2.SequencingoftheBARS .............................................................................................................. 41
8.3.3.NumberofPointsalongtheBARS ............................................................................................... 41
8.4. PresentationofFeedback .................................................................................................................... 42
8.5. MissingContent ................................................................................................................................... 42
8.6. Discussion ............................................................................................................................................ 42
8.7. ConclusionandResponsetoResearchQuestion ................................................................................. 43
9. ConclusionsandFutureWork......................................................................................................................43
References........................................................................................................................................................45
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario6
ListofFiguresFigure4-1:Overviewofthestudydesigndemonstratingwhichparticipantscompletewhichassessments,surveysandattendfocusgroups......................................................................................................................14
Figure4-2:Trainingsequenceinwhichtheteachingassistantsparticipatedpriortoprovidinganyassessmentsofstudentperformance...............................................................................................................17
Figure5-1:Numberofframeworkcompetenciesdiscussedinfeedbackreceivedbystudentsintheunstructuredfeedbackgroupinresponsetotheprompt“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject”.......................................20
Figure5-2:Percentofrelationalcontentinanunstructuredgroup’sstudent’sfeedbackbasedonthetotalamountoffeedbackreceived...........................................................................................................................21
Figure5-3:Distributionofresponsestothestatement“Ireceivedfeedbackon:abroadrangeoftopics”bygroup.................................................................................................................................................................22
Figure5-4:Distributionofperformanceagreementinstudentfeedbackforcompetenciesthatwerediscussedincommonbypeerassessors...........................................................................................................25
Figure6-1:Percentofpeer-feedbackreceivedbystudentsintheunstructuredfeedbackgroupthatdiscussedtheirstrengths..................................................................................................................................30
Figure6-2:Distributionofresponsestothestatement“thefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare”bygroup.......................................................................................32
Figure6-3:Numberofweaknessesidentifiedinstudents’end-of-termsurveysthattheycommentedasimprovingonbasedontheirfeedback,bygroup.............................................................................................34
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario7
ListofTablesTable3-1:The27competenciesofthefeedbackframeworkdividedintothreeaspectsofteammembereffectiveness.Thecompetencynumbersprecedingeachcompetencywillbeusedtoreferencethecompetenciesintheanalysis............................................................................................................................11
Table4-1:Demographicinformationoftheteachingassistants(TAs)whoparticipatedinthestudy,theirbackgroundsandtheirrespectivepreviousaffiliationswiththecourse..........................................................16
Table5-1:Contingencytableanalysiscomparinggroupagreementonreceivingabroadrangeoftopicsintheirfeedback...................................................................................................................................................22
Table5-2:DistributionofagreementbetweenpeerassessmentsofastudentasmeasuredusingICCsinatwo-wayrandomeffectsmodelforconsistency..............................................................................................26
Table5-3:Spearman’srankcorrelationbetweenstudents’self-andpeerassessmentsforeachaspectoftheframework........................................................................................................................................................26
Table6-1:Contingencytableanalysiscomparinggroupagreementonreceivingidentifiedweaknessesintheirfeedback...................................................................................................................................................31
Table6-2:Contingencytableanalysiscomparingthetoneofthefeedbackreceivedbythetwogroups.......33
Table7-1:FrequencyofagreementbetweenstudentandTAidentifiedthesamestrengthsandweaknessesintheunstructuredandframeworkfeedbackgroups......................................................................................36
Table7-2:Spearman’srankcorrelationbetweenstudents’peerassessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework..........................................................................................36
Table7-3:Spearman’srankcorrelationbetweenstudents’self-assessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework..........................................................................................37
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario8
AcknowledgmentsTheauthorswouldliketothankJasonFosterandRobIrishforallowingthisresearchtooccurintheircourse,JenniePhillipsforherassistanceinfacilitatingandtranscribingtheparticipantfocusgroups,LobnaElGammalforherassistancewiththedataanalysis,EstelleOliva-Fisherforherassistancewithquestiondevelopment,CindyRottmanforherguidancewiththeresearchmethodology,JasonFosterforhistechnicalassistancewiththesurveyingsystem,aswellasSusanMcCahan,JasonBazylakandPeterWeissfortheirconstructiveinputinthedesignandexecutionofthestudy.Theauthorswouldliketoespeciallythankallofthestudentsandteachingassistantswhomadethisstudypossible.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario9
1. IntroductionIntentionalinstructionofteamskillsisquicklyemergingasanimportantandmissingdimensionofengineeringeducation.Thisprojectinvestigatedandevaluatedanewinterventionthatsupportspersonalizedlearningofteameffectivenessinlargeundergraduatecourses.Specifically,weevaluatedtheeffectivenessofanewweb-basedtooltoprovidestudentswithstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Thisinterventionaimedtoprovidestudentswithasafe,virtualenvironmentinwhichtheycould:i)learnabouttheirteam-effectivenessandteamissues,andii)identifymethodstoimprovetheirareasofweaknessbeforetryingthemwiththeirteammembers.Afundamentalcomponentofthisinterventionisthefeedbackframeworkthatformsthebasisofthetool.Thefeedbackframeworkgroundstheassessmentsstudentsprovideandthefeedbacktheyreceivealongspecificcompetencies,guidingstudentstorecognizeandreflectonnecessaryteamworkskills.Thisstudyaimedtoinvestigatewhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,wouldincreaseastudent’sabilitytolearnaboutandimprovetheirteam-effectivenessintheirprojectteams.
2. BackgroundandMotivationEngineeringisateam-basedprofessionthatrequiresstudentstobebothtechnicallyproficientandeffectiveatteamwork.TheCanadianEngineeringAccreditationBoardmandatesthatallstudentswhograduatefromengineeringprogramsbecapableofworkingeffectivelyaloneaswellasinteams(CanadianEngineeringAccreditationBoard,2010).Upongraduation,whenstudentstransitiontoworkinginindustry,theywillworkonlarge-scale,complexproblemsthatrequiremultipleindividualsworkingeffectivelyinteamstodesignappropriatesolutions.Inendeavouringtopreparestudentsforthisenvironment,mostundergraduateengineeringprogramshavestudentsworkonteam-basedprojectsthatmodelwhatthestudentswillexperienceinindustry.Theprojectsendeavourtoinstillinstudentstheskillsnecessarytoworkeffectivelyintheseteamsaswellasintheirfuturecareers.However,studentshavetraditionallybeenexpectedtodevelopteamworkskillsimplicitlysimplybyparticipatingintheteamproject.AttheUniversityofToronto,engineeringstudentsareintroducedtoteamworkintheirfirst-yearengineeringdesigncourses.First-yearclasses,inwhichstudentslearnthefoundationalskillstheyneedfortheremainderoftheirdegreesandfortheircareers,aretraditionallythelargest.Forexample,classsizeintheengineeringprogramattheUniversityofTorontorangesfrom100to1,000students,withthetwofirst-yearengineeringdesignclassesbeingthelargestatapproximately280and1,000students.Inthesecourses,studentshavebeengroupedintoteamstosimulateindustryworkingconditions.However,thesizesoftheseteams(rangingfromthreetosevenstudents)aremainlyconstrainedbyfacilitiesandinstructor/teachingassistanttime,especiallygradingtime.Intheselargeclasses,providingfeedbackonstudentperformanceinatimelymannerisalwaysachallenge.Asaresult,thesestudentshavetraditionallyreceivedfeedbackonthequalityoftheirteam’sdeliverables(e.g.,designreports,presentationsandprototypes)fromtheirteachingassistants,butrarelyreceivefeedbackonhowtheyasindividualsareperformingasteam
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario10
members.Courseshavetraditionallyinformedstudentsthattheteachingassistantsareavailabletosupportbutnotassesstheirteam-effectiveness.Theteam-basedprojectenvironmentinfirst-yearengineeringdesigncoursesisahigh-stakesenvironment.Theprojectsaretoocomplexforonepersontoaddressontheirownandrequiresomelevelofcollaborationbetweenteammembers.Thestudentsdonotalwaysworkeffectivelytogetherbutarestillgradedtogether–thewholeteamreceiveswhatevergradethedeliverablereceives,regardlessofhoweffectivelytheyworkedtogether.Whenstudentsintheprojectteamsdonotpulltheirweight,theremainderofthestudentsintheteamlearnnottoliketeamworkandnottotrusttheirteammembers.Theyseeteamworkasanundesirableexperiencebeingforcedonthemthatjeopardizesthethingtheyvalue:theirgrades.Ourbeliefisthatmanyoftheseissuescouldbemitigatedbyprovidingstudentswithpersonalizedsupportinthedevelopmentoftheirteamworkskills.Thefirstpartofthissupportisprovidingstudentswithanawarenessofwhatishappeningintheirteams.Insmallclasses,aninstructorcandothisthankstothepersonalnatureofteacher-studentinteractionsresultingfromasignificantlydecreasedassessmentload.However,evenhere,theinstructormayonlyseepartofwhatisoccurringwithinateam.Thus,duetothewayinwhichstudentteamswork,webelievethatthebestpeopletoprovidethisfeedback,bothinlargeandsmallclasses,arethestudentsthemselves.Theyseetheirteammembersfortheentiredurationoftheproject,inparticularinteammeetingsoutsideofclassortutorialtimewheninstructorsarenotpresent.Ourobjectiveistouseselfandpeerassessmentwithinthestudentprojectteamstoprovidepersonalizedfeedbacktoeachoftheteammembersontheireffectivenessasateammemberduringtheproject.Approachestoconductingselfandpeerassessmentsofteamworkskillsinengineershavebeenapopularfocusofresearchoverthelastfiveto10years.Web-basedtoolshavebeendevelopedintheUS,mostnotablytheComprehensiveAssessmentofTeam-memberEffectiveness(Loughry,Ohland&Moore,2007),andintheUK,theWebPA(LoughboroughUniversity,2009).Thesetoolsoriginatedtoensurethatinstructorshavesufficientinformationtoknowiftheyneedtomodifyindividualstudents’gradesbasedontheirrelativecontributionstoateamdeliverable.Asthesetoolshaveexpandedtobecomeweb-basedresourcestofacilitatethelearningofteam-effectiveness,theyhaveapproachedteamworkfromastrongtask-focusedorientation.Studentsinengineeringarealreadytraditionallymoreorientedtowardsthetasksideofteamworkthantothepeopleside,sothereexistsagreaterneedtofacilitatetheirdevelopmentoftherelationalandcommunicationskillsofteamwork.Asaresult,ratherthanadoptingoneoftheexistingframeworks,wedevelopedourownthatpushesstudentstothinkaboutrelationalskillsinadditiontotaskskillswhenprovidingandreceivingfeedback.Wesoughtwiththisstudytoassesstheeffectivenessofthisselfandpeerassessmentfeedbackframework.Thisreportdocumentstheassessmentofourteammembereffectivenessfeedbackframeworkasafoundationtofacilitatestudentdevelopmentoftheircompetencyasteammembers.Feedbackoriginatingfromtheuseofthefeedbackframeworkwascomparedtofeedbackdevelopedfromanunstructuredfeedbackprompttoassesstheusefulnessandeffectivenessofthefeedbackframework.ThisreportfirstpresentstheframeworkinSection3,outlinesthestudydesignandresearchquestionsinSection4,
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario11
respondstotheseresearchquestionsindividuallyinSections5-8,andfinallyconcludeswithourfindingsaroundtheutilityofthefeedbackframework.AdiscussionofplannedfutureworkisoutlinedinSection9.
3. FeedbackFrameworkTheteam-effectivenessfeedbackframeworkthatwasusedinthisinterventionispresentedinTable 3-1. Table3-1:The27competenciesofthefeedbackframeworkdividedintothreeaspectsofteammembereffectiveness.Thecompetencynumbersprecedingeachcompetencywillbeusedtoreferencethecompetenciesintheanalysis.
OrganizationalAspects RelationalAspects CommunicationAspects
O1.Supportteamrules R11.BuildthetrustofteammatesC20.Exchangeinformationinatimelymanner
O2.Attendteammeetingsprepared R12.Motivateothersontheteamtodotheirbest
C21.Introducenewideas
O3.Contributetomakingmeetingseffective
R13.Raisecontentiousissuesinaconstructiveway
C22.Openlyexpressopinions
O4.Dotheirfairshareofthework R14.Solicitinputbeforeproceeding C23.Promoteconstructivebrainstorming
O5.DelivertheirworkontimeR15.Adoptsuggestionsfromothermembers C24.Activelylistentoteammates
O6.ProducehighqualityworkR16.Acceptfeedbackaboutstrengthsandweaknesses C25.Provideconstructivefeedback
O7.Helptoplan,setgoals,andorganizework
R17.Showrespectforotherteammates
C26.Makesurethatteammatesunderstandimportantinformationandinstructions
O8.Trackteamprogressvs.yourtimeline
R18.Demonstrateaccountability C27.Helptheteambuildconsensus
O9.Encourageprogresstomeetgoalsanddeadlines R19.Collaborateeffectively
O10.Displaydedicationanddetermination
Note:Thisframeworkhasbeenanalyzedinotherstudiesaswell.SeeSheridanetal.(2013)and(2014).Thisframeworkwasdevelopedfromfourexistinginventoriesanddesignedtoincludeagreaterfocusonnon-task-relatedcompetencies(Bushe&Coetzer,1995;Lingard,2010;Moore,Diefes-Dux&Imbrie,2006;Maxwell,2011).Asynthesisoftheseinventorieswasdevelopedandredundancybetweenbehaviourswaseliminated.Competencieswerecategorizedaccordingtothreeaspectsofteammembereffectiveness(Table 3-1):organizationalcompetencies,whichfocusontheprojectmanagementandtaskcompletionaspectofeffectiveteamwork;relationalcompetencies,whichfocusonhowstudentsbuildeffectivemeansofcollaboratingtogetherandtrustineachother;andcommunicationcompetencies,whichfocusonhowstudentsleveragetheirinteractionstoexchangeinformationandpromoteproductivediscussions.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario12
Foreachcompetency,thereisabehaviorallyanchoredratingscalethatdescribesastudent’slevelofengagementinutilizingthatcompetencytoimproveteamperformance.Studentsassessthemselvesandtheirteammatesaccordingtoa7-pointdescriptiveLikertscale,whereanassessmentof1-2representsanunengagedteammember,3-5aself-focusedteammember,and6-7ateam-focusedteammember.Toobtainateam-focusedassessmentalongeachcompetency,thestudenthastodemonstrateanabilitytoexhibitandpromotethebehavioursofthecompetencyintheirteammembers.ThebehaviourallyanchoredratingscaleforallcompetenciesisavailableinAppendixA.Bystructuringtheassessmentsusingabehaviourallyanchoredratingscaleforeachcompetency,studentsshould:i)beabletoassessusingacommonscale,increasingtheconsistencyinassessmentsreceivedbyastudent,andii)beabletoprovidefeedbackthatisspecificenoughthatateammatecanidentifytheirperformancelevelandhowtoimproveit.
4. StudyDesignandResearchMethodsTheobjectiveofthestudywastounderstandhowstudentsperceive,interpretanduseourfeedbackframeworktopromotelearningaboutteammembereffectivenessbehavioursintheirprojectteamsthroughtheuseofselfandpeerassessment.Ourgoalwastodesignaneffectiveframeworktoguidestudentstoprovidefeedbackinordertofacilitatethelearningofteamworkwithinstudentprojectteamsinlargeclasses,withminimalneedforadditionalcontact-hourresources.Ourhypothesiswasthat:
Studentscanbeguidedtoprovideusefulpersonalizedfeedbackonteam-effectivenesstotheirteammatesusingourteam-effectivenessframework.
AstudytoassesstheutilityoftheframeworkinfacilitatingfeedbackwastestedintheWinter2012termofPraxisII,a280-studentcornerstonedesigncourseinfirst-yearengineeringattheUniversityofToronto.Thiscoursepreviouslyhadstudentsprovidesomeunstructuredcommentsabouttheirteammemberstotheteachingteamasameansofexpressingconcernincaseswhereteammembersdidnotcompleteequitablesharesoftheprojectwork.Asprovidingunstructuredcommentsabouteachteammemberwasalreadyamodeofreflectionaboutteam-effectivenessinthecourse,wewantedtocompareourframeworktothepre-existingunstructuredfeedbackmethodtodeterminewhetherourframeworkcouldbemoreusefulinguidingstudentstoimprovetheirteam-effectivenessbehaviours.Toassessthis,weusedthefollowingfourresearchquestionstoguideourinquiry:
1. Doestheframeworkguidestudentstoprovideagreaterbreadth,quantityoraccuracyoffeedback?2. Dostudentsperceivethepeerfeedbackfromtheframeworktobemoreusefulthanunstructured
feedback?3. Canstudentsprovidefeedbacksimilartothatofatrainedobserver(courseteachingassistant)when
usingtheframework?4. Istheframeworkaccessible(e.g.,jargon,descriptions,levelsofcompetency)tostudentsandtrained
observers?
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario13
Thetheoreticalorientationtakentolearninginthisstudyisthatofsocialconstructivism.Thisisbecausetheteamactsasalearningcommunityinwhichtodevelopagreaterunderstandingofone’sownteammembereffectivenessbehaviours,aswellasagreaterunderstandingofteammembereffectivenessingeneral.Selfandpeerassessmentsofteammembereffectivenessactasscaffoldingtosupportstudentsinobservingandthereby developing a better understanding of team member effectiveness behaviours. Improvement inbehaviour isbelievedtobemotivatedboth intrinsicallyandextrinsically throughthedesiretobeabetterteammemberandtocreateabetterteam,respectively.4.1. CourseContextPraxisIIisafirst-yearengineeringdesigncourseinthefirst-yearEngineeringScienceprogramattheUniversityofToronto.PraxisIIisa280-studentcoursethattakesplaceinthesecondtermoftheschoolyearandbuildsontheintroductorydesign,communicationandteamworkprinciplescoveredintheirfirst-termcourse,PraxisI.ThecourseisbuiltonthepedagogicalmodelsofKolb,PerryandVygotskyinthedesignandexecutionofitscourselectures,tutorialsandassignments.Specificallyrelatedtothisstudy,theclass(studentsandteachingassistants)attendedthreehalf-hourlecturesonteam-effectiveness,wheretheywereintroducedtoTuckman’s(1965)andLencioni’s(2002)modelsofeffectiveteamwork,theaspectsandcompetenciesofourfeedbackframework(Table 3-1),andhowthesebehavioursmanifestinhighlyeffective,high-performanceteams.Thecourseisdesignedasaservice-,project-andteam-basedlearningcourseinwhichstudentsspendthefirsthalfofthe13-weektermengagingwithcommunitiesaroundtheCityofTorontotoidentifyengineeringdesignopportunitiestheycanaddress,andthesecondhalfofthetermaddressingasubsetofopportunitiesselectedbytheteachingteamandbasedonthoseidentifiedinthefirsthalfoftheterm.Thestudentsworkinself-selectedteamsofthreetofourstudentsfortheentire13weeksofthecourse.Studentscanselecttheirteamfromthe22to27studentsintheirtutorialsection;studentsmaynotbeabletoselecttoworkwiththeirfriendsiftheyarenotinthesametutorialsection.Asaresultofworkingwiththesameteamforthe13weeksofthecourse,studentshavesufficienttimetodeterminehowtheyandtheirteammembersworkinteams,aswellasimprovetheirteamworkskillsandbehavioursbasedonfeedbackduringthecourse.Thecoursehasateachingteamofninemembers:twocourseinstructors,oneofwhomisadesignexpertandtheotheracommunicationexpert,andseventeachingassistants,whoseareasofexpertisestraddleengineering,theartsandthehumanities.Studentsaregroupedintotutorialsof22to27studentsandarepairedwithtwoteachingassistantsofcomplementarybackgrounds.Theroleoftheteachingassistantsistochallengetheideasofthestudents,instillgooddesignandcommunicationpractices,modelconstructiveandcollegialdisagreement,anddemonstrateeffectivecollaborationintheirteachingstyle.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario14
4.2. StudyDesignThestudylookedattheeffectivenessofthefeedbackframeworkbycomparingittounstructuredfeedbackinarandomizedcontrolledexperimentwithinthesameclass.AnoverviewofthestudydemonstratingwhichparticipantscompletedwhichcomponentsisprovidedinFigure4-1.Figure4-1:Overviewofthestudydesigndemonstratingwhichparticipantscompletewhichassessments,surveysandattendfocusgroups
4.3. StudentParticipationTheentireclass’teamsweredividedrandomlyandapproximatelyequallyintotheunstructuredandframeworkfeedbackgroups.Individualstudentswerethenabletoconsenttolettingususetheirfeedbackforresearchpurposes.Studentteamswereusedtoseparatestudentsintotheexperimentgroupsinordertoensurethatallstudentswithinateamwereusingonetypeoffeedbackmechanism,andthatstudentsonlyreceivedfeedbackofonetype.Immediatelyafterthestudentteamssubmittedtheirfirstmajordeliverableinweek7,studentswereaskedtoprovideself-andpeerassessmentsoftheireffectivenessasteammembers.Studentsprovidedpeerassessmentswithintheirprojectteamsonlyfortheirteammembers.Forty-eightpercentoftheclasswasgroupedintotheframeworkgroupandcompletedthisassessmentusingourfeedbackframework(AppendixA).Thesestudentscompletedtheirassessmentsbyselectingthedescriptorthatmatchedtheirteammember’sbehaviourforeachquestion.Theremaining52%weregroupedintotheunstructuredgroupandcompletedthisassessmentbyprovidingunstructuredfeedbackthatrespondedtotheprompt:“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject.”
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario15
Studentsinbothgroupswereaskedtoprovidefeedbackfortheirteammembersonlineonsequentialpagesofasurvey.StudentscompletedthesesurveysonLimeSurvey,anopen-sourcesurveyingsoftwarethatwasimplementedonasecureserveroncampus.Studentswereprovidedalinktocompletetheirsurvey,anddependingonwhethertheywereintheframeworkorunstructuredfeedbackgroup,theysawandcompletedadifferentsurvey.Oneweekaftercompletingtheassessment,studentsreceivedtheirselfandpeerfeedbackonlineforreview.Studentsintheframeworkgroupreceivednumericalfeedbackfromtheirteammembersthatcorrespondedtothedescriptorsthestudentsusedtoprovidetheirassessments(AppendixA).Feedbackintheframeworkgroupidentifiedforeachstudenttheirlowest-rankedcompetenciesasweaknesses(byhighlightingtheirfeedbackalongthesecompetenciesinred)andtheirhighest-rankedcompetenciesastheirstrengths(byhighlightingtheirfeedbackalongthesecompetenciesingreen).Therankingofthesecompetencieswasdeterminedaccordingtoasortingalgorithmthattookintoaccountthevaluesoftheself-assessment,peerassessments,andthedifferencebetweenthetwo.Thealgorithmworkedtoidentifythreestrengthsandthreeweaknessesforthestudents.However,inthecaseofcompetencieswithtiedrankings,theminimumnumberofranksthatresultedinthreeormorecompetenciesbeinghighlightedwerecolouredasstrengthsorweaknesses,respectively.Forexample,iffourcompetenciesallhadthesamerankingthatwashigherthanallothercompetencies,onlythatrank(thosefourcompetencies)wouldbehighlighted.Includingallstudentswhohadtiedrankings,themaximumnumberofcompetenciesthatwaseverhighlightedasastudent’sstrengthsorweaknessesbasedonthesortingalgorithmwasfive.Studentsintheunstructuredgroupreceiveduneditedtextualfeedback,withnostrengthsandweaknessesintentionallyhighlighted.Studentsinbothgroupsalsoreceivedalistoftoolsandtechniquesthattheycouldusetoenhancetheireffectivenessforeachofthecompetenciesoutlinedintheframework.AnexampleofframeworkfeedbackisprovidedinAppendixB,andanexampleofunstructuredfeedbackisprovidedinAppendixC.Attheendofthecourse,studentsinbothgroupscompletedthesameend-of-termsurveyontheusefulnessofthefeedbacktheyreceived(AppendixD).Questionsinthesurveyanalyzedthequantity,depthandbreadthofthefeedback,aswellasitsabilitytomotivatestudentstoimprovetheireffectivenessinteamsbasedonthefeedback.Focusgroupswithstudentswhousedtheframeworkwereconductedaftertheend-of-termsurveytoassesstheframework’sutility,capturedataonhowparticipantsperceivedandusedit,andtheirperceptionsoftheutilityofthefeedbackitgenerated.ThefocusgroupswereconductedbyanexperiencedfocusgroupfacilitatorfromoutsideofengineeringandaskedthequestionsoutlinedinAppendixE.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario16
4.4. TeachingAssistantParticipationTeachingassistants(TAs)participatedinthestudybyusingtheframeworktoprovideassessmentsforallstudentsintheirtutorialsections.Teachingassistantswereusedastrainedobserverstoassesstheconcurrentvalidityoftheframework.TAsaretypicallytheindividualswhohaveidentifiedandaddressedteamdysfunctionsituationsandwhostudentsinthecourseapproachwhentheyhavequestionsorissueswiththeirteamwork.Additionally,withaclassof280students,theseweretheonlyindividualswhohadsufficientone-on-onecontactwiththestudentsintheirteams,andwerealreadyscheduledtobesupporting,andthereforeobserving,themintutorials.ThedemographicinformationofthesevenTAsisshowninTable4-1.TheseTAssupportedninetutorialsectionsinpairs,witheachTAstaffingatleasttwotutorialsandsomestaffingthree.ThepairingofTAsintutorialsprovidedafortuitousopportunitytoacquiretwoobservationsofeachstudentwithwhichtocomparethestudents’self-andpeerassessments.ThisallowedustoassesswhetherstudentsprovidedsimilarfeedbacktotheirtutorialTAs,aswellastheabilityoftheTAsastrainedobservers.TeachingassistantsweretrainedbeforeprovidingfeedbackabouttheirstudentsaccordingtotheprocessshowninFigure4-2.Afterattendingthesamelecturesonteam-effectivenessasthestudents,theTAsmetwiththedesignersoftheframeworkfor1.5hourstodiscusshowtheywouldbeusingtheframeworktoassesstheirstudents.AsalmostallTAshadpreviouslybeenTAsinthecourse,thegroupelectedtousethepreviousyear’steamsthattheyandtheframeworkdesignerwerefamiliarwithasthecasestudiestocalibratetheirassessments.TheseexemplarteamsweredescribedtothetwonewTAsinfrontoftheframeworkdesignerbytheotherfiveTAs,sothattheycouldallassesstheteamsbasedontheprovideddescriptions.Usingthesecasestudies,theTAsnegotiatedwhathighandlowperformancelookedlikeforeachframeworkcompetencyuntiltheirassessmentsofperformancewereinagreement.Finally,theTAsaskedtheframeworkdesigneranyquestionsofclarificationaboutwhatdifferentbehavioursintheframeworkcouldlooklikeoutsideoftheteamsdiscussed.Table4-1:Demographicinformationoftheteachingassistants(TAs)whoparticipatedinthestudy,theirbackgroundsandtheirrespectivepreviousaffiliationswiththecourse
TA1 TA2 TA3 TA4 TA5 TA6 TA7
Background Eng Eng Eng Eng Eng Arts Humanities
Gender Female Male Male Male Female Female Female
Previousexperiencewiththecourse
Student Student StudentandTA
StudentandTA
TA TA TA
NativeEnglishSpeaker Yes Yes Yes Yes Yes Yes Yes
Note:Forbackground,Engrepresentsengineering
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario17
Figure4-2:Trainingsequenceinwhichtheteachingassistantsparticipatedpriortoprovidinganyassessmentsofstudentperformance
TAswereaskedtocompletetheirassessmentsdifferentlythanwerethestudentsintworespects.First,teachingassistantscompletedtheassessmentstwoweeksafterthestudents.Asstudentsprovidedtheirfeedbackimmediatelyaftersubmittingtheirfirstmajorteamdeliverable,TAsweregradingatthattime.Approximatelytwoweeksafterthestudentsprovidedfeedback,theTAswereaskedtoindependentlyprovidefeedbackforeverystudentintheirtutorialsectionsusingtheframework.AsthismeantthateachTAhadbetween54and81studentstoassess,theyweregivenmoretimetocompletetheframeworkthanthestudentteamswhoonlyhadtoassessthreeorfourteammembers.TheTAsweregiventwoweekstocompletetheseassessments,meaningthatsomeTAassessmentswerecompleteduptoamonthfromthetimewhenthestudentscompletedthem.Second,atthetrainingsessionTAsrequesteda‘donotknow’optionforeachofthecompetencies.TheTAscommentedthattheywouldprefernottobeforcedtoprovideanincorrectassessmentforastudentwhentheyfeltthattheywerenotabletoassessthestudentalongthatcompetency.AfocusgroupwiththeTAswasconductedaftertheendoftermtoassesstheframework’sutility,capturedataonhowtheyperceivedandusedit,andtheirperceptionsoftheutilityofthefeedbackitgenerated.Thefocusgroupwasconductedbyanexperiencedfocusgroupfacilitatorfromoutsideofengineeringandaskedthesamequestionsasthestudentfocusgroup,outlinedinAppendixE.4.5. ParticipationRatesTherateofstudents’consenttoparticipateintheresearchstudywashighat77.5%,totalling218studentsfromtheclass–112fromtheunstructuredgroupand106fromtheframeworkgroup.Sincethestudentshadtocompletetheassessmentsoftheirteammembersforcoursepurposes,aspartofa2%completed/notcompleteddeliverable,therewasnoadditionaleffortonthepartofthestudentstoparticipateinthestudy.Studentswereincentivizedtoallowustousetheirdatainthestudythroughtheopportunitytowinone$100giftcard.Ofthe112studentsintheunstructuredgroupwhoagreedtoparticipateinthestudy,thedatafrom105studentswereusedinthisanalysis.Oneteamofthreestudentswascompletelydysfunctionalandasaresult,theyusedtheirunstructuredfeedbackasaplacetoventtheirangerratherthanprovidefeedback.Thesedatawereremoved,aswefeltthattheyhadthepotentialtosignificantlyskewtheresultsofthedepthandbreadthoftheunstructuredfeedbackcontent.Fourotherstudentswereremoved,aseachofthemhadonlyoneotherteammemberwhoconsentedtoparticipateinthestudy,thereforeeliminatingthe
Aqendcourselectureonteam-effecsvenesswithstudents
Negosatewhathighandlowperformancealongeachframeworkcompetencylookslike
Asktheframeworkdesignersquessonsofclarificasonandprocess
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario18
abilitytocompleteanyfeedbackcomparisonsbetweenpeers.All105studentsprovidedfeedbackandrespondedtotheend-of-termsurvey.Ofthe106studentsintheframeworkgroupwhoagreedtoparticipateinthestudy,thedatafrom105studentswereused.Onestudent’sfeedbackwasremoved,astheyweretheonlypersonintheirteamtoconsent,providingnopeerfeedbackforthisindividual.Only95ofthesestudentsrespondedtotheend-of-termsurvey.Allteachingassistantsinthecourseagreedtoprovidefeedbackontheirstudents,aswellasparticipateinafocusgroupontheiruseofthefeedbackframework.Asthiswasadditionalworkontopoftheteachingassistants’normalcourseduties,teachingassistantswerecompensatedattheuniversity’steachingassistantpayratefortheadditionalhoursrequiredtocompletetheframeworkfortheirstudents.Whiledatawereobtainedfromtheteachingassistantsfor100%ofthestudents,onlythedataforthe77.5%whoconsentedtoparticipatewereusedfortheanalysis.Eitherduetothelackofgrade-basedincentivesortothetimingafterexams,studentparticipationinthefocusgroupswassignificantlylowerthanexpectedatthreestudents,makingallfindingsfromthestudentfocusgroupinformativebutnotsignificant.4.6. ResearchMethodsThestudywasconductedasacontrol-conditionexperimentwithinthesameclass,thusallowingustocomparetheeffectsofbothmodesofprovidingfeedbackwithinthesamecontext.ThestudywasapprovedbytheUniversityofToronto’sResearchEthicsBoardforSocialSciences,Humanities,andEducation.Thisstudyfollowedamixedmethodsapproachthatusedbothqualitativeandquantitativeinquiry.AllqualitativeanalysiswascompletedinNVivo,withquantitativeanalysescompletedinSPSSandMicrosoftExcel.Feedbackthatstudentsreceivedinresponsetotheunstructuredfeedbackpromptwascodedaccordingtothefeedbackframework(Table3-1)toidentifywhichcompetencieswerediscussedincommonbetweentheunstructuredfeedbackandtheframeworkfeedback.Codesweresetupforthe27competencies,aswellasforfourothercompetencythemesthatemergedintheunstructuredfeedbackbutdidnotmapdirectlyontotheframework’scompetencies.AdiscussionofthesethemesisincludedinAppendixF.Thenumberofoccurrencesofeachofthecompetencycodeswasthencalculatedforcomparisonwiththeframeworkfeedbackandoverlapsbetweencodesinvestigated.Additionally,thetypeoffeedbackprovidedtoastudent(individualorteam-level)aswellasthecontentofthefeedback(designrelated,teamworkrelated,etc.)wasalsocoded.Feedbackthatstudentsreceivedfromtheframeworkwasanalyzedusingintra-classcorrelationstounderstandhowstudentswereusingtheframeworkforself-andpeerassessments.Trendsinassessmentpatternsweredeterminedandcomparedtothoseoftheunstructuredfeedback.AsTAsalsousedtheframework,TAassessmentswereanalyzedsimilarly.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario19
Theend-of-termsurveycontainedacombinationofbothqualitativeandquantitativequestionsandwascompletedbystudentsinboththeunstructuredandframeworkfeedbackgroups.Qualitativequestionswerecodedthematicallyandacomparisonofcommentsacrossthetwogroupswasperformed.Codingthemesfocusedonmotivation(ifstudentstriedtoimprove,whydidtheydoit),action(whatstudentsdidtoimprove)andimpact(whatresponsedidthefeedbackevokeinthestudentsintermsofbelieforcommitment).Quantitativequestionsmeasuredstudents’levelofagreementwithdifferentstatementsregardingthefeedbackandwerecomparedtoidentifystatisticallysignificantdifferencesbetweenthetwogroups’perceptionsoftheirfeedback.Thefocusgroupswereaudiorecordedandtranscribed.ThetranscriptionswerethencodedthematicallytoidentifysimilaritiesanddifferencesinstudentandTAresponses,andwereanalyzedtodeterminesimilaritiesanddifferences.
5. DifferencesintheQuantity,BreadthandAccuracyofStudentFeedback
Thissectionrespondstothefirstresearchquestion–doestheframeworkguidestudentstoprovideagreaterquantity,breadthoraccuracyoffeedbackthantheunstructuredprompt–bycomparingthetypesoffeedbackreceivedbystudentsinthetwogroups.5.1. QuantityofTeamMemberEffectivenessContentQuantitywasmeasuredbythenumberofdifferentteammembereffectivenesscompetenciesastudentreceivedintheirfeedbackthatmatchedthoseinourfeedbackframework.Allstudentsintheunstructuredgroupreceivedfeedbackonatleastthreecompetencies,withafewstudentsreceivingfeedbackonupto20competencies(Figure5-1)discussedinourframework.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario20
Figure5-1:Numberofframeworkcompetenciesdiscussedinfeedbackreceivedbystudentsintheunstructuredfeedbackgroupinresponsetotheprompt“Pleaseprovidefeedbacktoyourselfandyourteammembersbasedonyour/theirteam-effectivenessoverthecourseofthisproject”
Onaverage,studentsintheunstructuredgroupreceivedfeedbackonnineto10competencies.Allstudentsreceivedfeedbackonatleastoneorganizationalcompetency,97%ofstudentsreceivedfeedbackonacommunicationcompetency,butonly88%ofstudentsreceivedfeedbackonarelationalcompetency.Fourstudentsintheunstructuredgroup,includingthreefromthesameteam,chosetostructuretheirfeedbackbasedontheframework’scompetenciespresentedtostudentsduringthethreehalf-hourlecturesonteam-effectiveness.Eventhoughthesestudentschosetousetheframework’scompetency-languagetoprovidefeedback,theydidnotprovidefeedbackonall27competenciesbutprimarilyusedtheterminologyofthecompetenciestoframetheirfeedback.Bycomparison,allstudentsintheframeworkgroupreceivedfeedbackon27competencies,asstudentswererequiredtoprovidefeedbackonallcompetenciesincludedintheframework.Therefore,studentsintheframeworkgroupreceivedagreaterquantityofteammembereffectivenessfeedback.
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8 9 10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
Num
bero
fFramew
orkCo
mpe
tenciesd
iscussedin
Feed
back
Students
OrganizasonalItems RelasonalItems CommunicasonItems
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario21
5.2. BreadthofTeamMemberEffectivenessContentBreadthwasevaluatedbasedonwhetherstudentsreceivedfeedbackcoveringarangeofcompetenciesorwhetherthecommentsprovidedfeedbackonfewsimilarcompetencies.Competenciesdiscussedinthefeedbackweregroupedintothethreeaspectsofteammembereffectivenessintheframeworktoassessthebreadthofthefeedbackastudentreceived.Studentsintheunstructuredgroupgenerallyreceivedsignificantlymorefeedbackontheirorganizationalcompetenciesthanontheothertwoaspects’competencies.Studentsintheunstructuredgrouponaveragereceivedfeedbackonfiveorganizationalbehaviours(SD=1.7),tworelationalbehaviours(SD=1.6)andthreecommunicationbehaviours(SD=1.4).Thiscorrespondstoapproximately50%ofastudent’sfeedbackbeingonorganizationalcompetencies,22%onrelational,and28%oncommunicationcompetencies.Whiletheamountoforganizationalandcommunicationcontentbypercentagewasfairlystableregardlessofthenumberofcompetenciesonwhichastudentreceivedfeedback,studentswhoreceivedfeedbackonfewcompetenciesreceivedlittletonofeedbackonrelationalcompetencies(seeFigure5-2).Competenciesthatwerediscussedthemostinfeedback(over60%ofstudents)includedproduceshighqualitywork,doestheirfairshareofthework,deliverstheirworkontimeandintroducesnewideas.Competenciesthatwerediscussedtheleastinfeedback(lessthan10%ofstudents)includedsupportteamrulesandbuildthetrustofteammates.Figure5-2:Percentofrelationalcontentinanunstructuredgroup’sstudent’sfeedbackbasedonthetotalamountoffeedbackreceived
Studentsintheframeworkgroupreceivedfeedbackon10organizationalbehaviours,ninerelationalbehavioursandeightcommunicationbehaviours.Onaverage,studentsintheunstructuredgroupreceivedfeedbackonlessthanhalfasmanycompetenciesasstudentsintheframeworkgroup.
0%
5%
10%
15%
20%
25%
30%
35%
<5 5-9 10-14 15+
AveragePe
rcen
tofFeedb
ackon
Re
laso
nalCom
petencies
TotalNumberofCompetenciesonwhichFeedbackwasReceived
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario22
Atwo-waycontingencytableanalysiswasconductedtoevaluatewhetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthebreadthoftheirfeedbackinresponsetothesamequestionontheend-of-termsurvey,“Ireceivedfeedbackon:abroadrangeoftopics”(Table5-1).Thetwovariablesusedineachoftheseanalysesweretheexperimentalconditionwithtwolevels(unstructured,framework),andstudentresponsetothequestiononaLikertscale(stronglydisagree,disagree,slightlydisagree,neutral,slightlyagree,agreeandstronglyagree).Table5-1:Contingencytableanalysiscomparinggroupagreementonreceivingabroadrangeoftopicsintheirfeedback
Question Pearson’sχ2 Cramer’sV
Ireceivedfeedbackon:abroadrangeoftopics 20.4 .32**
Note:**p≤.01AscanbeseenfromTable5-1,therewasasignificantdifferenceinstudents’perceptionsofthebreadthoftheirfeedbackacrossthetwogroups.Lookingatthegraphofthestudentresponsestothisquestion,itcanbeseenthatstudentsintheunstructuredfeedbackgroupperceivedthattheydidnotreceivefeedbackonasbroadarangeoftopicsastheframeworkgroup(Figure5-3).Figure5-3:Distributionofresponsestothestatement“Ireceivedfeedbackon:abroadrangeoftopics”bygroup
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario23
5.3. OtherTypesofInformationintheFeedback Inadditiontothefeedbackdiscussedaboveonstudents’teammembereffectiveness,anumberofotheritemswerepresentedintheunstructuredfeedbackthatdidnotsupportstudentawarenessoftheirteammembereffectiveness.Twotypesoffeedbackfitthisdescription:unrelatedfeedbackandteam-levelfeedback. 5.3.1. UnrelatedFeedback Somestudentsintheunstructuredgroupreceivedfeedbackontheircourse-relatedknowledge/skills(17students)orwereprovidedwithadescriptionofwhatworktheirteammatesperceivedtheycontributedtotheproject(34students,ofwhich11receivednoteammembereffectiveness-relatedfeedback).Non-feedbackstatements,genericstatements(e.g.,“goodjob”)thatprovidednoinformationaboutthestudents’team-effectivenesscompetenciesorcourseskills,werereceivedbyninestudentsandprovidedthemwithnoinformationthattheycoulduseaseitherapointofreferenceorguidanceabouttheirperformance.Nostudentsintheframeworkgroupreceivedunrelatedfeedbackduetothenatureofthefeedbackframework. 5.3.2. Team-levelfeedbackSelf-assessmentsinparticularwerenotusedeffectivelybystudentsintheunstructuredgroup.Whenstudentswereaskedtoprovidefeedbackonthemselves,theyoftenreflectedontheirwholeteam’seffectivenessratherthanontheirownindividualeffectiveness.Twenty-fourpercent(24%)ofstudentsinthegroupdidthis,providingnospecificreflectionontheirownperformanceasateammember.Nostudentsprovidedteam-levelassessmentsaspeerfeedbackwithoutdiscussinghowthereceivercontributedtotheteam’seffectiveness–akeycomponentthatwasmissingfromtheself-assessments.Nostudentsintheframeworkgroupreceivedteam-levelfeedbackduetothenatureofthefeedbackframework.Asaresult,nostudentsintheframeworkgroupmissedoutonthisopportunitytocomparetheirself-andpeerassessments.5.4. Accuracy–AgreementbetweenSelf-andPeerAssessmentsAccuracywasmeasuredbyexaminingtheamountofunconsideredassessmentsprovidedbyeachgroup,theagreementbetweenself-andpeerassessments,aswellasthroughstudentperceptionsoftheaccuracyoftheirfeedbackasreportedintheirend-of-termsurveys.Assessmentsthatdemonstratednoconsideredreflectiononthepartoftheproviderwereunlikelytoprovideaccurateassessmentsofthereceiver’sperformance.Agreementintheunstructuredgroupwasmeasuredintermsofthenumberofcompetenciesdiscussedincommonacrossmultipleteammembers.Agreementintheframeworkgrouplookedat
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario24
correlationbetweenself-andpeerassessmentsusingintra-classcorrelationandSpearman’srankcorrelation.5.4.1. UnconsideredAssessmentsUnconsideredassessmentsfortheunstructuredgroupconsistedofnon-teamwork-relatedcomments,non-feedbackstatementssuchasgenericperformancestatements,anddescriptions(ratherthanassessments)ofastudent’scontributiontoteamdeliverables.Forty-sevenindividualassessmentswerejudgedtobeunconsidered,ofwhichthemajoritywereprovidedbyeightstudentstoeachoftheirteammembers.Thisresultedin45%ofstudentsintheunstructuredgroupreceivingatleastoneunconsideredassessment,outofatotalofthreeorfourassessmentsfromtheirwholeteam.Unconsideredassessmentsintheframeworkgroupwerethoseinwhichastudentgavethemselvesand/ortheirpeersthesamenumericassessmentacrossall27competencies.Twelveassessmentsmetthesecriteria,withsevenofthemcomingfromtwoindividuals.Onestudentprovidedthesameassessmentforallteammembersacrossallcompetencies,andasecondstudentprovidedtheirpeerswiththesameassessmentacrossallcompetenciesbutdemonstratedconsiderationintheirownassessment.Itwasunclearwhethertheotherfiveassessmentswereconsidered,astheirassessmentvaluedidnotdiffersubstantiallyfromtheconsideredassessmentsoftheirotherteammembers.Thus,sevenoutof94individualsintheframeworkgroupreceivedunconsideredassessments(approximately7.5%),substantiallylessthanthe45%intheunstructuredgroup.Theunconsideredassessments–12fromtheunstructuredgroupandsevenfromtheframeworkgroup–wereremovedbeforecompletingtheanalysesdiscussedbelow.5.4.2. AgreementintheUnstructuredFeedbackStudentsintheunstructuredgroupreceivedlimitedfeedbackthatdemonstratedagreementordisagreementontheirperformance,astheirfeedbackcontainedfewcompetenciesthatwerediscussedincommonbymorethanoneassessor.Thus,theabilitytoassessagreementislimitedtothecasesinwhicheitherthestudent’speersdiscussedatleastonecompetencyincommon,orinwhichastudentandatleastoneoftheirpeersdiscussedatleastonecompetencyincommon.Usingthe27competenciesthatwerecodedbasedonthefeedbackframework,14%ofallcompetency-codedfeedbackintheunstructuredgroupaddressedcompetenciesthatwerediscussedincommonbytwoormorepeers.Thisresultedin83studentsreceivingfeedbackthatdiscussedatleastonecompetencyincommon,withanagreementonthestudent’sperformanceonthatcompetency.Onaverage,24%ofastudent’sfeedbackdiscussedcompetenciesincommon.Themaximumpercentageoffeedbackdiscussingcompetenciesincommonwas63%.Competenciesdiscussedincommonweregroupedaccordingtowhetherthepeersagreedontheperformancelevelofthestudent,whetherthepeersdisagreedontheperformancelevel,orwhetheritwas
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario25
unclearwhethertherewasagreementordisagreementbetweenthepeerassessments.Thedistributionsoftheseperformance-levelagreementsonaperstudentbasiscanbeseenbelowinFigure5-4.Acrossthe83studentswhoreceivedfeedbackincommon,60studentsreceivedfeedbackthat,onaverage,agreedontheperformancelevelofthestudent,4receivedfeedbackthatdisagreedontheperformancelevelofthestudent,and6studentsreceivedfeedbackthatagreedontheperformancelevelofatleastonecompetencyanddisagreedonatleastoneother.Fortheremaining13students’competenciesdiscussedincommon,itwasunclearwhethertherewasagreementordisagreementbetweenassessments.Figure5-4:Distributionofperformanceagreementinstudentfeedbackforcompetenciesthatwerediscussedincommonbypeerassessors
5.4.3. AgreementintheFrameworkFeedbackToassesstheagreementofthefeedbackastudentreceivedintheframeworkgroup,theinter-raterreliabilitybetweenpeerassessorsandthecorrelationbetweenstudents’self-andpeerassessmentsweredetermined.Inter-raterreliabilitywasdeterminedusingtheintra-classcorrelationcoefficients(ICCs)forpeerassessments,usingatwo-wayrandomeffectsmodelforconsistency.Ninety-oneofthe95studentshadatleasttwopeerassessorsandwereconsideredinthisanalysis.PeerICCswerethengroupedaccordingtotheirlevelofagreementbetweenthepeerassessors(Table5-2).Thirty-onepercentofthefeedbackshowednoagreementbetweenthepeerassessors,and8.8%showedsubstantialagreementbetweenthepeerassessors.Themajorityofassessmentsshowedlimitedagreement,rangingfromslighttomoderateagreement.Therefore,itislikelythatthefeedbackshoweddifferinglevelsofagreementalongthedifferentaspectsoftheframework.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario26
Asaresultofthelimitedagreementamongpeers,thelevelofagreementbetweenself-andpeerassessmentswasdetermined.Lookingspecificallyintoourhypothesisthatthelimitedagreementmaybeaspect-specific,accuracywasdeterminedbycomparingtheaverageself-assessmentstothepeerassessmentsacrossthethreeaspectsoftheframework.UsingaSpearman’srankcorrelation(Table5-3),weseeasignificantcorrelationbetweenthestudents’self-andpeerassessmentsintheorganizationalaspect.However,thereisnosignificantcorrelationbetweenstudents’self-andpeerassessmentsintherelationalandcommunicationaspects.Thisstrongcorrelationinself-andpeerassessmentsalongonlytheorganizationalaspectmayexplainthevarianceinagreementseeninthepeeragreementdiscussedabove.Table5-2:DistributionofagreementbetweenpeerassessmentsofastudentasmeasuredusingICCsinatwo-wayrandomeffectsmodelforconsistency
LevelofAgreement NumberofStudents ICCvaluerange
Noagreement 28 <0
Slightagreement 18 0-0.2
Fairagreement 20 0.21-0.4
Moderateagreement 17 0.41-0.6
Substantialagreement 8 0.61-0.8
Perfectagreement 0 0.81-1
Table5-3:Spearman’srankcorrelationbetweenstudents’self-andpeerassessmentsforeachaspectoftheframework
Aspect Correlation
Organizational .45**
Relational .19
Communication .16
Note:Students’self-assessments(n=numberofcompetencies)andpeerassessments(n=numberofcompetenciesxnumberofpeers)wereaveragedseparatelytodetermineeachstudent’sself-assessedandpeer-assessedlevelofcompetencyfortheaspect.Theseaveragesforeachaspectwerethencorrelatedoverallthestudents.**-p≤.01.5.4.4. PerceivedAccuracyUsingatwo-waycontingencytableanalysis,therewasnosignificantdifferenceacrossthetwogroupsintheiragreementwiththestatement“ThefeedbackIreceiveddescribedmeexactlyhowIperceivedmyself”fromtheirend-of-termsurveys.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario27
Thissimilarityintermsofhowstudentsperceivedtheaccuracyofthefeedbackextendedintostudentsresponsestotheopen-endedquestionsontheend-of-termsurvey.Fourstudentsfromtheunstructuredfeedbackgroupandfivestudentsfromtheframeworkfeedbackgroupcommentedthattheyfelttheirfeedbackwasfakeandthatitwasnotaccurate.Asthisaccountedforthesamepercentageofstudentsineithergroup(5%),neithergrouponaverageperceivedthefeedbacktheyreceivedtobemoreorlessfakethantheothergroup.Additionally,anequalnumberofstudents(8)fromeachgroupchallengedtheirfeedbackfromtheirpeers,statingthattheyfeltthatitdidnotreflecthowtheybehavedintheirteam,orthattheirassessmentswerebasedononeincidentandnottheiraveragebehaviour.Onedifferencebetweenthegroups,however,wasintermsoftheperceivedconsistencyofthefeedback.Sevenstudentsintheframeworkgroupclaimedthattheirfeedbackpresentedtwodifferingperspectivesontheirperformanceandthatitthusbecamehardtoknowhowtheywereactuallyperformingintheirteam.Nosuchcommentswereprovidedbystudentsintheunstructuredfeedbackgroup.5.5. DiscussionInadditiontothedifferencesinfeedbackaroundquantity,breadthandaccuracyofteammembereffectivenesscompetencies,threeothernotabletrendsemergedintheanalysis.First,students,primarilythoseintheunstructuredfeedbackgroup,didnotalwaysreceiveenoughfeedbacktoprovidethemwithaclearassessmentoftheireffectivenessasateammember,limitingtheusefulnessofthefeedbacktheyreceived.Second,studentsinbothgroupsstronglyprivilegedorganizationalcompetenciesintheirfeedback.Third,studentsintheunstructuredfeedbackgroupdidnotalwaysusetheself-assessmentpromptcorrectly.5.5.1. LimitedQuantityofUnstructuredFeedbackWithintheunstructuredfeedbackgroup,students’limitedabilitytofindagreementamongtheirpeerassessmentspreventedthemfromgettingaclearpictureofhowtheywereperformingasateammember.Fewstudentshadcompetenciesdiscussedincommon,whichwaslimitedfurtherbytheamountofunrelatedfeedbackstudentsgeneratedduetotheunstructurednatureoftheprompt.Thislimitedquantityaffectedtheabilityofthefeedbacktoprovidethereceiverwithaconsistentmessageabouttheirperformance.Whiletherewasafairamountofagreementwithinthepeersaboutstudents’performancewhentheydiscussedcompetenciesincommon(66%ofallfeedbackdiscussedincommon),therewasnotsufficienttotalfeedbackincommon(14%oftotalfeedback)toprovidestudentswithasenseofconsistencyinpeerfeedback.Thisresultedinonly9.24%oftheunstructuredfeedbackdiscussingcompetenciesincommonwithagreementaboutperformance.Essentially,anunstructuredprompt(andhowstudentschoosetointerpretit)willmostlikelyproducepeerfeedbackthatraisesfewaspectsincommon.Studentsmaythusperceivethisfeedbackasrandombitsofinformationwithoutaconsistentmessage,makingiteasiertodismissasnotaccuratelydescribingthem.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario28
5.5.2. StudentPrivilegingofOrganizationalCompetenciesStudentfeedbackinboththeunstructuredandframeworkgroupsdemonstratedastrongidentificationwithandprivilegingoforganizationalcompetencies.Thisisnotsurprising,asanecdotallystudentsinengineeringhaveatendencytoprivilegework-contribution-relatedcompetencies.Thisprivilegingwasevidentinthedistributionofcompetenciesdiscussedintheunstructuredgroup’sfeedbackthroughthestrongprevalenceoforganizationalitemsintheunstructuredfeedback(approx.50%ofallfeedbackwasorganizational).Additionally,studentswhoreceivedlittlefeedbackreceivedalmostalloftheirfeedbackonorganizationalcompetencies.Asaresult,unstructuredfeedbackmayperpetuatestudentprivilegingoforganizationalcompetenciesandlimittheirabilitytogrowaseffectiveteammembersbynotbroadeningstudentunderstandingofteammembereffectiveness.Whenlookingatwhichitemswerediscussedintheunstructuredfeedback,wefindthesameorganizationalpredominance,withtheexceptionoftwocompetencies.Thetopfivemostdiscussedcompetenciesintheunstructuredfeedbackwerepredominantlyorganizational,withoneexception.Giventheprivilegingbyengineeringstudentsofwork-contribution-relatedcompetencies,thehighlevelofresponserelatingtotheintroducenewideascompetency(thirdmostdiscussed)islikelyaproductoftheteam’sprojectwork.Studentswhocameupwiththeideathatwasselectedtoaddressthedesignprojectwerelaudedextensivelyfordoingsointheirfeedback.Oneorganizationalcompetencythatwasveryminimallydiscussedwassupportteamrules.Thisisnotalarmingasinthecoursecontext,teamruleswerediscussedasnormsofpracticebutwerenotcodified.Asaresult,studentsinthecoursewouldnotbepredisposedtoreflectingonastudent’srolewithrespecttoteamrules.Thisprivilegingcanalsobeseenintheframeworkfeedbackinthecorrelationbetweenstudentself-andpeerassessments.Theonlysignificantlycorrelatedassessmentswerethosealongtheorganizationalaspect.Whilethisdoesnotaffectthefeedbackastudentreceives,itdoesimplythatstudentsmayperceivethefeedbackprovidedalongtheorganizationalaspecttobemoreaccuratethanthefeedbackalongtheothertwoaspectsastheyagreemorecloselywiththeirself-assessments.5.5.3. LimitedUtilityofSelf-assessmentsintheUnstructuredGroupStudentself-assessmentsintheunstructuredgroupwerenotusedeffectively,asmanystudentsuseditasaspacetoreflectontheirteam’seffectivenessratherthantheireffectivenessasateammember.Theself-assessmentwasmeanttoactasapointofcomparisonforthestudentwhenreceivingfeedbacktoseehowdifferentlythey,ascomparedtotheirpeerteammembers,perceivedtheireffectiveness.Withoutanindividual-levelself-assessment,thiscomparisonislesseffectiveandthefeedbackthereforelessuseful.Additionally,itdoesnotprovidethereceiverwithanyguidanceonhowtoimprovetheirbehaviour,aswhatisneededtoimprovetheteamasawholemightbedifferentfromwhattheyneedtoimproveasateammember.Thus,nothavinganyself-assessmentofteammembereffectivenesslimitedtheusefulnessofthefeedbackincreatingagreaterawarenessofhowthestudentswereperformingforthis23%oftheunstructuredgroup.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario29
5.6. ConclusionandResponsetoResearchQuestionComparingholisticallythefeedbackreceivedbetweenthetwogroups,studentsintheunstructuredgroupreceivedfeedbackonsubstantiallyfewerteamworkcompetenciesthandidthoseintheframeworkgroup,andreceivedtheirfeedbackprimarilyonorganizationalcompetencies.Studentsintheunstructuredfeedbackgroupreceivedsignificantlylessfeedbackonrelationalcompetenciesthandidthoseintheframeworkgroup–anareaknowntobeunderprivilegedinengineeringstudents.Studentsinthisgroupalsoreceivedasubstantialamountoftheirfeedbackonnon-teamwork-relatedissues/skills.Intermsoftheaccuracyofthefeedbackreceivedbystudentsineithergroup,theframeworkgroupreceivedlessfeedbackthatappearedtohavenoconsideredreflectiononthepartoftheproviderthandidtheunstructuredgroup.Consistencyinthefeedbackprovidedbystudentswasgreaterintheframeworkgroup,inpartduetothesetcompetenciesonwhichallstudentshadtoprovidefeedback,butitdidnotappeartolimittheabilitytoreceiveadiversityoffeedback,asseeninstudents’commentsonreceivingdifferentassessmentsfromtheirpeers.Therefore,inresponsetotheresearchquestion,theframeworkprovidesgreaterbreadth,quantityandaccuracyoffeedbackthantheunstructuredfeedbackprompt.
6. DifferencesinStudentPerceptionsoftheUsefulnessoftheirFeedback
Thissectionrespondstothesecondresearchquestion–dostudentsperceivethefeedbackfromtheframeworktobemoreusefulthanunstructuredfeedback?–bycomparingstudents’receivedfeedbackandtheircommentsontheend-of-termsurveys.Asstudentsinbothgroupsansweredthesamequestionsontheend-of-termsurvey,comparativeanalysesarepossible.Threenotabledifferencessurfacedintheanalysis:i)identificationofstrengthsandweaknessesinthefeedback,ii)perceivedamountofactionablefeedback,andiii)improvementbasedonfeedback.6.1. IdentificationofaStudent’sStrengthsandWeaknessesFeedbackwithclearstrengthsandweaknessesismoreusefultostudents,asitidentifieswhattheyaregoodatandwhattheyneedtoimprove.Thisallowsstudentstofocustheireffortsonimprovingcertainkeyareas.Feedbackfrombothgroupswascomparedtodeterminewhichmorereadilyidentifiedstrengthsandweaknessesinfeedback.6.1.1. UnstructuredFeedbackThefeedbackfromtheunstructuredgroupwassignificantlylessgranularintermsofperformancethantheframeworkfeedback.Studentswereeitherlaudedonwhattheydidwell,critiquedonwhattheydidpoorly,orinformedofwhattheydidordidnotdo.Asaresulttheunstructuredfeedback,whichhadalreadybeen
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario30
codedtoidentifyteammembereffectivenesscompetencies,wascodedagainaccordingtothesethreetypesoffeedback.Theunstructuredfeedbackreceivedbystudentsdiscussedthestrengthsofthestudentsmorecommonlythanitdidweaknesses.Twenty-fivestudents(24%)intheunstructuredgroupdidnotreceiveanyfeedbackonweaknessesorcritiqueoftheirbehaviour,andsixstudents(6%)receivednopraiseoridentifiedstrengthsintheirevaluations.Studentswithinbothofthesegroupsalsoreceivedneutralfeedback,whichsimplystatedwhattheycontributedtotheproject.Todeterminetheprevalenceofstrengthsinthefeedbackreceivedbystudents,thefeedbackwasbinnedintofivecategoriesaccordingtothepercentageofthefeedbackthatdiscussedstrengths.AscanbeseeninFigure6-1below,75%ofthestudentsintheunstructuredgroupreceivedfeedbackthatprimarilydiscussedtheirstrengthsasopposedtotheirweaknesses(opportunitiesforimprovement).Figure6-1:Percentofpeer-feedbackreceivedbystudentsintheunstructuredfeedbackgroupthatdiscussedtheirstrengths.
6.1.2. FrameworkFeedbackAllstudentsintheframeworkgrouphadasimilarnumberofidentifiedstrengthsandweaknesses.Thiswasaresultofthesortingalgorithmrankingstudentframeworkfeedback,suchthatstudents’strengthsandweaknessesweredeterminedandhighlightedforthem.Allstudentsintheframeworkgrouphadthreetofivecompetenciesidentifiedintheirfeedbackasstrengths,andthreetofivecompetenciesidentifiedas
0%
5%
10%
15%
20%
25%
30%
35%
40%
0-24% 25%-49% 50%-74% 75%-99% 100%Percen
tofStude
ntsintheUnstructuredGrou
p
PercentofFeedbackthatDiscussedStrengths
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario31
areasforimprovement.Thiscorrespondedtostudentshavingatleast11%oftheirfeedbackidentifyingstrengthsandatleastanother11%identifyingtheirareasforimprovement.Toensurethatthestrengthsandweaknesseswerenotjustarbitrarilyidentifiedbutactuallyflaggedcompetencieswithwhichthereceiverhadadistinctdifferenceintheirlevelofperformance,aRelated-SamplesWilcoxonsignedranktestwasconductedonthedifferencebetweentheaverageratingsofthecompetenciesidentifiedasstrengthsandtheaverageratingsofthecompetenciesidentifiedasweaknesses.Therewasasignificantdifference(p=.01)intheaveragesofthestrengthsandweaknessesidentifiedforthestudents,meaningthatonaveragestudentswerereceivingfeedbackthatdemonstrateddistinctdifferencesintheirperformancelevelacrossthecompetencies.6.1.3. StudentPerceptionoftheStrengthsandWeaknessesIdentifiedTherewasasignificantdifferenceintheperceptionoftheidentificationofstrengthsandweaknessesacrosstheunstructuredandfeedbackgroups.Whiletherewasnosignificantdifferencebetweenthegroupsastotheirperceptionoftheirstrengths,therewasasignificantdifferenceacrossthetwogroupsintermsoftheidentificationoftheirweaknesses.Atwo-waycontingencytableanalysiswasconductedtoevaluatewhetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthecontentoftheirfeedbackinresponsetothesamequestionontheirend-of-termsurvey(Table6-1).Thetwovariablesusedineachoftheseanalysesweretheexperimentalconditionwithtwolevels(unstructured,framework),andstudentresponsetothequestiononaLikertscale(stronglydisagree,disagree,slightlydisagree,neutral,slightlyagree,agreeandstronglyagree).Table6-1:Contingencytableanalysiscomparinggroupagreementonreceivingidentifiedweaknessesintheirfeedback
Question Pearson’sχ2 Cramer’sV
ThefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare
12.4 .25*
Note:*p≤.05AscanbeseenfromtheresponsedistributionsinFigure6-2,theframeworkgroupdemonstratedmoreoverallagreementwiththestatementthantheunstructuredgroup,indicatingthatstudentsintheunstructuredgroupnoticedthedistinctlackofweaknessesprovidedintheirfeedback.Thestudentsintheunstructuredgroupremarkedonthislackofcritiqueoftheirbehaviourintheopen-endedquestionsontheirend-of-termsurveys.Thestrongesttheme(greatestnumberofstudentscommentingonit)amongresponsesfromstudentswhoreceivedunstructuredfeedbackwasthattheydidnotreceivefeedbackonareasinwhichtheycouldimprove.Thirty-onestudents(30%)intheunstructuredgroupcommentedthattheywantedmorecritiqueintheirfeedback,comparedwithsevenstudents(7%)fromtheframeworkgroup.Thestrongestthemeamongthestudentswhoreceivedframeworkfeedbackwasthattheywantedmorejustification,commentsorexamplesfortheassessmentstheyreceived(23%).
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario32
Therefore,studentsintheframeworkgroupreceivedandperceivedthattheyreceivedmoreidentifiableweaknessesonwhichtoimprove,whilebothgroupsperceivedasimilarnumberofidentifiedstrengths.Figure6-2:Distributionofresponsestothestatement“thefeedbackIreceivedwasstructuredinsuchawaythatIfoundoutwhatmyweaknessesare”bygroup
6.2. ImprovementbasedonFeedbackImprovementbasedonfeedbackwasdifferentacrossthetwogroups.Motivationtoimproveperformanceappearedstrongerfortheunstructuredfeedbackgroup,duetothetoneoftheirfeedback.However,self-reportedimprovementalongspecificweaknessesidentifiedinthefeedbackwashigherfortheframeworkfeedbackgroup.Thetoneofthefeedbackappearedtohavealargeinfluenceonstudents’motivationtoimproveanddevelopasenseofteamcohesion.Atwo-waycontingencytableanalysiswasconductedtoevaluate
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario33
whetherthereweredifferencesbetweentheunstructuredandtheframeworkgroup’sperceptionsofthetoneoftheirfeedbackinresponsetothesamequestionontheirend-of-termsurvey(Table6-2).Thetwovariablesusedintheanalysisweretheexperimentalcondition(unstructured,framework)andstudentresponsetothequestiononathree-pointscale(Positive,Neutral,Negative).Table6-2:Contingencytableanalysiscomparingthetoneofthefeedbackreceivedbythetwogroups
Question Pearson’sχ2 Cramer’sV
Feedbacktomewasphrasedina{positive,neutral,negative}tone 8.5 .21**
Note:**p≤.01Therewasasignificantdifferenceintheperceptionofthetoneofthefeedback.Studentsintheunstructuredgroupfeltthattheirfeedbackwasphrasedmorepositively,whereasstudentsintheframeworkgroupweremostlysplitastowhethertheirfeedbackwaspresentedinaneutralorapositivetone.Twenty-threestudentsintheunstructuredgroup,comparedto12studentsintheframeworkgroup,commentedthatthefeedbacktheyreceivedincreasedtheircommitmenttotheirteamormadethemfeelthattheirteammembersvaluedtheircontributions.Commentsfromstudentsintheframeworkgroupseemedtobelookingforthispersonalconnectionintheirfeedback.Studentsintheframeworkgroupcommentedthattheywantedmorejustification,commentsorexamplesfortheassessmentstheyreceived(23students)(i.e.,sometextualfeedbackfromtheirpeers).Onedifference,however,betweenthetwogroupswasinthebehaviourstheyindicatedthattheyworkedonimprovingduringtheperiodbetweenreceivingtheirinitialfeedbackandcompletingtheend-of-termsurvey.Studentsfromtheframeworkgroupcitedmoreweaknessesthatwereidentifiedintheirfeedbackthandidstudentsintheunstructuredfeedbackgroup(Figure6-3).Bothgroupscitedworkingonotherimprovementsthatwerenotindicatedintheirfeedbackbutthattheythoughtwereequallyimportant.6.3. DiscussionWhilethetoneofthetextualfeedbackprovidedbystudentsintheunstructuredpromptincreasedstudentcommitmenttoimproving,thelackofidentifiedweaknesseslimitedtheusefulnessofthefeedbackasameanstopromoteimprovementandlearning.Sincetheunstructuredfeedbackgroupreceivedfeedbackthatwasphrasedinthewordsoftheirteammembersandmostlyfocusedonstrengthsinsteadofweaknesses,itmakessensethatstudentswouldperceivethisfeedbackasmorepositivethantheframeworkfeedbackreceivedbythatgroup.However,inafewcaseswherestudentswereprovidedconstructivecriticism,thecontextforthefeedbackwasprovidedtoexplaintheassessment.Thisaspectoftheunstructuredfeedbackstrengthenedtheutilityofthefeedbackforthesefewstudents.Thislackofidentifiedweaknessesmayalsoexplainwhythenumberofweaknessesonwhichstudentsidentifiedimprovingwasthesameacrossbothgroups,butthefeedback-identifiedweaknessesonwhich
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario34
studentsimprovedwaslessfortheunstructuredgroup.Havingfeweridentifiedweaknessescouldinhibitstudents’abilitytorecalland/orconsciouslyimprove.Additionally,sincetheamountofcompetency-specificfeedbackwasminimal(14%)fortheunstructuredfeedbackgroup,itislikelythatevenwhenaweaknesswasidentified,itwaswrittenoffasananomalyifitwasnotalsodiscussedbyotherpeers.Figure6-3: Numberofweaknessesidentifiedinstudents’end-of-termsurveysthattheycommentedasimprovingonbasedontheirfeedback,bygroup
6.4. ConclusionandResponsetoResearchQuestionInresponsetotheresearchquestion–dostudentsperceivethepeerfeedbackfromtheframeworktobemoreusefulthanunstructuredfeedback?–thefindingsaboveindicatethatstudentsdofindtheframeworkfeedbacktobemoreuseful.Fromtheseanalyses,itwasclearthattheframeworkguidedstudentstoidentifystrengthsandweaknessesmoreclearlythantheunstructuredfeedbackprompt,asstudentshadaprescribedsetofcompetenciestoassessintheframework.Studentsintheunstructuredgrouphadsignificantlylessagreementwiththestatementthattheirfeedbackidentifiedtheirweaknesses,andalsovocalisedthisissueintheiropen-endedresponsesintheend-of-termsurvey.Asaresult,studentsreceivingfeedbackfromtheframeworkreceiveagreateramountofcritiqueandthereforewereabletoidentifytheirweaknessessothattheycouldimproveonthem.Thiswasconfirmedwhenstudentsintheframeworkgroupcitedthattheyworkedonimproving
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario35
aspectsoftheirteam-effectivenessthatwereidentifiedasweaknessesintheirfeedbackmorefrequentlythandidstudentsintheunstructuredgroup.Therefore,becausetheframeworkfeedbackindicatesthepotentialtocreatemoreofthedesiredbehaviouralchangearticulatedbyteammembersintheirpeerfeedback,itmakestheframeworkfeedbackmoreuseful.However,basedontheimpactoftheunstructuredfeedback’stoneonteamcohesion,ahybridoftheframeworkandunstructuredfeedbackshouldbeinvestigatedinthefuturetoallowfortheinclusionoftextualcommentsintheframework.
7. DifferencebetweenStudentandTAAssessmentsThissectionrespondstothethirdresearchquestion–canstudentsprovidefeedbacksimilartothatofatrainedobserver(courseteachingassistant)whenusingtheframework?–bycomparingtheassessmentsprovidedbytheteachingassistants(usingtheframework)withthestudentsassessmentsinboththeunstructuredandframeworkgroups.Teachingassistantswereusedastrainedobserversbecauseofthequantityandqualityoftheirinteractionwiththestudentsinweeklytutorialsessions.Inendeavouringtoassesstheconcurrentvalidityoftheframework,itwasnotenoughforstudentstobeabletoprovidesimilarratingstotheirpeersusingtheframework.Inamoreabsolutesense,theyneededtoprovideratingsthatwereinfactarepresentationoftheassessedstudent’steammembereffectiveness.Aninvestigationofteachingassistants’abilitytoprovidefeedbackusingtheframeworkwasprovidedbySheridanetal.(2014).ThekeyfindingsaroundhowTAsusedtheframeworkwere:
1. TAsprovidedassessmentsthatwereonaveragelowerthanthoseofthestudents.2. TheaveragenumberofcompetenciesonwhichaTAprovidedfeedbackwas16(minimum2,
maximum27),duetotheiruseofthe‘donotknow’option.3. TAsprovidedasimilaramountoffeedbackacrossthethreeaspects.
Twonotabledifferencessurfacedinthiscomparisonofteachingassistanttostudentassessments:i)similarityofstrengthsandweaknessesidentification,andii)correlationbetweenteachingassistant,andself-andpeerassessments.7.1. SimilarityinIdentificationofaStudent’sStrengthsandWeaknessesThesamealgorithmusedonstudents’frameworkassessmentswasappliedtotheTAs’assessmentstosortoutthethreetofivetoprankedstrengthsandthreetofivebottomrankedweaknessesforeachstudent.TheseTA-identifiedstrengthsandweaknesseswerethencomparedtothestrengthsandweaknessesidentifiedbystudentsintheframeworkfeedbackgroup,andtothecompetency-codedstrengthsandweaknessesdiscussedinfeedbackfortheunstructuredfeedbackgroup.Whenlookingatthepercentageofstudentsineachgroup,theagreementbetweenstudentandTAfeedbackintermsofidentifyingatleastonestrengthorweaknessincommonwashigherfortheframework
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario36
group(86%)thanfortheunstructuredgroup(69%)(Table7-1).Additionally,whileTAsandstudentsrarelyagreedonbothastrengthandaweaknessforanygivenstudentinbothgroups,thisoccurredmorefrequentlyintheframeworkgroup(18%)thanintheunstructuredgroup(7%).Table7-1:FrequencyofagreementbetweenstudentandTAidentifiedthesamestrengthsandweaknessesintheunstructuredandframeworkfeedbackgroups
PercentofStudentsinFeedbackGroup
TypeofAgreement Unstructured Framework
1strengthOR1weakness 69% 86%
1strengthAND1weakness 7% 18%
7.2. CorrelationbetweenTAandStudentFrameworkAssessmentsTodeterminewhetherthestudentsprovidedsimilarfeedbacktotheTAswhengiventheframework,thecorrelationsbetweenstudentself-andpeerassessmentsandTAassessmentswerecomputed.Asthenumberofcompetencieswassubstantial,andgiventhatanumberofthewithin-aspectcompetenciesarehighlycorrelated,correlationbetweenstudentandTAassessmentstodeterminethesimilarityoftheirassessmentswasdoneattheaspect-leveloftheframework.Correlationswerecomputedacrossstudents’andTAs’averageorganizational,relationalandcommunicationassessments.TheseselfaveragesandpeeraveragesforeachaspectwerethencomparedtotheTAs’averageassessmentofthestudent’scompetencybycorrelatingoverallthestudents.ASpearman’srankcorrelationwasconductedbetweentheaspect-levelTAandpeerassessments(Table7-2)andTAandself-assessments(Table7-3)forallstudentsintheframeworkgroup.Whenusingtheframework,thepeerassessmentsandTAassessmentsofindividualstudentsweresignificantlycorrelatedacrossallaspects.However,theself-assessmentswereonlysignificantlycorrelatedwiththeTAassessmentsfortheorganizationalaspect–theaspectthatstudentsinboththeframeworkandunstructuredgroupweremorecomfortableassessing.ThisisthesametrendthatwesawinTable5-3,wherestudentself-andpeerassessmentswereonlysignificantlycorrelatedalongtheorganizationalaspectaswell.Table7-2:Spearman’srankcorrelationbetweenstudents’peerassessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework
Aspect Spearman’sRankCorrelation
Organizational .35**
Relational .27**
Communication .24*
Note:Students’peerassessments(n=numberofcompetenciesxnumberofpeers)wereaveragedseparatelytodetermineeachstudent’speer-assessedlevelofcompetencyfortheaspect.*p≤.05,**p≤.01
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario37
Table7-3:Spearman’srankcorrelationbetweenstudents’self-assessmentsandTAs’assessmentsofeachstudentalongthethreeaspectsoftheframework
Aspect Spearman’sRankCorrelation
Organizational .26**
Relational .00
Communication .16
Note:Students’self-assessments(n=numberofcompetencies)wereaveragedseparatelytodetermineeachstudent’sself-assessedlevelofcompetencyfortheaspect.**p≤.017.3. DiscussionInadditiontothedifferencesinteachingassistantandstudentassessments,threeothernotabletrendsemergedintheanalysis.First,teachingassistantswerehesitantabouttheassessmentcontext.Second,thequantityofunstructuredfeedbacklimitstheapplicabilityofthefindingsaroundagreement.Third,studentself-assessmentsbyaspectdemonstrateadifferentprivilegingthandoespeerassessments.7.3.1. TeachingAssistantHesitancyaroundtheAssessmentContextTheobjectiveofthispartofthestudywasinitiallytouseTAsasthegoldstandardagainstwhichtocomparestudents’feedbacktodetermineaccuracy.WhiletheresultsoftheassessmentsprovidedcorroborationthatstudentscanprovidepeerassessmentsthataresimilartothoseoftheirTAs,theteachingassistantswerehesitanttobeconsideredagoldstandard.TAsarethefront-linesupportforteamissuesandguidance,whichrequiresthemtobeattunedtohowtheteamsarefunctioning.However,theycommentedthattheyweremoretunedintotheteamsthantotheindividualswithintheteamsthemselves.Asaresult,theyweremorecomfortableandmoreconfidentassessingtheteamsthantheindividualstudents.PartoftheTAs’hesitationmayberelatedtothesizeofthefeedbackframework(27competencies),butbasedonTAcommentsfromthefocusgroupswehypothesizethatmostofthediscomfortisduetothecognitiveloadontheteachingassistantsduringtutorials.SincetheTAsneedtoprovidedifferentscaffoldingtoeachoftheteamstosupportthedevelopmentoftheirdesignandcommunicationskills,theirprimarymodeofobservationisduringtheseinteractions.Intheseconversations,theTAsneedtoassessthecurrentstateoftheteam’sdesignworkandwhatscaffolding(intheformofadvice,resourcesorcoaching)isnecessarytomovetheteam’sdesignworkforward.Asaresult,thissubstantiallylimitstheirabilitysimplytoobservethestudentsastheyworkintheirteamsandbiasestheirassessmentsagainstthosestudentswithwhomtheydonotfrequentlyinteractintheteams.However,giventhesignificantcorrelationsbetweenpeerandTAassessmentsattheaspectlevel,reducingtheTAfeedbackframeworktosubstantiallyfewercompetenciesortoanaspect-levelframeworkmaystrengthenthesecorrelations,asTAswouldthenbeabletoprovidemoreconfidentandhopefullymoreaccurateassessmentsoftheirstudents.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario38
7.3.2. LimitationsontheAgreementforUnstructuredAssessmentsAsdiscussedinTable7-1,strengthsandweaknessesidentifiedbyTAsagreedstronglywithstudentsassessmentsfromtheframeworkgroup.However,itmustbenotedthatthisfindingislimitedbythequantityofunstructuredteammembereffectivenessfeedback.Asthequantityofunstructuredfeedbackwassubstantiallylessthanthatoftheframeworkfeedback,therewasalowerlikelihoodofastudenthavingchosentodiscussastrengthorweaknessidentifiedbytheTA.Sinceallstudentsintheframeworkgroupcommentedonallframeworkcompetencies,therewasagreaterlikelihoodofagreementbetweenthisgroupandtheTAs.Therefore,whiletheagreementoftheTAs’assessmentswiththeframeworkgroupisgreater,therearenotsufficientdatafromtheunstructuredgrouptoclaimanydifferenceinagreementbetweenthetwogroups.7.3.3. Aspect-levelDiscrepanciesinAssessmentAsdiscussedinTable7-2,TA-peerassessmentsweresignificantlycorrelatedalongallaspectsoftheframework,whileself-assessmentswereonlycorrelatedwithTAassessmentsalongtheorganizationalaspect.ThisfindingsupportsthediscussioninSection0thatstudentsinthisclassprivilegeorganizationalcompetenciesandareabletoself-assessthemmoreaccuratelythanthoseintheotheraspects.TA-self-assessmentcorrelationfortherelationalaspectwas.00,whichdemonstratednocorrelationbetweenTAandself-assessmentsatall.Thismaybeaproductoftheteachingassistants’lackofcomfortwiththerelationalcompetenciesaswell.Theteachingassistantscommentedintheirfocusgroupthattherelationalaspectwasthehardesttoassess.However,therewerenocleartrendsacrossthethreeaspectswithrespecttothenumberofcompetenciesonwhichtheyprovidedfeedback(duetothe‘donotknow’option).Thismayalsobeaproductofthepreviouslydiscussedprivileging,asstudentsinengineeringarenottraditionallycomfortablewithrelationalcompetencies.Thiswasseenalsointheunstructuredfeedback,inwhichlessthan20%ofthecompetenciesdiscussedwererelational.Therefore,whiletheframeworkdoesprovideasubstantialbasisforstudentstogivefeedbacktotheirteammembersthatissimilartothatoftheirTAs,studentshaddifficultyself-assessingtheirabilitiesinregardstorelationalandcommunicationcompetencies.Thisinabilitytoself-assessaccuratelyspeaksfurthertothevalueofusingpeerfeedbackasateachingtooltohelpstudentsdeveloptheirself-awarenessofandcompetencyinteammembereffectiveness.7.4. ConclusionandResponsetoResearchQuestionComparingthefeedbackreceivedbetweenthetwogroupsandtheteachingassistants,wecanclaimthatstudentpeerassessorscanbeusedforintra-teamevaluationsmoreeffectivelythanteachingassistants.Agreementbetweenteachingassistantandstudentfeedbackintermsoftheusefulnessofthefeedback(identifiedstrengthsandweaknesses)isgreaterwhenstudentsusetheframeworkthanwhentheyuseanunstructuredprompt.Additionally,studentpeerassessmentsaresignificantlycorrelatedwithteachingassistantassessmentsacrossallthreeaspectsoftheframework.Studentpeerscanprovidefeedbacksimilar
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario39
tothatoftrainedTAobservers.Thesamediscrepancybetweenself-andpeerassessmentswasseenwithself-andTAassessments,furthercorroboratingthatpeerassessmentsprovidesimilarinformationwithwhichstudentscanbroadentheirawarenessoftheirteamworkskills.Therefore,inresponsetotheresearchquestion,theframeworkguidesstudents(aspeers)toprovidefeedbacksimilartothatoftrainedobservers(teachingassistants).
8. StudentandTAFeedbackontheUsabilityoftheFrameworkThissectionrespondstothefourthresearchquestion–istheframeworkaccessible(e.g.,jargon,descriptions,levelsofcompetency)tostudentsandtrainedobservers?–bydiscussingfourprominentthemesaroundtheusabilityoftheframework,asdiscussedbystudentsandTAsintheirend-of-termfocusgroups.Thesefourthemesare:i)accessibilityofthecompetencies,ii)accessibilityoftheratingscale,iii)presentationofthefeedback,andiv)missingcontent.8.1. HolisticAssessmentofFrameworkAccessibilityThelanguageofthefeedbackframeworkreceivednocomplaintsorcommentsfromeitherthestudentsortheteachingassistants.Studentswereparticularlyhappythattherewasno“engineeringjargon”intheframework,whichtheyfeltenhanceditsaccessibility.Studentsinthefocusgroupthoughtthattheframeworkwas“thorough.”Theydescribedtheframeworkas“pertinent”and“relevant,”andsaidthatit“touchedonmoredetailedelementsofteamsthan[surveys]usuallytouchon.”Studentsalsocommentedthattheframeworkwasaneffectivelearningtoolinandofitself,andremarkedthatcompletingtheframeworkatthemid-pointoftheirteamprojectsallowedthemtoreflectonwheretheywereorwerenotdoingwellandremindedthemofwhatcompetencieseffectiveteamworkrequired.BothstudentsandTAslikedthattheassessmentsweregroupedintothethreeaspectsofteam-effectiveness.Bothgroupscommentedthattheyfoundthishelpedthemgetintoacertainheadspaceandevaluateaspecificaspectoftheexperience.Thissimplifiedthinkingandhelpedtoclarifyanysourcesofconfusioninthequestions.Onestudentcommentedthattheywouldhavepreferredthecompetenciestohavebeenpresentedasquestionsratherthanstatements,asitwould“allow[thestudent]tofocusmoreonwhat[theyare]grading[theirteammember]on.”Theydidnotelaborateonwhytheythoughtthiswouldbethecase.Studentscommentedthatinsteadofratingeachteammemberalongthecompetenciessequentially,theywantedtoprovidefeedbackforeachteammemberalongthecompetenciessimultaneously–thestudentswantedtocompleterelativerankingsoftheirteammembers.Thestudentsfeltthatiftheywereratingalltheirteammembersatonce,theywouldbeabletoprovidemoreaccurateassessments,andasaresultofhavingfewer(perceived)questionstocompletetheywouldprovidemoreaccurateassessments.Studentscommentedthatcompletingtheentireframeworkforeachteammembersequentiallyresultedinfatiguebytheendofthesurvey,suchthattheywerenotfullyreadingthecompetencyorratingscaleandmaynothavebeenasaccurateforthelastteammemberassessed.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario40
8.2. AccessibilityoftheFramework’sCompetenciesStudentsinthefocusgroupdidnotcommentspecificallyonthelistofcompetencies,buttheTAsdidfeelthatsomecompetenciescouldnotbeassessedatanindividuallevel.Forcompetencieslikeproducehighqualitywork,theTAswouldonlybefamiliarwithteamperformanceandcouldnotrateteammembersindividually.TAsalsofeltthatthecompetenciesdisplaydedicationanddetermination,trackteamprogressvs.projecttimeline,demonstrateaccountability,exchangeinformationinatimelymanner,raisecontentiousissuesinaconstructivewayandsolicitinputbeforeproceedingwereallpoorlyworded,astheywerenotabletoassessthesecompetenciesthroughobservationduringthetutorials.TheTAssawthesecompetenciesastheproductofaseriesorsequenceofbehavioursthatcombinetogether.Giventhattheywerenotparticipatingintheteamsorobservingthemallthetime,theywereonlyabletoseesomeofthebehaviours,notallofthesequence,thatcombinetocreatethecompetency.Theycommentedthattheywouldhavepreferredtomarkoffspecificindividualbehaviourstheysawratherthantryingtodeducethestudents’performancealongthecompetency.8.3 AccessibilityoftheCompetencies’BehaviourallyAnchoredRatingScalesCommentsregardingtheaccessibilityofthebehaviourallyanchoredratingscales(BARS)foreachcompetencycentredaroundthreedifferentaspectsoftheBARS:theirdescriptions,theirsequencingandthenumberofratingpoints.8.3.1. BARSDescriptionsStudentsfoundtheBARSdescriptionslong.Giventhattheyhadtoprovidefeedbacktoeachteammemberindividually,theymentionedthattheywouldstopreadingthemafterawhile.Thestudentsproposedhavingalongerdescriptionofthecompetencyandafewwordsbrieflydescribingeachofthedifferentratingpoints.However,thiswouldnotprovideenoughinformationtodistinguishclearlythedifferentlevelsofperformanceforthestudents.TAsfeltthatthedescriptionsfortheBARSdidnotoutlinebehavioursthattheycouldclearlyobserveintutorialswithoutbeingamemberoftheteam.TAsfeltthatintheirassessmentcontext,theBARSrequireddeductionratherthanobservation,whichmadethemuncomfortablewhenanswering.Inparticular,theyfeltthatthedescriptionswerenotrepresentativeofthepersonalitytypesoftheirstudents.TAsperceivedthatthedescriptionsprivilegedextroversiontothedetrimentofintrovertedstudents,especiallywithrespecttothecommunicationaspect,astheonlywaytheycouldassessstudentsintroducingnewideasoropenlyexpressingopinionsintutorialswasverbally.Additionally,TAsfeltthatwhentheycompletedtheframeworkforteamstheyperceivedasstrong,theyfoundthemonlytobeperformingsatisfactorilybasedontheBARSdescriptions.TAsfoundinthesecasesthattheBARSdescriptionsdidnotmaptotheirperceptionsoftheteam’sfunctioning.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario41
8.3.2. SequencingoftheBARSStudentsdidnotexpressanyconcernsaroundthesequencingofthedescriptionsalongtheBARS,buttheTAswereunsureofthecontinuumsalongwhichtheBARSweredevelopedforeachcompetency.ForcompetencyR12–“motivateothersontheteamtodotheirbest”–TAsfoundthattheexamplesthatwereprovidedinthedescriptionsoftheBARSinplaceoftheword“motivate”sometimescausedtheretoappearasthoughthereweretwocontinuumsalongwhichthecompetencywasbeingassessed.TAsfoundthisdualcontinuumtooccurmostprominentlyintherelationalaspect,wheretheywouldagreewithonehalfofadescriptionbutnottheotherhalf.Inthissituation,TAswereunsureofhowtoratethestudentandwouldmostfrequentlypickthelowerratingoption.Fortherelationalaspect,theTAscommentedthattheyfoundthein-betweenoptionstobelargelyirrelevant,asthescalewasmorenominalthanordinal.TAsfeltthatthelowendoftheBARSwasnotsequencedappropriately,astheyfeltthatitwasbettertodosomethingwrongthantoavoiddoinganythingatall.Inparticular,withrespecttocompetencyR13–“raisecontentiousissuesinaconstructiveway”–theyfeltthatthedescriptionsforlevels1and3onthescaleshouldbeflipped,sothat“avoidedcontentiousissues”wouldbethedescriptionforlevel1and“raisedcontentiousissuesinadestructivemanner”wouldbethedescriptionforlevel3,asthisorganizationwouldshowaprogressionofengagementwiththeteamthatbettermapstotheconceptualframeworkoftheBARS.Onthelowendofthescaleaswell,theTAscommentedthattheyfeltthatthelanguageusedoftenimpliedthatthestudentswouldbeactingmaliciouslyordemonstratinga“self-interestinthedestructionoftheteam,”whichdidnotmapontotheirunderstandingofthestudents.Thiswasnottheintentionoftheframeworkdesigners,whomeanttocommunicatelackofengagementonthelowestendofthescaleandself-centrednessatthemid-rangeofthescale.TAsfoundthatthelanguagearounddoingsomethingto“serveone’sownpurposes”appearedmalicious,whereastheframeworkdesignersdidnotintenditthatway.StudentsfoundthehighendoftheBARStohavevagueandover-archingcompetencywording,insteadofthecollaborativeorientationintendedbytheframeworkdesigners.ThestudentscommentedthatlanguagesuchasthatinR19–“Collaboratedwithothersinamannerwhichpromotedopennessandunderstandingamongteammembers”–wasopentointerpretationandthereforewouldbeassesseddifferentlybyeveryone.Thestudentsinthefocusgroupcommentedthatadescriptionthat“narrowstheanswermore”andspecifiesamannerofpromotingopennesswouldensurethattheframeworkhadasingleinterpretation.8.3.3. NumberofPointsalongtheBARSTwostudentsinthefocusgroupandtheTAsbelievedthatthereweretoomanyratingoptionsforeachcompetency.Havingdescriptionsononlyfourofthesevenpointsonthescalecreatedadditionalconfusion,astheywerenotsurehowtointerpretthemiddleoptions.Themajorityoffocusgroupparticipantsrecommendedeliminatingtheseundescribedratingpointsasawayofreducingthenumberofpointsalongthescaleandmakingtheratingscaleclearer.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario42
8.4. PresentationofFeedbackStudentsinthefocusgroupwereaskedaboutthepresentationofthefeedback(seeexampleprovidedinAppendixB)todetermineiftherewerewaystomakethefeedbackeasiertoreadandunderstand.Nomarkedlynegativecommentswereprovidedaboutthepresentationofthefeedback,noranycommentssuggestingthatstudentswereunabletointerpretthefeedbackoridentifytheirstrengthsandweaknessesfromit.Studentsinthefocusgroupcommentedthattheylikedthecolourcoding,whichallowedthemtoknow“oninstantglance”whattheirstrengthsandweaknesseswere.However,whilehavingthestrengthsandweaknessesvisuallyseparablefromtheothercompetencieswasaplus,adifferentsetofcolourswasrecommendedbyacolour-blindstudent,whocommentedthatitwasdifficulttodistinguishbetweenthegreenandyellowthatdemarcatedstrengthsfromaveragecompetencies.8.5. MissingContent Finally,toensurethattheframeworkaddressedstudents’feedbackneeds,studentswereaskediftherewasanythingtheyperceivedasmissingfromtheirfeedback.Commentsfromthefocusgroupparticipantswerecombinedwithstudents’responsestoquestion9oftheend-of-termsurvey.Thestrongestthemeamongtheresponseswasthatofadesireforcommentsorexamplesfromthefeedbackproviders–somethingthatcouldguidethereceivertoknowexactlyhoworwhytheyreceivedthenumericassessmentsontheframeworkandhowtheirteammemberswishthemtoimprovetheirperformance.Studentsinthefocusgroupsawthisasawaytojustifyhighorlowassessments,whichwasalsoathemediscussedbytheTAs.TheTAsweremoreinclinedtohaveanoptionaljustificationboxaftereachquestioninordertojustifytheirassessmentand/ortheirconfidenceintheirassessmentwhentheyfeltitwasnecessary.Students,ontheotherhand,simplycommentedthattheywantedsomejustificationbutindicatednopreferencebetweenitbeingpercompetencyorperstudent.Thisideawassupportedbyallstudentsatthefocusgroupaswellasby23%ofthestudentsintheframeworkgroup,whocommentedonthisintheirend-of-termsurvey.Theywantedtoknowwheretheyexcelled,withspecificexamples,andtobeabletolearnbyknowingwhytheywereassessedthewaytheywere.8.6. DiscussionThemostcommonthemethatsurfacedunintentionallyinthefocusgroupswashowthefeedbackcouldbeleverageddifferentlyinfutureclassestobetterfacilitatestudentimprovement.Studentssuggestedsomeotherwaystorepresentthefeedback,includingasanaverageofteammembers’assessments,byshowingtheirperformancerelativetotheirteam’sortheirclass’averageperformance,orbyintegratingtheperformanceleveldescriptionsintothefeedbackinsteadofprovidingitasaseparatehandoutwiththefeedback.Givenengineeringstudents’competitivenature,thismightappealtotheirdesiretobethebestintheclass.Bycomparison,TAsprovidedmorefeedbackabouttheaccessibilityoftheframeworkasanassessmentinstrument.Thisislikelyduetotheirfamiliaritywithgradingrubricsasaresultoftheirteachingwork.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario43
Onerecommendationforimprovingtheeffectivenessofthefeedbackfromthefocusgroups,andsomethingthatwascommentedonbysomestudentsontheirend-of-termsurveys,wastoaddadebriefonthefeedbackduringtutorialtimewiththeteamasawhole.Thisstudentrecommendedtheinclusionofastructureddiscussionwiththeteamabouttheirweaknessestoencourageeveryonetoworkmoreeffectivelytogether.Thiscouldbedesignedasawalk-throughorscriptedexercisethatthestudentsrunthemselves,withtheTAsavailableforsupportasneeded.Thiswouldalloweachteamtoreceiveitsownpersonalizedfeedbackwithinthestructureofthelargeclass.Thisactivitycouldguidethestudentsthroughspecificexercisesoractivitiesfortheteamtoexploreareaswheretheyallareweak,ortolookatnewwaysinwhichtheycanleveragethestrengthsoftheteammembers.Havingthedifferentresourceslistedseparatelyfromthefeedbackonlinedidnotappeartobeadetriment,thoughitwasnotcommentedonpositivelyinthesurveyorfocusgroupseither.Amoreintentionalwaytointegratethefeedbackandmethodsofimprovement–eitherinpersonoronline–needstobeconceived.8.7. ConclusionandResponsetoResearchQuestionCombiningstudentfeedbackabouttheaccessibilityoftheframeworkfromtheirend-of-termsurvey,andstudentandTAfeedbackfromthefocusgroupsattheendoftheterm,weconcludethattheframeworkisaccessibleasafeedbackguideforintra-teamuse.FortheTAs,theframeworkistoolonganddifficulttocompletegiventheirinteractionswiththestudentsandthecognitiveloadoftheirteachingresponsibilities.Bothgroupsidentifiedanumberofwaysinwhichtheframeworkcouldbemademorerelevantandintuitive.
9. ConclusionsandFutureWorkThisprojectevaluatedtheeffectivenessofafeedbackframeworkasafoundationforanewweb-basedtoolthatprovidesstudentswithstructuredfeedbackfromteammates,alongwithpersonalizedexercisesandactionablestrategiesthatguidetargetedlearningintheareastherebyidentified.Specifically,thestudyinvestigatedwhetherthefeedbackframework,whenusedforintra-teamselfandpeerfeedback,wouldincreaseastudent’sabilitytolearnaboutandimprovetheirteam-effectiveness.Whenfeedbackfromtheframeworkiscomparedtofeedbackgatheredusinganunstructuredprompt,wefindthatfeedbackfromtheframeworkhasagreaterabilitytoincreasestudentself-awarenessandprovidethemwithinformationthatcanguidethemindevelopingteam-effectivenesscompetencies.Studentswhousedtheframeworkhadsignificantlydifferentstrengthsandweaknessesreliablyidentifiedforthem,andtheyreceivedconsistentassessmentsoftheirperformancefromtheirteammembers.Studentsintheunstructuredfeedbackgrouphadlessconsistentfeedbackthatmoreoftendiscussedstrengthsalonethanbothstrengthsandweaknesses.Additionally,asstudentsintheunstructuredfeedbackgroupdiscussedwhichevercompetenciestheyfeltwereimportant,thefeedbackislesstargetedandfocusesonhowtheteamperceivedthestudent’sperformance.Studentsinthefeedbackgrouphadsignificantlygreateragreementwiththestatementsthattheyreceivedfeedbackonabroaderrangeoftopicsandontheirweaknessesthandidstudentsintheunstructuredgroup.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario44
Whilestudentsintheunstructuredgroupdidnothavethesesamebenefits,theydidcommentthatthetextualfeedbackmadethemfeelmorecommittedtotheirteam,asitdemonstratedthattheirteammembershadaninterestintheirdevelopmentandinthatoftheteamasawhole.Basedonthisbenefitandonrequestsfromstudentsinthefeedbackgroupforexamplesandcommentsintheirfeedback,webelievethatahybridcombiningthefeedbackframeworkandsometextualfeedbackwouldbeideal.Thisdocumentaddressesanimportantcomponentofalarger,ongoingprojectlookingattheeffectivenessofourfeedbackframeworktofacilitatefeedbackthatguidesstudentstolearnaboutanddeveloptheircompetenceaseffectiveteammembers.Wehavealreadyincorporatedmuchofthelearningachievedthroughthisstudyintoourweb-basedtool.Forexample,thenumberofcompetenciesbeingaddressedhasbeensubstantiallyreducedandanoptionforholisticfreeformfeedbackincorporatedintotheframework.Further,asstudentsoftenworkindiverseteams,weareintheprocessofinvestigatinghowdiversityofpersonalitycharacteristicsinteamsaffectsthefeedbackastudentreceivesfromtheirteam.Thisinvestigationiscurrentlyinanalysisandwillbepublishedatalaterdate.
TeachingTeam-effectivenessinLargeClasses
HigherEducationQualityCouncilofOntario45
References
Bushe,G.R.,&Coetzer,G.(1995).AppreciativeInquiryasaTeam-DevelopmentIntervention:AControlledExperiment.JournalofAppliedBehavioralScience,31(1),13-30.
CanadianEngineeringAccreditationBoard(2010).AccreditationCriteriaandProcedures.Retrievedfromhttp://engineerscanada.ca/e/files/Accreditation_Criteria_Procedures_2010.pdf
Lencioni,P.M.(2002).TheFiveDysfunctionsofaTeam:ALeadershipFable.SanFrancisco,CA:Jossey-Bass.
Lingard,R.W.(2010).TeachingandAssessingTeamworkSkillsinEngineeringandComputerScience.JournalofSystemics,CyberneticsandInformatics,8(1),34-37.
LoughboroughUniversity(2009).Retrievedfromhttp://www.webpa.ac.uk/
Loughry,M.L.,Ohland,M.W.,&Moore,D.D.(2007).DevelopmentofaTheory-basedAssessmentofTeammemberEffectiveness.EducationalandPsychologicalMeasurement,67(3),505-524.
Maxwell,J.(2011,July).TeamEffectivenessQuestionaire.Retrievedfromhttp://www.firststepstraining.com/resources/teq/index.htm
Moore,T.J.,Diefes-Dux,H.,&Imbrie,P.K.(2006).AssessmentofTeamEffectivenessDuringComplexMathematicalModelingTasks.36thASEE/IEEEFrontiersinEducationConference.SanDiego,CA.
Sheridan,P.K.,Phillips,J.,ElGammal,L.,Evans,G.,&Reeve,D.A.(2013).TeamEffectivenessInventoryforGuidedReflectionandFeedback.AmericanSocietyofEngineeringEducationAnnualConferenceandExposition.Atlanta,Georgia.
Sheridan,P.K.,Reeve,D.,&Evans,G.(2014).UnderstandingTeachingAssistants'AssessmentofIndividualTeamworkPerformance.AmericanSocietyforEngineeringEducationAnnualConferenceandExhibition.Indianapolis,IN.
Tuckman,B.W.(1965).Developmentalsequenceinsmallgroups.PsychologicalBulletin,63(6),384-399.