Gyuris - Evaluating the Alignment FINAL - Adelaide
8
1 I-MELT, 11-13 December 2017 Evaluating the Alignment Between the RSD Framework and the Effectiveness of ‘Bolted on’ Research Skills Training Emma Gyuris 1 1 College of Science and Engineering, James Cook University Corresponding author email address: [email protected] A peer reviewed short paper for a presentation at the International conference on Models of Engaged Learning and Teaching (I-MELT), 11-13 December 2017. Available from www.imelt.edu.au Abstract This research considers the alignment of the curriculum and assessment design of the subject Research and Communication Skills for the Natural Sciences (SC5055) against the RSD matrix and asks which students benefit most from training in research and communication skills. The impact of SC5055 on students’ achievement and their ability to self-evaluate the development of their skills and understanding is also explored. Potential improvement for the RSD framework is highlighted. Introduction Research skills training is a focus of interest in higher education not only for students aspiring to be the producers of new knowledge and understandings but also for those who will access and use such new knowledge and understandings in the course of their professional activities. Critical thinking and written communication skills underlie almost every stage or component of the research endeavour [1, 2, 3]. But, while critical thinking is much discussed, clear definitions of what it is, how it may be developed, practiced and assessed, are seldom clear [1, 4]. In research training, the importance of the linkages between the development of critical thinking skills and critical reflection or reflective judgement, including reflection and judgement on one’s own development, is even less understood. At James Cook University the overwhelming majority of candidates in the Master of Science (Coursework) program expect to complete a minor thesis and then progress to a higher degree by research candidature. The subject SC5055:03 ‘Research and Communication Skills in the Natural Sciences’ (from hereon SC5055), was purposed to hone the research and, in particular, research communication skills of postgraduate coursework students. As part of their research training agenda, many Australian universities offer research methods subjects, focusing on the multiple facets of research communication as a key skill. This emphasis is due to the expectations that graduates of masters and doctoral degrees will author and co-author reports and research papers, propose new research, apply for funding, and be able to participate in the peer review
Transcript of Gyuris - Evaluating the Alignment FINAL - Adelaide
1
I-MELT, 11-13 December 2017
EvaluatingtheAlignmentBetweentheRSDFrameworkandtheEffectivenessof‘Boltedon’ResearchSkillsTraining
EmmaGyuris1
1CollegeofScienceandEngineering,JamesCookUniversity
Correspondingauthoremailaddress:[email protected]
ApeerreviewedshortpaperforapresentationattheInternationalconferenceonModelsofEngaged
LearningandTeaching(I-MELT),11-13December2017.Availablefromwww.imelt.edu.au
Abstract
Thisresearchconsidersthealignmentofthecurriculumandassessmentdesignofthesubject
ResearchandCommunicationSkillsfortheNaturalSciences(SC5055)againsttheRSDmatrix
andaskswhichstudentsbenefitmostfromtraininginresearchandcommunicationskills.The
impactofSC5055onstudents’achievementandtheirabilitytoself-evaluatethedevelopment
oftheirskillsandunderstandingisalsoexplored.PotentialimprovementfortheRSDframework
ishighlighted.
Introduction
Researchskillstrainingisafocusofinterestinhighereducationnotonlyforstudentsaspiringtobethe
producersofnewknowledgeandunderstandingsbutalsoforthosewhowillaccessandusesuchnew
knowledgeandunderstandingsinthecourseoftheirprofessionalactivities.Criticalthinkingandwritten
communicationskillsunderliealmosteverystageorcomponentoftheresearchendeavour[1,2,3].But,
whilecriticalthinkingismuchdiscussed,cleardefinitionsofwhatitis,howitmaybedeveloped,practiced
andassessed,areseldomclear[1,4].Inresearchtraining,theimportanceofthelinkagesbetweenthe
developmentofcriticalthinkingskillsandcriticalreflectionorreflectivejudgement,includingreflectionand
judgementonone’sowndevelopment,isevenlessunderstood.
AtJamesCookUniversitytheoverwhelmingmajorityofcandidatesintheMasterofScience(Coursework)
programexpecttocompleteaminorthesisandthenprogresstoahigherdegreebyresearchcandidature.
ThesubjectSC5055:03‘ResearchandCommunicationSkillsintheNaturalSciences’(fromhereonSC5055),
waspurposedtohonetheresearchand,inparticular,researchcommunicationskillsofpostgraduate
courseworkstudents.Aspartoftheirresearchtrainingagenda,manyAustralianuniversitiesofferresearch
methodssubjects,focusingonthemultiplefacetsofresearchcommunicationasakeyskill.Thisemphasisis
duetotheexpectationsthatgraduatesofmastersanddoctoraldegreeswillauthorandco-authorreports
andresearchpapers,proposenewresearch,applyforfunding,andbeabletoparticipateinthepeerreview
2
I-MELT, 11-13 December 2017
process[5,6].Furthermore,higherorderthinkingskills–criticalthinkingandreflectivethinking–are
prerequisitesfortheunambiguousandeffectivecommunicationthatisexpectedbyemployersand
academiaalike.
Here,wepresentexploratoryresearchthatconsiders
1)thealignmentofthecurriculumandassessmentdesignofSC5055againstWillison’sRSDmatrix[7],
2)whichstudentsbenefitmostfromtraininginresearchandcommunicationskills,and
3)theimpactofSC5055onstudents’achievementandtheirabilitytoevaluatethedevelopmentoftheir
ownskillsandunderstandings.
Academicsetting
AtJCU,themajorityofpostgraduatestudentsusetheMSc(coursework)asapathwaytothePhD.Hence,
anintroductorysubjectinresearchmethodsandcommunicationwasseenasanefficientplatformto
teach,practiceandassesscriticalthinking,criticalreflectionandsciencecommunicationskills.Theteaching
teamwaschallengedtoengageahighlydiversebodyofstudentswithstrikinglyvariableEnglishlanguage
skillsandwithlimitedexposuretoresearchandresearchtraining.Moststudentshavehadnoorminimal
exposuretoissuesrelatedtotheresponsibleconductofresearch.Further,studentshadhighlyvariableand
narrowlyfocusedresearchinterests.Asnotedpreviously[8],moststudentsexhibitedsignificantresistance
to“wastingtime”onlearningresearchandcommunicationskills.
Subjectdelivery,curriculumandassessment
Offeredasalimitedattendance,blendedlearningclass,SC5055wascompulsoryforallstudentsintheMSc
(Coursework).Priortoattendingscheduledclasses,studentswerepresentedwithane-foldercontaining
learningresourcesandpreparatorytaskstobecompleted.
Assessmentconsistedofanoriginalresearchproposal(40%oftotalmarks),anoralcommunicationpiece
(30%,andnotdiscussedfurtherinthispaper)andalearningportfolio(30%)thatrequiredself-evaluationof
thedevelopmentofskillsrelevanttosubjectlearningoutcomes[9].Thepreparationoftheseassignments
wasscaffoldedbyfourworkshops,plustwotwo-hourtutorialsbookendingthesemester.Throughout,
teamworkandcollegialityamongststudentswasstressed.
Atfirst,studentswereintroducedtothefunctionsandformatofscientificresearchproposalsandtotopics
ontheresponsibleconductofresearch.Thelevelofautonomyexpectedparalleledlevel4ofthefive-point
RSDframework,withstudentsdirectedtoobservetheguidelinesoftheJCURisingStarsLeadership
program(seehttps://www.jcu.edu.au/research/i-want-to/grants/internal-grant-schemes/research-
3
I-MELT, 11-13 December 2017
services).Theproposalwasgradedtwice.Thefirstsubmissionwasgradedandcommentedonbystaffand
waspeerreviewedbyatleasttwofellowstudents.Studentsgaveandreceivedpeerreviewsduringand
followingthesecondworkshopthatwasfocusedoncriticalreadingandwriting[10].Theprocessofwriting
anoriginalresearchgrantapplicationfullyengagedstudentswiththefacetsoftheRSDframework.This
includedethicalissuesastheyrelatetoresearchusinghumanornon-humansubjectsaswellastheethics
aroundpeerreviewandauthorship(Table1).
Table1.AlignmentofRSDlevel4indicatorsandassessmentcriteriafortheresearchproposal,oneofthemajor
assignmentsforSC5055.NumbersinbracketsincolumnthreedenotecorrespondencewithRSDfacets,alsonumbered.
FacetsofRSD
frameworkLevel4indicators
Alignmenttoindicators(made
evidentbyassessmentcriteria
below)
1. Embark&Clarify
Studentsgeneratequestions
/aims/hypothesesframedwithin
structuredguidelines.Anticipate
&prepareforECSTissues
• Theresearchquestionisclearly
originalandfitsclearlywithin
AustralianandNewZealand
StandardResearch
ClassificationCodesandresearch
type.(1)
• Ethicsandsafetyissuesare
consideredandareappropriateand
comprehensivefortheresearch(1,6)
• Themethodologyisappropriateto
addresstheresearchquestion.(2)
• Theproposalhasaclearproject
descriptionthatincludeswhy
theworkwarrantsfunding
basedontheidentificationofa
knowledgegaporcontradiction,
innovativeresearchandvery
clearlydefinedoutcomesand
outputs.(2,3,4,5)
• Thebudgetisfitforthepurposeof
theproposedresearchandistightly
2. Find&Generate
Studentscollect&recordself-
determinedinformation/data
choosinganappropriate
methodologybasedonparameters
set.3. Evaluate&Reflect
Studentsevaluate
information/data&theinquiry
processusingself-determined
criteriadevelopedwithin
parametersgiven.Reflectsto
refineothers’processes4. Organise&Manage
Studentsorganise
information/datausingself-
determinedstructures,&manage
theprocesses(includingteam
function)withintheparameters
set. 5. Analyse&Synthesise Studentsanalyse
4
I-MELT, 11-13 December 2017
information/data&synthesizeto
fullyintegratecomponents,
consistentwithparametersset.
Fillknowledgegapsthatare
statedbyothers
alignedwithproposedresearch
methods.(4)
• Trackrecordandleadership,relative
toopportunity,areclearly
articulated.(5)
• Alignmenttouniversitystrategic
intentmadeclear.(5)
• Theprojecthasaclearobjectiveand
thesummaryiswellarticulatedfor
thelayperson.(6)
• Qualityofwriting,observanceof
guidelinesandpresentation.(6)
6. Communicate&
Apply
Studentsusediscipline-specific
language&genrestodemonstrate
scholarlyunderstandingfora
specifiedaudience.Theyapplythe
knowledgedevelopedtodiverse
contextsandspecifyECSTissuesin
initiating,conducting&
communicating.
Performanceofthe2015studentcohort
ToexaminetheeffectivenessofSC5055onstudents’achievementwetooktheformativeandsummative
gradesobtainedfortheirresearchproposals.Weconsideredthatthedifferenceingradebetweenthetwo
reflectedtheimpactofengagingwithSC5055.Ofthe61studentswhosubmittedworkforboththe
formativeandsummativeassessments,5students’summativegradeswerebelowtheformativegrade(i.e.
below0onthe‘y’axis),while32studentsimprovedby10marksormore(Figure1).Studentsobtaining
failinggradesfortheirfirstsubmissionmadethegreatestimprovement.All6studentswhoimprovedby
≥20marksscored≤70marksfortheirformativegrade(Figure1),implyingthatweakerstudentsbenefited
themostfrominstructiononproposalwritingandcriticalreadingandwritingcoupledwiththeextensive
feedbackprovided(Pearson’sr=0.52,P<0.001).
-10
0
10
20
30
40
50
30 35 40 45 50 55 60 65 70 75 80 85 90
Improvem
entinscore
betweenfirstand
secondsubmissionof
proposal
5
I-MELT, 11-13 December 2017
Figure1.Improvementintheresearchproposalscore(verticalaxis)asafunctionofstudent’sformative
score(horizontalaxis).The10-pointimprovementishighlightedingreen.
Wethenprobedtoseewhetherstudentsshowingthemostimprovementweremoreawareoftheir
learningexperiencethanwerestudentswhodisplayedlessimprovement.Wetookthegradesobtainedfor
thereflectivelearningportfolioasaproxymeasureofstudents’abilitytounderstandtheirownlearning
andcorrelatedthosewiththescoresforthefirstsubmissionoftheproposal(Figure2a)andwiththe
improvementinscoresbetweenthetwosubmissions(Figure2b).Toaddressissuesofvalidityand
reliability[11],theportfolioconsistedof4individualreflectivepiecesandwasassessedby4different
academicteachingstaff.
Figure2a.Scoresonlearningportfolioasafunctionofstudent’sgradeforthefirstsubmissionoftheresearch
proposal).Studentsscoringabove65%(greenline)gainedgradesofCredit,DistinctionorHighDistinction.
20
30
40
50
60
70
80
90
100
30 35 40 45 50 55 60 65 70 75 80 85 90
Learningportfolioscore
Proposalscore,firstsubmission
Proposalscore,firstsubmission
6
I-MELT, 11-13 December 2017
Figure2b.Scoresonlearningportfolioasafunctionofstudent’simprovementintheresearchproposal.Students
scoringabove65%(greenline)gainedgradesofCredit,DistinctionorHighDistinction.
Students’abilitytothinkabouttheirownlearningprocesseswasonlyweaklyassociatedwiththeirscores
forthefirstsubmissionoftheirproposal(Fig.2a)(Pearson’sr=0.41,P<0.001)and,importantly,therewas
norelationshipbetweentheimprovementachievedforthesecondsubmissionandthelearningportfolio
score(Fig.2b)(Pearson’sr=0.003,P=0.98).Thisleadstothesuppositionthatalargeproportionofstudents,
irrespectiveofinitialperformanceandimprovement,arelackingawarenessoftheirownlearning,andthat
theprocessofwritingthelearningportfoliofailedtomakeexplicitthementalmodelsandskillawareness,
whichisanallegedbenefitofwritingreflectiveessaysandlearningportfolios[8].
Whileitiswellknownthatfutureperformanceisquitereliablypredictedbypastperformance,past
performanceisoftenapoorpredictorofimprovementinperformance[12].Thisisattributedtothe
performanceheuristic,meaningthatinsteadofcriticallyexaminingthepotentialforfutureimprovementin
performance,peopleexpectfuturesuccessorfailuretobecommensuratewiththeirpastperformance.
Suchexpectationspreventinsightsintofactorsthatcanshapefutureperformance.Thiscouldexplainwhy
studentswithapoorinitialperformancefortheproposaltendtofailtounderstandtheirsubsequent
improvementandshowlimitedawarenessoflearninganddevelopment,scoringlowforthelearning
portfolio.Manyothers,performingrelativelyhighlyintheproposal,alsodemonstratedpoorawarenessof
theirskillsandlearning.ThishasthepotentialtosignificantlydiminishtheeffectivenessofRSDprograms,
andtheomissionbytheRSDframeworkoftheabilitytocriticallyreflectonone’sowndevelopmentneeds
addressing.
20
30
40
50
60
70
80
90
100
-10 -5 0 5 10 15 20 25 30 35 40 45 50
Learningportfolioscore
Improvementinscorebetweenfirstandsecondsubmissionofproposal
7
I-MELT, 11-13 December 2017
Giventhelackofanyrelationshipbetweentheportfolioscoreandtheimprovementachievedforthe
secondsubmissionoftheproposal(seeFigure2b),howdoesoneaccountfortheobservationthatstudents
scoringlowerinthefirstsubmissiontendedtoshowthemostimprovementforthesecondsubmission?
Thereareseveralpossibleexplanationsforthis.Firstly,somestudents,disappointedwiththeirformative
grades,maysimplyseektheassistanceofacademicstudyskillsadvisorsand,afterimplementingtheadvice
given,obtainamuch-improvedgradefortheirsecondsubmission.Inthiscaselowgradeswouldbe
expectedfortheportfolio.Others,satisfiedwiththeirformativegradeandrelyingontheperformance
heuristic,wouldexpectanequallysatisfyinggradeforthesecondsubmission,thuslimitingtheir
improvementandscoringlowfortheportfolio(Fig.2a).Thelackofdemonstratedrelationshipbetween
reflectivethinkingandacademicperformance[13]wouldalsoexplainthewidespreadofscoresforthe
portfolioamongststudentsgainingadistinctionorhighdistinctionforthefirstsubmissionoftheproposal
(seeFigure2a).Furthermore,andasnotedbyHoseinandRao[8],itisuncertainifthemarkingcriteria
scaffoldedthereflectivewritingofthemoreacademicallysophisticatedstudents,thatis,thosereceiving
scores>80%fortheformativeproposalandscoringhighlyintheportfolio.
Thisexploratoryresearchscopedtherelationshipbetweenstudents’researchskills,asexpectedaccording
tolevel4oftheRSDframework,andtheirabilitytoreflectontheirdevelopmentofknowledgeandskill.
Manystudents,whileachievingmuchimprovementintheirassessmentscore,seemunawareofthe
processesthatallowedthemtoimprove.Weneedtoequipstudentswiththeskillstobecompetent,self-
regulatedlearners,understandingandpracticingthemetacognitiveskillsthatallowthemtoperformat
theirhighestlevel.Thisisespeciallyimportantinresearchtrainingasreflectingonandunderstandingthe
reasonsforsuccessorfailureintheresearchendeavourisacriticalattributeofaccomplishedresearchers.
8
I-MELT, 11-13 December 2017
References
DwyerCP,HoganMJ,StewartI.Anintegratedcriticalthinkingframeworkforthe21stcentury.Thinking
skillsandCreativity.2014;12:43-52.Epub18Jan.2014.
GhanizadehA.Theinterplaybetweenreflectivethinking,criticalthinking,self-monitoring,andacademic
achievementinhighereducation.HigherEducation.2017;74:101-14.
MooreT,MortonJ.Themythofjobreadiness?Writtencommunication,employabilityandthe'skillsgap'in
highereducation.StudiesinHigherEducation.2017;42:591-609.Epub30Jul.2015.
CargasS,WilliamsS,RosenbergM.Anapproachtoteachingcriticalthinkingacrossdisciplinesusing
performancetaskswithacoomonrubric.ThinkingskillsandCreativity.2017;26:24-37.Epub31May
2017.
CargillM,SmernikR.Embeddingpublicationskillsinscienceresearchtraining:awritinggroupprogramme
basedonappliedlinguisticsframeworksandfacilitatedbyascientist.HigherEducationResearch&
Development.2016;35:229-41.Epub25Sep2015.
McCarthyBD,DempseyJL.Cultivatingadvancedtechincalwritingskillsthroughagraduatelevelcourseon
writingreseachproposals.JournalofChemicalEducation.2017;94:696-7-2.
WillisonJ,Buisman-PijlmanF.PhDprepared:Reseachskilldevelopmentacrosstheundergraduateyears.
InternationalJournalforResearcherDevelopment.2016;7:63-83.
HoseinA,RaoN.Students’reflectiveessaysasinsightsintostudentcentred-pedagogieswithinthe
undergraduateresearchmethodscurriculum.TeachinginHigherEducation.2017;22:109-25.
NilsonLB.Creatingself-regulatedlearners.Sterling,Virginia:StylusPublishing.2013;56-58.
GyurisE,CastellL.TellThemorShowThem?HowtoImproveScienceStudents’SkillsofCriticalReading.
InternationalJournalofInnovationinScienceandMathematicsEducation.2013;21:70-80.
MonizT,ArntfieldS,MillerK,LingardL,WatlingC,RegehrG.Considerationsintheuseofreflectivewriting
forstudentsassessment:Issuesofreliabilityandvalidity.MedicalEducation2015;49:901-8.
CritcherCR,RosenzweigEL.Theperformanceheuristic:Amisguidedrelianceonpastsuccesswhen
predictingprospectsforimprovement.JournalofExperimentalPsychology:General.2014;143:480-
485.
PhanHP.Predictingchangeinepistemologicalbeliefs,reflectingthinkingandlearningstyles:Alongitudinal
study.BritishJournalofEducationalPsychology.2008;78:75-93.
