It summit 2016_combined
-
Upload
kevindonovan -
Category
Education
-
view
285 -
download
0
Transcript of It summit 2016_combined
Rubrics and Peer-Review in Canvas: Examples from FAS and HKS
Daniel Jamous, Senior Instructional Technologist, Academic Technology Services, Harvard University Information Technology
Alice Heath, Master in Public Policy Candidate, Harvard Kennedy School
Kelly Miller, Lecturer on Applied Physics, Harvard John A. Paulson School of Engineering and Applied Sciences
Salimah Samji, Manager of Building State Capability Program, Center for International Development, Harvard Kennedy School
Allison Pingree, Director of Professional Pedagogy, Strengthening Learning and Teaching Excellence (SLATE), Harvard Kennedy School
IT Summit June 2 2016
UsingCanvasforPeerReviewatHKS:PolicyAnalysis&Design(API-505)
AllisonPingree &AliceHeathHarvardITSummit
Thursday,June2,2016- 2:20– 3:15pm
CourseContext:API-505,PolicyAnalysis&Design• Required forall1st-yearMastersofPublicPolicyStudents
• 4sections,eachtaughtbyadifferent faculty
• Integratesothercorecourses, leadingtodeliverables thatreflectthepracticeofreal-worldpublicpolicy:memos,oralbriefings, slidedecks
• Onesection(coursehead,Prof.JohnDonahue,chairofMPPProgram)mademostextensiveuseofpeerreviewandrubricsinCanvas
• Twoothersectionsusedpeerreview inCanvaswithsimplifiedrubric;4thusedpeerreviewviae-mail
WhyUsePeerReviewwithaRubric?WhyinCanvas?
• WeknewfromSpringExercise[experiential capstone]thatthedraft-feedback-rewritecycleisagreattoolforlearning.
• Wealsoknewfrompriorversionsofthiscoursethatthefacultysimplycouldn’tgivedetailedfeedbackoneverydraft,especiallynotintimeforittobeuseful.
• We’dlearned fromobserving theNegotiationcoursethattheMPPs—especiallyincohorts—wereskilledatdealingwitheachothercandidlybutthoughtfully.
• Coursefocusitselfistheevaluationofoptionsalongvariouscriteria - rubricallowedustobemoretransparent aboutquality,andpracticewhatwepreach.
• Weneededawaytomanagethelogistical complexityofpairingupstudentsandtrackingtheirexchanges.
Overall,HowDiditGo?In-classpollofstudents:
• verystrongsupportfortheusefulness ofthepeerreviews (~40%extremelyuseful,30%definitelyuseful,20%somewhatuseful,and10%orlessnotuseful)
End-of-courseevaluations:mentioned5xasoneof“2-3mostvaluableaspects”ofcourse:
• “Requiringustoprovidepeerfeedbackonthefirstmemoassignmentwasveryhelpful--forcedustodosomethingthatwewouldnothavedoneotherwise,butwasreallyvaluable.”
• “Thepeerfeedbackmechanismwasexcellent.Ourgroupfounditsovaluablethatweorganizedpeerfeedbackforthelastassignment eventhoughitwasn'trequired.”
Faculty:
• “Ithinkitwasoneofthebestfeaturesofthecoursethisyear.... Canvas,whilenotwithoutitsglitches, isahugeleapforward.”
ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde
thequalityofthepeerreview.
2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.
• Lineedits• Attachfiles• Usearubric• Writeageneral comment
3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.
ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde
thequalityofthepeerreview.
2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.
• Lineedits• Attachfiles• Usearubric• Writeageneral comment
3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.
ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde
thequalityofthepeerreview.
2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review
• Lineedits• Attachfiles• Usearubric• Writeageneral comment
3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.
ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde
thequalityofthepeerreview.
2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.
• Lineedits• Attachfiles• Usearubric• Writeageneral comment
3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.
LessonsLearned• Strategies
• Ifyouwanttosetadeadlineorgradethepeerreview,createanewassignment.• Createadocumentwithinstructionsonhowtogiveandviewpeerfeedback.
• Usescreenshots.• Testtheinstructionsfromastudent’sperspective.
• Benefits• Tostudents:
• Peersofferedvaluableinsights• Givingfeedbacktoothersfocusedthemonassessment criteria,whichclarifiedhowtoimprovetheirownwork
• ToFaculty:• Logisticsfarmorestreamlinedandsimplified (vs.manual/email)• Easiertocheckcompletion• Requiredfewerdocuments– rubricintegrated• Abilitytouseanonymousfeedback
AAP 50 and Canvas
AP50lectures group work exams projects
• Physics as a Foundation for Science & Engineering
• new class in SEAS Sept. 2012
• 30 students first year, now 75 students
• undergraduate engineering students
• project and team based
• “Flipped classroom”
Applied Physics 50
Team & Project Based Learning
AP50lectures group work exams projects
• 3 projects per semester
• 3 groupings per semester
• final report for each project
• teaching staff provides feedback on reports via rubrics uploaded on Canvas
• teams resubmit reports (via Canvas) for a final grade
Canvas for Groups and Reports
Our$challenge:$How$to$diffuse$our$ideas$and$create$a$community$of$practice?
• Huge%demand%for%PDIA,%short%supply%of%faculty• Change%agents%in%developing%countries%cannot%afford%to%attend%an%executive%education%course– Sustainable%change%requires%teaching%teams%who%work%together
– There%is%a%spacing%effect%of%learning%in%one’s%own%environment%
• Why%not%experiment%with%a%MOOC?
Our$answer:$Canvas$Catalog
• We%already%had:– Blog:%International%following%(155%countries)– 34%high%quality%3P5%min%videos%%(why?)
• Canvas%Catalog– No%Harvard%ID%required%to%sign%up– Easy%to%use– Do%It%Yourself%(DIY)
What$did$we$do?
• Launched%our%first%MOOC%on%Nov.%8,%2015– 6%weeks–Weekly%assignments• Written%answers%(no%quiz)• 4%conceptual%and%2%openPended
–Weekly%reflection%exercises– Average%of%3P5%hours%of%work– Peer%review%and%rubric%tool%• to%grade%and%to%learn%from%each%other
Example$of$assignment$rubric
241$people$from$53$countries$completed$Part$I
Top$10$Countries
# ofStudents
United%States 34Nigeria 31Peru 21Great%Britain 15India 8Kenya 7Tanzania 7Pakistan 7Nepal 6TimorPLeste 6
What$did$we$learn?
79%$rated$the$Peer$reviewsgood$or$higher
“The$peer$reviews$were$truly$amazing."A"huge"part"of"it"was"helpful"and"usually"highlighted"areas"I"may"not"have"considered.”
“Very"useful"and"have"never$had$it$so$good$in$all$the$online$programs$i have"undertaken"in"the"past”
“I"think"the"peer"review"was"a"very"good"opportunity"to"engage"with"like=minded"persons"that"provided$competent$reflection$and$criticism.”
“Generally"good."In"fact"I"think"the"act"of"peer"reviewing"is"more"valuable"than"the"act"of"being"peer"reviewed."Doing"the"reviewing"forces"me"to"think"seriously"about"another"person's"take"on"the"question.”
“It"is$informative$to"read"others"reflections,"and"adds$to$the$community$feel$of"the"course.”
Student$Feedback$on$Peer$Reviews
“A"little"disappointing."A"couple"offered"interesting"comments"that"were"worth"thinking"about"but"others$were$delayed$or$not$completed.”
“Peer"review"assignments"were"very"useful"except"that"it"was"very"difficult"to"find"course"colleagues"to"comment"on"work"consistently."It"looks"like"people"were$very$busy$and$could$not$find$much$time$to$peer9review$work.”
“Sometimes,"a"reviewer"can"score"a"student"low,"because"of"his/her"limited$understanding$of$the$concepts.”
“Some"reviewers"do"not"simply"understand"the"concept"or"the"problem,"hence"they"provide"minimal"contribution."
Student$Feedback$(cont’d)
Valuable$feedback$and$peer$interaction
Generic$no$value$add
Limited$understanding$of$the$concepts
Limited$understanding$of$concept/language
Where$to$next?
• More%specific%questions/prompts%for%reviewer–What%did%you%like?–What%would%you%suggest%for%improvement?– How%does%this%differ%from%what%you%learnt?
• Assignments– Hired%graders%
• Flipped%classroom%Fall%2016– Opportunity%to%test%peer%review%in%residential%course%(calibrated%reviews)