Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course
-
Upload
christina-hendricks -
Category
Education
-
view
175 -
download
1
Transcript of Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course
![Page 1: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/1.jpg)
Longitudinal Analysis of Peer Feedback in a Writing-
Intensive Course: A Pilot Study
PI: Christina HendricksCo-PI: Jeremy Biesanz
University of British Columbia-Vancouver
Funded by the UBC Institute for the Scholarship of Teaching and Learning SoTL Seed Fund
Festival of Learning, June 2016Slides licensed CC-BY 4.0
![Page 2: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/2.jpg)
Literature on peer feedbackReceiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012)
Giving peer feedback improves writing(Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
![Page 3: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/3.jpg)
GAPS: Most studies look at revisions to a single essay, not changes across different essays
Draft 1
Draft 2
Draft 3
Essay 1 Essay 2 Essay 3 Essay 4 Essay …n
PFB
PFB
PFB
PFB PFB PFB
Few studies look at “dose-response curve”
![Page 4: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/4.jpg)
Pilot study research questions1. How do students use peer comments given and
received for improving different essays rather than drafts of the same essay?
2. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
3. Does the quality of peer comments improve over time?
![Page 5: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/5.jpg)
• Interdisciplinary, full year course for first-years
• 18 credits (English, History, Philosophy)• Students write 10-12 essays (1500-2000
words)• Peer feedback tutorials every week (4
students)
http://artsone.arts.ubc.ca
Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0
Osamu Tezuka, public domain on Wikimedia Commons
Jane Austen, public domain on Wikimedia Commons
Friedrich Nietzsche, public domain, Wikimedia Commons
![Page 6: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/6.jpg)
Data for pilot study 2013-2014
• 10 essays by 12 participants (n=120)
• Comments by 3 peers on essays (n=1218)
• Comments by instructor (n=3291)
• All coded with same rubric
![Page 7: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/7.jpg)
Coding RubricCategories(plus subcategories, for 11 options)
• Strength of argument• Organization• Insight• Style & Mechanics
Numerical value
1: Significant problem2: Moderate problem3: Positive comment/praise
E.g., STREV 2: could use more textual evidence to support your claims
Change for future
![Page 8: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/8.jpg)
Inter-coder reliability
Fleiss’ Kappa Intra-class correlation
Student comments (n=141)
All categories: 0.61 (moderate)Most used categories: 0.8 (excellent)
0.96 (excellent)
Essays (n=120) 0.71 (adequate)
3 coders:• Daniel Munro & Kosta Prodanovic
(undergrads, former Arts One)• Jessica Stewart (author, editor)
Change for future
![Page 9: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/9.jpg)
LOOKING AT TRENDS IN COMMENTS OVER TIME
![Page 10: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/10.jpg)
INSTRUCTORComments
-.28**
Strength
Style
Organiz.
Insight-.04*
Number of 2 comments over time
![Page 11: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/11.jpg)
STUDENT comments
Strength
Style
Organiz.
Insight
-.16**
Number of 2 comments over time
![Page 12: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/12.jpg)
INSTRUCTORComments .31***
Strength
Style
Organiz.
Insight
.08**
.19**
.11**
Number of 3 comments
![Page 13: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/13.jpg)
STUDENTComments
Strength
Style
Organiz.
Insight
Number of 3 comments over time
![Page 14: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/14.jpg)
HOW DOES ESSAY QUALITY CHANGE OVER TIME?
![Page 15: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/15.jpg)
Essay quality improves linearly
b = .038t(107) = 2.1p = .037
Essays rated on a 7-point scale
![Page 16: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/16.jpg)
MORE COMPLEX ANALYSES
![Page 17: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/17.jpg)
Cross-lagged panel design with auto-regressive structure
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
![Page 18: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/18.jpg)
Path A: Instructor Comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Ratings of 1 in Strength (-.12*) & Org. (-.23**)• Ratings of 2 in Strength (-.06*) & Style (-.08*)• Ratings of 3 in Str, (.11*), Insight (.35*), Style
(.15*) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
![Page 19: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/19.jpg)
Path A: Student comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Ratings of 2 in Insight (-.53*)• Ratings of 3 in Organization (.13*)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
![Page 20: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/20.jpg)
Path C: instructor commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects don’t show up if split out by category• Comments ratings of 1 (.29**)• Comments ratings of 2 (.23*)• Comments ratings of 3 (.21, p=.057)*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p
<.00001
See next slid
e for c
orrecti
on
![Page 21: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/21.jpg)
Path C: instructor commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects:• Rating of 3 in Strength (.34**) and Style (.30**)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
![Page 22: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/22.jpg)
Path C: student commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Comments rated 2 in Strength (.22*) & Style
(.33**)• Comments rated 3 in Style (.31*)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
![Page 23: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/23.jpg)
Path D: Student & Instructor comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationship ONLY if combine student & instructor comments, & only for comments rated 1 (all categories combined): (.05, p=.06)
![Page 24: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/24.jpg)
Research question 1How do students use peer comments given and received for improving different essays rather than drafts of same essay?
o Very little significant evidence of relationships in Path D
o No difference between comments given & received
![Page 25: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/25.jpg)
Research question 2Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
o No evidence that there is any change over time in path D
o No difference between comments given or received
![Page 26: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/26.jpg)
Research question 3Does the quality of peer comments improve over time?o No evidence of change over time in path
A
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
![Page 27: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/27.jpg)
Research Question 3, cont’d
Student/instructor agreement on average numerical ratings on each essay • tends to go down over time (-.04**)
• student ratings increase at only half the rate (.16*) that instructor’s ratings increase (.33*****)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
![Page 28: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/28.jpg)
Research Question 3, cont’dCorrelations on number of comments, students & instructor• No change in these relationships over
time
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Comment value 1
Comment value 2
Comment value 3
Strength 0.23*Organization 0.21* 0.17*Insight 0.17*Style
![Page 29: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/29.jpg)
Some conclusionsPilot study: feasible for larger sample? Yes, if:
o instructors code essay quality rather than coders
o “chunk” essays for cross-lagged analyseso have easy collection of comments
![Page 30: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/30.jpg)
References• Cho, K., & MacArthur, C. (2010). Student revision with peer and
expert reviewing, Learning and Instruction. 20, 328-338.• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving
comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting
in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426
• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229.
• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.
• Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
![Page 31: Pilot study: Longitudinal analysis of peer feedback in a writing-intensive course](https://reader036.fdocuments.us/reader036/viewer/2022062316/58ec93f81a28ab24298b46a1/html5/thumbnails/31.jpg)
Thank you! Christina Hendricks
University of British Columbia-Vancouver
Website: http://blogs.ubc.ca/christinahendricksBlog: http://blogs.ubc.ca/chendricksTwitter: @clhendricksbc
Slides available: https://is.gd/PeerFeedbackPilot_FOL16
Slides licensed CC-BY 4.0
Capitals needed
underscore