Imagine the possibilities Peg Balachowski Everett Community College
50
IMAGINE THE POSSIBILITIES PEG BALACHOWSKI EVERETT COMMUNITY COLLEGE
description
Imagine the possibilities Peg Balachowski Everett Community College. Using Improvement Science to Improve Student Outcomes. “There is no blueprint. We are inventing this together as we go along. “ Anthony Bryk. - PowerPoint PPT Presentation
Transcript of Imagine the possibilities Peg Balachowski Everett Community College
IMAGINE THE POSSIBILITIES
PEG BALACHOWSKIEVERETT COMMUNITY COLLEGE
USING IMPROVEMENT SCIENCE TO IMPROVE STUDENT OUTCOMES
“There is no blueprint. We are inventing this together as we go along. “
Anthony Bryk
• Thanks to Lawrence Morales, Associate for Improvement Science and Jane Muhich, Managing Director for Community College Programs, Director of Productive Persistence for the use of their work.
got persistence?
Outline• Improvement Science •The Model for Improvement•Tools in Improvement Science
•Driver Diagrams•PDSA Cycles•Run Charts
6
Carnegie’s Networked Improvement Community
ImprovementScience
Language & Literacy
Productive Persistence
Advancing TeachingAnalytics
Carnegie’s Pathways• Addressing the Developmental Math challenge with a Pathways approach:• Create a high quality Instructional System• Attend to Language and Literacy issues• Advance Quality Teaching• Sustain a Networked Improvement Community
• Promote Productive Persistence
Improvement Science• A field designed to provide a framework for research that is focused on improvement. • Poses questions about a change being tested• Proposes predictions about how the test will go
• Uses a testing mechanism to answer questions and verify predictions
• Collects data to support conclusions• Analyzes data to inform later iterations of tests
• Langley, G. J., et al (2009). The improvement guide: a practical approach to enhancing organizational performance (2nd ed.). San Francisco: Jossey-Bass.
The Model for Improvement (MFI)The Aim
The Measures
The “Change Ideas”
• Three Questions for Guiding Work–What are we trying to accomplish?
–How will we know that a change is an improvement?
–What changes can we make that will result in an improvement?
• One Method for Testing Change–The Plan-Do-Study-Act Cycle (PDSAs)
Tools in Improvement Science
1. Driver Diagrams
2. PDSA Cycles
3. Run Charts
Productive Persistence
•The tenacity plus good strategies that math students need in order to be successful.
14
Tool 1: The Driver Diagram
AimWhat?Who?
By When?Measures:1. 2. 3.
AIMWhat
are we trying to
do?
MEASURESHow will we
know a change is an improvement
?
15
The Driver Diagram
AimWhat?Who?
By When?Measures:1. 2. 3.
Driver 1
Driver 2
Driver 3
Primary Secondary
Driver A
Driver B
Driver C
Driver D
Driver EMEASURESHow will we
know a change is an improvement
?
AIMWhat
are we trying to
do?
16
The Driver Diagram
AimWhat?Who?
By When?Measures:1. 2. 3.
Driver 1
Driver 2
Driver 3
Primary Secondary
Driver A
Driver B
Driver C
Driver D
Driver EMEASURESHow will we
know a change is an improvement
?
Change Ideas
Change IdeaChange IdeaChange IdeaChange IdeaChange IdeaChange IdeaChange Idea
CHANGEWhat
changes can we test?AIM
What are we
trying to do?
Skills & Habits
Mindsets
Value
Social Ties
Faculty
Primary Drivers Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Aim:Students
continue to put forth
effort during challenges and when they do so they use effective
strategies.
Students have skills, habits and know-how to succeed
in college setting.
Students believe they are capable of learning math.
Students believe the course has value.
Students feel socially tied to
peers, faculty, and the course.
Faculty and college support students’ skills and
mindsets.
Aim:Students
continue to put forth
effort during challenges and when they do so they use effective
strategies.
Primary Drivers
Skills and Habits
Mindsets
Value
Social Ties
Faculty
Students have skills, habits and know-how to succeed
in college setting.
Students believe they are capable of learning math.
Students believe the course has value.
Students feel socially tied to
peers, faculty, and the course.
Faculty and college support students’ skills and
mindsets.
Aim:Students
continue to put forth
effort during challenges and when they do so they use effective
strategies.
Primary Drivers Secondary DriversHave accurate knowledge about succeeding in the course and navigating The Health Care Data Guide: Learning from Data for Improvement institution.
Use learning strategies that are appropriate for the academic challenge they are facing.
Have strategies for regulating anxiety.
Have the know-how and self-discipline to set and prioritize long and short-term goals over short-term desires and distractions
Believe they can actively grow their math ability with effort, help, and good strategies.
View math success as something “people like them” do, and not something “other people” do.
See that math isn’t just a set of algorithms to be memorized but a connected set of concepts that can be understood and applied.
Students believe the knowledge from the course is relevant to a personal or socially-valued goal.
Students feel as though they are completing academic tasks for personal reasons.
Students see how completion of this course is relevant to goals for degree/certificate completion.
Students feel comfortable asking questions
Students do not feel stigmatized due to membership in a negatively stereotyped group.
Faculty believe students can succeed if they develop more productive skills and mindsets.
Faculty integrate PP principles in how they talk to students and in the curriculum they assign.
Students feel that the professor cares that they, personally, succeed in the course and in college.
Faculty see helping their students to productively persist as part their role as an instructor.
Faculty know how to promote productive skills and mindsets.
Students do not question whether they belong.
Possible Measures:Attendance
Time on taskStrategy use Help-seekingRevising work
Challenge-seeking
Skills & Habits
Mindsets
Value
Social Ties
Faculty
Primary Drivers Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Secondary Drivers
Aim:Students
continue to put forth
effort during challenges and when they do so they use effective
strategies.
Social Ties
Secondary Drivers
Students feel the professor cares that
they succeed
Primary Driver
Students feel comfortable asking
questions
Students do not doubt their belonging
in the course
Change Ideas
Social Ties
Secondary Drivers
Students feel the professor cares that
they succeed
Primary Driver
Students feel comfortable asking
questions
Students do not doubt their belonging
in the course
Change Ideas
• Email routines• Use of names routines
• Your ideas?
Social Ties
Secondary Drivers
Students feel the professor cares that
they succeed
Primary Driver
Students feel comfortable asking
questions
Students do not doubt their belonging
in the course
Change Ideas
• Email routines• Use of names routines
• Your ideas?
• Answering routines• Data collection routine
• Your ideas?
Social Ties
Secondary Drivers
Students feel the professor cares that
they succeed
Primary Driver
Students feel comfortable asking
questions
Students do not doubt their belonging
in the course
Change Ideas
• Email routines• Use of names routines
• Your ideas?
• Answering routines• Data collection routine
• Your ideas?
• Group role routine• Group noticing routine
• Your ideas?
CHANGEHow do we know
these work?
Tool 2: PDSA Cycles
• “A way to turn change ideas into action and connect action to learning.”*
• A disciplined way to test ideas.
• Avoids implementing too soon.
*Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd edition). San Francisco: Jossey-Bass Publishers; 2009.
PDSA Cycles
• “A way to turn change ideas into action and connect action to learning.”
PLAN• What’s the change?
• What’s your prediction?
• Plan to conduct test DO• Execute
test• Collect
data, document observations
STUDY• Compare results to prediction
• What did you learn?
ACT• Next
steps: Adapt, adopt, abandon
CHANGEWhat
changes can we test?
PDSA Cycles in Productive Persistence
Aim: To measure student sense of belonging, develop an efficient way to gather data on the number of students asking questions
• Cycle 1: Feedback from ONE student.• Cycle 2: Test with ONE student in class.• Cycle 3: Test with MORE students in class.• Cycles 4-6: Further testing with more students, more than
one student at a time, and regular implementation.• Cycle 7: Inform whole class more fully.• Next Step?
28
References
Provost, L. P., & Murray, S. K. (2011). The health care data guide learning from data for improvement. San Francisco, CA: Jossey-Bass.
Tool 3: Run Charts• A graphical display of data to allow you to track outcomes over time.
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Median
Goal
Student Services Satisfaction
Date
Aver
age
Scor
e (O
ut o
f 5)
Run Chart Scenario• Around the 4th week of the term, an instructor is
noticing a lot of students are missing class. So she brainstorms and eventually has an idea for how to reduce absences. Her plan is to implement her idea in Week 8 of the semester and then compare data from Week 4 and Week 11 of the semester to see if her idea worked.
• Here are her data….
Week 4Data
Week 8Implement
Week 11Data
• Questions to Ponder:• Did this change lead to an improvement?• What questions do you have about the
data?• What other factors might be at play?
Before Change After Change02468
108
3
Attendance
# of
Abs
ent S
tude
nts
Week 8Implementatio
n
32
Group Activity 1) Look at the following 5 Run Charts:Note: Each one has the same data for Weeks 4 & 11 as the bar chart.
2) Decide what conclusions you can make for each case:Did the change lead to improvement? Why or why not?
33
Case 1
1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789
108
3
Attendance
Week #
# of
Abs
ent S
tude
nts
Conclusion?
Week 4Data
Week 8Implement
Week 11Data
34
Case 2
Conclusion?
1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789 8
3
Attendance
Week #
# of
Abs
ent S
tude
nts
35
Case 3
Conclusion?
1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789
108
3
Attendance
Week #
# of
Abs
ent S
tude
nts
36
Case 4
Conclusion?
1 2 3 4 5 6 7 8 9 10 11 12 13 140
2
4
6
8
10
12
8
3
Attendance
Week #
# of
Abs
ent S
tude
nts
37
Case 5
Conclusion?
1 2 3 4 5 6 7 8 9 10 11 12 13 140
2
4
6
8
10
12
8
3
Attendance
Week #
# of
Abs
ent S
tude
nts
How do we know there’s improvement?
In all of these Cases, we need to consider the question:
“When do the data give us a signal that we have improved on our outcome measure?”
Case 5 Revisited
1 2 3 4 5 6 7 8 9 10 11 12 13 140
2
4
6
8
10
12
8
3
Attendance
Week #
# of
Abs
ent S
tude
nts
Does this count as an
improvement?
40
Run Chart Rules for Signals
Shift: Six or more consecutive points all above or all below the median. (Skip points on the median.)
Trend: Five or more consecutive points all going up (increasing) or all going down (decreasing).
Astronomical Point: An obvious value different than others in the data set.
41
Example 1: Are there signals?
12/30/1899
9/2/1902
5/2/1905
1/2/1908
9/2/1910
5/2/1913
1/2/1916
9/2/1918
5/2/1921
1/2/1924
9/2/1926
5/2/1929
1/2/1932
9/2/1934
5/2/1937
1/2/1940
9/2/1942
5/2/1945
1/2/1948
9/2/1950
5/2/1953
1/2/1956
9/2/1958
5/2/1961
1/2/1964
9/2/1966
5/2/1969
1/2/1972
9/2/1974
5/2/1977
1/2/1980
9/2/1982
5/2/1985
1/2/1988
9/2/1990
5/2/1993
1/2/1996
9/2/1998
5/2/2001
1/2/2004
9/2/2006
5/2/2009
1/2/2012
40%
50%
60%
70%
80%
90%
100%
0.9
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
e
Note: 3 of 4 of the following examples
contain real data and the y-axis units have
changed.
12/30/1899
9/2/1902
5/2/1905
1/2/1908
9/2/1910
5/2/1913
1/2/1916
9/2/1918
5/2/1921
1/2/1924
9/2/1926
5/2/1929
1/2/1932
9/2/1934
5/2/1937
1/2/1940
9/2/1942
5/2/1945
1/2/1948
9/2/1950
5/2/1953
1/2/1956
9/2/1958
5/2/1961
1/2/1964
9/2/1966
5/2/1969
1/2/1972
9/2/1974
5/2/1977
1/2/1980
9/2/1982
5/2/1985
1/2/1988
9/2/1990
5/2/1993
1/2/1996
9/2/1998
5/2/2001
1/2/2004
9/2/2006
5/2/2009
1/2/2012
40%
50%
60%
70%
80%
90%
100%
0.9
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
e
• Shifts?• Trends?• Astronomical Points?
Example 1: Analysis
• None• None• None• No Signals
60%
65%
70%
75%
80%
85%
90%
95%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
eExample 2
60%
65%
70%
75%
80%
85%
90%
95%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
eExample 2
8/27/2012 9/3/2012 9/10/2012 9/17/2012 9/24/2012 10/1/2012 10/8/2012 10/15/2012 10/22/201240%
50%
60%
70%
80%
90%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
e
Median = 0.83
Example 3
8/27/2012 9/3/2012 9/10/2012 9/17/2012 9/24/2012 10/1/2012 10/8/2012 10/15/2012 10/22/201240%
50%
60%
70%
80%
90%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
e
Median = 0.83
Example 3
12/30/99
5/2/03
9/2/06
1/2/10
5/2/13
9/2/16
1/2/20
5/2/23
9/2/26
1/2/30
5/2/33
9/2/36
1/2/40
5/2/43
9/2/46
1/2/50
5/2/53
9/2/56
1/2/60
5/2/63
9/2/66
1/2/70
5/2/73
9/2/76
1/2/80
5/2/83
9/2/86
1/2/90
5/2/93
9/2/96
1/2/00
5/2/03
9/2/06
1/2/10
40%
50%
60%
70%
80%
90%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
e
Median = 0.78
Example 4
Median = 0.78
12/30/99
5/2/03
9/2/06
1/2/10
5/2/13
9/2/16
1/2/20
5/2/23
9/2/26
1/2/30
5/2/33
9/2/36
1/2/40
5/2/43
9/2/46
1/2/50
5/2/53
9/2/56
1/2/60
5/2/63
9/2/66
1/2/70
5/2/73
9/2/76
1/2/80
5/2/83
9/2/86
1/2/90
5/2/93
9/2/96
1/2/00
5/2/03
9/2/06
1/2/10
40%
50%
60%
70%
80%
90%
100%
Attendance (By Day)
Perc
ent o
f Stu
dent
in A
tten
danc
eExample 4
49
Reflections from users• “From hearing the group discussions in the last couple
modules—they were teaching themselves. The amount that I had to come to the group[s] and make sure they were on task went down. They were becoming self-learners…. It was wonderful to see.”
- Productive Persistence Faculty
• “It helped me focus on what was going on with the students, particularly on who was engaged in class. I enjoyed doing the Improvement work and I feel like it really helped me focus on teaching in a different way than I did before.”
- Productive Persistence Faculty
