IMAGINE THE POSSIBILITIES

PEG BALACHOWSKIEVERETT COMMUNITY COLLEGE

USING IMPROVEMENT SCIENCE TO IMPROVE STUDENT OUTCOMES

“There is no blueprint. We are inventing this together as we go along. “

Anthony Bryk

• Thanks to Lawrence Morales, Associate for Improvement Science and Jane Muhich, Managing Director for Community College Programs, Director of Productive Persistence for the use of their work.

got persistence?

Outline• Improvement Science •The Model for Improvement•Tools in Improvement Science

•Driver Diagrams•PDSA Cycles•Run Charts

6

Carnegie’s Networked Improvement Community

ImprovementScience

Language & Literacy

Productive Persistence

Advancing TeachingAnalytics

Carnegie’s Pathways• Addressing the Developmental Math challenge with a Pathways approach:• Create a high quality Instructional System• Attend to Language and Literacy issues• Advance Quality Teaching• Sustain a Networked Improvement Community

• Promote Productive Persistence

Improvement Science• A field designed to provide a framework for research that is focused on improvement. • Poses questions about a change being tested• Proposes predictions about how the test will go

• Uses a testing mechanism to answer questions and verify predictions

• Collects data to support conclusions• Analyzes data to inform later iterations of tests

• Langley, G. J., et al (2009). The improvement guide: a practical approach to enhancing organizational performance (2nd ed.). San Francisco: Jossey-Bass.

The Model for Improvement (MFI)The Aim

The Measures

The “Change Ideas”

• Three Questions for Guiding Work–What are we trying to accomplish?

–How will we know that a change is an improvement?

–What changes can we make that will result in an improvement?

• One Method for Testing Change–The Plan-Do-Study-Act Cycle (PDSAs)

Tools in Improvement Science

1. Driver Diagrams

2. PDSA Cycles

3. Run Charts

Productive Persistence

•The tenacity plus good strategies that math students need in order to be successful.

14

Tool 1: The Driver Diagram

AimWhat?Who?

By When?Measures:1. 2. 3.

AIMWhat

are we trying to

do?

MEASURESHow will we

know a change is an improvement

?

15

The Driver Diagram

AimWhat?Who?

By When?Measures:1. 2. 3.

Driver 1

Driver 2

Driver 3

Primary Secondary

Driver A

Driver B

Driver C

Driver D

Driver EMEASURESHow will we

know a change is an improvement

?

AIMWhat

are we trying to

do?

16

The Driver Diagram

AimWhat?Who?

By When?Measures:1. 2. 3.

Driver 1

Driver 2

Driver 3

Primary Secondary

Driver A

Driver B

Driver C

Driver D

Driver EMEASURESHow will we

know a change is an improvement

?

Change Ideas

Change IdeaChange IdeaChange IdeaChange IdeaChange IdeaChange IdeaChange Idea

CHANGEWhat

changes can we test?AIM

What are we

trying to do?

Skills & Habits 

Mindsets

Value 

Social Ties

Faculty

Primary Drivers  Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Aim:Students

continue to put forth

effort during challenges and when they do so they use effective

strategies.

Students have skills, habits and know-how to succeed

in college setting. 

Students believe they are capable of learning math.

Students believe the course has value.

 Students feel socially tied to

peers, faculty, and the course.

 

Faculty and college support students’ skills and

mindsets.

Aim:Students

continue to put forth

effort during challenges and when they do so they use effective

strategies.

Primary Drivers 

Skills and Habits 

Mindsets

Value

 Social Ties

Faculty

Students have skills, habits and know-how to succeed

in college setting. 

Students believe they are capable of learning math.

Students believe the course has value.

 Students feel socially tied to

peers, faculty, and the course.

 

Faculty and college support students’ skills and

mindsets.

Aim:Students

continue to put forth

effort during challenges and when they do so they use effective

strategies.

Primary Drivers  Secondary DriversHave accurate knowledge about succeeding in the course and navigating The Health Care Data Guide: Learning from Data for Improvement institution.

Use learning strategies that are appropriate for the academic challenge they are facing.

Have strategies for regulating anxiety.

Have the know-how and self-discipline to set and prioritize long and short-term goals over short-term desires and distractions

Believe they can actively grow their math ability with effort, help, and good strategies.

View math success as something “people like them” do, and not something “other people” do.

See that math isn’t just a set of algorithms to be memorized but a connected set of concepts that can be understood and applied.

Students believe the knowledge from the course is relevant to a personal or socially-valued goal.

Students feel as though they are completing academic tasks for personal reasons.

 

Students see how completion of this course is relevant to goals for degree/certificate completion.

Students feel comfortable asking questions

Students do not feel stigmatized due to membership in a negatively stereotyped group.

Faculty believe students can succeed if they develop more productive skills and mindsets.  

Faculty integrate PP principles in how they talk to students and in the curriculum they assign.

Students feel that the professor cares that they, personally, succeed in the course and in college.

Faculty see helping their students to productively persist as part their role as an instructor.

Faculty know how to promote productive skills and mindsets.  

Students do not question whether they belong.

Possible Measures:Attendance

Time on taskStrategy use Help-seekingRevising work

Challenge-seeking

Skills & Habits 

Mindsets

Value 

Social Ties

Faculty

Primary Drivers  Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Secondary Drivers

Aim:Students

continue to put forth

effort during challenges and when they do so they use effective

strategies.

Social Ties

Secondary Drivers

Students feel the professor cares that

they succeed

Primary Driver 

Students feel comfortable asking

questions

Students do not doubt their belonging

in the course

Change Ideas

Social Ties

Secondary Drivers

Students feel the professor cares that

they succeed

Primary Driver 

Students feel comfortable asking

questions

Students do not doubt their belonging

in the course

Change Ideas

• Email routines• Use of names routines

• Your ideas?

Social Ties

Secondary Drivers

Students feel the professor cares that

they succeed

Primary Driver 

Students feel comfortable asking

questions

Students do not doubt their belonging

in the course

Change Ideas

• Email routines• Use of names routines

• Your ideas?

• Answering routines• Data collection routine

• Your ideas?

Social Ties

Secondary Drivers

Students feel the professor cares that

they succeed

Primary Driver 

Students feel comfortable asking

questions

Students do not doubt their belonging

in the course

Change Ideas

• Email routines• Use of names routines

• Your ideas?

• Answering routines• Data collection routine

• Your ideas?

• Group role routine• Group noticing routine

• Your ideas?

CHANGEHow do we know

these work?

Tool 2: PDSA Cycles

• “A way to turn change ideas into action and connect action to learning.”*

• A disciplined way to test ideas.

• Avoids implementing too soon.

*Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd edition). San Francisco: Jossey-Bass Publishers; 2009.

PDSA Cycles

• “A way to turn change ideas into action and connect action to learning.”

PLAN• What’s the change?

• What’s your prediction?

• Plan to conduct test DO• Execute

test• Collect

data, document observations

STUDY• Compare results to prediction

• What did you learn?

ACT• Next

steps: Adapt, adopt, abandon

CHANGEWhat

changes can we test?

PDSA Cycles in Productive Persistence

Aim: To measure student sense of belonging, develop an efficient way to gather data on the number of students asking questions

• Cycle 1: Feedback from ONE student.• Cycle 2: Test with ONE student in class.• Cycle 3: Test with MORE students in class.• Cycles 4-6: Further testing with more students, more than

one student at a time, and regular implementation.• Cycle 7: Inform whole class more fully.• Next Step?

28

References

Provost, L. P., & Murray, S. K. (2011). The health care data guide learning from data for improvement. San Francisco, CA: Jossey-Bass.

Tool 3: Run Charts• A graphical display of data to allow you to track outcomes over time.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Median

Goal

Student Services Satisfaction

Date

Aver

age

Scor

e (O

ut o

f 5)

Run Chart Scenario• Around the 4th week of the term, an instructor is

noticing a lot of students are missing class. So she brainstorms and eventually has an idea for how to reduce absences. Her plan is to implement her idea in Week 8 of the semester and then compare data from Week 4 and Week 11 of the semester to see if her idea worked.

• Here are her data….

Week 4Data

Week 8Implement

Week 11Data

• Questions to Ponder:• Did this change lead to an improvement?• What questions do you have about the

data?• What other factors might be at play?

Before Change After Change02468

108

3

Attendance

# of

Abs

ent S

tude

nts

Week 8Implementatio

n

32

Group Activity 1) Look at the following 5 Run Charts:Note: Each one has the same data for Weeks 4 & 11 as the bar chart.

2) Decide what conclusions you can make for each case:Did the change lead to improvement? Why or why not?

33

Case 1

1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789

108

3

Attendance

Week #

# of

Abs

ent S

tude

nts

Conclusion?

Week 4Data

Week 8Implement

Week 11Data

34

Case 2

Conclusion?

1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789 8

3

Attendance

Week #

# of

Abs

ent S

tude

nts

35

Case 3

Conclusion?

1 2 3 4 5 6 7 8 9 10 11 12 13 140123456789

108

3

Attendance

Week #

# of

Abs

ent S

tude

nts

36

Case 4

Conclusion?

1 2 3 4 5 6 7 8 9 10 11 12 13 140

2

4

6

8

10

12

8

3

Attendance

Week #

# of

Abs

ent S

tude

nts

37

Case 5

Conclusion?

1 2 3 4 5 6 7 8 9 10 11 12 13 140

2

4

6

8

10

12

8

3

Attendance

Week #

# of

Abs

ent S

tude

nts

How do we know there’s improvement?

In all of these Cases, we need to consider the question:

“When do the data give us a signal that we have improved on our outcome measure?”

Case 5 Revisited

1 2 3 4 5 6 7 8 9 10 11 12 13 140

2

4

6

8

10

12

8

3

Attendance

Week #

# of

Abs

ent S

tude

nts

Does this count as an

improvement?

40

Run Chart Rules for Signals

Shift: Six or more consecutive points all above or all below the median. (Skip points on the median.)

Trend: Five or more consecutive points all going up (increasing) or all going down (decreasing).

Astronomical Point: An obvious value different than others in the data set.

41

Example 1: Are there signals?

12/30/1899

9/2/1902

5/2/1905

1/2/1908

9/2/1910

5/2/1913

1/2/1916

9/2/1918

5/2/1921

1/2/1924

9/2/1926

5/2/1929

1/2/1932

9/2/1934

5/2/1937

1/2/1940

9/2/1942

5/2/1945

1/2/1948

9/2/1950

5/2/1953

1/2/1956

9/2/1958

5/2/1961

1/2/1964

9/2/1966

5/2/1969

1/2/1972

9/2/1974

5/2/1977

1/2/1980

9/2/1982

5/2/1985

1/2/1988

9/2/1990

5/2/1993

1/2/1996

9/2/1998

5/2/2001

1/2/2004

9/2/2006

5/2/2009

1/2/2012

40%

50%

60%

70%

80%

90%

100%

0.9

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

e

Note: 3 of 4 of the following examples

contain real data and the y-axis units have

changed.

12/30/1899

9/2/1902

5/2/1905

1/2/1908

9/2/1910

5/2/1913

1/2/1916

9/2/1918

5/2/1921

1/2/1924

9/2/1926

5/2/1929

1/2/1932

9/2/1934

5/2/1937

1/2/1940

9/2/1942

5/2/1945

1/2/1948

9/2/1950

5/2/1953

1/2/1956

9/2/1958

5/2/1961

1/2/1964

9/2/1966

5/2/1969

1/2/1972

9/2/1974

5/2/1977

1/2/1980

9/2/1982

5/2/1985

1/2/1988

9/2/1990

5/2/1993

1/2/1996

9/2/1998

5/2/2001

1/2/2004

9/2/2006

5/2/2009

1/2/2012

40%

50%

60%

70%

80%

90%

100%

0.9

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

e

• Shifts?• Trends?• Astronomical Points?

Example 1: Analysis

• None• None• None• No Signals

60%

65%

70%

75%

80%

85%

90%

95%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

eExample 2

60%

65%

70%

75%

80%

85%

90%

95%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

eExample 2

8/27/2012 9/3/2012 9/10/2012 9/17/2012 9/24/2012 10/1/2012 10/8/2012 10/15/2012 10/22/201240%

50%

60%

70%

80%

90%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

e

Median = 0.83

Example 3

8/27/2012 9/3/2012 9/10/2012 9/17/2012 9/24/2012 10/1/2012 10/8/2012 10/15/2012 10/22/201240%

50%

60%

70%

80%

90%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

e

Median = 0.83

Example 3

12/30/99

5/2/03

9/2/06

1/2/10

5/2/13

9/2/16

1/2/20

5/2/23

9/2/26

1/2/30

5/2/33

9/2/36

1/2/40

5/2/43

9/2/46

1/2/50

5/2/53

9/2/56

1/2/60

5/2/63

9/2/66

1/2/70

5/2/73

9/2/76

1/2/80

5/2/83

9/2/86

1/2/90

5/2/93

9/2/96

1/2/00

5/2/03

9/2/06

1/2/10

40%

50%

60%

70%

80%

90%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

e

Median = 0.78

Example 4

Median = 0.78

12/30/99

5/2/03

9/2/06

1/2/10

5/2/13

9/2/16

1/2/20

5/2/23

9/2/26

1/2/30

5/2/33

9/2/36

1/2/40

5/2/43

9/2/46

1/2/50

5/2/53

9/2/56

1/2/60

5/2/63

9/2/66

1/2/70

5/2/73

9/2/76

1/2/80

5/2/83

9/2/86

1/2/90

5/2/93

9/2/96

1/2/00

5/2/03

9/2/06

1/2/10

40%

50%

60%

70%

80%

90%

100%

Attendance (By Day)

Perc

ent o

f Stu

dent

in A

tten

danc

eExample 4

49

Reflections from users• “From hearing the group discussions in the last couple

modules—they were teaching themselves. The amount that I had to come to the group[s] and make sure they were on task went down. They were becoming self-learners…. It was wonderful to see.”

- Productive Persistence Faculty

• “It helped me focus on what was going on with the students, particularly on who was engaged in class. I enjoyed doing the Improvement work and I feel like it really helped me focus on teaching in a different way than I did before.”

- Productive Persistence Faculty

Questions?

•Thanks for your time!•Peg Balachowski•mbalachowski@everettcc.edu