Bryk may 2014 using NICs to tackle practical problems in education
-
Upload
learningemerg -
Category
Education
-
view
189 -
download
0
description
Transcript of Bryk may 2014 using NICs to tackle practical problems in education
Anthony S. Bryk Master Class, University of Bristol
2
Triple Aims of Educational Improvement
EFFICIENCY
EFFECTIVENESS
ENGAGEMENT
Be0er Use of Resources
Ambi<ous Learning For All Students
More Relevance
How We Work Now: Tower of Babel Problem
3
The Educational R&D Problem
• Accelerate Improvement Efforts
• Aim for Quality, Reliably at Scale
4
How We Are Working on This
• Analogical Scavengers—The Gawande Inspiration
• Learning by Doing—Can we actually make the ideas work?
• Engaging a Larger Community
An Inspiration: Improvement Science in Healthcare Protecting 5 Million from Harm,
Saving 100,000 Lives
7
We can accomplish more together, than
even the best of us can do alone.
Complex systems problems that we now seek to solve
Power of Networks
Networked Improvement Communities: What are they?
Integrating Two Big Ideas:
• The discipline of Improvement Science
joined to
• The Power of Networks
Accelera'ng Learning in and through Prac'ce to Improve
Six Principles Guide the Work (plus useful tools to scaffold the activity)
9
Taken Together:
• Disciplined Inquiry
• Rudiments a scientific community
• Aim: systematic practice improvement
I. Problem-‐ & User-‐Centered
• What is the specific problem we’re trying to solve?
• What we tend to do now: a general issue comes into view and we jump on solu<ons
60-‐70% Students assigned to developmental math
course.
80% Percent of these
students that never get past this gate.
500,000 students
in every cohort will never complete college math
requirement.
11
The Problem
A Solu<on Framework: Integrated Pathways
12
Through college-‐level sta5s5cs
“To-and-through” college-level quantitative reasoning
Two 1-year pathways “to and through college math”
1
2
II. Varia5on in Performance is the problem to solve
• “What Works” is typically the wrong ques<on • Real Issue: Quality Improvement Ques<on
“How to advance effec<veness among diverse faculty engaging varied popula<ons of students and working in different organiza<onal contexts?”
• Goal: Achieve efficacy with reliability at scale
Trad
ition
al S
eque
nce
Stat
way
Effects: Time to Complete a College Level Math Course 1 Year 2 Years
Triple the
success rate in half the time.
6%
51%
15%
What is Next?
• Normal Course of Events: “It Works” – Tout success – Publish results – Hope others pick this up – Go onto our next project
Varia<on in Pathways Success Rates by College (n=19)
16
1
2 3
4 5
6 7
8
9
11
12 13
14
15
17
18 19
-50%
0%
50%
100%
0% 50% 100%
Stat
way
Stu
dent
s
Non-Statway Matched Comparisons
No improvement line
We also have a failure, why? What can we learn?
Triple success rate line
III. See the System to Improve it
• Put simply: It is hard to improve what we do not fully understand.
How Do We Heal Medicine? Atul Gawande April, 2012
Gawande’s Closing Observation
Making systems work is the great task of my generation of physicians and scientists. But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.
The Invisible Complexity Schooling
21
The Invisible Complexity of Schooling
60-‐70% Students assigned to developmental math
course.
80% Percent of these
students that never get past this gate.
500,000 students
in every cohort will never complete college math
requirement.
22
Returning to The Presen<ng Problem
The Orien<ng Problem
Extraordinarily high failure rates among students assigned to developmental math instruc<on
Consolidate the courses into a 1-‐year pathway
Real world problems from sta<s<cs as the organizer
Psycho-‐social interven<ons aimed at “produc<ve persistence”
Rapid analy<cs capacity
Faculty development
Causal Systems Analysis: Why do we con<nue to get the outcomes observed?
Primary Causes for High Failure Rates
Organizing Improvement Hypotheses
event
???
Ins$tu$onal*structures*don’t*support*student*success*
Students*are*not*engaged*or*mo$vated*
The*course*material/content*is*problema$c*
Instructors*lack*skills*and*beliefs*that*students*can*succeed*
State*policy*does*not*support*student*success*
Low*success*rates*in*development*
math*func$on*as*a*gatekeeper*to*opportunity*
Students*lack*the*skills*to*succeed*
Ineffec$ve*advising*system*
Ineffec$ve*learning*support*services*
Lack*of*social*$es*to*each*other*and*to*their*instructors*
Inaccurate*placement*
High*dropAout*rates*between*courses*
Don’t*believe*they*can*learn*
Math*and*tes$ng*anxiety*
Poorly*prepared*mathema$cally*
They*don’t*know*how*to*study*math*
They*don’t*know*how*to*“navigate”*the*college*world*
Not*interes$ng*or*relevant**
Not*seen*as*useful*
Text*is*inaccessible*
Does*not*leverage*what*we*know*about*how*students*learn*
Too*many*courses*in*the*developmental*sequence*
Few*opportuni$es*to*learn*from*others*
Lack*knowledge*of*learning*theory*
Weak*pedagogy*
Don’t*believe*suppor$ng**student*success*is*their*job*
Lower*reimbursement*for*developmental*math*
Tradi$onal*transfer*requirements*impede*innova$on*
Arcane*curricular*topics*create*needless*hurdles*
Funding*based*on*enrollment*rather*than*outcomes*
OOen*taught*by*adjunct*faculty**with*liPle*professional*support*
The Orien<ng Problem
Embedded literacy and language barriers
Extraordinarily high failure rates among students assigned to developmental math instruc<on
Lose large # of students at the transi<ons
Consolidate the courses into a 1-‐year pathway
Students mindsets undermine success
Real world problems from sta<s<cs as the organizer
Students “gone” before we know it
Psycho-‐social interven<ons aimed at “produc<ve persistence”
Rapid analy<cs capacity
Course material and instruc<on are not engaging
Faculty development
Analy<c Summary of Causal Systems Analysis
Primary Causes for High Failure Rates
Organizing Improvement Hypotheses
Eventually leads to a “Pathways Strategy”
Pathways Driver Diagram: Organizing a Networked Improvement Community
Aim: increase from 5% to 50%, students achieving college math credit within one year of continuous enrollment
Instructional System: Organized around productive struggle, explicit connections, and
deliberate practice.
Productive Persistence:
Students develop skills and maintain positive mindsets
Language and Literacy: Students
use language in understanding
problems, reason mathematically, and communicate results
Advancing Teaching: Effective
teaching within 2 years of
implementation
Reduce transitions + assure enrollment across semesters
Deliberate focus on “Starting Strong”
Promote students’ ties to peers, faculty,
pathway
Math that matters: students see material interesting, relevant
Enhance faculty’s beliefs and relational
practices
Opening lessons engage interest, assure early
success
Direct interventions to influence student mindsets
Real-time data tracking on student engagement
Detail supportive classroom norms and social
connections
Professional development on “Starting Strong”
A Community Explicates its Causal Thinking:
A Community Explicates its Causal Thinking:
A Driver Diagram
to Organize Its Major Improvement Hypotheses
Pathways Driver Diagram: Organizing a Networked Improvement Community
Aim: increase from 5% to 50%, students achieving college math credit within one year of continuous enrollment
Instructional System: Organized around productive struggle, explicit connections, and
deliberate practice.
Productive Persistence:
Students develop and maintain
positive mindsets
Language and Literacy: Students
use language in understanding
problems, reason mathematically, and communicate results
Advancing Teaching: Effective teaching within 2
years of implementation
Reduce transitions + assure enrollment across semesters
Deliberate focus on “Starting Strong”
Promote students’ ties to peers, faculty,
pathway
Math that matters: students see material interesting, relevant
Enhance faculty’s beliefs and relational
practices
Opening lessons engage interest, assure early
success
Direct interventions to influence student mindsets
Real-time data tracking on student engagement
Detail supportive classroom norms and social
connections
Professional development on “Starting Strong”
Elabora<ng Out The Driver Diagram
Produc<ve Persistence
IV. You cannot improve at scale what you cannot measure
• Measureable targets: “Some is not a number; soon is not a <me”-‐-‐Valued outcome measures – But, you just can not stand at the end of the line.
• We need process measures <ed to intermediate targets.
Produc<ve Persistence
Suppor<ve social rela<onships
Target: How do we measure it?
Mindsets about the value of math
Mindsets about poten<al to learn
math
Anxiety Regula<on
Study Skills Conceptual Task: reduce to 5 core ideas focus on underlying malleable causes + change evidence
Prac5cal Measurement:
reduce 900 items to 26 “you have 3 minutes”
V. Accelerate Improvement: Embrace Disciplined Inquiry
• Policy Romance of the Silver Bullet – Move quickly to large scale implementa<on, but…
• We typically don’t know whether: – We can make these ideas work at all; – We have capacity and will to execute with efficacy at
scale.
• Instead, a DEED orienta<on – Quick, minimally intrusive, an empirical warrant – Mantra: Learn Fast, Fail Fast, Improve Fast!
A System of Social Learning to Improve
Transla5onal Research
Interven5ons (Alpha Labs)
Will they work for community college students, and if so,
how?
Expert Prac55oner Knowledge (Subnet)
Building robust clinical
knowledge about effec<ve materials and instruc<onal prac<ces.
Learning from Network Data (Hub Analy5cs)
Learning from observed variability.
Discerning the unseen.
32
Transla5onal Research
Interven5ons (Alpha Labs)
• Will they work for community college students, and if so,
how?
Initial Alpha Lab: Mindset Intervention
• A carefully designed experimental intervention has changed student mindsets.
• But just because an intervention can work in one setting does not mean it will work in another.
• Need to engineer it to “fit” in instructional contexts. – Conduct rapid R&D using DEED methodology. – “Smell testing” – 4 months from small-scale test to larger scale use.
Rapid Iterative DEED cycles
• Research-Practitioner Team
• Testing – Small double-blind randomized
trial in Algebra course (n = 26)
– Larger double-blind experiment (n = 288)
• Introduce to faculty network, carefully study emerging results, continue to revise, refine, and extend.
34
Roberta Carew, Statway faculty
Valencia College
35
Learning from Network Data (Hub Analy5cs)
• Learning from observed
variability. Discerning the “unseen.”
36
!
1. Assessing Change: Initial Evidence of Efficacy of Starting Strong Package
2. Predictive Analytics—targeting support
(a simple at-risk indicator scoring 5 key items/item clusters-day 1)
37
% of who failed the end-‐of-‐term common assessment
Connections to Stereotype Threat
12% 13% 14%
28%
40%
7% 11% 14%
50%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Never Hardly Ever Some<mes Frequently Always
Pathways Drop
out
All students Black students
“How oqen, if ever, do you wonder: ‘Maybe I don't belong here?’”
N = 714 math students
39
Expert Prac55oner Knowledge (Subnet)
Building robust clinical
knowledge about effec<ve instruc<onal materials and prac<ces.
PDSA Cycle: Rapid, Small Experimental Trials
PLAN DO
ACT STUDY
The Three Ques5ons: • What specifically are we trying to accomplish?
• What change might we introduce?
• How will we know that the changes are an improvement?
Improving Instructional Routines in Support of Productive Persistence: PDSA Cycles
• Faculty routines and email scripts re: absent students
• Student group noticing routine
• Effective scaffolding for group roles (rich problems)
41
Sample Run Chart for a PDSA Cycle (Student Group Noticing Routine)
0%#
20%#
40%#
60%#
80%#
100%#
120%#
1/14/1
3#
1/21/1
3#
1/28/1
3#
2/4/13
#
2/11/1
3#
2/18/1
3#
2/25/1
3#
3/4/13
#
3/11/1
3#
3/18/1
3#
3/25/1
3#
4/1/13
#
4/8/13
#
4/15/1
3#
4/22/1
3#
4/29/1
3#
Perce
nt'of
'Stud
ent'in'A
/end
ance'
A/endance'(By'Day)'
Typical#a4endance#
Observed#a4endance#
n=44#Median:#0.85#
Median#
Developing a Quality Process Reliably at Scale
Develop A Change
Test under mul<ple condi<ons
Test under increasingly varied
condi<ons
Make the change permanent
Ini5al Hunches
System Changes
1 school 1 administrator
5 schools Many administrators
En<re ver<cal team A more diverse group of administrators
District Wide All administrators
Seeing Task Complexity
Seeing Organiza<onal Complexity
Learning to improve feedback conversa<ons between
principals and new teachers PLAN DO
ACT STUDY
A Developmental Dynamic
Hunches Theories Ideas
Ini<a<ng Resources
P D
S A
P D
S A
P D
S A P D
S A
Moving out toward More diverse condi<ons: “factor of 5 rule of thumb”
Aiming for Efficacy with Reliability at
Scale
VI. Accelerate Improvement: Tap the Power of Networks
• A source of innova<on – Dig into the details: what worked, how, for whom? – Can we adap<vely integrate this into other contexts?
• Mul<ple fast replica<on – Can we make this happen with efficacy, reliably at scale?
• Innova<on diffusion—it is largely about who is connected to whom and what they think and do
A Learning Educa'onal System
A A
Improvement Networks: Accelerate Learning in Prac<ce for Improvement
A
B
A A A B
A A A B
A A A B
C
(Englebart,1994)
It is all about accelerating how we learn in and through practice to improve.