A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two...
Transcript of A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two...
![Page 1: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/1.jpg)
1
The Ninth Annual International
Campbell Collaborative Colloquium
A Systematic Approach to Data-based Decision Making in Education
Randy Keyworth
Jack States
President Barack Obama
March 9, 2009
Promoting science…
is about ensuring that scientific
data is never distorted or
concealed to serve a political
agenda --and that we make
scientific decisions based on
facts,
not ideology.not ideology.
“
”
![Page 2: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/2.jpg)
2
Arne DuncanArne Duncan
Secretary of EducationSecretary of EducationWall Street Journal OpinionWall Street Journal Opinion
April 22, 2009April 22, 2009
“Without the data, we cannot even have the conversation, let alone discuss solutions.”
“We need solid, unimpeachable
information that identifies what's working
and what's not working in our schools.”
DOE DataDOE Data--Based Decision Making InitiativesBased Decision Making Initiatives
1. Improve quality of standardized tests and raise standards that are internationally benchmarked (national academic
standards)
![Page 3: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/3.jpg)
3
Comparison of State AYP RequirementsComparison of State AYP Requirements
QuickTime™ and aBMP decompressor
are needed to see this picture.
THOMAS B. FORDHAM I NSTITUTE (2008)
DOE DataDOE Data--Based Decision Making InitiativesBased Decision Making Initiatives
1. Improve quality of standardized tests and raise standards that are internationally benchmarked (national academic standards)
2. Build robust data systems to aggregate and disaggregate student outcome data
link teachers to students outcomes -- to distinguishbetween effective and ineffective teachers
to link effective teachers to their colleges of education -- todistinguish between effective and ineffective programs
3. Invest heavily in teacher and principal quality initiatives (including performance pay for effective teachers)
![Page 4: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/4.jpg)
4
TodayToday’’s Schedules Schedule
The Role of Data-Based Decision Making in Evidence-Based Education States
Critical Assumptions in Date-based Decision Making Keyworth
Types of Data States
Building a Data-Based Decision Making Culture Through Performance Management Keyworth
Key Indicator Systems: Why We Need to Systematize Decision Making States
30 years studying “research to practice” issues…
from the “practice” side
![Page 5: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/5.jpg)
5
The Wing InstituteThe Wing Institute
1978 1978 -- 20042004
operated "research based" special education services in
“real-world” settings…
…provided a “laboratory” setting for longitudinal study of
research to practice, implementation and sustainability
The Wing InstituteThe Wing Institute
2004 2004 -- presentpresent
independent, non-profit operating foundation
promote evidence-based education policies and practices
act as a catalyst to facilitate communication, cooperation and collaboration between individuals and organizations
currently engaged in evidence based education
![Page 6: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/6.jpg)
6
The Wing Institute’s Strategic Vision
Identify exemplars in evidence-based education
research individuals policies
models programs organizations
Develop networks to facilitate collaboration
Provide support for new ideas, research, and publications
Facilitate cross-discipline cooperation
The Wing Institute’s Strategic Vision
Increased focus on
Research to Practice
in the Real-world
in Real-timeimplementation
and
sustainability
![Page 7: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/7.jpg)
7
� Provide an expanded model for bridging the gap between research and practice
� Define the primary components of an evidence-based culture, their functions, and how they relate to each other.
� Illustrate the necessary and continuous reciprocal nature of influence between research and practice
““Research to PracticeResearch to Practice”” RoadmapRoadmap
The purpose of the “Roadmap” is to:
Research
Re
plic
ab
ilityS
usta
inab
ility
Evidence-based Education
What
works?
When does
it work?
Is it working?
Efficacy Effectiveness
ImplementationMonitoring
Practice
How do we
make it work?
Research to Practice
![Page 8: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/8.jpg)
8
The Role of
Data-Based Decision Making
in Evidence-Based Education
Critical Assumptions
in Data-Based Decision Making
![Page 9: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/9.jpg)
9
DataData--based Decision Makingbased Decision Making
has two distinct componentshas two distinct components
Decision MakingDecision MakingDataData--basedbased
meaningful and accurate
data being collected and
displayed
accurate interpretation of the
data and selection of effective
response
(problem solving)
Five Assumptions for EffectiveFive Assumptions for Effective
DataData--based Decision Makingbased Decision Making
Data-Based Assumptions
1. Reliable and valid data are available.
2. Interventions are implemented with integrity.
Decision Making Assumptions
3. Decision makers have the ability to analyze data.
4. Decision makers know what to do when data indicate a change is necessary.
5. Professional judgment is objective.
![Page 10: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/10.jpg)
10
Assumption 3: Assumption 3: Decision makers have the Decision makers have the
ability to base decisions on data.ability to base decisions on data.
Most student-level data displayed in graphs
Visual inspection of graphic data displays can be unreliable
Reviewers of behavioral journals(DeProspero & Cohen, 1979)
IOA = 61%
Assumption 3: Assumption 3: Decision makers have the Decision makers have the
ability to base decisions on data.ability to base decisions on data.
Accuracy can be improved with training and visual aids
Accuracy with criteria and training increasedfrom .46 to .81 (Hagopian, et al. 1997)
Accuracy visual aids and training increasedfrom .55 to .94 (Fischer, et al. 2003)
![Page 11: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/11.jpg)
11
Assumption 4: Assumption 4: Decision Makers Know What to Decision Makers Know What to
Do When Data Indicate Change is NecessaryDo When Data Indicate Change is Necessary
• Knowing there is a problem and knowing what to do
about it are very different things.
• Are we preparing educators to identify and implement
evidence-based interventions?
0
5
10
15
20
25
30
35
40
0 1 2 3 4 5
What Are Teachers Being Taught About Science of Reading?
Perc
en
t
# of Reading Elements TaughtNational Council on Teacher Quality, 2006
![Page 12: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/12.jpg)
12
Are We Training Educators to be Evidence-based?
Survey of School Psychology Directors of Training
29%
Knowledge of Evidence-based Interventions
Knowledge
(Shernoff, Kratochwill, & Stoiber, 2003)
Training41% programs
directors
Assumption 5:Assumption 5:
Professional judgment is objectiveProfessional judgment is objective
1. Data-based decision making is still an uncertain practice.
complex issues of cause and effect
incomplete data
2. False sense of accuracyprofessional arrogance
art vs. science
lack of feedback
3. Individual limitationsprofessional bias
errors in reasoning
![Page 13: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/13.jpg)
13
The fallibility of professional judgment: The fallibility of professional judgment:
Professional Arrogance: ignoring the fact that mistakes are inevitable
a false sense of accuracya false sense of accuracy
What percent of your decisions turn out to be "right"?
0
2
4
6
8
10
12
90% 85% 80% 75% 70% 65% 60% 55% 50%
% Decisions Correct
AVG = 77%
![Page 14: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/14.jpg)
14
The fallibility of professional judgment: The fallibility of professional judgment:
a false sense of accuracya false sense of accuracy
“art vs. science”
� decisions made on the basis of intuition
� empirical data not only waste of time, diminishes quality of service…it limits creativity and spontaneity
� professionals “trust” their professional judgment
The fallibility of professional judgment: The fallibility of professional judgment:
a false sense of accuracya false sense of accuracy
77% 22%Your intuition (gut feeling)
about what will be effective
39% 92%
Your demonstrated track record of success based on data you have gathered systematically and regularly
37% 92%Results of controlled experimental Studies
CRITERIA FOR
TREATMENT CHOICES CLIENT PHYSICIAN
Gambrill and Gibbs, 2002
![Page 15: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/15.jpg)
15
The fallibility of professional judgment: The fallibility of professional judgment:
a false sense of accuracya false sense of accuracy
“science vs. art”
� failure to recognize the limitations of existing scientific research
� failure to recognize the role and fallibility of professional judgment
The fallibility of professional judgment: The fallibility of professional judgment:
a false sense of accuracya false sense of accuracy
Lack of Feedback
� lack of systematic feedback from peers or colleagues
� lack of organizational systems for providing feedback on outcomes related to decisions
� difficulty with “complex systems” to ascertain cause and effect
![Page 16: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/16.jpg)
16
The fallibility of professional judgment: The fallibility of professional judgment:
Biases that can interfere with judging progress and causation
� Being swayed by hindsight
� Being overconfident
� Engaging in wishful thinking
� Having an illusion of control
� Overlooking the role of chance (coincidences)
� Seeking only data that support preferred views
� Overlooking the interaction between predictions and their consequences
� Mistaking correlation for causation
Critical Thinking in Clinical Practice : Improving the Quality of Judgments and Decisions, Eileen Gambrill
biasbias
The fallibility of professional judgment: The fallibility of professional judgment:
errors in reasoningerrors in reasoning
Common errors in reasoning that can effect perceptions and decisions.
Circular Reasoning
Non-Sequitur
Post Hoc
Red Herring
Equivocation
False Dichotomy
Lying
Authority
Shifting the Burden of Proof
Self-Referential Fallacy
Ad Hominem
Sidestepping/Avoiding the Question
Suppressed Evidence (Stacking the Deck)
Statistics
Jumping to Conclusions
Traditional Wisdom
Analogy
Humor
Extrapolation
Circumstantial Evidence
Straw Man
Guilt by Association
Best-in-Field Fallacy
![Page 17: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/17.jpg)
17
Five Assumptions for EffectiveFive Assumptions for Effective
DataData--based Decision Makingbased Decision Making
Data-Based Assumptions
1. Reliable and valid data are available.
2. Interventions are implemented with integrity.
Decision Making Assumptions
3. Decision makers have the ability to analyze data.
4. Decision makers know what to do when data indicate a change is necessary.
5. Professional judgment is objective.
![Page 18: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/18.jpg)
18
Good News / Bad News
Bad News: If any assumption is not true then our
ability to effectively implement data-based decision making is compromised.
Good News: There are emerging approaches that
make it possible for improve our performance in each
of these areas.
Types of Data
![Page 19: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/19.jpg)
19
Building a
Data-Based Decision Making Culture
Through Performance Management
DataData--based Decision Makingbased Decision Makingreliesrelies on Staff Performanceon Staff Performance
In order for a data-based decision making system to be effective, staff must:
implement it
compliance
correctly over time
competence sustainability
![Page 20: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/20.jpg)
20
DataData--based Decision Makingbased Decision Makingcompliance / competencecompliance / competence
• ask the right questions
• identify appropriate data to collect (validity)
• implement interventions according to plan (treatment integrity)
• collect data accurately (reliability)
• display / analyze data
• interpret data / draw correct conclusions
• give / receive feedback based on the data
• modify interventions based on data
DataData--based Decision Makingbased Decision Makingsustainability sustainability
• implemented with procedural fidelity and desired
outcomes (effectiveness) at the consumer level
• maintains over time
• maintains over generations of practitioners and
decision-makers
• operates within existing resources (financial, staff, materials) and existing mandates
• becomes institutionalized, routine…
National Implementation Research Network (NIRN)
“the way we do business”
![Page 21: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/21.jpg)
21
Sustainable PracticeSustainable Practice
requiresrequires::
a social / cultural change process across all levels of an organization
a long term, ongoing, developmental process
respect and accommodation of the uniquenessof every aspect of the culture
National Implementation Research Network (NIRN)National Implementation Research Network (NIRN)
Research on ImplementationResearch on Implementation
The most common forms of implementation…
paper implementation: new policies and procedures put in place
process implementation: new operating procedures put in place
information dissemination
training
supervision
have repeatedly been shown to be ineffective
performance implementation: monitoring activities and outcomes
and responding to the data
National Implementation Research Network (NIRN)
![Page 22: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/22.jpg)
22
Stakeholderspolicy makers
parents
school administrators
classroom staff
students
What isWhat is an Organizational Culture?an Organizational Culture?
The complex interaction of formal and informal contingencies governing the behavior of all stakeholders, embodied in:
External Contingencieslaws & regulations
fundingjob market
training and ideology
Internal Contingenciespoliciespractices
values
resource allocations
data systems
feedback systems reporting
requirements
program evaluationrecruitment & hiring
initiatives
job expectations
compensation
staff trainingstaff coaching
staff feedback
Building a Building a
DataData--based Decision Making Culturebased Decision Making Culture
All organizations have cultures…
most don’t support data-based decision making
![Page 23: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/23.jpg)
23
Baseline Cultural ObstaclesBaseline Cultural Obstacles
staff resistance to a data-based decision making culture:
long standing mistrust of the purpose of data
educator autonomy, implicit power relationships
cynicism about fads, new ideas, education reform
resistance to performance feedback
data collection is too difficult
data collection causes too much change
desired outcomes take too long to materialize
perceived costs exceed perceived benefits
Building a Building a
DataData--based Decision Making Culturebased Decision Making Culture
Data-based decision making cultures deliberately shape all cultural contingencies to reinforce the effective use of data in decision making.
� increase reinforcement for the target behaviors
� decrease aversive consequences for the target behaviors
� decrease reinforcement for competing behaviors
� increase aversive consequences for competing behaviors
![Page 24: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/24.jpg)
24
Performance management strategies are essential to accomplish this goal:
clear, measurable outcomes, goals, measures
effective data feedback system
deliberate, measurable intervention
ongoing monitoring
analysis
adaptation and innovation
Using Performance Management to BuildUsing Performance Management to Build a a
DataData--based Decision Making Culturebased Decision Making Culture
Overcoming Baseline Cultural Obstacles:Overcoming Baseline Cultural Obstacles:
Calibration, Process and EngagementCalibration, Process and Engagement
a “learner centered” culture (calibration)
focus on student learning and educational practices
establishing consensus on standards, definitions, goals
a culture of “inquiry” rather than “compliance” (process)
use of data to answer questions, problem solve
use of data-based decision making at all levels of the organization
not having all of the answers
a culture of “universal participation” (engagement)
wide-spread involvement (ownership, pride, participation)
collaboration across disciplines
giving, receiving, and using feedback
data analysis as positive, non-threatening experience
![Page 25: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/25.jpg)
25
Overcoming Baseline Cultural Obstacles:Overcoming Baseline Cultural Obstacles:
AlignmentAlignment
Alignment of all organizational cultural components so that contingencies consistently support data-based decision making
policies
practices
values
resource allocations
data systemsfeedback systems
reporting
requirements
program evaluationrecruitment & hiring
initiatives
job expectations
compensationstaff training
staff coaching
staff feedback
Using Performance ManagementUsing Performance Management
forfor cultural alignmentcultural alignment
Goals:
Definitions:
Outcomes:
Increase the number of “Qualified Staff”
staff meet regulatory qualifications
staff share common values about data,accountability, feedback and problem
solving
staff have technical skills in instruction, dataanalysis, problem solving…
staff positions filled by qualified staff
staff retention
![Page 26: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/26.jpg)
26
Using Performance ManagementUsing Performance Management
forfor cultural alignmentcultural alignment
X X Xstaff training
Process Measures
Outcome MeasuresStrategies
X X Xstaff feedback
X X Xstaff evaluation
X X Xrecruitment
X X Xselection
X X Xhiring
X X X
X X X
X X X
X X X
X X X
X X X
policies
practices
resource allocations
job expectations
compensation
initiatives
Strategies for ImprovingStrategies for Improving
Decision MakingDecision Making
Data Analysis Teams
1. Membershipcross discipline (administrator, support staff, classroom staff)
data facilitator
2. GoalsApply data-based decision making strategies to desired outcomes
Reinforce “inquiry” culture
Provide training and feedback
3. ToolsChecklists data displays
Scripts technology
4. ProcessOngoing meetings
Ongoing review
![Page 27: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/27.jpg)
27
• "Never doubt that a small group of thoughtful,
concerned citizens can change the world. Indeed it is the only thing that ever has.” Margaret Mead
• Evolution is chaos with feedback. �Joseph Ford
Key Indicator Systems: Why We Need
to Systematize Decision Making
![Page 28: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/28.jpg)
28
Efficacy ResearchEfficacy Research
(What Works?)(What Works?)
• Research conducted to identify promising practices
• Establishes a causal relationship between an intervention and its impact on behavior.
• Often conducted in highly structured and controlled laboratory settings to clearly demonstrate impact and causation
• Precision is often achieved with highly trained change agents, carefully screened participants, adequate resources, and close supervision.
• Currently, this is the most common form of published educationalresearch.
Practice
Research
Re
pli
ca
bil
ity
Su
sta
inab
ilit
y
What
works?
When
does it
work?
How do
we make
it work?
Is it working?
Implementation
Efficac y
Monitoring
Effectiveness
What
works?
Efficac y
Effectiveness ResearchEffectiveness Research
(When Does it Work?)(When Does it Work?)
• Research conducted to answer questions about the impact androbustness of interventions when taken to scale in more typical practice settings
• Primarily concerned with when an intervention works in the context of the following dimensions.
�characteristics of students, setting
�leadership and instructors
�resources, training available
�culture, level of commitment
• Less common than efficacy research
Practice
Research
Re
pli
ca
bil
ity
Su
sta
inab
ilit
y
What
works?
When
does it
work?
How do
we make
it work?
Is it working?
Implementation
Efficac y
Monitoring
Effectiveness
When
does it
work?
Effectiveness
![Page 29: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/29.jpg)
29
ImplementationImplementation
(How do we make it work?)(How do we make it work?)
• How do we make this intervention work in this particular setting?
• Translates effectiveness research to practice, from “general settings” to a “ particular setting”
• Explicit, systematic process for analyzing and addressing the critical variables necessary for an intervention to be successfully adopted, implemented and sustained in a particular setting.
• Analyzes the contingencies operating on various stakeholders in a particular practice setting and how they influence adoption and sustainability of an intervention.
Practice
Research
Re
pli
ca
bil
ity
Su
sta
inab
ilit
y
What
works?
When
does it
work?
How do
we make
it work?
Is it working?
Implementation
Efficac y
Monitoring
Effectiveness
How do
we make
it work?
Implementation
Performance MonitoringPerformance Monitoring
(Is it Working?)(Is it Working?)
• To assure that the intervention is actually being effective must
monitor the impact of the intervention in the setting (practice-based
evidence).
• Monitoring must occur:
� student level (to ensure progress and be able to modify components of
the intervention when necessary)
� systems level (to be able to make systems level decisions and policy
choices)
Practice
Research
Re
pli
ca
bil
ity
Su
sta
inab
ilit
y
What
works?
When
does it
work?
How do
we make
it work?
Is it working?
Implementation
Efficac y
Monitoring
Effectiveness
Is it working?
Monitoring
![Page 30: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/30.jpg)
1
Part II
The Role of Data-based Decision
Making
Jack States
The Wing Institute
QuickTime™ and a decompressor
are needed to see this picture.
What Is the Purpose of the Campbell Collaboration?
Mission
To help people make well-informed
decisions by preparing, maintaining
and disseminating systematic
reviews in education, crime and
justice, and social welfare.
![Page 31: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/31.jpg)
2
Why?��To Improve People’s Lives �Premise
Data-based decision making is indispensable to increasing the probability that socially important outcomes will be achieved.
How?
To help people make
well-informed decisions
by preparing, maintaining and disseminating
systematic reviews
in education, crime and justice, and social welfare.
![Page 32: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/32.jpg)
3
What Is Evidence-based Practice?“A Decision-making Model”
• EBP is a decision-making approach that places emphasis on evidence to:
�guide decisions about which interventions to use;
�evaluate the effects of an intervention.
Professional
Judgment
Best available
evidence
Client Values
Sackett et al (2000)
Professional JudgmentProfessional JudgmentBest Available EvidenceBest Available EvidenceClient ValuesClient Values
Research
Why do we select evidence-based practices?
Research tells us best
what works and when it works
Practice
The Key
Data-based Decision Making
![Page 33: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/33.jpg)
4
Why DataWhy Data--based Decision based Decision
Making?Making?
To Make Better Choices
1. Increase Effectiveness
2. Increase Efficiency
3. Provide Accountability
4. Guide Quality Improvement Efforts
What Are Critical Decision Points?
1. Selection of a practice or intervention
2. Assessment of the effectiveness of a practice
3. Assessment of treatment integrity
4. Selection of staff
5. Allocation of resources
![Page 34: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/34.jpg)
5
Assumption
• Assumption: There are sufficient
numbers of studies to answer questions
of efficacy and effectiveness
• Current Status: Unfortunately there are
not enough studies (of strength and
quality)
�Campbell Collaborative
�What Works Clearing House
Continua of Evidence
Quality of the Evidence
Personal Observation
Expert Opinion
Current “Gold Standard”High Quality
Randomized Controlled Trial
Uncontrolled Studies
General Consensus
Single Case Designs
Semi-Randomized Trials
Well-conducted Clinical Studies
Quantity of the Evidence
Janet Twyman, 2007
Meta-analysis (systematic review)
Single Case Replication (Direct and Parametric)
Single Study
Various Investigations
Repeated Systematic Measures
Convergent Evidence
Threshold of Evidence
![Page 35: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/35.jpg)
6
Assumption
• Assumption: If we implement an evidence-based practice we will achieve the desired results
• Current Status: Unfortunately, practices are not always implemented as designed
• Immediately following training treatment integrity begins to decline.
Mortenson & Witt, 1998
Noell, Witt, LaFleur, Mortenson, Ranier, & LeVelle, 2000
DiGennaro, Martens, & McIntyre, 2005
Assumption: Staff Can Implement Interventions with Integrity
Evidence-based drug education programs implemented with integrity only 19% of the time.
Hallfors & Godette (2002)
�This may be an overestimate.
� 52% reported that programs were modified or adapted.
No reason to believe that other curricula and social interventions are implemented with any better integrity.
![Page 36: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/36.jpg)
7
Positive NegativeH
igh
Lo
w
Continue
Intervention
Change
Intervention
Unknown Reason
Unknown Reason• Intervention problem?
• Implementation problem?
• Other life changes?
• Unknown intervention?
• Intervention is
effective?
OutcomeIn
tegrity
Positive NegativeH
igh
Low
Context
Progress monitoring is an effective method of
comprehensive school reform.
4
Yeh (2007)
10% increase in per pupil spending.
vouchers; increased accountability.
charter schools.
X
6 X
64 X
![Page 37: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/37.jpg)
8
Assumption
• Assumption: An evidence-based intervention works for all students
• Current Status: Unfortunately, few if any interventions work for everyone
A Prevention Model for Evidence-based Education
Academic Systems Behavioral Systems
1-5% 1-5%
5-10% 5-10%
80-90% 80-90%
Intensive, Individual Interventions
•Individual Students
•Assessment-based
•High Intensity
Intensive, Individual Interventions
•Individual Students
•Assessment-based
•Intense, durable procedures
Targeted Group Interventions
•Some students (at-risk)
•High efficiency
•Rapid response
Targeted Group Interventions
•Some students (at-risk)
•High efficiency
•Rapid response
Universal Interventions
•All students
•Preventive, proactive
Universal Interventions
•All settings, all students
•Preventive, proactive
![Page 38: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/38.jpg)
9
Conclusion
1. Data-based decision making is core to
building an evidence-based culture
2. An evidence-based practice cannot exist
without treatment integrity
3. Progress monitoring is an essential to
informing us that the evidence-based
intervention is working
Part III
Critical Assumptions in Data-based Decision Making
Randy Keyworth
![Page 39: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/39.jpg)
10
Part IV
TYPES OF DATA
Jack States
“In God we trust; all others must bring data.”
--W. Edwards Deming, physicist and quality improvement pioneer
Assumption: Reliable and Valid Data are Available
• Academic performance:
�Sufficient for reading and math at Elementary age
�Insufficient for majority of subjects for above elementary age
• Social behavior: Insufficient established measures
• Systems: Sufficient
![Page 40: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/40.jpg)
11
NationalStudent
Skill
Classroom
School
TeachersTest
Standards
Analyzed
Curriculum
Raw Data
FiscalGraduation
Attendance
“…analysis paralysis, or the process of being over-stimulated by data...” Joe McKendrick (2003)
National Mandates
Local Reforms
Parental Expectations
UnionsSchool Safety
Special Education
Regulations
School Improvement Plans
Employment Regulations
Regional
Reforms
InformationRaw Data
Data-Based
DecisionInformed
Action
Data-Based Decision
Model
•Organize
•Process
•Perspective
•Synthesize
•Prioritize
•Weigh
•Select
•Inform,
Identify, or
Clarify
•Act
•Define
•Collect
•Reliable
Organize Decision Making
![Page 41: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/41.jpg)
12
Assumption
• Assumption: It is enough to monitor student achievement through high stakes testing
• Current Status: High Stakes Tests
�Are infrequent
�Are not prescriptive for individual students (only groups)
Use Data That Drives The System
Three Categories of Data
1. Outcome Data - Are we achieving our Goals?
2. Process Data - Is the system working?
3. Satisfaction Data - Are we meeting stakeholders expectations?
![Page 42: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/42.jpg)
13
Outcome and Process Data
Formative
Data used to adjust instructional practices to maximize individual students’ learning, to gage progress, and direct instructional interventions.
− Lesson Quiz
− Chapter Test
− Curriculum Based Measures (CBM)
− Dynamic Indicators of Basic Early Literacy Skills (DIBELS)
Summative
Data collected after instruction has been completed, at the end course, or annually and used to make strategic program adjustments, curriculum decisions, and improve instructional practices.
− SAT, ACT
− Standardized test
− Exit Exams
− Graduation/Drop out
Process
Data collected on systems that support instruction.
− Demographic information
− Finances
− Teacher credentials
− Hiring
− Facility safety
− Treatment integrity
− School improvement plans
Understand How Your System Works
OutcomesInput
Feedback
Loop
Teaching
Performance
Curriculum
Rummler & Brache, (1990)
Improving Performance
Test
![Page 43: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/43.jpg)
14
Part V
Building a Data-based Decision Making Culture Through
Performance Management
Randy Keyworth
PART VI
Key Indicator Systems
“Why We Need to Systematize
Decision Making?”
Jack States
![Page 44: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/44.jpg)
15
Context is Everything
Data based decision makers are only as good as the systems in
which they are working.
Student
Classroom
State
District
School
Federal
Student Outcome Formative and
Summative
Process Outcome
Support, Teaching
and
Aggregated Student
Align Across
Levels
![Page 45: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/45.jpg)
16
What is a Key Indicator?
Discrete pieces of information that let us know how the system is running.
But, does not prescribe solutionsTells you where to begin to look
But, does not identify causeIndicates system going well or poorly
But, not a diagnostic toolIdentifies status or trends
What It Does Not DoWhat It Does
Thermometer
Dow Jones
Key Indicator Report
Ask the Right QuestionsAsk the Right Questions
1. What are you going to do with the data?
2. Can the data answer the stakeholders questions?
3. Are the data an indicator of key elements (input, process,
and outcomes)?
4. Are the data objective, measurable, and reliable?
5. Who needs to interact with the data?
6. Does the data drive decisions for improving the process?
7. Are the data provided timely?
8. Are the proper technologies available for storage and
delivery?
9. Are the data systems capable of making queries (drill-down)
Wayman, J., Stringfield, S. & Yakimowski, M. (2004) Software Enabling School Improvement…,
![Page 46: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/46.jpg)
17
Create a Key Indicator Report
1. Keep Report to two pages - limit items
• Staff do not have time
• Staff do not have the training
• Staff do not have the motivation
2. Data can be disaggregated (“drill down” capability)
3. Provide time for review of the report
4. Display data in easily understandable
5. Provide meaningful comparisons (status and trend)
- Identify current performance
- Identify progress over time
- Compare to standards
- Compare to peers
QuickTime™ and a decompressor
are needed to see this picture.
Office Referrals Per
Student
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Sept Oct Nov Dec Jan Feb Mar Apr May June
Months
Current Year
2007
District 2007
Indicator System Selection Criteria
Communicative Value
•Less is more: Gate keep and prune
•Utilize graphic displays
•Display simple to understand:
convey complex issues but simply
![Page 47: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/47.jpg)
18
Indicator System Selection Criteria
Informative Value
• Indicative of critical outcomes, process, and input
• Predictive of multiple factors
• Data can be disaggregated: Drill down
Indicator System Selection Criteria
Functional Value
• Confidence
• Timely
• Measurable
• Numerical
• Comparable
• Within available resources
• Align the data across levels
![Page 48: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/48.jpg)
19
Assumption
• Assumption: Technology exists and is
capable of supporting a Key Indicator
Report
• Current Status: Major gaps exist in the
storage and retrieval of data across data
bases
Key Indicator ReportKennedy Elementary School - 2009
90%
90%
87%
Feb
90%
90%
85%
Jan
80%
80%
81%
Nov
90%
50%
82%
Dec
100%80%80%Input: Fully
credentialed teachers
- % teachers current
credential
95%80%60%Process: Treatment
integrity check - %
items met per
observation
95%79%75%Outcome: Reading -%
Words/minute to
grade level
2008
YTD Ave.
OctSeptReport Item
Reading - Words/Minute - 2009
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Sept Oct Nov Dec Jan Sept
Months
Kennedy
District
Reading - Grades
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
Sept Oct Nov Dec Jan Sept
Months
Second Grade
Third Grade
Fourth Grade
Fifth Grade
![Page 49: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/49.jpg)
20
Assumption
• Assumption: Staff have the training and time to review data and adjust teaching strategies in response to data
• Current Status:
1. Teachers do not receive the technical training
2. Formative data is frequently not input into the systems
3. Time allotted for review of data is often not a priority
Conclusion
1. Key Indicators Reporting Systems (KIRS) can be potent tools in driving performance improvement
2. The necessary data and the technology to implement KIRS are available to school systems
3. Changes in the education culture are required to increase the time educatorsengage in the review of data
![Page 50: A Systematic Approach to Data-based Decision Making in ... · Data -based Decision Making has two distinct components Data -based Decision Making meaningful and accurate data being](https://reader033.fdocuments.us/reader033/viewer/2022052013/6029f5d3a3fc086fc635d109/html5/thumbnails/50.jpg)
21
The End
ResourcesData Based Decisions - The Tyranny of Small Decisions:Origins, Outcomes and Proposed Solutions - (2000) Bickel, W., & Marsch, L.
Data Based Decision Model - From Data to Wisdom: Quality Improvement Strategies Supporting Large Scale Implementation of Evidence-Based Services - (2005) Daleiden & Chorpita
Data Based Decision Models - Making Sense of Data-Driven Decision Making in Education -(2006) Marsh, J., Pane J, & Hamilton, L.
Effective Dashboard Designs - http://www.slideshare.net/hursman/effective-dashboard-design-presentation?from=email&type=share_slideshow&subtype=slideshow
Key Indicators Systems - Celio,B. and Harvey, J. (2005) Buried Treasure
Performance Improvement - Rummler, G., & Brache, A., (1990) Improving
Performance
Software and Hardware - Wayman, J., Stringfield, S. & Yakimowski, M. (2004) Software Enabling School Improvement Through Analysis of Student Data