Making Sense of Measuring Implementation in Educational Research

Amy Cassata & Jeanne Century MSPNet Webinar

February 25th, 2015

Today’s Agenda

•  Defini&ons,  Theory  and  Background  •  Implementa&on   Measurement   and   the  

Common  Guidelines  •  Design  and  Measurement  Approaches  •  Analysis  Strategies  

•  Study  #1:    Exploratory  -­‐  Nine  School  Districts  –  Science      1998-­‐2002  •  Study  #2:    Literature  Synthesis  2006-­‐2010  •  Study  #3:      FOI  of  Mathema&cs  and  Science  Materials    2007-­‐2010  •  Study  #4:      Implementa&on  Study  –  STEM  Schools  (2)    2010-­‐2013  •  Study  #5:      Implementa&on  Study  –  Elementary  Mathema&cs    2011  -­‐  2015  •  Study  #6:      Instrument  Development/Valida&on  Study  –  IES  2011-­‐2015  •  Study  #7:      Implementa&on  Study  –  Computer  Science  2013-­‐2016  

Previous and Current Work

•  What  is  an  innova&on?  An  interven&on?    

•  What  is  fidelity  of  implementa&on?  How  is  it  different  from  “implementa&on”  (use)?    

•  What  is  the  implementa&on  process?  

   

Defini&ons  

Intervention Component Framework  

Interven(on  Component  Framework

Categories  of  Cri&cal  Components

Structural Interac&onal

Procedural Educa&ve Interac&onal  (Leader)

Interac&onal  (Par&cipant)  (engagement)

This  work  was  funded  in  part  by  the  Na(onal  Science  Founda(on    

What is the “intervention?” What is the “IT?”  

FACTORS  

INTERVENTION  

OUTCOME  B  OUTCOME  A   OUTCOME  A’  

FACTORS  

FACTORS  

OUTCOME  B   OUTCOME  A’   OUTCOME  C  OUTCOME  B’  

FACTORS  

FACTORS  

Time  

INTERVENTION’  INTERVENTION’’  

characteris&cs  of  the  users  factors  

In  the  context  of  the  interven&on  •  self-­‐efficacy  •  understanding  of  the  interven&on  •  actude  toward  the  interven&on  •  intrinsic  mo&va&on  •  extrinsic  mo&va&on  

Descrip&ve  characteris&cs  •  demographic  •  educa&on  •  experience  

characteris&cs  of  the  users  factors  

NOT  in  the  context  of  the  interven&on  •  innova&veness  •  resourcefulness  and  coping  •  networked-­‐ness  •  &me  management  and  organiza&onal  skills  

Percep&ons  •  perceived  adaptability  •  perceived  visibility  •  ease  of  use  •  perceived  effec&veness  

characteris&cs  of  the  organiza&on  

factors  Descrip&ve  Characteris&cs  •  organiza&onal  structures  •  physical  environment  •  popula&on  characteris&cs  •  stakeholder  community  

support  •  presence  of  opportuni&es  for  

learning  inside  the  organiza&on  

People  in  the  Organiza&on  •  informa&on  sufficiency  •  informa&on  sharing  •  shared  beliefs  and  values  •  Resources  •  locus  of  decision  making  •  u&lity  of  learning  opportuni&es  •  organiza&onal  efficacy  •  collabora&on  •  leadership  

Organiza&onal  Strategies  •  ongoing  improvement  

structures  •  leveraging  •  implementa&on  strategy  •  family/community  

communica&on  

characteris&cs  of  the  environment  factors  

•  poli&cal  environment  •  community  beliefs  and  values  •  descrip&ve  community  characteris&cs  •  opportuni&es  for  learning  •  network  structures  •  extraneous  events  or  ini&a&ves  

•  Defini&on  of  fidelity  of  implementa&on    

•  Measuring  “dimensions”  of  implementa&on:  structure  and  process    

•  Need  to  iden&fy  the  interven&on  model  and  the  cri&cal  elements  &ed  to  intended  impacts  

Areas  of  Convergence  

Design  and  Development   Efficacy  Study   Effec&veness  Study   Scale-­‐up  Study  

Purpose   Draw  on  exis(ng  theory  and  

evidence  to  design  and  itera(vely  develop  interven(ons  or  strategies.  

Tes(ng  a  strategy  or  interven(on  under  “ideal”  circumstances    

Examine  a  strategy  or  interven(on  under  “typical”  circumstances  or  condi(ons  of  rou(ne  prac(ce  

Examine  a  strategy  or  interven(on  in  a  wide  range  of  popula(ons,  context,  and  circumstances  

Que

s&on

s  

Are  users  able  to  enact  the  interven(on  as  intended(feasibility  and  fidelity)  and  if  not,  why  (factors)?  

Does  it  work?  (fidelity)  

Does  it  work  on  a  large  scale?  (fidelity,  use,  factors)  

How  does  it  work  in  natural  seLngs  and  what  what  factors  support  and  inhibit  implementa(on?  (use,  factors)    

Implem

enta&o

n  Measuremen

t   •  Iden(fying  essen(al  components  

•  Organizing  components  •  Disentangling  interven(on  

components  from  general  quality  indicators  

•  Measures  of  appropriate  grain  size  

•  Instruments  for  both  treatment  and  control  condi(ons  

•  Affordable  designs  that  will  yield  sufficient  data  to  differen(ate  treatments  

•  Crea(ng  scales  and  indices  represen(ng  implementa(on  that  can  be  linked  to  outcomes  

•  Appropriate  analy(c  approaches  

•  Rela(onships  between  factors  and  implementa(on  

•  Rela(onships  between  implementa(on  and  outcomes  

•  Describe  implementa(on  over  (me  

•  Op(mal  implementa(on  for  par(cular  popula(ons  

Measuring  Implementa&on  in  Common  Guidelines  Studies  

Making  Sense  of  Implementa&on  

•  Implementa&on  measurement  is  more  than  just  “fidelity  of  implementa&on.”  

•  All  ini&a&ves  (  programs,  interven&ons,  reforms,  innova&ons)  are  comprised  of  components.    

•  Factors  influence  enactment  of  components.  

•  Components  and  factors  are  measurable.    •  Be  clear  about  what  the  “it”  is.  

 

Takeaways

Contact Information: acassata@uchicago.edu jcentury@uchicago.edu