Measuring Fidelity in Early ChildhoodScaling-Up Initiatives:
A Framework and Examples
Carl J. Dunst, Ph.D.
Orelena Hawks Puckett InstituteAsheville, North Carolina
Presentation made at the “Fidelity: How Much Devotion is Enough? (and Other Methodological Virtues)” session
at the Office of Special Education Program’s Project Directors Conference,Washington, DC, July 2008
Purposes of the Presentation
• Describe a framework for measuring fidelity as part of scaling-up early literacy learning practices.
• Define different types of fidelity at different levels of training and practice.
• Illustrate how variations in fidelity can be related to variations in practice adoption and outcomes.
Center for Early Literacy Learning
The main aims of CELL are to: (1) Synthesize researchevidence on effective early literacy practices, (2) developevidence-based practices from the research syntheses,(3) evaluate the validity and use of the practices, and (4)conduct both generalized and specialized technicalassistance promoting adoption and use of CELL practices.
From Evidence-Based Practices to Fidelity to Outcomes
Developing Evidence-Based Early Literacy Learning Practices
Scaling-Up the Use of the Practices
Measuring the Fidelity of Use of the Practices
Child Outcomes
Early Literacy Learning
Specialized Technical Assistance
Framework for Measuring Fidelity
Types of Fidelity
Implementation
Diffusion
Intervention
Implementation Fidelity
Implementation fidelity is defined as the degree to whichCELL training activities of known characteristics areimplemented as planned and promote participantunderstanding of the known characteristics of the evidence-based practices constituting the focus of training.
Diffusion Fidelity
Diffusion fidelity is defined as the degree to which secondand third generation technical assistance providers useCELL training methods and procedures of known characteristics to promote end-user understanding of thekey characteristics of evidence-based practices.
Intervention Fidelity
Intervention fidelity is defined as the extent to which endusers adopt and use instructional practices of knowncharacteristics for implementing evidence-based practicesof known characteristics.
Elements of Fidelity
Each type of fidelity includes two elements:
• Fidelity of the training processes (how)
• Fidelity of practice understanding and adoption (what)
Elements of Fidelity
Training processes are HOW we promote adoption and useof WHAT we want implemented by end-users. As state byFixsen et al. (2005), it is important to be aware of thedifference between the practices constituting the focus oftraining and the training methods used to promote adoptionof the practices.
Framework for Structuring Fidelity Assessments
CELL Fidelity Checklist for MeasuringImplementation and Diffusion Fidelity
Fidelity of both implementation and diffusion training isbeing measured by a CELL fidelity checklist that assessesthe degree to which CELL training processes and CELLpractices are accomplished with (1) sufficient dose (2)and with ample quality exposure to (3) key training and practice characteristics, and (4) are viewed by participantsas socially valid.
Measuring Fidelity of Intervention
Fidelity of intervention is measured in terms of practitioneror parent, or both, use of instructional practices andevidence-based practices of known characteristics as partof the provision of everyday early childhood intervention. Fidelity is measured by the frequency and number ofpractices that are used and the development-enhancingqualities (active ingredients, core features, etc.) of theintervention practices. The social validity of both thepractices and the child outcomes from using the practicesare also assessed.
0
10
20
30
40
50
60
70
80
Not At All the Case
Somewhat the Case
Mostly/Very Much the Case
DEVELOPMENT-ENHANCING CHARACTERISTICS
PE
RC
EN
T O
F P
AR
TIC
IPA
NT
S
Parents
Practitioners
Relationship Between Fidelity and Child Outcomes
Practice MeasuresLow Users High Users
Mean Mean F-Test Cohen’sa a d
Number of Weeks -.21 .16 1.26 .37
Number of Learning Games -.32 .32 5.70* .67
aMean standardized score.
*p < .05.
Game Characteristics -.61 .51 5.73* 1.32
Further Evaluation of the Effectivenessof the Practice
Additional information about the effectiveness of thepractices was determined by a post-test only quasi-experimental between group design that included twooutcomes, one of which was a “dependant variable that atreatment should affect” and a second dependantmeasure that was not expected to be related to the use ofthe practice (Shadish et al., 2002, p.184). Obtaining anexpected group by outcome measure interaction was usedto ascertain effectiveness.
Conclusions
• Fidelity is important as part of ascertaining the adoption and use of evidence-based practices.
• A fidelity framework can help structure the development and collection of fidelity measures.
• It is important to measure both implementation and intervention fidelity.
• Relating variation in fidelity to variation in outcomes can help identify what matters most in terms of targeted practices.
For More Information Visit
www.puckett.org for methods and procedures for identifying evidence-based practices
www.earlyliteracylearning.org for information on the Center for Early Literacy Learning.
Top Related