Using a Fidelity Index to Increase Program Attribution in Impact Evaluation Studies

Presented by Donna Smith-Moncrieffe

Canadian Evaluation Society-Ontario Chapter

October 4th-5th, 2010

Presentation Outline

� National Crime Prevention Centre (NCPC) mandate

� What is a Quantifiable Fidelity Index? Why is it Important?

� How can using a Quantifiable index increase Program Attribution?

� Steps: From Fidelity Tool Development to interpretations regarding program attribution

2

� Steps: From Fidelity Tool Development to interpretations regarding program attribution

� Sample Fidelity tool and output data for two Model programs

� Towards No Drug Program (results)� Stop Now and Plan Program ® (application of treatment intensity

measures)

� Summary

NCPC Mandate/Core Activities

Mission statement:

To provide national leadership on effective and cost-efficient ways to both prevent and reduce crime by addressing known risk factors in high risk populations and places

Core activities:

3

Core activities:

� Supporting targeted interventions in local communities

� Building and sharing practical knowledge with policy makers and practitioners

NCPC Priorities:

Provide funding to the following target groups/crime issues:

� Children and youth at risk

� Crime prevention in Aboriginal communities

4

� Prevent recidivism among high-risk groups

� Priority crime issues (youth gang, drug-related crimes)

Use of Evaluation in NCPC

NCPC Time-Limited Funding

Results contribute to Identify and encourage the development of

Disseminate resultsand encourage

the province and municipalities to replicate

effective programs

5

UseMulti-site

EvaluationsTo IncreaseKnowledge

Results contribute to Treasury Board

Reports

Encourage the development of Fidelity Tools to

Increase Attribution

Identify and encourage the development of Model and Promising Programs

What is Program Fidelity ?

� Fidelity in evaluation is used to describe the extent to which the initiative/intervention corresponds to the originally intended program

� The literature uses the following terms interchangeably

6

� Adherence� Compliance� Integrity

Benefits of a Fidelity Index

A Fidelity Index:

� Lists the elements that contribute to the success of the program

� Makes the important elements of the program visible and helps evaluators “bound” the program

When expected results are unfavourable, model programs can

7

� When expected results are unfavourable, model programs can attribute the results to low fidelity levels while still being able to maintain a reputation for being an effective program

� Identifies what elements practitioners should focus on to increase compliance

� Ensures that elements in the index will have a complementary quality assurance protocol and definitions for practitioners to review

What is a Quantifiable Fidelity Index?

� A quantifiable fidelity index is a set of measureable items that determine whether the program elements have been implemented as planned.

� Each item provides a quantifiable measure using ratio or interval level data that can be later used to test for statistical significance, changes in effect sizes or clinical significance

8

� The items in the index can be checked for construct validity. We can confidently determine if we are measuring what we think we are measuring. (i.e. Using Chronbach’s alpha and other reliability tests can be achieved by using quantitative measures)

� Composite indices and aggregate scores can be inserted into multivariate analysis to determine whether the fidelity related scores contribute to participant related changes.

How Can a Measureable Fidelity Index Contribute to Better Program Attribution?

� You can confidently answer the question, ‘Did the model program contribute to change or were additional unintended program elements contributing to changes ? (i.e. Was the inconsistent implementation of cognitive behavioural sessions responsible for the lower than expected changes in anger for boys in the SNAP program?)

� Each element that is quantified can be placed into a multivariate causal model. The evaluator can:� Isolate whether the program’s adherence levels contributed to

favourable or unfavourable change in the outcomes of interest

9

favourable or unfavourable change in the outcomes of interest� Isolate what specific aspect of the program contributed to greater effects

or changes in the outcomes of interest� Identify the need to find other explanatory variables that are contributing

to program impact (i.e. R squared adjusted result < .80 informs us that there are other factors that account for the change in reduction of drugs or offending for example)

� Identify if low fidelity to the program is contributing to variation in the variables of interest (i.e. Does low participant responsiveness in 25% of the TND classrooms affect the overall goal of preventing or reducing drug use in youth?)

Use of Fidelity in Four NCPC Model Programs(Based on literature prior to NCPC Implementation)

75% of model programs used fidelity for

Process evaluation only

25% of model programs used fidelity for

Outcome/Impact Evaluations

10

Method of Attribution is limited toInferences about how

Program is related to outcomes

Method of Attribution is stronger and can identify how much

Correlation exists between the Program elements and outcomes

Key Steps in Using a Quantifiable Fidelity Tool to Increase Attribution

STEP 1: Identify all the key program elements using a comprehensive framework

STEP 2: Apply appropriate measures

STEP 3: Construct the causal model

11

STEP 3: Construct the causal model

STEP 4: Calculate and interpret the coefficients (multivariate analysis)

STEP 5: Reporting: Enhance program attribution. Determine if program fidelity levels contributed to outcomes

STEP 1: Identify all key elements using a comprehensive framework

Based on Dane & Schneider Framework (1998)

Develop the fidelity tool and include all elements in the following four areas:

1. Implementation: Was the program delivered as intended? (i.e. Were all 8 sessions delivered on a weekly basis over a 2 month period?)

2. Dosage: How much of the program was delivered (i.e. Did the youth receive the 48 hours of individual case management?)

12

the 48 hours of individual case management?)

3. Quality: Are the main components of the program delivered clearly and correctly?

4. Participant Responsiveness: Does the program stimulate interest among the participants and practitioners? (i.e. Are teachers interested in implementing the curriculum to students?)

5. Monitoring Control/Comparison fidelity- What services did the comparison/control group receive? Did the program adhere to their intention to provide minimal services or “usual care”?

STEP 2:Apply Appropriate Measures

Durlak and Dupre 2008 reviewed 59 studies (1998-2005) that had some type of fidelity tool and found the following typical measures:

� 27/59 studies (46%) used categorical measures� Ordinal scales (i.e. assigning a definition to low, moderate and

high fidelity levels)

13

high fidelity levels)

� 32/59 studies (54%) used continuous variables/interval measures� Averages reported as percentages (i.e. observations of each

session identified that 65% of the youth showed interest in the counselling sessions)

� Likert scales results were converted to continuous variables� Used actual # of hours, # of sessions, # of months especially for

dosage related measures

STEP 2:Sample Treatment Intensity Measures

(Stop Now and Plan® Program)

Sample : Quantifiable fidelity checklist (note: this is not the full checklist)v Target Group Population Met: 8/10

v 12 Cognitive Behavioural group Sessions 10/12� Sessions delivered weekly within 3 months

� Delivered by a trained facilitator� Approved manuals used (weekly documentation)

14

� Approved manuals used (weekly documentation)

v 12 Cognitive Behavioural concurrent parenting sessions 10/12

Fidelity Level: High Total Score: 28/34 or 82.4%____________________________________________________________

Fidelity LegendNo Fidelity: 0-29%Low Fidelity: 30-69%

Moderate Fidelity: 70-79%High Fidelity: 80-100%

STEP 3: Construct the causal model

(Multiple Regression Analysis sample)

� A multiple regression equation can be used to make estimates about key program outcomes (i.e. recidivism) based on given values for a number of explanatory variables (i.e. levels of drug use, # of hours participating in treatment, and type of practitioner)

15

� Ensure all statistical assumptions are met before using the model

� Construct the causal model by using the following equation :

y = β0 + β1X1 + β2X2 + …. + β kXk + ε

STEP 4 : Calculate and interpret coefficients

Interpretations

Overall, we are looking at the marginal contribution that each X

(explanatory) variable is making on the Y (outcome) variable when the

other X variables are being held constant.

Review and interpret the output and consider the following

16

Review and interpret the output and consider the following questions:

� Are the elements from the fidelity tool statistically significant?

� What elements of the fidelity tool have higher values and contribute more to change in the key outcomes of interest?

� What is the overall contribution of the program elements on the outcome?

� How much does the impact of fidelity have on program outcomes?

STEP 4: Calculation and Interpretation of Coefficients

cont’d

B e c le a r a b o u t h o w yo u g o t fro m yo ur fid e l ity to o l to d isc us sing wh e th e r a d h e re n ce le v e ls h a d a n im p a ct o n p ro gra m o u tc o me s …

17

1 1

“I th ink y ou sho u ld be m ore exp lic it h e re

in ste p tw o .”

im p a ct o n p ro gra m o u tc o me s …

Th e re a re n o m ira cle s h e re … ..

Step 4: Calculate and Interpret Coefficients (sample data only)

Y= Aggressive Behaviour

Constant 5.274

Coefficient of X1

(CBS) Cognitive Behavioural Sessions (0-12 hours)0.4487

18

Coefficient of X2

(DSH) Duration of Service Hours (total program hours)0.3334

Coefficient of X3

(FPS) Levels of Fidelity (Participant Responsiveness)0.2448

R 2 (adj.) 0.7553

Step 4: Calculate and Interpret Coefficients

� For every increment of cognitive behavioural sessions, aggressive behaviour decreases by 0.4487

� For every increment of cognitive parental group sessions, aggressive behaviour decreases by 0.3334For every increment of fidelity (responsiveness), aggressive

Y= 5.274 - 0.4487 (CBS) - 0.3334 (CPG) - 0.2448 (FPS)

R Square (adj.) =.7553

19

� For every increment of fidelity (responsiveness), aggressive behaviour declines by 0.2448

� R square (adjusted) : The treatment intensity of the cognitive behavioural session, duration of service hours and total group responsiveness contributed to 76% of the variability in Aggression levels.

� Fidelity levels related to group responsiveness are positively correlated with the outcomes measuring changes in aggression levels but contribute less than treatment related factors

Step 4 Calculation cont’dUse Multi-Level Analysis

� Include fidelity scores in the ANOVA or regression equation

� Use a multilevel analysis approach:Group fidelity measure� Participant responsiveness� Implementation measures� Quality of Implementation

20

Quality of Implementation

Individual measures� Dosage (quantity)

� Ensure all key quantifiable fidelity scores are tested for separate cohorts (i.e. various classrooms, sites or session). Correlate fidelity scores with key results (i.e. changes in knowledge, attitudes and behavioural changes)

Step 5: Reporting ResultsWhat Really Contributed to the Results?

High Levels of Fidelity to the Program

Low levels of Fidelity to the Program

Favourable ChangesMade in the

Outcomes of InterestIncreased

Confidence that the Program is Effective

(It Works!!)

�Determine whether fidelity scores or core treatment measures are contributing to results�Qualify results by explaining how fidelity may have contributed to results

21

(It Works!!) results�Be cautious about reporting on program attribution

Unfavourable Changes

Made in the Outcomes of Interest

IncreasedConfidence that the

Program is Ineffective

Explore other sources of data to explain findings:� How many elements of the fidelity index were low?�What elements of the index were low?

STEP 5: Determine if Program Outcomes are related to Fidelity(Towards No Drugs sample)

Outcome FidelityProgram-specific knowledge 0.33 (0.09)***Beliefs: health-as-a-value 0.10 (0.04)**Beliefs: pro-drug myths -0.15 (0.06)**Cigarette intentions -0.08 (0.04)**

22

Cigarette intentions -0.08 (0.04)**Marijuana intentions -0.09 (0.04)**Alcohol intentions -0.07 (0.04)*

Multi-level models * p< .10 ** p < .05 *** p < .0001, one tailed

Rohrbach, L.A., Gunning, G., Sussman, S., & Sun, P. (May; 2008).Predictors of implementation in the Project Towards No Drug Abuse dissemination trial.

Summary

� Attempt to utilize 5 categories: Use Implementation, Dosage, Quality, Participant Responsiveness and Monitoring control group fidelity categories to ensure the tool/index is comprehensive

� Utilize a participatory approach to identify the key elements

� Use a statistical approach (backward or forward selection procedures)

� Evaluators should advocate for the development and use of a

23

� Evaluators should advocate for the development and use of a quantifiable fidelity tool to be implemented in the evaluation study

� Encourage the use of interval levels of measurement to each item in the tool. This type of measure can provide specific information about incremental changes in the outcomes of interest

� Qualitative information should also be collected to explain why the results may be favourable or unfavourable

Summary

� Construct a multivariate analysis model that will incorporate elements of the fidelity tool

� Consider using multilevel models that utilize more than one equation that will take into consideration different levels of data (i.e. Towards No Drugs fidelity tool required a multilevel model for the analysis: schools are nested in communities; pupils are nested in schools etc..)

� Report how the fidelity levels related to the outcomes of interest

24

� Where fidelity levels and expected results are low, ensure that triangulation with other data is used to verify program attribution levels

� Explore other elements of the regression or other multivariate equations used to isolate what program elements may or may not have contributed to the results.

Contact Information

Donna Smith-Moncrieffe, BSc., Crim Dip, MSc.Senior Evaluation Advisor

Public Safety CanadaNational Crime Prevention Center

25

Policy Research and Evaluation DivisionE-mail: donna.smith-moncrieffe@ps.gc.ca