Conceptualizing, Designing, and Conceptualizing, Designing, and Testing Interventions in Social Testing Interventions in Social

Work: Work: Substantive and Methodological Substantive and Methodological

IssuesIssues

Mark W. FraserMark W. FraserUniversity of North CarolinaUniversity of North Carolina

Chapel Hill, NCChapel Hill, NC

This presentation was given at the NIH Office of Behavioral and Social Sciences Research (OBSSR) Summer Institute on Social and Behavioral Intervention Research, July 18, 2005, in Washington, DC. Special thanks to Mary A. Terzian and Shenyang Guo for assistance in developing material on conceptual models, efficacy subset analysis, and propensity scores.

AgendaAgenda What is intervention research?What is intervention research? Substantive issuesSubstantive issues

Defining the problem in terms of prevalenceDefining the problem in terms of prevalence Creating theoretical and conceptual frameworksCreating theoretical and conceptual frameworks Identifying risk mechanismsIdentifying risk mechanisms

The (your) interventionThe (your) intervention Drawing conceptual frameworksDrawing conceptual frameworks Making treatment manualsMaking treatment manuals

Methodological issuesMethodological issues Three key questions Three key questions

What are the impacts?What are the impacts? ClusteringClustering AttritionAttrition SelectionSelection

What works for whom?What works for whom? ITT versus ESAITT versus ESA

What ‘causes’ the impacts?What ‘causes’ the impacts? Special issues (if time)Special issues (if time)

Treatment fidelity and adherenceTreatment fidelity and adherence Method varianceMethod variance Culture, gender, age, languageCulture, gender, age, language Ethics and successEthics and success

ChallengesChallenges

Big topic and goals are to:

•Describe a sequential process in the design and testing of interventions

•Review conceptual models

•Show features of a treatment manual

•Touch on core analytic questions and challenges

•Discuss special issues

INTERVENTION INTERVENTION

… … an attempt to induce change an attempt to induce change selectively … [based] not only on the selectively … [based] not only on the experiences of caseworkers … but on experiences of caseworkers … but on a broad range of studies bearing on a broad range of studies bearing on

the problems of induced change.the problems of induced change.

--Briar & Miller, 1971, p. --Briar & Miller, 1971, p. 173173

Note. Includes systematic changes in practice and policy – changes targeting individuals, families, schools and other organizations, neighborhoods, counties, states, countries, and other aggregations.

Etiological and interventive research

Research now exerts an influence on most Research now exerts an influence on most aspects of practice. It has added to our aspects of practice. It has added to our understanding of most of the psychosocial understanding of most of the psychosocial problems we face in our work with clients. It problems we face in our work with clients. It has helped create a range of empirically tested has helped create a range of empirically tested methods….methods….

The not-so-good news is that the amount of The not-so-good news is that the amount of influence that research exerts on practice is influence that research exerts on practice is still quite limited and that much practice is still still quite limited and that much practice is still little affected by research. little affected by research. Reid, W. R. (1997). Long-term trends in clinical social work. Social Service Review, 71, 202.

Status of Intervention Research in Social Work

To affect practice: Place of intervention To affect practice: Place of intervention researchresearch

Source: New Freedom Commission on Mental Health (2005, April). Subcommittee on Evidence-Based Practices: Background Paper (Pub. No. SMA-05-4007). Rockville, MD: DHHS, p. 11.

Sciences to Services Cycle

If you want to understand If you want to understand something, try to change it.something, try to change it.

--Albert Einstein--Albert Einstein

Okay, now…“R” stands for

research and the little bitty “i” stands for intervention

I think I am going to do

more intervention

research

This is good…so you are going to do

intervention research too?

Design and Development Design and Development ApproachApproach

1.1. Problem analysis and project planningProblem analysis and project planning

2.2. Information gathering and synthesisInformation gathering and synthesis

3.3. Design of interventionDesign of intervention

4.4. Early development and pilot testing Early development and pilot testing

5.5. Evaluation and advanced developmentEvaluation and advanced development

6.6. DisseminationDissemination

Rothman, J., & Thomas, E. J. (Eds.). (1994). Intervention research: Design and development for human service. New York, NY: Haworth

The Process of Building Programs from Research

Efficacy Testin

g

Epidemiology

Conceptualization

Program Design

Effectiveness Testin

g

Dissemination

Development of Making Choices

Programs

•Defining the Problem•Incidence•Prevalence

•Risk and Protective Factors•Theory and Risk Chains•Prior Intervention Research•Practice Experience

•Develop treatment manual•Pilot test with single cases•Review by experts/consumers•Collect qualitative data

“Transportability” Problem

Making Choices: Solving Social

Problems Social Skills – information processing and problem-solving skills Emotional regulation skills

understanding feelings, arousal self-talk and other techniques to control impulsive

behavior and arousal

Opportunities for involvement with prosocial peers

Build a sense of community in the classroom through supportive discussion and learning

For reports on Making Choices, see Fraser et al., 2004; Fraser et al., in press; Smokowski et al., 2004

Example: Quasi-Experimental Cluster Design

Setting: Two elementary schoolsClassrooms: 29 (3rd Grade)Teachers: 14Cohort/Cluster Design: (j=classrooms)

Year 1: Routine Services (j=9) Year 2: Making Choices Only (j=9) Year 3: Making Choices Plus (j=11)

Note. Complicated nesting structure of students within classrooms within teachers (raters) within schools.

Sociodemographic Characteristics by School

t=1.188.90 (0.42)8.95 (0.50)8.93(0.50)Age (Mean/SD)

49.3%% (101)51.9% (178)50.9% (279)Male

0.3550.7% (104)48.1% (165)49.1% (269)Female

Sex (%/n)

0.094.4% (9)5.0% (17)4.7% (26)Other

7.58**13.7% (28)23.3% (80)19.7% (108)African American

144.63***65.9% (135)15.5% (53)

34.3% (188)European American, Not Latino

85.44***14.6% (33)56.3% (193)41.2% (226)Latino

Race/Ethnicity (%/n)

65.3***24.8%81.6%52.9%Free or Reduced Lunch

χ2 statistic or t-statistic

School B(n=205)

School A(n=343)

Total Sample(n=548)

Demographic Characteristics

Substantive IssuesSubstantive Issuesin Intervention Researchin Intervention Research

Prevalence:Prevalence:How Big Is the Problem?How Big Is the Problem?

Are there routine surveys?Are there routine surveys? Are there longitudinal studies?Are there longitudinal studies? Are there epidemiological studies?Are there epidemiological studies?

Starting with…

Prevalence Rates of Psychiatric Disorder without Impairment Among School-Age Youth

Sample CharacteristicsSample Characteristics MethodsMethods % of Sample with DSM-III or IV Diagnosis % of Sample with DSM-III or IV Diagnosis

Disruptive Behavior Disruptive Behavior Emotional Emotional

Disorders Disorders

StudyStudy NN AgeAge SampleSample DSMDSM InstrumentInstrument ADHDADHD ODDODD CDCD GADGAD SADSAD DEPDEP

Egger et Egger et al., 2003al., 2003

14221422 9-169-16 NormativNormative sample e sample of rural of rural youthyouth

IVIV CAPACAPA(caregiver (caregiver and youth and youth report; 3 report; 3

mo mo prevalence)prevalence)

7.67.6 24.324.3 12.912.9 5.25.2 7.57.5 12.612.6

Simonoff Simonoff et al., et al., 19971997

27622762 8-168-16 NormativNormative sample e sample

of of CaucasiaCaucasian twinsn twins

III-RIII-R CAPA CAPA (caregiver (caregiver and youth and youth report; 3 report; 3

mo mo prevalence)prevalence)

2.42.4 3.93.9 6.66.6 10.810.8 7.27.2 1.31.3

Garland Garland et al., et al., 20012001

16181618 6-186-18 Indicated Indicated sample sample

of of service-service-involved involved

youthyouth

IVIV DISC-IV DISC-IV (caregiver (caregiver and youth and youth

report; 1 yr report; 1 yr prevalence)prevalence)

24.424.4 17.417.4 24.924.9 1.21.2 4.94.9 6.06.0

Teplin et Teplin et al., al., 2002*2002*

18291829 10-10-1818

Indicated Indicated sample sample

of of juvenile juvenile detaineedetainee

ss

III-RIII-R DISC 2.3 DISC 2.3 (youth (youth

report; 6 report; 6 mo mo

prevalence)prevalence)

18.318.3 15.615.6 38.838.8 7.27.2 15.015.0 12.012.0

Beals et Beals et al., 1997al., 1997

109109 13-13-1818

High-risk High-risk sample sample

of of American American IndiansIndians

III-RIII-R DISC 2.1cDISC 2.1c(youth (youth

report; 6 report; 6 mo mo

prevalence)prevalence)

10.610.6 2.92.9 3.83.8 1.91.9 1.91.9 4.74.7

Note: CAPA=Child and Adolescent Psychiatric Assessment; DISC=Diagnostic Interview Schedule for Children; ADHD=Attention Deficit/Hyperactivity Disorder; ODD=Oppositional Defiant Disorder; CD=Conduct Disorder; GAD=Generalized Anxiety Disorder; SAD=Separation Anxiety Disorder; DEP=Depressive Disorders. *Prevalence estimates reflect rates of diagnosis with impairment.

Copyright restrictions may apply.

Costello, E. J. et al. Arch Gen Psychiatry 2003;60:837-844.

Prevalence of Any Diagnosis and SED by Gender:Finding Compelling Research Questions in Data(SED = serious emotional disturbance/disorder)Why does

“any diagnosis”

for girls accelerate

when it does not for boys?

What could be done to prevent

the change in

trajectory?

Building Conceptual Building Conceptual FrameworksFrameworks

What are the boxes in SW?

Ecological Theory?Ecological Theory?Is it theory? Is it adequate?

Yes, it makes sense.

May be good start. But we need more…

Biological RisksParentingFamily-School Pre-School ClimateNeighborhood

School ReadinessProcessing SkillsParentingFamily-SchoolSchool ClimateNeighborhood

Peer RejectionAcademic FailureParentingFamily-SchoolSchool ClimateNeighborhood

Increasingly Broad Repertoire of Potentially Damaging and

Aggressive Behaviors

Cascades of Risk PerspectiveConceptual Model that includes

non-measured constructs.

Intervention

Features

Intensity

Social - Emotional

CompetenceBeliefs about aggression

Self-efficacyEmpathy

School ClimateEngagement

ConnectednessFeelings of safety

Parental involvementOrganizational climate

BehaviorResponsibilitySelf-regulationCooperation

AltruismAggression

DelinquencyConduct Problems

Disruptive BehaviorVictimization

Academic Achievement

Moderating Factors

Child risk statusParenting practicesHome atmosphere

Community risk and protection

Intervention fidelity/dosageControl school SACD-

activities

IES-US DOE Social and Character IES-US DOE Social and Character Development: Conceptual and Theoretical Development: Conceptual and Theoretical

ModelModelDebate: Should peer and family factors be included?

Slide prepared by Tamara M. Haegerich, Research Scientist, Teaching and Learning Division, National Center for Education Research, Institute of Education Sciences, U.S. Department of Education

Conceptual Model that includes all measured constructs

Simple Conceptual FrameworkSimple Conceptual Framework(also with measured constructs (also with measured constructs

only)only)

SocialInfo

ProcessSkills

EmoRegul

OvertAggre

PeerRejec

t

MakingChoices

Family Processes• Cognitive-emotional

• Problem-solving • Discipline/monitoring

• School involvement• Familism

Child Skills• Social competence

• Information processing• Beliefs about aggression

• Problem-solving

Italics = Program Goals

Social Relations• Social contact• Acceptance by prosocial peers•Teacher-child

closeness

Child Behavior• Emotion regulation • Social competence

• Authority acceptance • Prosocial behavior

• Concentration

Child Predisposition

• Gender• Shyness

• Overactivity• Other

SES

Ethnicity

Making Choices Project: Full Conceptual ModelPotentiating

FactorsPrecursors Distal OutcomesProgram Targets or

Proximal Outcomes

Promoting social development for school success…

Family Stress• Acculturation• Physical needs

• Family structure

• Family size

Parental Beliefs

Logic ModelsLogic Models

The Social Development StrategyThe Social Development Strategy

Individual Characteristics

Opportunities RecognitionSkills

Bonding•Attachment•Commitment

Healthy Behaviors

Healthy Beliefs &Clear Standards

Environmental Conditions Public Policies

Source: Social Development Research Group, School of Social Work, U of WA

Complicated Social Development Complicated Social Development ModelModel

(with only measured constructs)(with only measured constructs)

Source: Social Development Research Group, School of Social Work, U of WA

Bases for Conceptual Bases for Conceptual Frameworks:Frameworks:

Advances in Scientific Advances in Scientific KnowledgeKnowledgeMajor concepts and constructs from:Major concepts and constructs from:

Genetics and biologyGenetics and biology Cognition and neuroscienceCognition and neuroscience Prevention science and outcome Prevention science and outcome

studiesstudies Specific fieldsSpecific fields

Child DevelopmentChild Development AgingAging Substance AbuseSubstance AbuseFor reviews of intervention research in childhood and adolescence, see Allen-Meares, P., & Fraser, M.

W. (Eds.). (2004). Intervention with children and adolescents: An interdisciplinary perspective. Needham Heights, MA: Allyn & Bacon.

Developmental Psychopathology Developmental Psychopathology and Epidemiologyand Epidemiology

Risk factor -- Risk factor -- increase the probability increase the probability of onset, digression to a more serious of onset, digression to a more serious state, or maintenance of a problem state, or maintenance of a problem condition condition

Protective factor -- individual and Protective factor -- individual and environmental resources that reduce environmental resources that reduce riskrisk

Nonlinear Effect of Cumulative Nonlinear Effect of Cumulative RiskRisk

0102030405060708090

0 2 4 6 8

Number of Risk Factors

Poo

r So

cial

Fun

ctio

ning

Low ProtectionHigh Protection

Core Concept: At high levels of risk, few children may be resilient.

Growing Acceptance of a Growing Acceptance of a Risk Factor PerspectiveRisk Factor Perspective

Common language Common language Atheoretical – no particular causal Atheoretical – no particular causal

structuresstructures Cross-disciplinaryCross-disciplinary HoweverHowever: lack of conceptual clarity : lack of conceptual clarity

about risk versus protection (also assets about risk versus protection (also assets and strengths)and strengths) Protective interactionsProtective interactions Promotive main effectsPromotive main effects

For reviews, see Fraser, M. W. (Ed.) (2004). Risk and resilience in childhood. Washington, DC: NASW Press.

Drawing on TheoryDrawing on Theory

Growth of microsocial theories that Growth of microsocial theories that more clearly specify mediating more clearly specify mediating mechanismsmechanisms

Empirical advances in social Empirical advances in social disorganization theory – specifying disorganization theory – specifying mediating processes at the mediating processes at the community levelcommunity level

More than ecological theoryMore than ecological theory

Biopsychosocial Model: Three Biopsychosocial Model: Three FindingsFindings

1.1. Genetic risk is environmentally Genetic risk is environmentally mediatedmediated

2.2. Cognitive processes mediate early risk Cognitive processes mediate early risk and later developmental outcomesand later developmental outcomes

Social knowledgeSocial knowledge ScriptsScripts SchemaSchema Patterns in processing informationPatterns in processing information

3.3. Cumulative risk Cumulative risk maymay be more/as be more/as important than specific risk factorimportant than specific risk factor

Sources: Dodge & Pettit, 2003; Rutter, 2005; Sameroff & Gutman, 2004

Theoretical Bases: Integrating Social Information Processing Theory with Emotional Regulation

State the problem

Generate

potentialsolutions

Evaluate potentialsolutions

Select &enact the

best solution(s)

Assessoutcomes

Encode social cues

Interpretsocial cues

Arousal, Emotions,Social Knowledge

Setgoal(s)

Social Knowledge: Life experiences producing scripts, schemata, skills, and beliefs

Risk MechanismRisk Mechanism……a sequence or chain of risk factors a sequence or chain of risk factors

that significantly elevates vulnerability that significantly elevates vulnerability for a social or health problemfor a social or health problem Macro-social – the relationship of Macro-social – the relationship of

neighborhood cohesion to informal social neighborhood cohesion to informal social control and the development of youth control and the development of youth gangsgangs

Mirco-social – early behavioral and Mirco-social – early behavioral and communication patterns that lead to peer communication patterns that lead to peer rejection, association with rejected rejection, association with rejected children (exclusive classroom children (exclusive classroom atmospheres), and “deviant talk” that atmospheres), and “deviant talk” that reinforces problem thinking and behaviorreinforces problem thinking and behavior

Disrupting Risk Mechanisms: Disrupting Risk Mechanisms: Specifying Mediation in Specifying Mediation in

Intervention?Intervention? The process through which change is thought to The process through which change is thought to

occur; understanding how and why a treatment occur; understanding how and why a treatment effect is observedeffect is observed

Methodologically, it requires:Methodologically, it requires: Strong association – Baron & Kenny vs. Shrout & Strong association – Baron & Kenny vs. Shrout &

BoglerBogler Specificity – identifying and measuring key constructsSpecificity – identifying and measuring key constructs Gradient – showing dose-response relationshipGradient – showing dose-response relationship Experiment – factorial designs (stats aren’t enough)Experiment – factorial designs (stats aren’t enough) Time order – implies continuous measurementTime order – implies continuous measurement Replication – consistency across samples (SIP?)Replication – consistency across samples (SIP?) Credibility – plausible and coheres with theory?Credibility – plausible and coheres with theory?

Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy. Journal of Child Psychology and Psychiatry, 44, 1116-1129.

Why specify and study risk Why specify and study risk mechanisms?mechanisms?

Understand latent processes – the active Understand latent processes – the active ingredients – producing positive findingsingredients – producing positive findings

Optimize effects of treatment (i.e., by Optimize effects of treatment (i.e., by narrowing the focus of treatment)narrowing the focus of treatment)

Identify moderators – factors on which Identify moderators – factors on which success may depend (e.g., child age)success may depend (e.g., child age)

Describe helpful processes that may Describe helpful processes that may exist outside of treatment (i.e., effective exist outside of treatment (i.e., effective interventions may represent health-interventions may represent health-promoting, adaptational processes that promoting, adaptational processes that operate normatively to promote operate normatively to promote resilience) resilience) Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and

adolescent therapy. Journal of Child Psychology and Psychiatry, 44, 1116-1129.

Fish sticks for dinner?Fish sticks for dinner?

Risk Mechanism in the Strong Families program…

For a report on Strong Families, see Fraser et al., 2004

Coercion Risk MechanismCoercion Risk Mechanism1. Parents makes request.1. Parents makes request.2. Child responds coercively 2. Child responds coercively (aversive escalation).(aversive escalation).3. Parent acquiesces 3. Parent acquiesces (child escapes contingency).(child escapes contingency).4. Parent is frustrated. Anger builds as 4. Parent is frustrated. Anger builds as

child continues to ignore request.child continues to ignore request.5. Parent responds with force (and is 5. Parent responds with force (and is

increasingly distanced from child).increasingly distanced from child).6. But parent behavior is rewarded by the 6. But parent behavior is rewarded by the

child’s coerced compliance.child’s coerced compliance.

Challenge: Identify and interrupt risk mechanisms…

Implications for Practice: Implications for Practice: Must Disrupt Risk Mechanisms…Must Disrupt Risk Mechanisms… Assess for cumulative risk and protectionAssess for cumulative risk and protection Identify malleable risk/protective factors Identify malleable risk/protective factors

and mechanismsand mechanisms Focus on family with sensitivity to Focus on family with sensitivity to

race/ethnicity, culture, gender, agerace/ethnicity, culture, gender, age Strengthen cognitive and other skills that Strengthen cognitive and other skills that

mediate early experience and behaviormediate early experience and behavior Change environmental conditions that Change environmental conditions that

interact with bio-risk and elevate stressinteract with bio-risk and elevate stress

Stress-Vulnerability Model with Cognitive Features?

What is your conceptual What is your conceptual model?model?

What risk mechanism(s) to disrupt?What risk mechanism(s) to disrupt?

Treatment ManualsTreatment Manuals From risk mechanisms and the From risk mechanisms and the

conceptual framework to treatment conceptual framework to treatment activitiesactivities

Example from Making Choices Example from Making Choices ProgramProgram

For a discussion of issues in the development and use of treatment manuals, see: Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (in press). The art of group work practice with manualized curricula. Social Work with Groups.

Warning: It is easy to under estimate the difficulty of developing a treatment manual.

Recognizing Your Feelings

Objectives:

The learner will recognize that certain situations bring out feelings in all of us. The learner will practice recognizing their own feelings. The learner will use personal experiences and knowledge to interpret written

and oral messages. (SCS- LA 3.01) The learner will write structured, informative presentations and narratives

when given help with organization. (SCS- LA 4.08)

Materials:

Penguin Facts page, Response Sheets, Write About It worksheets A and B

Introduction

Review the idea that we all experience a variety of emotions and responses to emotions. Even when we experience the exact same situation, we may have different responses to the situation. Our responses to our feelings can cause us to do good things, but at times they can also cause us to do things that are not helpful.

Activity I: Pete the Penguin

Using two columns, list on the board the emotions presented in Lesson 1 of the book, The Way I Feel. Column I- Emotions that Feel Good: happy, silly, excited, proud, or

thankful Column II- Emotions that Don’t Feel Good: scared, sad, disappointed, bored, angry, or jealous Introduce the students to Pete the Penguin using the penguin puppet. Pass out the Penguin Facts page and discuss the factual information about penguins. Explain to the students that Pete has experienced events that have brought out many different emotions. Sometimes his emotions feel good, but at other times they don’t feel very good at all.

Review the emotions listed in the columns on the board. Then give each student four small pieces of paper (about the size of a note card). Read aloud the following events involving Pete the Penguin. After reading each event, ask the students, “How would you feel?” Give the students enough time to record their responses on one of

Grade 2Grade 2 Lesson Lesson 22

Activity 1

Overview

Review

PropAnswers

Process Tip

Standard Course

of StudyPrep Material

s

their blank pieces of paper. They can use the emotions on the board to express how they would feel or they may provide their own responses.

After you read each situation, collect a few responses randomly and read them aloud (so as not to bring attention to specific student responses). As you read through each response, discuss whether the event brought out a good feeling or a not-so-good feeling. The texts are ambiguous so that students can develop their own interpretations—not all students will feel the same way about each situation. Discuss the idea that everyone heard the same event, yet the feelings were different in many instances.

Today Pete walked in the classroom. As he walked to his desk, Pete noticed Susan and Tony talking quietly and laughing. They both looked up at Pete and giggled. If you were Pete, how would you feel?

When Pete was on the playground, he saw a group of students playing ball. He went to join them, and they told him he could play as soon as they started the next game. If you were Pete, how would you feel?

At lunch, Pete was sitting next to Jermaine. Jermaine opened his lunch and Pete looked inside. All he saw was two cookies and a drink box. If you were Pete, how would you feel?

Pete’s teacher told him he could play a game with Juan as soon as he finished his writing assignment. If you were Pete, how would you feel?

After discussing the above events, ask the students how they recognize when they are feeling certain emotions. “What happens when you start to feel angry?” “Happy?” “Frustrated?” and so on. (Example response: When you are getting angry- you might get hot, start to shake, get tense, grit your teeth, etc.)

Leave the list of emotions on the board to use in Activity II .

Activity II: Write About It

Give the students the Write About It page. On the top of the sheet, have students write about an event in their life that caused them to experience an emotion that made them feel good. On the bottom of the sheet they can write about an experience that caused an emotion that didn’t feel good. Each

narrative should describe the emotion, what caused it, and how they responded to the emotion. Students can refer to the columns on the board to choose the emotions they want to write about. Share the following examples aloud or on a transparency: Example 1: Once I felt excited when I was going to my friends party. I knew I felt this way because I was smiling and jumping around.

Avoid labeling

Scenarios

Activity 2: Write About It!

G r a d e 2 - L e s s o n 2

A good feeling: Once I felt ___________________ when ____________________________________ ____________________________________ ____________________________________ ____________________________________ I knew I felt this way because__________ ____________________________________ ____________________________________

Activity II: Sheet A NAME: _________________

A not so good feeling: Once I felt _____________________ when ___________________________________ ___________________________________ ___________________________________ ___________________________________ I knew I felt this way because__________ ___________________________________

___________________________________

Activity II: Sheet B NAME: ____________________

““Pete the Penguin” Poster for Pete the Penguin” Poster for Grade 2Grade 2

Methodological IssuesMethodological Issuesin Intervention Researchin Intervention Research

Three Key QuestionsThree Key Questions

1.1. What are the impacts on individuals What are the impacts on individuals (and organizations or other units of (and organizations or other units of analysis)?analysis)?

2.2. What works for whom and under What works for whom and under what conditions?what conditions?

3.3. Are impacts on mediators Are impacts on mediators consistent with impacts on longer consistent with impacts on longer term outcomes?term outcomes?

For a general reference, see Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.

Question 1:Question 1:What are the impacts?What are the impacts?

Assessing the Impacts:Assessing the Impacts:The Counterfactual QuestionThe Counterfactual Question

What would have happened to the people who received an intervention in the absence of the intervention?

(or counter to the fact that they received (or counter to the fact that they received an intervention)an intervention)

Related questions:

•What is the average treatment effect?

•What is the average treatment effect among the treated?

•What is the average treatment effect in the control group?

Is it plausible that the Is it plausible that the apparent impact is due to the apparent impact is due to the

intervention?intervention? Advances in plausibility argumentsAdvances in plausibility arguments Clustering – nesting of effects; group Clustering – nesting of effects; group

level random assignment (cluster level random assignment (cluster randomized designs) versus individual randomized designs) versus individual randomization; cluster multipliers; randomization; cluster multipliers; hierarchical linear modelshierarchical linear models

Attrition bias – “missingness” and Attrition bias – “missingness” and imputationimputation

Selection bias – designs when random Selection bias – designs when random assignment fails or is not possibleassignment fails or is not possible

ClusteringClustering Problem: The scores of individuals within the same Problem: The scores of individuals within the same

cluster may be correlatedcluster may be correlated Types of clusteringTypes of clustering

Individual – scores over timeIndividual – scores over time Classrooms – scores across childrenClassrooms – scores across children Organizations – scores within agencies, hospitals, etc.Organizations – scores within agencies, hospitals, etc. Geographic units – scores within neighborhoodsGeographic units – scores within neighborhoods Raters or workers – scores by same teacher, case Raters or workers – scores by same teacher, case

managermanager Intracluster correlation (ICC)Intracluster correlation (ICC)

Rho (Rho (ρρ) = the proportion of total variation explained by ) = the proportion of total variation explained by variation across clusters (variation between clusters = variation across clusters (variation between clusters = ττbb

22))

For an overview, see: Nash, J. K., Kupper, L. L., & Fraser, M. W. (2004). Using multilevel statistical models in social work intervention research. Journal of Social Service Research, 30(3), 35-54.

ρρ = = ττbb22/(/(ττbb

22 + + σσww22))

Degree to which variation in outcomes is stratified by cluster

Design or Cluster Effect Design or Cluster Effect MultiplierMultiplier

A high ICC means that clusters share A high ICC means that clusters share more variance – are more homogenousmore variance – are more homogenous

If SEs are computed as you might for a If SEs are computed as you might for a design with randomization at the design with randomization at the individual level, the SEs for the individual level, the SEs for the treatment parameter will be too small.treatment parameter will be too small.

This may produce a significant This may produce a significant treatment effect when none exists.treatment effect when none exists.

Note. When properly estimated, the SEs in a cluster randomized design will be larger than the SEs in an individual randomized design. The amount of inflation will be related to the number of clusters, the number of people within clusters, and the size of rho. To reduce the design effect (or inflation of the variance), it is more helpful to increase the number of clusters than to increase the size of clusters, holding rho constant.

Cluster Effect MultipliersCluster Effect Multipliers

Note. Size of the SE of the treatment parameter in a cluster randomized design (if properly estimated by adjusting for rho and size of cluster) relative to the size of the SE for the treatment effect if estimated in a randomized individual trial (without clustering).* Source: Bloom, 2005, p. 128

Cluster Size (n)Cluster Size (n)

ICC (ICC (ρρ)) 1010 2020 5050

.00*.00* 1.001.00 1.001.00 1.001.00

.02.02 1.091.09 1.171.17 1.411.41

.05.05 1.201.20 1.401.40 1.861.86

.08.08 1.311.31 1.591.59 2.222.22

.10.10 1.381.38 1.701.70 2.432.43

.20.20 1.671.67 2.192.19 3.293.29

Interpretation: Given an ICC of .10 and 20 students per

classroom, the standard error of

intervention parameter for

cluster randomization is

1.70 times as large as it is for

individual randomization.

SACD ICCs by School: Variation by SACD ICCs by School: Variation by Source?Source?

SourceSource MeasureMeasure Mean ICCMean ICC RangeRange

CaregiveCaregiverr

AggressionAggression .01.01 .00-.04.00-.04

Social Social CompetencCompetencee

.05.05 .00-.13.00-.13

ChildChild AggressionAggression .06.06 .01-.18.01-.18

Self Self EfficacyEfficacy

.01.01 .00-.03.00-.03

AltruismAltruism .05.05 .00-.12.00-.12

TeacherTeacher Conduct Conduct ProblemsProblems

.11.11 .06-.25.06-.25

Social Social CompetencCompetencee

.09.09 .05-.14.05-.14

AltruismAltruism .10.10 .09-.14.09-.14

How does CEM affect MDE?How does CEM affect MDE?Smallest detectable effect given α, power, df, 1 or 2-tailed test, and SE

MDEMDE

(at (at ρρ=.05)=.05) Size of ClustersSize of Clusters

# Clusters# Clusters 1010 2020 5050 100100

44 2.042.04 1.671.67 1.411.41 1.311.31

66 1.161.16 .95.95 .80.80 .74.74

88 .90.90 .74.74 .62.62 .58.58

1010 .77.77 .63.63 .53.53 .49.49

2020 .50.50 .41.41 .35.35 .32.32

3030 .40.40 .33.33 .28.28 .26.26

4040 .35.35 .28.28 .24.24 .22.22Source: Bloom, 2005, p. 133 (based on power at .80, α=.05, 2-tailed tests, and randomization of half the clusters to the treatment condition)

Medium effect size

Better effect size but requires 800 more

participants (40 new participants per cluster x

20 clusters)Same effect size but

requires only 200 more participants (20

additional clusters x 10 participants per cluster)

Accounting for ICCs in Analysis: 2-Level Accounting for ICCs in Analysis: 2-Level HLMHLM

Level 1 (individual): POSTij = β0j + β1j Preij + β2j AFRij + β3j LATij + β4j MALEij + rij

Level 2 (classroom – random intercept model): β0j = π00 + π01MCj + π02MCPj + π03 SCHj + u0j

And…for models with random intercepts and slopes Cognitive concentration: β2j = π20 + u2j (random slope for race/ethnicity)

1j = π10 + π11MCj + π12MCPj (interaction with pretest)

Authority acceptance, social contact, CBCL aggression: β1j = π10 + u1j (random slope for pretest)

Note. The effects of MC and MC+ are estimated as one-tailed tests at the classroom level. For a comparable example, see Raudenbush and Bryk (2002, pp. 112-113).

Covariates reduce cluster multiplier by reducing unexplained variance

Intervention at Level 2: Do you have a df problem?

Notes: Cluster Randomized Notes: Cluster Randomized DesignsDesigns

Determine intervention levelDetermine intervention level Estimate ICC (in advance and empirically)Estimate ICC (in advance and empirically) Adjust for cluster effect multiplier (i.e., Adjust for cluster effect multiplier (i.e.,

big SE—low power = add more clusters?) big SE—low power = add more clusters?) Consider adding covariates (at cost of Consider adding covariates (at cost of

one df per one df per clustercluster characteristic) characteristic) Consider one-tailed tests for Consider one-tailed tests for

interventions interventions Consider effects sizes using a different Consider effects sizes using a different

(non-Cohen) metric(non-Cohen) metric

For review, Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom, H. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115-172). NY: Russell Sage Foundation.

Attrition:Attrition:Are impacts due to

differential loss of high risk participants or differential

missing data?

Types of AttritionTypes of Attrition

Measurement – loss of dataMeasurement – loss of data Treatment – loss of Treatment – loss of

participantsparticipants“Missingness”

For an example of multiple imputation, see Orlando, M., Ellickson, P. L., McCaffrey, D. F., & Longshore, D. L. (2005). Mediation analysis of a school-based drug prevention program: Effects of Project ALERT. Prevention Science, 6(1), 35-46.

Multiple imputation

This slide is drawn from a presentation by Dr. Joseph L. Schafer, Pennsylvania State University, Department of Statistics and NIDA Center for the Study of Prevention through Innovative Methodology. See: http://www.stat.psu.edu/~jls/

Create m new data sets of simulated

valuesEstimate parameters by averaging across

parameter estimates from each simulated data set

This slide is drawn from a presentation by Dr. Joseph L. Schafer, Pennsylvania State University, Department of Statistics and NIDA Center for the Study of Prevention through Innovative Methodology. See: http://www.stat.psu.edu/~jls/

Estimate parameters

in simulated data set #1

Estimate parameters

in simulated

data set # 2

Estimate parameters

in simulated data set #m

Selection Effects:Selection Effects:Are impacts due to (unobserved) pre-

intervention and/or post-randomization differences between treatment and

control (comparison) conditions?

Selection BiasSelection Bias When the average person in an When the average person in an

intervention condition differs from an intervention condition differs from an average person in a control or comparison average person in a control or comparison conditioncondition Failed random assignmentFailed random assignment ““Randomization” biasRandomization” bias ““Treatment substitution” biasTreatment substitution” bias Treatment attritionTreatment attrition OtherOther

Hawthorne Effects – in treatment groupHawthorne Effects – in treatment group John Henry Effects – in control groupJohn Henry Effects – in control group Contamination or Spillover Effects – in control groupContamination or Spillover Effects – in control group

For review, Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. NY: Houghton Mifflin.

There is a sizable divergence between the theoretical

capabilities of evaluations based on random assignment

and the practical results of such evaluations. Heckman & Smith, 1995, p. 107

James J. Heckman, Ph.D.

Nobel Prize Laureate, 2000

Lots of action and Lots of action and controversy…controversy…

Traditional OLS with covariatesTraditional OLS with covariates Matching on observed variables (1:1, 1:n)Matching on observed variables (1:1, 1:n) Instrumental variables – 2SLSInstrumental variables – 2SLS Propensity score matching onPropensity score matching on

1:11:1 1:n1:n Local geographic area vs. notLocal geographic area vs. not In quintilesIn quintiles Differentially weightedDifferentially weighted

For review, see Bloom, H. S., Michalopoulos, & Hill, C. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 173-235). NY: Russell Sage Foundation. For instrumental variables: http://en.wikipedia.org/wiki/Instrumental_variables_estimation

Propensity ScorePropensity Score

… …the predicted probability of the predicted probability of being in the treatment or being in the treatment or comparison groupcomparison group

Or…trying to model the selection bias and then using it to create more balanced conditions by matching.

Caliper MatchingCaliper Matching Nearest neighborNearest neighbor ““Kernel” matching within bandwidthKernel” matching within bandwidth Forced nearest neighbor matchingForced nearest neighbor matching IssuesIssues: :

Use of multiple versus single matchesUse of multiple versus single matches Use of replacement/resamplingUse of replacement/resampling Bias correction when matching is not exactBias correction when matching is not exact Variance estimationVariance estimation Weighting to produce population parametersWeighting to produce population parameters

For a review, see Abadie, A., Drukker, D., Herr, J. L., & Imbens G. W. (2001). Implementing matching estimators for average treatment effects in Stata. The Stata Journal, 1, 1-18.

Participants Nonparticipants

Predicted Probability

Range of matched cases.

Cases excluded

Cases May Be Excluded at Cases May Be Excluded at Both Ends of the Propensity Both Ends of the Propensity

ScoreScore

Heckman’s Difference-in-Heckman’s Difference-in-Differences Matching EstimatorDifferences Matching EstimatorDifference-in-differencesApplies when each participant matches to multiple nonparticipants.

Participant i in the set of common-support.

Participant i in the set of common-support.

pp SIj

jttjSIi

ittiKDM YYjiWYYn

0

'

1

' )})(,(){(1

00011

Multiple non-participants who are in the set of common-support (matched to i).

Multiple non-participants who are in the set of common-support (matched to i).

Difference Differences…….in……………

Total number of participants

Total number of participants

Weight (estimated different ways)

Weight (estimated different ways)

Question 2:Question 2:What works for whom?What works for whom?

Is the impact different for Is the impact different for different participants?different participants?

Is the impact different at Is the impact different at different dosage levels?different dosage levels?

Intent-To-Treat (ITT)Intent-To-Treat (ITT)

Inclusion of all participant data in an Inclusion of all participant data in an analysis, regardless of the degree of analysis, regardless of the degree of compliance or participation in an compliance or participation in an intervention*intervention* Includes low exposure to interventionIncludes low exposure to intervention Includes high exposure to intervention in control Includes high exposure to intervention in control

group (experimental contamination)group (experimental contamination) Preserves equivalence based on randomizationPreserves equivalence based on randomization Produces “honest” parameter estimates for clinical Produces “honest” parameter estimates for clinical

trials (FDA and med journals)trials (FDA and med journals) Has high “ecological validity” for when an Has high “ecological validity” for when an

intervention is brought to scale as public policyintervention is brought to scale as public policyNoteNote. Exceptions include death, withdrawal of consent, and lost to follow-. Exceptions include death, withdrawal of consent, and lost to follow-

up.up.

Critique of ITT approachCritique of ITT approach

Not a true clinical estimate of the effect (as intended by developers) – Unfair!

Confounds motivation to participate and adherence with effect estimate

Confounds institutional environment with effect estimate

ITT does not produce valid effect estimates (anyway) because efforts to increase adherence in RCTs (e.g., food, transportation) cannot be replicated in real worldGross, D., & Fogg, L. (2004). A critical analysis of the intent to treat principle Gross, D., & Fogg, L. (2004). A critical analysis of the intent to treat principle in prevention research. in prevention research. Journal of Primary Prevention, 25Journal of Primary Prevention, 25, 475-489., 475-489.

Efficacy Subset AnalysisEfficacy Subset Analysis

Selection of a subset of participants Selection of a subset of participants based on criteria identified post-based on criteria identified post-randomization to examine the effect of randomization to examine the effect of experimental treatment (Lachin, 2000).experimental treatment (Lachin, 2000).

Key feature: Subset excludes participants with little Key feature: Subset excludes participants with little or no exposure to treatment to test performance or no exposure to treatment to test performance under optimal conditions.under optimal conditions.

Note.Note. Similar terms are “per protocol analysis,” “explanatory Similar terms are “per protocol analysis,” “explanatory analysis,” “analysis by treatment received,” and “treatment of the analysis,” “analysis by treatment received,” and “treatment of the treated.”treated.”

Making ChoicesMaking Choices ESA Findings ESA Findings (2-level HLMs with Dichotomous Treatment Variables for MC & MC+)(2-level HLMs with Dichotomous Treatment Variables for MC & MC+)

ITT (>0%) >0% >75% >80% >85% >90% >95%

  MC MC+ MC MC+ MC MC+ MC MC+ MC MC+ MC MC+ MC MC+

AA .03 .12 .05 .18†† .05 .18†† .05    .18†† .05 .18†† .05  .18†† .05 .20*

CC .19 .30* .25†† .32* .24†† .32*  .24††  .31* .25†† .35* .28*  .36**  .26†† .37*

SC .29* .36* .28†† .37* .30* .36*  .30* .37*  .29†† .37*  .28†† .37*  .31* .40**

N 

548  539  

524   

 522 

513  

 487 

445 

AA=Authority Acceptance; CC=Cognitive Concentration; SC=Social Competence

One-tailed test of significance at alpha=.05.

So, which analyses to believe: So, which analyses to believe: ITT or one of the ESAs? ITT or one of the ESAs?

Issues with ESA

Selection Bias – Findings could be due to the loss of higher-risk children and not due to increased exposure to intervention

Power and MDE – Sample size, effect size, and variance all change; this may cause a loss of power and an increased MDE (Minimal Detectable Effect)

How to proceed?How to proceed?

Need 1: Analyze causal effects of Need 1: Analyze causal effects of treatment on treated (TOT), andtreatment on treated (TOT), and

Need 2: Reduce selection biasNeed 2: Reduce selection bias

Propensity Score Matching???

Problem: How to select the proper subset Problem: How to select the proper subset from the control (or comparison) group when from the control (or comparison) group when there is no ready way to determine who there is no ready way to determine who would have participated at, say, the 90-100% would have participated at, say, the 90-100% exposure levelexposure level

MC only (Pretest)MC only (Pretest) LogitLogit MC Plus (Pretest)MC Plus (Pretest) LogitLogit

Authority Authority AcceptanceAcceptance 1.19*

Authority Authority AcceptanceAcceptance 0.88

Cognitive Cognitive ConcentrationConcentration 0.11

Cognitive Cognitive ConcentrationConcentration 0.01

Social ContactSocial Contact 0.07 Social ContactSocial Contact -0.43*

Social CompetenceSocial Competence -0.25 Social CompetenceSocial Competence 0.21

Social AggressionSocial Aggression 0.28 Social AggressionSocial Aggression 0.28

CBCL AggressionCBCL Aggression 1.18 CBCL AggressionCBCL Aggression 1.57

* p<.05, ** p<.01, *** p<.0001* p<.05, ** p<.01, *** p<.0001

Logistic Regression to Test for Selection Logistic Regression to Test for Selection Bias Bias MC only v. Comparison and MC PlusMC only v. Comparison and MC Plus

v. Comparison with 90-100% subsetv. Comparison with 90-100% subset

DID findings for MC only and MC PlusDID findings for MC only and MC Pluswith the 90-100% Subsetwith the 90-100% Subset

PSM/DID for PSM/DID for MC only (N only (NMC MC

= 141)= 141)PSM/ DID for MC Plus PSM/ DID for MC Plus

(N(NMC+MC+ = 169)= 169)

Outcome Change est. 95% CI^ est. 95% CI^

Authority Acceptance .062 (-.075, .168) .175* (.082, .246)

Cognitive Concentration .185 (-.007, .303) .282* (. 088, .636)

Social Competence .247 (-.063, .389) .418* (. 236, .556)

Social Contact .379* (.148, .558) .337* (. 191, .642)

Social Aggression -.047 (-.201, .185) -.232* (-.316, -.074)

CBCL Aggression -.081* (-.153, -.020) -.080* (-.141, -.031)

**pp<.05<.05^ 95% bias-corrected CI estimated by bootstrapping method^ 95% bias-corrected CI estimated by bootstrapping method+Comparability between PSM/DID and other methods based on coefficient direction & significance+Comparability between PSM/DID and other methods based on coefficient direction & significance

Draft: Do Not Cite or Distribute

Question 3:Question 3:

Mediation Mediation Are impacts on proximal outcomes consistent with

impacts on long term outcomes?

(Or: What “causes” the observed impacts?)

Defining Experimental Defining Experimental MediationMediation

Where,

X = intervention

Y = outcome

M = mediatorNote: Mediation is said to occur when c’ < c Source: Shrout & Bolger, 2002, p.

423.

c = total effect

c’ = direct effect controlling for a x b

Efficacy test(Does

intervention affect the

outcome?)

Intervention Test(Does treatment

change mediator?)

Mediator Change Test(Does mediator affect distal

outcome?)

Mediation Test(Does c’ decrease

after controlling M?)

Distal outcomeMediator or proximal outcome – risk mechanism

Source: Shrout & Bolger, 2002, p. 433.

America Online 9.0.lnk

Many Mediational ProcessesSuppression

Trouble: When the sign

of c ’ is opposite the sign of a x b

Source: Shrout & Bolger, 2002, p. 438.

Bootstrap-based versus Standard Normal Theory-based Inference

•SE of a x b not normally distributed

•Ignoring this reduces power

•Solution: Bootstrap estimates and CIs

Skip if theoretically relevant and more distal outcome?

Do you have a mediational conceptual framework, based on a specified risk mechanism?

Mediation Decision Tree

Mediation vs. ModerationMediation vs. Moderation

Mediation and Mediation and TransportabilityTransportability

If mediational processes can be If mediational processes can be identified, may improve identified, may improve transportability of interventionstransportability of interventions More focused treatment manualsMore focused treatment manuals Training based on principles, e.g., must Training based on principles, e.g., must

improve SIP skills (any way you can!)improve SIP skills (any way you can!) Principle based training allows for local Principle based training allows for local

adaptationsadaptations

For an example of mediation analysis, see Orlando, M., Ellickson, P. L., McCaffrey, D. F., & Longshore, D. L. (2005). Mediation analysis of a school-based drug prevention program: Effects of Project ALERT. Prevention Science, 6(1), 35-46.

Special Issues in Intervention Special Issues in Intervention ResearchResearch

Treatment fidelityTreatment fidelity Method varianceMethod variance Culture and construct validityCulture and construct validity EthicsEthics

Treatment Fidelity and Treatment Fidelity and AdherenceAdherence

Did you complete

the protocol?

Treatment ImplementationTreatment Implementation

Clarity of program elementsClarity of program elements Training and clinical supervision of Training and clinical supervision of

staffstaff Management of organizational Management of organizational

contingenciescontingencies Tenacious program monitoring and Tenacious program monitoring and

data collection on implementationdata collection on implementation ALSO ALSO adherence to a program by adherence to a program by

participantsparticipants

Unobserved Adherence Unobserved Adherence Model:Model:

Treatment Manual Is Not Treatment Manual Is Not EnoughEnough

What the practitioner says when What the practitioner says when she first meets the parents, she first meets the parents,

How she responds to the first How she responds to the first few things the parent says, few things the parent says,

Whether the parent regards her Whether the parent regards her as having good intentions,as having good intentions,

Unobserved Adherence Unobserved Adherence ModelModel

Whether the parent has more Whether the parent has more pressing problems, pressing problems,

Whether the parent’s children are Whether the parent’s children are being adequately supervised, being adequately supervised,

Whether the time of the training Whether the time of the training is convenient, is convenient,

Whether transportation is Whether transportation is available, and so on.available, and so on.

To improve adherence…To improve adherence… Personalized approach to recruitment Personalized approach to recruitment

and retentionand retention Culturally-sensitive, indigenous staffingCulturally-sensitive, indigenous staffing Timing and targeting of program Timing and targeting of program

effortsefforts to key transitions (e.g., to key transitions (e.g., graduation to middle school)graduation to middle school)

Matching of participants’ interests to Matching of participants’ interests to program contentprogram content (e.g., promoting (e.g., promoting children’s achievement)children’s achievement)

Extensive monitoring (weekly data?)Extensive monitoring (weekly data?) Source: Source: Prinz, Smith, Dumas, Laughlin, White, & Barron, Prinz, Smith, Dumas, Laughlin, White, & Barron,

20012001

Method VarianceMethod Variance

Method VarianceMethod Variance

……variation in scores due to variation in scores due to the reporting agent, context, the reporting agent, context, strategy, scaling protocol, strategy, scaling protocol, and length of recall required and length of recall required of raters.of raters.

Source: Eddy, Dishion, & Stoolmiller, 1998, p. Source: Eddy, Dishion, & Stoolmiller, 1998, p. 6161

Method Variance = Complicated Findings and Fuzzy Plausibility

Culture and Construct Culture and Construct ValidityValidity

DefinitionDefinition

Culture – values, norms, Culture – values, norms, expectations, and behaviors expectations, and behaviors that are at once transmitted that are at once transmitted from parents to children and from parents to children and influenced by the changing influenced by the changing characteristics of the contextcharacteristics of the context

Source: Source: Hughes & Seidman, Hughes & Seidman, 20022002

Construct Validity and Construct Validity and CultureCulture

Linguistic equivalence Linguistic equivalence Do words carry the same meanings and nuances?Do words carry the same meanings and nuances?

RelevanceRelevance Design of interventions – NabilaDesign of interventions – Nabila Selection of measuresSelection of measures

Structural (scale) equivalenceStructural (scale) equivalence Same number of constructs?Same number of constructs? Same relationships across constructs?Same relationships across constructs? Comparable factor loadings?Comparable factor loadings?

Source: Source: Hughes & Seidman, 2002Hughes & Seidman, 2002

Culturally-based Culturally-based Constructs?Constructs?

Relational aggression Relational aggression FamilismFamilism PersonalismoPersonalismo Racial socialization Racial socialization Ethnic identityEthnic identity Many othersMany others … …

Ethical IssuesEthical Issues Competing financial and other interestsCompeting financial and other interests

in developing ‘successful’ interventionsin developing ‘successful’ interventions Accrued academic status and honorsAccrued academic status and honors Royalties and speaking honorariaRoyalties and speaking honoraria Consulting and training feesConsulting and training fees Privatization of training and other fiscal gain (equities)Privatization of training and other fiscal gain (equities)

Goal: Minimize the extent to which financial Goal: Minimize the extent to which financial considerations affect design and conduct of considerations affect design and conduct of studiesstudies Disclosure of financial interestsDisclosure of financial interests Role of fiduciary (trusted to act on behalf of others for Role of fiduciary (trusted to act on behalf of others for

their benefit)their benefit) Insure sound scienceInsure sound science Maximize protection of human subjectsMaximize protection of human subjects

Key Question: Will a competing interest create or have reasonable potential to create a bias?

Programs of Sequential Programs of Sequential ExperimentationExperimentation

In the course of your career…In the course of your career… Write at least one treatment manualWrite at least one treatment manual Refine and enlarge it slowly with many studiesRefine and enlarge it slowly with many studies

Characteristics of needed interventions…Characteristics of needed interventions… Conceptual framework rooted in Conceptual framework rooted in

research/theory identifying core constructs and research/theory identifying core constructs and risk mechanismsrisk mechanisms

Treatment manual rooted in conceptual Treatment manual rooted in conceptual framework – disrupts risk processesframework – disrupts risk processes

Measurement model based on proximal/distal Measurement model based on proximal/distal outcomes from conceptual frameworkoutcomes from conceptual framework

Address three key analytic questions (impact, Address three key analytic questions (impact, different impacts by…, proximal-distal different impacts by…, proximal-distal mediation)mediation)

… …group experimental research has group experimental research has represented a major departure fromrepresented a major departure from earlier experiments in social work, inearlier experiments in social work, in which researchers were cast primarilywhich researchers were cast primarily in the role of evaluators with littlein the role of evaluators with little involvement in the design and operation of theinvolvement in the design and operation of the service programs… A particular strength of theservice programs… A particular strength of the [emerging] model is that [emerging] model is that it enables researchers it enables researchers to design and shape their own interventions andto design and shape their own interventions and then test themthen test them. . (p. 180)(p. 180)

-- Bill Reid, 1928-2003-- Bill Reid, 1928-2003

Reid, W. J. (1994). The empirical practice movement. Social Service Review, 68, 165-184.

Practice-Research Challenge: Design and Develop Interventions

To do this, you have to know a little bit about everything – the problem, potential interventions

(the “big” idea), conceptual frameworks, treatment manuals, research design, statistics,

treatment fidelity, politics (in working with agencies), and ethics. It is very hard…but very rewarding. I hope the challenge (and puzzle) of

intervention research becomes part of your career!

Developing Interventions with Developing Interventions with Advanced Research MethodsAdvanced Research Methods

ReferencesReferencesAbadie, A., Drukker, D., Herr, J. L., & Imbens G. W. (2001). Implementing matching estimators for

average treatment effects in Stata. The Stata Journal, 1, 1-18.Allen-Meares, P., & Fraser, M. W. (Eds.). (2004). Intervention with children and adolescents: An

interdisciplinary perspective. Needham Heights, MA: Allyn & Bacon.Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom

(Ed.), Learning more from social experiments (pp. 115-172). NY: Russell Sage Foundation.Bloom, H. S., Michalopoulos, & Hill, C. (2005). Using experiments to assess nonexperimental

comparison-group methods for measuring program effects. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 173-235). NY: Russell Sage Foundation.

Briar, S., & Miller, H. (1971). Problems and issues in social casework. NY: Columbia University Press.

Crick, N. R., & Dodge, K. A. (1994). A review and reformulation of social information-processing mechanisms in children's social adjustment. Psychological Bulletin, 115, 74-101.

Dodge, K. A., & Pettit, G. S. (2003). A biopsychosocial model of the development of chronic conduct problems in adolescence. Developmental Psychology, 39, 349-371.

Eddy, J. M., Dishion, T. J., Stoolmiller, M. (1998). The analysis of intervention change in children and families: Methodological and conceptual issues embedded in intervention studies. Journal of Abnormal Child Psychology, 26, 53-69.

Fraser, M. W., Day, S. H., Galinsky, M. J., Hodges, V. G., & Smokowski, P. R. (2004). Conduct problems and peer rejection in childhood: A randomized trial of the Making Choices and Strong Families programs. Research on Social Work Practice, 14(5), 313-324.

Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., & Guo, S. (in press). Social information-processing skills training to promote social competence and prevent aggressive behavior in third grade. Journal of Consulting and Clinical Psychology.

Fraser, M. W. (2004). Intervention research in social work: Recent advances and continuing challenges. Research on Social Work Practice, 14(3), 210-222.

Fraser, M. W. (Ed.). (2004b). Risk and resilience in childhood (2nd ed.). Washington, DC: NASW Press.

Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (in press). The art of group work practice with manualized curricula. Social Work with Groups. Gross, D., & Fogg, L. (2004). A critical Gross, D., & Fogg, L. (2004). A critical analysis of the intent to treat principle in prevention research. analysis of the intent to treat principle in prevention research. Journal of Primary Journal of Primary Prevention, 25Prevention, 25, 475-489., 475-489.

Gross, D., & Fogg, L. (2004). A critical analysis of the intent to treat principle in prevention Gross, D., & Fogg, L. (2004). A critical analysis of the intent to treat principle in prevention research. research. Journal of Primary Prevention, 25Journal of Primary Prevention, 25, 475-489., 475-489.

References (continued)References (continued)Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation

estimator: Evidence from evaluating a job training programme. The Review of Economic Studies, 64, 605-654.

Heckman, J. J., & Smith, J. A. (1995). Assessing the case for social experiments. Journal of Economic Perspectives, 9, 85-110. .

Hughes, D. L., & Seidman, E. (2002). In pursuit of a culturally anchored methodology. In T. A. Revenson, A. D’Augelli, S. E. French, D. L. Hughes, D. Livert, E. Seidman, M. Shinn, & H. Yoshikawa (Eds.), Ecological research to promote social change: Methodological advances in community psychology (pp. 243-255). New York, NY: Kluwer Academic Press.

Lachin, J. M. (2000). Statistical considerations in the intent-to-treat principle. Controlled Clinical Trials, 21, 167-189.

Lemerise, E. A., & Arsenio, W. F. (2000). An integrated model of emotion processes and cognition in social information processing. Child Development, 71, 107-118.

Nash, J. K., Kupper, L. L., & Fraser, M. W. (2004). Using multilevel statistical models in social work intervention research. Journal of Social Service Research, 30(3), 35-54.

Peck, L. S. (2003). Subgroup analysis in social experiments: Measuring program impacts based on post-treatment choice. American Journal of Evaluation, 24, 157-187.

Prinz, R. J., Smith, E. P., Dumas, J. E., Laughlin, J. E., White, D. W., & Barron, R. (2001). Recruitment and retention of participants in prevention trials involving family-based interventions. American Journal of Preventive Medicine, 20(Suppl. 1), 31-37.

Rothman, J., & Thomas, E. J. (Eds.). (1994). Intervention research: Design and development for human service. New York, NY: Haworth.

Rutter, M. (2005, April). Identified gene-specific environment interactions in relation to psychopathology. Presentation at the Biennial Meeting of the Society for Research in Child Development. Atlanta, GA.

Sameroff, A. J., & Gutman, L.M. (2004). Contributions of risk research to the design of successful interventions. In P. Allen-Meares & M.W. Fraser, Intervention with children and adolescents: An interdisciplinary perspective. Boston: Pearson Education, Inc.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.

Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422-445.

Smokowski, P. R., Fraser, M. W., Day, S. H., Galinsky, M. J., & Bacallao, M. L. (2004). School-based skills training to prevent aggressive behavior and peer rejection in childhood: Evaluating the Making Choices program. Journal of Primary Prevention, 25(2), 233-251.