Developing Evidence-Based Products Using the Systematic

Review Process Session 3/Unit 9:

Inclusion and Exclusion Criteria

Screening and Coding Studies

Julia LavenbergOctober 30, 2007

NCDDR training course for NIDRR grantees

2

Outline

Study selection vs. data extraction

Requirements for each Exploration of screening and coding via a worked example

Application of inclusion and exclusion criteria and codes to planned reviews

3

Study Selection vs. Data Extraction

The process by which one chooses studies for inclusion in a systematic review.

Accomplished by instituting specific and detailed inclusion and exclusion criteria.

The process by which one locates and transcribes information from a primary study.

Accomplished by applying specific and detailed criteria to the information in a primary study.

4

Eligibility criteria

Provide readers with an idea of the research domain of interest

Aid the systematic reviewer in applying consistent and objective standards throughout the selection process

Clearly circumscribe the review

5

Inclusion and Exclusion Criteria: Categories

Publication types Study design Population Intervention Outcomes Measures

6

Publications Identify the types of reports to be included

Determine any geographic or linguistic limitations

“Studies eligible for this review may be published or unpublished reports (e.g.,

dissertations theses, government reports, school district reports, etc.) of school-based interventions conducted

in any country and reported in any language.”

(Lavenberg, 2007)

7

Study design Specify the research designs to be included

Also identify the research designs to be excluded

Address the rationale for inclusion and exclusion Consider theoretical framework Consider available evidence

8

Population Indicate the desired target population Stipulate any required characteristics Additionally, distinguish characteristics that would make the target population ineligible

“Studies of interventions that target children and youth who are enrolled in kindergarten through grade 12 (or the international equivalents) at public, private, parochial, or alternative schools and are between the ages of 4 and 20 years will be included in this review. Persons identified as attending ‘preschool’ or ‘college’ will not be included, even if they are within the acceptable age range.” (Lavenberg, in process)

9

Intervention

Detail the required characteristics of the intervention

Provide definitions List examples of what would be included and what would be excluded

10

Outcomes Indicate outcome variables of interest

“At least one quantitative measure of aggressive behavior must be reported in each study….[citations to literature here]… Therefore, both physical and verbal aggressive behaviors will be considered acceptable outcomes.”

(Lavenberg, in process)

11

Measures Address acceptable measures

“Standardized measures of aggressive behavior (e.g., Child Behavior Checklist – Teacher Report Form….) and unstandardized measures with adequate face validity (e.g., local administrative records…) will both be considered acceptable forms of reporting aggressive behavior. Measures reported in the Buros Institute of Mental Measures Yearbook…will be considered standardized measures; all others will be considered unstandardized measures.”

(Lavenberg, in process)

12

Example: Application to a review with > 900 records retrieved (Lavenberg, 2007)

Two phase screening process developed Phase I – title and abstract

Goal: differentiate potential studies from clearly inappropriate studies

Evaluate each study for topical relevance, setting, participants, and general study design

Phase II – method section Goal: create the pool of eligible studies Evaluate each study for intervention, design details, and outcome

13

Selection process for including reports

Potentially relevant reports identified

(n>900)

Reports retrieved for further review

(n=155)

Reports progressing to next level(n=109)

Reports used to code studies for inclusion

(n=50)

Reports excluded at Phase II screening(n=59)

Reports excluded at Phase I screening(n=46)

Reports excluded by title and

abstract review(n=750)

14

Data Extraction (i.e., “coding”)

Initiated after identifying appropriate pool of studies for the systematic review

Accomplished by means of a coding form and codebook

Goal: ensure reliable and orderly extraction of information from each study report

15

Forms… May be either paper-based or computer-based

Are best completed by trained coders Should have items organized such that the order reflects the manner in which information is presented in the study report

Contain the same categories of information as inclusion and exclusion criteria

16

Coding categories

That is, the data extracted will address: Publication type Design Participants Interventions Outcomes Measures

17

. . . in coding, much more detail

is required.

The data that is extracted from each study report is the basis of all subsequent analyses.

BUT:

18

Overarching principle:

Conserve as much of the original information as possible.

Fewer coder judgments = fewer errors

19

Publication characteristics

Document publication year Distinguish between types Create mutually exclusive categories

Must assign numerical values to these categorical variables

20

Study design and methodological characteristics

Consider identifying the recruitment pool to provide additional context

Describe the unit of assignment (e.g., individual, family, school, clinic, workplace)

Address the mechanism of allocation (e.g., random assignment, matching on pretest measures of outcome variables, etc.)

List unit of analysis Note whether unit of assignment matches unit of analysis

21

Participant characteristics Age

May need to capture data in multiple ways (e.g., school grade, chronological age, mental age, age range)

Gender Race/ethnicity Socioeconomic status Specific characteristics relevant to review topic (e.g., official diagnoses, developmental categories, baseline level of behavior)

Co-morbidity

22

Intervention characteristics Describe important aspects of the intervention Characteristics or features Total length of intervention, number of sessions, length of each session

Method of delivery Personnel implementing intervention Fidelity of implementation

Identify comparison group conditions

23

Outcome and measure characteristics

Identify outcome variables of interest Allow for and document variations

Identify outcome informant or source of information Determine if informant was blind to assignment

Identify personnel administering measure

Specify outcome measures used Indicate time of outcome assessment

24

Effect size characteristics Extract information required to calculate an effect size Sample size (intervention and control groups)

Pretest and posttest Attrition

Means, standard deviations d index F value Others: Chi-square, p value, t value, etc.

25

Worked example #1

Approaches to Parent Involvement for Improving the

Academic Performance of Elementary School Age Children

Nye, Turner, & Schwartz (2006)

http://www.campbellcollaboration.org/doc-pdf/Nye_PI_Review.pdf

26

Definition of parent involvement

“…the parent has a direct interaction with the child in either the delivery or monitoring of the program or intervention.” (page 11)

27

Inclusion criteria

Design: limited to randomized controlled trials (RCTs)

Rationale: “…in order to provide the least biased estimate of the effect of parent involvement on student achievement and to control for variation in threats to internal validity.” (pg. 12)

28

Inclusion criteria, continued

Participants: described in detail in the coding categories section of the review (pg. 13)

Age Grade Gender SES Ethnicity

29

Inclusion criteria, continued

Interventions: “…studies reported the following characteristics of the intervention program: 1) Parent involvement with their child in academic support activities outside of school (e.g., reading or completing supplemental math problems with the child), and 2) Parent involvement [as defined earlier] for a minimum of 20 days” (pg. 11)

30

Inclusion criteria, continued

Outcomes: “Included studies reported the following outcomes on children’s academic performance in: reading, mathematics, spelling, writing, language arts, or science. (pgs. 11-12)

31

Inclusion criteria, continued

Measures: not specified as an inclusion criteria in Nye, et al. Again, specification of this characteristic will depend on the topic and decisions made by the review team

e.g., depression as measured by Beck inventory

32

Nye, Turner, & Schwartz Coding categories and coding sheet (unique elements)

Report characteristics Subject characteristics

Treatment and comparison groups separately addressed

Sample source – types of schools Classroom assignments – high achieving, underachieving, average, academically at-risk

Developmental categories – emotionally disabled, physically disabled, hearing impaired, speech/language impaired, etc.

http://www.ncddr.org/training/1NyeTurnerSchwartz_PI_CodingForm_2006.doc

33

Coding categories, continued Intervention characteristics

Coder description required, in addition to numerical data (e.g., total length of intervention program, length of time per day of PI activity, etc.)

Fidelity of implementation Outcome measures – norm referenced vs criterion referenced vs rating scale vs survey

Design characteristics Recruitment pool Specific method of randomization and identification of who implemented the process

Effect size characteristics

34

Review topics Weight loss intervention to reduce physical disability in older adults

Improving the ambulatory transition of pre- and post-adolescents with cerebral palsy

Improving cultural competency to increase the employment of persons with disabilities

35

Review topics, continued

Use of assistive technology and employment supports in the employment of working age people with disabilities

Health promotion programs that improve primary access for adults with disability

36

Contact information

Julia LavenbergEmail: j.lavenberg@gmail.com

37

SUPPLEMENTAL SLIDES

38

Worked example #2

Effects of School-Based Cognitive-Behavioral Anger

Interventions: A Meta-Analysis

(Lavenberg, 2007)

39

Primary Research Question:

Are school-based cognitive-behavioral anger

interventions effective in reducing child and adolescent aggressive behavior in the

school setting?

40

Secondary Research Questions… Involved looking at whether the following were related to the effect size: Methodological issues (e.g., random assignment)

Setting Geographic location (urban, suburban, rural)

Delivery personnel Teachers vs. counselors

Stand-alone vs. embedded anger interventions

Participant characteristics (e.g., gender, age, baseline level of aggressive behavior)

41

Define and operationalize

critical elements:

Aggression Anger Cognitive-behavioral

Theory Intervention

Anger management intervention School-based intervention

42

Screening Forms and Manual

Copy sent to Joann Starks for posting

http://www.ncddr.org/training/3Lavenberg_2007_Screening_Coding_FormsManuals.doc

43

Coding Form and Manual

Copy sent to Joann Starks for posting

http://www.ncddr.org/training/3Lavenberg_2007_Screening_Coding_FormsManuals.doc