Early Identification of Children At Risk for Costly Mental Health Service Use

P1: GVM/GCY/LCR/GCV/GCQ

Prevention Science [PREV] pp654-prev-454095 October 14, 2002 12:39 Style file version Nov. 04, 2000

Prevention Science, Vol. 3, No. 4, December 2002 ( C© 2002)

Early Identification of Children At Riskfor Costly Mental Health Service Use

Damon Jones,1,5 Kenneth A. Dodge,2 E. Michael Foster,3 Robert Nix,1and Conduct Problems Prevention Research Group4

Children and adolescents with serious and persistent conduct problems often require largepublic expenditures. Successfully diverting one high risk child from unfortunate outcomes mayresult in a net savings to society of nearly $2 million, not to mention improving the life of thatchild and his or her family. This figure highlights the potential of prevention, which often restson the ability to identify these children at a young age. This study examined the ability of a shortconduct-problems screening procedure to predict future need for mental health assistance,special education services, and the juvenile justice system during elementary school ages. Thescreen was based on teacher and parent report of child behavioral habits in kindergarten,and was used to identify children as either at risk or not at risk for behavioral problems.Service outcomes were derived from a service-use assessment administered to parents at theend of the sixth grade, while special education information was gathered through a surveyof school records. Study participants (463 kindergarten children; 54% male, 44% AfricanAmerican) were from economically disadvantaged neighborhoods in four diverse communitiesacross the United States. Results indicated that, while controlling for demographic backgroundvariables, the risk indicator strongly predicted which children would require services relatedto conduct disorder or behavioral/emotional problems. Additional analyses revealed that thedichotomous high risk indicator was nearly as strong as the continuous screening variable inpredicting the service-use outcomes, and that the screening of both parents and teachers maynot be necessary for determining risk status.

KEY WORDS: prevention; behavioral disorders; service utilization.

This study determined whether a short conduct-problems screening instrument, administered inkindergarten, could predict children’s involvementwith the mental health, special education, and juve-nile justice systems by the time they entered middle

1Department of Psychology and Human Development, VanderbiltUniversity, Sewanee, Tennessee.

2Center for Child and Family Policy, Duke University.3Department of Health Policy and Administration, PennsylvaniaState University.

4Kenneth Dodge’s colleagues in the Conduct Problems PreventionResearch Group are, in alphabetical order, Karen L. Bierman,PhD (Pennsylvania State University), John D. Coie, PhD (DukeUniversity), Mark Greenberg, PhD (Pennsylvania State Univer-sity), John E. Lochman, PhD (University of Alabama), RobertJ. McMahon, PhD (University of Washington), and Ellen E.Pinderhughes, PhD (Vanderbilt University).

school. If high cost children can be identified reliably,early intervention may be cost-effective.

A small minority of children initially display con-duct problems as early as preschool, and these prob-lems persist well into adulthood (Moffitt, 1993). Thisgroup, called “early starters,” constitutes about 5 or6% of the population but is responsible for 50–60%of all known crimes (Blumstein et al., 1986). The mostreliable and robust predictor of whether a child willdisplay this pattern of serious and persistent con-duct problems is extreme and stable oppositional andaggressive behaviors early in life (Achenbach et al.,

5Correspondence should be directed to Damon Jones, Departmentof Psychology and Human Development, Vanderbilt University,417 New College Drive #6, Sewanee, Tennessee 37375; e-mail:[email protected].

2471389-4986/02/1200-0247/1 C© 2002 Society for Prevention Research



248 Jones, Dodge, Foster, Nix, and Conduct Problems Prevention Research Group

1998; Lipsey & Derzon, 1998; Loeber, 1982, 1991).Early starters can be contrasted with children whodemonstrate problems primarily during adolescence(Moffitt, 1993). These adolescents generally commitonly property crimes (Bartusch et al., 1997), and theydo so irregularly. Their conduct problems dissipate inearly adulthood.

Serious and persistent conduct problems resultin great public expense. A life of crime costs societybetween $1.3 and $1.5 million; a heavy drug user costssociety between $370,000 and $970,000; a high schooldropout costs society between $243,000 and $388,000(Cohen, 1998). Most of these costs are related to vic-tim losses, court adjudication, incarceration, medicaltreatment, rehabilitation, decreased productivity, andforegone earnings. Children showing early conductproblems are at risk for each, and possibly all three, ofthese outcomes (i.e., criminal activity, substance use,and school dropout). The value to society of success-fully diverting one high-risk child from serious con-duct problems may be as much as $1.7–$2.3 million(Cohen, 1998). This figure is conservative: it does notcapture many inherent benefits for the children andfamilies involved. The potential benefits of preventingconduct disorder are, therefore, quite large.

In spite of the known stability of conduct-problem behaviors, virtually no studies have exam-ined the predictability of service system utilizationfrom early conduct-problem behavior. This studysought to fill that gap by addressing four questions.First, how well does a measure of conduct problems atschool entrance predict future involvement with themental health, special education, and juvenile justicesystems? Clearly, the initial step in any successfulprevention effort is the accurate identification ofhigh-risk individuals. Second, does the measure’spredictive ability vary by child sex, race, or ecologicalcontext? If screening efforts are differentially effec-tive for subgroups of children, then the empirical casefor prevention may also vary across children. Third,what are the relative advantages of using a moreparsimonious dichotomous risk status designationwhen compared to a continuous measure of conductproblems? And, fourth, is a screening measure moreaccurate if it combines both parent and teacherreports of behavior? This issue is important becauseof the added costs of collecting information from twosources. We consider these questions using data fromthe Fast Track Project (Conduct Problems Preven-tion Research Group, 1992), a multicohort, multisitelongitudinal intervention designed to prevent seriousconduct problems in high-risk children.

METHOD

Participants

This study followed 463 children participating ina longitudinal study being conducted as part of theFast Track project.6 Data are available on these chil-dren through the end of the sixth grade. The sam-ple includes children who were identified as “highrisk” (as described below) as well as a representative/normative sample of children who also were recruitedfrom the study schools. It is important to note that nochildren in these analyses received Fast Track inter-vention services; children identified as high risk forthis study are part of the Fast Track control group(described below).

Study participants were drawn from four studysites: Durham, NC; Nashville, TN; rural centralPennsylvania; and Seattle, WA. Approximately 54%of the children were boys; about 52% were Euro-pean American, 44% were African American, and4% were of other ethnic backgrounds. The mean so-cioeconomic status (SES) of their families was 25 onthe Hollingshead SES scale (Hollingshead, 1979) [be-tween lower and lower-middle class] with a standarddeviation of 13; 24% of the families did not include aparent who was employed full-time; and 41% of thefamilies included only one parent.

Schools were selected for the study using a pro-cedure described in more detail elsewhere (Con-duct Problems Prevention Research Group, 1992;Lochman & Conduct Problems Prevention ResearchGroup, 1995). Using crime records, poverty statistics,and high school dropout rates, each site identified be-tween 10 and 17 high risk schools in its community.Half of the schools were randomly assigned to receiveintervention; the other half (n = 27) were assignedas controls, and it was the students in these controlschools that were the participants in this study.

Within the intervention and control schools, chil-dren were identified as high risk using informationfrom both teachers and parents. Each kindergartenteacher in these schools completed a brief 10-itemscreening instrument on each child. The instrument

6Fast Track research projects such as this are supported by Na-tional Institute of Mental Health (NIMH) Grants R18 MH48043,R18 MH50951, R18 MH50952, and R18 MH50953; and inpart by the Department of Education, Grant S184U30002 andNIMH Grants K05MH00797 and K05MH01027. For additionalinformation concerning Fast Track (via internet), please visitwww.fasttrackproject.org.



Predicting Child Mental Health Service Use 249

was adapted from the Teacher Observation of Class-room Adaptation-Revised7 (TOCA-R; Werthamer-Larsson et al., 1991). The teacher rated each child as towhether they displayed behaviors such as “trouble ac-cepting authority; disobedient” and “harms others.”The internal reliability for this measure was high(Cronbach’s α = .87). If a child received a score inthe top 40th percentile (within site) on this teacher-report measure, a project staff member contacted andinterviewed the child’s parent. The parent completeda 24-item measure, based on questions from the ChildBehavior Checklist (Achenbach, 1991) and the Re-vised Problem Behavior Checklist (Quay & Peterson,1987), which assessed children’s conduct problems athome. The internal reliability for this measure alsowas high (Cronbach’s α = .87).

Scores on the teacher-report and parent-reportmeasures were standardized within site and summed.The 10% of the population with the highest combinedscores (n = 155, not including Fast Track interventiongroup children) were designated as “high risk.” Theremaining 90% of children were stratified within siteaccording to child’s sex, race, and decile scores on theteacher-report measure. From these stratified blocks,308 children were chosen randomly to represent therest of the populations in these schools and were as-signed to the non-high risk, normative group.8 Whencombined, the resulting sample included 463 children.

Of the original sample, 391 were still participat-ing by the seventh year of the project (84% of total)and were able to provide data for this study. When thekey outcome data were collected, the children had amean age of 12.0 years. Because of the intentionaloverrepresentation of high risk children as well as thepossible selectivity in participation (attrition) by theseventh year, alternative weighted analyses were per-formed that accounted for having a nonrepresentativesample. Details are given below.

Outcome Measures

Once children were part of Fast Track, theirteachers and parents were interviewed on an an-nual basis. Child school records were also reviewed

7These items were taken from the Authority Acceptance subscaleof the TOCA-R.

8The official Normative sample for the Fast Track project actuallyincludes an appropriate proportion of high risk individuals. Thus,the comparison sample in this paper will be referred to as thenon-high risk sample as opposed to the normative sample.

annually by a member of the Fast Track staff. Twoof the outcome measures in this study were derivedfrom school records: whether a child had ever had anindividual education plan (IEP) or received specialeducation services; and whether a child had ever re-peated a grade in school. A third outcome was takenfrom the annual survey of parents: whether a child hadever taken medications, such as methylphenidate, be-cause of behavior or emotional problems.

The remaining outcomes for this study were de-rived from an assessment of service use in the sam-ple. During the summer of the seventh year of thestudy (after the end of sixth grade for most of the chil-dren in the sample), mothers were interviewed usinga modified, 30-minute version of the Service Assess-ment for Children and Adolescents (SACA; Stiffmanet al., 2000). Developed for epidemiological research,the SACA assessed children’s use of a variety of men-tal health services and their contact with the juvenilejustice system during their lifetime. For this study, theSACA provided dichotomous (yes/no) scores on eachof the following six variables:

• Has the child ever received treatment from aprovider who specialized in mental health ser-vices, such as an outpatient therapist or an in-home family preservation worker, or receivedtreatment in a mental health facility, such as apsychiatric hospital, a drug and alcohol treat-ment unit, a residential treatment center, a daytreatment program, or a group home?• Has the child ever used any general medical

service for emotional or behavioral problems,including talking to a family doctor, going toan emergency room, or staying in a medicalhospital?• Has the child ever stayed overnight in a men-

tal health treatment facility, including a psy-chiatric hospital, general hospital, drug andalcohol treatment unit, residential treatmentcenter, emergency shelter, group home, fosterhome, detention center, or jail?• Has the child ever used any inpatient or

outpatient medical/professional service foremotional or behavioral problems, excludingtalking to a school counselor or having policecontact?• Has the child ever talked to a school coun-

selor as a result of emotional or behavioralproblems?• Has the child ever been picked up, arrested,

or given a warning by the police?




Analyses

Examination of the predictive potential of thehigh risk screening variable was aided by sensitivityand specificity indices. Sensitivity is the proportionof participants using a given service who had beengiven a high risk designation in kindergarten (i.e.,“true positives” who are correctly identified by thescreen). Specificity is the proportion of participantsnot using a given service who had not been labeledhigh risk at the screening (i.e., “true negatives” whoare correctly identified).9 Any screening instrumentbalances sensitivity and specificity by identifying moreor fewer children as “at risk.” By raising or loweringthat threshold, one can raise sensitivity or specificitywhile lowering the other. To use an extreme example,one could identify all children as at risk and achieve100% sensitivity. Of course, specificity would be zeroin that case. The relative desirability of high sensitivityversus high specificity depends on the relative costli-ness of failing to provide intervention or treatmentearly to at-risk children versus the costs of misiden-tifying children as at risk (Glascoe et al., 1997). Thelatter might involve stigma or the costs of treatmentprovided unnecessarily.

Prior research provides a context for assessingthe figures presented below. In a survey of studieson the prediction of conduct disorder, Bennett et al.(1999) found that sensitivity rates based on risk in-dicators assigned at kindergarten or first-grade wereunlikely to exceed 50% for such outcomes, whereasspecificity rates were unlikely to exceed 90%. It isdiscouraging to think that one might only expect toidentify and potentially treat 50% of the cases thatwould otherwise end up achieving a certain unfortu-nate outcome. An intervention, however, is unlikelyto prevent all future behavioral problems; a low sen-sitivity rate for one outcome does not mean an in-tervention will not prevent the occurrence of anotheroutcome (for instance, it may prevent a child fromfuture need of overnight mental health assistance,while not preventing future need of an IEP at somepoint). And as some figures indicate that only 20%of children in need of care typically receive it (Insti-tute of Medicine, 1994; as cited in Bennett et al., 1999,p. 1067), figures close to 50% might actually be rel-atively respectable. It is also worth pointing out that

9Sensitivity = [# for which outcome present and risk indicatorpresent]/[# for which outcome present]; Specificity = [# for whichoutcome absent and risk indicator absent]/[# for which outcomeabsent].

sensitivity and specificity rates reflect the nature ofthe outcome. In general, higher specificity rates aremore attainable as the likelihood of not achieving theoutcomes featured in studies of this nature is ratherhigh. As with any proportion, larger denominatorswill likely translate into smaller ratios; so the morelikely the outcome has occurred at some point in apopulation of children regardless of risk status, themore likely a sensitivity rate will be lower (helpingto explain lower sensitivity numbers for such cate-gories as ever had an IEP/special education services,or “any of the above services”). At the same time,lower specificity rates are often associated with out-comes that have an extremely high likelihood of notoccurring in a population (such as inpatient mentalhealth assistance).

To compare the predictive power of these modelsfor each outcome measure, receiver operating char-acteristics (ROC) curves were generated for someanalyses (Agresti, 1990). The ROC curve plots sen-sitivity against 1 minus the specificity given a vary-ing predicted probability cutoff. The area under thiscurve indicates the predictive power of the model.Possible values range from 0.5 to 1, with higher val-ues indicating greater predictive power. This value ishelpful in summarizing the effectiveness of a screen-ing classification regarding a particular outcome aswell as the degree to which a screening indicator hasbeen both sensitive and specific in respect to thatoutcome.

Because all outcome variables used in this studywere dichotomous, logistic regressions were used forstatistical analyses. Predictors in these models in-cluded dummy variables to denote high risk status(no vs. yes) as well as gender (whether child wasfemale) and minority status (whether child was anethnic minority).10 Also included in the models werethree dummy variables to denote site (with Durhamas the omitted site).11 In order to address the sec-ond research question introduced above, two-way

10Of the sample involved in this study (n = 391), over 92% of thoseclassified as Minority are African American.

11In addition to including variables to control for site, we were alsoconcerned with a likely clustering of error variance due to simi-lar characteristics shared by children attending the same school.In order to control for this type of error structure, a robust esti-mator of error variance was derived by specifying a clustering atschool level within our models (StataCorp, 1999). Because chil-dren may have attended more than one school within the 6-yearspan of this study, the children’s school in first grade was speci-fied as the clustering unit. It was deemed most likely to representschool clustering effects.




interaction terms between the high risk dummy vari-able and the other explanatory variables were initiallyincluded in the models and retained if statistically sig-nificant (p < .05).

To improve interpretation of the effect of the ex-planatory variables, marginal effects are presented inthe Results section in place of the conventional logitcoefficients. As is well known, interpreting the coef-ficient estimates from logistic regression is somewhatawkward—those estimates represent the effect of theexplanatory variable on the log-odds ratio. One op-tion is to present odds ratios, but a more informa-tive alternative is to present the so-called “marginaleffects” (Greene, 2000). The marginal effect repre-sents the effect of the explanatory variable in mean-ingful terms—the likelihood that the individual ex-periences the outcome of interest. In this case, themarginal effect is the impact of risk status (or anyother variable) on the predicted probability that theindividual uses the service, holding all other vari-ables in the analyses constant. The marginal effectis analogous to the ordinary regression coefficientbut allows for the fact that the dependent variable isdichotomous.

As discussed above, there was some concern re-garding the degree to which having a nonrepresenta-tive sample (oversampled on high risk children) mightalter the generalizability of the results. In addition,there was some concern about the small yet nonignor-able level of attrition (approximately 15% of the ini-tial sample) that had occurred by the time the SACAwas administered. As widely discussed in the litera-ture, selection effects and differential attrition in fieldstudies can reduce both internal and external validity(e.g., Cook & Campbell, 1979).

To examine the potential impact of overselec-tion for high risk children and possible differentialattrition, we performed alternative weighted analyses(Levy & Lemeshow, 1991). The weight was a functionof two probabilities. The first probability involved thelikelihood that a child was selected for the study. Be-cause of the oversampling, that probability was aboutthree times as great for high risk children as for thenon-high risk children. A second probability was cal-culated based on an attrition model that predictedparticipation in the seventh year of the study (whenthe SACA was administered) based on important in-dividual characteristics. To allow for interactions be-tween high risk status and the predictors of attrition,we estimated the attrition model separately for highrisk and non-high risk groups. We used the resultingcoefficient estimates to calculate predicted probabil-

ities of participation. The probability weight used inthese alternative analyses is the inverse of the productof these two probabilities. These weights were used toprovide population estimates of the service-use rates.They were also used in main effects logistic regressionmodels that were computed for all outcome measures.When the results of those analyses were comparedto the results of main effects models without weights,they were virtually identical. Therefore, we concludedthat selection and participation issues were not no-ticeably altering our outcomes and proceeded usingnonweighted models.

RESULTS

Question 1: Does High Risk Status Predict FutureInvolvement With the Mental Health, SpecialEducation, and Juvenile Justice Systems?

Table 1 presents rates of service utilizationthrough the end of elementary school. The first col-umn provides weighted service utilization rate esti-mates. (These figures are representative of children ofthis age living in poor neighborhoods across the coun-try.) The figures in this column show that, although therate of children receiving at least one of these servicetypes is above 50%, use is predictably low for othercategories such as inpatient mental health use (lessthan 2%). The next columns present rates by high riskstatus. For all outcome measures, children identifiedas high risk in kindergarten had much higher rates ofservice use across the subsequent 6 years. Over halfof children identified as high risk later received somekind of professional service for emotional or behav-ioral problems, whereas approximately 16% of thenon-high risk children received such services. Over-all, 82% of high risk children had received assistancein at least one of the defined categories.

Table 1 also presents the sensitivity and speci-ficity indices. Results show that the kindergarten riskindicator was a relatively sensitive predictor of lateruse of psychotropic medication, later use of men-tal health services from a school counselor or gen-eral medical professional, and future overnight place-ment for mental health problems. For instance, threeout of four children who had received inpatient ser-vices for mental health by the seventh year had beenidentified in kindergarten as at high risk for futurebehavioral/emotional problems. Specificity was highfor all variables, indicating that children not identifiedas at risk for behavioral problems were very likely not




Table 1. Rates of Lifetime Service Use (Whether Ever Used) 7 Years After Risk Status Designation (N = 391)

Population estimatea High risk Non-high risk Sensitivityb Specificityb

Had an IEP/special education services 36.3% 56.3% 33.2% 36% 83%Medicationsc 9.0 30.6 5.3 62 80Repeated a grade 23.4 31.9 22.7 33 79Specialized mental healthc 18.1 46.5 13.0 52 83General medicalc 6.5 18.1 4.5 59 79Overnight mental healthc 1.8 7.6 0.8 75 77Any service for mental health (SACA)c 21.3 52.1 15.8 50 84School counselorc 9.0 26.4 5.7 61 80Police contact 6.9 16.7 5.3 52 78Any of above services 57.7 82.6 53.9 33 90

aPopulation estimate weighted to account for overrepresentation of high-risk subjects in the sample and higher rate of dropout fornon-high risk subjects by the seventh year of the study. The represented population reflects the Fast Track sample (i.e., childrenliving in poor neighborhoods).

bSensitivity and specificity figures based on normative sample (N = 326).cService provided for emotional, behavioral, drug, or alcohol problems.

to require services for such problems within the nextseven years.

As noted above, logistic regressions were usedto examine how high risk status predicted serviceuse while controlling for the key demographic fac-tors of sex, race, and variations due to study site (allof which might be considered to have a strong im-pact on variation in the outcomes). The results ofthese analyses are presented in Table 2. High risk chil-dren were significantly more likely than non-high riskchildren to have received mental health assistance inthe forms of professional services (specialty mentalhealth or general medical for mental health reasons),medication (p < .01), or inpatient mental health ser-vices (p < .05). The relation between high risk sta-tus and the likelihood of receiving any inpatient oroutpatient service for mental health was marginallysignificant (p = .06). For school-related services, highrisk children were significantly more likely to have re-ceived special education services (IEP or special ed-ucation minutes) or received mental health-relatedschool counseling (p < .01). Although to a lesser de-gree, high risk status also was related to the likelihoodof repeating a grade in school (p = .08). Finally, highrisk children were significantly more likely to havehad contact with the police (p < .05).

The marginal effects make these results relativelyeasy to understand. For instance, the significant resultfor the IEP/special education outcome indicates thathigh risk children were 20 percentage points morelikely (on average) to have received that service bythe seventh year of the study than non-high risk chil-dren, controlling for gender, minority status, and site.This effect varies among outcomes, with the most

substantial difference occurring for the likelihood ofusing a specialty mental health provider (high riskchildren were more than 26 percentage points morelikely to use such a service) and the smallest effectoccurring for the likelihood of staying overnight formental health services (high risk children were justover 3% more likely to do so). Note that the differ-ence in the effect of high risk status as presented inTable 2 versus Table 1 is mostly due to the inclusionof demographic variables in the logistic regressionmodels.

Question 2: Does the Effect of High Risk Status Varyby Child Sex, Race, or Ecological Context?

As noted, two-way interaction terms betweenhigh risk status and the other predictors were includedin initial models in order to examine whether the in-fluence of being at risk for behavioral problems onfuture service use might vary across levels of the childsex, minority classification, or geographic region ofthe country. The only significant interaction detectedacross all outcomes involved the interaction betweenhigh risk status and study site for the likelihood ofhaving received any inpatient or outpatient servicefor mental health assistance. The specific interactionterms (given in the footnotes below Table 2) revealwhat is behind this significant effect: The high risk chil-dren at the Pennsylvania and Washington sites wereclose to 20 percentage points more likely to have usedany inpatient or outpatient service for mental healthproblems than the high risk children in Durham orNashville.




Tabl

e2.

Log

isti

cR

egre

ssio

nsof

Men

talH

ealt

h(M

H)

Serv

ice

Use

Out

com

es(C

olum

ns)

onP

redi

ctor

s(R

ows)

—M

argi

nalE

ffec

tsP

rese

nted

IEP

/spe

cial

Med

icat

ions

Gen

eral

med

ical

Ove

rnig

htIn

pati

ent/

outp

atie

ntSc

hool

serv

ices

educ

atio

nfo

rM

HR

epea

ted

grad

eSp

ecia

lty

MH

for

MH

for

MH

for

MH

for

MH

Polic

eco

ntac

t

Out

com

epr

edic

tor

Eff

ect

ZE

ffec

tZ

Eff

ect

ZE

ffec

tZ

Eff

ect

ZE

ffec

tZ

Eff

ect

ZE

ffec

tZ

Eff

ect

Z

Nas

hvill

e−0.1

67†

1.70−0.0

020.

050.

221∗∗

2.85

0.05

40.

640.

076∗∗

2.83

0.00

40.

330.

100

0.99

0.14

8∗∗

4.83

0.09

6∗∗

3.32

Pen

nsyl

vani

a−0.2

80∗∗

2.72−0.0

200.

43−0.0

090.

09−0.0

330.

350.

055†

1.75−0.0

130.

72−0

.089

0.76

0.04

10.

860.

043

1.18

Was

hing

ton

−0.1

95∗

2.43−0.0

380.

78−0.2

60∗∗

3.85

0.02

60.

330.

058∗

2.26−0.0

060.

43−0

.087

0.69

0.10

6∗∗

3.21−0.0

160.

46

Site

effe

ct8.

52∗

1.10

30.6

2∗∗

3.74

8.97∗

1.40

11.7

4∗∗‡

37.9

2∗∗

22.3

6∗∗

Fem

ale

−0.1

45∗∗

2.63−0.0

68∗

2.19

0.01

30.

31−0.1

03†

1.95−0.0

77∗∗

3.33−0.0

171.

53−0

.127∗

2.47−0.0

40†

1.90−0.0

80∗∗

3.60

Min

orit

y−0.0

460.

61−0.0

371.

180.

214∗∗

3.20−0.0

96†

1.77−0.0

180.

77−0.0

020.

17−0

.098†

1.70−0.0

260.

890.

020

0.92

Hig

hri

sk0.

205∗∗

4.25

0.16

8∗∗

5.38

0.07

6†1.

730.

269∗∗

5.82

0.06

8∗∗

3.44

0.03

1∗2.

220.

217†

1.86

0.12

8∗∗

5.52

0.03

5∗2.

13C

onst

ant

0.07

90.

83−0.1

97∗∗

4.41−0.3

53∗∗

4.90−0.2

34∗∗

2.88−0.1

79∗∗

3.93−0.0

63∗

2.20

−0.1

90∗

2.00−0.2

70∗∗

7.34−0.1

51∗∗

3.27

Not

e.Z=

Abs

olut

eva

lue

ofz-

stat

isti

c;de

gree

sof

free

dom

for

all

mod

els=

390;

join

tsi

teef

fect

sar

ech

i-sq

uare

stat

isti

cs.∗

sign

ifica

ntat

5%;∗∗ s

igni

fican

tat

1%;†

sign

ifica

ntat

10%

.‡si

gnifi

cant

HR×

site

inte

ract

ion

for

“inp

atie

nt/o

utpa

tien

tfo

rM

H”

outc

ome,χ

2=

8.10

(mar

gina

lef

fect

s:H

R×

Nas

hvill

e=−0.0

36,H

R×

Pen

nsyl

vani

a=

0.18

4,H

R×

Was

hing

ton=

0.22

1).




Table 3. Comparison of Logistic Regressions for Models WithHR Status Only Versus Models With HR Status Plus Continu-ous Screening Variables—Predictive Power (Area Under ROC

Curve)a

Dichotomous HR variable+HR variable screening variables

IEP/special education .68 .69Medication for MH .78 .80Repeated grade .78 .79Specialty mental health .75 .77General medical for MH .78 .79Overnight for MH .81 .88Inpatient/outpatient for MH .75 .77School services for MH .79 .80Police contact .83 .85

aEntire output for logistic models used to generate these numbersare available by request from the first author.

Question 3: What are the Relative Advantagesof Using a More Parsimonious Dichotomous HighRisk Designation When Compared to a ContinuousMeasure of Conduct Problems?

Designating someone as “high risk” or “not-high risk” provides a practical way to determine whoshould or should not receive an intervention. How-ever, any categorization of individuals into groupsdiscards potentially important information: Not allchildren in the high risk group are equally at risk,and not all children in the non-high risk group areequally invulnerable. For that reason, we considerwhether the discarded information provides any ad-ditional ability to explain the key outcomes. Thus, thenext analyses sought to determine whether any infor-mation remained in these continuous scores—onceaccounting for the information captured in the di-chotomous high risk designation—that might provideincreased accuracy in predicting service use. Main ef-fects logistic regression models were calculated thatincluded high risk status, sex, race, site, plus the twocontinuous scores from the screening measures de-scribed above.12 The results of these analyses are pre-sented in Table 3. The left column gives the predictivepower for models that only included the high risk di-chotomous designation (in addition to the other de-mographic predictors) whereas the right column givesthe predictive power for models that also included theteacher and parent continuous scores. By looking at

12Two subjects who received the SACA were missing data for theparent screening measure, and thus there is a slightly lower N formodels including the continuous predictors.

Table 4. Numbers of Subjects With Teacher+ Parent High Risk(HR) Designation Versus Teacher Only High Risk Designation

Parent+ Teacher HR designation

Teacher HR designation Non-high risk High risk Total

Non-high risk 215 35 250High risk 32 109 141

Total 247 144 391

the measure of model fit based on the ROC numbers,one can see that, for almost all of the outcomes, thetwo continuous indicators add virtually no explana-tory power to the model—the possible exception be-ing the model for inpatient mental health service use.

Question 4: Do Parental Reports Improvethe Accuracy of the Screening Procedure?

To this point, the results suggest that the high riskindicator is an effective predictor of service use. Be-cause obtaining reports from both parents and teach-ers can be costly, it is important to know whether infor-mation from both sources is needed to predict serviceuse.

To examine the potential effectiveness of theteacher screening score alone as a determinant of highrisk status, a teacher-high risk indicator was createdfrom the kindergarten TOCA scores with percentagesof high risk children based on the teacher score set tomatch the percentages of children already designatedas high risk at each site.13 Table 4 shows the degree towhich the combined high risk designation (based onboth parent and teacher scores) overlaps with the ret-rospective teacher-high risk designation: Almost 83%of the time these designations agree.

In order to compare the predictive power of thehigh risk designation based on teacher report onlyto high risk designation based on teacher and parentreport, main effects logistic regression models for theoutcome measures were calculated, and ROC curveswere generated. The numbers showing the predictivepower of the logistic regressions for the two sets ofmodels are presented in Table 5. In most cases theeffect of the combined-source high risk designationis equal to or slightly higher than that of the teacher-only high risk designation. The difference is minimal,

13We set the threshold in this way to make sure the percentage ofchildren deemed at risk did not vary across screening alternatives.Doing so makes comparisons of sensitivity and specificity of themeasures more straightforward.




Table 5. Comparison of Logistic Regressions for Models With HRStatus as Derived From Parent and Teacher Scores Versus HR Sta-tus Derived Only From Teacher Scores—Predictive Power (Area

Under ROC Curve)a

Combined HR Teacher HRvariable variable

IEP/special education .68 .68Medication for MH .78 .77Repeated grade .78 .78Specialty mental health .75 .73General medical for MH .78 .78Overnight for MH .81 .84Inpatient/outpatient for MH .75 .72School services for MH .79 .76Police contact .83 .84

aEntire output for logistic models used to generate these numbersis available by request from the first author.

however. In fact, all values are within 0.03. For two ofthe outcomes, inpatient mental health service use andpolice contact, the predictive power is slightly higherwhen using the teacher report only.

DISCUSSION

This study has shown that costly social service uti-lization through early adolescence can be predictedto a considerable degree by the time a child finisheskindergarten. By evaluating children’s conduct prob-lems with a brief screening measure, kindergartenteachers and, to a lesser extent, parents are able todiscern which children will become involved in themental health, special education, and the juvenile jus-tice systems up to six years later. This study has shownthat an early designation of high risk status can effec-tively predict service use (and thus predict emotionaland behavioral problems serious enough to requireservices), and that the predictability of this high riskstatus does not appear to vary substantially by childsex, child race, or geography as represented by thefour Fast Track sites. Although service utilization pat-terns varied by child sex and race and across projectsites, kindergarten ratings of conduct problems re-mained a robust predictor of later service utilizationin all groups of children. The broad generalizabilityof this pattern suggests that early conduct problemsrepresent an obvious risk factor among most popula-tions of children. We also found that a dichotomousclassification of risk status can be as effective a pre-dictor for service-use likelihood as a continuous pre-dictor of risk, and that a high risk classification de-rived from teacher reports alone might be as effective

in predicting service-use likelihood as a classificationbased on both teacher and parent reports.

If we know which young children are most likelyto become involved with the mental health, special ed-ucation, and juvenile justice systems, the rationale fordeveloping effective prevention and intervention ser-vices is made stronger. There is evidence suggestingthat the overall effectiveness of mental health treat-ment is greater for young children than adolescents(Weisz et al., 1987). Comprehensive reviews havefound that conduct problems can be prevented whenhigh risk children and their families receive earlyeducation and support services (Yoshikawa, 1994).Targeted prevention and intervention studies havedemonstrated that the savings to society from reduc-ing the incidence of serious and persistent conductproblems exceed the costs of high quality and effec-tive programs (Barnett, 1996).

It is important to note that the high risk designa-tion used in this study was moderately sensitive buthighly specific. The relative advantages of high sen-sitivity versus high specificity depend on a variety offactors, including the likelihood that a child identifiedas high risk is subsequently treated, and the costs ofthat treatment (Glascoe et al., 1997). When screeningmeasures are not specific enough, too many childrenare falsely designated as high risk. One consequenceof this is that certain children will receive evaluationand treatment that is unnecessary. However, becauseof the extremely high costs to society of children andadults with serious and persistent conduct problemsand the relatively moderate costs of targeted pre-vention programs, such a screening procedure maybe warranted and still yield cost-effective outcomes.In other words, the combination of a high exclusionrate for children not in need of services (high speci-ficity) with the potential reduction in service-use costsof the correctly identified children (despite moder-ate levels of sensitivity) likely translates into a cost-effective prevention program. Another consequenceof a less specific high risk designation is that certainchildren may be subject to undeserved labeling. How-ever, as long as the negative consequences of inclusionin a prevention program are avoided, children whoare mistakenly labeled as high risk might still ben-efit from intervention. To be successful in any pop-ulation, a prevention program must deliver servicesin a manner that minimizes stigmatization and maxi-mizes participant receptiveness to treatment. Servicessuch as tutoring, social skills training, and parent train-ing might enhance positive outcomes even amongchildren with subclinical levels of conduct problems.




Thus, even a loose criterion for screening may be cost-effective.

This study does not address whether receivingtreatment at the time of screening would have pre-vented the need for or substantially reduced the useof costly social services. This study also does not ad-dress the critical importance of a well-formulated andeffective intervention irrespective of when it is pro-vided. Clearly, the accurate designation of children ashigh risk is only the first step in the process of gettingthem the help they need. To that end, however, thisstudy has demonstrated that an inexpensive screeningin kindergarten can identify those children at risk foremotional and behavioral problems as manifested intheir use of mental health, special education, and/orjuvenile justice services within the following six years.

REFERENCES

Achenbach, T. M. (1991). Manual for the Child BehaviorChecklist/4–18 and 1991 Profile. Burlington, VT: Departmentof Psychiatry, University of Vermont.

Achenbach, T. M., Howell, C. T., McConaughy, S. H., & Stanger,C. (1998). Six-year predictors of problems in a national sam-ple: IV. Young adult signs of disturbance. Journal of the Amer-ican Academy of Child and Adolescent Psychiatry, 37, 718–727.

Agresti, A. (1990). Categorical data analysis. New York: John Wiley& Sons.

Barnett, W. S. (1996). Lives in the balance: Age-27 benefit-cost anal-ysis of the High/Scope Perry Preschool Program. Ypsilanti, MI:High/Scope Educational Research Foundation.

Bartusch, D. R. J., Lynam, D. R., Moffitt, T. E., & Silva, P. (1997).Is age important? Testing a general versus a developmentaltheory of antisocial behavior. Criminology, 35, 13–48.

Bennett, K. J., Lipman, E. L., Brown, S., Racine, Y., Boyle, M.H., & Offord, D. R. (1999). Predicting conduct problems: Canhigh risk children be identified in kindergarten and grade 1?Journal of Consulting and Clinical Psychology, 67, 470–480.

Blumstein, A., Cohen, J., Roth, J. A., & Visher, C. A. (Eds.) (1986).Criminal careers and career criminals. Washington, DC: Na-tional Research Council, National Academy Press.

Cohen, M. A. (1998). The monetary value of saving a high riskyouth. Journal of Quantitative Criminology, 4, 5–33.

Conduct Problems Prevention Research Group (1992). A develop-mental and clinical model for the prevention of conduct disor-der: The Fast Track Program. Development and Psychopathol-ogy, 4, 509–527.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation:Design & analysis issues for field settings. Boston: HoughtonMifflin Company.

Glascoe, F. P., Foster, E. M., & Wolraich, M. L. (1997). An economicanalysis of developmental detection methods. Pediatrics, 99(6),830–837.

Greene, W. H. (2000). Econometric analysis, 4th ed. Upper SaddleRiver, NJ: Prentice-Hall.

Hollingshead, A. A. (1979). Four-factor index of social status. Un-published manuscript, Yale University, New Haven, CT.

Institute of Medicine (1994). Reducing risks for mental disorder:Frontiers for preventive intervention research. Washington, DC:National Academy Press.

Levy, P. S., & Lemeshow, S. (1991). Sampling of populations: Meth-ods and applications. New York: Wiley.

Lipsey, M. W., & Derzon, J. H. (1998). Predictors of violent or se-rious delinquency in adolescence and early adulthood: A syn-thesis of longitudinal research. In R. Loeber & D. P. Farrington(Eds.), Serious and violent juvenile offenders: Risk factors andsuccessful interventions (pp. 86–105). Thousand Oaks, CA:Sage.

Lochman, J. E., & Conduct Problems Prevention Research Group(1995). Screening of child behavior problems for preventionprograms at school entry. Journal of Consulting and ClinicalPsychology, 63, 549–559.

Loeber, R. (1982). The stability of antisocial and delinquent childbehavior: A review. Child Development, 53, 1431–1446.

Loeber, R. (1991). Antisocial behavior: More enduring thanchangeable? Journal of the American Academy of Child andAdolescent Psychiatry, 30, 393–397.

Moffitt, T. E. (1993). Adolescence-limited and life-course-persistent antisocial behavior: A developmental taxonomy.Psychological Review, 100, 674–701.

Quay, H. C., & Petersen, D. R. (1987). Manual for the Re-vised Behavior Problem Checklist (Available from H.C.Quay, University of Miami, Box 248074, Coral Gables, FL33124).

StataCorp. (1999). Stata user’s guide: Release 6. College Station, TX:Stata Press.

Stiffman, A. R., Horwitz, S. M., Hoagwood, K., Compton, W.,Cottler, L., Bean, D. L., Narrow, W. E., & Weisz, J. R.(2000). The Service Assessment for Children and Adolescents(SACA): Adult and child reports. Journal of the AmericanAcademy of Child and Adolescent Psychiatry, 39(8), 1032–1039.

Weisz, J. R., Weiss, B., Alicke, M. D., & Klotz, M. L. (1987). Ef-fectiveness of psychotherapy with children and adolescents: Ameta-analysis for clinicians. Journal of Consulting and ClinicalPsychology, 55, 542–549.

Werthamer-Larsson, L., Kellam, S. G., & Wheeler, L. (1991). Ef-fects of first-grade classroom environment on shy behavior,aggressive behavior, and concentration problems. AmericanJournal of Community Psychology, 19, 585–602.

Yoshikawa, H. (1994). Prevention as cumulative protection: Effectsof early family support and education on chronic delinquencyand its risks. Psychological Bulletin, 115, 28–54.

Early Identification of Children At Risk for Costly Mental Health Service Use

Documents

Transcript of Early Identification of Children At Risk for Costly Mental Health Service Use