A Series of the

Tracking Behavioral Health Services to Children and Adolescentsand Their Families in Publicly-Financed Managed Care Systems

Promising Approachesfor Behavioral Health Services to Childrenand Adolescents and Their Familiesin Managed Care Systems

HCRT

P

4: Accountability andQuality Assurancein Managed CareSystemsMary I. Armstrong

HCRTP Health Care ReformTracking Project

1

Promising Approachesfor Behavioral Health Services to Children and Adolescents

and Their Families in Managed Care Systems

4: Accountability and Quality Assurancein Managed Care Systems

Mary I. Armstrong

IntroductionHealth Care Reform Tracking ProjectSince 1995, the Health Care Reform Tracking Project (HCRTP) has been trackingpublicly-financed managed care initiatives and their impact on children with mental healthand substance abuse (i.e., behavioral health) disorders and their families. The HCRTPis co-funded by the National Institute on Disability and Rehabilitation Research in theU.S. Department of Education and the Substance Abuse and Mental Health ServicesAdministration in the U.S. Department of Health and Human Services. Supplemental fundinghas been provided by the Administration for Children and Families of the U.S. Department ofHealth and Human Services, the David and Lucile Packard Foundation, and the Center forHealth Care Strategies, Inc. to incorporate a special analysis related to children involved in thechild welfare system. The HCRTP is being conducted jointly by the Research and TrainingCenter for Children’s Mental Health at the University of South Florida, the Human ServiceCollaborative of Washington, D.C. and the National Technical Assistance Center for Children’sMental Health at Georgetown University.

The HCRTP Promising Approaches Series is comprised of a number of thematic issuepapers that describe promising strategies, approaches, and features of service delivery tochildren and adolescents with behavioral health disorders and their families (particularly forchildren with serious and complex disorders) in publicly-financed managed care systems.The series draws on the findings of the HCRTP to date, highlighting relevant issues andapproaches that have surfaced through the HCRTP’s all-state surveys and in-depth impactanalyses in a smaller sample of 18 states. The papers are intended to be technical assistanceresources for states and communities as they refine their managed care systems to betterserve children and families.

HCRTP Health Care ReformTracking Project

2

Methodology for Study of Promising ApproachesThe strategies and approaches described in the Promising Approaches Series wereidentified by key state and local informants via the state surveys and site visit interviewsconducted as part of the HCRTP. More detailed information was then gathered about thesepromising approaches and features by members of the HCRTP team, including researchers,family members, and practitioners. In some cases, sites were visited so that targeted interviewscould be conducted with key stakeholders, such as system purchasers and managers,managed care organization representatives, providers, family members, and other child-servingagency representatives. In other cases, telephone interviews were conducted with key stateand local officials and family members, and supporting documentation was gathered andreviewed.

The series intentionally avoids using the term, “model approaches.” Although strategies,approaches, and features of managed care systems described in the series are perceivedby a diverse cross-section of key stakeholders to support effective service delivery for childrenwith behavioral health disorders and their families, the HCRTP has not formally evaluated theseapproaches. In addition, none of these approaches or strategies is without problems andchallenges, and each would require adaptation to individual state and local circumstances.Also, a given state or locality described in the series may be implementing an effective strategyor approach in one part of its managed care system and yet be struggling with other aspectsof the system.

The series does not describe the universe of promising approaches that are underway instates and localities. Rather, it provides a snapshot of promising approaches that have beenidentified through the HCRTP to date. New, innovative approaches are continually surfacing asthe public sector continues to experiment with managed care.

Each approach or strategy that is described in the series is instructive in its own right. At thesame time, the commonalities that exist across these strategies and approaches help illustratehow effective service delivery systems are organized within a managed care environment forthis population.

Each paper in the series focuses on a specific aspect of publicly-financed managed caresystems. This paper is on Promising Approaches to Managed Care Accountability andQuality Assurance.

3

OverviewPromising Approaches 4: Accountability and QualityAssurance in Managed Care SystemsThis paper identifies a number of managed care accountability and quality assuranceapproaches that support effective service-delivery to children with serious emotional problemsand their families. These promising approaches include both statewide approaches focused ona total population (Delaware, Pennsylvania, Iowa, and Utah) and local sites (HealthChoices inAllegheny County, PA; Dawn Project in Marion County, IN; and Delaware County, PA) focusedon a specific geographic area (see Table 1).

The paper addresses four areas of accountability: (1) availability of data for decisionmaking, (2) types of system performance information collected and tracked, (3) measurementof clinical and functional outcomes, and (4) quality measurement. The discussion of each areabegins with a brief review of the accountability and quality assurance issues related to publicsector behavioral health managed care that have been identified through the Health CareReform Tracking Project.

The discussion of accountability and quality assurance issues highlights differencesbetween carve out and integrated designs, design differences which the HCRTP has beenanalyzing since its inception. Following the review of each accountability issue, the paperdescribes related promising approaches. The final section for each issue summarizesthe common characteristics and challenges described by key stakeholders as well asrecommendations to other states and communities. The paper concludes with a list of resourcecontacts for the promising approaches, and a list of national organizations addressingthese issues.

Table 1Promising Approaches to Accountability andQuality Assurance in Managed Care Systems

Statewide Approaches Local Approaches

Delaware Allegheny County, PA

Iowa Delaware County, PA

Pennsylvania Marion County, Indiana

Utah

4

I. Availability of Data for Decision-MakingSince 1995 the Tracking Project has explored the extent to which adequate data are availablein managed care systems to guide decision-making regarding children’s behavioral healthservices. In both impact analyses, key stakeholders in most states reported that managementinformation systems (MIS) were insufficient to meet the accountability needs of managed caresystems. They also cited problems with obtaining encounter data from managed careorganizations (MCOs), a lack of electronic data reporting systems, and a lack of resourcesto manage and analyze data in a timely manner for either decision-making or qualityassurance purposes.

According to the 2000 State Survey, which was based on a sample of 35 managed caresystems in 34 states, adequate data were available to guide decision making in 59% of thereforms, leaving 41% of the reforms without adequate data. The survey also found thatadequate data related to child behavioral health services are more likely to be available in carveouts (63%) than in integrated reforms (43%).

In reforms without adequate data to guide managed care decision-making, the mostfrequently cited reasons for data inadequacies, as shown on Table 2, were inadequate MISsystems and lack of encounter data from MCOs.

Promising Approaches for Making Data Availablefor Decision Making

• Delaware’s Family and Child Tracking System (FACTS)Delaware’s Department of Services for Children, Youth, and their Families is an integratedstate agency for children’s services. The Department includes four divisions: (1) childmental health services, (2) child welfare, (3) juvenile justice, and (4) a support division.The MIS system is situated within the support division and is used by the children’s mentalhealth, child welfare, and juvenile justice divisions. Development of the Family and ChildTracking System (FACTS) was initiated in 1993. Three quarters of the development and

Table 2Reasons for Lack of Adequate Datain Reforms without Adequate Data

2000

Carve Out Integrated Total

Lack of encounter data 50% 50% 50%

Lack of staff capacityto analyze data 40% 25% 36%

Inadequate MIS system 50% 75% 57%

No tracking children’sbehavioral health services 20% 25% 21%

Other 30% 0% 21%

5

startup costs for FACTS were paid by Statewide Automated Child Welfare InformationSystems (SACWIS), an initiative authorized by Congress in 1993 to help states meet datacollection and reporting requirements of the Social Security Act. SACWIS continues tofund half the maintenance costs for FACTS.

The data indicators for FACTS include demographic information, identified problems,screening data, level of care recommendations, service providers, and service utilizationand cost data. The system includes a service admission form (with specific goals),screening information, treatment progress notes, and service plans. Care managersbecome familiar with a newly-assigned case by reviewing FACTS data (e.g., the serviceadmission form includes previous service history, previous contacts with child welfare and/or juvenile justice, and whether the child is active now in either system). Althoughbehavioral healthcare providers develop treatment plans that are not entered into FACTS,they are expected to incorporate the service plan goals that appear in FACTS into theirtreatment plan. In addition, the service plan is used for service authorization, andcounselors from the child welfare and juvenile justice divisions are able to access theprogress notes for children who are on their caseloads. Multi-division access to client datais designed to enhance service integration for the child and family receiving services.

In addition to its use as a management information system, FACTS is also a caremanagement tool. Care managers receive automatic reminders of due dates for ProgressReviews, Clinical Necessity Reviews (certification of medical necessity), etc. Delinquencyreports are sent to supervisors on a regular basis.

In Delaware, the Division of Child Mental Health Services (DCMHS) serves as themanaged care organization for children in the public sector who use more than a basicbehavioral health benefit (defined as 30 hours of mental health and/or substance abuseservices annually). Providers who are members of the Division’s provider network submitdata to FACTS on a monthly basis and receive a bundled case rate of $4,239/client/month.If data are not submitted on a client, the provider does not receive payment for that client.

DCMHS uses FACTS to submit cost recovery claims to Medicaid. The division receivesdivision-wide reports and team-level reports on cost and service utilization from FACTS ona quarterly basis.

FACTS data entry is PC (personal computer) based. According to users, data entry iseasy, and it takes about 20 minutes to enter data on a new child. Data can be submitted24 hours a day, 7 days a week, including through a remote access CITRIX system.Delaware is exploring transition to a web-based format using the Enterprise system,however there are concerns about confidentiality of data. There are routine reports for usein system management and there is also an ad hoc reporting capability where staff andproviders can request specific or one-time reports from FACTS but the process is slowerthan desired due to the limited number of staff who have the training and experience toquery the system. Providers can request a limited number of reports from FACTS, but theprocess reportedly is cumbersome.

FACTS is available free-of-charge to other states and communities. In Maryland,juvenile justice uses FACTS as its MIS system.

6

Advice to other states and communities regardingdata management systems:

� Make the system PC-based for data entry and distributed via a clientserver system which is not dependent on Local Area Networks.

� Include an easy way for users to correct errors and maintain databaseintegrity.

� Get advice from potential users during system development.

� Make sure that users can easily retrieve data and reports.

� Provide training and support to users during implementation.

� Include funds for a maintenance budget (e.g., funds are needed to makeFACTS compliant with the new HIPAA requirements).

7

II. Types of System Performance InformationCollected and Tracked

The findings of the 2000 State Survey are consistent with previous survey findings with regardto the types of performance indicators most likely to be tracked by public managed caresystems. These indicators are: (1) service utilization, (2) access (measured by child behavioralhealth penetration rates), and (3) total cost of child behavioral health services. Carve outs aresimilar to integrated systems in the rate at which they track service utilization and penetration.However, there is a dramatic difference in the rate at which they track the cost of children’sbehavioral health services. As shown in Table 3, nearly all carve outs (96%), but very fewintegrated systems (21%), track the total cost of children’s behavioral health services. Also,carve outs are more likely to track the service utilization of children involved in the childwelfare system.

Table 3Percent of Reforms Tracking and Using Various Types

of System Performance Information

2000

System Information Not Tracked Carve Out Integrated Total

Child behavioral healthpenetration rates 15% 86% 83% 85%

Child behavioral healthservice utilization 0% 100% 100% 100%

Child behavioral healthservice utilization by culturallydiverse group 25% 79% 60% 75%

Behavioral health servicesutilization by childrenin child welfare 26% 78% 50% 74%

Behavioral health servicesutilization by childrenin juvenile justice 54% 45% 50% 46%

Total aggregate cost ofchildren served withbehavioral health services 7% 96% 21% 93%

Cost per child served withbehavioral health services 21% 87% 50% 79%

Cost shifting amongchild-serving systems 84% 13% 25% 16%

8

The two system-level performance indicators least likely to be tracked by managed caresystems reportedly are service utilization by children in the juvenile justice system, tracked byfewer than half of the reforms (46%), and cost shifting among child serving systems, tracked byonly 16% of the reforms. Despite the low rate of systematic tracking of cost shifting, allegationsof cost shifting resulting from managed care reforms have been widespread and were made bystakeholders in both the 1997 and 1999 impact analyses as well as being reported in the 2000State Survey.

Although the few systems that reportedly collect information on cost shifting use theinformation for system planning, data collected for most of the other performance indicators isused for system planning in only about one-third to one-half of the reforms. The 1999 ImpactAnalysis also found that few data were available in states in the sample, despite reports thatperformance information was being collected. Furthermore, the 2000 State Survey indicates agap between information that is tracked and information that is used for system planning, and acontinuing problem in the capacity of managed care systems to generate data in a format andtime period that is helpful for planning and decision-making.

Promising Approaches in Tracking SystemPerformance Information

• Pennsylvania’s Performance/Outcomes ManagementSystem (POMS)HealthChoices is Pennsylvania’s statewide Medicaid managed care program that is beingimplemented in stages across the state. The goals of HealthChoices are to improve accessto care, quality of care, continuity of care, and management of scarce Medicaid resources.Behavioral health services are administered and financed separately from physical healththrough a behavioral health carve out. Counties have the right of first opportunity to act astheir own managed care entity through a contractual arrangement with the state Office ofMental Health and Substance Abuse Services. Counties may choose to subcontractmanaged care functions to nonprofit or commercial organizations.

The state Office of Mental Health and Substance Abuse Services has created aperformance monitoring system, tied to a Continuous Quality Improvement (CQI)process. The Performance/Outcomes Management System (POMS) consists of adatabase that is updated on a periodic basis through batch data file extracts that areobtained from a variety of data sources, including encounter data, enrollee eligibility anddemographic data, consumer/family satisfaction reports, behavioral health organizations'quarterly status files, and performance indicator reports. Work has begun to obtainsecondary data through data exchange agreements with other state agencies, as feasible,such as the education system and the juvenile justice system (see Figure 1).

9

The POMS serves three primary functions:

• Provides accountability for public funds expended through the State’s capitationpayments to the behavioral health organizations (BHOs)

• Provides a fair and objective evaluation of the BHOs that can be used for outcome-oriented incentives and sanctions

• Supports the Department and the BHOs in the implementation of a collaborativecontinuous quality improvement process

The integrated database provides the basis for producing quantitative performanceindicators related to the following system-level outcomes:

• Increase community tenure and less restrictive services

• Increase vocational and educational status

• Reduce criminal/delinquent activity

• Improve health care

Figure 1Performance/Outcome Management System (POMS)for Behavioral Health Managed Care Organizations

MH Consumer Data

Claims

Providers

D&A Consumer Data

D&A Consumer Data

Bureau of Drug & Alcohol

CIS Data

Satisfaction Data

Consumers Families & Persons in Recovery

Consumer Data

Reports

Aggregate & OtherReports

CQIReports

Selected Data Sets• Education System• Criminal Justice System• Juvenile Justice System• Law Enforcement System• Others

Secondary Data Sources

Person LevelEncounter Data Behavioral Health

Managed Care Organizations

Eligibility & Person Level Encounter Data

PA Office of Information

Systems

POMSReports

POMSDatabase

Commonwealth of PennsylvaniaHealthChoices–Lehigh-Capital BH RFP

10

• Increase penetration rates

• Increase consumer/family satisfaction

• Implement continuous quality improvement actions

• Increase the range of services and improve utilization patterns

Each outcome is measured by a set of performance indicators. For example, theoutcome regarding increasing vocational and educational status is measured by thefollowing indicators: (1) increasing school attendance, (2) increasing school retention,(3) increasing school performance, and (4) improving school behavior.

The POMS is used for a number of purposes, including production of anEarly Warning monitoring system. The Early Warning system is a quarterly report thatcompares all HealthChoices counties with one another on a number of quality indicatorsand reports on quarterly trends on a statewide basis. The indicators include:

• Rates of authorization by service categories

• Percent of members with a service denial

• Grievances

• 30-day inpatient readmissions

• Service authorizations for racial minorities

• Feedback from stakeholders (consumers, family members, providers, organizations)

• HealthChoices in Allegheny County, PennsylvaniaThe Department of Human Services in Allegheny County subcontracts with two entities,Allegheny HealthChoices and Community Care Behavioral Health, for theadministration and operation of behavioral health managed care. The County isresponsible for governance, financial risk, and administrative oversight of programoperations. Community Care provides the managed care functions for the County, withan emphasis on management of high risk and priority populations. Community Care alsoassumes financial risk on behalf of the County. The role of Allegheny HealthChoices isto monitor the performance and outcomes of the managed care functions performed byCommunity Care.

Allegheny HealthChoices and Community Care share responsibility for the tracking andmonitoring of system performance indicators. Allegheny HealthChoices tracks andanalyzes program trends in the areas of: health outcomes; consumer and familysatisfaction; quality of service; provider performance; financial management and outcomes;and coordination of care. Data sources include: claims data; complaints and grievances;ombudsman activities; consumer/family satisfaction data; clinical/fiscal chart audits; andfocus studies of special topics. Quality measures for children include: access standards(different requirements for various children's services); average length of stay in intensiveservices; utilization family-based services as an in-plan alternative; and the quality oftreatment and discharge planning.

Community Care has developed a Children's Team, which oversees the entirespectrum of services for children and adolescents. The team's mission includescommunication and collaboration with child serving systems, maintaining children in theleast restrictive and least intrusive environment, and securing the safety and needs of allchildren. Its decision-making process is based upon Pennsylvania's medical necessity

11

criteria and the application of the system of care principles of child-centered, family-focused, community-based, multi-system, culturally competent, and least-restrictiveenvironment. Managed care strategies used by the Children's Team of Community Careinclude care management, a physician advisor review process, team approval of newprograms and services, the development of performance standards for levels of care,provider meetings and training, utilization management, and quality reviews. For example,the monitoring plan for children's behavioral health rehabilitation services includes the useof outcomes data through the Columbia Impairment Scale, review of monthly reports ofservice utilization, and quality reviews.

• Iowa’s Monitoring Plan for Performance IndicatorsIn 1999 Iowa's Medicaid agency and the Department of Public Health integrated twoseparate carve outs for mental health and substance abuse services into one statewidebehavioral health care program, the Iowa Plan for Behavioral Health. The two stateagencies contract with one for-profit managed care organization on a prepaid capitationbasis to provide comprehensive mental health and substance abuse services statewide.

The Iowa Plan for Behavioral Health contract for fiscal year 2003 includes a set ofperformance indicators that are divided into three broad categories: (1) indicators withmonitoring only, (2) indicators carrying financial incentives, and (3) indicators with financialpenalties. The performance indicators that are “monitored only” (i.e., without fiscalincentives or penalties) fall within the domains of: (1) consumer involvement and qualityof life, (2) network management, (3) access and array, (4) quality and appropriateness,(5) integration and interface, (6) quality of care, and (7) administrative accountability.Examples of performance indicators with monitoring only are:

• Consumer satisfaction surveys shall be conducted at least two times percontract period.

• 98% of all enrollees who request any Iowa Plan service will be offered a service.

• The number of Iowa Plan enrollees, reported overall and separately for children andadults, for whom wraparound and rehabilitation and support services were providedduring the month.

For each of the nine performance indicators carrying financial incentives, the contractspecifies the annual dollar amount the contractor shall be paid if the indicator is attained($110,000 to $120,000). Examples of performance indicators with financial incentives are:

• The percent of involuntary admissions for mental health treatment to 24 hourinpatient settings shall not exceed 20% of all children admissions and 15% of alladult admissions.

• Based on claims during the contract period, the contractor shall provide services toat least 13.5% of Iowa Plan enrollees.

• At least 4.5% of mental health service expenditures will be used in the provision ofintegrated services and supports, including natural supports, consumer runprograms, and services delivered in the home of the enrollee.

For the 10 performance indicators with financial penalties, the state assesses damagesif the contractor fails to comply with the minimum performance expectations for any twoquarters in a contract period. The first occurrence of non-compliance for a performance

12

indicator is assessed a penalty of $5,000, the second occurrence penalty ranges from$10,000 to $25,000, and the penalty for a third occurrence is either $15,000 or $50,000.Performance indicators with financial penalties include:

• The percentage of enrollees under the age of 18 discharged from a mental healthinpatient setting to a homeless or emergency shelter shall not exceed 3% of allmental health inpatient discharges of children under the age of 18.

• The contractor shall arrange or participate in at least 20 Joint Treatment Planningconferences per month.

• Medicaid claims shall be paid or denied within the following time periods:

– 85% within 12 calendar days– 90% within 30 calendar days– 100% within 90 calendar days

The perception of the Iowa Plan administrators is that the use of financial incentivesand penalties tied to performance indicators in the managed care contract has a number ofadvantages. First, the performance indicators provide an opportunity for the managed careorganization to influence provider behaviors. Second, the indicators direct attention of themanaged care organization and the provider network to client outcomes. Third, spendinglevels can be monitored because the percentage of services offered that are authorized isreported on a monthly basis. Finally, the data that are collected regarding the performanceindicators are useful in refuting myths about both process and outcomes.

Advice to other states and communities regarding trackingsystem performance information:

� Know what performance outcomes and indicators you want to collect beforethe system is implemented.

� Make sure that the performance indicators include what is needed forwaivers and other administrative requirements.

� Re-organize at the state level one year before implementation so thatthe infrastructure (e.g., Management Information System) is in place.

� Engage all stakeholders (advocates, county government, legislators,consumers/families, and providers) early in the development process.

� Recognize that performance outcomes systems can be burdensomeand time-intensive for providers.

� Develop a strategy for providing feedback to MCOs, providers, and parents.

� Commitment to quality improvement entails resources for increasedpersonnel and different skill sets.

� Be cautious about making changes post-implementation in system levelperformance outcomes.

� Define each indicator with the managed care organizations, includingdata elements (numerator and denominator) and the data source.

� Recognize and address the challenge of balancing quality and cost.

13

III. Measurement of Clinical and FunctionalOutcomes

The 2000 State Survey found a continuing increase in the measurement of child-level clinicaland functional outcomes related to children's behavioral health in managed care systems.As shown in Table 4, the percentage of systems that reportedly measure child outcomesrelated to children's behavioral health increased from 52% in 1995, to 63% in 1997-98,and to 90% in 2000.

Despite the steady increase in reforms measuring clinical and functional outcomes, thefindings of the impact analyses indicated that the outcome measurement systems in mostmanaged care systems were at an early stage of development. The early developmental stageis still evident in the 2000 State Survey.

As shown in Table 5, only about one fourth (26%) of the reforms (all carve outs) haveresults from their outcome measurement systems. Another 26% described their outcomemeasurement systems as implemented, but with no results available yet. Finally, in 44% of thereforms, outcome measurement was described as being at an early stage of development, andan additional 4% reported that the system was developed but not yet implemented. Carve outsreportedly are ahead of integrated systems in the measurement of children's behavioral healthclinical and functional outcomes. More integrated systems are at early stages of this process orhave outcome measurement systems but no results as yet.

Table 4Measurement of Clinical and Functional Outcomes

in Managed Care Systems Related to Children’s Behavioral Health

Percent of Change1995 1997–98 2000 1995 1997/98Total Total Carve Out Integrated Total –2000 –2000

Managed care reformmeasures clinical andfunctional outcomes 51% 63% 96% 71% 90% +39% +27%

Managed care reformdoes not measureclinical and functionaloutcomes 49% 37% 4% 29% 10% -39% -27%

14

Promising Approaches in Measurement of Clinical andFunctional Outcomes

• The Dawn Project, Marion County (Indianapolis), IndianaThe Dawn Project is a behavioral health carve out serving children who have seriousemotional problems in Marion County, Indiana. The target population includes children whoare involved in at least two child-serving systems, are at risk of or in residential placement,and have serious functional impairments at home, in school, and in the community. Severalstate and county agencies contract with Indiana Behavioral Health Choices, a non-profitmanaged care entity, to administer the Dawn Project's performance information system aswell as the clinical and financial processes. Choices uses an individualized, strengths-based approach to serve children and families, using child and family teams and acommunity resource network.

Choices has developed a technological infrastructure that provides fiscal and clinicalaccountability to its key stakeholders, including payers, providers, and children andfamilies. In 1999, Dawn worked with an Outcome Committee of its CommunityConsortium to develop a set of performance indicators and outcome measures. After usingthese indicators for a three-year period, there was agreement that some outcomes weredifficult to measure. The Outcome Committee re-convened to examine outcome definitionsand measures, data collection, and data interpretation. In February 2002 the Dawn Projectadopted a new set of performance indicators, including clinical and functional outcomes.The organizing framework for the new outcome measures is made up of the followinggoals:

• Provide high quality care that results in improved outcomes for the child and family

• Include parents/families in decision-making

• Decrease the cost of serving children with the most disturbed and disturbingconditions in Marion County

• Be accountable to all stakeholders

Table 5Stage of Development of Measurement of

Clinical and Functional Outcomes

2000

Carve Out Integrated Total

In early stage of developingmeasurement system 41% 60% 44%

Developed but not yet implementedmeasurement system 5% 0% 4%

Implementing measurementsystem but do not yet have results 23% 40% 26%

Implementing measurementsystem and have results 32% 0% 26%

15

Table 6 shows the clinical and functional outcomes for the first goal noted above.As shown in Table 6, a set of outcomes defines each goal, and performance indicatorsmeasure progress towards each outcome.

Table 6The Dawn Project: Outcome Measures

Goal #1: Provide high quality care, which results in improved outcomes for the child and family.

I. Improved child and family functioning

A. Improved school functioning

1. Grade reports, attendance reports, behavior reports, suspension/expulsion reports2. The Clinical Manager Treatment Plan level rating

B. Improved records with the child welfare and the juvenile justice system

1. Percent of families with no further substantiated incidences of child abuse or neglect,which results in removal of the child from the home during involvement.

2. Percent of families with no further substantiated incidences of child abuse or neglect,which results in removal of the child from the home for a period of six and twelvemonths from disenrollment.

3. Percent of children with no further incidences of delinquency, runaway or truancycharges, or violation of terms of probation which results in placement failure duringenrollment.

4. Percent of children with no further incidences of delinquency, runaway or truancycharges, or violation of terms of probation which results in placement failure for aperiod of six and twelve months from disenrollment.

C. Improved records for community supervision for Department Corrections youth

1. Number of youth with arrests for offenses more severe than original offense2. Number of youth with arrests for offenses less severe than original offense3. Number of youth with technical violations other than for placement4. Number of youth with technical violations that include placement failure5. Number of youth with technical violations for placement failure only.6. Number of youth with arrests in first year after disenrollment.

D. Improved CAFAS scores

1. Measured with CAFAS at intake, every six months, and dischargeE. Progress in Service Coordination Plan

1. Measured by monthly team report and The Clinical Manager Treatment Plan level ratingF. Fewer days in out of home placement

1. Measured by Dawn national evaluation placement data

II. Increased family autonomy

A. Decrease in number of paid providers

1. Measured by service usage and payment dataB. Caregiver Strain Questionnaire

1. Measured by Questionnaire at intake, every 6 months until discharge, and12 monthsafter discharge

16

In the Dawn Project, data collection is the responsibility of the Service Coordinators.Service Coordinators input data (demographics, case history, treatment plans, progressnotes) into Clinical Manager, a software program that serves as both a clinical medicalrecord and a fiscal record. Each Service Coordinator has a desktop computer, and eachsupervisor has two laptop computers available for the unit. For those Service Coordinatorswho prefer to do data entry out of the office, laptops can be signed out and data entryaccomplished through a dial-up system.

There are several characteristics of the Dawn Project that help to ensure accurate andtimely data:

• Data entry is addressed in the productivity standards for the Service Coordinators,partly because the Progress Notes are used to bill Medicaid and other payers.

• The model is supervisory-intensive (7: 1). Caseloads are relatively small (8 to 10).

• Supervisors review treatment plans weekly with the Service Coordinators, and apsychiatrist reviews plans every 60 days.

• Service Coordinators are observed “in action” on a quarterly basis.

• Supervisors are responsible for pre-authorization of services and for managing theirunit's budget.

• Supervisors recognize the value of “real-time data” as a management tool.

Advice to other states and communities regarding clinicaland functional outcome systems:

� Clinical and functional outcome measurement is always a “workin progress.” A system needs to be in place for continuous review andmodification.

� Key stakeholders need to be involved in the development and monitoringof clinical and functional outcomes. Families, for example, need to beasked what is important for them regarding their child's progress.

� Incentives need to be in place that facilitate and support data collection.

� Data need to be available for quality assurance, management, andplanning activities.

17

IV. Quality MeasurementThe 1997–98 State Survey found that the majority of managed care systems (88%) incorporatesome child-specific quality measures related to behavioral health, with carve outs more likelyto do so than reforms with integrated designs. The majority of reforms responding to the 2000State Survey also reported including some child-specific measures related to behavioral healthin their quality measurement systems (71%), although this represents a 17% decrease inreforms with child-specific quality measures (see Table7).

The Tracking Project has also identified the methods that states use to measure quality.The 1999 Impact Analysis identified the following processes:

• On-site reviews and audits of MCOs

• Focus groups with consumers and family members

• Report card with standard indicators for each MCO, enabling comparisons amongMCOs

• Committees and work groups focusing on quality

• Review and analysis of grievances, appeals, and complaints

• Requirements that each MCO develop and implement its own quality measurementand improvement process

• Contract with external entity to conduct quality reviews and studies

Promising Approaches in Quality Measurement

• Utah’s Prepaid Mental Health PlanUtah operates one Medicaid managed care carve out program for mental health

services, known as the Prepaid Mental Health Plan, for TANF, disabled populations andmedically needy individuals residing in 25 of its 29 counties. The state Medicaid authorityhas sole-source contracts with eight Community Mental Health Centers using a capitationpayment method. Each PMPH subcontracts with hospitals for inpatient psychiatric care,and to varying extents, with community providers for outpatient mental health services.

Table 7Percent of Reforms Incorporating Quality Measures Specific

to Child and Adolescent Behavioral Health Services

1997–98 2000 Percent of ChangeTotal Carve Out Integrated Total 1997/98–2000

Managed care system incorporateschild-specific behavioral healthquality measures 88% 74% 57% 71% -17%

Managed care system does notincorporate child-specificbehavioral health quality measures 12% 26% 43% 29% +17%

18

Utah's quality monitoring approach includes a children's team that visits each PMHPannually. For a sample of clients, team members review records, meet with staff, and meetwith families to assess the quality of care. During the development of the monitoringprocess, the Division of Mental Health contracted with Utah Allies as Families, thestatewide family organization and Utah's chapter of the Federation of Families forChildren's Mental Health, to develop a survey instrument for interviewing families.The family organization worked with a children's mental health researcher to develop andvalidate the instrument, known as the Family Perception of Care Scale. Initially, Familiesas Allies was subcontracted to conduct the site visits, including interviewing parents andadministering the survey. Survey data were analyzed and reported for each PMHP and ona statewide basis. The Division of Mental Health subsequently developed a similar surveyinstrument. Currently family members are members of the monitoring teams and therevised survey instrument is used for family interviews.

• Delaware County's Family Satisfaction TeamAs noted earlier about Pennsylvania, in HealthChoices Behavioral Health Services,

counties operate Pennsylvania's behavioral health managed care system. Counties havethe choice of either subcontracting with a private sector BHO or forming their own nonprofitmanaged care organization to manage care for both mental health and substance abuseservices. Each managed care entity is responsible to implement a comprehensiveapproach for the measurement of consumer and family satisfaction, including a FamilySatisfaction Team Program.

In Delaware County, Pennsylvania the county Department of Human Servicessubcontracts with Magellan Behavioral Health to perform the managed care functions.The county's Office of Behavioral Health and Magellan have developed a DelCare QualityImprovement Plan for HealthChoices. Evaluation of consumer and family perceptionsand experience is a critical component of this quality improvement process. The countyOffice of Behavioral Health subcontracts with the Parents Involved Network of theMental Health Association of Southeastern Pennsylvania for the operation of theFamily Satisfaction Team.

The Family Satisfaction Team is composed of a team leader and three family members.The team assesses family satisfaction with service delivery and the process for accessingthe services. The team's motto is, “evaluating satisfaction through family interaction,” andits goal is to ensure that services provided to children and adolescents are child-centeredand family-driven. The process developed by the team focuses on surveying parents whohave recently participated in an interagency service planning team meeting. The domainscovered by the survey include: (1) provision of information to families beforehand regardingthe interagency team meeting process, (2) whether the family felt comfortable during themeeting, (3) quality of the assessment and service plan, and (4) the accessibility of themeeting. Findings from the survey are compiled and reported on a regular basis to Magellan,the County, and providers. As a result of this process, the following actions have taken place:

• Providers have developed methods to ensure that families can bring an advocateand/or support person to team meetings.

• Meetings are scheduled at times convenient for the family.

• Magellan has taken steps to ensure that the same care manager follows anassigned child throughout treatment.

19

ResourcesMore detailed information about each of the accountability approaches described in this papercan be obtained from the following individuals:

Allegheny County, PennsylvaniaMary Fleming, CEOAllegheny HealthChoices, Inc.444 Liberty Avenue, Suite 240Gateway 4Pittsburgh, PA 15222Phone: (412) 325-1100, Ext. 771E-mail: mfleming@ahci.org

DelawareNancy WiddoesDelaware Department of Services forChildren, Youth and Families1825 Faukland RoadWilmington, DE 19805Phone: (302) 633-2603E-mail: nwiddoes@state.de.us

PennsylvaniaMichael RootCommonwealth of PennsylvaniaMental Health and Substance AbuseServicesBeechmont Building #32PO Box 2675Harrisburg, PA 17105Phone: (717) 772-7992E-mail: miroot@state.pa.us

IowaDennis Janssen, Bureau ChiefBureau of Managed Care & ClinicalServicesDepartment of Human ServicesHoover State Office BuildingFifth Floor, NEDes Moines, IA 50319Phone: (515) 281-8747E-mail: djansse@dhs.state.ia.us

Dawn ProjectKnute RottoIndiana Behavioral Health Choices4701 North Keystone Avenue, Suite 150Indianapolis, IN 46205Phone: (317) 205-8202E-mail: krotto@kidwrap.org

UtahLori CerarUtah Allies with Families450 East 1000 North, Suite 311North Salt Lake, UT 84054Phone: (801) 292-2515Email: awfamilies@msn.com

Delaware County, PAChristina CorpDelaware County Parents Involved Network135 Long Lane, 3rd FloorUpper Darby, PA 19082-3116Phone: (610) 713-9401E-mail: ccorp@mhasp.org

20

The Children's Mental HealthBenchmarking ProjectThe goal of this project is to gather,compare, and disseminate administrativedata from states and counties so thatpolicymakers can compare their ownlocality to others in regard to certain keyindicators. The project focuses ondeveloping indicators in the domains ofaccess, utilization, expenditures andintersystem involvement.

Contact Information:Richard H. DoughertyDougherty Management Associates, Inc.(721) 865-8003E-mail: public@doughertymanagement .comWeb: http://www.doughertymanagement.com/

Outcomes Roundtable forChildren and FamiliesThe mission for the Outcomes Roundtableis to bring together multiple perspectivesand expertise to provide leadership thatstimulates culturally competent and datadriven improvement in policy, practice, andresearch for children and youth withemotional and behavioral problems andtheir families. The current focus is thedevelopment of an appropriate outcomeaccountability system within child servicesystems.

Contact Information:Ann DoucetteE-mail: adoucette@aol.com

Trina OsherE-mail: tosher3@comcast.net

The following national groups and organizations have useful information and resourcesin the areas of accountability and quality assurance in managed care systems.

The Forum on Performance Measures inBehavioral Healthcare and RelatedService SystemsThe purpose of the Forum is to facilitatecommon approaches to the development,testing, and adoption of performancemeasures in behavioral healthcare andrelated services. The Forum provides avenue for collaboration, coordination, andcommunication among various initiatives,both public and private, which are workingto measure service access and delivery,quality, and outcomes.

Contact Information:John Bartlett(404)942-3616E-mail: johnbarlett@performancemeasures.org

21

AppendixAll reports of the HCRTP are available from the Research and Training Center for Children'sMental Health, University of South Florida (813) 974-6271:

Armstrong, M. I., (2002). Health care reform tracking project (HCRTP): Promisingapproaches for behavioral health services to children and adolescents and their families inmanaged care systems – 4: Accountability and quality assurance in managed care systems.Tampa, FL: Research and Training Center for Children’s Mental Health, Department of Childand Family Studies, Division of State and Local Support, Louis de la Parte Florida MentalHealth Institute, University of South Florida. (FMHI Publication #211-4)

Hepburn, K. & McCarthy, J. (2003). Health care reform tracking project (HCRTP): Promisingapproaches for behavioral health services to children and adolescents and their families inmanaged care systems – 3: Making interagency initiatives work for the children and families inthe child welfare system. Washington, DC: National Technical Assistance Center for Children'sMental Health, Georgetown University Center for Child and Human Development. (GeorgetownUniversity Publication #211-3)

McCarthy, J. & McCullough, C. (2003). Health care reform tracking project (HCRTP):Promising approaches for behavioral health services to children and adolescents and theirfamilies in managed care systems – 2: A view from the child welfare system. Washington, DC:National Technical Assistance Center for Children's Mental Health, Georgetown UniversityCenter for Child and Human Development. (Georgetown University Publication #211-2)

Pires, S.A (2002). Health care reform tracking project (HCRTP): Promising approaches forbehavioral health services to children and adolescents and their families in managed caresystems – 1: Managed care design & financing. Tampa, FL: Research and Training Center forChildren's Mental Health, Department of Child and Family Studies, Division of State and LocalSupport, Louis de la Parte Florida Mental Health Institute, University of South Florida. (FMHIPublication #211-1)

Stroul, B.A., Pires, S.A, & Armstrong, M.I. (2001). Health care reform tracking project:Tracking state health care reforms as they affect children and adolescents with behavioralhealth disorders and their families – 2000 State Survey. Tampa, FL: Research and TrainingCenter for Children's Mental Health, Department of Child and Family Studies, Division of Stateand Local Support, Louis de la Parte Florida Mental Health Institute, University of South Florida.(FMHI Publication #198)

Pires, S.A., Stroul, B.A., & Armstrong, M.I. (2000). Health care reform tracking project:Tracking state health care reforms as they affect children and adolescents with behavioralhealth disorders and their families – 1999 Impact Analysis. Tampa, FL: Research and TrainingCenter for Children's Mental Health, Department of Child and Family Studies, Division of Stateand Local Support, Louis de la Parte Florida Mental Health Institute, University of South Florida.(FMHI Publication #183)

Pires, S.A., Armstrong, M.I., & Stroul, B.A. (1999). Health care reform tracking project:Tracking state health care reforms as they affect children and adolescents with behavioralhealth disorders and their families – 1997/98 State Survey. Tampa, FL: Research and TrainingCenter for Children's Mental Health, Department of Child and Family Studies, Division of Stateand Local Support, Louis de la Parte Florida Mental Health Institute, University of South Florida.(FMHI Publication #175)

22

Stroul, B.A., Pires, S.A., & Armstrong, M.I. (1998). Health care reform tracking project:Tracking state health care reforms as they affect children and adolescents with behavioralhealth disorders and their families – 1997 Impact Analysis. Tampa, FL: Research and TrainingCenter for Children's Mental Health, Department of Child and Family Studies, Division of Stateand Local Support, Louis de la Parte Florida Mental Health Institute, University of South Florida.(FMHI Publication #213)

Pires, S.A., Stroul, B.A., Roebuck, L., Friedman, R.M., & Chambers, K.L. (1996). Healthcare reform tracking project: Tracking state health care reforms as they affect children andadolescents with behavioral health disorders and their families – 1995 State Survey. Tampa, FL:Research and Training Center for Children's Mental Health, Department of Child and FamilyStudies, Division of State and Local Support, Louis de la Parte Florida Mental Health Institute,University of South Florida. (FMHI Publication #212)

The following special analyses related to the child welfare population are available from theNational Technical Assistance Center for Children's Mental Health, Georgetown University(202) 687-5000:

McCarthy, J. & Valentine, C. (2000). Health care reform tracking project: Tracking statehealth care reforms as they affect children and adolescents with behavioral health disordersand their families – Child Welfare Impact Analysis – 1999. Washington, D.C.: National TechnicalAssistance Center for Children's Mental Health, Georgetown University Child DevelopmentCenter.

Schulzinger, R., McCarthy, J., Meyers, J., de la Cruz Irvine, M., & Vincent, P. (1999). Healthcare reform tracking project: Tracking state health care reforms as they affect children andadolescents with behavioral health disorders and their families – Special Analysis – ChildWelfare Managed Care Reform Initiatives. Washington, D.C.: National Technical AssistanceCenter for Children's Mental Health, Georgetown University Child Development Center.

Promising Approachesfor Behavioral Health Services to Children and Adolescents

and Their Families in Managed Care Systems

4: Accountability and Quality Assurancein Managed Care Systems

Mary I. ArmstrongFeaturing

• Introduction to HCRTP• Methodology for Promising

Approaches• Overview of Accountability and

Quality Assurance in ManagedCare Systems

• Availability of Data forDecision-Making

• Delaware’s Family and ChildTracking System (FACTS)

• Types of System PerformanceInformation Collectedand Tracked

• Pennsylvania’s Performance OutcomesManagement System (POMS)

• HealthChoices in Allegheny County, PA• Iowa’s Monitoring Plan for

Performance Indicators• Measurement of Clinical and

Functional Outcomes• The Dawn Project, Marion County, IN• Quality Measurement• Utah’s Prepaid Mental Health Plan• Delaware County’s Family Satisfaction Team• Resources• Appendix

FMHI Library/Technical PublicationsLouis de la ParteFlorida Mental Health InstituteUniversity of South Florida13301 Bruce B. Downs BoulevardTampa, FL 33612-3899Phone: 813-974-4471Fax: 813-974-7242SunCom Phone: 574-4471Sun Com Fax: 574-7242Email: cone@fmhi.usf.edu

__________________________________________Name Title

__________________________________________Organization/Agency

___________________________________________Street Address Suite

___________________________________________City State Zip

___________________________________________Country

( )______________ ( )_______________Phone Fax

__________________________________________Email Web Address

To Order, Mail or Fax this Form with Payment to:FMHI Library/Technical PublicationsLouis de la Parte Florida Mental Health Institute (FMHI)The University of South Florida13301 Bruce B. Downs Blvd. • Tampa, FL 33612-3899

Fax: 813-974-7242 SunCom Fax: 574-7242Phone: 813-974-4471 SunCom Phone: 574-4471 Email: cone@fmhi.usf.edu

� Please send ___ copies of the HCRTP: Promising Approaches 4:Accountability and Quality Assurance in Managed CareSystems — FMHI Technical Document #211-4. ($6 each copy–Including Shipping and Handling)Payment accepted by Purchase Order, Check, or Money Order payable to:Louis de la Parte FMHI Auxiliary Fund

Suggested APA Citation:Health care reform tracking project (HCRTP): Promising approaches for behavioral healthservices to children and adolescents and their families in managed care systems – 4:Accountability and quality assurance in managed care systems. Tampa, FL: Research andTraining Center for Children’s Mental Health, Department of Child and Family Studies, Divisionof State and Local Support, Louis de la Parte Florida Mental Health Institute, University ofSouth Florida. (FMHI Publication #211-4)

Produced in Cooperation with:

Research and Training Center for Children’s Mental HealthDepartment of Child and Family Studies

University of South FloridaTampa, Florida

National Technical Assistance Center for Children’s Mental HealthGeorgetown University Center for Child and Human Development

Washington, D.C.

Human Service CollaborativeWashington, D.C.

28 Pages

Request

FMHI Technical

Document

#211-4

$6ooWhileSuppliesLast

HCRT

P: O

rder

For

mH

ealth

Car

e R

efor

m T

rack

ing

Proj

ect

HCRPT Technical Publications in printand/or on-line as Acrobat PDF files:http://www.fmhi.usf.edu/institute/pubs/bysubject.html

• Promising Approaches Series– Accountability & Quality Assurance – Doc. #211-4 (Purple Bk)– Interagency Initiatives – Doc. #211-3 (Green Bk)– Child Welfare View – Doc. #211-2 (Burgundy Bk)– Managed Care Design & Financing – #211-1 (Blue Bk)

• 2000 State Survey – Doc. #198 (Purple Bk)• 1999 Impact Analysis – Doc. #183 (Green Bk)• 1997–98 State Survey – Doc. #175 (Blue Bk)• 1997 Impact Analysis – Doc. #213 Print only (Red Bk)

HCRTP Health Care ReformTracking Project

Florida Mental Health InstituteFMHI Louis de la Parte

FMHILouis de la Parte

InstituteMental HealthFlorida

Florida Mental Health InstituteFMHI Louis de la Parte

© 2003 The Louis de la Parte Florida Mental Health Institute

Research and Training Center for Children’s Mental HealthDepartment of Child and Family Studies

University of South FloridaTampa, FL

National Technical Assistance Center for Children’s Mental HealthGeorgetown University Center for Child and Human Development

Washington, DC

Human Service CollaborativeWashington, DC

Florida Mental Health InstituteFMHI Louis de la Parte