Draft Conference Paper - pdx.edu Carl… · Draft Conference Paper ... Intergovernmental barriers...
Transcript of Draft Conference Paper - pdx.edu Carl… · Draft Conference Paper ... Intergovernmental barriers...
1
2nd International Conference on Government Performance Management and Leadership
Draft Conference Paper
October 1st-2nd, 2011
Title: Intergovernmental barriers and the development of state performance management
systems: Results from the federal National Outcome Measures project
Author: Carl Foreman, Doctoral Student Portland State University
Abstract
Federal and state agencies must cooperate to provide mental health services, which
creates a complex implementation environment. This environment can cause significant
challenges for measuring agency performance and accountability. Complex
implementation chains are a functional component of federal programs. This requires
states to perform a central role in the reporting and ensuring of accountability. Viewed
from a state perspective, federal programs create a dilemma for reporting agencies. In
particular, federal block grants have various levels of funding that are not always aligned
with the reporting requirements. Misalignment between funding and reporting can cause a
2
disproportionate impact on agency resources. Federal programs that require significant
reporting through administrative data sets are the most easily accessible input for the
development of state-level performance management systems. There are limitations in the
use of administrative data in estimating program impact although the data has been proven
useful in identifying policy levers (Heinrich, 2002). This study examines the usefulness of
federally collected data for the development of state performance management systems.
Despite efforts to collect standardized national data at the federal level, states were found
to not sufficiently report on evidence-based practice measures to provide useful, in-depth
information for performance management systems. However, the results do provide
information regarding the lack of monitoring of evidence-based practices. This information
is useful to states in their decision to allocate resources when conflicting state and federal
goals are present.
Introduction In Int
SAMSHA began tracking ten National Outcomes Measures in FY 2004 with the goal
of improving state performance management systems (Manderscheid, 2006). These
measures were included in the state’s federal block grant reporting requirements. There
are two federal block grants associated with the provision of behavioral health services.
The Community Mental Health Block Grant (CMHBG) is administered through the Center
for Mental Health Services (CMHS), under the Substance Abuse and Mental Health Services
Administration (SAMSHA, 2011a). The second behavioral health related block grant is the
Substance Abuse Prevention and Treatment (SAPT) grant which was administered by the
3
Center for Substance Abuse Treatment (CSAT) and Center for Substance Abuse Prevention
(CSAP) (SAMSHA, 2011a). Beginning in FY 2012, the two grants will be combined into a
uniform block grant application (SAMHSA, 2011). In 2009, the National Association of
Mental Health Program Directors National Research Institute (NRI) redeployed the state
level National Outcomes Measures data collection in nine states to increase the quality of
captured data (National Association of State Mental Health Program Directors Research
Institute, Inc., 2009). The National Outcome Measures were planned to be used in
conjunction with a decision support system that was never fully implemented
(Manderscheid, 2005). Evidence-based practices represent one domain in the National
Outcome Measures; evidence-based practices have also been a policy consideration of the
states (Rieckmann, et al., 2008). The State of Oregon began a titrated implementation of a
state legislative mandate requiring a 25% increase in purchasing of evidence-based
practices by biennium (Oregon Revised Statute 182.525). The state of Oregon had the
similar goal of increasing the use of evidence-based practices as SAMSHA but chose a
different implementation plan. This plan mandated the purchase of evidence-based
practices while the federal agency did not use a mandate and selected ten evidence-based
practices that were chosen through a national consensus process.
The results point to the need for further efforts to increase data quality in order to
provide sufficient data for state and federal decision-making. This exploratory analysis
focused on the quality of the national outcome data in providing useful performance
information to the states. The results point to a larger issue in the sustainability of
performance management systems such as increasing the usefulness of performance data
across intergovernmental agencies.
4
Performance Management Systems: Definitions and Requirements
Bird et al. (2005) identify three functions of performance monitoring in the public
sector: determining the policy impact, determining the level of agency performance, and
stewardship of public funds. Pidd (2007) adds the support of central control and symbolic
action as the reasoning for the pervasive use of performance monitoring in the public
sector. Specific to health care, Commission on Behavioral and Social Sciences and Education
(CBASSE) describes good performance measures as aimed at specific objectives and
supported by adequate data. They should also be sufficiently valid, reliable, and responsive
to changes in the environment (Perrin & Koshel, 1997). Third-party governance adds
complexities into the use of performance measures to ensure accountability or
performance (Posner, 2002). In third-party governance, performance occurs with a
contracted entity and does not necessarily reflect the performance of the state agency
monitoring the contract. In the practical world of system delivery in a third-party
governance structure, completely valid and reliable data is not necessarily available.
Therefore, performance management requires providing information for administrative
decision-making from sufficiently accurate data sources from within and outside the
agency.
Performance Measures
Performance Measures need to be able to meet the internal management needs
across multiple purposes (Behn, 2003). Dubnick and Frederickson (2009) developed six
categories of accountability used in the Department of Human Services. Dubnick and
Fredrickson characterize federal performance measures as relating management
5
techniques to political processes (2009). Additionally, Dubnick and Fredrickson (2009)
state that there is a lack of evidence that performance measures truly impact issues of
equity in public law. Issues of equity and individual rights are a fundamental component in
the provision of mental health services. There are definite limits to the expectations for
performance measures and management. While performance measures and management
provide information regarding certain aspects of performance, they do not provide all of
the information regarding the activities of the agency. These limitations are especially
pronounced in areas where there are long implementation chains, and the state agency is
accountable to multiple stakeholders. Wright views the relationship of federal, state, and
local governments through an overlapping authority model. In this model, each layer of
government is interdependent, has overlapping relationships and exerts authority through
bargaining with the other intergovernmental layers (Wright, 1988). There are some
challenges in the interaction of intergovernmental layers. For example, federal grants
provide the federal government with a voice in state decision-making; however, this role
causes a diffusion of responsibility among federal and state representatives with their
respective constituents (Derthick, 1970, p.214).
Third-Party Implications
Performance measures are directed to efficiencies and effectiveness, but they do
not capture the total tasks of the agency or the interaction of agencies with third-party
contractors. Mechanisms for addressing equity have similar challenges in capturing
adequate data in order to make informed decisions. For example, a grievance process for
state contracted services can be initiated at several different access points. For mental
6
health services, which provide services to a vulnerable population, there are several outlets
for informal grievances. Included are: the provider, the managed care organization,
advocacy council, the managed care ombudsman and various other agencies depending on
the exact nature of the complaint. For Medicaid clients there is also an annual survey
required for customer satisfaction. These multiple avenues for the grievance process
provide for the opportunity of quick informal resolution of the grievance. However,
multiple grievance referral sources create challenges for quantification and may not
provide an accurate portrayal of the exact nature of the grievance. Customer surveys are
amenable to performance measurement, but capture a sample of the population and are
susceptible to response bias. The performance management aspect of the grievance
procedure that is most relevant to the state agency is the presence of timely controls that
can address issues that are addressed in grievances. For issues such as access to services,
the grievance procedure is a symptom of a systemic problem which, when quantified, may
not satisfactorily capture the nature of the grievance to suitably provide the necessary
action to address the systemic flaw. Quantifying the number of grievances provides a data
element for managerial decisions, but it is not solely adequate to make administrative
decisions. The difficulty is determining the sufficient data elements at the system level
required to make useful decisions.
Framework for Performance Measures
Performance measures have a vital role in quality assessment in behavioral health.
Performance measures are included with activities such as audits and accreditation which
provide external accountability and grievance systems. Performance measures accomplish
7
multiple goals measures focused on internal improvement and monitoring external and
internal processes (Edmunds, M. et al.,1997, p.97). Each of the performance measures in
this complex framework interaction to varying degrees with agencies in the
intergovernmental system. Performance measures are tracked differently at these levels
which can cause redundancies in reporting. While this complex framework provides
sensitivity to the reporting needs of the various reporting agencies, the collectors or
monitors of the information normally reside at the state agency. While survey information
can be populated automatically if used from an external data source, performance data
especially the tracking of innovative practices such as evidence-based practices requires
data collection at the provider level. In this framework, the state agency is in the
predominate role of verifying data quality. The ability of the state agency to assure the
quality of the data is limited by the capacity of the state agency to track data. The ability of
the state to track data quality is limited by the capacity of the providers to collect data.
Challenges to performance measurement
There are several challenges to the implementation of performance measures in the
public sector. One of the fundamental challenges is that performance measures can be
used for multiple purposes that may not be accurately reflected in the measure.
Performance measures are used for accountability and performance, but their ability to
meet both of these purposes is difficult. For example, efforts to use performance measures
to increase performance through accountability can have deleterious effects on
performance (Halachmi, 2002). Increases in accountability do not necessarily translate into
8
rises in agency performance. Dedicating resources to become more accountable takes
resources away from performance tasks. Performance measures are used to provide
information at several levels of government for accountability and performance purposes
which is further complicated by their use as measures of accountability and performance.
Several of these performance measures are mandated by the funding agencies. Mandated
performance measures can cause greater challenges than recommended measures
(Halachmi, 2002). Mandated performance measures provide information relevant to the
recipient agency but are not sensitivity to implementation and factors and constraints at
the state level. There are multiple roles for measures in the public sector.
Internal and External Focused Measures
One level of distinction between performance measures is internal focused
measures and measures of external focus. In public health, external measures provide view
of community health which relate to internal performance but involve contextual factors
and areas that cross multiple agency agendas. Community level measures provide
information regarding the contextual environment that state agencies operate. While
external measures track the overall health of the community they are not necessarily
factors of the agency performance or accountability. State agencies can make decisions on
resource allocation based on information obtained from performance measures but if the
performance measures track domains beyond the reach of the agency then there is a
disincentive to incorporate decision-making based on performance management and
instead focus on other administrative or political decision-making strategies. Objective
information regarding broad community metrics does not provide the agency with a
9
distinct policy lever from which to improve performance or accountability. In fact broad
based external measures may serve to disengage in the use of performance measures as an
element for decision-making and instead serve as a political justification for more funding.
Performance measures are also functions of political and contextual variations
present at each governance level. Federal performance measures provide information for
the executive branch and congress while state level performance measures are directed
toward state legislatures. State agencies and the providers that they monitor have to collect
information to meet the reporting requirements for both federal and state legislative
bodies. State administrative agencies have limited resources and bounded rationality
regarding management decisions. In order to meet the demands from these multiple
stakeholders and internal management decisions, choices have to be made on how to
approach implementation monitoring. Political decisions at the state and federal level
impact what a state agency can focus on. Even when there is agreement in the goals
between the state and federal levels, there may be disagreement on what elements of
performance to track. When there are distinctions between the federal and state measures,
the state must make resource allocation decisions based on the relative importance of the
performance measure.
Structural Barriers
There are several structural factors that impact public health performance. Mays et.
al. (2006) found that public health performance had varying rates of impact based on the
size, financial resources, and organizational resources of the local public health system.
When using federal performance measures several contextual factors along multiple
10
governance layers can impact the performance measures. There is also uncertainty
regarding the exact mechanism in which structural factors impact public health outcomes
(Mays et al., 2009). One of the challenges is separating changes in program eligibility and
programs that occur through legislative decisions and those related to agency discretion.
Statutory barriers also impact the ability of an agency to impact performance measures or
may spur the development of performance measures to track legislative compliance. There
are many political and contextual reasons beyond management decision-making that
influence performance measure selection. There are methods to address the contextual
factors associated with agency performance. Adjusted performance measures which use
contextual factors that impact agency performance (Rubenstein, 2003). While adjusted
performance measures are sensitive to contextual factors they are unable to account for the
political influences at each level of the complex implementation cycle
State and Federal goal alignment
One for the challenges presented to state mental health agencies is goal ambiguity at
the federal level. Ho (2007) describes the differences in the definition of performance
between the branches of government with regards to the Substance Abuse and Mental
Health Agency. The executive branch viewed performance through the lens of cost-
effectiveness whereas congress viewed performance as a measure of meeting stakeholder
needs (Ho, 2008, p.329). A variety of definitions of performance is also present at the state
level. There have been numerous efforts to monitor state and federal performance, and
some of these efforts are not implemented with incentives. In a model without incentives,
accountability professionals can serve as the instrument of performance monitoring. The
11
task of the accountability agency is to objectively determine if the implementation of the
performance measure will impact performance or accountability. However, there are
challenges in depending on external accountability professionals to determine agency
performance. One of the challenges is the differences in values between accountability
professionals and agency staff (Power, 1997). From a practical programmatic standpoint,
without a performance linked incentive, there is little reason to assume that a performance
report will result in program improvement. Given the number of mandated performance
measures in which agencies must track, measures without incentives by definition become
less of an agency concern. However, there are instances when the agency actions are
aligned with the federal performance measures. Given the resources required at the state
level to ensure that providers track data and the performance of data integrity on the
collected data it is instances when incentives are used that the state and federal agency are
the most likely to be aligned.
Innovation and performance
One of the uses of performance measures is to track the implementation of a service
innovation. Tracking an innovation has additional challenges due to the fact that innovation
is also moderated by internal agency factors and interagency networks. In their analysis of
factors contributing to the adoption of matrix management at organizations Burns and
Wholey (1993) determined that task diversity, socio-metric location, dissemination of
information, and the cumulative force of adoption influenced the adoption of innovative
management practices. Federal performance measures are able to have a limited impact
on the dissemination of information and the cumulative force of adoption. The limited
12
impact of performance measures on the organizational factors that influence change in
combination with the lack of a performance incentive serves to mitigate the impact of the
performance measure. Additional questions regarding the configuration of the
performance management system are the number of measures and the alignment of
measures at various governance levels. Public accountability takes many forms which
require different administrative actions, one of which is performance monitoring and
management. Dubnick & Frederickson, (2009) note that federal performance measures
represent accountability to the executive branch and there but there is limited evidence on
impact on issues of equity and justice.
Agency Motivation
State mental health agencies receive signals from national reporting systems such as
the Uniform Reporting System regarding the value of the types of performance data.
However, without incentives or administrative support, the measures tracked in the URS
data are of limited use to the state and agency decision-making. In an area such as
evidence-based practices which is an innovative practice, there are difficulties associated
with its measurement. The tracking of the use of innovative measures requires changes to
state administrative systems which maybe legacy systems. Similarly, providers may not
have a method of tracking these new measures. Tracking these measures can require the
agency to invest in supplemental datasets or other methods to obtain the performance
information. These supplemental systems may have limited protections regarding the
validity and reliability of the submitted data. The performance measures may be tracking
13
the ability of an administrative system to track the innovative measures rather than the
diffusion of innovative practices.
Data Quality
Data quality is a concept that is constructed of multiple components and lacks a
standardized definition (Pipino et al., 2002; Mitchell and Brown, 2002). The nature of
performance measures make maintaining data quality a challenge. Radin (2000) highlights
the challenges of finding valid data that meets the needs of the various stakeholders
representing multiple levels of governance. In addition to the federal state interface, federal
block grants also interact with local governments. While information may report up to the
superordinate agency, information does not always flow downward to the subordinate
agency. This lack of integration of data reduces the probability that the data will fulfill the
needs of the subordinate agency. In instances where the subordinate agency is unable to
receive adequate information regarding the measure, the opportunity to improve
performance is diminished. Sensitivity to agency discretion must therefore be applied
across numerous levels of governance which in turn dilutes the impact of the federal need
for information. Without appropriate incentives or resource assistance, the development of
reliable and valid data that meets the needs of all stakeholders represents a challenge.
Coffey et al. (2008) identify several challenges in the collection of mental health and
substance abuse data and identify practical policy solutions to improve data quality. The
most significant barrier identified was improving the interoperability of datasets
(Coffey,2008). One of the most significant challenges specific to mental health and
substance abuse treatment is the issue of confidentiality and the use of Health Information
14
and Portability and Accountability Act (HIPAA) compliant client identifiers (Coffey et al.,
2008). Another potential challenge to data is not having the technological ability to track
innovations. Changes at the professional or federal level are difficult to track in the legacy
systems used by state agencies.
Performance information regarding agency performance can be implemented with
positive results even if the information is not ideal (Coffey, 1999). However, the
information needs to be involved in a continuous improvement process with dual feedback
loops. One of the influencing contextual factors is the influence of goal ambiguity on
performance measurement. In Chun and Rainey’s (2005) descriptive study of goal
ambiguity at federal agencies, the author’s highlighted the need for efforts to increase goal
clarity to account for agency differences (p.24). Hall and Handley (2011) used a national
survey of city government officials and found mixed results regarding the implementation
of federal performance measures at the local level. Performance measures are likely to
provide incomplete but sufficient information regarding the specified domain of use. The
challenge resides in using the data for multiple purposes beyond the initially intended use.
Given resource limitations, flexibility in the use of performance measures provides
agencies with the ability to meet the needs of multiple stakeholders.
Impact on State
Challenge of Multiple Levels Performance Measures
15
The performance data collected for multiple agencies creates difficulty in making
management decisions at the state level. While there are tertiary benefits in stakeholder
accountability, it serves to provide a complex message to agencies trying to initiate change
especially when the agency has multiple stakeholders at the federal, state, and local levels.
In the case where there are related policies it becomes difficult for an agency to prioritize
and measurement decisions may be the result of responding to the agent that is most
actively engaging the agency. There are other reporting mechanics that may influence the
reporting of the agency such as the presence of an incentive, the frequency of reporting,
and intensity of reporting serve to impact the quality of the performance measure. Courty
and Marschke (2007) use the federal job training program to explain the necessary
feedback loop between performance measure development and agency reporting. The
process requires a continual process improvement for performance measures which
incorporates the responses to the performance measures. In addition to these
measurements of reporting intensity there are also factors related to the type of report.
One possible sustainable solution to the multiple levels of performance measures is the use
of performance information capture to meet the accountability requirements of one level of
the intergovernmental system at separate intergovernmental level.
Challenge of Missing Data in Performance Measures
The results of this study highlight missing data as one of the major barriers to the
states using federal performance measures to benchmark performance with other states.
Federal performance measures serve as a measure on the use of federal expenditures.
However, performance measures serve as an important resource for the development of
16
performance management systems and benchmarking at the state level. In order to be
useful to the states, performance measures need to provide accurate and reliable data from
which states can use the data for national comparisons. For most of the population served,
the state agency is removed from the process of service provision and while it has access to
the Medicaid billing data, information regarding key management decisions resides at the
level of program management. There is a delicate balance between the amounts of data and
the state authority requires meeting contract requirements and the amount of extra
information regarding performance. Performance measures can be integrated into the
contract but the inclusion of measures requires additional dedication of resources and
political negotiation.
Data
The Substance Abuse and Mental Health Services Administration developed the National
Outcome Measures and developed a national collection of outcome measures developed
along ten domains. The ten domains are associated with related outcomes and measures
that represent mental health, substance abuse. Substance abuse measures are separated
into treatment and prevention categories. The broad coverage of the domains across the
behavioral health spectrum creates a situation where the measures are not applicable
across all practice domains. Using information from this national database for FY 2007 and
FY 2009, the impact of national efforts on evidence-based practices will be analyzed. The
goal is to identify the interaction between Oregon’s efforts to monitor and increase the
amount of evidence-based practices at the state and county-level with the reporting on
national level outcome measures.
17
Reviewing the performance of 50 states using 2007 and 2009 national outcome
measures for adult and children’s evidence-based practice implementation provides
information on the reporting levels of various state mental health systems. The results of
this analysis identify areas that highlight incongruences in information uptake at the state
and federal level. There is a demonstrated lack of reporting of practices at the state-level.
There are ten evidence-based practice measures contained in the National Outcome
Measures (NOMs). Seven of these measures are for the adult population, and three are for
the children’s population. The evidence-based practices were identified through a
professional research consensus panel convened by Robert Woods Johnson Foundation in
1998 (SAMSHA, 2008). The practices identified by the panel included six practices: Illness
Management and Recovery, Supported Employment, Family Psycho-education, Assertive
Community Treatment, Integrated Treatment for Co-Occurring Disorders, and Medication
Management are represented in the NOM measures (SAMSHA, 2008). In addition to
tracking these measures, SAMSHA provides free implementation information for each of
the practices. These practices have been shown to be impacted by contextual factors that
limit their implementation such as the financing and regulation of individual practices as
well as leadership and training and quality issues (Isett et al., 2007). There are also
variations in the implementations of individual evidence-based practices. Given the level of
variability in the system, it is difficult to standardize implementation across state agencies.
This variation is reflected in the reported URS data. The results of the analysis is the level of
variability in the use of innovative practices and serves as a signal to states that results
based on the implementation of all tracked evidence-based practices might be muted at
best. The lack of reliable reporting across states provides states with general information
18
regarding the tracking of evidence-based practices for a state performance management
system. The information is based on the level of report and not the levels reported in the
data. Stated differently, the states gain information on the capacity of other states to report
and the level in which this innovation needs to be a priority. Lacking federal incentive, the
states learn the baseline for report and can make administrative decisions based on this
information. In the case of Oregon, the state mandate provided a mechanism in which to
focus on evidence-based practices from a more general approach and not focus on the six
evidence-based practices required for the Uniform Reporting System.
The amount of data missing in the Uniform Reporting System across states for the
ten identified evidence-based practices is significant and limits the analysis available to
state decision-making systems. In 2007 and 2009 the State of Oregon reported on half of
the ten reported evidence-based practices. While the challenges of missing data preclude
any complicated analysis they do provide information to state performance management
systems regarding the level of tracking and implementation of selected evidence-based
practices across state mental health agencies. The agency can make administrative
decisions regarding the amount of investment in the implementation of national evidence-
based practices and instead invest resources into the state mandate to track the amount of
funding on evidence-based practices. Given that state mental health agencies operate in a
resource limited environment and are subject to goals from different levels of the
intergovernmental system, this information may be important in making decisions on
resource allocation.
Frequency Distributions
19
The frequency distributions for the reported evidence-based practices for the years
2007 and 2009 were analyzed using STATA 11.0. Supported Employment and Assertive
Community Treatment had 27 instances of missing data; Supported Housing displayed 32
instances out of 102 observations of missing data for the years 2007 and 2009. Family
psycho-education and medication management had 75 instances of missing data; Illness
management had 69 instances of missing data. Dual Diagnosis treatment experienced 56
instances and Therapeutic Foster Care experienced 52 instances of missing data. Multi-
Systemic Therapy experienced 69 instances of missing data. Functional Family care
experienced 79 instances of missing data.
The goal of these measures was to increase the use of these evidence-based
practices so it is assumed that the practices would have low initial representation that rises
over time. The challenge encountered was the extent of missing data across all measures
and states for 2007 and 2009. Louisiana, Tennessee, and Washington were the only states
to report on all of the evidence-based practice measures which occurred in 2009. The state
of Wyoming did not report on any of the performance measures for 2007 and 2009. Alaska,
California, Louisiana, and Mississippi did not report on any of the evidence-based practice
measures in 2007. The level of missing data reflects a general lack of reporting, tracking,
and implementation of evidence-based practices across states over the 2007 to 2009 time
period.
SAMSHA has addressed issue in data quality experienced in the URS data. In 2007,
SAMHSA provided up to $142,200 per year in total costs to states for the purpose of
improving data quality (SAMSHA, 2007). One of the possible improvement opportunities
20
identified was the use of web based mechanisms to increase reporting (SAMSHA, 2007).
There are many challenges to the use of the Uniform Reporting System database as a
function of performance management. One of the biggest challenges is the issue of non-
report on measures. Given that non-response is more highly attributed to a lack of report
missing values can be categorized as Non-Ignorable (NI) error. Non-ignorable missing data
can be modeled but require significant information regarding the parameters of the
missing data (Luke, et al., 2002). Lacking information on the data parameters, the database
is of limited use. The importance of the National Outcome Measures is the ability to track
progress on particular program aspect with other states. The inability to track across states
limits the use of the national outcome measure system.
Oregon Specific Implementation
Descriptive data for the adoption of evidence-based practices for the years 2007 and
2009 for the state of Oregon reveal trends in reporting of evidence-based practices.
Medications Management is the only practice with a reported value over 1%. Medications
Management grew from 27.94% in 2007 to 34.3% in 2009. Several of the practices had no
report for any of the three years. There was a considerable amount of indication of a lack
of report of evidence-based practices. Of total 1,530 observations over a three year period,
825 (54%) of the reports had no reported data. The highest level of report was for
Functional Family Therapy with 118 out of 153 observations. The second most non-
reported evidence-based practice was Illness Self-Management with 111.
21
Discussion
While the National Outcome Measures provided information regarding state
progress in the tracking of the six national evidence-based practices, they do not capture
state efforts to increase the use of evidence-based practices. The six national evidence-
based practices were derived through an expert consensus practice in 1998, and they are
not sensitive to improvements in determining evidence-based practices overtime. There
are some inconsistencies at the federal agency level. SAMSHA maintains the National
Registry of Evidence-Based Programs and Practices (NREPP) which rates the effectiveness
of evidence-base practices along multiple domains. The NREPP effort is similar to the State
of Oregon process in which evidence-based practices are submitted by providers and
assessed based on the available research. Lacking a consensus on the specific practices
employed at the state levels there is the need to seek additional information. This process
allows the agency to gain information on the current practices and monitor the influx of
new practices.
As of July 27, 2011, the NREPP has tracked over 200 practices (SAMSHA, 2011).
Where the National Outcome Measures provide a specific focus on particular practices,
they are not sensitive to the overall proliferation of evidence-based practices. A state may
increase performance on the narrowly defined evidence-based practice measures but have
limited impact overall on the mental health service delivery process. The greatest challenge
to the tracking of evidence-based practices at the national level is the lack of data capacity
related to tracking the use of the particular evidence-based practice.
22
Variations in Implementation
The differences in the reporting at the federal and state level provide different
pictures of the same policy implementation. The federal performance monitoring focuses
on the amount of the population receiving a particular service, the Oregon evidence-based
practice implementation tracked the agency level of adoption. There are advantages and
disadvantages to both approaches. The Oregon implementation model highlights local
contextual factors that impact the interpretation of criteria in performance monitoring. At
the local level, evidence-based practices were interpreted widely and developed through a
consensus process. At the national level, a narrowly defined standardized criterion was
adopted. The goal of the SAMHSA URS is to provide accountability to congress on mental
health spending. The impetus for the Oregon implementation was the state-legislature
which focused its concern on the amount of state funds purchasing evidence-based
practices. In both cases, the respective legislature guided the performance measurement
system. One of the possible advantages available to the state agency is to use both sets of
information to develop a performance management system that monitors both state and
federal requirements.
Potential Improvements
One promising development is the use of electronic health records which can
increase the transmission of data from the provider to the monitoring agency
automatically. Additional protections such as incentives for performance on measures are
an alternative to increase data reliability and report. While at the federal level it is
23
advantageous to have information on performance on a wide variety of measures, at the
agency level there is limited incentive to provide reliable data when there are competing
demands for data report.
While the state of Oregon did not track all of the evidence-based practices for the
URS, the state developed a separate process for tracking the implementation of all
evidence-based practices. In 2005, Oregon had identified 80 evidence-based practices. In
2008, Oregon tracked 110 evidence-based practices. In 2005, the most reported evidence-
based practice was Motivational Interviewing with a 75 counties identifying that the
practice was purchased. The Oregon survey used a different unit of analysis compared to
the SAMSHA URS report. The Oregon survey focused on county purchasing of an evidence-
based practice where the SAMSHA URS database tracks the percent of individuals that
received a particular service.
National Performance Management Guidelines
The National Performance Management Advisory Commission provides a
performance management framework for state and local government. The focus of this
framework is to provide general guidelines for the development of public performance
management systems. The framework focuses on a continual cycle of development which
involves stakeholders and standard principles in the development of performance
management systems that address accountability and the capacity for learning and
improvement (National Performance Management Advisory Commission, p.1). This
framework involves the normative performance management theories. These management
theories focus on the use of information in decision-making but remain unclear on the
24
practicality of monitoring and tracking data with reduced resources and third-party
governance structures. In instances where federal performance measures do not overlap
with state measures, states may make the rational decision and invest resources with the
stakeholder of greatest influence. Performance measures that are not a core element of the
mission have less chance of completion. Accountability professionals play an important
role in assisting agencies in developing performance measure systems that interact with
federal systems and provide useful baseline information that improves the functioning of
the agency. The decision to delegate resources to collection of performance measures is a
management decision which accounts for the transaction cost of the monitoring effort.
Accountability professionals can provide guidance in developing performance management
systems at the state agency level that allow state agencies to utilize existing data collection
systems and integrate these mechanisms to increase the information available for
managerial decisions. Performance data that serves the function of reporting to the
congress is not necessarily effective to the state agency. The most important level of
information in which program decisions can be made rests at the interface between the
provider and the state agency. State comparisons are effective for the state agency to make
decisions about provider practices but require sufficiently valid and reliable data.
25
References
Behn, R. D. (2003). Why Measure Performance? Different Purposes Require
Different Measures. Public Administration Review, 63(5), 586-606.
Bird, S. M., Sir David, C., Farewell, V. T., Harvey, G., Tim, H., & Peter C., S. (2005).
Performance indicators: good, bad, and ugly. Journal of the Royal Statistical Society: Series A
(Statistics in Society), 168(1), 1-27.
Brolin, M., Seaver, C., & Nalty, D. Performance Management: Improving State Systems
through Information-based Decision making. DHHS Publication No. 05-3983. Rockville, MD:
Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services
Administration, 2004.
Chun, Y. H., & Rainey, H. G. (2005). Goal Ambiguity in U.S. Federal Agencies. Journal
of Public Administration Research and Theory, 15(1), 1 -30.
Coffey, R. M. (1999). Case Mix Information in the United States: Fifteen Years of
Management and Clinical Experience. CaseMix Quarterly, 2(3), 7-16.
Coffey, R. M., Buck, J. A., Kassed, C. A., Dilonardo, J., Forhan, C., Marder, W. D., &
Vandivort-Warren, R. (2008). Transforming Mental Health and Substance Abuse Data
Systems in the United States. Psychiatric Services, 59(11), 1257-1263.
Courty, P., & Marschke, G. (2007). Making Government Accountable: Lessons from a
Federal Job Training Program. Public Administration Review, 67(5),
26
Derthick, M. (1970). The influence of federal grants: public assistance in
Massachusetts. Harvard University Press.
Dubnick, M. J., & Frederickson, H. G. (2009). Accountable Agents: Federal
Performance Measurement and Third-Party Government. Journal of Public Administration
Research and Theory, 20(Supplement 1),
Edmunds, M., Frank, R., Hogan, M., McCarty, D., Robinson-Beale, R., and Weisner, C.
(Eds); Committee on Quality Assurance and Accreditation Guidelines for Managed
Behavioral Health Care, Institute of Medicine. (1997). Managing Managed Care: Quality
Improvement in Behavioral Health. Washington, D.C.: The National Academies Press.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of Innovations
in Service Organizations: Systematic Review and Recommendations. The Milbank Quarterly
2004, 82:581-629.
Ganju V: Mental Health Quality and Accountability: The Role of Evidence-Based
Practices and Performance Measurement. Administration and Policy in Mental Health and
Mental Health Services Research 2006, 33:659-665.
Gunsteren, H. R. van. (1976). The quest for control: a critique of the rational-central-
rule approach in public affairs. London: Wiley.
Halachmi, A. (2002). Performance measurement: a look at some possible
dysfunctions. Work Study, 51(5), 230-239.
Halachmi, A. (2005). Performance measurement: test the water before you dive in.
International Review of Administrative Sciences, 71(2), 255 -266.
27
Hall, J., & Handley, D. (2011). City Adoption of Federal Performance Measurement
Requirements. Public Performance & Management Review, 34(4), 443-466.
Ho, A. (2007). The Governance Challenges of the Government Performance and
Results Act A Case Study of the Substance Abuse and Mental Health Administration. Public
Performance & Management Review, 30(3), 369-397.
Ingraham, P. W., Joyce, P. G., & Donahue, A. K. (2003). Government performance: why
management matters. Baltimore: The Johns Hopkins University Press.
Isett, K. R., Burnam, M. A., Coleman-Beattie, B., Hyde, P. S., Morrissey, J. P.,
Magnabosco, J., Rapp, C. A., et al. (2007). The state policy context of implementation issues
for evidence-based practices in mental health. Psychiatric Services (Washington, D.C.), 58(7),
914-921.
Jacobsen, K., & McGuire, T. G. (1996). Federal Block Grants and State Spending: The
Alcohol, Drug Abuse, and Mental Health Block Grant and State Agency Behavior. Journal of
Health Politics, Policy and Law, 21(4), 751-768.
Julnes, P. D. L., & Holzer, M. (2001). Promoting the utilization of performance
measures in public organizations: An empirical study of factors affecting adoption and
implementation. Public Administration Review, 61(6), 693-708.
Light, P. C. (1997). The tides of reform: making government work, 1945-1995. Yale
University Press.
28
Lilford, R., Mohammed, M. A., Spiegelhalter, D., & Thomson, R. (2004). Use and
misuse of process and outcome data in managing performance of acute medical care:
avoiding institutional stigma. The Lancet, 363(9415), 1147-1154.
Luke, D. A., Marsh, L., Smithson, M., Allison, P. D., Chen, P. Y., Best, S. J., Rudas, T., et al.
(2001). Missing data. SAGE.
Mitchell, Sandra and Brown, Ann. Performance Measurement Data: Is It Fit For Use?
Data Quality Initiatives At CIHI. Proceedings of Performance Measurement in Healthcare:
Responding to the Need for Greater Accountability. Toronto, Ontario, Canada. May 15-16,
2002.
Manderscheid, R. (2006). Promoting Self-Assessment and Accountability:
Introduction to Quality Improvement Measures In Center for Mental Health Services. Mental
Health, United States, 2004. Manderscheid, R.W., and Berry, J.T., eds. DHHS Pub no. (SMA)-
06-4195. Rockville, MD: Substance Abuse and Mental Health Services Administration.
Mays GP, McHugh MC, Shim K, Perry N, Lenaway D, Halverson PK, Moonesinghe
R.(2006). Institutional and economic determinants of public health system performance.
American Journal of Public Health. Mar;96(3):523-31.
Mays GP, Smith SA, Ingram RC, Racster LJ, Lamberth CD, Lovely ES (2009). Public
health delivery systems: evidence, uncertainty, and emerging research needs. American
Journal of Preventative Medicine. Mar;36(3):256-65.
29
National Association of State Mental Health Program Directors Research Institute,
Inc. (2005). Results of a survey of state directors of adult and child mental health services on
implementation of evidence-based practices. Alexandria, VA.
National Association of State Mental Health Program Directors Research Institute,
Inc. (2009). User Guide: Compiling and Reporting Client Level Data For The Mental Health
NOMS (For use by pilot States),Version 4.0, July 2009. Last Accessed on 8/09/2011 at
http://www.nri-inc.org/projects/SDICC/forms/Client_Level_Data_Users_Guide_1-14-
10.pdf .
National Association of State Mental Health Program Directors Research Institute,
Inc. (2011). State Data Infrastructure Coordinating Center (SDICC). Retrieved August 5,
2011, from http://www.nri-inc.org/projects/SDICC/
National Performance Management Advisory Commission (2010). A Performance
Management Framework for State and Local Government: From Measurement and Reporting
to Management and Improving. Last Accessed on 7/27/2011 at
http://www.pmcommission.org/
New Freedom Commission on Mental Health. (2003). Achieving the promise:
transforming mental health care in America. Final report. (DHHS Pub. No. SMA-03-3832).
Rockville, MD.
Oregon Revised Statute. 182.525 Available at:
http://www.leg.state.or.us/ors/182.html
30
Perrin, E.B.,Koshel, J.J. Editors; Panel on Performance Measures and Data for Public
Health Performance Partnership Grants, National Research Council. (1997). Assessment of
Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington,
D.C.: The National Academies Press.
Pidd, M. (2007). Perversity in public service performance measurement. Business
Performance Measurement (Second edition). Cambridge University Press.
Pipino,L. L., Lee,Y., W., and Wang, R. Y. 2002. Data quality assessment. Commun. ACM
45, 4 (April 2002), 211-218.
Posner, P. L. (2002). Accountability challenges of third-party government. In The
tools of government: A guide to the new governance, ed. L. M. Salamon, 523–51. New York,
NY: Oxford Univ. Press.
Power, M. (1997). The Audit Society: Rituals of Verification. Oxford: Oxford
University Press.
Radin, B. A. (2000a). Intergovernmental Relationships and the Federal Performance
Movement. Publius, 30(1), 143-158.
Radin, B.A. (2000b). The Government Performance and Results Act and the tradition
of federal management reform: Square pegs in round holes? Journal of Public
Administration Research and Theory, 10(1), 111–135.
Rieckmann, T. R., Kovas, A. E., Fussell, H. E., & Stettler, N. M. (2008). Implementation
of Evidence-Based Practices for Treatment of Alcohol and Drug Disorders: The Role of the
State Authority. The Journal of Behavioral Health Services & Research, 36(4), 407-419
31
Rubenstein, R., Schwarfz, A. E., & Stiefel, L. (2003). Better than Raw: A Guide to
Measuring Organizational Performance with Adjusted Performance Measures. Public
Administration Review, 63(5), 607-615.
Substance Abuse and Mental Health Services Administration (2008). Assertive
Community Treatment: How to Use the Evidence-Based Practice KITs. DHHS Pub. No. SMA-
08-4344, Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental
Health Services Administration, U.S. Department of Health and Human Services
Substance Abuse and Mental Health Services Administration. (2011a). SAMHSA
Block Grants. Last Accessed on August 5, 2011, from
http://www.samhsa.gov/grants/blockgrant/
Substance Abuse and Mental Health Services Administration. (2011). NREPP. Last
Accessed on August 5, 2011, from http://nrepp.samhsa.gov/
Substance Abuse and Mental Health Services Administration. (2011b). Leading
Change: A Plan for SAMHSA’s Roles and Actions 2011-2014. HHS Publication No. (SMA) 11-
4629. Rockville, MD: Substance Abuse and Mental Health Services Administration.
SAMHSA 2009 Uniform Reporting System (URS) Output Tables. (n.d.). . Retrieved
August 5, 2011, from http://www.samhsa.gov/dataoutcomes/urs/urs2009.aspx.
Substance Abuse and Mental Health Services Administration. (2007). $21 Million
Awarded to Help States Enhance Mental Health Data Recording and Reporting Systems. Press
Release 9/26/2007. Last accessed on 8/6/2011.
http://www.samhsa.gov/newsroom/advisories/0709265213.aspx
32
U.S. Department of Health and Human Services. (1999). Mental Health: A Report of
the General—Executive Summary. Rockville, MD: U.S. Department of Health and Human
Services, Substance Abuse and Mental Health Services Administration, Center for Mental
Health Services, National Institutes of Health, National Institute of Mental Health.
United States. General Accounting Office. GAO/HEHS-95-74 Block Grants Health,
Education, and. Human Services Division. B-260302. February 9, 1995
U.S. General Accounting Office, Managing for Results: Agency Progress in Linking
Performance Plans with Budgets and Financial Statements (GAO-02–236) (Washington, DC:
U.S. Government Printing Office, 2002 (ADD)
Westphal, J.D., R. Gulati, and S.M. Shortell. 1997. Customization or Conformity? An
Institutional and Network Perspective on the Content and Consequences of Total Quality
Management Adoption. Administrative Sciences Quarterly 42(366):394.
Wildavsky, A. (1974). The Politics of the Budgetary Process. Boston; Little Brown
and Company. Behn, R. D. (2003). Why Measure Performance? Different Purposes Require
Different Measures. Public Administration Review, 63(5), 586-606.
Wright, D. S. (1988). Understanding Intergovernmental Relations (3rd ed.). Pacific
Grove,CA: Brooks Cole.