Measuring What Counts in HIS - Balanced Scorecards
-
Upload
sudhendu-bali -
Category
Documents
-
view
685 -
download
2
description
Transcript of Measuring What Counts in HIS - Balanced Scorecards
international journal of health planning and management
Int J Health Plann Mgmt 2010; 25: 74–90.
Published online in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/hpm.1004
Designing a balanced scorecard for a tertiarycare hospital in Pakistan: a modified Delphigroup exercise
Fauziah Rabbani1,2*, Syed M. Wasim Jafri3y,z, Farhat Abbas4x,�,Mairaj Shah5k,#, Syed Iqbal Azam1yy, Babar Tasneem Shaikh1yy,Mats Brommels6,7zz,xx,�� and Goran Tomson2,7kk,##
1Department of Community Health Sciences, Aga Khan University, Karachi, Pakistan2Department of Public Health Sciences, IHCAR Div International Health, KarolinskaInstitutet, Stockholm, Sweden3Department of Medicine and Department of Continuing Professional Education , Aga KhanUniversity, Karachi, Pakistan4Department of Surgery, Aga Khan University, Karachi, Pakistan5Aga Khan University Hospital, Karachi, Pakistan6Department of Public Health, University of Helsinki, Finland7Medical Management Centre at Karolinska Institutet, Stockholm, Sweden
SUMMARY
Balanced Scorecards (BSC) are being implemented in high income health settings linkingorganizational strategies with performance data. At this private university hospital in Pakistanan elaborate information system exists. This study aimed to make best use of available data forbetter performance management. Applying the modified Delphi technique an expert panel ofclinicians and hospital managers reduced a long list of indicators to a manageable size.Indicators from existing documents were evaluated for their importance, scientific soundness,appropriateness to hospital’s strategic plan, feasibility and modifiability. Panel membersindividually rated each indicator on a scale of 1–9 for the above criteria. Median scores wereassigned. Of an initial set of 50 indicators, 20 were finally selected to be assigned to the four
*Correspondence to: Dr F. Rabbani, Professor Dept of Community Health Sciences, PO BOX 3500,Stadium Road, Karachi, Pakistan and doctoral student Dept of Public Health Sciences IHCAR DivInternational Health, Nobels vag 9, Karolinska Institutet, SE 171 77 Stockholm, Sweden.E-mails: [email protected]; [email protected] of Medicine.zAssociate Dean.xProfessor in the Section of Urology.�The COO (Chief Operating Officer—on site).kManager Clinical Affairs.#CME.yyAssistant Professor.zzProfessor of Health Services Management.xxGuest Professor.��Director.kkProfessor.##Director.
Copyright # 2010 John Wiley & Sons, Ltd.
DESIGNING BSC USING MODIFIED DELPHI 75
BSC quadrants. These were financial (n¼ 4), customer or patient (n¼ 4), internal business orquality of care (n¼ 7) and innovation/learning or employee perspectives (n¼ 5). A need forstringent definitions, international benchmarking and standardized measurement methods wasidentified. BSC compels individual clinicians and managers to jointly work towards improvingperformance. This scorecard is now ready to be implemented by this hospital as a performancemanagement tool for monitoring indicators, addressing measurement issues and enablingcomparisons with hospitals in other settings Copyright # 2010 John Wiley & Sons, Ltd.
key words: balanced scorecard; performance management; indicators; modified Delphi;
private hospital in Pakistan
INTRODUCTION
Hospital performance assessment is becoming increasingly important for different
stakeholders such as health care providers, decision makers, and purchasers of
health care. This is in response to growing demands to ensure transparency,
control, and reduce variations in clinical practice (Groene et al., 2008). With
hospitals consuming more than half of overall health care budget (McKee et al.,
2002), recent hospital reforms are highlighting a quest for achieving more efficient
and effective hospital care. This can be achieved through generalizable,
standardized interpretable, and useable information for clinicians or health service
managers (Willis et al., 2008).
Hospitalmanagement teams receivevoluminous information fromawidevariety of
sources. Despite thewidespread use of performance indicators, there is little research
evidence on how to select the essential data to make evidence-informed decision
making (Ovretveit and Al Serouri, 2006). The worldwide health community,
therefore, needs to focus on improving measurement of a small set of priority areas
(Murray, 2007). Formal consensus methods are a set of techniques that synthesize
expert or stake holder opinion to guide and prioritize group decisions in situations
where information is lacking, contradictory or where there is an overload of
information (Campbell et al., 2002). Threemainmethods have been used in the health
field: the Delphi, the Nominal Group Technique and the Consensus Development
Conference. The comparative advantage of the Delphi technique over other strategies
is the enhanced opportunity for all participants to contribute greater number of ideas
than other group processes, minimizing domination of the process by more confident
or outspoken individuals, the ease of interpreting the results (ideas aregenerated, voted
on/ranked, aggregated, and evaluated at the session itself), a greater sense of
accomplishment formembers (results are available immediately after the session), and
minimal resource requirements with efficient use of time (Murphy et al., 1998).
Conceptualization of hospital functioning is a diverse and complex phenomenon.
WHO strategic orientations are encompassed into six interrelated dimensions:
clinical effectiveness, safety, patient centeredness, responsive governance, staff
orientation, and efficiency (Veillard et al., 2005). Though no performance
management tool is ideal, this multidimensional approach of hospital performance
is captured in the balanced scorecard (BSC), in four different perspectives with an
equal weightage: (i) labeled learning and growth (staff orientation and satisfaction),
(ii) internal processes (clinical outcomes and management of health services),
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
76 F. RABBANI ET AL.
(iii) customer (patient) satisfaction, and (iv) financial efficiency/performance
(Castaneda-Mendez et al., 1998). BSC serves as a dashboard for meaningful decision
making and quality improvement and relates results to external references while
promoting internal comparisons overtime (Veillard et al., 2005).
The advantage that BSC has over other performance measurement tools is that it is
less of a diagnostic control system for highlighting abnormal activities and more of
an interactive system for providing signals to the organization about management
objectives, stimulating debate, improving quality, and achieving organizational learn-
ing (Gordon and Geiger, 1999). Through its use various healthcare organizations in
high income countries (HICs) improved in their recruitment and retention processes
and employees gained a better understanding of organizational strategies leading to
overall improvement in performance including reduction in costs, better clinical
outcomes, increased staff, and patient satisfaction (Curtright et al., 2000; Kaplan and
Norton, 2000; Hospital Report, 2003; Mannion et al., 2005).
The implementation of management models is considered a step towards
maturity and a change discourse aiming for an efficient and modern organization
(Schalm, 2008). In the context of low income countries (LICs), however, we know
little about successful models to promote greater management effectiveness at the
hospital level (Hartwig et al., 2008). Evidence about BSC usage in LICs is
deficient mainly due to lack of committed leadership, cultural readiness, quality
information systems, viable strategic plans, and optimum resources (Rabbani et al.,
2007). Simple dissemination of written guidelines in LICs is proving to be often
ineffective (Rowe et al., 2005) and health managers face significant challenges in
developing and managing appropriate systems (Green and Collins, 2003). Such
faulty information systems result in a clear lack of knowledge regarding where to
focus priorities; where improvement is needed and whether ongoing initiatives
were having a positive impact (Murray, 2007; Malqvist et al., 2008). Although a
partnership-mentoring model for enhancing management capacity in Ethiopian
hospitals has been tested (Hartwig et al., 2008), to our knowledge BSC specifically
has not been implemented in hospital settings in LICs. Recently BSC was applied
at a national (macro) level to demonstrate how provinces and the country are doing
in delivering the basic package of health services in Afghanistan (Peters et al.,
2007). This innovative adaptation of the BSC in Afghanistan at a macro level has
provided a useful tool to summarize the multidimensional nature of health services
and enabled managers to benchmark performance and identify strengths and
weaknesses in the Afghan context.
There is heavy emphasis on curative services and hospitals consume 45% of the
meager health budget in Pakistan (Abrejo et al., 2008). Despite this the quality of
health care in public hospitals is dismal and 70% health care is being provided by
private facilities (Ghaffar et al., 2000). We conducted this study at a private
university hospital in Karachi Pakistan to assess the feasibility of the modified Delphi
group technique to reach consensus about the indicators of an institutional level BSC,
identify the strengths, and weaknesses of data being generated and recommend ways
to improve hospital performance measurement.
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
DESIGNING BSC USING MODIFIED DELPHI 77
METHODS
This study was conducted at a large private university hospital in Karachi (largest and
most populous city of Pakistan) in 2006. The hospital offers quality care to
outpatients and inpatients of all socio-economic classes (Rafique et al., 2006). It
operates with 542 beds in operation and offers a broad range of secondary and
tertiary services to over 38 000 hospitalized patients and to over 500 000 outpatients
annually. Inpatients have an average length of stay of 3.9 days (AKUH Quality
Manual, 2007: internal document). There are currently 400 trainees (interns,
residents, and fellows) affiliated with the hospital. Clinical services offered by this
university hospital (with staffing details) are listed in Table 1. According to the
hospital’s strategic plan (Health Sciences Centre Committee, 2002; internal
document), the vision is to provide (i) compassionate, accessible, and good quality
care that meets or exceeds expectations (ii) provision of a work environment that
fosters committed and motivated staff, and (iii) enabling leadership in research and
education that improves national health.
This hospital has an extensive health information system in place. An internal
situation analysis, however, identified the need for better integration of information
collected for evidence informeddecisionmaking (Health SciencesCentreCommittee,
2002: internal document). This report recommended that academicians and
administrators develop a road map together and foster a culture of team work,
shared vision, and institutional ownership. BSC serves as a road map for self-
assessment and continuous improvement towards excellence (Ruiz et al., 1999).
Table 1. University hospital–clinical services
Anesthesia �(28) Surgery (52)� Cardiothoracic surgery
Family medicine (12) � Dental-oral and maxillofacial surgery� General surgery
Medicine (54) � Neurosurgery� Cardiology � Ophthalmology� Diabetes, Endocrinology, and Metabolism � Orthopedic surgery� Gastroenterology � Otolaryngology� General internal medicine � Pediatric Surgery� Hematology and Oncology � Urology� Neurology� Pulmonary and Critical care medicine Ambulatory care services� Emergency medicine
Allied health servicesObstetrics and Gynecology (15) � Pharmacy
� PhysiotherapyPediatrics (23) � NutritionPathology and Microbiology (33) Diagnostic services
� CardiopulmonaryPsychiatry (7) � Clinical laboratories
� Neurophysiology� Radiology (21)
�Number in parenthesis represents total full time faculty. Non-faculty employees are not listed.
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
78 F. RABBANI ET AL.
Therefore, in 2006 a multidisciplinary team comprising of hospital leadership
(including DG and CEO of the hospital, Medical Director, Chief Operating Officer)
agreed that the hospital needs to produce aBSC incorporating clinical and non-clinical
metrics for better clinical outcomes andperformancemanagement. In 2008anewVice
President (VP) was appointed for health services with the past experience of working
as an Executive Director at Guy’s and St. Thomas NHS Foundation Trust in London.
The newly appointed VP was responsible for corporate and clinical governance,
clinical operations and organization-wide performance measurement and manage-
ment. Under the leadership of VP, BSC was envisaged as an organizational
performance management pyramid (Figure 1) empowering all levels (from executive
to operational) with varying metrics and details. It would serve to link the hospital’s
strategic plan and individual department objectives. The frontline level was to look at
details with a large set of indicators tracked on a monthly/quarterly basis and
concerned with problem solving and improvements whereas the Board and executive
management would be more aligned towards long-term global trends, summary
reports generated biannually and concerned with overall strategy and governance.
Following an assessment for cultural readiness to implement the BSC (Rabbani
et al., 2008), a systematic development plan was used to design the BSC at this
hospital. The cultural assessment showed that the required prerequisites for BSC
implementation particularly conducive leadership, viable strategic plan, and a
functional management information system already existed at this hospital. The steps
used to design the BSC were in line with those outlined in earlier studies (Kaplan and
Figure 1. Proposed approach to develop a balanced scorecard
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
DESIGNING BSC USING MODIFIED DELPHI 79
Norton, 1996; Wachtel et al., 1999; Oliveira, 2001; Peters et al., 2007). These are;
(i) building the business case at the executive leadership level of the hospital (Vice
President hospital services, Chief Operating Officer and Medical Director of the
hospital were involved in this study) (ii) identifying strategies and tactical objectives,
(iii) identifying performance measurements, (iv) identifying data sources, (v),
building consensus around the indicators to create the BSC based on the background
material obtained from internal documents (vi) develop communication tools and
targets for each measure based on benchmarking (vii) implementation, refinement,
evaluation, and reusing the BSC.
After working through initial steps this study focused on the creation of BSC for
use at the Medical Directorate level (Figure 1). A subsequent study (in progress) will
report on the adaptation and implementation of BSC at the frontline departmental
level. It is anticipated that results of the latter would be available in 2010 after which
the BSC would be cascaded upward to the CEO and Board level.
In order to select indicators for BSC, a multistage modified Delphi consensus
process developed by RAND (Marshall et al., 2006) was used. We used the modified
Delphi technique so that face-to-face panel discussions with experts in the field could
be conducted and face validity of indicators established. The face validity of the
indicators was defined as whether its meaning and relevance to the assessment under
consideration was self-evident and it superficially appeared to measure what it was
supposed tomeasure (McBurney, 2001).Apanel of nine expertswas selected based on
guidelines of the Delphi technique (Campbell et al., 2002). The group of experts was
identified from a variety of professional disciplines and the required range of
professional backgrounds. The panel represented hospital domains of marketing
(managers who conduct quarterly patient satisfaction surveys), clinical quality
assurance (clinicians, physician, and nurse managers who monitor quality care
indicators), human resource management (staff and managers who conduct annual
staff satisfaction surveys), and budget and planning (financial managers furnishing
financial reports). As per recommendation of other studies (Chung et al., 2008) it was
ensured that experts committed time and involvement until the process was complete.
Ethical approval for the study was received from the Ethical Review Committee of
the Pakistani hospital where this study was implemented.
RESULTS
Short listing of indicators by the expert panel
Following an extensive review of existing internal documents (periodical quality
assurance, patient and employee satisfaction surveys, and financial reports), a
preliminary list of 50 indicators was formulated in line with hospital’s strategic plan.
No indicators were removed from consideration at this phase of the activity. The next
step was to prioritize key performance indicators based on the criteria of importance,
scientific soundness (credibility), appropriateness to hospital’s strategic plan,
feasibility (i.e., whether the measure was available easily as part of management
information system, could be collected accurately, reliably and at a reasonable cost)
and modifiability of the clinical outcome measures.
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
80 F. RABBANI ET AL.
The panel used the modified Delphi technique (over a period of 6 months) during
face to facemeetings to individually rate each indicator on a scale of 1–9 for the above
criteria. All criteria were given equal weightage. If an indicator was thought not to be
amenable to action it was dropped. Median scores and measures of disagreement for
the whole panel and individual ratings were discussed, in subsequent meetings. Panel
members were given an opportunity to change their ratings after the discussions.
Indicators receiving final scores of 7–9were regarded as robust, 4–6 as equivocal, and
1–3asweak (Figure 2).All indicators receiving scores of 7ormore (facevalidity)were
included in the final set. In addition, a small number of indicators which received
scores of 4–6 were retained if the panelists considered the indicators essential to
contribute to the overall balance and comprehensiveness of the final set.
Twenty indicators (receiving a median score of 7 or more) were finally selected
(Table 2) and organized by the expert panel into the four BSC quadrants: financial,
customer (patient), internal business, and innovation/learning.
Indicators for innovation and learning quadrant of the BSC
The indicators for employee satisfaction (innovation/learning quadrant of BSC) were
selected from the annual faculty and staff surveys and included (i) satisfaction with
job; dimensions of training and skills, work load including double duties performed,
maximum use of staff abilities, decision-making authority, and motivation to strive
for excellence, (ii) collegial satisfaction (helping each other in times of need, respect
from the colleagues, discussion with colleagues to mutually resolve issues),
(iii) satisfaction with supervisor; dimensions of friendly working relationship, regular
feedback, satisfaction with appraisal system, recognition for doing a good job,
openness to suggestions, and good ideas, (iv) satisfaction with organization; annual
faculty and staff turnover, fair treatment without gender and religious discrimination,
opportunities for growth and improvement, proud to work, viewing organization as a
Figure 2. Short listing indicators for BSC using a modified Delphi process
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
Table 2. Shortlisted set of indicators for the BSC using the modified Delphi technique
Indicators
Financial perspective (FP) Median Mean Std. deviation
� Average charges (inpatient) 8.00 8.22 0.44� Length of stay (inpatient) 7.00 6.89 0.78� Daily census (inpatient) 8.00 8.11 0.60� Net operating margin 9.00 8.67 0.50� Overall FP Median 8.00Internal business perspective (IBP): clinical outcomes (efficiency and quality)� Laboratory report turnaround time 8.00 7.89 0.60� Radiology film rejection rate 8.00 8.33 0.50� Unplanned stay after day care procedure 7.00 6.78 1.20� Incidence of blood transfusion reaction 7.00 7.22 1.09� Nosocomial infection 8.00 8.00 0.71� Cross match to transfusion ratio 7.00 7.33 0.87� Needle stick injuries 8.00 7.89 1.17� Overall IBP Median 8.00Human resource perspective (HRP)� Satisfaction with job 7.00 7.33 0.71� Satisfaction with colleagues 8.00 7.78 0.83� Satisfaction with on campus facilities 8.00 7.44 0.73� Satisfaction with organization 7.00 6.89 0.93� Satisfaction with supervisors 7.00 7.00 1.12� Overall HRP Median 7.00Patient satisfaction perspective (PSP)� Satisfaction with physicians 7.00 7.11 0.78� Patient Complaints (inpatient) 8.00 7.56 0.88� Satisfaction with nursing services 7.00 7.33 0.87� Proportion of patients recommending
this hospital to their families and friends8.00 7.89 0.60
� Overall PSP Median 7.50
DESIGNING BSC USING MODIFIED DELPHI 81
long-term career choice, balance between work and personal life, and (v) satisfaction
with various on- campus staff facilities (sports and gymnasium, utility shops, child
day care center, cash withdrawal facilities, payment of utility bills etc.).
Some of the indicators reviewedwere not finally selected as theywere considered to
be more specific to the Human Resources Department (HRD) and not directly
influencingclinical serviceprovisionat thehospital.These included (i) theperformance
of HRD obtained through staff surveys (e.g., assistance provided to new employees,
managing employee discipline cases, promptness in responding to queries etc.), (ii)
indicators for employee safety and emergency preparedness obtained through Safety
andSecurityDepartment reports (e.g., fireemergency response time,numberof injuries
per100full-timeemployees) and (iii) indicatorsofworkforcemanagement suchas staff
absenteeism (doctors and nurses), number and type of employee illnesses.
Indicators for internal business quadrant of the BSC
Indicators for this quadrant were shortlisted from a larger list of quality care
indicators which the hospital (medical directorate) is monitoring through various
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
82 F. RABBANI ET AL.
quality assurance teams for ongoing accreditation by the Joint Commission on
Accreditation of Healthcare Organizations USA (JCAHO). The selected indicators
included three indicators of efficiency; laboratory report turnaround time (number of
samples reported within acceptable time limit per total number of samples analyzed),
radiology film rejection rate (number of rejected films out of total number used and
indicates film wastage rate with implications for appropriate training of radiology
staff in patient positioning and exposure techniques) and cross match to transfusion
ratio (proxy for indicating the actual need for carrying out blood transfusions). There
were two indicators of quality of care and efficiency; unplanned stay after day care
procedure (number of unplanned overstay following day care surgery out of patients
undergoing day care surgery) and incidence of blood transfusion reactions (number
of blood transfusion associated reactions out of total units of blood transfused).
Needle stick injuries were selected as an indicator of staff compliance with safety
techniques and quality procedures and is also an indicator of employee safety. Rate
of nosocomial infections (central line associated blood stream infections in intensive
care units in relation to device days (following CDC guidelines of National Nosocomial
Infection Surveillance) was selected as the indicator of infection control.
The Health Management Information System provided information on mortality
related to anesthesia (using American Society of Anaesthesia guidelines), however,
desirable information on adjusted case fatality rate for tracer conditions was not
available. Similarly peri-operative mortality (in elective procedures) and returns to
operation theatre during the same episode were considered as indicators of operative
safety however due to residual confounding effects (age, severity of illness etc.), these
were not finally selected. Hospital acquired pressure ulcers, adverse event reporting,
patient fall rates, unplanned descents to the floor /1000 patient days were initially
selected as indicators of nursing quality but the panelists were of the opinion that in-
depth discussion was required with Division of Nursing Services and individual
clinical units should consider these while developing their customized scorecards.
Indicators for customer (patient) satisfaction quadrant of the BSC
Quarterly patient satisfaction surveys by the hospital’s Marketing department and
quality reports from the Department of Clinical Affairs were used to select four
indicators of patient satisfaction The patient satisfaction survey captures, analyzes and
monitors patient satisfaction with outpatient, inpatient, diagnostic and emergency
services. For the purpose of this study only inpatient service indicatorswere discussed.
These indicators are; (i) satisfaction with nursing services (dimensions: provision of
adequate information on health condition/medicines/ follow up care, courtesy,
listening, prompt response to call bell, respect for privacy, skilful insertion of cannulas/
IV lines, proper dressing, provision of special help when needed) (ii) satisfaction with
physicians (dimensions: daily visit of consultant, proper explanation and information
given and respect for privacy) (iii) recommending this service to family and friends (a
proxy indicator for a satisfied client), and (iv) percentage of patient complaints
(dimensions: care, delays, environment, attitude, availability of health staff,
communication, billing system, quality of food, cleanliness of washrooms, level of
noise, scheduled tests and investigation procedures on time etc.).
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
DESIGNING BSC USING MODIFIED DELPHI 83
It is important to mention that the Department of Clinical Affairs and the
Hospital’s Risk Management Forum are already taking specific actions against those
patient complaints classified as ‘sensitive’ (resulting in potential defamation,
litigation, or compensation) and hence the latter was excluded from our list of
generic BSC indicators. Moreover ‘overall patient satisfaction’ was a composite
indicator shortlisted but not finally selected. The patients included in the sample are
requested to respond to the question’ were you overall satisfied with the quality of the
service you received? in terms of strongly agree, agree, neutral, disagree, and
strongly disagree. Responses obtained on a likert scale are converted to mean values
and reported as a percentage. Those who respond to this question with strongly agree
and agree are classified as overall satisfied patient. Sample size calculations for this
indicator needed statistical refinement and the panel could not rule out the element of
recall bias in these telephonically conducted interviews. Rates of medication error
(including prescribing, administration, and dispensing) and rationale use of
antibiotics were considered as indicators of patient safety, however, since there
were already specific quality control committees monitoring these on a quarterly
basis they were dropped as key indicators for the institutional level BSC.
Indicators for financial quadrant of the BSC
Four indicators were finally selected (i) average charges per inpatients (total inpatient
revenue earned against total number of patients admitted over a period) is a measure
of the accessibility of patients to hospital and also the cost-effectiveness of services
provided. It is computed by comparing the increase in average charges per inpatient
with the average price increase and the inflation rate, (ii) inpatient length of stay
(indicator of efficiency), (iii) average daily census (average number of patients
occupying bed per day), and (iv) net operating margin (margin on gross revenues
before interest and depreciation: indicator of cost and productivity) were some
indicators routinely generated by the department of Budget and Planning and later
shortlisted for the financial quadrant of the BSC.
FTE per adjusted occupied bed (an indicator of patient staff ratios and efficiency
monitored for JCAHO), % capex expenditure versus planned (highlights the total
capital budget consumed against the annual budget), percentage of referrals from
CHC; the low cost outpatient clinic (another indicator of financial accessibility of
this hospital for all socioeconomic groups) were selected in the initial list of 50
indicators but were dropped later due to lack of available national and regional
benchmarking and because some of this information was not considered relevant for
general public disclosure.
DISCUSSION
To the best of our knowledge, this was the first time that experts (managers, academicians,
and clinicians) were together involved in a scientific process (the modified Delphi group
technique) to develop a BSC for a hospital in a LIC setting. Integrating the activities
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
84 F. RABBANI ET AL.
of different departments is a difficult task for the management of the organization
(Axelsson and Axelsson, 2006). Designing the BSC was possible in our study
because most of the necessary prerequisites for successful BSC implementation in
LICs (committed leadership, viable strategic plans and information systems etc.)
were already in place and cultural readiness for BSC usage had previously been
assessed (Rabbani et al., 2007; Rabbani et al., 2008). The modified Delphi method
successfully incorporated views from health personnel and specialists in the
development of a BSC. The same has been reported from Canadian hospitals
(Robinson et al., 2003). Another study of nine health provider organizations in USA
emphasized the importance of using a lot of teaching, discussion and consensus
building to ensure a successful BSC implementation (Inamdar et al., 2002).
Moreover consensus techniques such as modified Delphi have been utilized in
Thailand to develop trauma care indicators (Suwaratchai et al., 2008). The recent use
of formal consensus methods in Pakistan’s neighboring Muslim country Iran to
identify outcome-based indicators for rational drug prescribing and effective
academic leadership in order to increase the validity of the findings is also
encouraging (Bikmoradi et al., 2008; Esmaily et al., 2008). It is noteworthy that the
criteria (importance, scientific soundness, feasibility etc.) used by other studies
(Idanpaan-Heikkila, 2006; Marshall et al., 2006; McLoughlin et al., 2006) to short
list indicators also worked well in our setting.
Interestingly the composition of the multidisciplinary panel in our study was
found to be quite similar to other studies on development of a BSC for hospitals in
HICs. In an Australian study, experts in the Delphi panel included both hospital
managers and clinical practitioners (Xiao et al., 1997). In Taiwan (Huang et al.,
2004) the team that worked to develop the BSC included president, Vice President
and all department directors of the study hospital. In another recent study on
implementation of BSC in a community hospital at USA besides others the panel of
experts consisted of directors from patient care services and quality management
(Lorden et al., 2008). It is envisaged that this involvement of staff at different levels
within an organization during the development of BSC will enhance the acceptance
of the scorecard when it is implemented.
The Balanced Scorecard provided an opportunity to capture indicators in four
aspects of hospital performance. This demonstrates that data which are routinely
collected by the hospital can be used to develop an integrated core of
multidisciplinary indicators. This institutional level BSC has been designed for
use at the level of medical directorate. Subsequent studies can contextualize and
customize BSC by each of the implementing frontline clinical units so that specialty–
specific BSCs can be developed. Other studies have also used existing documents in a
similar fashion to create effective BSC (Wachtel et al., 1999; Idanpaan-Heikkila,
2006; Marshall et al., 2006; McLoughlin et al., 2006). In a recently concluded study
BSCwas used to track certain nursing indicators in acute care Ontario hospitals using
secondary data (Hall et al., 2008).
There was relatively a high level of agreement about the usefulness of the 20
indicators which were finally selected in our study. These indicators were distributed
across all 4 quadrants of BSC: financial perspective (n¼ 4), internal business (n¼ 7),
human resource perspective (n¼ 5) and patient satisfaction perspective (n¼ 4). The
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
DESIGNING BSC USING MODIFIED DELPHI 85
indicators are similar to the ones shortlisted in both high (Baker and Pink, 1995) and
low income (Hansen et al., 2008) health settings. In the former study revenue
generated, patient volumes, patient and employee satisfaction, nosocomial
infections, average length of stay, and routine laboratory test turnaround time were
included among the 23 indicators shortlisted for the BSC developed for Canadian
hospitals. In the latter study in Afghanistan domains of patient and community, staff,
capacity for service, service provision, financial system, and overall vision were used
to monitor 29 indicators at provincial level. The latter are quite similar to the ones
selected in our setting. In another recent Canadian study following initial steps of
building executive commitment and strategic alignment, 23 indicators were
shortlisted which could be compared across various care centers using the same BSC
dimensions (Schalm, 2008).
The results from this investigation also reflect limitations of routinely collected
data. The Delphi process highlighted that certain indicators selected for BSC
(Table 3) had relatively lower face validity as assessed by their median ratings. This
was mainly due to lack of standardized definitions and measurement techniques,
reliable instruments, adequate sample sizes and response rates etc. Other studies
(Baker and Pink, 1995; Zeitlin et al., 2003) have also noted that methodological
shortcomings of many indicators have generated skepticism about the data sources,
consistency of reporting, derivation of the numbers, and their usefulness in offering
analogous estimates. It is to be noted that Pakistan does not have a national hospital
database. Comparable national/regional targets, benchmarking and a balance
between process and outcome indicators was recommended. It was also noted that
for subsequent designing of BSC for front line clinical departments disaggregation of
information by each clinical specialty would be needed.
It is possible that despite efforts to capture all relevant indicators through publicly
available surveys and documents, certain valuable indicators may have been
overlooked. Some of the shortlisted indicators in the western studies included
allocative efficiency, vertical equity, survival rates and age, sex and disease specific
mortality, and morbidity ratios (Wachtel et al., 1999; Robinson et al., 2003; ten
Asbroek et al., 2004; Schalm, 2008). The BSC developed in our setting did not have
some of the more analytical indicators listed above. Such lack of in-depth outcome
data has been listed as a BSC implementation barrier elsewhere (Schalm, 2008). It is
important to mention that to date at this hospital only diagnostic services (laboratory,
radiology) and pharmacy (data on medications prescribed and dispensed) are fully
computerized. Although each patient visiting this hospital has a unique medical
record number and information related to patient characteristics (age, gender,
diagnosis, length of stay, and clinical intervention performed) is computerized,
however, presenting complaints, co-morbid conditions, and discharge summaries are
still available only on paper files. Similar issues of patient data records have been
reported from Ethiopian hospitals (Hartwig et al., 2008). Moreover it has been shown
in Iran that hospitals are collecting a lot of financial and clinical information in a
fairly computerized but not in a well-organized format (Ghaffari et al., 2008). A need
for an electronic patient record system (an e–health initiative) was, therefore,
emphasized to overcome some of these methodological barriers and come up with
more robust indicators in our setting.
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
Table
3.Measurementissues
highlightedduringtheDelphiprocess
Nam
eofindicator
Sourceofdata
Issues
Financial
perspective
�Length
ofstay
(inpatient)
Monthly
reportsofdepartm
ent
ofbudget
andplanning
�Cannotstratify
byseverityofillness,number
ofcomplications
andco-m
orbidities
�Thisindicatorcanbeinfluencedbyfactors
beyondthehospital
environment(e.g.,absence
ofnursinghomes
andhomecare
facilities
postdischarge)
�Length
ofstay
canalso
increase
because
ofmanagem
entdelays
(schedulingofinvestigationprocedures,timingofconsultant
wardrounds,weekendadmissionsetc.)
Internal
business(efficiency
andquality)
�Unplanned
stay
afterday
care
procedure
Quarterlyqualityim
provem
ent
reports
�Cannotdifferentiatebetweensurgical
complicationsand
managem
entdelays(e.g.,delay
inavailabilityofoperation
theatreandlow
staffinglevels)
�Cannotcommentonwhichprocedure
causesmaxim
um
delay
�Incidence
ofbloodtransfusion
reaction
Quarterlyqualityim
provem
ent
reports
�Needto
setatarget
based
oninternational
comparison
�Cross
match
totransfusionratio
�Only
trendbeingmonitoredcurrently
Human
resourceperspective
�Satisfactionwithjob
Annual
employee
satisfaction
survey
�Low
response
rate
tothesurvey
�Notdisaggregated
bydepartm
entordesignation
�Only
quantitativeinform
ationcollectedonaslidingscale
�Satisfactionwithorganization
�Datacollectioninstrumentnotvalidated,questionslikelyto
evoke
apositiveresponse
�Satisfactionwithsupervisors
�Nointernational/regional
comparisonsortargets
Patientsatisfactionperspective
�Satisfactionwithphysicians
Quarterlypatientsatisfaction
surveys
�Does
notconvey
underlyinginform
ationaboutsatisfactionwith
physician
andnursingservices
byeach
clinical
departm
entand
therefore
delaysaction
�Satisfactionwithnursing
services
Measurementissues
relatedto
9/20indicators
oftheBSC
arehighlighted.Theseindicators
received
arelativelylower
rating(m
edian¼7).
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
86 F. RABBANI ET AL.
DESIGNING BSC USING MODIFIED DELPHI 87
Despite these practical limitations, the Delphi group process led to a pragmatic
interpretation of existing data resulting in the design of a scorecard with
comprehensive indicators in multiple dimensions. It has been reported that for
hospital managers and those developing health policies, studies such as this provide
insight into the factors influencing hospital performance (Xiao et al., 1997). The 20
indicators which emerged from our study using the modified Delphi process have
highlighted the methodological challenges faced during the design of BSC. As a next
step this scorecard will now be customized for individual clinical departments of this
hospital and later implemented at the executive and board level. The BSC would
facilitate rational organization and management of data collection systems and serve
as an evaluation framework for monitoring improvement of clinical outcomes and
quality. A greater cohesion within the hospital units is expected simultaneously.
Lessons learnt will have important bearing for hospital performance measurement
initiatives in other settings.
ACKNOWLEDGEMENTS
The authors thank the senior Aga Khan University (AKU) and hospital (AKUH)
leadership—President Firoz Rasul, Vice President Health Services Dallas Ariotti,
Dean of the Medical college AKU, Mohammad Khurshid, Director General and
CEOAKUH Nadeem Khan, Dean of Research and Graduate Studies AKU, El-Nasir
Lalani, Director Hiuman Resources AKU, Navroz Surani, and chair Dept of
Community Health Sciences (CHS) Dr Gregory Pappas—for encouraging us to
proceed with the work related to Balanced Scorecard (BSC) at AKU. This study is a
component of BSC studies underway. We thank Dr Naushaba Mobeen, Senior
Instructor Community Health Sciences and Dr Wasif Shahzad Manager Dept of
Medicine for assisting in initial meetings. We also express our gratitude to Mr Zafar
Tahir (CHS), Aslam Fareed and Muhammad Feisal (Marketing Department AKUH),
Ms Salma Jaffer (Manager JCIA Coordination AKUH), Ms Shamim Nayani (Senior
Manager Employee Relations, Rehman Hirani, and Khurram Jamal (Dept of Budget
and Planning AKU). Ms Saira Nigar (CHS) assisted us in data analysis and
Ms Shafaq Ambreen, administrative officer (CHS) rendered untiring secretarial assist-
ance. We thank Bo Badr Saleem Lindblad, Professor Emeritus of International Child
Health, Department of Public Health Sciences, Division of International Health
(IHCAR), Karolinska Institutet Medical University, Stockholm, Sweden, and visit-
ing professor, AKU, Karachi, Pakistan, for his overall support. Thanks also go to
Mr Thomas Mellin at IHCAR, Department of Public Health Sciences, Karolinska
Institutet (KI), Sweden for connecting the first author to various information
technology resources during her visits to KI. The authors acknowledge our various
grant sources: Swedish South Asian Network (SASNET: grant ID; EPG05S:06),
WHO EMRO (project ID #: RPC 04/60) and Swedish Institute (Si: Id # 05655/2005).
The major support for this study came from AKU University Research Council
(URC, project ID 052013 CHS).
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
88 F. RABBANI ET AL.
REFERENCES
Abrejo FG, Shaikh BT, Saleem S. 2008. ICPD to MDGs: missing links and common grounds.Reprod Health 5(4): DOI: 10.1186/1742-4755-5-4
Axelsson R, Axelsson SB. 2006. Integration and collaboration in public health—a conceptualframework. Int J Health Plann Manage 19: 383–398. DOI: 10.1002/hpm.826
Baker GR, Pink GH. 1995. A balanced scorecard for Canadian hospitals. Healthc ManageForum 8: 7–21.
Bikmoradi A, Brommels M, Shoghli A, Sohrabi Z, Masiello I. 2008. Requirements foreffective academic leadership in Iran: a nominal group technique exercise. BMC Med Educ8: 24.
Campbell SM, Braspenning J, Hutchinson A, Marshall M. 2002. Research methods used indeveloping and applying quality indicators in primary care. Qual Saf Health Care 11: 358–364. DOI:10.1136/qhc.11.4.358
Castaneda-Mendez K, Mangan K, Lavery AM. 1998. The role and application of the balancedscorecard in healthcare quality management. J Healthc Qual 20: 10–13.
Chung KP, Lai MS, Cheng SH, et al. 2008. Organization-based performance measures ofcancer care quality: core measure development for breast cancer in Taiwan. Eur J CancerCare (Engl) 17: 5–18. DOI: 10.1111/j.1365-2354.2007.00796.x
Curtright JW, Stolp-Smith SC, Edell ES. 2000. Strategic performance management: devel-opment of a performance measurement system at mayo clinic. J Healthc Manag 45: 58–68.
Esmaily HM, Savage C, Vahidi R, Amini A, Zarrintan MH, Wahlstrom R. 2008. Identifyingoutcome-based indicators and developing a curriculum for a continuing medical educationprogramme on rational prescribing using a modified Delphi process. BMCMed Educ 8(33):DOI: 10.1186/1472-6920-8-33
Ghaffar A, Kazi BM, SalmanM. 2000. Health care systems in transition III. Pakistan, Part I. anoverview of the health care system in Pakistan. J Public Health Med 22: 38.
Ghaffari S, Doran C, Wilson A, Aisbett C, Jackson T. 2008. Investigating DRG cost weightsfor hospitals in low resource countries: an Iranian example. Int J Health Plann Manage1–14. DOI: 10.1002/hpm.948
Gordon D, Geiger G. 1999. Strategic management of an electronic patient record project usingthe balanced scorecard. J Healthc Inf Manag 13: 113–123.
Green A, Collins C. 2003. Health systems in developing countries: public sector managers andthe management of contradictions and change. Int J Health Plann Manage 18: S67–S78.DOI: 10.1002/hpm.721
Groene O, Skau JKH, Frolich A. 2008. An international review of projects on hospitalperformance assessment. Int J Qual Health Care 20(162): DOI: 10.1093/intqhc/mzn008
Hall LM, Peterson J, Baker GR, et al. 2008. Nurse staffing and system integration and changeindicators in acute care hospitals: evidence from a balanced scorecard. J Nurs Care Qual 23:242–252. DOI: 10.1097/01.NCQ.0000310655.60187.92
Hansen PM, Peters DH, Niayesh H, Singh LP, Dwivedi V, Burnham G. 2008. Measuring andmanaging progress in the establishment of basic health services: the Afghanistan healthsector balanced scorecard. Int J Health PlannManage 23: 107–117. DOI: 10.1002/hpm.931
Hartwig K, Pashman J, Cherlin E, et al. 2008. Hospital management in the context of healthsector reform: a planning model in Ethiopia. Int J Health Plann Manage 23: 203–218. DOI:10.1002/hpm.915
Hospital Report. 2003. Acute Care. A joint initiative of the University of Toronto, OntarioHospital. Association and the Government of Ontario. Canadian Institute for HealthInformation Ontario.
Huang SH, Chen PL, Yang MC, Chang WY, Lee HJ. 2004. Using a balanced scorecard toimprove the performance of an emergency department. Nurs Econ 22: 140–146, 107.
Idanpaan-Heikkila U. 2006. Selecting indicators for the quality of cardiac care at the healthsystem level in organization for economic co-operation and development countries. Int JQual Health Care 18: 39–344. DOI: 10.1093/intqhc/mzl028
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
DESIGNING BSC USING MODIFIED DELPHI 89
Inamdar N, Kaplan RS, Bower M. 2002. Applying the balanced scorecard in healthcareprovider organizations. J Healthc Manag 47: 179–196.
Kaplan RS, Norton DP. 1996. The balanced scorecard: translating strategy into action.Harvard Business School Press: Boston, MA.
Kaplan RS, Norton DP. 2000. The Strategy-Focused Organization: How Balanced ScorecardCompanies Thrive in the New Business Environment. Harvard Business School Press:Boston MA.
Lorden A, Coustasse A, Singh KP. 2008. The balanced scorecard framework-A case study ofpatient and employee satisfaction: what happens when it does not work as planned? HealthCare Manage Rev 33(145): DOI: 10.1097/01.HMR.0000304503.27803
Malqvist M, Eriksson L, Nga NT, et al. 2008. Unreported births and deaths, a severe obstaclefor improved neonatal survival in low-income countries; a population based study. BMC IntHealth Hum Rights 8: 1–7. DOI: 10.1186/1472-698X-8-4
Mannion R, Davies H, Marshall M. 2005. Impact of star performance ratings in English acutehospital trusts. J Health Serv Res Policy 10: 18–24.
Marshall M, Klazinga N, Leatherman S, et al. 2006. OECD health care quality indicatorproject. The expert panel on primary care prevention and health promotion. Int J QualHealth Care 18: 21–25. DOI: 10.1093/intqhc/mzl021
McBurney D. 2001. Research methods. Wadsworth Thomson Learning: Belmont, CA,Stamford, USA.
McKeeM, Healy J, Edwards N, Harrison A. 2002. Pressures for change.Hospitals in changingEurope. World Health Organisation (WHO): European observatory on health care systemsseries 36–58.
McLoughlin V, Millar J, Mattke S, et al. 2006. Selecting indicators for patient safety at thehealth system level in OECD countries. Int J Qual Health Care 18: 14–20. DOI: 10.1093/intqhc/mzl030
Murphy MK, Black NA, Lamping DL, et al. 1998. Consensus development methods, and theiruse in clinical guideline development. Health Technol Assess 2: 1–88.
Murray CJL. 2007. Towards good practice for health statistics: lessons from the millenniumdevelopment goal health indicators. The Lancet 369: 862–873. DOI: 10.1016/S0140-6736(07)60415-2
Oliveira J. 2001. The balanced scorecard: an integrative approach to performance evaluation.Healthc Financ Manage 55: 42–46.
Ovretveit J, Al Serouri A. 2006. Hospital quality management system in a low income Arabiccountry: an evaluation. Int J Health Care Qual Assur 19: 516–532. DOI: 10.1108/09526860610686999
Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. 2007. A balancedscorecard for health services in Afghanistan. Bull World Health Organ 85: 146–151.
Rabbani F, Jafri W, Abbas A, et al. 2009. Culture and quality care perceptions in a Pakistanihospital. Int J Health Care Qual Assur 22: 498–513. DOI: 10.1108/09526860910975607
Rabbani F, Jafri W, Abbas F, Pappas G, Brommels M, Tomson G. 2007. Reviewing theapplication of the balanced scorecard with implications for low-income health settings.J Healthc Qual 29: 21–34.
Rafique G, Azam SI, White F. 2006. Diabetes knowledge, beliefs and practices among peoplewith diabetes attending a university hospital in Karachi, Pakistan. East Mediterr Health J12: 590–598.
Robinson VA, Hunter D, Shortt SE. 2003. Accountability in public health units: using amodified nominal group technique to develop a balanced scorecard for performancemeasurement. Can J Public Health 94: 391–396.
Rowe AK, de Savigny D, Lanata CF, Victora CG. 2005. How can we achieve and maintainhigh-quality performance of health workers in low-resource settings? The Lancet 366:1026–1035. DOI: 10.1016/S0140-6736(05)67028-6
Ruiz U, Simon J, Molina P, Jimenez J, Grandal J. 1999. A two-level integrated approach toself-assessment in healthcare organisations. Int J Health Care Qual Assur 12: 135–142.
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm
90 F. RABBANI ET AL.
Schalm C. 2008. Implementing a balanced scorecard as a strategic management tool in a long-term care organization. J Health Serv Res Policy 13: 8–14. DOI: 10.1258/jhsrp.2007.007013
Suwaratchai P, Sithisarankul P, Sriratanban J, Chenvidhyamd D, Phonburee W. 2008. Utilizethe modified Delphi technique to develop trauma care indicators. J Med Assoc Thai 91: 99–103.
ten Asbroek AHA, Arah OA, Geelhoed J, Custers T, Delnoij DM, Klazinga NS. 2004.Developing a national performance indicator framework for the Dutch health system. Int JQual Health Care 16: i65–i71. DOI: 10.1093/intqhc/mzh020
Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset AL. 2005.A performance assessment framework for hospitals: the WHO regional office for EuropePATH project. Int J Qual Health Care 17(487): DOI: 10.1093/intqhc/mzi072
Wachtel TL, Hartford CE, Hughes JA. 1999. Building a balanced scorecard for a burn center.Burns 25: 431–437. DOI: 10.1016/S0305-4179(99)00024-8
Willis CD, Stoelwinder JU, Cameron PA. 2008. Interpreting process indicators in trauma care:construct validity versus confounding by indication. Int J Qual Health Care 20: 1–8. DOI:10.1093/intqhc/mzn027
Xiao J, Douglas D, Lee AH, Vemuri SR. 1997. A Delphi evaluation of the factors influencinglength of stay in Australian hospitals. Int J Health Plann Manage 12: 207–218. DOI:110.1002/(SICI)1099-1751(199707/09)12:3
Zeitlin J, Wildman K, Breart G, et al. 2003. Selecting an indicator set for monitoring andevaluating perinatal health in Europe: criteria, methods and results from the PERISTATproject. Eur J Obstet Gynecol Reprod Biol 111: 5–14. DOI: 10.1016/j.ejogrb.2003.09.002
Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.
DOI: 10.1002/hpm