Psychometric Evaluation of an Inpatient Consumer Survey Measuring Satisfaction with Psychiatric Care

Psychometric Evaluation of an InpatientConsumer Survey Measuring Satisfactionwith Psychiatric CareGlorimar Ortiz and Lucille Schacht

National Association of State Mental Health Program Directors Research Institute (NRI), Inc., Falls Church,

VA, USA

Abstract Background: Measurement of consumers’ satisfaction in psychiatric settings

is important because it has been correlated with improved clinical outcomes

and administrative measures of high-quality care. These consumer sat-

isfaction measurements are actively used as performance measures required

by the accreditation process and for quality improvement activities.

Objectives: Our objectives were (i) to re-evaluate, through exploratory factor

analysis (EFA) and confirmatory factor analysis (CFA), the structure of an

instrument intended to measure consumers’ satisfaction with care in psychi-

atric settings and (ii) to examine and publish the psychometric characteristics,

validity and reliability, of the Inpatient Consumer Survey (ICS).

Methods: To psychometrically test the structure of the ICS, 34 878 survey

results, submitted by 90 psychiatric hospitals in 2008, were extracted from the

Behavioral Healthcare Performance Measurement System (BHPMS). Basic

descriptive item-response and correlation analyses were performed for total

surveys. Two datasets were randomly created for analysis. A random sample

of 8229 survey results was used for EFA. Another random sample of 8261

consumer survey results was used for CFA. This same sample was used to

perform validity and reliability analyses.

Results: The item-response analysis showed that the mean range for a dis-

agree/agree five-point scale was 3.10–3.94. Correlation analysis showed a

strong relationship between items. Six domains (dignity, rights, environment,

empowerment, participation, and outcome) with internal reliabilities between

good to moderate (0.87–0.73) were shown to be related to overall care sat-

isfaction. Overall reliability for the instrument was excellent (0.94). Results

from CFA provided support for the domains structure of the ICS proposed

through EFA.

Conclusion: The overall findings from this study provide evidence that the

ICS is a reliable measure of consumer satisfaction in psychiatric inpatient

settings. The analysis has shown the ICS to provide valid and reliable results

ORIGINAL RESEARCH ARTICLEPatient 2012; 5 (3): 163-173

1178-1653/12/0003-0163/$49.95/0

Adis ª 2012 Springer International Publishing AG. All rights reserved.

and to focus on the specific concerns of consumers of psychiatric inpatient

care. Scores by item indicate that opportunity for improvement exists across

healthcare organizations.

Key points for decision makers

� The Inpatient Consumer Survey (ICS) is an effective tool for evaluating psychiatric inpatientcare using dimensions deemed important to the consumers of that care

� The ICS has been in use for 10 years; its fundamental qualities were validated

� The ICS poses little effort from a hospital’s staff as the survey is self-administered. The formatis easy for consumers to use and for hospitals to calculate domain scores for quality im-provement

� Hospitals can compare domain scores with other measures of treatment to identify clinicalpractices having a positive effect on consumers

Introduction

It has been demonstrated that consumer sat-isfaction with care is a multidimensional con-cept.[1-3] Measurement of consumers’ satisfactionin psychiatric settings is important because it hasbeen correlated with improved clinical outcomesand administrative measures of high-qualitycare.[4,5] The perception of care gathered from aconsumer standpoint is traditionally used by ad-ministrators as an internal tool[6] for the improve-ment, planning, and delivery of services, and morerecently the information is shared not only withstaff but also with consumers as a way of increas-ing their knowledge regarding the performance ofthe provider. As administrators and managementstaffs value these measurements, outcome datacan even be found publicly posted on displayboards available for the general public as well.

The Joint Commission (TJC) is the standard-setting and accrediting body that accredits andcertifies more than 19 000 healthcare organiza-tions and programs in the US and is recognized asa symbol of quality that reflects an organization’scommitment to meeting certain performance stan-dards.[7] In 1997, TJC approved an initiative calledORYX� to integrate outcomes and other per-formance measures into the accreditation pro-cess.[8] Healthcare organizations are required to

collect performance data related to outcomes oftheir patient care and submit these data back toTJC, through an authorized listed vendor or per-formance measurement system, on a continuingbasis as part of the accreditation cycle.[9] TheORYX� initiative was designed to increase the rel-evance and value of accreditation; support systemand process improvement in accredited organiza-tions; and permit comparative performance eval-uations among healthcare organizations.[8]

In 1998, the National Association of StateMental Health Program Directors Research In-stitute (NRI), Inc. developed the BehavioralHealth-care Performance Measurement System (BHPMS)to assist psychiatric hospitals in meeting theORYX� requirements, and in 1999, the BHPMSbecame an ORYX� authorized vendor. Currently,the BHPMS also offers enhanced technical assis-tance, data quality reviews, and specialized reportsfor about 200 state and private participatingpsychiatric hospitals. In 2001, NRI developedand pilot tested a 43-item Inpatient ConsumerSurvey (ICS). NRI recognized the importance ofintegrating consumers’ perceptions of care intostandard performance measures for psychiatrichospitals,[10] as these measures are an integralpart of the ORYX� requirements of the TJC ac-creditation process.[8] NRI also realized the ne-cessity of an instrument able to produce national

164 Ortiz & Schacht

Adis ª 2012 Springer International Publishing AG. All rights reserved. Patient 2012; 5 (3)

and standardized comparative data across allpsychiatric hospitals.

During 2008, 90 psychiatric hospitals used theICS and submitted their data on a monthly basisto the BHPMS for performance measurementand comparison reports. While the reliability ofthe instrument has been tested internally, the de-tails of the psychometric properties of the 43-iteminitial instrument or the 28-item revised instru-ment have not been published. At the time of thepilot study, the NRI’s main goal was to developadditional measures to meet changing ORYX�

requirements. NRI’s publication of the pilot re-sults included brief statements on the results ofthe factor analysis and the decision for definingperformance measures.[10] Over the past few years,there has been increased interest in using vali-dated instruments as the foundation for commonmeasures. While NRI has internally validated theinstrument for its client hospitals, publication ofthe details of the psychometric properties of thefinal 28-item instrument is needed for nationalutilization of benchmarking data. The empiricalinvestigation of consumer satisfaction assumes,at a minimum, that the measures used to assess itare reliable and valid.[11] Even more, the impor-tance of showing that a test instrument has goodpsychometric properties is widely recognized.[1]

Therefore, the ultimate objectives of this researchstudy were (i) to further re-evaluate, through ex-ploratory factor analysis (EFA) and confirmatoryfactor analysis (CFA), the ICS structure; and(ii) to examine and publish the psychometriccharacteristics, validity and reliability, of the ICS.We hypothesized that the current ICS structureand its domains will replicate those found duringthe analysis of pilot data that used the 43-itemquestionnaire. We also hypothesized that the ICSis a reliable and valid instrument to measure sat-isfaction with psychiatric inpatient care.

Methods

All statistical analyses were conducted usingSPSS statistical tool version 17[12] and Amos 18.[13]

To comply with the Health Insurance Portabilityand Accountability Act (HIPAA) and the federalprivacy standards, all data extracted from the

BHPMS were de-identified and anonymous. Ap-proval was received from the NRI InstitutionalReview Board.

Instrument

The ICS is a proprietary 28-item survey used bypsychiatric inpatient hospitals to measure con-sumers’ satisfaction with care (see the survey inthe Supplemental Digital Content [SDC], http://links.adisonline.com/PBZ/A41). A pilot ICS of43 items was developed through a focus group in-volving a number of consumers and staff of psy-chiatric inpatient services and using the outpatientMental Health Statistics Improvement Program(MHSIP) Consumer Survey as a foundation. Theinstrument includes items related to medication,treatment, qualities of self-awareness, interactionwith staff, and hospital stay, and seven consumerdemographic questions to obtain a description ofthe individuals served. The items use a Likert scaleranging from 1 (‘strongly disagree’) to 5 (‘stronglyagree’). A ‘not applicable’ option is also included.The instrument is written to be understood at thefifth-grade level, it is appropriate for use with ad-olescents and adults, and it is intended to be self-administered at discharge and at annual review.Higher scores reflect higher satisfaction with thecare received. NRI permits any hospital to use thesurvey, independent of enrollment in the BHPMS.In January 2011, the National Quality Forum, aleading national organization whose efforts striveto improve the quality of American healthcare, en-dorsed the ICS as an outcome measure to assessthe results and thereby improve the care providedto people with mental illness.[14]

The results from EFA and CFA of the pilotimplementation indicated five domains. It wasfound that hospitals scored better on the dignitydomain than on other domains, and that 67% ofthe consumers at the participating hospitals feltsome degree of satisfaction with aspects of theirinpatient care.

Study Population

This secondary data analysis included an anon-ymous dataset (n = 34 878) extracted from theBHPMS for the period of 1 January to 31December

Psychometric Evaluation of a Psychiatric Care Consumer Survey 165


2008. The survey results were from individualsthat received psychiatric care in 90 psychiatrichospitals representing 24 states. Hospitals wereexcluded from analysis if at the time of the studythey were not enrolled in the BHPMS. As the ICSis not appropriate for children, the study analysisincluded data from individuals aged 13 years andolder. Individuals were not excluded based ongender/ethnic/race categories. Since the universalpopulation of the surveys was large (n = 34 878),study cohorts were selected using a random pro-cess of 15% of surveys completed at dischargeand all surveys performed at annual review forbetter interpretation of outcomes. Study cohortswere not selected based on demographic variations.

Descriptive Analysis

Analyses of skewness, peakedness, and missingvaluewere performed for each item in the ICS usingthe following criteria: skewness or asymmetry(less than 1.5 or greater than 1.5); kurtosis, thatis, more peaked or flat than Normal distribution(less than 1.5 or greater than 1.5); and analysis ofmissing values (i.e. greater than 20%). Items notmeeting these criteria were dropped from ana-lysis.[15] Item 28 was identified as an anchor itemand was dropped from EFA. Analysis of corre-lation between items was performed using a cut-off value of 0.30 or higher for all items.

Item Generation

Beginning in the spring of 2000, NRI invitedconsumers of psychiatric inpatient care and theMHSIP Policy Group to assist the NRI in for-mulating an inpatient version of the MHSIP Con-sumer Survey. A workgroup was formed consistingof representatives from these two groups, a re-search consultant, and NRI-BHPMS staff. Theoutcome of a series of meetings was an instrumentconsisting of 43 items in total, organized around sixconceptual domains, and a plan for implemen-tation and analysis.[10]

Item Reduction

The original 43-item ICS was pilot tested by15 state psychiatric hospitals during November

2000 through February 2001. A total of 1027 con-sumers completed the survey. EFA and CFAfrom the original pilot testing reduced the origi-nal 43-item instrument to 28 items. The workgroupcommented on wording, readability, length, re-dundancy, and items that were not clear on thenew instrument. Five domains using 18 items wereidentified. The remaining items, even though theydid not load in a particular domain, were kept inthe instrument as they address general satisfac-tion, staff, and medications.[10]

One of the interests of this research was to re-explore, replicate, and summarize the under-lying structure for the 28-item ICS. EFAwas basedon an independent, random sample of 15% of alldischarges combinedwith all cases completed at anannual review (n= 8229). Principal componentanalysis (PCA) was performed to identify thenumber of domains to be retained through ana-lysis of eigenvalues >1, scree plot, and percent-age of variance explained at a cut-off of 70%or above.[16] Parallel analysis (PA) was also per-formed as this method has become more preva-lent and standard, and it is one of the mostaccurate methods for determining the number ofdomains to retain.[17]

After the number of domains to be retained wasdetermined, the principal axis factor (PAF)methodusing maximum likelihood was performed. Thesolution obtained was then rotated for inter-pretation. The assumption that domains were notindependent led for an oblique rotation (promaxcriterion).[18] A cut-off equal to or greater than0.30 was used to determine the items underlying aspecific domain structure. Items with loadingsless than 0.30 or not loading in any domain weredropped from the instrument, and PCA, PA, andPAF were re-run.

Validity

CFA was used for construct validation.[19] Anindependent sample of 15% of discharges was com-bined with all cases completed on annual reviewto form the sample for CFA. A co-variance ma-trix was created to perform CFA. Using Amos18,[13] a model was programmed containing sixdomains or latent variables and the group of items

166 Ortiz & Schacht


associated to each one of them. Then, the modelidentification was determined, the parameterswere estimated, and the model fit was assessed.The number of latent variables and the observedvariables were set a priori along with a constraintto the latent variables (e.g. fixed coefficient of oneindicator).[20] Eleven goodness-of-fit indices wereused for the evaluation of model fit[21] (see ap-pendix G in the SDC).

Pearson’s correlation examined the instru-ment convergent validity, or the extent to whichthe instrument correlates with similar constructs.Multiple regression analysis examined criterion-related (predictive) validity. External validity wasinvestigated as demographic information fromsample cohorts was compared with demographicinformation for the discharged population in2008.

Reliability

Cronbach’s coefficient a examined the inter-nal consistency of the total instrument and itsdomains. The internal reliability estimates wereevaluated against the cut-off score of 0.70.[22] Thedomains, each latent variable with their associateditems, were created after CFA.

Results

Of the 94 hospitals that used the ICS during2008, 90 were still active at the time of the study,permitting use of their data for analysis. Therewere 34 878 surveys completed in calendar year2008, 3808 were categorized as annual review andthe remaining surveys were categorized as dis-charge. Only 13 hospitals did not reach at leasta 25% response rate, while 17 hospitals had aresponse rate that exceeded 70%; the averageresponse rate was nearly 51%.

Table I presents the sample characteristics.Seventy-nine percent of individuals were 18–54 yearsold, 57% were male, 52% were never married,39% had a civil commitment, 53% were White,73% received acute care, 84% completed thesurvey at discharge, and 27% received assistancewhile completing the survey. Seventy-nine per-cent of the surveys were collected at the hospitals.

Table I. Sample characteristics (n =34 878)

Characteristic n (%)

Gender

Male 19 840 (57)

Female 12 991 (37)

Missing 2 047 (6)

Age

13–17 y 1354 (4)

18–54 y 27 397 (79)

55–64 y 3059 (9)

65 y and older 1 122 (3)

Missing 1 946 (5)

Race

African American 7520 (22)

Hispanic 3 232 (9)

White 18 347 (53)

Othera 2 776 (8)

Missing 3 003 (8)

Marital Status

Never married 18 253 (52)

Now married 4 138 (12)

Formerly marriedb 9 931 (29)

Missing 2 556 (7)

Legal status

Voluntary 12 206 (35)

Involuntary – civil 13 758 (39)

Involuntary – otherc 4 204 (12)

Missing 4 710 (14)

Length of stay

3 mo or less 25 400 (73)

More than 3 mo 7133 (20)

Missing 2 345 (7)

Completed the survey at discharge 29 419 (84)

Anonymityd 24 879 (71)

Assisted in completion 9 390 (27)

Returned the survey to the hospital 27 585 (79)

a Includes Native American/Alaskan Native, Asian/Pacific Islander,

and other.

b Includes divorced, widowed, and separated.

c Includes involuntary criminal and juvenile justice.

d This characteristic refers to the distribution of the survey at a

hospital. All surveys received at the BHPMS are anonymous;

however, within hospitals there could be a mechanism to link the

survey back to the patient (see the survey in the Supplemental

Digital Content, http://links.adisonline.com/PBZ/A41). BHPMS staff

have no way of linking surveys received back to specific patients.

BHPMS=Behavioral Healthcare Performance Measurement System.



Descriptive Analysis

No item was dropped due to its assessment ofskewness and missing values. All items werepositively skewed; therefore, peakedness was notexamined due to a non-Normal distribution.[15]

Table II displays the mean and standard deviation(SD) data for the 28-item instrument. In general,the items were above the ‘I am neutral’ position,close to the ‘Agree’ statement. The mean rangewas from 3.10 to 3.94, in a disagree/agree five-point scale. The item on the instrument closer to amean equal to 4.0 (3.94) was related to staff be-liefs that clients could grow, change, and recover(item 7). The lowest mean (3.10) corresponds to aclient’s opportunity of talking with the doctor ortherapist from the community prior to discharge(item 18). The inter-item correlation analysisshowed all items scoring above the 0.30 cut-offscore, indicating a strong relationship betweenitems. No item was dropped due to low or ex-tremely high (collinearity) inter-item correlation.

Item Reduction

Data from a random sample of 8229 consumersin psychiatric hospitals were used for EFA. Mostconsumers (78%) were adults between 18 and54 years. Only 2%were adolescents (13–17 years).Sixty-five percent were male, 56% were White,and 61% were never married. The majority (78%)were receiving acute care services.

Before performing EFA, an examination of thesample appropriateness was conducted. TheKaiser-Meyer-Olkin measure of sampling adequacy was0.97, and Bartlett’s test of sphericity was significant[w2 (df = 351, N = 8229) = 131 320.42; p< 0.0001].Both tests indicated the sample and the matrixwere appropriate for EFA.[23] PCA revealed thatone, three, or eight domains should be retained(eigenvalues >1 = 3 [appendix A in the SDC],scree plot = 1 [appendix B in the SDC], 71% ofvariance explained = 8 [appendix A in the SDC]).The PA (appendix C in the SDC) concluded thatthree domains should be retained. A final deci-sion of retaining six domains was made based onconceptual clarity and interpretability.

Six items were dropped from further analyses(3, 10, 11, 12, 23, 24). While two items (3 and 10)

were related to medication, the other items eachfocused on a different construct: item 11 relatedto medical conditions being addressed, item 12related to agreement that hospitalization was ne-cessary, item 23 related to sensitivity to culturalbackground, and item 24 related to visitors. These

Table II. Descriptive statistics (mean and standard deviation) for

each Inpatient Consumer Survey item (n =34 878)

Item Mean (SD)

1. I am better able to deal with crisis 3.83 (1.28)

2. My symptoms are not bothering me as much 3.83 (1.31)

3. The medications I am taking help me control

symptoms that used to bother me

3.66 (1.47)

4. I do better in social situations 3.66 (1.33)

5. I deal more effectively with daily problems 3.71 (1.31)

6. I was treated with dignity and respect 3.86 (1.29)

7. Staff here believed that I could grow, change

and recover

3.94 (1.28)

8. I felt comfortable asking questions about my

treatment and medications

3.87 (1.29)

9. I was encouraged to use self-help/support groups 3.84 (1.33)

10. I was given information about how to manage my

medication side effects

3.46 (1.52)

11. My other medical conditions were treated 3.42 (1.62)

12. I felt this hospital stay was necessary 3.61 (1.46)

13. I felt free to complain without fear of retaliation 3.55 (1.45)

14. I felt safe to refuse medication or treatment during

my hospital stay

3.14 (1.63)

15. My complaints and grievances were addressed 3.26 (1.65)

16. I participated in planning my discharge 3.69 (1.46)

17. Both I and my doctor or therapist from the

community were actively involved in my hospital

treatment plan

3.22 (1.78)

18. I had an opportunity to talk with my doctor or

therapist from the community prior to discharge

3.10 (1.83)

19. The surroundings and atmosphere at the hospital

helped me get better

3.56 (1.39)

20. I felt I had enough privacy in the hospital 3.49 (1.39)

21. I felt safe while I was in the hospital 3.71 (1.34)

22. The hospital environment was clean and

comfortable

3.75 (1.32)

23. Staff were sensitive to my cultural background 3.46 (1.58)

24. My family and/or friends were able to visit me 3.47 (1.69)

25. I had a choice of treatment options 3.28 (1.55)

26. My contact with my doctor was helpful 3.64 (1.47)

27. My contact with nurses and therapists was helpful 3.80 (1.39)

28. If I had a choice of hospitals, I would still choose

this one

3.35 (1.57)

SD = standard deviation.

168 Ortiz & Schacht


items were retained in the instrument for furtheranalysis as they are important to patients andproviders. Item 28 was designated the anchoritem to measure overall satisfaction with care andwas not included in EFA although it was retainedfor validity analysis. These procedures resulted in21 items that accounted for 62.16% of the var-iance in the ICS scores after rotation, containingsix domains (outcome, rights, dignity, participation,environment, and empowerment).

When the assumption that domains will cor-relate is made, it is imperative to analyze thepattern matrix along with the structure matrix.Appendix D in the SDC summarizes the domainpattern (P) and structure (S) matrices rotated tothe promax criterion (k = 4), the communalities,and the percentage of variance post-rotation.Communalities were generally high. Large co-efficients were obtained for all the items in the sixdomains. As expected, the rotated domains werehighly correlated (appendix E in the SDC). Ap-pendix F in the SDC summarizes the general de-scriptive statistics for EFA.

Validity

Face and content validity of the instrumentwere performed during the original pilot studywhere consumers and stakeholders participatedin the creation and design of the initial 43-itemICS, and of the revised 28-items ICS.[10] The re-validation of the 28-item instrument in the cur-rent research was examined in different steps.Data from a sample of 8261 consumers in psy-chiatric hospitals were used to evaluate constructvalidity of the ICS. This sample included an in-dependent group of cases at discharge and allcases on annual review. Most consumers (78%)were adults (18–54 years old). Only 3% were ad-olescents (13–17 years). Sixty-five percent weremale, 57%wereWhite, and 61%were nevermarried.Forty-two percent of consumers were receivingacute care services.

From the proposed model (figure 1), a w2 =3658.414, df = 174, and p < 0.001 were obtainedindicating that the model can be estimated andtested.[20] In order for a proposed model to beidentified at least two conditions must be sat-

isfied.[21] The t-test rule requires that the numberof pieces of information in the model shall be atleast as large as the number of parameters to beestimated. In our model, we proposed 57 distinctparameters to be estimated and 85 pieces of in-formation; therefore, the first condition was met.The second condition was also met as all latentvariables in the model, and the error terms, had ascale assigned to each one of them.

Table III contains the goodness-of-fit indicesfor the model. The results of the CFA providedsupport for the ICS domains structure proposedthrough EFA; all indices met the minimum re-quirements.

Pearson’s correlation examined the instru-ment convergent validity. The correlation be-tween each item and the one anchor item (item28) resulted in significant correlations rangingfrom 0.31 to 0.58. Convergent validity was alsovalidated when the total ICS scores and the sumof the scores for all items except the anchor item(item 28) were highly correlated (0.99). The bestpredictors of overall high satisfaction (the anchoritem 28) were hospital environment (item 22)and contact with nurses and therapists (item 27)[r = 0.58].

Outcome

• Item 1• Item 2• Item 4• Item 5

Dignity


Rights• Item 13• Item 14• Item 15

Participation• Item 16• Item 17• Item 18

Environment


Empowerment• Item 25• Item 26• Item 27

Satisfaction with care received

Fig. 1. Proposed model for satisfaction with care received for in-patients in psychiatric hospitals.



Multiple regression analysis examined crite-rion-related (predictive) validity. The regressionmodel that included the scores for each item inthe six domains accounted for 50% of the var-iance in the anchor item. Table IV summarizesthe correlation coefficients from the multiple re-gression by domain and the overall anchor item(item 28). The models for each of the six domainsshowed significant criterion-related validity withoverall consumer satisfaction.

Reliability

Cronbach’s coefficient a examined the internalconsistency of the total ICS and the six domains.The sample used for this test was the one used forvalidity analysis. Table V summarizes the maincharacteristics of the final 21 items in the ICS thataligned into six domains as well as the overallperception of care. The reliability coefficients forthe ICS domains were between good tomoderate:0.87 for outcome, dignity, and environment; 0.83for empowerment; 0.81 for participation; and0.73 for rights. The coefficient a for the total ICSwas excellent (0.94). While only 3% of the samplewere adolescents, 230 responses were available foranalysis. To test appropriateness for the adolescentpopulation, Cronbach’s coefficient awas examined.

The a values by domain for the adolescents were0.84 for environment, 0.78 for empowerment,0.77 for outcome, 0.74 for dignity, 0.64 for par-ticipation, and 0.63 for rights. Floor effects (per-centage with minimum score) for the domainsranged from 2.0% to 5.4%, and ceiling effects (per-centage with maximum score) ranged from 4.0%to 16.4%. Only 2.6% of the total responses weremissing, ranging from 3.0% for the dignity do-main to 9.2% for the participation domain.

Discussion

This study psychometrically tested a consumersurvey designed for psychiatric hospitals that hasbeen significantly revised after an initial pilot test.The overall findings from this study provide evi-dence, as hypothesized, that the ICS is a reliablemeasure of consumer satisfaction in psychiatricinpatient settings. The original five domains (dig-nity, rights, environment, participation, and out-come) identified by EFA and CFA on the original43-item pilot ICS were replicated in the currentEFA and CFA. All items aligned to the samedomain as they did in the pilot ICS and a newdomain was revealed. The new domain is basedon rewording of several items in the original pilotICS. Items that did not load significantly withany domain in the current analysis also did notload in the pilot analysis. These items are main-tained in the ICS for their intrinsic value to con-sumers, as evidenced by the low rate of missing

Table III. Summary of fit indices from confirmatory factor analysis

Fit index Value

GFI 0.958

AGFI 0.944

NFI 0.965

RFI 0.958

IFI 0.966

TLI 0.959

CFI 0.966

PRATIO 0.829

PNFI 0.799

PCFI 0.801

RMSEA 0.049

AGFI =adjusted goodness-of-fit index; CFI = comparative fit index;

GFI = goodness-of-fit index; IFI = incremental fit index; NFI =normed

fit index; PCFI = parsimony comparative fit index; PNFI = parsimony

normed fit index; PRATIO = parsimony ratio; RFI = relative fit index;

RMSEA = root mean square error of approximation; TLI =Tucker-Lewis index.

Table IV. Results of multiple regression analysis examining ability

of the Inpatient Consumer Survey domains to predict overall con-

sumer satisfaction with carea

Domain Pearson’s

correlation coefficient

R2

Outcome 0.465 0.215

Dignity 0.566 0.320

Rights 0.517 0.267

Participation 0.406 0.165

Environment 0.664 0.440

Empowerment 0.642 0.412

Total ICS 0.707 0.499

a Item 28 was designated the anchor item to measure overall

satisfaction with care.

ICS = Inpatient Consumer Survey.

170 Ortiz & Schacht


data for all items. From the current EFA andCFA,it was found that the items in the dignity domain(items 6, 7, 8, and 9) have the highest mean scoresand that the domain itself has the highest corre-lation between rotated domains in concordancewith results from the initial pilot study testingEFA and CFA where hospitals scored better inthe dignity domain. Given the significant changesin the ICS instrument between the pilot and cur-rent versions, re-validation of the domains pro-vides credibility to the performance benchmarksthat can be established from these domains.

The 28-item ICS is used as a standardizedmea-sure of consumer satisfaction with psychiatricinpatient services. State and private psychiatrichospitals, as well as community psychiatric hos-pitals, distribute surveys each month to consumersprepared for discharge or annual review. Domainscores are actively used as performance measuresby hospitals in their accreditation process andquality improvement activity. Response rates havesignificantly improved through the efforts of hos-pitals to encourage consumers to provide activefeedback and to demonstrate how that feedbackis used by the hospital. Consumers’ involvementin the initial development of the items and theircontinued involvement at hospitals to use thesurvey results for program planning demonstrateadditional validity of the instrument and provideconsistency with recommendations in the liter-ature on generation of items based on the con-sumer’s point of view.[24]

During the late 1990s and the early 2000s, nu-merous consumer satisfaction instruments weredeveloped. Some of the instruments contain onlytwo or three dimensions of mental healthcare, or

the statements included were not derived fromconsumers’ feedback, which has been highly stressedby advocates; others use a ‘yes/no’ response format,which does not allow for dispersion of responsesat the positive end of the scale; others includednegatively worded items; and others only includedEFA and lack structure validation.[1,11,24-27] Manyinstruments also were targeted to service areas(community or hospital), greatly focusing onissues related to cost containment and speed ofservice. The ICS is distinctive as its focus stemmedfrom the need to evaluate consumers’ interactionsand perceptions about the care they received. TheICS also takes into consideration the principlesfor construction of psychiatric inpatient instru-ments such as inclusion of an unbalanced, broaderfive-point Likert scale, examination of constructvalidation, utilization of a large sample for externalvalidation, and provision of anonymous data, whichdiminishes distortion in consumer responses[28]

providing reliability in the use of the data forservice evaluation and planning. The constructsof dignity, rights, participation in treatment, en-vironment, empowerment, and personal outcomesaddress the fundamental desires of consumers andthe interpersonal aspect of treatment.

The ICS was initially developed to assist psy-chiatric hospitals with meeting accreditationrequirements of TJC, which requires healthcareorganizations to gather consumers’ opinions aboutservices and to use the information to improvecare. While TJC does not mandate a specific toolfor this activity, they do require that the tool betested and data assessed for reliability without aspecific protocol for either. Through this study,the ICS has been rigorously tested for domain

Table V. Statistics for the 21 items in the Inpatient Consumer Survey that aligned into six domains and the overall perception of care

Domain (no. of items) Min. Max. Mean (SD) Floor [%] Ceiling [%] Missing [%] a

Outcome (4) 4 20 14.6 (4.5) 2.0 12.8 3.3 0.87

Dignity (4) 4 20 15.1 (4.5) 2.0 16.4 3.0 0.87

Rights (3) 3 15 9.6 (3.8) 4.1 10.1 3.9 0.73

Participation (3) 3 15 9.1 (4.6) 5.4 13.8 9.2 0.81

Environment (4) 4 20 14.1 (4.7) 2.4 12.3 3.2 0.87

Empowerment (3) 3 15 10.1 (3.7) 2.8 14.6 4.6 0.83

Overall perception of care (21) 21 105 73.0 (21.0) 0.6 4.0 2.6 0.94

Max. =maximum; Min. =minimum; SD = standard deviation.



construction, validation of the domains throughCFA on an independent sample, and comparisonof pilot and final instruments, counteracting ar-guments about poorly designed inpatient satisfac-tion instruments that suffer from questionablevalidity and reliability.[25] The analysis has shownthe ICS to provide valid and reliable results andto focus on the specific concerns of consumers ofpsychiatric inpatient care. Scores by item indicatethat opportunity for improvement exists acrosshealthcare organizations. Further research in-cludes item-response examination by domain, scoreanalysis by demographic characteristics, andmethodof administration, healthcare organization, andunit specialty. Identification of differences in scoreswhen completing the survey at discharge or atannual review, the impact on scores if individualsare assisted in completion, and the impact onscores depending on the return method used willbe reviewed. Finally, the individual’s character-istics that predict overall satisfaction with care bydomain will be explored.

Limitations

The ICS has been validated for psychiatric in-patient settings, the majority of those being state-operated or supported psychiatric hospitals, al-though community hospitals and private hospitalsare also included. When respondents categorizedas annual review were removed, the sample had asmaller proportion of never married and a greaterproportion of voluntary admissions, which isdemographically similar to individuals who woulduse private psychiatric hospitals.

The survey focuses on the consumer’s evaluationof the hospital’s ability to meet his/her needs andto treat him/her with dignity and respect, and theconsumer’s ability to participate in his/her recov-ery. The survey does not include items to addressspeed or cost effectiveness of service; these aspectsmay be better addressed against external criteria.

Consumer satisfaction is one aspect whenmeasuring quality of mental healthcare. The ICSdoes not address specific treatments or adverseconditions in hospital environments. Fuller eval-uation of the quality of mental healthcare shouldinclude evaluation of adherence to best practices

for specific conditions (such as depression orschizophrenia), staffing competencies based onstandards, and occurrence of adverse events.

Response rates across hospitals averaged 51%.While this is a marked improvement over the sev-eral years that the ICS has been in use, the impactof non-respondents is unknown. Nevertheless, a51% response rate is exceptional for consumerself-administered, voluntary tools. The ICS isa self-administered and anonymous survey. Aconsumer may self-disclose his/her identity on theform but only the hospital would be able to linkthe survey data to identify non-respondents.

Conclusion

The overall findings from this study provideevidence that the ICS is a valid and reliable mea-sure of consumer satisfaction in psychiatric in-patient settings. This is critical to maintain theconfidence consumers, administrators, and othersplace in the integrity of data used for comparativepurposes. The analysis has also shown the ICS tofocus on the specific concerns of consumers ofpsychiatric inpatient care. Scores by item indicatethat opportunity for improvement exists acrosshealthcare organizations.

Acknowledgments

The study design, collection, analysis, and interpretationof the data, and the writing, review, and approval of themanuscript were fully funded by the BHPMS. The BHPMS isfunded by state and private psychiatric hospitals.

Glorimar Ortiz was responsible for the study protocol anddesign and the statistical analyses, and drafted the manuscript.Glorimar Ortiz also acts as the guarantor for the overall con-tent of this paper. Lucille Schacht contributed to theDiscussionand Conclusion sections, and revision of the final manuscript.

The authors declare no potential conflicts of interest.

References1. Barker DA, Orrell MW. The Psychiatric Care Satisfaction

Questionnaire: a reliability and validity study. Soc Psychi-atry Psychiatr Epidemiol 1999; 34: 111-6

2. Davies AR,Ware JE. Involving consumers in quality of careassessment. Health Aff 1988; 7: 33-48

3. Edgman-Levitan S, Clearly PD. What information do con-sumers want and need? Health Aff 1996; 15 (4): 42-56

4. Pellegrin KL, Stuart GW, Maree B, et al. A brief scale forassessing patients’ satisfaction with care in outpatientpsychiatric services. Psychiatr Serv 2001 Jun; 52 (6): 816-9

172 Ortiz & Schacht


5. Woodring S, PolomanoRC, Haagen BF, et al. Developmentand testing of a patient satisfaction measure for inpatientpsychiatric care. J Nurs Care Qual 2003 Dec 8; 19 (2): 137-48

6. Eisen S, Shaul J A, Clarridge B, et al. Development of aconsumer survey for behavioral health services. PsychiatrServ 1999 June; 50 (6): 793-8

7. The Joint Commission. About The Joint Comission [online].Available from URL: http://www.jointcommission.org/about_us/about_the_joint_commission_main.aspx [Accessed2011 Nov 2]

8. The Joint Commission on Accreditation of Healthcare Or-ganizations. ORYX� outcomes: the next evolution in ac-creditation. Oakbrook Terrace (IL): The Joint Commission,1997

9. Friedman MM. ORYX�: the next evolution in accred-itation. Home Healthc Nurse 1998 Apr; 16 (4): 236-9

10. Schacht L. NRI/MHSIP Inpatient Consumer Survey: resultsof pilot implementation. Alexandria (VA): National As-sociation of State Mental Health Program Directors Re-search Institute, Inc., 2001

11. Kolb SJ, Race KEH, Seibert JH. Psychometric evaluation ofan inpatient psychiatric care consumer satisfaction survey.J Behav Health Serv Res 2000 Feb; 27 (1): 75-86

12. SPSS Inc. SPSS� Statistics [computer program]. Version 17.Chicago (IL): SPSS Inc., 2008

13. IBM. Amos [computer program]. Version 18. Chicago (IL):IBM, 2009

14. National Quality Forum. NQF endorses mental health out-comemeasures [media release]. 2011 Jan 26 [online]. Availablefrom URL: http://www.qualityforum.org/News_And_Resources/Press_Releases/2011/NQF_Endorses_Mental_Health_Outcome_Measures.aspx [Accessed 2011 Jan 26]

15. MunroBH. Statisticalmethods for health care research. 5th rev.ed. Philadelphia (PA): Lippincott Williams & Wilkins, 2005

16. Fabrigar LR, Wegener DT, MacCallum RC, et al. Evalu-ating the use of exploratory factor analysis in psychologicalresearch. Psychol Methods 1999; 4 (3): 272-99

17. Hayton JC, Allen DG, Scarpello V. Factor retention deci-sions in exploratory factor analysis: a tutorial on parallelanalysis. Organ Res Methods 2004 Apr; 7 (2): 191-205

18. Kieffer KM. Orthogonal versus oblique factor rotation:a review of the literature regarding the pros and cons.Annual meeting of the Mid-South Educational ResearchAssociation; 1998 Nov 4; New Orleans (LA)

19. Hurley AE, Scandura TA, Schriesheim CA, et al. Ex-ploratory and confirmatory factor analysis: guidelines, is-sues, and alternatives. J Organiz Behav 1997 Feb 18; 18:667-83

20. Blunch NJ. Introduction to structural equation modellingusing SPSS and Amos. London: SAGE Publications Ltd,2008

21. Hu L, Bentler PM. Cutoff criteria for fit indexes in co-variance structure analysis: conventional criteria versusnew alternatives. Struct Equ Modeling 1999; 6 (1): 1-55

22. Nunnally JC. Psychometric theory. 2nd rev. ed. New York:McGraw-Hill, 1978

23. Ang RP, Huan VS. Academic expectations stress inventory:development, factor analysis, reliability, and validity. EducPsychol Meas 2006 Jun; 66 (3): 522-39

24. Boyer L, Baumstarck-Barrau K, Cano N, et al. Assessmentof psychiatric inpatient satisfaction: a systematic review ofself-reported instruments. Eur Psychiat 2009; 24: 540-9

25. Meehan T, Bergen H, Stedman T. Monitoring consumersatisfaction with inpatient service delivery: the InpatientEvaluation of Service Questionnaire. Aust NZ J Psychiatry2002; 36: 807-11

26. Eisen SV, Wilcox M, Idiculla T, et al. Assessing consumerperception of inpatient psychiatric treatment. Jt CommJ Qual Improv 2002 Sep; 28 (9): 510-26

27. Holcomb WR, Adams NA, Ponder HM, et al. The devel-opment and construct validation of a consumer satisfactionquestionnaire for psychiatric inpatients. Eval ProgramPlann 1989; 12: 189-94

28. Lebow J. Consumer satisfaction with mental health treat-ment. Psychol Bull 1982; 91 (2): 244-59

Correspondence: Mrs Glorimar Ortiz, 3141 Fairview ParkDr., Suite 650, Falls Church, VA 22042, USA.E-mail: [email protected]



Psychometric Evaluation of an Inpatient Consumer Survey Measuring Satisfaction with Psychiatric Care

Documents

Transcript of Psychometric Evaluation of an Inpatient Consumer Survey Measuring Satisfaction with Psychiatric Care