Client Satisfaction Evaluations -...

38
1 Workbook 6 Clent Satisfaction Evaluations WHO/MSD/MSB 00.2g Workbook 6 Client Satisfaction Evaluations

Transcript of Client Satisfaction Evaluations -...

Page 1: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

1Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Workbook 6

ClientSatisfactionEvaluations

Page 2: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

2 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

WHOWorld Health Organization

UNDCPUnited Nations International Drug Control Programme

EMCDDA European Monitoring Center on Drugs and Drug Addiction

World Health Organization, 2000c

This document is not a formal publication of the World Health Organization (WHO) and all rights are reserved by theOrganization. The document may, however, be freely reviewed, abstracted, reproduced and translated, in part or inwhole but not for sale nor for use in conjunction with commercial purposes. The views expressed in documents bynamed authors are solely the responsibility of those authors.

Page 3: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

3Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

(Canada) edited the workbook se-ries in earlier stages. JoAnne Epping-Jordan (Switzerland) wrote furthertext modifications and edited theworkbook series in later stages.Munira Lalji (WHO, SubstanceAbuse Department) and JenniferHillebrand (WHO, Substance AbuseDepartment) also edited the work-book series in later stages. MaristelaMonteiro (WHO, Substance AbuseDepartment) provided editorial inputthroughout the development of thisworkbook.

The World Health Organizationgratefully acknowledges the contri-butions of the numerous individualsinvolved in the preparation of thisworkbook series, including the ex-perts who provided useful com-ments throughout its preparation forthe Substance Abuse Department,directed by Dr. Mary Jansen. Finan-cial assistance was provided byUNDCP/EMCDDA/Swiss FederalOffice of Public Health. Cam Wild(Canada) wrote the original text forthis workbook and Brian Rush

Acknowledgements

Page 4: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

4 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Page 5: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

5Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Table of contents

Overview of workbook series 6

What is a client satisfaction evaluation? 7

Why do a client satisfaction evaluation? 7

How to do a client satisfaction evaluation? 8

Conclusion and practical recomendation 17

Comments about case examples 19

Case examples of client satisfaction evaluationPart A: An evaluation of satisfaction with a statedrinker driver treatment program 21

Part B: Client satisfaction with residential substancetreatment programmes 26

Part C: The case of community methadonstreatment programs 32

Page 6: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

6 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Overview ofworkbook seriesThis workbook is part of a series in-tended to educate programme plan-ners, managers, staff and other deci-sion-makers about the evaluation ofservices and systems for the treatmentof psychoactive substance use disor-ders. The objective of the series is toenhance their capacity for carrying outevaluation activities. The broader goalof the workbooks is to enhance treat-ment efficiency and cost-effectivenessusing the information that comes fromthese evaluation activities.

This workbook discusses the assess-ment of client satisfaction. It focuseson:

l reasons for assessing client satis-faction

l the use of client satisfaction mea-sures for programme improvement

l measures of client satisfaction

Introductory WorkbookFramework Workbook

Foundation WorkbooksWorkbook 1: Planning EvaluationsWorkbook 2: Implementing Evaluations

Specialised WorkbooksWorkbook 3: Needs Assessment EvaluationsWorkbook 4: Process EvaluationsWorkbook 5: Cost EvaluationsWorkbook 6: Client Satisfaction EvaluationsWorkbook 7: Outcome EvaluationsWorkbook 8: Economic Evaluations

Page 7: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

7Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

What is a clientsatisfaction evaluation?

Client satisfaction evaluations arean excellent opportunity to involveclients or patients in the process ofevaluating your programme.

Client satisfaction evaluations can address

1 . the reliability of services, or the assur-ance that services are provided in aconsistent and dependable manner;

2 . the responsiveness of services or thewillingness of providers to meet cli-ents/customer needs;

3 . the courtesy of providers; and

4 . the security of services, including thesecurity of records.

Specific questions may assess clients’views about :

l the physical setting of services

l the helpfulness of support staff

l information resources

l the competence of counsellors

l the costs of service

l the relevance of services to their needs

l the accessibility of services

l waiting times for service components

l frequency of appointments

l time spent with counsellor

l the ‘humanness’ of services

l the effectiveness of services in ame-liorating their problems

Client satisfaction occupies an ‘interme-diate’ step in establishing a healthy cul-ture for evaluation within a programmeor a setting. It often follows processevaluation and cost analysis, and pre-cedes outcome and economic evalua-tions. Accordingly, measures of clientsatisfaction lie somewhere between ‘pro-cess’ and ‘outcome’ measures. When theconcern is with the extent to which cli-ents are satisfied with the context, pro-cesses, and perhaps the costs of a treat-ment service or network, the relevantmeasures of satisfaction can be viewedas process measures. However, when theconcern is with the extent to which cli-ents view the programme as having beenhelpful in resolving their problems, cli-ent satisfaction becomes a proxy out-come measure.

Page 8: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

8 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

The assessment of client satisfaction addsan important ‘consumer’perspective toevaluations of PSU treatment services andsystems. Client satisfaction evaluationscan be viewed as an opportunity to‘consult’with clients about their experi-ences in your programme. Client satisfac-tion surveys may provide the only meansfor clients to express concerns about theservices received, and to express theirviews about new services that are needed.

Client satisfaction ratings have beencriticised as indicators of the quality ofhuman services because they may reflectunrealistic expectations. While this criti-cism may be valid in some instances, re-

Why do a clientsatisfaction evaluation?

Clientsatisfactionsurveys mayprovide theonly means forclients toexpressconcernsabout theservicesreceived.�

search with clients of mental health ser-vices suggests that they can effectivelydiscriminate between services that aredifferent in quality (Lebour, 1983;Sheppard, 1993). It is, however, impor-tant to recognise that evidence of posi-tive client satisfaction is not, in itself,sufficient to establish the effectivenessor accessibility of treatment. Clients withno base for comparison may be satisfiedwith services that are ‘ineffective’as de-termined by more objective outcomeevaluations. On the other hand, clientsmay be displeased with services thatachieve the objective of reducing theirPSU but employ rigid or authoritarianapproaches.

Client satisfaction with treatment pro-cesses may both influence, and be influ-enced by, treatment outcomes. Clientswho are not satisfied with a service mayhave worse outcomes than others becausethey miss more appointments, leaveagainst advice or fail to follow throughon treatment plans. On the other hand,clients who do not do well after treatmentmay have less than favourable attitudestowards a treatment service, even if it wasof high quality by other criteria. In prac-tice, these mutual influences may be dif-ficult to disentangle. It is worth keeping

in mind that satisfaction with the treat-ment processes, treatment compliance,and positive treatment outcomes are in-ter-related.

Ratings of different dimensions of satis-faction have been highly correlated insome studies, and scores on these dimen-sions have been added to yield overallsatisfaction ratings. However, responsesto specific items are of interest to ser-vice providers who want to find out howa particular aspect of the service couldbe improved.

It is worthkeeping inmind thatsatisfactionwith thetreatmentprocesses,treatmentcompliance,and positivetreatmentoutcomes areinter-related.

�... evidence ofpositive clientsatisfaction isnot, in itself,sufficient toestablish theeffectiveness oftreatment.�

Page 9: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

9Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

How to do a clientsatisfaction evaluation?The most common method for assessingclient satisfaction is with self-adminis-tered questionnaires. These may be givento clients as they enter or leave services,or at various times in between. They canalso be administered at some point aftertreatment has been completed, when theoutcomes of treatment are more clear tothe client. Client satisfaction question-naires can be completed at the time theyare distributed, or at a later date selectedby the client or program personnel.Stamped, return envelopes can be pro-vided if questionnaires are to be returnedby mail. Satisfaction questionnaires alsocan be mailed to former clients withstamped, return envelopes. A cover lettershould explain why the questions are be-ing asked and how the information willbe used. The cover letter should also in-dicate if individual replies will be consid-ered confidential or anonymous, and whatsteps will be taken to ensure that this isthe case. For ethical reasons, risks to cli-ents should be made clear. It should bestated that their responses will not in anyway affect present or future treatment.

Programme managers typically want thequestionnaire to identify the respondentso that they can follow-up with these in-dividuals who express concerns about theservices received. If this is the case, clearprovisions for confidentiality must bemade, including, for example, removal ofthe information identifying the client prior

to data analysis by computer or other tabu-lar means. For more information aboutthese ethical issues, see Workbook 2 ofthis series, Step 1A, entitled ‘ManageEthical Issues.’

Client satisfaction also can be assessedin face-to-face or telephone interviewsor focus groups. These strategies aremore expensive than self-completedquestionnaires. If interviews or focusgroups are used, it is preferable to havethem conducted by someone who is notconnected directly with the service. Thismay be an independent evaluator, vol-unteers or former clients themselvestrained to take on this role. If interviewsor focus groups must be done by a man-ager or staff member, it is best not to havethe individual’s principal therapist askabout client satisfaction because clientsmay be reluctant to comment negativelyabout their treatment directly to theirtherapist. Interviews may be highlystructured, perhaps guiding the clientthrough the same type of questionnaireused on a self-administered basis in othersituations. Other interviews, and cer-tainly focus groups, will be much lessstructured and the resulting informationwill be analysed qualitatively. Workbook1 provides guidance for conducting fo-cus group and semi-structured/unstruc-tured interviews. Workbook 2 offers ad-vice on analysing the resultinginformation.

Client satisfaction surveys are most use-ful when they are designed to meet spe-cific objectives and when they use appro-priate methods and measures. This

involves choices of sampling procedures,timing, cultural acceptability, and sensi-tivity of the questions to various levels ofsatisfaction.

The design and conduct of client satisfaction surveys

Page 10: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

10 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Choosing samples of clients

Your strategy for selecting clients for a satis-faction survey can influence the kinds of re-sults you obtain. If the surveys are limited toclients who complete treatment, the resultswill probably differ from those obtained insurveys that include people who havedropped out of the programme. There areno right or wrong ways to choose samples inclient satisfaction surveys. However, it is im-portant that your sample be consistent withthe evaluation objectives. If the objective isto learn about client satisfaction among thosewho complete treatment then there will beno need to involve treatment drop-outs.However, if the aim is to find how, in gen-eral, clients feel about the programme, a rep-

resentative sample of all clients completingthe intake process would be more appropri-ate. Regardless of the sample chosen, youmust be sure to clearly describe the samplein subsequent reports. Limitations to thegeneralizability of results must be stated. Forexample, are your results biased due to theexclusion of early drop outs?

Once you have decided which types of cli-ents will be involved in satisfaction surveys,you have a number of options for choosingparticular clients, including a random or sys-tematic sample. These and other options forsampling are discussed in Workbook 2.

There are noright or wrongways to choosesamples in clientsatisfactionsurveys.However, it isimportant thatyour sample beconsistent withthe evaluationobjectives.

The timing of client satisfaction surveyscan influence your results. Clients withpositive views during or immediately fol-lowing treatment may change their mindsif they later relapse. On the other hand, cli-ents may gain a greater appreciation of ser-vices as their value becomes evident in anincreasing number of real life situations.There is no ‘best’ timing for these surveys,except to ensure consistency with the ob-jectives of the evaluation. If the objectiveis to find out what clients feel at the timeof discharge, then ask clients to complete

There is no�best� timing forthese surveays,except to ensureconsistency withthe objectives ofthe evaluation.

Choosing samples of clients

a satisfaction questionnaire as they areabout to leave. However, if the aim is tofind out if clients are satisfied as part of anoutcome evaluation, wait until some pe-riod of time has passed before askingformer clients to complete a satisfactionquestionnaire. The timing of surveysshould be clearly indicated in reports andany associated biases should be discussed.If, for example, clients complete satisfac-tion questionnaires following an emotional‘graduation’ ceremony this could bias at-titudes in favour of the programme.

Cultures differ with respect to expecta-tions of feedback on public and privateservices. In jurisdictions where ‘consum-erism’ is firmly established, frank verbalor written feedback may be freely given.However, direct negative feedback insome cultures may be considered impo-lite and complaints may only be sharedwith intimate acquaintances. Direct andchallenging questions also may be cultur-

Culture sensituvity

ally inappropriate (NIDA, 1993). Expe-riences with (and attitudes toward) the useof questionnaires, interviews, focusgroups and other methods of inquiry alsodiffer between cultures. Methods for so-liciting client feedback must take intoaccount the prevailing cultural norms andseek to ensure the use of appropriatemethods that assess client beliefs andopinions.

Page 11: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

11Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Measures validated in one culture maynot be appropriate in others. Simpletranslation of questionnaire items doesnot guarantee that the items will havethe same meaning across cultures(Attkisson and Greenfield, 1994). Con-siderable effort may be required to gen-erate new, culturally appropriate ques-tions. Clients, or people advocating ontheir behalf, should be involved in thisprocess of questionnaire design to en-

sure that the measures will provide a validindication of client satisfaction.

Some groups of clients may also find particu-lar methods for assessing client satisfactionmore acceptable than others. For example,those with poor cognitive or reading skills mayprefer personal interviews over a written ques-tionnaire. However, clients who are shy orhave low self-esteem may prefer question-naires over interviews.

Many satisfaction surveys of clients ofhealth and social services have shownhigh levels of satisfaction partly becausethey have used insensitive measures(Ruggeri, 1994). An example would beusing questionnaire items that only havetwo response options (satisfied/not satis-fied). Such items tend to invite a ‘satis-

...if satisfactionis rated on afive-point scale,the proportionsof clients whoare �verysatisfied�,�somewhatsatisfied� or�neutral� can bebetterdiscriminated...

Sensitivity to different levels of satisfaction

fied’ response, even from those who areneutral of even mildly dissatisfied. How-ever, if satisfaction is rated on a five-pointscale, the proportions of clients who are‘very satisfied’, ‘somewhat satisfied’ or‘neutral’ can be better discriminated, ascan the proportions who are ‘somewhat’or ‘very’ unsatisfied .

Clients of human service agencies have atendency to be grateful for the attention theyreceive, and to be reluctant to criticise inthe event that this leads to negative conse-quences. Clients with low self-esteem, orwho are conscious of status differencesbetween themselves and service providers,may feel especially obliged to show thatthey are grateful and satisfied with the ser-vices provided. These tendencies can beovercome if clients are assured that theirhonest feedback is being sought and thatthere will be no consequences for those whocriticise the services in question. This canbe made clear in verbal or written instruc-tions for completing satisfaction question-naires or participating in interviews or fo-cus groups. Confidentiality of the results

Seeking out expressions of dissatisfaction

Clients ... mayfeel especiallyobliged to showthat they aregrateful andsatisfied with theservicesprovided.

should be assured as strongly as possible.It is desirable to actively seek out sourcesof discontentment by asking the followingkinds of questions:

l Are there any parts of the programmethat you liked more than others?

l Have you any suggestions for ways inwhich the programme can be improved?

Also, look for behavioural indicators ofdissatisfaction, for example, high drop-outor no-show rates within specificprogrammes, or for specific counsellors.While many factors may contribute to lowparticipation, low client satisfaction maybe involved.

Page 12: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

12 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

When choosing a questionnaire for yourevaluation, you first need to considerwhether all dimensions of client satis-faction are relevant to the service com-ponents being evaluated. It is often thecase that one treatment agency providesdifferent types of services and activities.You will have to decide whether yourclient satisfaction questionnaire willprovide feedback about individual ser-vice components, or whether you willfocus on a more global level ofprogramme participation. This will bean issue to resolve in the assessment ofsatisfaction with services receivedacross a large network of agencies. Ifthe intention is to use the resulting in-formation to suggest highly specific ar-eas for service or system enhancement,you may need to customise your selec-tion of client satisfaction measures to fitparticular service or system compo-nents. This may ultimately involve achoice between a standardised, globalmeasure of satisfaction available frompublished literature (see below), andquestionnaires tailored to your specificinformation needs.

If you are going to use a structured, self-administered questionnaire, you may se-lect one from the published literature.Such measures in the public domain willlikely have data available on reliabilityand validity in a particular setting. Thisis a big advantage, but must be consid-ered in light of cultural variations be-tween the culture in which the question-naire was validated and the culture inwhich you intend to use it now . In addi-tion, standardised questionnaires may betoo general to give you the kind of de-tailed feedback you need for making im-provements to specific parts of the pro-gram. Feedback unique to your programcan be derived from a specially-tailoredquestionnaire, although issues of reliabil-ity and validity will be of concern. Open-

ended questions can also be added to aself- administered questionnaire and thenanalysed qualitatively.

A questionnaire which can be used to as-sess client satisfaction is the Client Satis-faction Questionnaire (CSQ-8). This is awidely used instrument with publisheddata on reliability and validity (Greenfieldand Attkisson, 1989). The instrument isavailable in several languages, includingEnglish, Spanish, Dutch and French (deBrey, 1983; Roberts et al., 1984; Sabourinet al., 1987). Case examples of evalua-tions that used the CSQ-8 also are re-ported at the end of this workbook.

Workbook 1, Appendix 2 also contains fourother examples of questionnaires that canbe used to assess client satisfaction. Thereare no data on the reliability and validity ofthese other instruments. However, they maybe helpful in your situation or stimulate ideasfor the development of a questionnaireunique to your needs.

A report from the National Institute onDrug Abuse (1993) entitled ‘How Goodis Your Drug Abuse TreatmentProgram?’contains a series of client satis-faction questions used in an AIDS RiskReduction Project. Two other measures ofclient satisfaction appropriate for PSU ser-vices are the Service Satisfaction Scale(SSS-30) (Attkisson and Greenfield,1984), and the Verona Service Satisfac-tion Scale (VSSS) (Tansella, 1991). TheSSS-30 is a 30-item multi-dimensionalscale developed on the basis of experiencewith the Client Satisfaction Scale. The firstcase example at the end of this workbook(Part A: by Thomas Greenfield) describesthe SSS-30 in greater detail. The VSSS isan 82-item scale which covers seven di-mensions — overall satisfaction, profes-sional skills and behaviours, information,access, efficacy of interventions and rela-tive improvement.

Established questionnaires for assessing client satisfaction

Page 13: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

13Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

A good starting place for the develop-ment of a new client satisfaction ques-tionnaire that is tailored to your indi-vidual service or treatment system willbe your programme logic model and ac-companying written descriptions ofyour programme (see Workbook 1).These will identify the main compo-nents, activities and treatment processesfor which client satisfaction ratingscould be developed. In addition, clientscould be asked to rate their satisfactionwith the staff, comprehensiveness of theservices provided and aspects of thephysical environment. It would also beuseful to convene small groups of cur-rent and former clients to explore is-sues most relevant to their needs. Thesegroups may be helpful in testing ideasfor questionnaire items and response

Developing your own client satisfaction questionnaires

options. The instruments contained inWorkbook 1, Appendix 2 also will beuseful.

To help validate measures of client satis-faction, the ratings can be compared withverbal reports or satisfaction ratings fromfamily members or others that are familiarwith the services received. You can alsocompare the results using your new ques-tionnaire with the results of an instrumentlike the CSQ-8 completed by the samepeople. Client satisfaction ratings can alsobe compared with actual behaviours thatsignify satisfaction with services. Com-parisons could be made, for example, be-tween client satisfaction ratings and theirrecord of keeping appointments, complet-ing treatment, or returning for further treat-ment following a relapse.

Once reliable client satisfaction mea-sures are available, they can be usedfor routine or periodic ‘check-ups’ onthe quality of services from the clients’perspective. They also can be used toassess client reactions to changes inservice delivery being implemented.

Using client satisfaction measures during times of change inservice delivery

For example, changes may be plannedto increase the efficiency of a servicebut there are concerns that these couldlead to decreased client satisfaction.Measures of satisfaction taken before orafter the changes are introduced willshow if this has been the case.

It is possible to assess client satisfactionwith services received across a networkof programmes, rather than focusing onthe client’s experience with only one ser-vice provider. Not all clients will haveexperience with other services in thetreatment network. However, clients ofall services may have useful perspectiveson system-wide issues. For example,

Measuring client satisfaction in evaluations across two ormore agencies

[Clients] couldbe asked if theyfeel satisfiedwith theinformation thatis available onthe range ofservices in thecommunity.

they could be asked if they feel satisfiedwith the information that is available on therange of services in the community. Theycould also be asked to rate their satisfac-tion with recommendations for referralgiven the options that were presented. Didthey like this referral? Was it too far awayfor them? Do they feel satisfied with beingreferred to a residential service when they

Page 14: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

14 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

might have gone to a day treatment or out-patient service (and vice versa)? A reviewof a logic model for the treatment systemmay suggest other topics to be included inclient satisfaction surveys. For example,waiting times for moving from one ser-vice component to another may be of par-ticular concern. The issue of duplicationof services is also important to explore,for example, whether agencies in the sys-tem duplicate the collection of assessmentinformation when the client moves fromone service to another.

Clients who have experienced two or moreservices in a network may have valuableperspectives on the degree to which theseservices are co-ordinated. Sample ques-

tions concerning client satisfaction withinter-service co-ordination are:

l How satisfied are you with the way that(name of both services) exchangedtreatment information about your prob-lems?

l How satisfied are you with the infor-mation that (name of both services) pro-vided to you about each other’s treat-ment programmes?

l How satisfied are you with the waysthe treatment staff of (name of both ser-vices) worked together to help you withyour problems?

l Based on your experience, how well do(name of both services) work together?

It�s your turn

Put the information from this workbookto use for your own organisation or treat-ment network. Complete these exercisesbelow.

Remember to use the information fromWorkbooks 1 and 2 to help you completean evaluation plan. Review that informa-tion now, if you have not already done so.

Exercise 1

Think about your treatment programmeor local treatment network. List five gen-eral areas in which you want to know theviews of clients or patients.

Example:

What do clients think about the helpful-ness of our clinicians?

1)2)3)4)5)

Exercise 2

Using the information provided in thisworkbook about how to design and con-duct a client satisfaction evaluation, makethe following decisions:

l Decide what modality you will use tocollect the data (questionnaires, inter-views, focus groups)

l Choose a sampling procedure forchoosing clients to survey

l Decide the timing of the evaluation

Page 15: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

15Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

l Develop a procedure for ensuring cli-ents’ confidentiality and promotingtheir honesty in answering questions

l Decide who will help you administer thequestionnaires/interviews/focus groups.

Example (from above):

l Data will be collected using a self-re-port questionnaire.

l All clients checking in for appoint-ments during the week of December10th will be handed the survey to com-plete while waiting for their appoint-ments.

l Data will be collected over a one weekperiod of time only, from 10-15 De-cember.

l Clients will be given envelopes inwhich to place their completed ques-tionnaires before returning them to thecollection box. The following state-ment will appear at the top of the ques-tionnaire:

‘Please help us improve our program-me by answering some questions aboutthe services you have received. We areinterested in your honest opinion,whether it is positive or negative. Toensure your confidentiality, please donot write your name on this form.When you are finished, place the formin the envelope (provided) and seal itclosed, then place it in the collectionbox in the waiting area.’

l Because the questionnaire assesses cli-ent satisfaction with staff, it is not fea-sible for staff to be involved with dis-tributing or collecting questionnaires.A outside research assistant will behired to hand out the introductory let-ters, consent forms, and questionnaires.The assistant also will remove the ques-tionnaires from the collection box andkeep them in a safe place to ensure theirconfidentiality from the staff.

Now it’s your turn. Follow the same pro-cedure for your evaluation questions.

You will need to prepare an introductoryletter and consent form that explains thepurpose of your study. Review Section 1Aof Workbook 2, entitled Manage Ethi-cal Issues, for more information about theimportant topic of participants’ rights inevaluation research.

In general, all participants should be askedpermission ahead of time before beingenrolled in the study. When you do this,your should explain the purpose, nature,and time involved in their participation.No person should be forced or coerced toparticipate in the study.

A standard practice is to have each par-ticipant sign a consent form, which:

Exercise 3

l describes the purpose and methods ofthe study

l explains what they will need to do ifthey participate

l explains that participation is voluntary

Note that an ethical committee maywaive the requirement of a signed con-sent form if the research contains mini-mal risk. In these cases, researchers stillneed to provide full information to par-ticipants. A consent form is included inthe following example for the sake ofcompleteness.

Example (from above):

Page 16: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

16 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Introductory Letter:

We are asking your help in improving ourprogramme by filling out a 2 page ques-tionnaire about the services you have re-ceived here. The questions will ask aboutyour views regarding our staff members.They will take about 10 minutes to com-plete. All information that you provideus will remain strictly private and confi-dential.

If you agree to participate, please read andsign the consent form (attached) and re-turn it to the research assistant who gaveyou this packet when you arrived. Thankyou for your time.

Sincerely,Dr. XDirector, Treatment Programme

Consent Form

You agree to participate in a client sur-vey of satisfaction with our staff. You will

complete a 2 page questionnaire today,which will take about 10 minutes to com-plete. Your participation is completelyvoluntary. You can refuse to answer anyquestions and/or withdraw from the studyat any time without a problem to you oryour treatment here. All your responseswill remain strictly confidential:programme staff members will not haveaccess to your responses , your name willnot appear on your questionnaire, andyour responses will not be linked to youridentity at any time.

I have read the information above andagree to participate.

Signature:Date:

Now it’s your turn. Using the exampleabove, and the additional informationprovided in Workbook 2, section 1A,write your own introductory letter andconsent form.

Run a pilot test of your evaluation mea-surement and procedures to ensure thateverything runs smoothly. Review sectionIC of Workbook 2 entitled Conduct aPilot Test for specific information abouthow to do this. In general, pilot tests as-sess these questions:

l Do the questions provide useful infor-mation?

l Can the questions be administered prop-erly? For example, is it too long or toocomplicated to be filled out properly?

l Can the information be easily managedby people responsible for compiling thedata?

l Does other information need to be col-lected?

Exercise 4

Example (from above):

A pilot test will be run during one clinicday: 3 November. During this day, 10-15patients checking in will be given thequestionnaire. Afterwards, their responseswill be examined to determine whetherthey seemed to understand the questionsand were answering honestly. All personsinvolved with distributing the forms andcompiling the data will be interviewed todetermine their views on any improve-ments that could be made in the processand/or to the forms.

Now it’s your turn. Write down how youwill pilot test your evaluation study. Don’tforget to review Workbook 2 first!

Page 17: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

17Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Conclusion and practicalrecommendation

In this workbook, we have outlined thebasic principles and practices in theevaluation of client satisfaction withPSU services and systems. After com-pleting your evaluation, you want toensure that your results are put to prac-tical use. One way is to report your re-sults in written form (described inWorkbook 2, Step 4). It is equally im-portant, however, to explore what theresults mean for your programme. Dochanges need to happen? If so, what isthe best way to accomplish this?

Return to the expected user(s) of theevaluation with specific recommenda-tions based on your results. List your rec-ommendations, link them logically toyour results, and suggest a period forimplementation of changes. The ex-amples below illustrate how to managetwo different kinds of results using thistechnique.

Unfavourable findings

Based on the finding that over 1/4 of cli-ents were ‘very dissatisfied’or ‘somewhatdissatisfied’with the friendliness of theclinical staff, we recommend that the

programme institute a 2 hour client satis-faction training workshop for all cliniciansto attend. The workshop could happen inMarch, which traditionally is a low-cen-sus month for the programme, and be runby Dr. Z, who is well-liked and respectedby the staff.

Favourable findings

The results indicate that clients are ‘verysatisfied’ overall with the helpfulness ofthe clinical staff. Therefore, we recom-mend that the composition of the clinicalstaff remain unchanged, and that thesefavourable findings are publicly acknowl-edged at the next programme-wide staffmeeting.

Remember, clients provide an invalu-able perspective on the success of yourprogramme. It is important to use theinformation that they provide to im-prove treatment services. Through care-ful examination of your results, you candevelop helpful recommendations foryour programme. In this way, you cantake important steps to create a ‘healthyculture for evaluation’ within yourorganisation.

Page 18: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

18 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

References

Addiction Research Foundation. A Directoryof Outcome Measures. Toronto: AddictionResearch Foundation, undated.

Attkisson, C.C., & Greenfield, T.K. The Cli-ent Satisfaction Questionnaire-8 and the Ser-vice Satisfaction Questionnaire-30. In:Maruish, M. (ed.). Psychological testing:treatment planning and outcome assessment.Lawrence Erlbaum Associates: San Fran-cisco, 1994: 402-420.

de Brey, H. A cross-national validation of theClient Satisfaction Questionnaire: The Dutchexperience. Evaluation and Programme Plan-ning, 1983, 6: 395-400.

Greenfield, T.K., & Attkisson, C.C. Steps to-ward a multifactorial satisfaction scale forprimary care and mental health services.Evaluation and Programme Planning, 1989,12:271-278.

Kurtz, L.F. Measuring member satisfactionwith a self-help association. Evaluation andProgramme Planning, 1990, 13: 119-124.

Larsen, D.L. Enhancing client utilization ofcommunity health services. Dissertation Ab-stracts International, 39, 4041B. (UniversityMicrofilms No. 7904220), 1979.

Larsen, D.L., Attkisson, C.C., Hargreaves,W.A., & Nguyen, T.D. Assessment of client/patient satisfaction: Development of a gen-eral scale. Evaluation and Programme Plan-ning, 1979, 2: 197-207.

Lebour, J.L. Similarities and differences be-tween mental health and health care evalua-tion studies assessing consumer satisfaction.Evaluation and Programme Planning, 1983,6: 237-245.

Lettieri, D.J., Nelson, J.E., & Sayers, M.A.(Eds.) NIAAA treatment handbook series:Alcoholism treatment assessment researchinstruments. Rockville, MA: National Insti-tute on Alcohol Abuse and Alcoholism, 1985.

Levois, M., Nguyen, T.D., & Attkisson, C.C.Artifact in client satisfaction assessment:Experience in community mental health

settings.Evaluation and Program- mePlan-ning, 1981, 4: 139-150.

Moos, R.H., & Finney, J.W. Alcoholismprogramme evaluations: The treatment do-main. In D.J. Lettieri (Ed.), Research strate-gies in alcoholism treatment assessment(pp.31-51). New York: Haworth Press, 1988.

National Institute on Drug Abuse. How goodis your drug abuse program? ResourceManual. National Institute on Drug Abuse,Rockville, MD, 1993.

Nguyen, T.D., Attkisson, C.C., & Stegner,B.L. Assessment of patient satisfaction: De-velopment and refinement of a Service Evalu-ation Questionnaire. Evaluation andProgramme Planning, 1983, 6: 299-314.

Roberts, R., Attkisson, C., & Mendias, R.M.Assessing the Client Satisfaction Question-naire in English and Spanish. Hispanic Jour-nal of Behavioural Science, 1984, 6: 385-396.

Ruggeri, M. Patientsl ‘ and relatives’ satis-faction with psychiatric services; the state ofthe art of its measurement. Social PsychiatryPsychiatric Epidemiology, 1994, 29: 212-227.

Sabourin, S., Gendreau, P., & Frerette, L. Leneveau de satisfaction des cas d’abandon dansun service universitaire de psychologie. Ca-nadian Journal of Behavioural Science, 1987,19: 314-323.

Sheppard, M. Client satisfaction, extendedintervention and interpersonal skills in com-munity mental health. Journal of AdvancedNursing, 1993, 18: 246-259.

Tansella, M. (ed.) Community-based psychia-try: Long-term patterns of care in SouthVerona. Psychological Medicine Supplement,19, Cambridge University Press, Cambridge,1993.

Zwick, R.J. The effect of pretherapy orienta-tion on client knowledge about therapy, im-provement in therapy, attendance patterns,and satisfaction with services. Masters Ab-stracts, 1982, 20: 307. (University MicrofilmsNo. 13-18082).

Page 19: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

19Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Comments aboutcase examples

Each of the following case examples de-scribes evaluations comparing clientsatisfaction across sites. The first evalu-ation (Part A) presents a client satisfac-tion evaluation of a state-sponsoredtreatment, primarily for people con-victed of drunk driving. Satisfaction wasmeasured using two scales discussed inthis workbook: the CSQ-8 and SSS-30.Whereas the CSQ-8 provides a singlesatisfaction score, the SSS-30 assessesseveral aspects of client satisfaction.Results were used to guide site proce-dural improvements.

The second evaluation (Part B) is a goodexample of how client satisfaction evalu-ations can be completed with limited re-sources. In this case, a mental health in-tern wanted to examine client satisfactionacross several residential PSU treatmentprogrammes. With limited assistance, hewas able to plan and successfully imple-ment his evaluation. Differences in satis-faction across sites were detected by themultidimensional SSS-30.

The third evaluation examined client sat-isfaction across three community metha-done treatment sites in Australia. Evalua-tors used the CSQ-8 and two qualitative,open-ended questions to assess satisfac-

tion. They found significant differences insatisfaction across sites. These differenceswere given to clinic managers to makeprocedural improvements.

While each of these cases generated use-ful information about client satisfaction,it is noteworthy that none attempted toprovide information about client outcomeor treatment effectiveness. As describedearlier in this workbook, measurement ofclient satisfaction is useful yet distinct frommeasurement of client outcome or treat-ment effectiveness. On occasion, clientscan be satisfied with treatment that is in-effective in reducing PSU. On the otherhand, certain treatments can be effectivebut unpopular with clients. Evaluatorsmust remember that client satisfaction andclient outcome are distinct evaluation con-cepts.

Each of the following case examples de-scribes evaluations comparing client sat-isfaction across sites. The first evaluation(Part A) presents a client satisfactionevaluation of a state-sponsored treatment,primarily for people convicted of drunkdriving. Satisfaction was measured usingtwo scales discussed in this workbook: theCSQ-8 and SSS-30. Whereas the CSQ-8provides a single satisfaction score, the

Page 20: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

20 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

SSS-30 assesses several aspects of clientsatisfaction. Results were used to guidesite procedural improvements.

The second evaluation (Part B) is a goodexample of how client satisfaction evalu-ations can be completed with limited re-sources. In this case, a mental health in-tern wanted to examine client satisfactionacross several residential PSU treatmentprogrammes. With limited assistance, hewas able to plan and successfully imple-ment his evaluation. Differences in satis-faction across sites were detected by themultidimensional SSS-30.

The third evaluation examined clientsatisfaction across three communitymethadone treatment sites in Austra-lia. Evaluators used the CSQ-8 andtwo qualitative, open-ended questionsto assess satisfaction. They found sig-

nificant differences in satisfactionacross sites. These differences weregiven to clinic managers to make pro-cedural improvements.

While each of these cases generated use-ful information about client satisfaction,it is noteworthy that none attempted toprovide information about client out-come or treatment effectiveness. As de-scribed earlier in this workbook, mea-surement of client satisfaction is usefulyet distinct from measurement of clientoutcome or treatment effectiveness. Onoccasion, clients can be satisfied withtreatment that is ineffective in reducingPSU. On the other hand, certain treat-ments can be effective but unpopularwith clients. Evaluators must rememberthat client satisfaction and client out-come are distinct evaluation concepts.

Page 21: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

21Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Case examples of clientsatisfaction evaluations

Part A: An evaluation ofsatisfaction with a state drinkerdriver treatment program

byThomas K. Greenfield, Ph.D.Senior Scientist and Area Director forPopulation SurveysNIAAA National Alcohol Research CenterAlcohol Research Group2000 Hearst Ave., Suite 300Berkeley, California 94709-2130 USA

There are a number of reasons why cli-ents’ satisfaction with services is such acritical variable in an overall substanceabuse treatment outcome evaluation ef-fort. First, substance abuse treatmentprogramme directors and managers areoften required to justify their programs.They find consumer satisfaction is a con-cept readily understood by their clients,the public, government bodies, or otherfunding agencies (Greenfield, 1983).This was the case for a private providerin a small Eastern U.S. state licensed to

provide a brief outpatient counselling ser-vice to the ‘chemically dependent client’.The programme is licensed by the stateto provide its services in response to theDriving Under the Influence (DUI) prob-lem. It of course interacts closely withcourts, corrections, and the Departmentof Motor Vehicles (DMV).

Its managers knew that the state and in-volved agencies would need a positiveresponse from its clientele, in addition toobjective outcomes, for continued refer-rals and relicensing. In addition, the man-agers wanted data on specific aspects oftheir programme so that improvement ef-forts might target areas of greatest con-cern to clients. The multidimensionalSSS-30 questionnaire was selected for thisreason based on prior experience in giv-ing useful feedback to student services,primary care, and EAP managers(Attkisson & Greenfield, 1994; Greenfield& Attkisson, 1989a).

Who is asking thequestions and why dothey want thisinformation?

Purposes for the clientsatisfaction evaluation

The author alone isresponsible for theviews expressed in thiscase example.

Page 22: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

22 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Specific programme

In the programme, referrals are pri-marily (but not exclusively) individu-als guilty of a second DUI, and re-quired by law to receive treatment.The programme operates solely onclient fees totalling U.S. $495 at thetime of the study. After two indi-vidual sessions, a treatment plan isdeveloped with suitable, eligible cli-ents. Eligibility requirements include(a) willingness to explore drinkingand drug taking, allowing for a ‘nor-mal degree of denial’, (b) agreementto participate and remain sober anddrug free, (c) commitment to be in-volved and work toward ‘reasonabletreatment goals’, and (d) no overtpsychiatric difficulties implying aprimary mental health problem. Theprogramme involves approximately25 contact hours, including four in-dividual sessions, six educationallyoriented group sessions, and eight90 minute group sessions (under 16members in each group). Additionalindividual and family counsellingmay be included if needed. Condi-tional driving privileges may be re-stored by the court after 16 hours.The programme has some coerciveelements: clients unwilling to par-t icipate meaningfully are deemed‘noncompliant’, with paperwork in-dicating this sent back to the refer-ring agency. The court must act toclear this up before the client mayagain be enrolled in the programme.Upon completion of the treatment,an aftercare plan is developed in thedischarge session. The programme’soffices are located in various coun-ties and the evaluation focused onthree sites to assess their clients’ sat-isfaction. For a more complete de-scription of the programme and thebas is fo r c l i en t se lec t ion , seeGreenfield (1989) and Greenfield(1994).

What resources wereneeded to collect andinterpret theinformation?

The two case studies use direct client sat-isfaction measures, providing examples ofeach of the two measurement strategies.Both questionnaires were designed to bebroad enough to assess satisfaction witha range of human services including sub-stance abuse treatment. The two are theClient Satisfaction Questionnaire-8(Nguyen, Attkisson & Stegner, 1983;

How were the datacollected?

Questionnaires were handed out to cli-ents upon arrival for their final sessionby programme staff. Completed formswere collected daily during the surveyperiod. Thus, little additional effort wasrequired for administration. A copier wasused to duplicate the questionnaires sothat the main resource needed was fordata entry, accomplished by office staffusing the existing dBase software. Thiswas the software package used for main-taining client records (questionnaireswere not identified, so no linkage to otherclient data was possible or attempted).Data entry required approximately twominutes per questionnaire and was doneby the office staff responsible for the cli-ent information system. Because theprogramme did not have analysis soft-ware or capacity, the dBase files weresent on diskette to the scales’ authors foranalysis inSPSS. (Scoring keys for in-agency use, and SPSS syntax for read-ing data from common spredsheet or re-lational database file formats, areavailable from the scale’s firs author). Ateach of the two phases, analysis time in-volved about a day of work with an ad-ditional day needed for report writing.

Page 23: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

23Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Attkisson & Greenfield, 1994; Attkisson& Greenfield, 1995b), a widely-used, brief(8 item), general satisfaction measure,used in the first case study, and the Ser-vice Satisfaction Scale-30 (Attkisson &Greenfield, 1994; Attkisson & Greenfield,1995a, 1995b; Greenfield & Attkisson,1989a) a 30-item multidimensional ver-sion with derivative forms available forcase management and residential settings(Greenfield et al., 1996) including a fam-ily member version (Greenfield &Attkisson, 1989b). Italian (Ruggeri &Greenfield, 1995) and Spanish transla-tions are available.

There is evidence that these human servicemeasures are suitable for use in substanceabuse treatment. The research done suggestsgood psychometric performance of thesetwo measures (SSS-30 and CSQ-8)(Attkisson & Greenfield, 1994; Greenfield,1989; Greenfield, 1994; Greenfield &Attkisson, 1989a). The scales are both prod-ucts of the University of California, SanFrancisco (UCSF), Department ofPsychiatry’s research programme on clientsatisfaction which has extended over a quar-ter of a century (Attkisson & Greenfield,1995b). Permission to use these copyrightscales may be obtained from Dr. C.C.Attkisson (CSQ-8) at the UCSF GraduateDivision, 200 West Milberry Union, 513Parnassus Avenue, San Francisco, Califor-nia 94143-0404 USA (FAX+1-415-476-9690) and Dr. Greenfield (SSS-30) at theAlcohol Research Group, 2000 Hearst Ave.,Berkeley, California, 94709 USA (FAX+1-510-642-7175).

Both scales can be scored and simplyanalysed using common statisticalprogrammes such as SPSS or EPI INFO,or spreadsheet or national data base man-agement software such as Lotus 1-2-3,Excel, dBase, or Paradox, among othersoftware programmes. If programmingcapability is not available in-hours, ormore sophisticated analyses are needed,analysis can be done by an evaluator oranalyst. SPSS scoring keys and syntax are

available from the scale’s first author(Thomas Greenfield, INTERNET:[email protected]).

The CSQ has been included in a compen-dium of instruments assembled by the U.S.National Institute on Alcohol Abuse andAlcoholism (NIAAA), the NIAAA Treat-ment Handbook Series 2: AlcoholismTreatment Assessment Research Instru-ments (Attkisson, et al., 1985). Both theCSQ and the SSS have been included inLloyd Sederer and Barbara Dickeyl ‘sOutcomes Assessment in Clinical Practice(Attkisson & Greenfield, 1995a) whichincludes the scales as appendices. Normsand psychometric results for the SSS-30and CSQ-8 are available in a chapter inMark Maruish’s useful book Psychologi-cal Testing: Treatment Planning and Out-come Assessment (Attkisson &Greenfield, 1994). A second edition of thisbook contains updated chapters on theCSQ-8 (Attkisson & Greenfield, in press)and the SSS-30 (Greenfield & Attkisson,in press).

Because this was a relatively new appli-cation to substance abuse treatment, theproject involved two phases. In the first,it was decided to conduct factor analysesof the SSS-30 data to confirm the factor-based scales previously developed in men-tal health and primary care programmes(Attkisson & Greenfield, 1994; Greenfield& Attkisson, 1989a). In the second phase,data were collected from three programmesites in different locations. In both phases,for validation purposes, the CSQ-8 scalewas administered at the same time as theSSS-30. The dimensional analyses andcomparison of results with the two mea-sures may be considered to be the meth-odological aims of the study.

For practical reasons, it was only possibleto obtain data for people completing theprogramme. The two questionnaires were tobe completed during the last session and leftin a box at the door prior to departure. Ques-tionnaires were filled out anonymously. In

Page 24: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

24 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

the first phase, demographics were not col-lected due to an oversight. In the secondphase, the SSS-30’s standard demographicssection was added including gender, age, in-come, ethnicity, distance from programme,and number of sessions attended.

How were the dataanalysed?Analyses were done using SPSS PC andSPSS for Windows7'. Data were read infrom dBase7' files provide by theprogramme (SPSS can read such files di-rectly). Reversed items were recoded 5=1,4=2, 3=3, 2=4, 1=5 prior to analysis andscoring, based on the published SSS-30subscales (scoring key and code availablefrom T. K. Greenfield at the Alcohol Re-search Group, 2000 Hearst Ave., Berkeley,California 94709, USA). In the first psycho-metric and confirmatory phase (n=1027),item descriptive analyses, factor analysesand reliability analyses (using SPSS factorand reliability routines) of the SSS-30 itemswere done, comparing similarity of factorsolutions using Harmonl ‘s (1970) coeffi-cient of congruence calculated using asimple spreadsheet. The SSS-30 Total Scalescore was correlated (SPSS correlate) withthe CSQ-8 score. These preliminary analy-ses helped assure that the scale functionedwell in a substance abuse programme.

In the second evaluation phase (n=720),demographic profiles of clients at each ofthe three programme sites were first com-pared. Overall satisfaction was then as-sessed by examining item and subscale dis-tributions using the SPSS frequenciesroutine. Subscales were again scored andthese scores compared across the threeprogramme sites using SPSS anova (Analy-sis of Variance) routine. This allowed satis-faction across sites to be compared whilecontrolling for gender differences, sincemen tend to be more willing to indicatelower satisfaction than women (or are ac-tually less satisfied). Finally, the CSQ-8 andSSS-30 total scores were again correlated.

What did they find out?

Phase 1

The two major SSS-30 factors found ear-lier in mental health and primary caresamples (Attkisson & Greenfield,1994)were confirmed. Practitioner Manner andSkill and Perceived Outcome factors werehighly congruent with equivalent onesfrom earlier studies (Harmon coefficients.88 - .93). These two standard factor-basedsubscales were, therefore, useful for as-sessing client satisfaction in this substance

Table 1: Internal reliability of the four SSS-30 subscales.

SubstanceAbuse Program

Published normgroups*

Nº Items

Reliability coefficient (Cronbach� Alpha)SSS-30 Subscale

9

8

5

4

.83

.83

.74

.60

.89

.83

.74

.67

Practicioner manner and skill

Perceived outcome

Office procedures

Accessibility

* Based on 3 Norm groups - Four health Clinics, a Mental Health Service, and an Employee Assistance

Programme (see Attkisson & Greenfield, 1994).

Page 25: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

25Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

abuse programme. There was also someconfirmation for the earlier establishedAccessibility and Office Procedures fac-tors, so these subscales too were con-structed. Internal reliabilities for the fourscales were acceptable (see Table 1).

Finally, the SSS-30 Total score and theCSQ-8 score correlated .70, which pro-vides some added validity to the newerscale. When all items are combined toassess general satisfaction, the total scalescore may be used as a general satisfac-tion measure.

Substantive findings indicated that thesesubstance abuse programme’s court man-dated treatment clients were quite satis-fied with both their counsellor’s Mannerand Skill (programme level = 38.0 + 5.4versus the norm of 38.4 + 5.0 in mentalhealth counselling) and a bit less so forPerceived outcome (program level = 29.6+ 5.6 versus the norm of 32.4 + 4.0 inmental health counselling). Althoughlower by half a standard deviation, themean satisfaction was high (mean-item-mean=4.1), equivalent to a l ‘Mostly Sat-isfied’ response. Item-level results showedthat many programme completers (41%)were dissatisfied (‘Mostly Dissatisfied’or’Terrible’responses) with cost. It will berecalled that this coerced group of peoplecharged with driving while intoxicatedwere required by the courts to pay for theirsubstance abuse treatment programme.Facility location and accessibility werealso sources of dissatisfaction to 17% andit will be recalled that many had their driv-ers licenses suspended. Otherwise few cli-ents (under 10%) were dissatisfied withremaining programme aspects althoughsomewhat more were l ‘Mixed’in theirresponses.

It was important to demonstrate congru-ence between the factors found for sub-stance abuse treatment and those in earlierprimary care and mental health normgroups. Although not unexpected, thiscomparative dimensional analysis con-

firmed the appropriateness of retaining theoriginal subscale composition, makingcomparison with findings and norms fromother human services realistic and appro-priate. In addition, the correlation betweenthe widely used and well validated CSQ-8general satisfaction measure and the SSS-30 composite scale lends construct validityto the newer instrument as a measure of cli-ent satisfaction. The SSS-30 being a multi-dimensional measure adds to its value toprogramme managers who find itssubscales provide relevant feedback onprogramme strengths and weaknesses. Inaddition, its scaling results in less skeweditem distributions, leading to more normallydistributed scale scores than with the CSQ-8 (Greenfield & Attkisson, in press). It alsomakes the use of this outcome measure asa dependent variable in multivariate analy-ses controlling for demographics and othervariables more appropriate, increasing itssensitivity to differences in programme per-formance.

Phase 2

Clientele were mostly male (84%) withmodal age 26-35 years old (46%). Theywere predominantly of Caucasian origin(77%) with African Americans making up10%, typically high school graduates(48%) with modal income US $20-40,000(39%) and tended to live 6-10 miles fromthe programme site. Some clientele dif-ferences were seen across sites withwomen under-represented at one of them.In the cross-site analyses, controlling forgender (which was a significant predic-tor), Manner and Skill satisfaction did varysignificantly (p<.05), though not strongly,between sites (together, gender and site ac-counted for only 3% of the variance, soprogramme managers were cautioned notto over interpret the statistically significantdifference). For Perceived Outcome gen-der was again significant, but only a trendtoward significance (p=.06) was seen bysite. Contrasting these minimal differ-ences, stronger differences were found for

Page 26: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

26 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

How were the resultsused?

It is important to assure that data providedby clients are actually used to improveservices, so that the commitment to theclient to use her or his input for this pur-pose is carried through, justifying thesmall burden of completing the measure.

Results were provided to programmemanagers in a graphic form, showing

Office Procedures subscale satisfactionscores. One site had markedly higher sat-isfaction with office personnel, proce-dures, referrals, collaboration betweenstaff, and record handling, suggesting sup-port staff were functioning well from theservice consumers’ viewpoint. Accessibil-ity satisfaction, though not significantlydifferent, favoured the same site. How-ever, one of the other two sites had cli-ents who lived further away.

overall subscale and item means, as wellas cross site comparisons on the subscalesfor men and women client separately.Subscale score distributions, shown asaveraged item means, were given so thatthe relative number of dissatisfied of‘mixed’responses could readily be seen.Managers can easily share such resultswith staff. In this case, most of the feed-back was positive, allowing for reinforce-ment of ‘a job well done’, much neededin substance abuse services where staffburnout tends to be high. In addition, posi-tive office personnel and procedures in theone site could be identified and emulatedacross sites via cross-site training and se-lective procedural ’tune-ups.’ Resultswere also used in presenting findings toreferring agencies, the courts, Departmentof Motor Vehicles and accreditation bod-ies. The methodological results fromphase 1 were used to assure the managersand evaluators that the Service Satisfac-tion Scale was a reliable and valid toolfor assessing satisfaction with substanceabuse services.

Page 27: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

27Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Part B: Client satisfaction withresidential substance treatmentprogrammes

byThomas K. Greenfield, Ph.D.Senior Scientist and Area Director for Population SurveysNIAAA National Alcohol Research CenterAlcohol Research Group2000 Hearst Ave., Suite 300Berkeley, California 94709-2130 USA

An intern in a county’s umbrellaorganisation responsible for funding andsupervising the management of publiclyfunded drug and alcohol treatmentprogrammes had concerns about how tomeasure client satisfaction given an earlierexperience with a general satisfaction scalein the county’s community mental healthcenters (CMHCs). In the earlier study, thevast majority of clients in a range ofCMHCs, both those thought by the countyto be excellent and those deemed weaker,had indicated mostly satisfied. The clientsatisfaction measure was seen as insensi-tive. In fact, the Research Director hadgiven up on satisfaction questionnaires asa means of obtaining valid client feedback(Nebeker, 1992, p.2). The intern knew ofthe newly developed, multidimensionalscale, the SSS-30 (Greenfield & Attkisson,1989a) and approached its first author toconsult with him on its use in assessing cli-ent satisfaction in residential substanceabuse problems. He had hopes that the mul-tidimensional measure, unlike global mea-sures that the county had given up on, might

Who is asking thequestions and why dothey want thisinformation?

produce sufficient variation in the data tobe able to differentiate one facility fromanother Nebeker (1992, p.1). The internwanted to find a way to reduceAreactivity@ which Lebow (1983, 1983a;1983b) has discussed as biasing satisfac-tion responses upward when therapists col-lect data, or questions are read to clientsrather than answered by paper-and-pencil.He wanted to achieve high response ratesand potentially study the effect of responserate on satisfaction levels. Substantively,he wished to see if there were measurabledifferences in satisfaction between samplesof active residents in four different residen-tial programmes.

The author alone isresponsible for theviews expressed inthis case example.

What resources wereneeded to collect andinterpret theinformation?The intern reproduced the SSS-30 in-strument himself and served as the ad-ministrator. First, he secured countyagreement for the pilot study. Then heobtained agreement from programmedirectors to administer the scale himself.He used the sampling and administra-tion method described in the next sec-tion which required a medium sized

Page 28: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

28 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

How were thedata collected?

How were thedata analysed?

Data were analysed in the same way asdescribed in Phase 2 of the previous casestudy. The standard subscale scores werecomputed and used to compare the meanscores for each subscale across the fourresidential programmes, using ANOVA,with post-hoc tests to indicate the sourceof any difference if found.

Client satisfaction with the programmesdiffered between facilities on some butnot other subscales. One dimension Of-fice Procedures and Personnel showeda significant overall difference betweensites (F(3m 147)= 4.79; p<.01). In post-hoc comparisons, one specif icprogramme was found to differ from an-other on this dimension (Mean-item-mean 3.8 vs 3.3). Perceived Outcomeshowed an overall trend toward a dif-ference (p=.11) and again post-hocanalysis showed the same two residen-tial programmes differed (M = 3.8 vs3.4) significantly (p<.05). In terms ofAccessibility, the same level of trendtoward overall difference was observed.This time the post-hoc pairwise com-parisons showed a different programmeprovided highest satisfaction, statisti-cally (p<.05) higher (Mean-item-mean= 3.7) than the residence that had shownlower satisfaction on the other twosubscales as well (M = 3.3). Only Coun-

What did they find?

cardboard box. Once obtained, he en-tered the data in a word processor, sav-ing the final file as an ASCII text file,which can be read by SPSS (or otherstatistical programmes). Lastly, he se-cured the services of the evaluator toassist him with data analysis using anIBM compatible microcomputer versionSPSS and wrote up the results as a Mas-ters Thesis (Nebeker, 1992). Consider-able independent effort was required ofhim over the course of a year to accom-plish this research project.

The intern developed what he called aAgroup momentum@ method of data col-lection which he described as follows:AThrough trial and error [in pilot work]the following method emerged: (1) thestaff called a meeting [of residents] for theexplicit purpose of filling out the question-naire; (2) I explained that participation inthe study was anonymous and voluntary;(3) the clients were instructed to place thecompleted forms in a cardboard box witha slit cut into the top of the box; (4) I leftthe room so that the clients filled out thequestionnaire without the presence of staffor a test administrator; (5) I removed thebox as soon as the last client had com-pleted the questionnaire (Nebeker, 1992,pp.27-28). The intern comments furtherthat AInstances where the clients filledout the questionnaire individually, out-side the group, produced a lower re-sponse rate@ (p.28).

Page 29: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

29Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

sellor Manner and Skill showed no sig-nificant differences between the facili-ties. Samples obtained during the pilotphase at two facilities had lower re-sponse rates than those obtained at eachusing the Agroup momentum@ ap-proach, which achieved 90-97% re-sponse rates. As hypothesised, thesamples involving low response rates(20-41%) showed higher satisfactionlevels, signif icant ly so for twosubscales: Counsellor Manner and Skill(p<.01) and Accessibility (p<.05). Atone of the facilities the low-response-rate sample gave a Manner and Skillmean value of 4.1, or .5 higher than themean from the more complete sample.

An important finding from this smallstudy was the illustration of the fact thatwhen insufficient efforts are made to as-sure the highest possible response ratesin a nonreactive climate, or when datacollection is haphazard and possibly leftin the hands of staff, results are likelyto be seriously biased toward greatersatisfaction. From an ethical view point,it is essential for evaluators, whether in-ternal or external, to help assure a neu-tral setting for completion of question-naires where there is minimalprogramme staff involvement and influ-ence. In other studies that have achievedexcellent response rates, a volunteer hasserved as the individual approachingwaiting room clients in an open, friendlymanner, explaining the purpose of thestudy and encouraging candid feedbackas most useful for improving theprogramme (Attkisson & Greenfield,1994, Greenfield & Attkisson, 1989a).

How were theresults used?

The results were provided to programmemanagers and to the county umbrellagroup’s staff. Anecdotal evidence suggeststhe county personnel were not surprisedby the differences observed which con-firmed informal observations of theprogramme sites. However, the main re-sult of the study was to demonstrate thefeasibility of using a multidimensionalsatisfaction scale designed to have greatersensitivity than global scales in morewidespread use, and to show the impor-tance of obtaining high response rates forunbiased estimates of client satisfaction,especially so if the intent is cross-programme comparison.

It is also important to select a scale thathas the sensitivity needed to assess realdifference in degree of client satisfaction(Greenfield & Attkisson, in press). Glo-bal scales in widespread use are brief andattractive to programme administrators,but seldom have the requisite psycho-metric qualities to allow genuine differ-ences in satisfaction to be detected, ex-cept with extremely large samples,sometimes gathered in years of routinemonitoring (Greenfield, 1983). Forsmall sample studies a sensitive instru-ment is absolutely essential.

Page 30: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

30 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

It�s your turnWhat are the strengths and the weaknesses of the presented case example? List threepositive aspect and three negative aspects:

Strengths of the case study

1

2

3

Weaknesses of the case study

1

2

3

Page 31: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

31Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Attkisson, C.C., & Greenfield, T.K. The Cli-ent Satisfaction Questionnaire-8 and theService Satisfaction Scale-30. In: M.A.Maruish (Ed.). Psychological testing:Treatment planning and outcome Assess-ment. Hillsdale, N.J.: Lawrence ErlbaumAssociates, 1994.

Attkisson, C.C., & Greenfield, T.K. TheClient Satisfaction Questionnaire (CSQ)Scales and the Service Satisfaction Scale-30 (SSS-30). In: L.I a. D. Sederer (Eds.).Outcomes Assessment in Clinical Prac-tice (pp. 120-127). Baltimore, MD.: Wil-liams & Wilkins, 1995a.

Attkisson, C.C., & Greenfield, T.K. TheClient Satisfaction Questionnaire (CSQ)Scales: A history of scale development anda guide for users. (Unpublished Report)Department of Psychiatry, University ofCalifornia at San Francisco, 1995b.

Attkisson, C.C., & Greenfield, T.K. TheUCSF client satisfaction scales: I. TheClient Satisfaction Questionnaire-8. In:M.A. Maruish (Ed.). Psychological test-ing: Treatment planning and outcomeAssessment (2nd ed.). Mahwah, N.J.:Lawrence Erlbaum Associates, in press.

Greenfield, T.K. The role of client satis-faction in evaluating university counsel-ling services. Evaluation and ProgramPlanning, 1983, 6: 315-327.

Greenfield, T.K. Consumer satisfactionwith the Delaware Drinking Driver Pro-gram in 1987-1988. Department of Psy-chiatry, University of California, 1989.

Greenfield, T.K. Consumer satisfactionwith the Delaware Drinking Driver Pro-gram in 1993. Berkeley, CA, Alcohol Re-search Group, 1994.

Greenfield, T.K., & Attkisson, C.C. Stepstoward a multifactorial satisfaction scalefor primary care and mental health ser-vices. Evaluation and Program Planning,1989, 12: 271-278.

References for case example Parts A and B

Greenfield, T.K., & Attkisson, C.C. Fam-ily satisfaction with services (Report toNorthwest Residential services, Inc.).University California, San Francisco,Department of Psychiatry, 1989b.

Greenfield, T.K., & Attkisson, C.C. TheUCSF Client client satisfaction scales: II.The Service Satisfaction Scale-30. In:M.A. Maruish (Ed.). Psychological test-ing: Treatment planning and outcomeAssessment (2nd ed.) Mahwah, N.J.:Lawrence Erlbaum Associates, in press.

Greenfield, T.K., Stoneking, B.C., &Sundby, E. Two community support pro-gram research demonstrations in Sacra-mento: Experiences of consumer staff asservice providers. The Community Psy-chologist, 1996, 29: 17-21.

Harmon, H.H. Modern factor analysis. Chi-cago: University of Chicago Press, 1970.

Lebow, J.L. Research assessing consumersatisfaction with mental health treatment:A review of findings. Evaluation and Pro-gram Planning, 1983a, 6: 211-236.

Lebow, J.L. Client satisfaction with men-tal health treatment. Evaluation Review,1983b, 7: 729-752.

Nebeker, H. Client satisfaction in four alco-hol and drug treatment programs. mastersThesis, John F. Kennedy University, 1992.

Nguyen, T.D., Attkisson, C.C., & Stegner,B.L. Assessment of patient satisfaction:Development and refinement of a ServiceEvaluation Questionnaire. Evaluation andProgram Planning, 1983, 6: 299-313.

Ruggeri, M., & Greenfield, T. The Italianversion of the Service Satisfaction Scale(SSS-30) adapted for community-basedpsychiatric patients: Development, fac-tor analysis and application. Evaluationand Program Planning, 1995, 18: 191-202.

Page 32: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

32 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

Part C: The case of communitymethadons treatment programs

byJeff WardDivision of PsychologyAustralian National UniversityCanberra, ACT 0200 Australia

Who is asking thequestions and why dothey want thisinformation?

Three publicly funded methadone main-tenance clinics located in Sydney, Aus-tralia participated in an evaluation con-ducted by the Australian National Drugand Alcohol Research Center (NDARC).The purpose of the study was to exam-ine relationships between treatment re-ceived, client characteristics, client sat-isfaction and treatment outcome in termsof heroin use, crime and HIV risk-takingbehaviour. In this case study, the outcomeof interest is how satisfied clients are withthe treatment they have been receivingand what, if any, variables are related totheir level of satisfaction.

The questions were being asked by re-searchers at NDARC as part of a largerresearch effort into the clinical aspectsof methadone maintenance treatment.Client satisfaction with treatment was in-vestigated, because it has become an im-portant outcome of interest to policymakers and treatment providers (Stallard,1996). Furthermore, previous researchhas found client satisfaction to be asso-ciated with client characteristics, serviceutilisation and better treatment outcomein other areas of health care (Pascoe,1983; Tanner 1981).

As the data collected were to be used instatistical analyses, an interview ques-tionnaire was required that preferablywould provide a single quantitative es-timate of client satisfaction. It was alsodesirable that the questionnaire be quickto administer and have established va-lidity and reliability (i.e. that it has beendemonstrated that the questionnairemeasures what it claims to measure andthat it does so consistently with differ-ent populations and over different situ-ations). Such a questionnaire is the 8-item version of the Client SatisfactionQuestionnaire (CSQ-8; (Attkisson &Zwick, 1982). The questionnaire was in-corporated in a much larger interviewschedule that included questions andquestionnaires assessing a range of othervariables related to clients’ histories,current functioning and recent treatmentexperiences.

Approximately 350 interview scheduleswere printed and 348 clients attending thethree methadone clinics were interviewedby trained interviewers from NDARC.

What resources wereneeded to collect andinterpret the data?

The author alone isresponsible for theviews expressed inthis case example.

Page 33: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

33Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Trained interviewers from NDARC visitedeach of the three clinics and interviewedclients on-site for the evaluation project.Clients were told that the information col-lected would not be communicated to clini-cal staff in any way that would identifyindividuals. This was to ensure that theywould not modify their answers to ques-tions in order to either please staff or avoidretribution.

The CSQ-8 was filled out by the clientsthemselves and took approximately 5minutes to complete. Each of the eightitems that make up the CSQ-8 has fiveresponses to choose from ranging frombeing very dissatisfied to very satisfiedto which a score of between 0 and 4 isattached. A higher score indicates moresatisfaction with treatment, and the eightscores are summed to yield a single over-all measure of satisfaction. The CSQ-8score for each client interviewed was en-tered into a computer using SPSS forWindows software. Scoring and data en-try for the CSQ-8 took approximately 6hours. As well as the 8 individual itemsthat make up the CSQ-8, there are twoquestions that allow for a more open-ended response. These two items ask re-spondents to complete the sentences: ’Thething I like best about this agencyis•’..’and ‘If I could change one thingabout this agency, it would be•’..’Theseitems were scanned for consistent re-sponses specific to the clinic concernedand recorded for feedback to the clinicstaff. This took an additional 10 hours.

How were the datacollected?

How were the dataanalysed?

The data was entered and analysed usingSPSS for Windows Version 6.0. When thequestionnaires were scored, it was found

that 6 of the clients had not completed theforms properly. The responses of these 6clients were not included in the analysis;this left 342 scores on the CSQ-8 for theanalysis.

The data analysis proceeded through foursteps, each designed to answer a differ-ent question. The four basic questionsthat informed this analysis were:

l What form does the distribution ofCSQ-8 scores take?

l Is there a difference in client satisfac-tion at each of the three clinics?

l If the three clinics differ in level of cli-ent satisfaction, why and how do theydiffer?

l What aspects of the treatment programwere identified as being in need ofchange in the clients’ responses to theopen-ended questions?

A preliminary step in analysing the datawas to inspect the distribution of theCSQ-8 scores to see if the pattern fol-lowed that observed in other studies,where a majority of study participantsindicate more, rather than less, satisfac-tion with the treatment they receive(Stallard, 1996). The distribution of theCSQ-8 scores for the study of the publicmethadone clinics is set out below in Fig-ure 1. Figure 1 is a histogram which rep-resents graphically the number of clientswho returned each of the scores on theCSQ-8.

As Figure 1 (on next page) shows, theshape of the distribution suggests that, asin previous studies, clients tended to ex-press more, rather than less, satisfactionwith the treatment they received.

What form does thedistribution of scores take?

Page 34: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

34 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

As can be seen from Table 1, the meanCSQ-8 scores for the clients attending thethree methadone clinics were differentfrom each other, with the clinic indicatedby A having the lowest score and the clinicindicated by C having the highest. To de-termine whether these differences weresimply due to chance or not, the next stepwas to subject them to a statistical test.

The appropriate statistical test for assessingwhether the differences in the scores on theCSQ-8 were due to chance or not is a one-way analysis of variance (ANOVA). TheF-ratio, which assesses the statistical significanceof the ANOVA was found to be equal to15.18, which was statistically significant with pset at 0.05 (F = 15.18; df =2,340; p = .000).This means that the differences observed werenot simply due to chance.

To determine which clinics differed fromeach other, the least significant differencetest was employed with adjustment formultiple tests. This revealed that clientsattending clinic C were more satisfied thanthose attending clinics B and A and thatthose attending clinic B were more satis-fied than those attending clinic A.

Do clients attending the threemethadone clinics differ intheir level of satisfaction withtreatment?

Distribution of CSQ-8 scores (N=342)

60

50

40

30

20

10

09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 2930 31 32

tnuoC

Having determined that clients attending thethree methadone clinics tended to be morerather than less satisfied with the treatment theywere receiving, an important subsequent ques-tion was whether there were any differencesin satisfaction with treatment across the threeclinics. In order to answer this question themeans on the CSQ-8 for each of the threeclinics were calculated and can be found inTable 1.

Figure 1

Table 1: Mean CSQ-8 scores forthe three methadone clinics

Clinic Mean CSQ-8Score

A 22.8B 24.9C 26.3

Page 35: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

35Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

Having found out that there were statisti-cally significant differences between thethree clinics in satisfaction with treatment,as measured by the CSQ-8, the questionarose as to why these differences were ob-served.

In order to answer this question and toexamine whether the previously observedrelationships between age, serviceutilisation, treatment outcome and satis-faction would be found with the metha-done clients, a multiple linear regressionmodel was developed. In a regressionmodel, an outcome (in this case the scoreson the CSQ-8) is predicted by a set ofvariables often referred to as predictorvariables. The model allows us to estimateto what extent any given predictor vari-able in the model is related to the outcomeafter taking into account the contributionof all of the other variables in the model.In this case study, a simple model is de-veloped as an example. The procedurefollowed in developing the model is theone recommended by Kleinbaum(Kleinbaum, Kupper, & Muller, 1988).

As noted at the beginning, client charac-teristics and treatment outcome have beenfound in previous research to be related

If the three clinics differ inlevel of client satisfaction,why and how do they differ?

to client satisfaction. In this case study,we will use gender and reported crime inthe past month as examples. Typically,previous research has shown women tobe more satisfied with health care servicesthan men (Pascoe, 1983; Tanner, 1981),while one of the major outcomes expectedof methadone maintenance treatment isthat it will reduce crime (Ward, Mattick,& Hall, 1994).

The regression model is set out below inTable 2. In Table 2, variables markedClinic B and Clinic C are known as‘dummy’variables and indicate the ex-tent of the relationship between these twoclinics and the CSQ-8 when comparedwith Clinic A which is used as a refer-ence category.

The F statistic at the bottom of the tableindicates that the model, as a whole, isassociated with client satisfaction asmeasured by the CSQ-8. Looking at thevariables in the model, it can be seen thatthere appears to be no difference betweenmen and women in their level of satis-faction. Similarly, the outcome fromtreatment indicated by whether the cli-ent reported committing a crime in themonth prior to interview, is also unrelatedto satisfaction with treatment. However,in the case of crime, the p value (.059) isclose to the significance level (.05) andsuggests a closer look at this relationship

Table 2: Multiple regression model for predicting client satisfaction with methadone treatment

Standard error tRegressioncoefficient

Variables in model

22.83

3.59

2.24

-0.99

0.37

.58

3.59

2.24

-0.99

0.37

39.36

5.60

3.45

-1.90

0.75

(Constant)

Clinic B

Clinic C

Crime reported in past month

Gender (1=male)

P

.000

.000

.001

.059

.455

F=8.663, p=.000

Page 36: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

36 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

What aspects of thetreatment program wereidentified as being in needof change in the clients�responses to theopen-ended questions?

In terms of the variables selected for in-vestigation in this case study, it has beenshown that while clients attending threepublic methadone clinics in Sydney,Australia tend to be satisfied with thetreatment they have been receiving,there are statistically significant differ-ences between the level of satisfactionat these three clinics. Unlike studies inother areas of health care, gender and

What did the studyfind out?

How where theresults used?The clients’ responses to the questions con-

cerning what they liked and what theythought needed changing concerning theirtreatment were read and sorted into majorthematic groups. There is insufficient spacein this context to elaborate fully on theseresponses. However, an example will suf-fice. One of the main client concerns werethe restricted hours that the clinics wereopen for methadone dosing. By far the mostcommon thing that clients would changeif they could was the times at which theclinics were open.

In the first instance, the results were com-municated to staff at the participating clin-ics. As an example of the way in which theresults were used by the clinic managers,one aspect of the survey’s use by the man-ager of Clinic A will be discussed. Themanager of Clinic A, which had the leastsatisfied clientele was not surprised by theresults. A recent change in the clinic’s lo-cation and changes to staffing levels werethought to be the cause of the client’s un-happiness. This was reflected in the clientsanswers to the open-ended questions at theend of the CSQ-8. The manager requesteda copy of the data set which was madeavailable so that future surveys conductedby clinic staff could be compared with theresults of this first survey. In this way, themanager would be able to assess whetherchanges that were being planned wouldimprove clients’ satisfaction with theirmethadone treatment.

The results of the study will also be pub-lished in an appropriate journal and willcontribute to the scientific literature onmethadone maintenance clinics and onclient satisfaction in general.

might be warranted. After adjusting forgender and crime, we find that there arestill significant differences between thethree clinics. The meaning of the statis-tically significant relationships for theclinics is that when compared with ClinicA as a reference category, clients attend-ing both Clinic B and Clinic C are moresatisfied with the treatment they havebeen receiving.

treatment outcome were not found to berelated to level of satisfaction, althoughthe treatment outcome selected (crime)was very close to being statistically sig-nificant, suggesting that further investi-gation may be warranted to determine ifand under what circumstances it may berelated to satisfaction with treatment. Itis important to note, however, that thisanalysis has intentionally been restrictedto a small number of variables for thepurposes of this cases study and that therelationships investigated may be differ-ent when placed in the context of thelarger number of variables included inthe study.

Page 37: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

37Workbook 6 · Clent Satisfaction Evaluations

WHO/MSD/MSB 00.2g

It�s your turnWhat are the strengths and the weaknesses of the presented case example? List threepositive aspect and three negative aspects:

Strengths of the case study

1

2

3

Weaknesses of the case study

1

2

3

Page 38: Client Satisfaction Evaluations - WHOapps.who.int/iris/bitstream/10665/66584/7/WHO_MSD_MSB_00.2g.pdf · Workbook 6 • Clent Satisfaction Evaluations 7 WHO/MSD/MSB 00.2g What is a

38 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2g

References for case example Part C

Attkisson, C. C., & Zwick, R. (1982). Theclient satisfaction questionnaire: Psycho-metric properties and correlations withservice utilization and psychotherapy out-come. Evaluation and Program Planning,5, 233-237.

Kleinbaum, D. G., Kupper, L. L., &Muller, K. E. (1988). Applied regressionanalysis and other multivariable methods.(Second ed.). Boston: PWS-Kent.

Pascoe, G. C. (1983). Patient satisfactionin primary health care: A literature reviewand analysis. Evaluation and ProgramPlanning, 6, 185-210.

Stallard, P. (1996). The role and use ofconsumer satisfaction surveys in men-tal health services. Journal of MentalHealth, 5, 333-348.

Tanner, B. A. (1981). Factors influenc-ing client satisfaction with mentalhealth services. Evaluation and Pro-gram Planning, 4, 279-286.

Ward, J., Mattick, R. P., & Hall, W.(1994). The effectiveness of metha-done maintenance treatment: An over-view. Drug and Alcohol Review, 13,327-336.