Collaboration From a Distance: Collaboration From a Distance: The Consequences of Implementing The Consequences of Implementing

an Evaluation Study by Proxyan Evaluation Study by Proxy

Jeffrey Sheldon, M. A., Ed. M.Jeffrey Sheldon, M. A., Ed. M.School of Behavioral & Organizational Sciences School of Behavioral & Organizational Sciences

Claremont Graduate UniversityClaremont Graduate UniversityThe Claremont CollegesThe Claremont Colleges

June 2006June 2006

22

Evaluation OverviewEvaluation Overview

Evaluation site: Private hospital, Durban, South Africa.Evaluation site: Private hospital, Durban, South Africa. Clients: Psychology DepartmentClients: Psychology Department Evaluation Component: Process and Outcomes.Evaluation Component: Process and Outcomes. Evaluand: Patient satisfaction with psychological services.Evaluand: Patient satisfaction with psychological services. Unit of Analysis: Patients.Unit of Analysis: Patients. Method/Data Source: Survey/Patients.Method/Data Source: Survey/Patients. Comparison Group: None.Comparison Group: None. Data Collection: Psychology Department Staff.Data Collection: Psychology Department Staff. Sample Size: 11.Sample Size: 11. Evaluation period: 15 – 30 November 2005.Evaluation period: 15 – 30 November 2005.

33

Evaluation NeedsEvaluation Needs

Little known about hospital patients’ thoughts, Little known about hospital patients’ thoughts, beliefs, and behaviors towards psychological services beliefs, and behaviors towards psychological services received. received.

Documentation and analysis of patient satisfaction Documentation and analysis of patient satisfaction with processes and outcomes of psychological with processes and outcomes of psychological services. services.

44

Evaluation QuestionsEvaluation Questions

How satisfied are patients with Psychology Department services? How satisfied are patients with Psychology Department services?

Does high satisfaction correlate with high program quality? Does high satisfaction correlate with high program quality?

55

Evaluation UseEvaluation Use Generate knowledge about the department’s Generate knowledge about the department’s

effectiveness in providing services to patients. effectiveness in providing services to patients.

Program improvement (QIP -Quality Improvement Program improvement (QIP -Quality Improvement Procedures)Procedures)

Primary intended users:Primary intended users: Hospital Psychologist; Hospital Psychologist; Intern Clinical Psychologists; and Intern Clinical Psychologists; and Counseling Psychology Intern. Counseling Psychology Intern.

66

BackgroundBackground Hundreds of outcome studies on the effectiveness of counselling and Hundreds of outcome studies on the effectiveness of counselling and

psychotherapy. psychotherapy.

Few carried out specifically in a primary care setting such as hospitals. Few carried out specifically in a primary care setting such as hospitals.

Hemmings (1997) surveyed 96 patients in an in-house counseled group, Hemmings (1997) surveyed 96 patients in an in-house counseled group, asking their opinions about received services.asking their opinions about received services.

Hemmings as precedent:Hemmings as precedent:

Collected and analyzed data about patient satisfaction with the Collected and analyzed data about patient satisfaction with the outcomes of a hospital – based psychological service outcomes of a hospital – based psychological service

Point of departure: collected and analyzed data about the peripheral Point of departure: collected and analyzed data about the peripheral services and environment (processes) of those same services. services and environment (processes) of those same services.

77

Survey DevelopmentSurvey Development

No formally articulated program theory to guide the No formally articulated program theory to guide the evaluation, evaluation,

Survey developed on the basis of:Survey developed on the basis of: Departmental criteria and standards by which Departmental criteria and standards by which

counseling services are conducted; counseling services are conducted; Department’s knowledge needs; andDepartment’s knowledge needs; and Patient satisfaction and service quality theory from Patient satisfaction and service quality theory from

extant literature. extant literature.

88

Patient SatisfactionPatient Satisfaction

Relationship w/ psychologistRelationship w/ psychologist Expected patient OutcomesExpected patient Outcomes Treatment – the counselling processTreatment – the counselling process Structure – peripheral services and environmental factorsStructure – peripheral services and environmental factors

(Donebedian, 1989)(Donebedian, 1989)

Helpfulness of counselling; Helpfulness of counselling; Being understood by counselor; Being understood by counselor; Having enough time to talk to counselor; andHaving enough time to talk to counselor; and Counsellor easy to talk with. Counsellor easy to talk with.

(Hemmings, 1997)(Hemmings, 1997)

99

Service QualityService Quality

A function of overall satisfaction and prior experience in a therapy A function of overall satisfaction and prior experience in a therapy setting:setting:

What patients think and feel about their therapy sessionWhat patients think and feel about their therapy session Attitudes and behaviors toward therapyAttitudes and behaviors toward therapy How well they understand and follow their psychologist’s How well they understand and follow their psychologist’s

instructions instructions The outcome of their therapyThe outcome of their therapy

1010

Theoretical ModelTheoretical Model

The following is a conceptual model of the program theory:

FIGURE 1

Therapy session

Attitudes & behaviors toward therapy

Following instructions

Outcome of therapy

General satisfaction

Service quality

Previous experience

Knowledge of therapy

Expectation of length of therapy

1111

MethodsMethods Initiated contact with Intern Clinical Psychologist to ascertain Initiated contact with Intern Clinical Psychologist to ascertain

Psychology Department evaluation needs.Psychology Department evaluation needs. Positive and enthusiastic response to conducting an evaluation. Positive and enthusiastic response to conducting an evaluation. Needs articulated.Needs articulated. Designation of an on-site co-researcher coordinator.Designation of an on-site co-researcher coordinator. Intern Clinical Psychologist with research experience Intern Clinical Psychologist with research experience

designated supervisor.designated supervisor. Memos of expectations and instructions sent via email. Memos of expectations and instructions sent via email. Some input given on survey instruments.Some input given on survey instruments. Surveys and instructions were to be administered between 22 Surveys and instructions were to be administered between 22

October and 22 November 2005.October and 22 November 2005. Summary of evaluation with recommendations sent via email Summary of evaluation with recommendations sent via email

January 2006.January 2006.

1212

ParticipantsParticipants English speaking.English speaking. 18 years and older.18 years and older. A current hospital patient (either in or out-patient).A current hospital patient (either in or out-patient). Have the ability to fully understand and give informed consent.Have the ability to fully understand and give informed consent. No psychological condition precluding them from fully understanding and No psychological condition precluding them from fully understanding and

giving informed consent.giving informed consent. Could not present with any acute and extreme psychological condition that Could not present with any acute and extreme psychological condition that

would necessitate immediate intervention on the part of a psychologist or would necessitate immediate intervention on the part of a psychologist or render them incapable of actually filling out a survey. render them incapable of actually filling out a survey.

No other distinguishing characteristics.No other distinguishing characteristics.

1313

SamplingSampling

Venue-based and convenient. Venue-based and convenient.

Patients could have arrived for counselling in different ways. Patients could have arrived for counselling in different ways.

Patients were to have been solicited for participation at the end of a Patients were to have been solicited for participation at the end of a counselling visit. counselling visit.

1414

MaterialsMaterials

Informed consent form developed with input from the Hospital Ethics Informed consent form developed with input from the Hospital Ethics Committee. Committee.

Survey:Survey: Sixty-two Likert-scale items about patients’ experiences with Sixty-two Likert-scale items about patients’ experiences with

counselling at the hospital. counselling at the hospital. Nine questions to compare the overall quality of care received with Nine questions to compare the overall quality of care received with

that provided at another institution (if applicable). that provided at another institution (if applicable).

Dillman’s (2000) principles for survey item construction and Dillman’s (2000) principles for survey item construction and formatting. formatting.

1515

Planned Analysis StrategyPlanned Analysis Strategy Factor Analysis Factor Analysis

Correlations:Correlations: between constructs and with moderatorsbetween constructs and with moderators Between moderators and service qualityBetween moderators and service quality

Regression: Regression: of 6 constructs and moderators on service qualityof 6 constructs and moderators on service quality

ANOVA: ANOVA: of satisfaction (low & high) and previous experience (yes or of satisfaction (low & high) and previous experience (yes or

no/good or bad) on 4 constructsno/good or bad) on 4 constructs … … previous experience on 2 constructsprevious experience on 2 constructs

Content Analysis: Content Analysis: of open-ended questions on 6 constructs of interestof open-ended questions on 6 constructs of interest

1616

Expected ResultsExpected Results

In keeping with patient satisfaction with health services studies in general, In keeping with patient satisfaction with health services studies in general, it was expected that counseling services would be viewed positively it was expected that counseling services would be viewed positively (Hemmings, 1997). (Hemmings, 1997).

Studies have consistently found patients to report reasonable satisfaction Studies have consistently found patients to report reasonable satisfaction with their mental health treatment (MacPherson et al, 1998). with their mental health treatment (MacPherson et al, 1998).

1717

Actual ResultsActual Results Few of the evaluation questions the clients wanted answered were Few of the evaluation questions the clients wanted answered were

answered. answered.

The only “comfortable” result: patients perceive the six constructs The only “comfortable” result: patients perceive the six constructs positively. positively.

Therapy outcome had the strongest correlation to perceived service qualityTherapy outcome had the strongest correlation to perceived service quality the only construct found to significantly contribute to the prediction of the only construct found to significantly contribute to the prediction of

service quality if all other constructs were accounted for. service quality if all other constructs were accounted for. The only supported hypothesis (from the extant literature). The only supported hypothesis (from the extant literature). A positive perception of therapy outcomes is the strongest predictor of A positive perception of therapy outcomes is the strongest predictor of

perceived quality of service based on the variables in the study. perceived quality of service based on the variables in the study. This finding based on an extreme response set and very low numbers. This finding based on an extreme response set and very low numbers.

1818

Unanticipated ResultsUnanticipated Results Provided information about feasibility of conducting a full-scale evaluation Provided information about feasibility of conducting a full-scale evaluation

of psychological services. of psychological services.

Tested the survey instrument and proxy process. Tested the survey instrument and proxy process.

Tested the proxy process. Tested the proxy process.

Provided preliminary information about hospital patients’ thoughts, beliefs, Provided preliminary information about hospital patients’ thoughts, beliefs, and behaviors towards psychological services received.and behaviors towards psychological services received.

1919

A Different Kind of CollaborationA Different Kind of Collaboration

Could not be physically present, relied on a proxy to implement the Could not be physically present, relied on a proxy to implement the evaluation. evaluation.

A belief distance evaluation could work given strong initial A belief distance evaluation could work given strong initial enthusiasm and commitment of clients.enthusiasm and commitment of clients.

Lack of total control over the process.Lack of total control over the process.

Blurred lines of distinction between client and evaluator. Blurred lines of distinction between client and evaluator.

2020

A Different Kind of CollaborationA Different Kind of Collaboration

Unforeseen logistical and collaborative challenges necessitating Unforeseen logistical and collaborative challenges necessitating multiple procedural trade-offs. multiple procedural trade-offs.

Trade-offs impinged upon the validity of findings, compromises made Trade-offs impinged upon the validity of findings, compromises made likely had some deleterious effect. likely had some deleterious effect.

Would have been easier to take a pass than continue, but…Would have been easier to take a pass than continue, but…

2121

Communication ChallengesCommunication Challenges Limited and inconsistent communication with program staff. Limited and inconsistent communication with program staff.

Communication with only one staff member who relayed Communication with only one staff member who relayed information to key decision-makers then relayed back information to key decision-makers then relayed back decisions made.decisions made.

Diminished communication frequency over time.Diminished communication frequency over time.

Prohibitive costs of calling South Africa and time difference Prohibitive costs of calling South Africa and time difference made communication feasible only through email. made communication feasible only through email.

One internet-connected PC available for staff use at the One internet-connected PC available for staff use at the hospital. hospital.

2222

Implementation ChallengesImplementation Challenges Detailed email memos sent in hopes of negotiating Detailed email memos sent in hopes of negotiating in good faithin good faith, but in the , but in the

end, they implemented the survey without consulting with us. end, they implemented the survey without consulting with us.

No on-site training of proxy. No on-site training of proxy.

Reliance on written instructions to provide survey implementation training Reliance on written instructions to provide survey implementation training – not sure if they were adhered to. – not sure if they were adhered to.

No oversight of the implementation process. No oversight of the implementation process. Unable to see process of survey in action and make mid-course Unable to see process of survey in action and make mid-course

corrections to enhance quality of implementation. corrections to enhance quality of implementation. No direct communication with person responsible for survey planning No direct communication with person responsible for survey planning

and implementation so anything we might have heard came too late to and implementation so anything we might have heard came too late to intervene accordingly. intervene accordingly.

2323

Implementation ChallengesImplementation Challenges Staff work-load distribution inequitable.Staff work-load distribution inequitable.

Primary decision maker taking a long time to make decisions and then Primary decision maker taking a long time to make decisions and then convey the resultsconvey the results

Availability of contact person limited at bestAvailability of contact person limited at best

Never found out:Never found out: How patients were recruited How patients were recruited What respondents were told about informed consent or survey What respondents were told about informed consent or survey Who administered the surveys Who administered the surveys Where the surveys were completed Where the surveys were completed How long it took for surveys to be completed How long it took for surveys to be completed How they were collected and stored (for confidentiality). How they were collected and stored (for confidentiality).

2424

Bureaucratic ChallengesBureaucratic Challenges Misunderstanding of intentions (applied v. basic research) by the hospital’s Misunderstanding of intentions (applied v. basic research) by the hospital’s

Ethics Committee:Ethics Committee:

Delayed an affirmative decisionDelayed an affirmative decision Limited sample size to ten Limited sample size to ten Turned planned evaluation into a pilotTurned planned evaluation into a pilot Wanted an unspecified reassessment after “pilot”Wanted an unspecified reassessment after “pilot” Wanted survey to be translated into isiZuluWanted survey to be translated into isiZulu No incentives to respondentsNo incentives to respondents

Couldn’t do a wire transfer of funds for return postage or incentives. Couldn’t do a wire transfer of funds for return postage or incentives.

2525

Logistical ChallengesLogistical Challenges

Survey packet delayed for two weeks by South African customs Survey packet delayed for two weeks by South African customs

Only two weeks to conduct the evaluation (half of anticipated time-frame).Only two weeks to conduct the evaluation (half of anticipated time-frame).

2626

Self-Imposed ChallengesSelf-Imposed Challenges Unable to meet with Psychology Department staff to develop clearly Unable to meet with Psychology Department staff to develop clearly

articulated program theory: articulated program theory:

Overall scale and subscales tied to some of the more commonly Overall scale and subscales tied to some of the more commonly operationalized constructs of patient satisfaction from the literature. operationalized constructs of patient satisfaction from the literature.

No pilot-testing with a small sample of the patient population to assess No pilot-testing with a small sample of the patient population to assess cognitive processing and language difficulties. cognitive processing and language difficulties.

Population selection bias: patient population low income, no Population selection bias: patient population low income, no access to another health institution, thus influencing their access to another health institution, thus influencing their responses and biasing results toward the positive (Clark, et al responses and biasing results toward the positive (Clark, et al 2004). 2004).

2727

Self-Imposed ChallengesSelf-Imposed Challenges No qualitative phase to better understand underlying subscale constructs. No qualitative phase to better understand underlying subscale constructs.

Did not tie in prior patient psycho-metric assessments to survey to Did not tie in prior patient psycho-metric assessments to survey to determine potential psycho-emotional confounds. determine potential psycho-emotional confounds.

No demographic data collected, e.g., age, education level, patients No demographic data collected, e.g., age, education level, patients expectations, intention to recommend the hospital, etc… that could have expectations, intention to recommend the hospital, etc… that could have been used to control for response bias during the analysis.been used to control for response bias during the analysis.

2828

Analytical ChallengesAnalytical Challenges Small sample size and missing items.Small sample size and missing items.

Could not run factor analysis.Could not run factor analysis. Couldn’t be certain particular constructs were actually being measured. Couldn’t be certain particular constructs were actually being measured. Further testing could not be based on calculated average rating score Further testing could not be based on calculated average rating score

for each construct based on the factor analysis separating each item for each construct based on the factor analysis separating each item into a construct - Correlations, regressions, and T-test not trustworthyinto a construct - Correlations, regressions, and T-test not trustworthy

Potential sampling error and response biases. Potential sampling error and response biases.

Aside from general descriptions of the data and calculated averages, results Aside from general descriptions of the data and calculated averages, results from tests run were unreliable. from tests run were unreliable.

2929

Distance Evaluation Might Work By…Distance Evaluation Might Work By…

Setting up an effective, two-way communication structure.Setting up an effective, two-way communication structure.

Having a longer, more realistic time-frame with appropriate, mutually Having a longer, more realistic time-frame with appropriate, mutually agreed upon deadlines.agreed upon deadlines.

Having a signed M. O. A.Having a signed M. O. A.

Having adequate funding.Having adequate funding.

Having realistic knowledge of organization’s bureaucratic Having realistic knowledge of organization’s bureaucratic mechanisms.mechanisms.

Using technology to train the proxy (availability dependent).Using technology to train the proxy (availability dependent).

3030

Distance Evaluation Might Work By…Distance Evaluation Might Work By…

Involving more staff and maintaining their involvement through Involving more staff and maintaining their involvement through incentives.incentives.

Creating the program theory together on-line.Creating the program theory together on-line.

Having greater control over process by requiring frequent updates.Having greater control over process by requiring frequent updates.

Going through site IRB first.Going through site IRB first.

Having a thorough understanding of department/organization culture.Having a thorough understanding of department/organization culture.

Planning for delays.Planning for delays.

Making sure clients understand what is being asked of them as proxy.Making sure clients understand what is being asked of them as proxy.

3131

For more information contact:For more information contact:

Jeffrey Sheldon, Ed. M.Jeffrey Sheldon, Ed. M.

School of Behavioral & Organizational SciencesSchool of Behavioral & Organizational Sciences

Claremont Graduate UniversityClaremont Graduate University

1.909.447.54741.909.447.5474

jeffrey.sheldon@cgu.edujeffrey.sheldon@cgu.edu