TELEMEDICINE JOURNAL AND e-HEALTHVolume 10, Number 4, 2004© Mary Ann Liebert, Inc.

Perspective

Telepsychiatry Treatment Outcome ResearchMethodology: Efficacy versus Effectiveness

B. CHRISTOPHER FRUEH, Ph.D.,1,2 JEANNINE MONNIER, Ph.D.,1,2 JON D. ELHAI, Ph.D.,3ANOUK L. GRUBAUGH, Ph.D.,1,2 and REBECCA G. KNAPP, Ph.D.1

ABSTRACT

The use of videoconferencing technology to provide mental health services (telepsychiatry)offers hope for addressing longstanding problems regarding work force shortages and accessto care, especially in remote or rural areas. However, data on treatment outcomes (i.e., basedon randomized clinical trials) from telepsychiatry applications are virtually nonexistent, rep-resenting an important gap in the literature. An important methodological decision point indeveloping treatment outcome research is whether to take an efficacy or effectiveness ap-proach. Efficacy approaches offer enhanced internal validity; however, they may have lim-ited generalizability to real-world settings. Effectiveness approaches offer enhanced externalvalidity. But, they are typically less controlled than efficacy studies, thereby limiting the as-sumptions that can be made about causality. The current state of telepsychiatry research ne-cessitates efficacy studies, the outcomes from which can be used to inform future effective-ness studies.

1Medical University of South Carolina, Charleston, South Carolina.2Ralph H. Johnson Veterans Affairs Medical Center, Charleston, South Carolina.3Disaster Mental Health Institute, University of South Dakota, Vermillion, South Dakota.

455

INTRODUCTION

AGROWING BODY OF LITERATURE suggests thatthe use of videoconferencing technology

to provide mental health services (telepsychia-try) offers hope for solving longstanding prob-lems regarding work force shortages and ac-cess to care, especially in remote or rural areas.Recent reviews of this literature1–4 indicatethat: (1) there are many novel case studies anddescriptive reports of telepsychiatry programsfrom around the world, (2) there is solid em-

pirical evidence that clinical assessments can bereliably and validly conducted via videocon-ferencing technology, and (3) there is moder-ate empirical evidence that mental health con-sumers and clinicians in many settings aresatisfied with telepsychiatry services. How-ever, data on clinical outcomes from telepsy-chiatry applications are lacking. In fact, thereare virtually no extant randomized clinical tri-als (RCTs) at this point, representing an im-portant gap in the literature. A critical method-ological decision point in developing the

5422_e07_p455-458 1/17/05 12:48 PM Page 455

needed treatment outcome research is whetherto take an efficacy or effectiveness approach tothe questions at hand. In this paper, we discussthe relative strengths and limitations of eachapproach as applied to the current telepsychi-atry literature and discuss methodological im-plications for future telepsychiatry outcomesresearch.

EFFICACY AND EFFECTIVENESS INMENTAL HEALTH INTERVENTIONS

RESEARCH

Studies on mental health treatment outcomegenerally represent either an efficacy or effec-tiveness methodological approach. In fact, thisis a continuum of study design features ratherthan a discrete dichotomy. Furthermore, wehave yet to reach full agreement with regard toa precise definition of these terms. A commonset of general principles for efficacy and effec-tiveness have emerged and can be discussed inreference to telepsychiatry research.5–8

In telepsychiatry research, efficacy studiesare essentially carefully controlled laboratoryresearch projects that typically include all ormost of the following: narrow subject inclu-sion/exclusion criteria; randomized assign-ment of individual subjects to treatment con-ditions; inclusion of rigid control groups;careful stratification of patients across groupsby relevant variables, such as race or psychi-atric comorbidity; highly specific treatmentprotocols (e.g., manualized psychotherapy,fixed-dose medications); fixed duration oftreatment and follow-up; ongoing training andtreatment integrity ratings to ensure care is pro-vided as specified in the research protocol;comprehensive and standardized individualoutcome assessment batteries that focus onclinical symptoms and functioning; andblinded clinician outcome ratings.

Effectiveness studies are generally less con-trolled research projects conducted in “real-world” practice settings and are characterizedby some or all of the following features: broadsubject inclusion/exclusion criteria, so as torepresent the clinical practice setting popula-tion of interest; randomization of sites or clin-icians, as well as randomization of individual

subjects; more flexible treatment protocols;more flexible treatment duration periods; useof actual practice setting clinicians to providecare; less comprehensive (i.e., more feasible)outcome assessment of individuals on clinicalvariables; examination of systemic outcomevariables, such as hospitalization rates, med-ication compliance, and other variables relatedto costs and cost-effectiveness, as well as indi-vidual outcomes.

Efficacy studies generally enhance internalvalidity, while effectiveness studies enhanceexternal validity. The former are necessary toestablish causation under highly controlledconditions, while the latter are necessary to es-tablish that treatments actually work and facil-itate dissemination and translation of inter-ventions to real-world practice settings. Thus,the two approaches are complementary, withefficacy studies usually preceding effectivenessstudies.

AN EFFICACY APPROACH TO TELEPSYCHIATRY

At this early stage of the telepsychiatry in-terventions literature, where there is virtuallyno clinical outcome data to support the use ofmental health services via videoconferencing,a good argument can be made for the use of ef-ficacy designs. The strengths of such an ap-proach allow for a more controlled study andenhanced internal validity, which are espe-cially important in the early stages of literaturedevelopment; enhanced feasibility regardingsubject recruitment, informed consent, assess-ment, and randomization; and the opportunityto work out a range of potential difficulties ina more controlled “laboratory” setting.

Because there are no empirical outcome dataon the use of telepsychiatry to provide treat-ment for specific psychiatric populations (e.g.,those with post-traumatic stress disorder), itseems prudent to conduct such studies in amanner that is as controlled as possible. Be-cause treatment for subjects in both conditionscan be provided at the same site, researcherscan be sure they are recruiting a subject sam-ple that is representative of the patients treatedat a particular clinic, and that they are receiv-

FRUEH ET AL.456

5422_e07_p455-458 1/17/05 12:48 PM Page 456

ing the same treatment in each condition, withthe only difference being mode of treatment de-livery. In other words, there is not a natural se-lection bias that might result if, for example,rural patients formed the “telepsychiatry”group, whereas more urban patients formedthe “same room” comparison group. Addi-tionally, researchers can be sure that all sub-jects face a similar transportation “burden” andhospital experience. In this way, variables suchas differences in driving times, traffic or park-ing hassles, or the social context of the medicalcenter do not have a differential impact on thetwo groups.

Feasibility and patient safety concerns arealso major issues in a decision to follow an ef-ficacy approach in the early stages of researchdevelopment. In an efficacy study, researchproject staff can recruit subjects, obtain in-formed consent, conduct study assessments,and implement randomization assignments inperson, without the difficulties and costs asso-ciated with having to travel to remote sites. Ad-ditionally, trained personnel (e.g., study thera-pists) can be sitting in the room with patientsin the “telepsychiatry” condition, as opposedto an inexperienced and unfamiliar clinician ata remote site, serving as a “safety net” in caseof a patient emergency during a session. Withexperience and empirical data to support theuse of telepsychiatry with various patient pop-ulations, these concerns will become lessprominent and allow for future research to es-tablish the effectiveness of this treatment de-livery mode.

Although this efficacy approach has a num-ber of strengths, several important limitationsmust be recognized. These include the possi-bility that the therapeutic context may be too“artificial” for subjects to feel comfortable, dif-ferent cost–benefit ratios for telepsychiatry con-ditions when the travel-saving features are notevident, and reduced external validity or gen-eralizability. While an efficacy approach wouldtell us something about the efficacy of telepsy-chiatry under very controlled conditions, it tellsus little about how this mode of service deliv-ery will work in the “real world.” In fact, it iseven conceivable that the artificial nature of theconditions could lead to some distortion in theresults. For example, subjects in “telepsychia-

try” may learn that their therapist is relativelynearby. They may, therefore, not take the treat-ment sessions seriously if the treatment modeseems unnecessarily complicated or lacking incredibility. In a related issue, because subjectsin the “telepsychiatry” group will not havebeen saved meaningful travel time and costsordinarily associated with the use of telepsy-chiatry under usual circumstances, they mayfind it more difficult to appreciate some of thereal benefits of this mode.

AN EFFECTIVENESS APPROACH TO TELEPSYCHIATRY

An effectiveness approach also has severalnotable strengths and limitations in the contextof telepsychiatry research. It may be more fea-sible to implement in some ways; it also offersenhanced external validity. Furthermore, treat-ment effectiveness is the ultimate goal of anyline of treatment outcome research. On theother hand, it may be quite costly due to thenature of multisite studies with large samples,offers less control over extraneous variablesand reduced internal validity, and poses feasi-bility challenges to a number of research issues(including informed consent, randomization,and management of acute psychiatric emer-gencies) and relevant ethical concerns relatedto patient safety and confidentiality.2 For thesereasons, and despite the apparent limitations ofthe efficacy approach, the use of effectivenessapproaches at this stage of research in thetelepsychiatry arena in many instances may bepremature. However, assuming the efficacy oftelepsychiatry treatment can be established, thenext logical research step will be to evaluate theeffectiveness of “telepsychiatry” service deliv-ery in applied practice settings.

Future research is likely to involve the use oftelepsychiatry to provide specialized treatmentto various patient populations through suchsystems as departments of mental health andVA Medical Centers. This would allow appro-priately trained and experienced clinicians toprovide specialty mental health care in areaswhere expert clinicians do not exist. Study de-signs could incorporate methodological strate-gies that randomize by site or by subject. Out-

TELEPSYCHIATRY TREATMENT OUTCOME 457

5422_e07_p455-458 1/17/05 12:48 PM Page 457

FRUEH ET AL.458

come variables could include systemic vari-ables (e.g., use of other health services, phar-macy records), and could include cost analysis.Such research would benefit from reduced ar-tificiality and enhanced external validity, but itmight also be costlier to conduct due to theneed for research staff at (or strong coordina-tion and cooperation from) remote sites, theneed for purchasing additional videoconfer-encing equipment, and the need for larger sam-ple sizes associated with multisite studies.

IMPLICATIONS FOR FUTURETELEPSYCHIATRY RESEARCH

Treatment outcome research on telepsychia-try is needed for both general and specific psy-chiatric disorders, across a range of settings(e.g., rural, military, prison settings). Both effi-cacy and effectiveness approaches have some-thing to offer, and both are necessary. How-ever, at this early stage of development, anefficacy approach may make more sense inmost cases because of enhanced internal valid-ity, feasibility, and cost advantages. Mixed ap-proaches (i.e., selecting features of both) mayalso make a great deal of sense, and, in the end,design considerations will depend on the na-ture of the specific research questions of in-terest and the development of the extant liter-ature. Ultimately, the vast majority of telepsy-chiatry treatment outcome research will leantoward the effectiveness end, but there is roomfor efficacy studies at this early stage to pro-vide empirical support for the risks and costsinherent in an effectiveness approach.

ACKNOWLEDGMENTS

This work was supported by grantsMH01660 from the National Institute of Men-tal Health (NIMH) and HS11642 from theAgency for Healthcare Research and Quality(AHRQ).

REFERENCES

1. Capner M. Videoconferencing in the provision of psy-chological services at a distance. J Telemed Telecare2000;6:311–319.

2. Frueh BC, Deitsch SE, Santos AB, et al. Procedural andmethodological issues in telepsychiatry research andprogram development. Psychiatric Serv 2000;51:1522–1527.

3. Hailey D, Roine R, Ohinmaa A. Systematic review ofevidence for the benefits of telemedicine. J Telemed Tele-care 2002;8(Suppl 1):1–30.

4. Monnier J, Knapp RG, Frueh BC. Recent advances intelepsychiatry: An updated review. Psychiatric Serv2003;54:1604–1609.

5. Armitage P. Attitudes in clinical trials. Statist Med1998;17:2675–2683.

6. Chambless DL, Hollon SD. Defining empirically sup-ported therapies. J Consult Clin Psychol 1998;66:7–18.

7. Hollon S. The efficacy and effectiveness of psy-chotherapy relative to medications. Am Psychol 1996;51:1025–1030.

8. Seligman MEP. The effectiveness of psychotherapy.Am Psychol 1995;50:965–974.

Address reprint requests to:B. Christopher Frueh, Ph.D.

VAMC/MHS (116)109 Bee Street

Charleston, SC 29401

E-mail: fruehbc@musc.edu

5422_e07_p455-458 1/17/05 12:48 PM Page 458