Leisure Competence Measure: Development and Reliability ...

Leisure Competence Measure:Development and ReliabilityTesting of a Scale to MeasureFunctional Outcomes inTherapeutic RecreationMarita Kloseck, Richard G. Crilly, Gary D. Ellis, and Elsa Lammers

Measurement of patient-specific outcomes continues to be a growing concern for the thera-peutic recreation profession. This paper examines the Leisure Competence Measure (LCM),a behaviour anchored rating scale designed to document current levels of leisure functioningand to demonstrate change over time. The conceptual background and development ofthe LCM, and its validation with a geriatric rehabilitation population will be outlined.Generalizability theory provides the framework for data analysis. Results support the LCMas an instrument from which viable and dependable inferences regarding leisure functioningmay be drawn.

KEY WORDS: Leisure Competence Measure, Outcome Measurement, GeneralizabilityTheory, Therapeutic Recreation

Marita Kloseck, M.S., CTRS is a doctoral student completing a joint Ph.D. in HealthStudies and Recreation and Leisure Studies at the University of Waterloo. Ms. Kloseck isthe former Director, Department of Recreation, Parkwood Hospital, London, Ontario,Canada. Richard G. Crilly, M.D., M.R.C.P.(U.K.), F.R.C.P.C. is an associate professor inthe Department of Medicine and Chairman of the Division of Geriatric Medicine at theUniversity of Western Ontario. Dr. Crilly is also Program Director for the SouthwesternOntario Regional Geriatric Program. Gary D. Ellis, Ph.D., is a professor and Director ofGraduate Studies in the Department of Recreation and Leisure at the University of Utah.Elsa Lammers, Hons. B.A., is Director of Recreation Services at St. Thomas PsychiatricInstitute, St. Thomas, Ontario, Canada.

First Quarter 1996 13

Demonstration of measurable outcomesis essential in health care professions to en-sure treatment effectiveness as well as com-petitiveness in an environment of shrinkingresources. Increasingly, documentation offunctional gain by patients is required to jus-tify professional services and to determineprospective payment. This is particularlytrue in the United States, but is a trend inCanada and other countries as well (Ameri-can Health Consultants, 1993; Boschen,1985; Forer, 1982; Hamilton & Granger,1987; Johnston, Keith & Hinderer, 1992).

As a component of the health care sys-tem, therapeutic recreation professionalsmust also perform outcomes evaluation.Such evaluations are critical to determiningthe effects of therapeutic recreation interven-tion on individuals with disabilities and arerequired to ensure compliance with the Com-mission on Accreditation of RehabilitationFacilities (CARF) and the Joint Commissionon Accreditation of Healthcare Organiza-tions (JCAHO) requirements. Quantifiabledata are needed, therefore, to objectivelygauge the functional status of individual pa-tients with respect to recreation and leisureand thereby to allow evaluation of changesin functional status that may result fromtreatment. A significant challenge for thera-peutic recreation professionals, then, is toacquire and appropriately utilize instrumentsthat accurately measure leisure functioning.Despite the growing demand, a dearth ofsuch instruments exists (Burlingame &Blaschko, 1990; Dunn, 1984; Howe, 1984;Stumbo, 1990). The Leisure CompetenceMeasure (LCM) was developed in responseto this demonstrated need.

The LCM is a multidimensional tool de-signed to address competencies required forsuccessful leisure functioning. The purposeof this paper is to: (1) outline the develop-ment of the LCM; and (2) determine its relia-bility under different measurement scenariosand demonstrate the LCM's potential use-fulness under the scenario most likely to per-tain under normal clinical practice. Includedis a discussion of the conceptual foundationof the LCM as well as a description of anempirical investigation of its dependability(reliability) and validity of inferences thatcan be drawn from LCM scores.

Conceptualization of the LCMThe LCM was developed by two of the

authors (Kloseck & Lammers) in responseto a need to demonstrate change in leisurefunctioning resulting from therapeutic recre-ation intervention. The underlying concep-tual framework of the LCM can be found inthe leisure-based philosophy for therapeuticrecreation practice which is operationalizedthrough the Leisure Ability Model (Pe-terson & Gunn, 1984), as well as the behav-ioural construct of competence outlined inthe literature (Gerard, 1959; Hogg & Heller,1990; Lawton, 1972; Norris, 1991; Olson &Freedman, 1979; Scofield, Berven & Har-rison, 1981). For practical purposes, theLCM was designed to be consistent with theFunctional Independence Measure (Hamil-ton, Granger, Zielezny & Tashman, 1987) toafford therapeutic recreation professionals,employed in rehabilitation facilities, a con-sistent and acceptable means of reporting atteam rounds. The Functional Independence

Information regarding the Leisure Competence Measure may be obtained from: LeisureCompetence Measure® Data System, 9 Mount Pleasant Avenue, London, Ontario, CanadaN6H 1C8. Telephone: (519) 679-1833.

The authors would like to acknowledge the support of the Ministry of Culture, Tourismand Recreation, Ontario, Canada and the Parkwood Hospital Foundation, London,Ontario, Canada in providing funding to conduct reliability and validity tests of theLeisure Competence Measure.

14 Therapeutic Recreation Journal

Measure (FIM) is a widely accepted and uti-lized instrument for outcome measurementin medical rehabilitation. The FIM assessespersonal independence in activities of dailyliving including self-care, elimination, mo-bility, ambulation, communication and cog-nition. The FTM does not, however, addressleisure functioning.

Various theory-based conceptualizationsof competence currently exist in the litera-ture. One very influential approach is the ex-amination of competence from a motiva-tional construct. This perspective infers thatall persons have a need to feel competent,that this need motivates behaviour and thatsuccessful interaction with one's environ-ment and resulting feelings of inherent plea-sure are key elements of this process (Harter,1978; White, 1959). Baltes and Wahl (1992),in examining the social environmental con-ditions influencing dependent and indepen-dent behaviour, support the strong relation-ship between the environment and individualcompetence, particularly in the elderly.Other related competence constructs includeperceived competence, perceived control,self-efficacy and learned helplessness. Theseconstructs focus on individual perceptionsand subjective appraisal of personal capabili-ties (Ford, 1985).

The multifaceted nature of competencehas made it difficult for researchers to reachconsensus on interpretation and measure-ment of this concept. Ford (1985) examinescompetence in terms of abilities and recom-mends "reserving the use of the term compe-tence for outcome descriptions, with alterna-tive labels being used for the various pro-cesses (e.g. motivational factors) that helpproduce desired outcomes" (p. 7). He con-tends that "this strategy allows one to defineand assess competence in terms of specifiedgoals and standards that make it clear whatit is a person must be or do in order to beconsidered competent'' (p. 7). Likewise, Ge-rard (1959) and Lawton (1972) suggest thatcompetence may be hierarchically arrangedwith increasing levels of behavioural com-

plexity. Norris (1991), in examining the con-struct of competence, identifies necessary el-ements for demonstration, observation andassessment of performance. These elementsinclude a description of the desired behav-iour, the situation in which it is to occur andprecise delineation of what capability is tobe inferred from the performance under ob-servation. Norris operationally defines com-petence as "something a person is or shouldbe able to do" (p. 332). Similarly, Ford(1985) argues that "a competent person isone who possesses certain specific skills orabilities" and that "these capabilities repre-sent a person's behavioral possibilities" (p.5). Ford also suggests that competence isdependent upon the attainment of goalswithin predetermined parameters or condi-tions. Albee (1980), as well as Cohen, Ridleyand Cohen (1985) support skill acquisitionand the development of competencies asnecessary antecedents in ensuring optimalfunctioning of persons with disabilities. An-thony and Farkas (1982) equate outcomes inrehabilitation with performance of skill bythe patient, in a specified environment "inwhich expected benefits will accrue'' (p. 18).

Baltes (1988) investigated conditions as-sociated with dependency in the elderly, in-cluding physical impairments, learned help-lessness and underestimation of abilities bythe patient. She suggests that with increasingage and biological vulnerability, the elderlyare required to selectively forego certain ac-tivities so that they may optimally performin others. While maximizing performance insome domains is possible, Baltes argues thata general level of high performance cannotbe maintained by the elderly. Baltes, Mayr,Borchelt, Maas and Wilms (1993) furtherdifferentiate between basic and expandedcompetence. The former is defined as routin-ized activities necessary for everyday living;the latter "individual preferences and moti-vations that turn mere existence into pur-poseful living" (p. 659). Baltes et al. (1993)examined competencies ranging from per-sonal care to engagement in leisure and so-


cial activities, required by the elderly for ev-eryday living and a quality life. Consistentwith the selective dependency model out-lined by Bakes (1988), the researchers arguethat a predictive relationship exists betweenbasic and expanded competence and that' 'high levels of functioning in basic compe-tence are a necessary prerequisite for the ex-panded level of competence" (p. 675). Ac-tivity profiles of the elderly further revealed"that the more time required for basic com-petence, the less time is available for ex-panded levels of competence" (p. 675).These findings suggest that due to the neces-sity of basic self-care requirements, the el-derly may "select out" mastery of person-ally satisfying leisure or social activities (ex-panded competence) contributing to qualityliving. However, the researchers recommendfuture longitudinal studies to further investi-gate the relationship between basic and ex-panded competence.

The concept of competence ranges fromthe attainment of the necessary skills andknowledge through to the actual perfor-mance of a function in the real-life environ-ment. Most measurement instruments usedin clinical environments employ surrogatemeasures of function to predict function un-der different circumstances. Thus, for exam-ple, most functional scales measure perfor-mance in the rehabilitation environment topredict function in the home, the latter notbeing directly measured. Likewise, for theLCM it is not always feasible for the thera-peutic recreation professional to directly ob-serve performance under field conditions.The LCM is therefore a clinical tool whichmeasures the range of competencies fromnecessary prerequisites for successful func-tion in the real-life environment, to actualperformance within the clinical or therapeu-tic environment in an attempt to predictlikely performance within a non-therapeuticsetting.

Within the leisure-based philosophy fortherapeutic recreation practice exists a con-tinuum of service designed to optimize lei-

sure ability through the development of func-tional and leisure-related skills. Central tothis philosophy is the premise that ability toengage in meaningful and satisfying leisureexperiences is dependent upon individualskills, attitudes and knowledge related to lei-sure (Peterson & Gunn, 1984; Mundy &Odum, 1979). Riddick (1986) further sup-ports the need for requisite leisure-relatedskills and competencies in order for individ-uals to achieve intrinsically motivated lei-sure experiences. Searle and Mahon (1993)demonstrated the prolonged effect of leisureeducation intervention on the perceivedcompetence of elderly day hospital patients.Since maximizing independence in leisurefunctioning is one of the overall goals of thetherapeutic recreation process, the ability oftherapeutic recreation professionals to accu-rately measure leisure competence is crucialin determining the most appropriate type ofintervention. The LCM identifies specificskills, knowledge and behaviour which in-dividuals must possess in order to functionindependently and successfully in their lei-sure. Deficiencies in competencies identifiedthrough the LCM may then be worked on toimprove or develop behaviours necessary formore independent living and to ensure thatindividuals enjoy their leisure optimally. TheLCM does not measure all competencies re-lated to leisure, nor does it assess skills inspecific activities. Rather, it samples repre-sentative behaviours in seven leisure do-mains identified as necessary indicators ofcompetence in leisure. These domains areassessed to provide a foundation upon whichspecific activity skills may be developed.The LCM has been designed for use in multi-ple settings, including institutions, commu-nity-based agencies or within an individual'shome and with the full spectrum of adult agepopulations. The practical utility of the LCMin widely diverse settings, however, remainsto be demonstrated. This paper concentrateson the use of the LCM with an elderly popu-lation and conclusions concerning the valid-ity and generalizability of the scale are, of


necessity, applicable only to that age group.Studies in other age groups are underway.

The LCM consists of a total of seven sub-scales or items, assessing the domains of lei-sure awareness, attitude, skills, social appro-priateness, group interaction skills, socialcontact and community-based participation(see Table 1).

The first five subscales address the pre-requisites for successful involvement in lei-sure, whereas the remaining two subscales(Social Contact and Community Participa-tion) more directly address actual leisure en-gagement. This introduces a concern thatthere may be a lack of internal consistencywith the LCM and this is addressed in thereliability testing outlined later.

The LCM provides the therapeutic recre-ation professional with a tool to identify initiallevels of patient functioning, establish spe-cific rehabilitation goals and monitor progressand outcomes of therapeutic recreation inter-vention. Each LCM subscale is divided intoseven levels to be consistent with the subdivi-sions of the FIM outlined in Table 2.

These functional levels are consistentacross all seven LCM subscales, althoughspecifically defined within each of the sub-scales. Ratings of 5 or below represent vary-ing degrees of dependent functioning, andratings of 6 and 7 represent degrees of inde-pendent leisure functioning. The individualshould consistently demonstrate attributes orbehaviours outlined in the particular level ofthe LCM subscale. The LCM should be usedto categorize and summarize informationgained through the initial therapeutic recre-ation assessment process. Upon completionof the initial assessment a ranking is assignedto each of the LCM subscales. Patient infor-mation should be obtained from a variety ofsources (e.g. medical record, use of otherstandardized tools, confirmation provided bythe family, direct observation, informationgathered at team rounds, etc.) to ensure themost accurate ratings possible. To achievea score of 7 (complete independence), thepatient must be able to function indepen-

dently without verbal or physical interven-tion. Adaptive devices are not considered aform of dependency if utilized indepen-dently. Intervention strategies may then bedeveloped consistent with identified levelsof functioning on each subscale. Interventionpriorities should be those domains havingthe greatest negative impact on immediateleisure functioning and then proceed to thosedomains having a lesser impact.

Method

LCM DevelopmentThe LCM is being developed using an

iterative process consisting of four phases(pilot, trial, implementatidn and revision).This process is patterned upon the develop-mental process for the FIM (Hamilton,Granger, Zielezney & Tashman, 1987). Thepilot phase, completed September 1990, ad-dressed initial face validity, identified prob-lems therapeutic recreation professionals hadin learning how to use the LCM and exam-ined time to learn and administer the instru-ment. The trial phase, completed February1993, reassessed validity and reliability ofthe LCM with a geriatric rehabilitation popu-lation. The LCM was modified after eachphase based on therapeutic recreation prac-titioner and educator responses.

During the pilot phase, initial face valid-ity was assessed by 5 therapeutic recreationeducators, 2 physicians and 18 therapeuticrecreation practitioners from 7 rehabilitationand psychiatric inpatient facilities. The aver-age number of years of clinical therapeuticrecreation experience of practitioners was6.0. Face validity of the LCM was assessedby asking experienced clinicians who admin-istered the LCM four questions:

1. Were there items that were difficult tounderstand?

2. Were there items that were unneces-sary?

3. Were there items that should beadded?


Table 1.Leisure Competence Measure

LCM Items DescriptionBehaviours Exhibited by

the Client

Leisure Awareness

Leisure Attitude

Leisure Skills

Social Appropriateness

Group Interaction Skills

Social Contact

Community Participation

Client's knowledge andunderstanding of leisure.

Behaviours exhibited and/orfeelings expressed by theclient which suggestattitude toward leisureinvolvement.

Skills possessed by theclient which affect leisureinvolvement.

Specific social behavioursexhibited by the clientwhich affect ability tofunction in leisureactivities.

Client's ability toparticipate in varioustypes of group situations.

Type and duration of socialcontact client has withothers.

Client's overall leisureparticipation pattern.

• personal beliefs• knowledge of leisure

opportunities• awareness of strengths and

weaknesses• realistic expectations• initiative• self-directedness• willingness to develop

new skills and hobbies• demonstration of

enjoyment• ability to make choices• activity skills necessary to

participate in chosenactivities

• ability to identify, locateand access leisureresources

• manners• dress• hygiene• tolerance of others

• sharing; co-operation• task is the focus; minimal

interaction with others• 1:1 interaction• withdrawal; isolation• type, duration, and

frequency of social contact• custodial vs. social contact• type, duration, and

frequency of leisureinvolvement

4. How would you rate the LCM as ameasure of leisure competence on a scale of1 to 5, with 1 = poor, 3 = average, 5 =excellent? (Hamilton, Granger, Zielezny &Tashman, 1987).

Content-related evidence of validity ofthe original six scales appeared to be good.Sixty-seven percent of respondents reportedthat all items were easy to understand; 75%reported that all items were necessary, and;


Table 2.LCM Functional Levels

LEVELS

7 Complete Independence6 Modified Independence5 Modified Dependence4 Modified Dependence with Minimal Assistance3 Modified Dependence with Moderate Assistance2 Modified Dependence with Maximal Assistance1 Total Dependence with Total Assistance

NOHELPER

HELPER

100% reported that no items should be addedto the LCM. After a 3-year trial period ofutilization, however, the need for some mod-ification became apparent (see below). As anadequate measure of leisure competence, theLCM received an overall rating of 4.0 on ascale of 1 (poor) to 5 (excellent). Suggestedrevisions were received from the jury of ex-perts and incorporated into the most recentedition of the LCM. Interrater reliability, as-sessed on a total of 45 patients, was consis-tent and acceptable based on Kendall's Coef-ficient of Concordance. Kendall's W for theoriginal six subscales follow: leisure attitude.82; leisure awareness .80; leisure skills .85;social appropriateness .51; clinical participa-tion .89; social contact .81. These are allsatisfactory with the exception of social ap-propriateness. The reason for this is not clearand requires further study.

Feasibility, the extent to which utiliza-tion of the LCM is possible, practical, andworthwhile was also determined to be sat-isfactory. In an attempt to determinewhether the information obtained by utiliz-ing the LCM justified the cost and effortrequired, therapeutic recreation prac-titioners were asked to record: 1) initialtime to learn to use the LCM; and 2) aver-age time, per patient, to complete the LCM.Average time required, initially, to learnto administer the LCM was 1 hour and 30minutes. Once learned, time required tocomplete the LCM Data Sheet was, on av-erage, 10 minutes or less per patient once

the therapeutic recreation assessment pro-cess was completed. In addition, the juryof experts was asked how useful they con-sidered the LCM to be for use in their facil-ity. Usefulness was rated as a 4.5 on a scaleof 1 (poor) to 5 (excellent). As notedabove, although the LCM appeared to havea solid foundation it was subsequentlymodified. The clinical participation scale,measuring participation in hospital-basedprograms was felt to be less valuable,whereas the need to assess interactionwithin group situations (group interactionskills) and current participation within acommunity setting (community participa-tion) were felt to be significant deficienciesrequiring the development of two new sub-scales.

The trial phase entailed extensive multi-site use of the seven LCM subscales togain practical experience in its usefulness.During the trial phase more extensive va-lidity and reliability testing of the LCMwas conducted with a geriatric rehabilita-tion population (Kloseck, Crilly, Hof-stede & Kerr, 1992). During this phase,practitioners using the LCM reported itsusefulness in documenting functionalgains achieved by patients and providinga patient-centred guide to intervention.Following completion of the therapeuticrecreation assessment, the average time re-quired to complete LCM ratings, by thera-pists during trial phase testing was reportedas five minutes or less per patient.


Reliability (Dependability) andValidity Tests of the LCM

As a preliminary to the implementationphase, we have used data collected duringthe trial phase to evaluate the reliability ofthe LCM under different possible clinicalscenarios. This information will serve toguide clinicians in the use of the scale insubsequent clinical applications and studies.Generalizability theory has been used for thisanalysis. Generalizability theory providesfor a two phase analysis: (1) to determinethe degree of variance that can be attributedto the rater, to the scale and to the studysubjects; and (2) the use of this informationto assess the reliability of the scale underdifferent measurement scenarios (e.g., differ-ent number of raters).

Research ParticipantsParticipants in the study were 114 pa-

tients (47 male, 67 female), mean age 77± 7.63 years (SD), admitted to either theGeriatric Rehabilitation Unit or GeriatricDay Hospital of Parkwood Hospital (Lon-don, Canada). Participant ages ranged from56 to 96 years. Forty-two percent of partici-pants were geriatric rehabilitation admis-sions, while fifty-eight percent were day hos-pital admissions. Primary diagnoses in-cluded cerebrovascular accident (27%);other neurologic conditions (15%); ortho-paedic conditions (12%); arthritis (9%); de-pression (5%); other conditions such as de-mentia, pain syndrome, cardiac or pulmo-nary conditions and alcoholism comprisedthe remaining thirty-two percent. At time ofadmission 79% of participants were living athome, while 21% were admitted from anacute care facility. Usual living companionsprior to admission varied from living alone(44%); with a spouse (41%); with a daughter(8%); son (3%); other relatives (1%); to liv-ing with an attendant (3%).

The Geriatric Rehabilitation Unit is a 34-bed unit providing comprehensive assess-ment and rehabilitation for patients prior to

their return home or to a long-term care insti-tution. The average length of stay on the unitis 30 days. The Geriatric Day Hospital is anout-patient program with an overall purposeof enabling the elderly to live as indepen-dently aS possible. Patients are referred bytheir attending/family physician. The pri-mary focus is on diagnostic and functionalassessment, intensive rehabilitation and man-agement of the medical and functional needsof elderly persons living at home. The aver-age length of stay is 3 months, with patientsattending twice weekly.

ProcedureFor a 12-month period all admissions to

the Geriatric Rehabilitation Unit and the DayHospital were considered for inclusion in thestudy. The only eligibility criterion was antic-ipated discharge into the community. Uponadmission, measures of cognitive function-ing, depression, leisure competence and lifesatisfaction were obtained. All measureswere administered by the research assistantimmediately upon admission.

The research assistant was a therapeuticrecreation specialist, currently employed bythe hospital, with a four year degree in recre-ation and leisure studies and six years clini-cal experience. The research assistant wasreplaced with a second therapeutic recreationspecialist, possessing similar qualifications,for the final five months of the study. Toensure consistency in data collection the re-placement research assistant received four,2-hour training sessions in the administrationof the instruments prior to fulfilling the re-placement role. Inter-rater reliability testingwas conducted by the two research assistantson 47 subjects (41% of the sample).

InstrumentationLife satisfaction was measured by the

Life Satisfaction Index (Neugarten, Havig-hurst & Tobin, 1961), one of two scalesavailable which have been rigorously testedand refined for use with the elderly (Kane &


Kane, 1989). For the purposes of this studythe Life Satisfaction Index B (LSIB) wasused. The LSIB was derived from the origi-nal LSI and consists of 12 open-ended ques-tions scored on a 3-point scale. Depressionwas assessed using the Geriatric DepressionScale (Yesavage, Brink, Rose, Lum, Huang,Adey & Leirer, 1983). The GDS is a self-rating scale consisting of 30 questions re-quiring yes/no answers. A maximum scoreof 30 is possible. Cognitive functioning wasassessed by the Folstein Mini-Mental StateExamination (Folstein, Folstein & McHugh,1975), which gives a maximum score of 30.The Leisure Competence Measure (LCM)was applied to all cases.

Method of Data AnalysisDependability of the LCM. Generalizabil-

ity theory (Cronbach, Gleser, Nanda, & Ra-jaratnam, 1972) provided the framework fordata analysis. This theory provides importantadvantages over classical test theory, partic-ularly for situations in which different mea-surement scenarios (e.g., numbers of raters)are possible (Suen, 1990). Classical test the-ory assumes that measurements reflect onlytwo components: variance attributable totrue score and variance attributable to er-ror (Cronbach, 1990; Feldt & Brennan, 1989;Nunnally, 1978; Shavelson & Webb, 1991;Suen 1990). In classical test theory, tests thatproduce large percentages of "true score"variance are considered to be reliable andtests that produce large portions of error vari-ance have low levels of reliability. In con-trast, generalizability theory allows investi-gators to account for select portions of theerror variance by partitioning various "fac-ets' ' of measurement (e.g., number of raters)that would otherwise be uncontrolled andpooled into the ' 'error component'' if classi-cal test theory procedures were used to as-sess the precision of the measurement (Cron-bach, Gleser, Nanda, & Rajaratnam, 1972;Shavelson & Webb, 1991; Suen, 1990). Withknowledge of the effects of these facets onthe variance of test scores, test users may

make decisions about what measurementscenarios to use to achieve desired levels ofprecision, or "dependability" in their as-sessments (Cronbach, Gleser, Nanda, & Ra-jaratnam, 1972; Shavelson & Webb, 1991;Suen, 1990).

In the case of the current study, measure-ment scenarios involving different numbersof raters and different numbers of dimen-sions of leisure competence were of interest.The effect of the raters' facet was of particu-lar interest because it is clearly impracticalin most clinical situations to have more thanone assessor available. The question aroseas to how much precision (dependability)(Cronbach, Gleser, Nanda, & Rajaratnam,1972; Shavelson & Webb, 1991; Suen,1990) is lost in measuring leisure compe-tence of clients using one as opposed to twoor three raters.

The number of item facets was of concernbecause the items of the LCM, whilefounded in components of the Leisure Abil-ity Model that have been suggested in theliterature (e.g., Peterson & Gunn, 1984) maynot have a sound enough theoretical founda-tion to assume that a sufficient degree ofinternal consistency exists among subscales.If no systematic tendency exists for clientswho rate high on some LCM items to alsohave high ratings on others, a lack of depend-ability in the measure exists and ratings ona greater number of items would be requiredto achieve a measure that provides an accept-able level of precision (dependability). Thus,in addition to assessing the effects of differ-ent numbers of raters (one, two, and threeraters), the generalizability analysis also en-abled us to estimate the dependability ofmeasurements based on the following sce-narios: (1) use of all seven items of the LCM,(2) use of all five items of the LCM clinicalscale, and, (3) use of the two items of theLCM community scale.

Consistent with generalizability theoryprocedures, the analysis followed twophases: (1) a "G," (generalizability) study,and, (2) a "D," (decision) study (Cronbach,


Gleser, Nanda, & Rajaratnam, 1972). In theG study, analysis of variance procedureswere used to estimate the contributions ofresearch participants, items, raters, and theirinteractions to the variance of leisure compe-tence in the population from which the sam-ple was drawn. In generalizability theory,these contributions are referred to as "vari-ance components." Large variance compo-nents reflect a greater contribution of thefacet to variance in the overall measure andsmall variance components reflect lesser con-tributions of the facet to variance in the mea-sure. Typically, a large variance componentfor the research participants is desirable be-cause that component is a function of individ-ual differences in the construct being mea-sured. Small variance components associatedwith the other facets of measurement are typ-ically desired because those components re-flect sources of error in measurement.

The variance components that were cal-culated in the G study phase were then usedin the D study to determine the dependabilityof LCM scores associated with different sce-narios. These procedures are consistent withdescriptions of G and D studies provided bySuen (1990), Shavelson and Webb (1991),as well as in Cronbach, Gleser, Nanda andRajaratnam (1972). As Shavelson and Webb(1991) point out "information from the Gstudy about the magnitude of various sourcesof measurement error (may be used) to eval-uate the effectiveness of alternative designsfor minimizing error and maximizing relia-bility" (p. 12). Specifically, the D study in-volved using the variance components to es-timate "G coefficients" for each measure-ment scenario (Suen, 1990, p. 65). Gcoefficients are analogous to the alpha relia-bility coefficients that are used in classicaltest theory. They represent the extent towhich measures obtained through a givenmeasurement scenario may be considered adependable estimate of individuals' "uni-verse scores". Like alpha reliability esti-mates, G coefficients greater than .85 aredesirable for measurement scenarios that are

used for client assessment purposes. Lesserdegrees of dependability may be acceptablefor tools to be used for research.

In the current study, G coefficients werecalculated through the following formula(Suen, 1990, p. 65):

G = (<7S2 + <Tsi

2/Y)/(<Xs2 + <Tsi

2/

Y + aJ/X + asir2/XY)

In that formula, the s, i, and r subscripts referto the effect of individual differences amongresearch participants ("subjects"), items,and raters, respectively. The variable X rep-resents the number of raters for that particu-lar scenario and Y represents the number ofquestionnaire items in the scenario. Differentvalues for X and Y are substituted into theformula to calculate G coefficients for thevarious combinations of numbers of ratersand items that are of interest. In the currentstudy, G coefficients for nine scenarios werecalculated: One, two, and three rater sce-narios were estimated for the total LCM, theclinical LCM, and the community LCM. Forcomparison purposes, classical test theoryprocedures, including an examination of (1)interrater reliability using the Intra ClassCorrelation and (2) internal consistencymeasured by the Corrected Item Correlation,were also utilized.

Validity. In addition to dependability,analyses were undertaken to generate crite-rion-related evidence of validity of infer-ences that can be made from LCM scores.Within the geriatric population cognitive sta-tus and depression are determinants of func-tion and would be expected to impact onleisure competence. Leisure involvement canbe seen as an important component of lifesatisfaction, and a relationship between thesewould be expected. These analyses involvedcalculations and interpretation of zero ordercorrelations between LCM total, clinical andcommunity scales and scores from the Geri-atric Depression Scale (Yesavage, Brink,Rose, Lum, Huang, Adey & Leirer, 1983),the Life Satisfaction Index (Neugarten, Hav-


Table 3.G Study and D Study Results: Leisure Competence Measure

Raters

123

SubjectsItemsRatersS x IS X RI X RS X I X R

Total LCM

.926

.962

.974

n

11472

————

G Coefficients (D Study Results)

Clinical LCM

.897

.945

.963

Community LCM

.911

.954

.969

Variance Components (G Study Results)

«•(%)

1.115 (39%).896 (31%).000 (0%).505 (18%).053 (2%).003 (.1%).291 (10%)

«•(%)

1.171 (52%).375 (17%).000 (0%).335 (15%).082 (4%).003 (.1%).302 (13%)

o-(%)

1.417 (49%).757 (26%).001 (0.01%).487 (17%).081 (3%).007 (.2%).161 (6%)

ighurst & Tobin, 1961, and the Mini-MentalState Examination (Folstein, Folstein &McHugh, 1975).

Results

Classical Test TheoryResults from classical test theory proce-

dures support the LCM as an instrumentfrom which dependable measures may betaken. Interrater reliability testing was con-ducted on 47 patients. The Intra Class Corre-lation was quite satisfactory for the totalLCM scale (r = .91). Rater agreement forthe seven individual subscales of the LCMranged from r = .71 to r = .91. Internalconsistency of the LCM as measured byCronbach's Alpha Reliability Coefficientwas also quite satisfactory (a = .92).

Generalizability TheoryResults of the dependability analyses are

summarized in Table 3. As can be seen, G

coefficients suggest that dependable mea-sures can be taken from each of the measure-ment scenarios. Coefficients ranged from.897 for the one rater, clinical LCM measureto .974 for the three rater, total LCM measure.

These results suggest that one rater withqualifications similar to those of the ratersused in this study is sufficient to producedependable LCM measurements in a geriat-ric rehabilitation setting, for the total LCMand either of the clinical or communityscales separately.

ValidityEach of the three zero order correlation

coefficients was significant and in the hy-pothesized direction as outlined in Table 4.Correlations between the total LCM scoreand the GDS, LSI, and MMSE ranged from.40** to .54**. For the clinical scale, the co-efficients ranged from .37** to .44**, andfor the community scale from .23* to .62**(*p < .01; **p < .001). These results suggest


Table 4.Correlations between LCM and

Validation Measures

GDS MMSE LSI

LCM-Total .54**LCM-Clinical .44**LCM-Community .62**

.40** .47**43** .37**.23* .59**

*p < .01; **p < .001.GDS: Geriatric Depression Scale. MMSE:

Mini-Mental State Examination. LSI: Life Satis-faction Index.

that valid inferences may be made about lei-sure competence based on LCM scores ob-tained with similar patient groups and raters.

DiscussionThe generalizability theory analysis sup-

ports the use of the LCM employing onerater and all seven subscales of the LCM.The primary purpose for constructing theLCM was to (1) provide a consistent meansof identifying requisite skills for leisurefunctioning, (2) guide intervention, and (3)provide a systematic method of gauging anddocumenting changes in patient functioning.A tremendous need exists for standardizedinstruments in therapeutic recreation. Evi-dence presented here suggests the LCM maymeet this need.

In order to be of use, the scale must be:a) accurate (i.e. measure what it purports tomeasure); b) reproducible (i.e. produceclosely similar measures from time to timeand with different evaluators) and; c) de-pendable (i.e. have internal consistency andbe constant across scenarios likely in clinicalpractice). In the use of scales, accuracy isaddressed through the concept of validity.Of particular importance are content and cri-terion-related evidence of validity. The for-mer was addressed through the panel of ex-perts approach and the latter by demonstra-ting the scale correlated to a reasonable

degree with other parameters which mightbe expected to relate to leisure competence.Significant correlations were found withmeasures of depression and cognition. TheLCM in turn correlated significantly with lifesatisfaction. These correlations are based onconceptual linkages that we have establishedbetween leisure competence and these vari-ables. The conceptual linkages are not, how-ever, well founded in theory. As a result suchmoderate correlations are to be expected.

Both the generalizability analysis and theclassical test theory analyses indicated thatthe LCM produces reliable scores. From thegeneralizability analysis it is clear that theLCM maintains dependability with only onerater and does so for the total scale as wellas for both the clinical and community sub-scales. G coefficients of greater than .89were found for the total LCM and subscalesfor the worse case scenario (only one rater)indicating high internal consistency of thetotal scale and subscales. The high Cron-bach's alpha coefficient that was found in theclassical test theory analysis provides furtherevidence of internal consistency of the LCM.

The LCM allows identification of basiccompetencies, or lack thereof, required forsuccessful leisure functioning. Changes infunctioning may be measured, analyzed andcompared with similar patient groupings. Areview of data collected, over time, mayhighlight areas warranting further explora-tion through research. Analysis of factorssuch as type of impairment, living arrange-ments, support networks, etc. may identifytrends which will have significant impact onthe scope of therapeutic recreation servicesoffered. The LCM may prove to be usefulwith other populations and in other settings.Testing in this study has, however, been lim-ited to a geriatric rehabilitation population.Much remains to be done to confirm the ap-plicability of the LCM across a variety ofpopulations. Future studies are needed to ex-plore the variability of the effectiveness ofintervention with different disability group-ings. Evaluation of gain scores and the de-


velopment of normative data will enablemore accurate prediction of rehabilitationpotential. Similarly, further research is nec-essary to establish a definite link betweenleisure competence, leisure satisfaction andlife satisfaction. This is particularly im-portant for those individuals whose physical,behavioural or cognitive deficits precludecommunity vocational placement.

In order to support therapeutic recreationas an essential professional service, docu-mentation of behavioural and functionalchange is critical. As a tool from which validinferences may be drawn, the LCM has di-rect practical application in clinical practice,resource allocation and utilization, programevaluation, monitoring quality of interven-tion and justifying accreditation. Scientifi-cally established evidence has, over theyears, become a driving force behind healthcare decisions. Measurement of patient-spe-cific outcomes provides a means of de-termining the effect and efficiency of reha-bilitative intervention on individuals withdisabilities. The current emphasis on the de-velopment of outcome measurement systemsattests to the need for more intensive investi-gation in this area.

References

Albee, G. W. (1980). A competency model mustreplace the defect model. In L. A. Bond & J. C.Rosen (Eds.), Competence and coping duringadulthood (pp. 75-104). Hanover, N.H.: Uni-versity Press of England.

American Health Consultants (1993). Computerprograms facilitate outcomes management, pa-tient evaluation. Hospital Rehabilitation: TheMonthly Management Advisor for Rehabilita-tion Professionals, 2(5), 53-56.

Anthony, W. A. & Farkas, M. (1982). A clientoutcome planning model for assessing psychi-atric rehabilitation intervention. SchizophreniaBulletin, 8, 13-38.

Baltes, M. M. (1988). Etiology and maintenanceof dependency in the elderly: Three phasesof operant research. Behavior Therapy, 19,301-319.

Baltes, M. M., Mayr, U., Borchelt, M., Maas, I. &Wilms, H. U. (1993). Everyday competencein old and very old age: An inter-disciplinaryperspective. Aging and Society, 13, 657-680.

Baltes, M. M. & Wahl, H. W. (1992). The behaviorsystem of dependency in the elderly: Interac-tion with the social environment. In M. G. Ory,R. P. Abeles & P. D. Lipman (Eds.), Aging,health and behavior (pp. 83-106). NewburyPark, California: Sage.

Boschen, K. A. (1985). Client outcome measuresin rehabilitation. The Ontario Psychologist,17(3), 3-9.

Burlingame, J. & Blaschko, T. M. (1990). Assess-ment tools for recreational therapy. Seattle:Frontier Publishing.

Cohen, B. F., Ridley, D. E. & Cohen, M. R. (1985).Teaching skills to psychiatric disabled persons.In M. A. Marlowe Jr. & R. B. Weinberg (Eds.),Competence development theory and practicein special populations (pp. 118-145). Illinois:Charles C. Thomas.

Cronbach, L. (1990). Essentials of psychologicaltesting (5th ed.). New York: Harper & Row.

Cronbach, L., Gleser, G., Nanda, H., & Rajarat-nam, N. (1972). The dependability of behav-ioural measurements: Theory of generalizabil-ity of scores and profiles. New York: JohnWiley.

Dunn, J. K. (1984). Assessment. In C. Peterson, &S. Gunn (Eds.), Therapeutic recreation pro-gram design: Principles and procedures (2nded.) (pp. 270). Englewood Cliffs: Prentice-Hall.

Feldt, L., & Brennan, R. (1989). Reliability. In R.Linn (Ed.), Educational measurement (3rd ed.)(pp. 105-146). New York: MacMillan.

Folstein, M. R., Folstein, S., & McHugh, P. R.(1975). Mini-mental state: A practical methodfor grading the cognitive status of patients forthe clinician. Journal of Psychiatric Research,12, 189-204.

Ford, M. E. (1985). The concept of competence:Themes and variations. In M. A. Marlowe Jr. &R. B. Weinberg (eds.), Competence develop-ment theory and practice in special popula-tions (pp. 3-49). Illinois: Charles C. Thomas.

Forer, S. K. (1982). Functional assessment instru-ments in medical rehabilitation. Journal of theOrganization of Rehabilitation Evaluators,2(4), 29-41.

Gerard, R. (1959). Aging and levels of organiza-


tion. In J. Birren (Ed.), Handbook of aging andthe individual (pp. 264-273). Chicago: Uni-versity of Chicago Press.

Hamilton, B. B., Granger, F. S., Zielezny, M., &Tashman, J. S. (1987). A uniform national datasystem for medical rehabilitation. In M. J. Fuh-rer (Ed.), Rehabilitation outcomes: Analysisand measurement (pp. 137-147). Baltimore:Paul H. Brookes.

Harter, S. (1978). Effectance motivation reconsid-ered: Toward a developmental model. HumanDevelopment, 21, 34—64.

Hogg, J. R., & Heller, K. (1990). A measure ofrelational competence for community-dwellingelderly. Psychology and Aging, 5(4), 580-588.

Howe, C. Z. (1984). Leisure assessment instru-mentation in therapeutic recreation. Therapeu-tic Recreation Journal, 18(2), 142-152.

Johnston, M. V., Keith, R. A., & Hinderer, S. R.(1992). Measurement standards for interdisci-plinary medical rehabilitation. Archives ofPhysical Medicine and Rehabilitation, 73, S—3-S-23.

Kane, R: A., & Kane, R. L. (1989). Assessingthe elderly: a practical guide to measurement.Massachusetts: Lexington.

Kloseck, M., Crilly, R. G., Hofstede, K. & Kerr,P. (1992). Determinants of life satisfaction ina geriatric rehabilitation population. Journal ofthe American Geriatrics Society, 40(10),SA52. (From Abstracts of AGS/AFAR Papers,1992, Abstract No. P130).

Lawton, M. P. (1972). Assessing the competenceof older people. In D. P. Kent, R. Kasten-baum, & S. Sherwood (Eds.), Research, plan-ning and action for the elderly: The power andpotential of social science (pp. 122-143). NewYork: Behavioral Publications.

Mundy, J., & Odum, L. (1979). Leisure education:Theory and practice. New York: John Wiley.

Neugarten, B. L., Havighurst, R. J., & Tobin, S. S.

(1961). The measurement of life satisfaction.Journal of Gerontology, 16, 134-143.

Norris, N. (1991). The trouble with competence.Cambridge Journal of Education, 21(3),331-341.

Nunnally, J. (1978). Psychometric theory. NewYork: McGraw-Hill.

Olson, P. A. & Freeman, L. (1979). Denning com-petence in teacher licensing usage. In P. S.Pottinger & J. Goldsmith (Eds.), Defining andmeasuring competence (1-10). San Francisco:Jossey-Bass.

Peterson, C, & Gunn, S. (1984). Therapeutic rec-reation program design: Principles and proce-dures. (2nd ed.). Englewood Cliffs: Prentice-Hall.

Riddick, C. C. (1986). Leisure satisfaction pre-cursors. Journal of Leisure Research, 18(4),259-265.

Scofield, M. E., Berven, N. L., & Harrison, R. P.(1981). Competence, credentialling and the fu-ture of rehabilitation. Journal of Rehabilita-tion, 47(1), 34.

Searle, J. & Mahon, M. (1993). The effects of aleisure education program on selected vari-ables: A 3 month follow-up investigation.Therapeutic Recreation Journal, 27(1), 9-21.

Shavelson, R., & Webb, N. (1991). Generalizabil-ity theory. Newbury Park: Sage.

Stumbo, N. J. (1990). Overview of state creden-tialling concerns: A focus on licensure. Thera-peutic Recreation Journal, 24(2), 40-59.

Suen, H. (1990). Principles of test theories. Hills-dale: Lawrence Erlbaum and Associates.

White, R. W. (1959). Motivation reconsidered:The concept of competence. PsychologicalBulletin, 66, 297-323.

Yesavage, J. A., Brink, T. L., Rose, T. L., Lum,O., Huang, V., Adey, M., & Leirer, O. (1983).Development and validation of a geriatric de-pression screening scale: A preliminary report.Journal of Psychiatric Research, 17(1), 37-49.


Leisure Competence Measure: Development and Reliability ...

Documents

Transcript of Leisure Competence Measure: Development and Reliability ...