Evaluating a Homemaking Assessment for Broader … fileIt has also been suggested that fraudulent...
Transcript of Evaluating a Homemaking Assessment for Broader … fileIt has also been suggested that fraudulent...
Evaluating a Homemaking Assessment for Broader Application to Practice
Nicole Matichuk, BKIN1, Liv Brekke, MPA1, Hilary Drummond, OT(C) 2, & Susan Forwell,
PhD, OT(C) 1
1Department of Occupational Science & Occupational Therapy, University of British Columbia
2Creative Therapy Consultants
1
This manuscript was prepared for the purpose of RSOT 547, in accordance with CJOT
article guidelines, with the exception of blinding the manuscript. The manuscript was not
blinded for RSOT 547 project clarity.
2
Abstract
Background. In response to third party payer needs, an assessment was developed to provide a
fair, objective, and consistent approach to quantifying the assistance an individual requires with
homemaking tasks as a result of accident or disability. Purpose. To evaluate psychometric
properties of this homemaking assessment for use with community-dwelling adults with injury or
disability. Methods. A retrospective chart review of 113 cases examined content validity and
internal consistency, and a prospective design of 9 cases tested interrater reliability. Findings.
Retrospective data demonstrated that assessment subscales have good internal consistency, items
are categorized appropriately, and the assessment measures a single construct overall. Several
latent constructs representing types of task demands were identified across assessment items.
Prospective data showed early evidence of acceptable interrater agreement. Implications.
Preliminary evidence is established for the psychometric integrity of a novel homemaking
assessment, providing the foundation for broader use of this assessment in community practice.
Keywords: Housekeeping, Measurement, Occupational Therapy, Psychometrics, MVA (Motor
Vehicle Accident)
3
Introduction
Instrumental activities of daily living (IADLs), such as homemaking, are integral to daily
life. IADLs are habits or routine tasks that allow individuals to be independent and engaged
within their homes and communities (Clemson, Bundy, Unsworth & Singh, 2009). They go
beyond personal care to include tasks requiring higher level cognitive processes and interactions
with varying environments and people. The ability to perform these everyday tasks has been
directly related to quality of life and well-being (Clemson et al., 2009). Aging, falls, motor
vehicle accidents (MVAs), and other injuries and disabilities have the potential to impact a
person’s ability to perform IADLs (Clemson et al., 2009). Decreased performance on IADL
tasks, like homemaking, can also be an indicator of higher care needs or supports (Clemson et
al., 2009). Occupational therapists are essential in assessing IADL function of their clients, yet
no assessment exists that evaluates homemaking in depth.
The CTC Homemaking Assessment was developed in direct response to the need to
provide a fair, objective, and consistent approach to assessing clients who might require supports
for housekeeping and home management tasks due to accident or disability. It is a
comprehensive, in-depth assessment tool designed by occupational therapists to determine the
percentages of housework tasks an individual can perform. While other IADL assessments
existed at the time of its development, there was no tool which focused on homemaking
specifically and which provided a concrete percentage of tasks an individual could complete. A
percentage was necessary because the majority of clients undergoing the CTC Homemaking
Assessment had been involved in MVAs, and the primary insurer provides supports only for
those individuals who can no longer perform the majority (i.e. 50 percent) of their previous
homemaking tasks.
4
The population of individuals injured in an MVA is large and increasing. Between July
2014 and June 2015, the number of reported new injury claims as a result of an MVA in British
Columbia increased by 10 per cent over the previous 12 months, to a total of 67,700 new claims
(ICBC, 2015). In addition, the costs associated with these claims (which cover payouts for pain
and suffering, future care, and loss of wages) have increased by 64 per cent since 2008 (ICBC,
2015). It has also been suggested that fraudulent claims may be on the rise (ICBC, 2015).
Together, the increase in bodily injury claims, the associated costs on the system, and concerns
about fraud all reinforce the need for valid and reliable assessments to be used in determining
what benefits an injured individual should receive. To date, however, there has been no
consensus on what method is best used for determining the need for these supports. Since its
development, the CTC Homemaking Assessment has been well accepted by clinicians, lawyers
and insurers; however, the measurement properties of the assessment have yet to be
systematically evaluated.
Issues in IADL Measurement
To date, there has been no research into the development and application of assessment
instruments specifically related to homemaking. For the purposes of informing this study, we
instead examined the literature on the assessment of IADLs, a broader construct which generally
includes homemaking tasks. Within the literature, what exactly constitutes IADLs remains the
subject of debate. IADLs are typically defined as activities that help facilitate independent living,
such as meal preparation, taking medication, managing money, using a telephone, shopping, and
housekeeping (Gold, 2012; Law, 1993; Law, Baum, & Dunn, 2005). Various ways of further
categorizing these activities have been discussed. Some authors have sorted IADLS into three
categories: getting about, household-based activities, and leisure-oriented activities (Law, Baum,
5
& Dunn, 2005). Leung et al. (2011) highlighted that IADLs have been theoretically divided into
physical and more complex cognitive tasks. However, empirical studies on the physical-
cognitive division of IADLs have been inconclusive; some show that IADLs can be expressed as
two separate constructs, while others show that IADLs are highly correlated and therefore
unidimensional (Leung, Chi, & Leung, 2011). Reuben et al. (1990) proposed advanced activities
of daily living (AADLs), another construct distinct from IADLs. These activities “are volitional,
influenced by cultural and motivational factors, expressing a personal engagement in satisfying
activities which are beyond what is needed to be independent” (De Vriendt et al., 2012, p. 975).
Examples of AADLs have included the use of household appliances (e.g. washer and dryer) and
advanced kitchen activities such as preparing a meal with a large number of ingredients (De
Vriendt et al, 2012). In contrast to all these construct theories, the CTC Homemaking
Assessment categorizes homemaking tasks according to task difficulty. It includes 29 items
divided into three categories: light, medium, and heavy tasks. (See Table 1 for a complete list of
assessment items).
Existing IADL assessments have been criticized for the number of different tests for
different populations, reliance on self-report rather than clinician observation, being cumbersome
and lengthy to administer, being based on varying conceptual frameworks, and relying on
outdated or specific cultural perspectives (Bottari, Dassa, Rainville & Dutil, 2010; Law, 1993;
Letts et al., 1994). Unlike the CTC Homemaking Assessment, which has been used with
individuals with a wide range of physical, cognitive, and/or mental health issues, the content of
many assessments is often developed for specific impairments, not on a broader disability basis
(Law, 1993). Some IADL assessments have also received only limited psychometric testing
(Clemson et al., 2009; Law & Letts, 1989). Instrument studies typically need a sample size of at
6
least 50 to support reliable conclusions, yet few occupational therapy instrument studies have
met this threshold (Yuen & Austin, 2014). In short, no reliable gold standard for IADL
assessment has yet to clearly emerge (Law, Baum, & Dunn, 2005).
IADLs can be evaluated via self-report, proxy report, direct observation, and/or medical
chart data extraction (Law, Baum & Dunn, 2005). Performance-based assessments are often
considered more effective. However, some evidence has suggested that these are not
psychometrically superior to self-report questionnaires, and that self-report measures generally
correlate well with performance-based ones when measuring the same domain of disability
(Coman & Richardson, 2006; Myers, Holliday, Harvey & Hutchinson, 1993). A more recent
review of the psychometric properties of instruments used to assess workers applying for
disability benefits concluded that “a combination of patient self-reports, performance tests and
medical examination […] seems the most solid tool for assessing functional limitations in work
disability benefits” (Spanjer et al., 2011, p. 2149). In keeping with this perspective, the CTC
Homemaking Assessment relies on information obtained both through therapist observation of
client performance in some homemaking tasks, as well as client-therapist interview.
Another important criterion for developing a measurement strategy is relevance and
appropriateness to the client’s life (Law, Baum, & Dunn, 2005). Research comparing assessment
methods highlights the importance of developing measurement tools which are ecologically
valid. From an occupational therapy perspective, this reflects a core assumption that engagement
in activity occurs within an individual’s unique physical, social, institutional, cultural and
temporal context. Therefore, “when conducting an evaluation [...] the environmental context in
that situation will have an influence on the person’s performance” (Law, Baum, & Dunn, 2005,
p. 25). In examining the effect of assessment environment, Rogers et al. (2003) found that self
7
and proxy reports were more concordant with a patient’s actual in-home performance of IADLs
than clinical judgment or performance testing in clinic. For physical IADLs specifically, they
found that self-report, proxy report, clinical judgment and clinic-based performance all
significantly underestimated a patient’s actual abilities. This suggests that, ideally, IADL
assessments should be conducted in the environment in which the client performs them. In
addition, IADL assessments need to account for the fact that tasks can be both gender and culture
specific (Law, Baum, & Dunn, 2005) and that individuals demonstrate different patterns of
occupational performance across their day and lifetime (Law, 1993). The CTC Homemaking
Assessment is administered in the client’s own home and is therefore grounded in the client’s
own context. Similarly, the assessment can account for the particular performance patterns of the
individual client. If an item is not applicable or appropriate for a client, it is scored as “not
applicable” and simply not used in calculating the assessment results.
Instrument Development and Psychometrics
In order to systematically examine the reliability and validity of any IADL assessment,
the purpose of the tool must first be established; which specific psychometrics to evaluate for a
certain instrument depends, in part, on the purpose of the instrument itself (Law, 1987; Law &
MacDermid, 2014). Descriptive measures give a snapshot of a person at one moment in time,
predictive measures set criteria against which to compare a person’s current status, and
evaluative measures assess change over time (Law & Letts, 1989). Many IADL tools can be used
for more than one purpose. The CTC Homemaking Assessment has historically been used for
both descriptive and evaluative reasons. This study focused on examining the measurement
properties of the CTC Homemaking Assessment primarily from a descriptive perspective.
8
Descriptive instruments should have evidence of content and construct validity, internal
consistency, and interrater reliability (Law, 1987).
The validity of an instrument is the degree to which an assessment measures what it is
intended to measure (Law & Letts, 1989). Content validity assesses the extent to which an
instrument’s items comprehensively represent all the characteristics of a construct it is designed
to measure (Law, 1987; Mokkink et al., 2010; Yuen & Austin, 2014). Methods of establishing
content validity include judgment by clinicians or patients, or statistical methods, such as
exploratory or confirmatory factor analysis (Law, 1987; Mokkink et al., 2010). With the CTC
Homemaking Assessment, a number of occupational therapists were involved in an iterative
process of item development and refinement over the course of several years. In addition, the
assessment has been well accepted by insurers and lawyers. However, statistical tests of content
validity have not yet been conducted.
Reliability speaks to the stability, consistency, and dependability of an instrument’s
measurements (Law, 1987), and includes properties such as internal consistency and interrater
reliability. Internal consistency refers to the “homogeneity of items or scores within an
instrument” (Law & MacDermid, 2014), while interrater reliability assesses how consistently an
instrument performs when administered by different practitioners (Vroman & Stewart, 2014).
Neither of these properties has yet to be examined in the CTC Homemaking Assessment.
Study Purpose
While the CTC Homemaking Assessment has been used with strong acceptance to date,
there is increasing potential for it to be used beyond a single private practice. Systematic
evaluation of the measure is necessary so that other clinicians can understand the properties of
the assessment and determine whether they wish to adopt it in their own practice. Therefore the
9
purpose of this study was a preliminary evaluation of aspects of the validity and reliability of the
CTC Homemaking Assessment for use by occupational therapists with community-dwelling
adults. Specifically this study sought to answer the following questions: (1) what characteristics
and performance patterns can be identified among clients with whom the assessment has been
used; (2) what constructs can be identified among the assessment items; (3) to what extent are
scores on items within each of the assessment’s subscales homogenous; and (4) how consistently
does the assessment perform when scored by different practitioners?
Methods
Study Design
This study focused on instrument testing of the CTC Homemaking Assessment using
quantitative methods. The overall design included both a retrospective chart review and a
prospective component.
Participants
The target population for this study was community-dwelling adults with injury or
disability. Two convenience samples were used, comprised of past and present adult (19+)
clients of Creative Therapy Consultants who had the CTC Homemaking Assessment
administered to them at least once. The sample for the retrospective phase included inactive
clients for whom records were available, while the sample for the prospective phase included
current clients who provided consent for their data to be used. The study excluded participants
under the age of 19. Participants in the prospective component were recruited by CTC clinicians
from among their caseload. Prior to being assessed, potential prospective participants received an
introductory letter describing the study and provided signed, informed consent.
Data Collection
10
All data collection was facilitated by CTC clinicians and staff and provided to the
research team in de-identified form. Unique identifiers were created for each participant and
clinician, and the research team was not able to associate these unique identifiers to client or
clinician names.
Retrospective Phase. A CTC research assistant (RA) reviewed charts of inactive clients
assessed between 2005 and 2016 in order to identify participants who met the study criteria; a
total 113 charts were identified during the study timeframe and all were included in the study
analysis. From these charts, the RA extracted relevant demographic information and captured it
in an audit form which was then provided to the research team, along with de-identified copies of
the assessment results. For retrospective participants, all assessments had been administered and
scored by trained CTC clinicians.
Prospective Phase. With participant consent, a CTC clinician administered the
Homemaking Assessment to clients (n=9) with an occupational therapy student present to serve
as a second rater. The student was oriented to the assessment by a CTC clinician, and was
provided with written scoring instructions. The student observed and independently scored the
assessment, and did not share or discuss their findings with the clinician. Fully independent
assessment administration was not feasible within the study timelines, although it is recognized
that it is recommended practice (Mokkink et al., 2010).
Similar to the retrospective phase, the CTC RA extracted relevant demographic
information from client charts and captured it in an audit form which was then provided to the
research team, along with de-identified copies of the assessment results from both the clinician
and the student.
Data Analysis
11
Descriptive statistics were calculated using Microsoft Excel 2010 (Microsoft
Corporation, 2010). All other statistics were completed with IBM SPSS Statistics Version 24
(IBM Corporation, 2016).
Retrospective. Demographic data, including age, gender, and reasons for referral, were analysed
from the sample. Detailed diagnosis information was reviewed and summarized into six general
categories: pain, fracture and/or joint issues, soft tissue injuries, mental health diagnoses, sensory
changes, head injuries and/or cognitive issues, and chronic disease.
The distribution of total weighted scores on the assessment was examined using a
histogram. Performance trends were also identified and recorded for each participant from their
unweighted percentage scores on each assessment subscale. For example, if a participant’s score
on light tasks was higher than their score on medium tasks, and their score on medium tasks was
in turn higher than their score on heavy tasks, then the performance trend was recorded as
“consistent downward trend.” The number of participants whose results demonstrated each
identified trend was totalled in order to characterize typical performance across assessment
subscales.
For all further statistical calculations, variables (assessment items) with greater than 45
percent missing data were removed from the analysis in order to achieve significant results. Ten
variables were excluded from analysis in total, including: hand-washing clothes, using the
microwave, using the oven, and drying dishes (light tasks); ironing, handling pots and pans,
reaching into upper and lower cupboards, and using the dishwasher (medium tasks); and
vacuuming stairs, and cleaning the freezer (heavy tasks). Some missing values were the result of
the sample including two slightly different versions of the assessment; therefore, not all cases
included all variables. Other missing values were the result of the design of the assessment tool
12
itself, which does not assess an individual’s performance on tasks they typically do not perform.
As a result, tasks such as ironing, drying dishes, and using the dishwasher were scored as “not
applicable” for a large percentage of study participants.
Using the remaining 24 variables (see Table 1), Pearson correlation coefficients
(Pearson’s r) were calculated between all assessment items and between unweighted percentage
scores on each subscale, and the results provided in correlation matrices. Production of these
matrices was designed to enable the research team to identify the extent to which items in a
single subscale were related to each other, as well as the extent to which items in different
subscales did not correlate, in order to help inform the categorization of scale items. R values
around 0.10 were interpreted as a weak relationship, 0.30 as moderate, and 0.50 and above as
strong (Kellar & Kelvin, 2013). The significance level was defined as p > 0.01.
[Table 1 here].
An exploratory factor analysis was performed to help assess the unidimensionality of the
assessment and of each subscale, and to examine for latent constructs within the data and
(Mokkink et al., 2010; Yuen & Austin, 2014). Due to the necessity of excluding a number of
variables, further item analysis and reduction was not carried out prior to running the factor
analysis. A principal axis factor extraction and Varimax rotation were used in order to help
simplify interpretation (Kellar & Kelvin, 2013). The required eigenvalue for a factor was set at 1
and missing data was excluded listwise. The researchers then examined the produced factor
loadings from an activity analysis perspective (Crepeau, 2014) in order to come to agreement on
the potential underlying constructs that might be represented by these factors.
Internal consistency was analyzed by calculating Cronbach’s alpha, the most commonly
used statistic for this purpose (Law & MacDermid, 2014; Mokkink et al., 2010; Vroman &
13
Stewart, 2014). As recommended by Mokkink et al. (2010), Cronbach’s alpha was calculated
separately for each subscale in the assessment in order to determine how well the subscale items
measured the same construct (i.e. light tasks, medium tasks, heavy tasks). Missing data was
excluded listwise. For the purposes of this study, a Cronbach’s alpha value above 0.70 was
considered adequate, 0.8-0.9 good, and above 0.9 excellent (Vroman & Stewart, 2014).
Prospective. Basic participant demographics only (gender and age) were analysed for the
prospective sample (n=9). Due to the small sample size, Cohen’s kappa was not calculated in
order to assess interrater reliability as might be expected (Mokkink et al., 2010; Vroman &
Stewart, 2014). Instead, a table was created allowing for visual comparison of the two raters’
results. The mean difference between scores on each subscale was also calculated, excluding
observations from one case which was determined to be an outlier.
Findings
Retrospective
Demographics. Participant demographic information is summarized in Table 2. Mean
participant age was 50.6 ± 15.2 years. The youngest participant was 19 years old, while the
oldest was 92. Three quarters of participants were female. The most frequent reasons for referral
to CTC for a Homemaking Assessment were MVA and cost of future care assessment. The
majority of clients had diagnoses of pain, fracture and/or joint issues, and soft tissue injuries. Of
the sample analyzed, 86.7 per cent of clients had two or more diagnoses at the time of
assessment.
[Table 2 here].
Overall patterns of response. Total weighted scores for participants were normally
distributed, with the greatest clustering of scores between the 40 to 60 per cent mark. Five
14
distinct performance patterns were identified across assessment subscales: consistent downward
trend, partial downward trend, no trend, flat trend, and consistent upward trend. Consistent
downward trend indicated that performance diminished as task difficulty increased, meaning
clients generally performed best on light tasks and worst on heavy tasks. Over two-thirds of
participants (67.3 per cent) performed with a consistent downward trend. A partial downward
trend was identified among 4.4 per cent of participants, indicating that their performance level
diminished only with heavy tasks.
No trend was identified across 22.1 per cent of participants. For these participants,
performance varied across light, medium, and heavy tasks. For example, a participant might
score well on light tasks, poor on medium tasks, and mid-range on heavy tasks. A flat trend,
indicating consistent scores across all three subscales, was identified for 5.3 per cent of
participants. Finally, 0.9 per cent of participants (n=1) performed with a consistent upward trend,
meaning their performance improved as subscale difficulty increased.
Intercorrelation of subscales and items. All three assessment subscales (light, medium,
heavy tasks) were significantly and positively correlated with each other. The light and heavy
subscales were moderately correlated (r of 0.491). The light and medium subscales demonstrated
a stronger relationship (r of 0.649), while the medium and heavy subscales were also strongly
correlated (r of 0.773).
In terms of assessment items, all tasks within the light subscale were weakly to
moderately positively correlated with each other and most of these relationships were significant.
The most strongly related items were folding clothes and putting clothes away (r of 0.573).
Generally, items in the medium subscale were also weakly to moderately positively correlated
with each other, with the strongest significant relationship between taking clothes out of the
15
dryer and transferring clothes from the washer to the dryer (r of 0.572). The relationship between
cleaning the sink and transferring clothes from washer to dryer was negative (r of -0.024) but
very weak and not statistically significant. All items in the heavy subscale were also weakly to
moderately positively correlated and virtually all of these relationships were significant.
Several particularly strong positive relationships were also observed between items from
different subscales. Cleaning the tub (a heavy task) and cleaning baseboards (a medium task)
reported an r value of 0.694. Sweeping floors (a light task) and washing floors (a medium task)
had an r value of 0.601. Only a single negative correlation was observed between items across
different subscales (cleaning windows and cleaning mirrors); however, this relationship was very
weak (0.10) and not statistically significant (See Supplementary Materials for complete
correlation matrices).
Extracted factors. Table 3 shows the results of the rotated Varimax factor loadings. Six
factors were extracted in total, accounting for 68.4 per cent of the cumulative variance.
Approximately half of the analyzed variables loaded onto Factor 1, suggesting a common
underlying construct among the tasks on the assessment. Variables which loaded onto Factor 1
were similar in that they were all tasks which could require bending at the waist or stooping to
complete. Factor 1 was therefore assigned the label of “Bending Tasks.” These bending tasks
included a mix of items from both the heavy and medium subscales.
[Table 3 here].
Four variables loaded most strongly onto Factor 2. These variables were determined to be
tasks that involved primarily use of the hands and arms to complete (without bending). Thus
Factor 2 was labelled “Upper Extremity Tasks.” These upper extremity tasks were primarily
items from the light subscale, with one medium task. Two variables loaded onto Factor 3,
16
representing perhaps a distinct category of “Clothing Tasks.” These variables did not load
strongly onto any other factors, and were both items from the light subscale. Similarly, a further
two variables loaded onto Factor 4. No commonalities could be surmised from these variables
and therefore Factor 4 was not assigned a label and was not considered useful in instrument
development considerations. Factor 5 was labelled “Load Tasks.” Both variables which loaded
most strongly onto this factor involved the manipulation of a potentially heavy load (i.e. full
garbage bags and wet laundry). Factor 6 was labelled “Planning Tasks” to reflect the potentially
more cognitively complex nature of the activities, e.g. grocery shopping, as well as the potential
need to plan these activities further in advance (e.g. cleaning the windows might require
preparing special equipment and/or determining a day in the future to complete this activity).
In general, the exploratory factor analysis suggested alternative groupings of items on the
CTC Homemaking Assessment. However, these results were not used to definitively change the
organization of the assessment itself. This was due to the fact that the sample size did not reach
the recommended minimum ratio of 10 cases to each variable (Kellar & Kelvin, 2013).
Reliability of Subscales. All three existing subscales demonstrated good internal consistency.
Cronbach’s alpha values were as follows: 0.78 for light tasks; 0.77 for medium tasks; and 0.80
for heavy tasks. No items were identified that, if deleted, would improve the alpha value of their
respective subscales.
Prospective
Demographics. Mean age for the prospective sample (n=9) was 45.1 ± 9.5 years, with a range of
27 years. Two-thirds were female (n=6).
17
Interrater agreement. Table 4 compares the subscale scores assigned by each rater to the same
client. The smallest mean subscale difference between the two raters was 6.3 ± 6.3 per cent, and
the largest was 8.1 ± 11.5 per cent.
[Table 4 here].
Discussion
The results of this study suggest that the CTC Homemaking Assessment reliably
measures the occupation of homemaking across different levels of difficulty. Based on tests of
internal consistency, each of the three assessment subscales can be said to reliably assess the
constructs of light, medium, and heavy homemaking tasks. Moreover, over two-thirds of clients
with whom the assessment has been administered demonstrated decreasing performance as task
difficulty increased, thus reinforcing the distinction between the subscales. In addition, the three
subscales are not perfectly correlated with each other, and the most strongly positively correlated
items occurred within a single subscale.
That being said, the results also suggest that the entire assessment measures a single
construct overall and that, rather than being distinct categories, task difficulty occurs along a
continuum. All three subscales demonstrated a moderate to strong positive correlation, while
individual assessment items also showed consistently positive correlations, suggesting that the
scale overall is generally unidimensional (i.e. assesses homemaking and nothing else). Similarly,
over half of the assessment items loaded strongly onto one factor. This is consistent with
previous studies which suggest that IADLs are all in fact a single construct (Leung et al., 2011).
However, the correlations between subscale scores also showed that there is a stronger
correlation between light and medium scores, and between medium and heavy scores, suggesting
a gradient in task difficulty.
18
In terms of occupational therapy practice, these results suggest that clinicians can use the
CTC Homemaking Assessment and expect it to reliably measure their client’s performance in
homemaking tasks across different levels of difficulty. Future research could focus on looking
for those items which help distinguish better between individuals’ homemaking abilities. The
distribution of total scores clustered around the 50 percent mark which is the cut-off for
homemaking supports from the primary motor vehicle insurance agency. Because of this, and
because descriptive measures should discriminate effectively between individuals (Mokkink et
al., 2010; Law, 1987), finding ways to make the assessment more discriminatory is warranted.
While the study provided support for the reliability and validity of the assessment
subscales related to task difficulty, additional latent constructs were identified from among the
study sample’s results. These underlying constructs, or factors, suggested a second dimension to
the assessment items: type of task demands. These included bending, upper extremity work,
manipulating loads, and planning. To some degree, this is consistent with studies that have made
the distinction between cognitive and physical IADLs (Leung et al., 2011). However, this study
suggests that even finer distinctions can be made among primarily physical tasks, even within the
narrower construct of homemaking. In terms of the difference between IADLs and AADLs
(Reuben et al., 1990), this study’s results were inconclusive about whether this distinction also
occurs within the construct of homemaking. Assessment items that might have been considered
AADLs were excluded from analysis due to a high number of missing values. With a larger
sample size, future studies might be able to provide more insight into whether this IADL-AADL
distinction is applicable to homemaking or not. Finally, the emergence of a “clothing task” factor
might suggest that individuals being assessed view folding clothes and putting clothes away as
very different from cleaning and home management tasks. We speculate that these tasks could
19
potentially be perceived as a more personal or individual, rather than a task that serves the needs
of the household in general. Although a range of latent constructs were identified within the data,
the structure of the assessment was not modified, due to an insufficient sample size. It would be
prudent to re-run the factor analysis with a larger sample before making any changes to the
existing assessment.
The content of the assessment was also not changed based on findings from the item
correlation matrix. These results showed that two items could potentially be removed due to their
strong and significant correlation with another item in the same subscale. These pairs, which also
loaded strongly together onto the same factors, were: folding clothes and putting clothes away;
and cleaning tub and cleaning baseboards. Removing one item from each of these pairs might
appear to be in keeping with recommendations that the number, range, and content of items in a
measurement tool be clearly justifiable (Yuen et al., 2014). However, removing items would
undermine a key strength of the CTC Homemaking Assessment, which is its client-centeredness.
Having a broader range of tasks structure the assessment helps clinicians better account for the
individual occupational performance patterns of clients. If an item is not applicable to a client, it
is simply not scored; therefore, a larger number of items is not inappropriate in these
circumstances.
Rather than changing the construction of the assessment itself, the results of this study
should inform how the assessment is administered and interpreted by clinicians. In particular, our
results have implications for the selection of tasks for observation. As per the design of the CTC
Homemaking Assessment, items are scored based on either self-report or direct observation of
task performance. Which tasks are to be observed is left to the discretion of the clinician. This
decision is particularly important as the literature suggests that direct observation in a client’s
20
home can be superior to, or an important adjunct to, self-reported performance status (Spanjer et
al., 2011). Based on our results, clinicians should consider not only the task difficulty, but also
the underlying type of task demand when selecting tasks to observe. Clinicians should observe
homemaking tasks that place varying types of physical and cognitive demands on clients in order
to obtain the most comprehensive picture of the client’s overall homemaking ability. For
example, if a client struggles to complete a medium bending task (e.g. cleaning the toilet), the
clinician administering the assessment may wish to also observe a medium upper extremity-
based task (e.g. washing dishes) because the two tasks may reveal different performance
abilities. Moreover, if clinicians are aware of these underlying task type constructs, it can assist
them in identifying and summarizing the specific functional limitations of individual clients.
In terms of whether the assessment can be administered reliably by different individuals,
very preliminary analysis is promising, suggesting it can be used by student occupational
therapists as well as clinicians with many more years of experience. Results from our interrater
comparison showed generally small differences in scores between raters for most participants.
The outlier that was excluded from our calculations was only the second assessment performed
by the student rater. Thus the difference in scores might be attributable to a learning curve effect
which has been observed among novices (Vroman & Stewart, 2014). In order to minimize future
differences in scoring, a more explicit standardization of the assessment (e.g. the creation of an
administration handbook) would be beneficial, particularly for new or emerging clinicians and/or
clinicians outside of CTC. Further research should also be conducted to enable the calculation of
formal statistics on the assessment’s interrater reliability.
With respect to generalizability, our results are promising in terms of the CTC
Homemaking Assessment’s use as a more global measure of disability for the occupation of
21
homemaking specifically. Results of this study suggest that the CTC Homemaking Assessment
can discriminate between a range of performance abilities, as the total weighted scores were
normally distributed. In addition, the assessment has been used with clients with a wide variety
of diagnoses and for a variety of referral reasons. This is unlike many other assessments that are
designed for administration with a very specific population or are more general but, to date, have
only be tested with a subset of potential populations (Law, 1993).
Though our study’s sample included a diversity of diagnoses, referral reasons, gender,
and ages, not all aspects of these were significantly represented in the sample population. Our
sample included primarily female participants. This was not unexpected because the assessment
is used mainly to determine the need for homemaking supports after an MVA. In BC, an
individual is only eligible for these supports if they were the primary homemaker prior to their
accident (Insurance [Vehicle] Regulations, 2014). Thus many individuals assessed are women.
In addition, the largest number of referrals in our sample was for individuals injured in an MVA,
closely followed by cost of future care assessments. Finally, diagnoses of pain, fracture/joint
issues, and/or soft tissue injuries were most frequently observed within the study’s sample. In
short, our study’s findings should be considered most generalizable to women, those who have
experienced an MVA, and for individuals with pain and/or musculoskeletal injuries or
conditions.
Study Limitations
The CTC Homemaking Assessment has been modified over time, and therefore the
retrospective sample included results from two slightly different versions of the measure. Most
of the assessments included in our study were from a previous version, meaning that some items
on the more recent version could not be included in our analysis due to a limited number of
22
cases. Therefore this study cannot make conclusions as to the effect of these additional variables
on the assessment’s factor structure and internal consistency. The sample size for our exploratory
factor analysis also limited our ability to make conclusive recommendations about altering the
structure of the CTC Homemaking Assessment. In addition, the sample size for the prospective
phase of this study did not allow us to make statistically significant conclusions about the
assessment’s interrater reliability. Further data collection to establish a larger overall sample
would ensure that the CTC Homemaking Assessment is generalizable to both men and women,
and would allow for a more conclusive factor analysis. If more data is collected using the current
version of the assessment, this would also allow for analysis of the additional variables that had
to be excluded by our study. Finally, further research could be helpful for determining interrater
reliability.
Conclusion
The CTC Homemaking Assessment is an ecologically valid, client-centered, and
occupation-based measure that fills an identified gap in the tools available to occupational
therapists to evaluate disability in a specific IADL domain. Its unique focus on homemaking
tasks and the provision of a percentage score have made it a valuable tool for clinicians working
in the legal system and with third party insurers. The results of our study provide preliminary
support for the psychometric integrity of the CTC Homemaking Assessment, suggesting that it is
a valid and generally reliable measure of homemaking ability, particularly among women and
those who have experienced an MVA. Given the increasing number of MVAs in Canada, the
need for such an assessment is only likely to continue to grow, especially within occupational
therapy private practice. With further study, the CTC Homemaking Assessment has the potential
23
to also be applied in other clinical settings to facilitate conversations with clients about their need
for supports, and to enhance the consistency and credibility of home support decisions.
Key Messages
● The CTC Homemaking Assessment is a novel, ecologically valid tool that can be used to
quantify a person’s ability to complete homemaking tasks ranging in difficulty and
performance demands.
● The assessment can identify a range of performance abilities on homemaking tasks and
occupational therapists with varying degrees of experience can administer the measure.
● Preliminary evaluation of this assessment demonstrates promising reliability and validity,
particularly when used with women and those who have experienced an MVA.
24
References
Bottari, C. L., Dassa, C., Rainville, C. M., & Dutil, E. (2010). The IADL profile: Development,
content validity, intra- and interrater agreement. Canadian Journal of Occupational
Therapy, 77(2), 90-100. doi:10.2182/cjot.2010.77.2.5
Clemson, L., Bundy, A., Unsworth, C., & Singh, M. F. (2009). Validation of the modified
assessment of living skills and resources, an IADL measure for older people. Disability &
Rehabilitation, 31(5), 359-369. doi:10.1080/09638280802105881
Coman L, Richardson J. Relationship between self-report and performance measures of function:
A systematic review. Can J Aging 2006;25:253e270.
Crepeau, E. B., et al. (2014). Analyzing occupations and activity. In Willard & Spackman’s
Occupational Therapy, B.A.B Schell, G. Gillen, & M. E. Scaffa (Eds). Philadephia:
Wolter Kluwers Health/Lippincott Williams & Wilkins.
De Vriendt, P., Gorus, E., Cornelis, E., Velghe, A., Petrovic, M., & Mets, T. (2012). The process
of decline in advanced activities of daily living: A qualitative explorative study in mild
cognitive impairment. International Psychogeriatrics, 24(6), 1-13.
doi:10.1017/S1041610211002766
Fisher, A. (1993). The assessment of IADL motor-skills - an application of many-faceted rasch
analysis. American Journal of Occupational Therapy, 47(4), 319-329.
Gold, D. (2012). An examination of instrumental activities of daily living assessment in older
adults and mild cognitive impairment. Journal of Clinical and Experimental
Neuropsychology, 34(1), 11-34.
Kottorp, A., Bernspång, B., Fisher, A. G., Arbetsterapi, Medicinska fakulteten, Institutionen för
samhällsmedicin och rehabilitering, & Umeå universitet. (2003). Validity of a
25
performance assessment of activities of daily living for people with developmental
disabilities. Journal of Intellectual Disability Research, 47(8), 597-605.
Insurance Corporation of BC (ICBC). (2015, August 31). ICBC starts 2015 basic rate
application process. Retrieved from http://www.icbc.com/about-
icbc/newsroom/Pages/2015Aug31.aspx
Insurance (Vehicle) Regulation, (2014). B.C. Reg. 447/83. Retrieved from
http://www.bclaws.ca/Recon/document/ID/freeside/447_83_00
Kellar, S. P., & Kelvin, E. A. (2013). Munro’s statistical methods for health care research.
Philadelphia: Wolter Kluwers Health/Lippincott Williams & Wilkins.
Law, M. C., & MacDermid, J. (2008). Evidence-based rehabilitation: A guide to practice.
Thorofare, NJ: Slack.
Law, M. (1993). Evaluating activities of daily living - directions for the future. American Journal
of Occupational Therapy, 47(3), 233-237.
Law, M. (1993). Evaluating activities of daily living - directions for the future. American Journal
of Occupational Therapy, 47(3), 233-237.
Law, M., & Letts, L. (1989). A critical-review of scales of activities of daily living. American
Journal of Occupational Therapy, 43(8), 522-528.
Law, M. (1987). Measurement in occupational therapy: Scientific criteria for evaluation.
Canadian Journal of Occupational Therapy, 54(3), 133-138.
doi:10.1177/000841748705400308
Law, M. C., Baum, C. M., & Dunn, W. (2005). Measuring occupational performance:
Supporting best practice in occupational therapy. Thorofare, NJ: SLACK Inc.
26
Letts, L., Law, M., Rigby, P., Cooper, B., Stewart, D., & Strong, S. (1994). Person-environment
assessments in occupational-therapy. American Journal of Occupational Therapy, 48(7),
608-618.
Leung, D., Chi, I., & Leung, A. (2011). An evaluation of the factor structure of the instrumental
activities of daily living involvement and capacity scales of the minimum data set for
home care for elderly chinese community dwellers in hong kong. Home Health Care
Services Quarterly, 30(3), 147-159.
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., . . . de
Vet, H. C. W. (2010). The COSMIN checklist for assessing the methodological quality of
studies on measurement properties of health status measurement instruments: An
international delphi study. Quality of Life Research, 19(4), 539-549. doi:10.1007/s11136-
010-9606-8
Myers, A., Holliday, P., Harvey, K., & Hutchinson, K. (1993). Functional performance-measures
- are they superior to self-assessments. Journals of Gerontology, 48(5), M196-M206.
Neistadt, M. E. (2000). Occupational therapy evaluation for adults: A pocket guide. Baltimore,
Md: Lippincott Williams & Wilkins.
Rogers, J., Holm, M., Beach, S., Schulz, R., Cipriani, J., Fox, A., & Starz, T. (2003).
Concordance of four methods of disability assessment using performance in the home as
the criterion method.
Arthritis & Rheumatism-Arthritis Care & Research, 49(5), 640-647.
doi:10.1002/art.11379
27
Reuben, D. B., Laliberte, L., Hiris, J. and Mor, V. (1990). A hierarchical exercise scale to
measure function at the Advanced Activities of Daily Living (AADL) level. Journal of
the American Geriatrics Society, 38, 855–861
Spanjer, J., Groothoff, J. W., & Brouwer, S. (2011). Instruments used to assess functional
limitations in workers applying for disability benefit: A systematic review. Disability &
Rehabilitation, 33(22-23), 2143-2150. doi:10.3109/09638288.2011.570413
Tarbell, M., Henry, A., & Coster, W. (2004). Psychometric properties of the scorable self-care
evaluation. American Journal of Occupational Therapy, 58(3), 324-332.
doi:10.5014/ajot.58.3.324
Vroman, K., & Stewart, E. (2014). Occupational therapy evaluation for adults: A pocket guide.
Baltimore, Md: Lippincott Williams & Wilkins
Wales, K., Clemson, L., Lannin, N. A., & Cameron, I. D. (2012). Functional assessments used
by occupational therapists with older adults at risk of activity and participation
limitations: A systematic review and evaluation of measurement properties. Systematic
Reviews, 1(1), 45-45. doi:10.1186/2046-4053-1-45
Yuen, H. K., & Austin, S. L. (2014). Systematic review of studies on measurement properties of
instruments for adults published in the American Journal of Occupational Therapy, 2009-
2013. The American Journal of Occupational Therapy : Official Publication of the
American Occupational Therapy Association, 68(3), e97. doi:10.5014/ajot.2014.011171
28
Appendix A - Tables
Table 1. Tasks (Variables) on Current Version of CTC Homemaking Assessment
Variable Subscale Included in Statistical
Analysis?
Handwashing clothes Light
Folding clothes Light √
Putting clothes away Light √
Dusting Light √
Cleaning mirrors Light √
Sweeping the floor Light √
Chopping, cutting, putting away food Light √
Drying dishes Light
Transfer clothes from washer to dryer Medium √
Take clothes out of dryer Medium √
Ironing Medium
Cleaning toilet Medium √
Cleaning sink Medium √
Cleaning baseboards Medium √
Cleaning windows Medium √
Washing floors Medium √
Making beds, stripping sheets Medium √
Washing dishes Medium √
Loading/unloading dishwasher Medium
Reach into upper and lower cupboards Medium
Vacuum Heavy √
Vacuuming stairs Heavy
Cleaning bathtub Heavy √
Cleaning oven Heavy √
Washing walls Heavy √
Cleaning fridge Heavy √
Cleaning freezer Heavy
Grocery shopping Heavy √
Taking out garbage Heavy √
29
Table 2. Demographics for Retrospective Sample (n=113)
Characteristic (n= 113) Finding
Age (years) Mean ± SD Range
50.6 ± 15.2
73
Gender Male Female
28 (24.7%)
85 (75.2%)
Reason for referral* MVA Cost of Future Care Medical-legal Fall Functional Capacity Evaluation Return to Work
80
73
21
7
2
5
Diagnoses* Pain Fracture/Joint Issues Soft Tissue Injuries Mental Health Sensory Changes Head Injury/Cognitive Issues Chronic Disease
69
49
48
42
34
26
16
*Note: Many participants had multiple diagnoses and reasons for referral
30
Table 3. Rotated Factor Matrix for CTC Homemaking Assessment
Factor 1:
Bending
tasks
Factor 2:
Upper
extremity tasks
Factor 3:
Clothing
tasks
Factor 4
Factor 5:
Load
tasks
Factor 6:
Planning
tasks
Vacuuming .531 .216 .242 .107 .111 .147
Cleaning fridge .530 .229 .271 .261 .041 .196
Washing walls .642 -.003 .037 -.013 .128 .175
Cleaning oven .515 .126 .035 .181 .065 .158
Cleaning bathtub .751 .121 .167 .073 .225 -.018
Making beds, stripping
sheets
.459 .273 .381 .178 .231 .030
Washing floors .660 .263 .036 .144 .107 .192
Cleaning baseboards .741 .080 .118 .092 .016 -.027
Cleaning toilet .483 .263 .017 .480 .104 -.013
Dusting .408 .564 .096 .162 -.092 .132
Washing dishes .080 .387 .342 .143 -.001 .143
Cleaning mirrors .222 .798 .143 .037 .173 -.081
Sweeping floor .472 .483 .120 .263 .054 .004
Folding clothes .208 .209 .701 .225 .191 .230
Putting clothes away .136 .119 .895 .096 .020 .023
Cleaning sink .140 .293 .194 .610 -.002 .079
Take clothes out of dryer .277 -.093 .265 .725 .438 .131
Transfer clothes from
washer to dryer
.152 .062 .024 .160 .880 .074
Taking out garbage .429 .197 .278 .013 .524 .110
Cleaning windows .299 -.054 .069 .027 .034 .588
Grocery shopping .207 .461 .269 .053 .254 .478
Chopping, cutting, putting
away food
-.100 .476 .191 .307 .110 .515
*Kaiser-Meyer-Olkin Measure of Sampling Adequacy = 0.807
**Bartlett’s Test of Sphericity: p value less than 0.000
31
Table 4. Interrater Comparison of Subscale Percentage Scores
Light Tasks
Medium Tasks Heavy Tasks
Client # Rater 1 Rater 2 Rater 1 Rater 2 Rater 1 Rater 2
A1 87.5 100 90 90 41.7 41.7
A2 100 100 100 100 100 100
A3 100 83.3 95.5 83.3 78.6 80.1
A4 62.5* 100* 40* 100* 33.3* 100*
A5 80 90 85 100 66.7 80
A6 100 94.4 100 100 100 92.8
A7 95 100 72.7 90 28.6 62.5
A8 94.4 94.4 85 80 83.3 83.3
A9 88.9 88.9 90.9 93.2 91.7 83.3
Mean difference: 6.3 ± 6.3
Mean difference: 6.5 ± 7.3
Mean difference: 8.1 ±11.5
32
Appendix B - Supplementary Materials
Supplemental Table 1. Subscale Correlation Matrix
LIGHT % MEDIUM % HEAVY %
LIGHT % Pearson Correlation 1 .649** .491**
Sig. (2-tailed) .000 .000
N 112 112 112
MEDIUM
%
Pearson Correlation .649** 1 .773**
Sig. (2-tailed) .000 .000
N 112 112 112
HEAVY % Pearson Correlation .491** .773** 1
Sig. (2-tailed) .000 .000
N 112 112 112
** Correlation is significant at the 0.01 level (2-tailed).
Supplemental Table 2. Interitem Correlation Matrix
Due to the size of this table it is available upon request to the authors.