Post on 16-Oct-2021
Running head: DISTRESS TOLERANCE SCALE
1
Tables: 3
Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis
Travis A. Rogers1, Joseph R. Bardeen1, Thomas A. Fergus2, & Natasha Benfer1
1Department of Psychology, Auburn University, United States
2Department of Psychology and Neuroscience, Baylor University, United States
NOTICE: this is the author’s version of a work that was accepted for publication in Assessment.
Changes resulting from the publishing process, such as peer review, editing, corrections,
structural formatting, and other quality control mechanisms may not be reflected in this
document. Changes may have been made to this work since it was submitted for publication. A
definitive version was subsequently published in Assessment.
Author Note
Correspondence concerning this article should be addressed to Joseph R. Bardeen,
Department of Psychology, Auburn University, Auburn, AL 36832. Voice: 334-844-6647; E-
mail: jbardeen@auburn.edu.
DISTRESS TOLERANCE SCALE
2
Abstract
The Distress Tolerance Scale (DTS) is a self-report measure of perceived capacity to withstand
aversive emotions. Initial factor analysis of this measure suggested a structure comprising one
higher-order factor and four lower-order domain-specific factors. However, there is limited
evidence in support of the DTS’s purported multidimensionality, and despite use of the DTS
subscales, research has yet to assess their incremental utility. The current investigation sought to
rectify the paucity of evidence in support of the DTS’s factor structure and independent use of
DTS subscales via bifactor analysis. In the present study (N = 826 community adults), a bifactor
model of the DTS provided the best fit to the data. However, an examination of statistical indices
associated with bifactor modeling, as well as results from an examination of incremental utility,
suggest that the domain-specific factors are largely redundant with the general factor and do not
provide incremental utility in predicting relevant clinical constructs beyond the general factor.
Measurement invariance between sexes was confirmed. Taken together, results support use of a
DTS total score, but not subscale scores.
Keywords: emotional distress intolerance, distress tolerance scale, bifactor analysis,
psychometrics
DISTRESS TOLERANCE SCALE
3
Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis
Distress tolerance is broadly defined as one’s capacity to withstand aversive physical and
psychological states (Leyro, Zvolesnky, & Bernstein, 2010). Although there is not yet consensus
regarding the operational definition of distress tolerance, compelling conceptualizations have
been put forward that propose that distress tolerance is made up of several distinct, but related,
facets (e.g., uncertainty, physical discomfort; Zvolensky, Vujanovic, Bernstein, & Leyro, 2010).
One facet of the broader construct of distress tolerance is emotional distress intolerance (EDI),
which reflects the capacity to withstand the distress associated with aversive emotions (Simons
& Gaher, 2005). Conceptual models of psychopathology position EDI as a central explanatory
factor in the pathogenesis and maintenance of a wide variety of pathological presentations
(Leyro, Zvolensky, & Bernstein, 2010; Zvolensky & Otto, 2010), including substance use
disorders, disordered eating, and posttraumatic stress (Bardeen & Fergus, 2016; Bernstein,
Vujanovic, Leyro, & Zvolensky, 2011; Vujanovc, Bernstein, & Litz, 2011). Moreover, EDI has
been identified as a potentially important mechanism of change in psychological interventions
for a wide variety of symptom presentations (see Zvolensky, Bernstein, & Vujanovic, 2011),
thus resulting in the development of clinical interventions that directly target EDI in the service
of symptom relief (e.g., Dialectal Behavior Therapy [DBT]; Linehan, 1993). Given EDI’s
transdiagnostic status and clinical utility, it is important to develop psychometrically sound
measures of this construct.
The assessment of EDI has garnered substantive interest in the extant literature, including
through the development of self-report and behavioral methods. As reviewed by Leyro et al.
(2010), Simons and Gaher's (2005) self-report measure, known as the Distress Tolerance Scale
(DTS), is the most commonly used index of EDI.1 Since Leyro et al.'s review, McHugh and Otto
DISTRESS TOLERANCE SCALE
4
(2012) developed a self-report measure that includes items from the DTS and other measures in
an attempt to refine the assessment of the construct. A limitation surrounding McHugh and Otto's
(2012) measure is that theoretically (e.g., Leyro et al., 2010) and empirically (e.g., Bardeen,
Fergus, & Orcutt, 2013b) distinct components of the distress tolerance construct are lumped
together to produce a composite score, thereby limiting the use of this measure as an index of
EDI per se. Veilleux, Pollert, Zielinski, Shaver, and Hill (in press) developed a behavioral
measure of EDI. Because behavioral methods often assess more "state-like" tendencies and self-
report measures often assess more "trait-like" tendencies, Veilleux et al.'s (in press) behavioral
measure holds promise for offering a complementary approach to the DTS for assessing EDI.
Overall, additional research is needed to identify optimal ways to assess EDI and, because of its
widespread use, further examination of the DTS is warranted, particularly related to the factor
structure and resulting operationalization of the DTS item scores.
The DTS was originally validated in two large, independent samples of university
students. Specifically, Simons and Gaher (2005) generated a pool of 16 items to be consistent
with their conceptualization of EDI. These items were submitted to an exploratory factor analysis
(EFA). Fifteen items were retained for further testing, and the EFA supported a four-factor
solution. Confirmatory factor analysis (CFA) supported a hierarchical structure, in which four
lower-order factors loaded onto a higher-order factor. The lower-order factors of the DTS
purportedly reflect (1) the perceived capacity to tolerate or endure negative emotions (i.e.,
Tolerance; “Feeling distressed or upset is unbearable to me”), (2) the extent to which attention is
occupied by emotional distress (i.e., Absorption; “When I feel distressed or upset, all I can think
about is how bad I feel”), (3) the subjective appraisal of negative emotions (i.e., Appraisal; “My
DISTRESS TOLERANCE SCALE
5
feelings of distress or being upset are not acceptable”), and (4) efforts aimed at managing
emotional distress (i.e., Regulation; “I’ll do anything to stop feeling distressed or upset”).
Psychometric support for scores on the 15-item measure has been provided, including
evidence of internal consistency for the total score (α range = .82 – .91) and subscale scores (α
range = .72 – .82; Leyro, Bernstein, Vujanovic, McLeish, & Zvolensky, 2011; Simons & Gaher,
2005), as well as retest reliability for the total score over a six-month interval (r = .61; Simons &
Gaher 2005). In addition, the DTS has been used across multiple domains of psychological
research, and evidence has accumulated in favor of (a) its concurrent validity (Bernstein,
Marshall, & Zvolensky, 2011; Simons & Gaher 2005), and (b) incremental utility in accounting
for variance within relevant clinical constructs (e.g., symptoms of bulimia, impulsivity) beyond
that explained by anxiety, depression, and other self-report and behavioral measures related to
EDI (e.g., Anestis et al., 2012). However, some evidence calls the validity of the subscales into
question. Cougle, Bernstein, Zvolensky, Vujanovic, and Mcatee (2013) found that the DTS total
score provided a significant incremental contribution in predicting tolerance and perceived threat
of anger, fear, sadness, and disgust in response to emotionally evocative film clips, even after
accounting for sex, emotional intensity, and anxiety sensitivity. In contrast, DTS subscales
exhibited inconsistent associations with constructs to which they should be theoretically related.
For example, the DTS Tolerance subscale was incrementally associated with perceived threat
and tolerance for the anger film clip, but not for three other film clips used to elicit sadness, fear,
and disgust. Such results run counter to the generally supported validity of these sub-domains of
emotional distress intolerance.
Cougle et al.’s (2013) results notwithstanding, the extant literature has been largely
supportive of the utility of the DTS total and subscale scores. However, the structure of this
DISTRESS TOLERANCE SCALE
6
measure has yet to be adequately examined (see Table 1). Leyro et al. (2011) provided the only
known CFA of the DTS after its initial publication and found several areas of possible model
strain that raise concerns about (a) the conceptual distinctions between the DTS factors (i.e.,
multiple potential cross loadings, strong inter-factor correlations [rs > .80]), and (b) the under-
performance of some DTS items relative to others (i.e., standardized item-factor loading range =
.29 – .86). Furthermore, Leyro et al. used a 14-item version of the DTS that was developed prior
to Simons and Gaher’s (2005) publication. Specifically, item 14 of the Regulation subscale
(“When I feed distressed or upset, I must do something about it immediately”) was omitted.
Leyro et al. also found that correlations between three of the DTS subscales (Tolerance,
Absorption, and Appraisal) were strong enough (rs > .80) to raise concerns about the conceptual
distinctions between them.
Following Leyro et al.’s (2011) CFA, Hsu, Collins, and Marlatt (2013) examined the
factor structure of the DTS via principal component analysis (PCA). Contrary to Simons and
Gaher’s (2005) original publication, Hsu et al.’s results yielded one unitary factor, suggesting
that the DTS might be better modeled as a unidimensional measure of EDI. It should also be
noted that PCA is primarily a data reduction technique (see Abdi & Williams, 2010). Thus, Hsu
et al.’s results are less likely to generalize to other factor analytic work (see Suhr, 2005). Put
succinctly, these limited investigations leave a number of questions about the structure of the
DTS unanswered.
The paucity of empirical evidence in support of the factor structure of the DTS remains a
significant limitation to future work with this measure. Studies using the DTS have tended either
to use the DTS total score as the sole index of EDI (e.g., Anestis, Bagge, Tull, & Joiner, 2011;
Anestis et al., 2012; Bernstein, Marshall, et al., 2011; Vujanovic, Bonn-Miller, Potter, Marshall,
DISTRESS TOLERANCE SCALE
7
& Zvolensky, 2011) or to interpret both the total and subscale scores (e.g., Ameral, Palm Reed,
Cameron, & Armstrong, 2014; Leyro et al., 2011; Simons & Gaher, 2005; Shorey et al., 2017;
Timpano, Buckner, Richey, Murphy, & Schmidt, 2009). These approaches make at least two
assumptions about the nature of the DTS. First, use of a total score assumes that each DTS
domain-specific factor represents an aspect of the same overarching general construct. Second,
use of subscale scores assumes that the lower-order factors capture unique information beyond
the total score. However, the existing literature base for the DTS has yet to adequately address
the tenability of these two approaches. While investigation of the purported hierarchical structure
of the DTS is useful for examining the first of the two assumptions, it cannot directly test the
assumption that items of domain-specific factors provide unique information beyond a higher-
order or general factor (Reise, 2012). Such analyses are necessary because, as noted, a number of
investigations have used DTS subscales and found differential relations with constructs of
interest. Thus, it is important to use an analytic approach that can simultaneously address both of
these assumptions (i.e., bifactor analysis).
Bifactor models permit investigation of the presence of a general factor and the degree to
which each domain-specific factor is meaningfully distinct from the general factor (Reise,
Bonifay, & Haviland, 2013; Reise, Moore, & Haviland, 2010; Rodriguez, Reise, & Haviland,
2016). Moreover, bifactor models have demonstrated good fit to transdiagnostic self-report
measures in prior research (e.g., Ebesutani, McLeish, Luberto, Young, & Maack, 2014;
Ebesutani et al., 2011). To our knowledge, the DTS has yet to be examined via a bifactor
modeling approach. Such an analysis is important and would likely yield new insights into the
performance of the DTS. For example, redundancy between a general factor and domain-specific
factors may suggest that a total score should be used in lieu of subscale scores (Reise, 2012).
DISTRESS TOLERANCE SCALE
8
Therefore, the primary aim of the current investigation was to examine the fit of a bifactor model
of the DTS in a large adult sample. Consistent with standard methodology for comparing model
fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model to competing models
(i.e., a correlated four-factor model, a one-factor model, and a hierarchical model). Some have
suggested that the utility of comparing bifactor models to alternative models is limited because
bifactor models often provide better fit to the data due to their inherent qualities (e.g., high
flexibility; Bonifay, Lane, & Reise, 2017; Reise, Kim, Mansolf, & Widaman, 2016). As such, a
number of additional statistical indices, developed for use with the bifactor approach, were
examined (Rodriguez, Reise, & Haviland, 2016). These statistics provide information central to
the aims of the present study (e.g., determining the degree to which domain-specific factors have
value beyond the general factor, determining the stability and replicability of the factors).
An additional benefit of bifactor modeling is that the relation between domain-specific
factors and criterion variables can be examined while holding the general factor constant
(Brown, 2015). Thus, we also examined the incremental utility of the DTS domain-specific
factors in predicting theoretically relevant constructs (i.e., depression, anxiety, and stress) after
accounting for a general EDI factor. Depression, anxiety, and stress were selected as outcome
variables because research suggests that emotional distress intolerance plays an important role in
the etiology and maintenance of emotional disorders (e.g., Bernstein, Vujanovic, et al., 2011;
McHugh et al., 2014).
Another unaddressed question regarding the structure of the DTS relates to whether it is
similar between males and females. In the original validation study, females reported
significantly poorer tolerance for emotional distress than males (Cohen’s d = 0.32; Simons &
Gaher, 2005). This effect remained significant in a regression analysis partialling out the shared
DISTRESS TOLERANCE SCALE
9
variance between sex and negative affectivity, indicating that mean differences in DTS total
scores could not be attributed to differences in negative affect. This difference may reflect either
a true difference in emotional distress tolerance between males and females or may be a function
of item content that promotes differential responding. Significant mean differences across sexes
have been reported in other investigations (e.g., Cougle et al., 2013; Dennhardt & Murphy, 2011)
and have indicated that females tend to report poorer emotional distress tolerance than males. An
investigation of sex invariance within the DTS can speak to the potential need for differential
scoring and modeling of DTS items among females and males. For example, if the loadings of
DTS items on a general factor and domain-specific factors differ as a function of sex, it may be
possible that certain DTS domain-specific factors are independent and incrementally useful in
one sex but not the other. Therefore, in addition to an examination of (a) the purported
multidimensionality of the DTS and (b) the viability of using the DTS domain-specific factors in
predicting clinically relevant constructs, the present study also sought to investigate (c)
measurement invariance of DTS items across males and females.
Methods
Participants and Procedure
One thousand and nine participants were recruited using Amazon’s Mechanical Turk
(MTurk), an online labor market where individuals can participate in research in exchange for
financial compensation. Previous research has supported the use of MTurk samples, citing their
reliability and diverse demographics relative to other online and undergraduate samples (e.g.,
Behrend, Sharek, Meade, & Wiebe, 2011; Buhrmester, Kwang, & Gosling, 2011; Chandler &
Shapiro, 2016). To minimize the effect of random responding or inattentiveness, three catch
questions (e.g., “Select ‘Much’ if you are paying attention right now”) were imbedded in the
DISTRESS TOLERANCE SCALE
10
online survey (Oppenheimer, Meyvis, & Davidenko, 2009; Paolacci, Chandler & Ipeirotis,
2010). Participants who answered fewer than two of the three catch questions correctly (n = 144;
14.27% of original sample) were excluded from analyses (for precedence, see Bardeen, Fergus,
Hannan, & Orcutt, 2016; Bardeen, Fergus, & Orcutt, 2013a, 2013b; Bardeen & Michel, 2017).
Additionally, 39 participants (3.87% of the original sample) who failed to provide responses for
the majority of DTS items (eight out of fifteen [at least 50%; see Graham & Schafer, 1999]) were
removed from the sample. The proportion of missing cases within all variables of interest was ≤
1%, and covariance coverage was high across all variables, ranging from .985 to 1.00. The final
sample (N = 826) was 60.5% female. Five participants elected not to report their sex. The
average age of the sample was 33.68 years (SD = 12.54, range = 18 to 73). Concerning race,
81.6% of the final sample identified as White, 6.5% as Black, 6.3% as Asian, 1.0% as American
Indian or Alaskan Native, 0.1% as Native Hawaiian or Pacific Islander, 2.8% as “Other,” and
1.7% preferred not to respond. Additionally, 89.1% of the final sample identified as Non-
Hispanic, 6.8% as Hispanic, and 4.1% preferred not to report ethnicity.
All study procedures were approved by the local institutional review board. Informed
consent and self-report measures could be completed from any computer with internet access.
The median time to complete the online battery was 29.19 minutes. Upon completion,
participants were compensated with $0.50, an amount consistent with compensation provided to
MTurk workers who completed studies of similar length (e.g., Buhrmester et al., 2011).
Measures
Distress Tolerance Scale (DTS). As described, the DTS (Simons & Gaher 2005) is a 15-
item measure designed to assess individual differences in EDI. Items are rated on a 5-point scale
from 1 (Strongly agree) to 5 (Strongly disagree). Higher total scores are indicative of greater
DISTRESS TOLERANCE SCALE
11
tolerance for emotional distress, while lower scores reflect a relative intolerance of emotional
distress.
The Depression Anxiety Stress Scales – 21-item version (DASS-21). The DASS-21
(Lovibond & Lovibond, 1995) is a 21-item measure with subscales assessing symptoms of
depression, anxiety, and stress during the past week. Each subscale comprises seven items rated
on a 4-point Likert scale ranging from 0 (Did not apply to me at all) to 3 (Applied to me very
much, or most of the time); thus, higher scores suggest greater symptomatology. The DASS-21
has evidenced good psychometric properties in a number of studies (Bardeen, Fergus, & Orcutt,
2014; Henry & Crawford, 2005; Lovibond & Lovibond, 1995) and has evidenced strong
convergent validity with other measures of depression and anxiety (Antony, Bieling, Cox, Enns,
& Swinson, 1998). The DASS-21 subscales evidenced good internal consistency in the current
sample, with Cronbach’s αs of .93, .90, and .86 for depression, stress, and anxiety, respectively.
Data Analytic Strategy
Confirmatory Factor Analysis. The following four models were examined via CFA.
The first model was a correlated four-factor model proposed by Simons and Gaher (2005), with
three items loading onto the Tolerance factor, three items loading onto the Absorption factor, six
items loading onto the Appraisal factor, and three items loading onto the Regulation factor.
Correlations between all four factors were modeled, and no item had a secondary loading.
Standard procedures for conducting model comparisons, especially for measures that produce a
total score, requires examination of a one-factor model to ensure that a unidimensional model
does not provide better fit to the data (Brown, 2015; Reise et al., 2010). As such, the second
model was a one-factor model, within which all items of the DTS loaded onto one factor. The
third model was a hierarchical model, in which correlations between factors were omitted and
DISTRESS TOLERANCE SCALE
12
direct pathways were modeled from a higher-order factor onto each domain-specific factor (i.e.,
identical to that proposed in the original DTS publication). The fourth model was a bifactor
model, in which all 15 items loaded onto a general factor as well as onto their respective domain-
specific factor, and covariance terms between all factors were held to zero (Brown, 2015).
Model Estimation and Comparison. All models were tested using MPlus 7.4 (Muthén
& Muthén, 2015). Missing data were handled using full information maximum likelihood, which
is the default method for imputing missing data in MPlus. Robust maximum likelihood (MLR)
estimation was used to test all models, as MLR is robust to violations of the assumption of
normality (Brown, 2015; Kaplan, 2009). Therefore, all reported chi-square (χ2) values represent
the Satorra-Bentler scaled χ2 (Satorra & Bentler, 1994). Four commonly recommended fit
statistics were used to evaluate the fit of each model to the data: the comparative fit index (CFI),
the Tucker-Lewis fit index (TLI), standardized root mean square residual (SRMR), and root
mean square error of approximation (RMSEA; Brown, 2015; Kline, 2016). The following
guidelines were used to evaluate model fit: CFI and TLI should be near .95; SRMR should be
less than .08 (Hu & Bentler, 1999); RMSEA should be near .06, and the upper limit of the 90%
RMSEA confidence interval (CI) should not exceed .10 (Kline, 2016). In addition to an
evaluation of model fit, model comparisons were evaluated by way of scaled χ2 difference
testing. However, due to the sensitivity of χ2 distributions to large samples, χ2 difference testing
might suggest significant differences between models when differences in the magnitude of
parameter estimates are trivial (Brown, 2015; Kline, 2016). Thus, alternative tests for comparing
model fit were used, including an examination of differences in sample-size corrected Akaike
Information Criterion (AICc; Akaiki, 1974; Sugiura, 1978), which provides a more quantitative
rather than qualitative (e.g., better/worse) estimate of model comparison. Additionally, RMSEA
DISTRESS TOLERANCE SCALE
13
90% CIs were examined. Relative differences in model fit were considered meaningful if models
differed in AICc by 10 or more (see Burnham, Anderson, & Huyvaert, 2011, for a discussion),
and had non-overlapping RMSEA 90% CIs (e.g., Wang & Russell, 2005).
Bifactor Model Evaluation. In addition to a basic examination of fit statistics between
models, the following statistical indices were examined to further evaluate the bifactor model
(see Rodriquez et al., 2016). All additional bifactor model indices were calculated using the
bifactor indices calculator provided by Dueber (2016). OmegaH (ωH) reflects the proportion of
variance in DTS scores attributable to a general factor, while OmegaHS (ωHS) reflects the
proportion of variance in scores explained by each domain-specific factor after removing
variance explained by the general factor. Both ωH and ωHS are best understood as indices of
factor reliability. Explained common variance (ECV) is calculated by dividing variance
attributable to the general factor by variance attributable to both general and specific factors;
thus, ECV serves as a better index of unidimensionality within a measure than ωH. Percentage of
uncontaminated correlations (PUC) is interpreted along with ECV and serves as an indicator of
the percentage of DTS item correlations contaminated by variance attributed to the general and
domain-specific factors. Item-level ECV (I-ECV) indicates the amount of common variance
from each item that is attributable to the general factor (Stucky, Thissen, & Edelen, 2013), thus
serving as an index of unidimensionality at the item level. Construct replicability (H) reflects the
degree to which a factor is well defined by its indicators. Factor determinacy (FD) is the
correlation between factor scores and the factors and serves as an indication of whether factor
scores are valid for independent use. Finally, average relative parameter bias (ARPB) serves as
an indication of bias across parameters when items are forced into a unidimensional structure.
DISTRESS TOLERANCE SCALE
14
Structural Regression Model. A structural regression model was used to examine
whether the domain-specific factors of the DTS relate to clinically relevant constructs when
simultaneously accounting for a general emotional distress intolerance factor. Specifically, the
general factor and domain-specific factors from the bifactor model were simultaneously
regressed onto the three uncorrelated domains of the DASS-21. Domain-specific factors from the
bifactor model that exhibited clear redundancy with the general factor (e.g., non-significant
residual variance) were removed from the model. Path coefficients from the general factor and
remaining domain-specific factor(s) to each of the DASS scales were freely estimated.
Measurement Invariance. A multiple-group CFA was used to test measurement
invariance, with restrictive models testing for (1) configural invariance (i.e., equal structural
form), (b) metric invariance (i.e., equal factor loadings), and (c) scalar invariance (i.e., equal
indicator intercepts) between males (n = 321) and females (n = 500). To test configural
invariance, the adequacy of the final DTS bifactor structure was examined independently and
simultaneously in both sexes. The test of metric invariance constrained the factor loadings within
the final DTS bifactor model to equality across both sexes. Scalar invariance was examined by
constraining indicator intercepts to equality across sexes. Because χ2 is sensitive to large samples
and interpretation of multiple-group CFA results is complicated by unequal group sizes (Brown,
2015), RMSEA 90% CIs and CFI were also examined between invariance models. Decrements
in model fit were considered meaningful when parameter constraints resulted in either (a) non-
overlapping RMSEA 90% CIs or (b) changes in CFI (∆CFI) greater than .01 (Cheung &
Rensvold, 2002).
Results
DISTRESS TOLERANCE SCALE
15
Model Estimation and Comparison. Goodness-of-fit statistics from the measurement
models are presented in Table 2. The correlated four-factor model evidenced adequate fit to the
data. CFI and TLI approximated the specified criteria of adequate model fit, and RMSEA,
RMSEA 90% CIs, and SRMR of the four-factor model indicated good fit. As the one-factor
model is nested within the correlated four-factor model (Brown, 2015), the one-factor model was
examined next for comparative purposes. Relative to the four-factor model, the one-factor model
provided significantly worse fit to the data, as evidenced by a significant change in χ2, ∆χ2(6) =
296.66, p < .001, differences in AICc of 528.70, and non-overlapping RMSEA 90% CIs. The
hierarchical model was fitted to the data next. CFI and TLI approximated the specified criteria of
adequate model fit, and RMSEA, RMSEA 90% CIs, and SRMR indicated good fit. In
comparison to the correlated four-factor model, there was a significant change in χ2, ∆χ2(2) =
13.15, p < .01, and meaningful change in AICc of 15.23. In contrast, RMSEA 90% CIs of the
correlated four-factor and hierarchical models were overlapping.
The bifactor model provided adequate to good fit to the data; all fit indices met or
approximated specified criteria. Hierarchical models are nested within bifactor models and
therefore can be directly compared (Brown, 2015). χ2, AICc values, and RMSEA 90% CIs were
compared across models. χ2 difference testing indicated a significant difference between the fit
of the (a) correlated four-factor and bifactor models, ∆χ2(9) = 42.78, p < .001, (b) one-factor and
bifactor models, ∆χ2(15) = 347.00, p < .001, and (c) hierarchical and bifactor models, ∆χ2(11) =
55.46, p < .001. Differences in AICc values between the correlated four-factor and bifactor
models, ∆AICc = 53.91, between the one-factor and bifactor models, ∆AICc = 582.61, and
between the hierarchical and bifactor models, ∆AICc = 69.14, also favored the bifactor model.
However, RMSEA 90% CIs were overlapping between all models. As two of three comparative
DISTRESS TOLERANCE SCALE
16
indices suggested better fit in the bifactor model, it was retained for further examination.
The standardized factor loadings from the bifactor model are presented in Table 3. While
nine of 15 items exhibited significant factor loadings (p < .05) onto their domain-specific factors,
factor loadings of all items of the Absorption factor and items 6, 9, and 10 of the Appraisal factor
were rendered non-significant in the bifactor model. In contrast, all items loaded significantly
onto the general factor. Redundancy between the Tolerance and Absorption factors and the
general factor was evidenced by non-significant residual variances within the bifactor model.
The Appraisal and Regulation factors were the only domain-specific factors to exhibit significant
residual variance.
Bifactor Model Evaluation. Additional indices derived from the bifactor model are
presented in Table 3. According to Stucky and Edelen (2015), I-ECV values greater than .80 or
.85 indicate unidimensionality at the item level. The majority of DTS items (nine out of 15)
demonstrated I-ECV values greater than .80. Within the domain-specific factors, two out of three
Tolerance items, two out of three Absorption items, and five out of six Appraisal items,
evidenced content redundancy with the general factor. In contrast, the Regulation factor was the
only domain-specific factor whose items demonstrated domain-specific content, as evidenced by
I-ECV values ≤ .72. As reviewed by Rodriguez et al. (2016), ω and ωS serve as indices of the
reliability of the general and domain-specific factors, respectively. Adequate reliability was
demonstrated by both the general factor (ω = .95) and domain-specific factors (ωS range = .83 –
.88). The general factor explained a substantial amount of variance in DTS scores (ωH = .91).
However, after partitioning out variance explained by the general factor, the domain-specific
factors accounted for a small proportion of variance within their respective items (ωHS range =
.07 – .35).
DISTRESS TOLERANCE SCALE
17
According to Rodriguez et al. (2016), ECV and PUC reflect the degree to which
parameter estimates are biased when multidimensional constructs are forced into a
unidimensional model. When ECV and PUC are greater than .70, common variance within a
model can be regarded as essentially unidimensional. Within the bifactor model, ECV of the
general factor was .80, and PUC was .77, supporting a unidimensional rather than
multidimensional conceptualization of the DTS. When H is high (H > .70; see Hancock &
Mueller, 2001), a latent variable may be considered well defined by its items, and thus, will
demonstrate greater stability across studies. Within the bifactor model, the general factor
evidenced acceptable construct replicability, H = .95. However, no domain-specific factor
demonstrated adequate construct replicability, with H values ranging from .25 to .55. Moreover,
Gorsuch (1983) recommends that factor score estimates should be used only when FD is greater
than .90. Per this criterion, the general factor estimate may be interpreted, with FD = .96.
However, no domain-specific factor met Gorsuch’s guideline, with FD values from domain-
specific factors ranging from .67 to .84. Lastly, ARPB of the bifactor model was .03, suggesting
minimal differences between parameter estimates in a unidimensional and bifactor solution.
ARPB less than 0.10 or 0.15 suggests that multidimensionality within a measure is not
substantial enough to negate the tenability of a unidimensional solution (Muthén, Kaplan, &
Hollis, 1987; Rodriquez et al., 2016).
Modification indices derived from the bifactor model yielded 12 potential item cross-
loadings (i.e., expected change in parameter estimates of > 10; Brown & Moore, 2012; Muthén
& Muthén, 2015). Specifically, allowing all three Tolerance items, all three Absorption items,
two of the Appraisal items (items 7 and 10), and item 13 of Regulation to cross-load onto another
domain-specific factor would substantially improve model fit. Modification indices also
DISTRESS TOLERANCE SCALE
18
suggested that model fit would benefit from allowing three items (2, 7, and 15) to cross-load on
three factors. These results suggest that DTS items may reflect unidimensional rather that
domain-specific content.
Structural Regression Model. Because three of the four DTS domain-specific factors
(Tolerance, Absorption, and Appraisal; see Table 3) exhibited poor performance within the
bifactor model (i.e., non-significant residual variance and/or a preponderance of items that
exhibit content redundancy with the general factor), only the Regulation factor and general factor
were included as predictors in the structural regression.2 The structural regression model
provided adequate fit to the data, with all fit indices approximating specified guidelines: χ2(585)
= 1981.16, p < .001; RMSEA = .05 (90% CIs = .051 – .056); CFI = .91; TLI = .90; SRMR = .05.
The general factor significantly predicted all outcome variables at p < .001, with standardized
coefficient estimates in the expected direction: depression, β = -0.46; anxiety, β = -0.38; stress, β
= -0.41. After accounting for the general factor, the Regulation factor also significantly predicted
all outcome variables at p < .001; however, all standardized coefficient estimates were in a
theoretically inconsistent direction: depression, β = 0.76; anxiety, β = 0.83; stress, β = 0.84.3
In order to test for possible net suppression effects, the structural regression was
modified, such that depression, anxiety, and stress were regressed onto only the Regulation
domain-specific factor (i.e., the general factor was modeled but did not serve as a predictor of
these outcomes). This second structural regression yielded a pattern of results in opposition to
the first. Significant negative associations between the Regulation factor and criterion variables
(depression, β = -0.86; anxiety, β = -0.91; stress, β = -0.94, ps < .001) suggest that the
unexpected positive relationships between the Regulation domain-specific factor and these
clinical outcomes, when controlling for the general factor, most likely represent a suppression
DISTRESS TOLERANCE SCALE
19
effect. Further, the high standardized beta weights between this domain-specific factor and
criterion variables suggests construct overlap.4, 5
Measurement Invariance. The final DTS bifactor model (the general factor and the
Regulation domain-specific factor) was retained for an analysis of measurement invariance. Fit
indices of the configural, metric, and scalar invariance models are displayed in Table 2. The
bifactor model exhibited adequate fit in both males and females when modeled separately and
simultaneously (i.e., evidence of configural invariance). While there was a change in AICc
greater than 10, no significant decrement in model fit was observed between the configural and
metric models as evidenced by Δχ2 (Δχ2[16] = 14.56), overlapping RMSEA 90% CIs, and ∆CFI
of .002. χ2 differences between the metric and scalar models (Δχ2[13] = 41.76) and the scalar and
configural models (Δχ2[29] = 58.70) were significant at p < .001. However, RMSEA CIs across
all invariance models overlapped, and ∆CFI did not exceed Cheung and Rensvold’s (2002)
guideline. Thus, overlapping RMSEA CIs and ∆CFI < .01 were interpreted as evidence of scalar
invariance between males and females. In addition, latent mean scores of the DTS general factor
and Regulation domain-specific factor were compared between sexes. Significant differences
were not observed when comparing the latent general factor (difference = 0.11, p = .14) and
latent Regulation factor means (difference = 0.06, p = .17) between males and females.
Discussion
The primary aim of this investigation was to address the paucity of empirical evidence in
support of the multidimensional structure of the DTS. Consistent with standard methodology for
comparing model fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model of the
DTS to a correlated four-factor model, a one-factor model, and a hierarchical model, and thereby
moved beyond the measurement models examined in previous studies. The adequacy of the
DISTRESS TOLERANCE SCALE
20
hierarchical model proposed in the original publication of the DTS (Simons & Gaher, 2005) was
replicated in this study; however, the hierarchical model and correlated four-factor model did not
differ in their fit to the data. The bifactor model of the DTS, in which all indicators loaded onto a
general factor in addition to their respective orthogonal domain-specific factors, evidenced better
fit relative to all other tested models. In evaluating the bifactor model further, item loadings onto
the domain-specific factors were substantially attenuated, and residual variance of two of the
four domain-specific factors (i.e., Tolerance and Absorption) was rendered non-significant after
accounting for the general factor. Moreover, the majority of DTS items demonstrated content
redundancy at the item level. The general factor accounted for most of the variance in DTS
scores, and no domain-specific factor met specified benchmarks of adequate construct
replicability or factor determinacy. Succinctly, results provide support for a strong general factor
within the DTS but do not support the use of purported domain-specific factors.
This investigation also sought to test the incremental utility of the DTS domain-specific
factors in predicting clinical constructs after accounting for the general factor. To that end, a
structural regression model, in which depression, anxiety, and stress served as outcome variables,
was performed. The Regulation factor was the only domain-specific factor whose residual
variance remained significant in the bifactor model and whose items could be regarded as
providing enough unique variance beyond that of the general factor to be considered a separate
domain. Thus, only the Regulation factor and the general factor served as predictors in the
structural regression. The general factor significantly predicted depression, anxiety, and stress,
indicating that emotional distress intolerance shares a significant relationship with these clinical
outcomes.
DISTRESS TOLERANCE SCALE
21
After accounting for the general factor, the Regulation factor of the DTS also
significantly predicted depression, anxiety, and stress, but in a theoretically inconsistent
direction. These unexpected results necessitated an examination of net suppression. While
domain-specific factors cannot exhibit suppression effects on other domain-specific factors due
to their orthogonal specification in the bifactor model, content redundancy may still emerge from
the simultaneous loadings of all items on both the general factor and respective domain-specific
factors. In other words, the Regulation factor may still have limited practical value on its own.
As reviewed by Paulhus, Robins, Trzesniewski, and Tracy (2004), suppression and/or
redundancy effects can occur in regression models when multiple predictor variables with strong
correlations are entered simultaneously (for an example of the effect of suppression in a bifactor
framework, see Patrick, Hicks, Nichol, & Krueger, 2007). All latent factors of the DTS have
evidenced strong, positive correlations in past studies (e.g., Ameral et al., 2014; Leyro et al.,
2011), and standardized correlations between the domain-specific factors in the current study
were also quite high, falling within a range of .72 to .94. A second structural regression in which
variance from the DTS general factor was not controlled for yielded a pattern of results in
opposition to the first regression, providing strong evidence for net suppression. Further, the
magnitude of the associations between the Regulation domain-specific factor and our outcome
variables suggests content redundancy. That is, the Regulation factor likely has little to no
incremental utility beyond the general factor in predicting depression, anxiety, or stress, because
it may in fact be redundant with measures of these clinically relevant outcomes. Results such as
these are not new to researchers of individual differences (see Wolgast, 2014).
This study provided the first investigation of measurement invariance in the DTS across
males and females. Past research has suggested that females may demonstrate lower scores on
DISTRESS TOLERANCE SCALE
22
this measure (i.e., poorer tolerance of emotional distress) than males (Cougle et al., 2013;
Dennhardt & Murphy, 2011). To our knowledge, however, no other investigation has sought to
explore whether this difference reflects a true deficit in emotional distress tolerance in females or
measure-specific variance of the DTS items across sexes. Our results provide evidence that the
DTS exhibits adequate measurement invariance across males and females and supports the use of
identical scoring and modeling of the DTS in both sexes. Contrary to past findings, no significant
differences were found in the latent means of the DTS’s general factor or Regulation domain-
specific factor between sexes.
Results of the current investigation support a unidimensional conceptualization of the
DTS, contraindicate independent use of the DTS domain-specific factors, and provide evidence
of measurement invariance across males and females. However, it should be noted that fit of the
unidimensional model was relatively poor, suggesting a potential need for improvements to this
commonly used measure of emotional distress intolerance. For instance, given that modification
indices suggested multiple cross loadings, and that two of the four domain-specific factors
exhibited non-significant residual variance after modeling the general factor (i.e., redundant item
content), future investigations might prioritize refining the DTS by removing truly redundant and
underperforming items. Refinement of the DTS in this way would likely result in a shorter, more
parsimonious measure with greater ease of interpretability and potentially better unidimensional
model fit.
Results should be viewed in light of certain limitations. First, although research supports
the use of MTurk for data collection (Chandler & Shapiro, 2016), and a substantial amount of
literature exists to aid researchers in quality-control procedures and other issues related to online
data collection (see Paolacci et al., 2010, for a discussion), MTurk samples should not be
DISTRESS TOLERANCE SCALE
23
considered representative of the general population. For example, evidence suggests that MTurk
samples tend to be more highly educated and younger than the general population (Paolacci &
Chandler, 2014). As such, replication of study results in both general population and clinical
samples, with more individuals scoring at the high end of the continuum of anxiety and
depression, will help to ensure that results generalize. Second, the current sample, though large,
comprised a female majority and was relatively homogeneous in terms of race and ethnicity. The
acceptability of online sampling notwithstanding, future research with the DTS would do well to
examine the psychometric performance of this measure and to cross-validate its structure in
samples with greater diversity. Third, the current investigation was cross-sectional in design.
Future research would benefit from longitudinal data and experimental paradigms that allow
researchers to better characterize the relationship between emotional distress intolerance and
relevant clinical constructs. For example, research designs that recruit participants prior to
developmentally sensitive periods (e.g., late adolescence, early 20s) and follow such participants
across time would be better suited to examine the prospective relationship between emotional
distress intolerance and the psychogenesis of various forms of psychopathology. Further, in line
with a diathesis-stress model, longitudinal designs across multiple time points could help to
expound on past findings related to emotional distress intolerance and the occurrence of stressful
life events (e.g., traumatic exposure and the development of PTSD).
Lastly, an additional limitation of the current study pertains to the growing body of
research on distress tolerance as a multifaceted, transdiagnostic individual difference factor in
the etiology and maintenance of psychopathology. As reviewed by Zvolensky, Vujanovic,
Bernstein, and Leyro (2010), distress tolerance is believed to comprise five facets of experience
that elicit psychological distress: uncertainty, ambiguity, frustration, physical discomfort, and
DISTRESS TOLERANCE SCALE
24
emotional distress (for empirical support for this conceptualization, see Bardeen et al., 2013b).
The DTS was developed to reflect only emotional distress intolerance, and thus, represents only
a part of the expansive work on the broader construct of distress tolerance. The purpose of this
study was to evaluate the psychometric performance of the DTS within different models and
across sexes. In doing so, we have contributed to this expansive literature base, but must
acknowledge the narrowness of our focus relative to the breadth of the distress tolerance
literature.
DISTRESS TOLERANCE SCALE
25
References
Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary
Aeviews: Computational Statistics, 2(4), 433-459.
Akaiki, H. (1974). A new look at the statistical model identification. IEEE Transactions on
Automatic Control, 19, 716-723.
Ameral, V., Palm Reed, K. M., Cameron, A., & Armstrong, J. L. (2014). What are measures of
distress tolerance really capturing? A mixed methods analysis. Psychology of
Consciousness: Theory, Research, and Practice, 1, 357-369.
Anestis, M. D., Bagge, C. L., Tull, M. T., & Joiner, T. E. (2011). Clarifying the role of emotion
dysregulation in the interpersonal-psychological theory of suicidal behavior in an
undergraduate sample. Journal of Psychiatric Research, 45, 603-611.
Anestis, M. D., Lavender, J. M., Marshall-Berenz, E. C., Gratz, K. L., Tull, M. T., & Joiner, T.
E. (2012). Evaluating distress tolerance measures: Interrelations and associations with
impulsive behaviors. Cognitive Therapy and Research, 36, 593-602.
Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric
properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in
clinical groups and a community sample. Psychological Assessment, 10, 176-181.
Bardeen, J. R., & Fergus, T. A. (2016). Emotional distress intolerance, experiential avoidance,
and anxiety sensitivity: The buffering effect of attentional control on associations with
posttraumatic stress symptoms. Journal of Psychopathology and Behavioral Assessment,
38, 320-329.
DISTRESS TOLERANCE SCALE
26
Bardeen, J. R., Fergus, T. A., Hannan, S. M., & Orcutt, H. K. (2016). Addressing psychometric
limitations of the Difficulties in Emotion Regulation Scale through item modification.
Journal of Personality Assessment, 98, 298–309.
Bardeen, J. R. Fergus, T. A., & Orcutt, H. K. (2013a). Experiential avoidance as a moderator of
the relationship between anxiety sensitivity and perceived stress. Behavior Therapy 44,
459-469.
Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2013b). Testing a hierarchical model of distress
tolerance. Journal of Psychopathology and Behavioral Assessment, 35, 495-505.
Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2014). The moderating role of experiential
avoidance in the prospective relationship between anxiety sensitivity and
anxiety. Cognitive Therapy and Research, 38, 465-471.
Bardeen, J. R., Kumpula, M. J., & Orcutt, H. K. (2013). Emotion regulation difficulties as a
prospective predictor of posttraumatic stress symptoms following a mass shooting.
Journal of Anxiety Disorders, 27, 188-196.
Bardeen, J. R., & Michel, J. S. (2017). The buffering effect of religiosity on the relationship
between intolerance of uncertainty and depressive symptoms. Psychology of Religion and
Spirituality, 9, 90-95.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of
crowdsourcing for survey research. Behavior Research Methods, 43, 800-813.
Bernstein, A., Marshall, E. C., & Zvolensky, M. J. (2011). Multi-method evaluation of distress
tolerance measures and construct(s): Concurrent relations to mood and anxiety
psychopathology and quality of life. Journal of Experimental Psychopathology, 2(3),
386-399.
DISTRESS TOLERANCE SCALE
27
Bernstein, A., Vujanovic, A. A., Leyro, T., & Zvolensky, M. J. (2011). Distress tolerance:
Research synthesis and future directions. In M. J. Zvolensky, A. Bernstein, & A.
Vujanovic (Eds.), Distress tolerance: Theory, research, and clinical applications (pp.
263-280). New York, NY: Guilford Press.
Bonifay, W., Lane, S. P., & Reise, S. P. (2017). Three concerns with applying a bifactor model
as a structure of psychopathology. Clinical Psychological Science, 5, 184-186.
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York,
NY: Guilford Press.
Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selection and
multimodel inference in behavioral ecology: some background, observations, and
comparisons. Behavioral Ecology and Sociobiology, 65, 23-35.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source
of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.
Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced
convenience samples. Annual Review of Clinical Psychology, 12, 53-81.
Cheung, C. W., & Rensvold, R. B. (2002). Evaluating good- ness-of-fit indexes for testing
measurement invariance. Structural Equation Modeling, 9, 233-255.
Corstorphine, E., Mountford, V., Tomlinson, S., Waller, G., & Meyer, C. (2007). Distress
tolerance in the eating disorders. Eating Behaviors, 8, 91-97.
Cougle, J. R., Bernstein, A., Zvolensky, M. J., Vujanovic, A. A., & Macatee, R. J. (2013).
Validation of self-report measures of emotional and physical distress tolerance. Journal
of Psychopathology and Behavioral Assessment, 35, 76-84.
DISTRESS TOLERANCE SCALE
28
Dennhardt, A. A., & Murphy, J. G. (2011). Associations between depression, distress tolerance,
delay discounting, and alcohol-related problems in European American and African
American college students. Psychology of Addictive Behaviors, 25, 595-604.
Dueber, D. M. (2016, December). Bifactor Indices Calculator: A Microsoft Excel-based tool to
calculate various indices relevant to bifactor CFA models. Available at
http://sites.education.uky.edu/apslab/resources.
Ebesutani, C., McLeish, A. C., Luberto, C. M., Young, J., & Maack, D. J. (2014). A bifactor
model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of
Psychopathology and Behavioral Assessment, 36, 452-464.
Ebesutani, C., Smith, A., Bernstein, A., Chorpita, B. F., Higa-McMillan, C., & Nakamura, B.
(2011). A bifactor model of negative affectivity: Fear and distress components among
younger and older youth. Psychological Assessment, 23, 679-691.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
Graham, J. W., & Schafer, J. L. (1999). On the performance of multiple imputation for
multivariate data with small sample sizes. In R. Hoyle (Ed.), Statistical strategies for
small sample research (pp. 1-29). London, England: Sage Publications.
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable
systems. In R. Cudeck, S. Du Toit, & D. Sorbom (Eds.), Structural Equation Modeling:
Present and Future (pp. 195-216). Lincolnwood, IL: Scientific Software International.
Henry, J. D., & Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress
Scales (DASS-21): Construct validity and normative data in a large non-clinical sample.
British Journal of Clinical Psychology, 44, 227–239.
DISTRESS TOLERANCE SCALE
29
Hsu, S. H., Collins, S. E., & Marlatt, G. A. (2013). Examining psychometric properties of
distress tolerance and its moderation of mindfulness-based relapse prevention effects on
alcohol and other drug use outcomes. Addictive Behaviors, 38, 1852-1858.
Hu., L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling: A
Multidisciplinary Journal, 6, 1-55.
Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.).
Thousand Oaks, CA: Sage Publications.
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). New York,
NY: The Guilford Press.
Leyro, T., Bernstein, A., Vujanovic, A. A., McLeish, A., & Zvolensky, M. J. (2011). Distress
Tolerance Scale: A confirmatory factor analysis among daily cigarette smokers. Journal
of Psychopathology and Behavioral Assessment, 33, 47-57.
Leyro, T. M., Zvolensky, M. J., & Bernstein, A. (2010). Distress tolerance and
psychopathological symptoms and disorders: A review of the empirical literature among
adults. Psychological Bulletin, 136, 576-600.
Linehan, M. (1993). Cognitive-behavioral treatment of borderline personality disorder. New
York, NY: The Guilford Press.
Lovibond, S. H., & Lovibond, P. F. (1995). Manuals for the Depression Anxiety Stress Scales
(2nd ed.). Sydney, Australia: Psychology Foundation.
McHugh, R. K., Kertz, S. J., Weiss, R. B., Baskin-Sommers, A. R., Hearon, B. A., &
Björgvinsson, T. (2014). Changes in distress intolerance and treatment outcome in a
partial hospital setting. Behavior Therapy, 45, 232-240.
DISTRESS TOLERANCE SCALE
30
McHugh, R. K., & Otto, M. W. (2012). Refining the measurement of distress tolerance. Behavior
Therapy, 43, 641-651.
Muthén, B., Kaplan, D., & Hollis, M. (1987). On structural equation modeling with data that are
not missing completely at random. Psychometrika, 52, 431-462.
Muthén, L. K., & Muthén, B. O. (2015). MPlus (Version 7.4) [Computer software]. Los Angeles,
CA: Muthén & Muthén.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks:
Detecting satisficing to increase statistical power. Journal of Experimental Social
Psychology, 45, 867–872.
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a
participant pool. Current Directions in Psychological Science, 23, 184-188.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon
Mechanical Turk. Judgment and Decision Making, 5, 411–419.
Patrick, C. J., Hicks, B. M., Nichol, P. E., & Krueger, R. F. (2007). A bifactor approach to
modeling the structure of the Psychopathy Checklist-Revised. Journal of Personality
Disorders, 21, 118-141.
Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004). Two replicable
suppressor situations in personality research. Multivariate Behavioral Research, 3, 303-
328.
Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral
Research, 47, 667-696.
DISTRESS TOLERANCE SCALE
31
Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological
measures in the presence of multidimensionality. Journal of Personality Assessment, 95,
129-140.
Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring
the extent to which multidimensional data yield univocal scale scores. Journal of
Personality Assessment, 92, 544-559.
Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating
and interpreting statistical indices. Psychological Methods, 21, 137-150.
Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in
covariance structure analysis. In C. C. Clogg, & A. von Eye (Eds.), Latent variables
analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA:
Sage Publications.
Shorey, R. C., Strauss, C., Elmquist, J., Anderson, S., Cornelius, T. L., & Stuart, G. L. (2017).
Distress tolerance and intimate partner violence among men in substance use treatment.
Journal of Family Violence, 32, 317-324.
Simons, J. & Gaher, R. (2005). The Distress Tolerance Scale: Development and validation of a
self-report measure. Motivation and Emotion, 29, 83-102.
Stucky, B. D., & Edelen, M. O. (2015). Using hierarchical IRT to create unidimensional
measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook
of item response theory modeling: Applications to typical performance assessment (pp.
183-206). New York, NY: Routledge.
Stucky, B. D., Thissen, D., & Edelen, M. O. (2013). Using logistic approximations of marginal
trace lines to develop short assessments. Applied Psychological Measurement, 37, 41-57.
DISTRESS TOLERANCE SCALE
32
Sugiura, N. (1978). Further analysts of the data by akaike's information criterion and the finite
corrections: Further analysts of the data by akaike's. Communications in Statistics-Theory
and Methods, 7, 13-26.
Suhr, D. D. (2005). Principal component analysis vs. exploratory factor analysis. SUGI 30
Proceedings. Retrieved from https://pdfs.semanticscholar.org/ 936e/
76d8f2203cb534a3952d3ea7032fc57f8ad0.pdf
Timpano, K. R., Buckner, J. D., Richey, J. A., Murphy, D. L., & Schmidt, N. B. (2009).
Exploration of anxiety sensitivity and distress tolerance as vulnerability factors for
hoarding behaviors. Depression and Anxiety, 26, 343-353.
Veilleux, J. C., Pollert, G. A., Zielinski, M. J., Shaver, J. A., & Hill, M. A. (in press). Behavioral
assessment of the negative emotion aspect of distress tolerance: Tolerance to emotional
images. Assessment.
Vujanovic, A. A., Bernstein, A., & Litz, B. (2011). Traumatic stress. In M. Zvolensky, A.
Bernstein, & A. Vujanovic (Eds.). Distress tolerance: Theory, research, and clinical
applications. New York, NY: The Guilford Press.
Vujanovic, A. A., Bonn-Miller, M. O., Potter, C. M., Marshall, E. C., & Zvolensky, M. J. (2011).
An evaluation of the relation between distress tolerance and posttraumatic stress within a
trauma-exposed sample. Journal of Psychopathology and Behavioral Assessment, 33,
129-135.
Wang, M., & Russell, S. S. (2005). Measurement equivalence of the Job Descriptive Index
across Chinese and American workers: Results from confirmatory factor analysis and
item response theory. Educational and Psychological Measurement, 65, 709-732.
DISTRESS TOLERANCE SCALE
33
Wolgast, M. (2014). What does the Acceptance and Action Questionnaire (AAQ) really
measure? Behavior Therapy, 45, 831-839.
Zvolensky, M. J., Bernstein, A., & Vujanovic, A. A. (Eds.). (2011). Distress tolerance: Theory,
research, and clinical applications. New York, NY: The Guilford Press.
Zvolensky, M. J., & Otto, M. W. (2010). Affective intolerance, sensitivity, and processing:
Advances in clinical science introduction [Special series]. Behavior Therapy, 38, 228-
233.
Zvolensky, M. J., Vujanovic, A. A., Bernstein, A., & Leyro, T. (2010). Distress tolerance:
Theory, measurement, and relations to psychopathology. Current Directions in
Psychological Science, 19, 406-410.
DISTRESS TOLERANCE SCALE
34
Footnotes
1A measure by the same name (Corstorphine, Mountford, Tomlinson, Waller, & Meyer,
2007) has been developed as another measure of distress tolerance. However, this measure was
developed and validated in a sample of treatment-seeking females with disordered eating, and
there are notable conceptual differences between the purported constructs of this measure and the
DTS of interest in the present study (e.g., an emphasis on emotion regulation strategies rather
than tolerance of emotional distress per se). Therefore, Corstorphine et al.’s measure and
confirmatory factor analytic work were not considered in our literature review.
2Confirmatory factor analysis was used to test the post-hoc hypothesis that a modified
bifactor model of the DTS (i.e., with the general factor and the Regulation domain-specific
factor) would provide adequate fit to the data. Model fit for the modified bifactor model was as
follows: χ2(87) = 556.97 (p < .001), RMSEA = .081 (90% CI = .075 - .087), CFI = .918, TLI =
.901, SRMR = .042. Significant Δχ2[12] of 140.13 (p < .001) suggested that the modified
bifactor model provided significantly worse fit to the data than the full bifactor model. However,
RMSEA 90% CIs between the two models were overlapping.
3Readers are directed to supplemental Figure 1 for a depiction of the structural regression
model and parameter estimates.
4Depression, anxiety, and stress shared positive correlations that were large in magnitude,
rdep-anx = .81, rdep-stress = .83, ranx-stress = .86, ps < .001. Thus, consistent with previous research
(Bardeen, Kumpula, & Orcutt, 2013), the DASS-21 was also modeled as a general distress
construct in a second structural regression model. Specifically, general distress was modeled as a
second-order factor; correlations between lower-order factors (i.e., anxiety, depression, stress)
were omitted and direct pathways were modeled from the second-order factor (i.e., general
DISTRESS TOLERANCE SCALE
35
distress) onto each lower-order factor. The DTS general factor and domain-specific factors, from
the bifactor model, were simultaneously regressed onto the general distress latent factor, as
measured via the DASS-21. Consistent with the first structural regression model, only the
Regulation factor and general factor were included as predictors in the model. The results of this
regression were as follows. Model fit was adequate, χ2(586) = 1734.62 (p < .001), RMSEA =
.049 (90% CI = .046-.051), CFI = .926, TLI = .921, SRMR = .052. The general factor of the
DTS significantly predicted general distress in a theoretically consistent direction, β = 0.45, p <
.001, while the Regulation domain-specific factor significantly predicted general distress in a
theoretically inconsistent direction, β = -0.13, p < .01.
5The full item-level correlation matrix is presented in supplemental Table 1.
DISTRESS TOLERANCE SCALE
36
Table 1
Review of Psychometric Literature
Simons & Gaher, 2005 Leyro et al., 2011 Hsu et al., 2013
Sample size
and
composition
Sample 1: 642 students
(70% female);
Sample 2: 823 students
(67% female)
173 adult cigarette
smokers from
community (45%
female)
168 adults seeking
treatment for substance
use (36% female)
Statistical
approach
EFA and CFA
CFA PCA
Items used 1 - 15 1 - 13, 15
(item 14 omitted)
1 - 5, 7 - 15
(item 6 omitted)
Structure
derived
1 general factor, 4
specific factors
(Tolerance,
Absorption, Appraisal,
and Regulation)
1 general factor, 4
specific factors
(Tolerance, Absorption,
Appraisal, and
Regulation)
1 general factor
Areas of
concern
Mean differences in
general distress
tolerance between
males and females
Omission of item 14;
standardized residuals
≥ 1.96; potential item-
cross loadings; strong
inter-factor correlations
(rs > .80)
Omission of item 6;
generalizability of PCA
results with past CFA
work
Note. EFA – exploratory factor analysis; CFA – confirmatory factor analysis; PCA – principle
component analysis.
DISTRESS TOLERANCE SCALE
37
Table 2
Model Comparisons
Model AICc χ2 (df) Scaling
Factor
RMSEA [90%
CI] CFI TLI SRMR
Correlated Four-
Factor 34895.60 432.12* (84) 1.308 .071 [.064-.078] .939 .924 .039
One-Factor 35424.30 824.78* (90) 1.343 .099 [.093-.106] .871 .850 .054
Hierarchical 34910.83 445.61* (86) 1.313 .071 [.065-.078] .937 .923 .041
Bifactor 34841.69 390.70* (75) 1.256 .071 [.064-.078] .945 .923 .036
Configural
invariance 34874.90
663.56*
(174) 1.304 .083 [.076-.090] 0.915 .898 .045
Metric
invariance 34847.24
692.92*
(190) 1.267 .080 [.074-.087] 0.913 .904 .049
Scalar
invariance 34857.43
735.82*
(203) 1.249 .080 [.074-.086] 0.908 .908 .052
Configural-
Metric Δ 27.66 14.56 (16) .003 .002 -.006 -.004
Configural-
Scalar Δ 17.47 58.70* (13) .003 .007 -.010 -.007
Metric-Scalar Δ -10.19 41.76* (29) .000 .005 -.004 -.003
Note. AICc = sample-size corrected Akaike information criterion; * indicates p < .001; df =
degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence
interval; CFI = comparative fit index; TLI = Tucker-Lewis fit index; SRMR = standardized root
mean square.
DISTRESS TOLERANCE SCALE
38
Table 3
Bifactor Evaluation Indices and Standardized Factor Loadings
General Factor Tolerance Absorption Appraisal Regulation
ω/ωS .95 .83 .88 .87 .84
ωH/ ωHS .91 .11 .07 .10 .35
ECV .80 .16 .12 .16 .42
H .95 .27 .25 .42 .55
FD .96 .67 .69 .75 .84
Residual variance .89 .37^ .44^ .06 .26
Item Number I-ECV Factor Loadings
1 .71 .72 .46
3 .90 .80 .27
5 .97 .66 .13
2 .70 .75 .49^
4 1.00 .83 .06^
15 .97 .81 .13^
6† 1.00 .42 .02^
7 .93 .66 .19
9 .97 .76 .13^
10 .97 .82 .14^
11 .53 .63 .59
12 .83 .73 .34
8 .72 .65 .40
13 .50 .64 .64
14 .54 .53 .49 Note. ω = omega; ωS = omega subscale; ωH = omega hierarchical; ωHS = omega hierarchical subscale; ECV =
explained common variance; H = Hancock and Mueller (2001) replicability indicator; FD = factor
determinacy; all factor loadings and residual variances significant at p < .05, except for those identified by ^;
^ indicates p > .05; † indicates reverse-scored item.
Table S1 Full Inter-Item Correlation Matrix
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 DASS1 1.00 2 DASS2 0.42 1.00 3 DASS3 0.54 0.47 1.00 4 DASS4 0.43 0.45 0.53 1.00 5 DASS5 0.47 0.36 0.56 0.43 1.00 6 DASS6 0.50 0.35 0.54 0.43 0.49 1.00 7 DASS7 0.40 0.37 0.43 0.48 0.32 0.43 1.00 8 DASS8 0.54 0.38 0.50 0.48 0.46 0.51 0.54 1.00 9 DASS9 0.44 0.37 0.52 0.47 0.49 0.55 0.51 0.59 1.00 10 DASS10 0.45 0.36 0.70 0.39 0.57 0.48 0.39 0.47 0.51 1.00 11 DASS11 0.54 0.38 0.58 0.42 0.55 0.62 0.42 0.58 0.55 0.58 1.00 12 DASS12 0.65 0.36 0.57 0.40 0.53 0.55 0.41 0.61 0.52 0.53 0.73 1.00 13 DASS13 0.52 0.38 0.70 0.44 0.61 0.53 0.38 0.48 0.49 0.71 0.62 0.61 1.00 14 DASS14 0.52 0.36 0.53 0.40 0.41 0.48 0.39 0.50 0.43 0.46 0.57 0.55 0.50 1.00 15 DASS15 0.52 0.40 0.57 0.51 0.43 0.52 0.48 0.60 0.62 0.52 0.57 0.60 0.55 0.53 1.00 16 DASS16 0.51 0.39 0.71 0.46 0.62 0.51 0.39 0.50 0.51 0.74 0.61 0.62 0.72 0.53 0.62 1.00 17 DASS17 0.47 0.39 0.67 0.41 0.53 0.50 0.40 0.48 0.56 0.73 0.55 0.53 0.70 0.48 0.58 0.70 1.00 18 DASS18 0.54 0.35 0.55 0.37 0.49 0.60 0.41 0.52 0.48 0.52 0.66 0.62 0.59 0.55 0.56 0.58 0.57 1.00 19 DASS19 0.38 0.42 0.45 0.57 0.37 0.40 0.47 0.44 0.43 0.39 0.41 0.40 0.40 0.38 0.50 0.43 0.40 0.38 1.00 20 DASS20 0.47 0.39 0.56 0.56 0.46 0.52 0.54 0.56 0.60 0.51 0.50 0.50 0.50 0.45 0.63 0.55 0.57 0.49 0.54 21 DASS21 0.45 0.38 0.68 0.41 0.52 0.51 0.45 0.47 0.53 0.74 0.53 0.51 0.68 0.44 0.53 0.68 0.78 0.53 0.41 22 DTS1 0.13 0.07 0.23 0.12 0.18 0.19 0.08 0.19 0.22 0.23 0.23 0.21 0.25 0.16 0.19 0.21 0.25 0.17 0.09 23 DTS2 0.17 0.09 0.20 0.13 0.24 0.20 0.07 0.18 0.19 0.24 0.23 0.25 0.25 0.14 0.18 0.21 0.24 0.17 0.10 24 DTS3 0.19 0.13 0.28 0.17 0.23 0.25 0.15 0.22 0.29 0.29 0.29 0.29 0.29 0.19 0.27 0.29 0.30 0.23 0.13 25 DTS4 0.27 0.17 0.33 0.22 0.27 0.32 0.18 0.27 0.33 0.33 0.31 0.30 0.38 0.25 0.33 0.33 0.37 0.26 0.17 26 DTS5 0.09 0.10 0.13 0.06 0.11 0.09 0.03 0.11 0.12 0.17 0.15 0.14 0.18 0.06 0.13 0.14 0.18 0.09 0.04 27 DTS6 0.23 0.18 0.32 0.21 0.21 0.31 0.17 0.21 0.30 0.29 0.25 0.24 0.31 0.22 0.28 0.29 0.30 0.20 0.21 28 DTS7 0.13 0.11 0.22 0.07 0.18 0.15 0.10 0.20 0.20 0.19 0.17 0.17 0.20 0.14 0.17 0.20 0.23 0.15 0.05 29 DTS8 0.10 0.07 0.15 0.05 0.11 0.11 0.08 0.13 0.17 0.17 0.14 0.12 0.17 0.10 0.14 0.15 0.16 0.09 0.04 30 DTS9 0.23 0.16 0.30 0.13 0.27 0.27 0.10 0.23 0.29 0.31 0.28 0.26 0.34 0.18 0.25 0.31 0.32 0.23 0.14 31 DTS10 0.23 0.12 0.30 0.16 0.27 0.31 0.16 0.23 0.31 0.33 0.33 0.31 0.34 0.20 0.27 0.30 0.32 0.26 0.13 32 DTS11 0.23 0.17 0.26 0.15 0.24 0.23 0.18 0.25 0.32 0.29 0.29 0.27 0.30 0.20 0.28 0.29 0.35 0.23 0.14 33 DTS12 0.25 0.17 0.31 0.22 0.26 0.27 0.24 0.33 0.37 0.33 0.32 0.32 0.34 0.25 0.36 0.33 0.34 0.24 0.20 34 DTS13 0.15 0.08 0.21 0.11 0.15 0.16 0.11 0.20 0.22 0.20 0.18 0.17 0.21 0.15 0.22 0.23 0.20 0.12 0.10 35 DTS14 0.06 0.03 0.10 0.04 0.10 0.08 0.00 0.10 0.13 0.11 0.12 0.11 0.12 0.09 0.10 0.14 0.10 0.06 0.00 36 DTS15 0.24 0.16 0.31 0.19 0.29 0.28 0.15 0.28 0.34 0.31 0.32 0.31 0.34 0.22 0.30 0.34 0.32 0.24 0.17 37 SEX 0.00 -0.04 -0.07 -0.06 0.03 0.02 -0.08 -0.05 -0.04 -0.01 -0.04 -0.01 0.04 -0.03 -0.01 0.00 -0.03 0.02 -0.07
Mean 1.04 0.92 0.74 0.50 1.15 0.92 0.41 0.76 0.68 0.86 1.08 1.14 1.01 0.80 0.59 0.82 0.77 0.94 0.71 SD 0.99 1.04 0.94 0.84 1.03 0.98 0.76 0.94 0.95 1.05 0.98 1.04 1.03 0.94 0.90 1.02 1.05 0.97 0.93 n 820 818 821 821 821 821 819 818 821 821 821 820 819 819 821 820 822 820 820
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 1 DASS1 2 DASS2 3 DASS3 4 DASS4 5 DASS5 6 DASS6 7 DASS7 8 DASS8 9 DASS9
10 DASS10 11 DASS11 12 DASS12 13 DASS13 14 DASS14 15 DASS15 16 DASS16 17 DASS17 18 DASS18 19 DASS19 20 DASS20 1.00 21 DASS21 0.60 1.00 22 DTS1 0.19 0.25 1.00 23 DTS2 0.14 0.25 0.63 1.00 24 DTS3 0.25 0.30 0.70 0.68 1.00 25 DTS4 0.30 0.36 0.58 0.65 0.70 1.00 26 DTS5 0.07 0.15 0.54 0.54 0.56 0.58 1.00 27 DTS6 0.24 0.26 0.29 0.28 0.31 0.39 0.18 1.00 28 DTS7 0.19 0.22 0.52 0.43 0.55 0.53 0.47 0.27 1.00 29 DTS8 0.10 0.20 0.51 0.53 0.50 0.46 0.53 0.19 0.56 1.00 30 DTS9 0.24 0.34 0.51 0.55 0.60 0.63 0.45 0.39 0.55 0.47 1.00 31 DTS10 0.23 0.34 0.56 0.57 0.63 0.70 0.51 0.39 0.56 0.55 0.68 1.00 32 DTS11 0.27 0.30 0.46 0.43 0.52 0.52 0.37 0.27 0.54 0.40 0.55 0.59 1.00 33 DTS12 0.33 0.34 0.55 0.49 0.58 0.62 0.45 0.30 0.53 0.45 0.58 0.67 0.66 1.00 34 DTS13 0.19 0.23 0.48 0.43 0.49 0.47 0.48 0.23 0.49 0.67 0.47 0.53 0.41 0.53 1.00 35 DTS14 0.06 0.11 0.42 0.39 0.38 0.38 0.44 0.13 0.40 0.54 0.36 0.46 0.35 0.40 0.65 1.00 36 DTS15 0.26 0.31 0.58 0.67 0.61 0.69 0.52 0.34 0.50 0.51 0.64 0.68 0.55 0.64 0.55 0.50 1.00 37 SEX -0.11 -0.06 0.03 0.08 0.03 0.03 0.06 -0.03 -0.07 0.06 0.04 0.07 0.03 0.01 0.06 0.11 0.10 1.00
Mean 0.55 0.66 2.82 2.87 2.59 2.48 2.87 2.58 2.58 2.85 2.69 2.56 2.51 2.33 2.83 3.00 2.81 1.61 SD 0.88 1.01 1.31 1.36 1.32 1.42 1.36 1.27 1.28 1.27 1.37 1.37 1.40 1.35 1.29 1.28 1.37 0.49 n 822 819 824 824 825 822 823 823 826 822 825 825 822 822 824 823 823 821
Note: DASS – Depression Anxiety Stress Scales; DTS – Distress Tolerance Scale; SD – standard deviation; n – number of cases with available data per variable
Figure 1. Structural regression with the DTS modified bifactor model.
Note: dts# = DTS items; gendt = general factor of the DTS bifactor model; reg = Regulation domain-specific factor of
the DTS bifactor model; dep = Depression subscale of the DASS-21; anx = Anxiety subscale of the DASS-21; str =
Stress subscale of the DASS-21; dass# = DASS-21 items; all parameter estimates fully standardized (i.e., STDXY
standardization); all estimates significant at p < .01.