MEASUREMENT: VALIDITY

Lu Ann Aday, Ph.D.The University of Texas School of Public Health

VALIDITY: Definition Extent of systematic departure

in answers to questions in relationship to the meaning of the concept itself (content), equivalent measures of the same concept (criterion), or hypothesized associations with measures of similar or different concepts (construct).

VALIDITY: Types

Content validityCriterion validityConstruct validity

VALIDITY: Computation Requires analyzing the

association between survey measures and documented definitions of the underlying concept (content); an equivalent “gold standard” of the survey measures (criterion); or theoretical associations with other abstract concepts (construct).

VALIDITY: Content

Definition: extent to which the survey measures adequately represent the concept of interest

VALIDITY: Content

Factors affecting: Failure to thoroughly conduct background research on the definition & measurement of the study concept

Failure to clearly define the study concept

VALIDITY: Content Computation:

1. Identify domain or universe of interest through a thorough review of the literature.

2. Select or construct questions based on the review.

3. Ask subject matter experts to review the items you have selected or constructed.

VALIDITY: Content Validity criteria:

1. Is there at least one item to represent the concept?

2. Are there enough items? 3. Are the items well accepted, based on the literature & expert opinion?

4. Are they reliable?

VALIDITY: Criterion

Definition: extent to which the survey measure predicts (predictive) or agrees with (concurrent) a criterion indicator of the concept of interest

VALIDITY: Criterion Factors affecting:

Underreporting due to problems with recalling non-salient or frequently occurring events

Overreporting due to problems with telescoping salient events

VALIDITY: Criterion- Predictive: Correlation Analysis

Computation: Compute correlation coefficient between answers to survey question and future criterion source value for same concept :Respondent Survey Question MD Observation1 ADLs=3 ADLs=3 2 ADLs=2 ADLs=2 3 ADLs=1 ADLs=1 44 ADLs=0ADLs=0 ADLs=3ADLs=35 ADLs=0 ADLs=0

VALIDITY: Criterion- Concurrent: Correlation Analysis

Computation: Compute correlation coefficient between answers to survey question and current criterion source value for same concept:

Respondent Survey Question Clinic Record1 BP=140/90 BP=140/90 2 BP=150/80 BP=150/80 3 BP=145/95 BP=145/95

44 BP=145/95BP=145/95 BP=120/80BP=120/805 BP=140/90 BP=140/90

VALIDITY: Criterion-Concurrent: Sensitivity-Specificity Analysis

CRITERION /SURVEY

IS (+)Hypertensive

IS NOT (-)Hypertensive

IS (+)Hypertensive

a = true +Sensitivity =a/a + c

b = false +

IS NOT (-)Hypertensive

c = false - d = true –Specificity =d/b + d

VALIDITY: Criterion-Concurrent: Sensitivity-Specificity Analysis

CRITERION /SURVEY

IS (+)Hypertensive

IS NOT (-)Hypertensive

IS (+)Hypertensive

a = 130Sensitivity =130/180= 72.2%

b = 1,000

IS NOT (-)Hypertensive

c = 50 d = 64,000Specificity =64,000/65,000=98.5%

VALIDITY: Construct Definition: extent to which

relationships between measures agree with relationships predicted by theories or hypotheses Convergent – Ho: Related Discriminant – Ho: Not Related

VALIDITY: Construct

Factors affecting: Inadequate conceptualization or measurement of study constructs

Inadequate theory to guide specification of hypothesized relationships

VALIDITY: Construct: Correlation Analysis

Computation: Compute correlation coefficient between survey question or scales measuring construct and related (or unrelated) constructs :Respondent Physical Mental Overall

1 Good FairFair Good2 Good FairFair Good3 Good FairFair Good44 Good FairFair Good5 Good FairFair Good

VALIDITY: Construct- Known Groups Validity Analysis (McHorney, et al., 1993, Table 3, p. 255)

COMPARISON GROUPS/MOS-SF 36 SUMMARY SCALES

MINOR MEDICAL

SERIOUS MEDICAL

PSYCHIATRIC ONLY

PSYCHIATRIC& SERIOUSMEDICAL

PhysicalFunctioning

80.53 57.35 80.62 46.37

MentalHealth

82.49 77.59 52.75 56.90

VALIDITY: CONSTRUCT-Factor Analysis(McHorney, et al., 1993, Table 1, p. 251)

FACTOR LOADINGS (HYPOTHESES)/SUBSCALES

Physical Mental

Physical functioning 0.88 (+) 0.04

Role-physical 0.78 (+) 0.30

Bodily pain 0.77 (+) 0.24

Mental health 0.12 0.90 (+)

Role-emotional 0.19 0.81 (+)

Social Functioning 0.44 (~) 0.71 (+)

Vitality 0.59 (~) 0.57 (~)

General health perceptions

0.68 (~) 0.32 (~)

VALIDITY: Construct—Multi-Trait, Multi-Method

Trait X Method

A B C A B C A B C

A (.90)

B (.85)

C (.80)

A .75 (.80)

B .55 (.75)

C .35 (.80)

A .80 .85 (.95)

B .55 .55 (.90)

C .30 .30 (.65)

VALIDITY: Effect of Reliability

Theoretical Variable can be no more valid than it is reliable.

Empirical Rt = theoretical correlation between study &

criterion variables (unobservable) Rx = reliability of study variable (computed) Ry = reliability of criterion variable (computed) Rt * sqrt [ Rx * Ry] = Max. achievable

correlation 1.00 * sqrt [.60 * .80] = .69

WHEN TO UNDERTAKE VALIDITY ANALYSIS

VALIDITY/DIMENSIONS

CONTENT CRITERION CONSTRUCT

QUESTIONS All questions Factual questions for which “gold standard” is available

Summary scales of attitudes or other abstract concepts

STUDIES All types of surveys

Esp. used in behavioral surveys

Esp. used in attitudinal surveys

STAGES Question development

Pretest or final study

Pretest or final study

REFERENCES DeVellis, Robert F. (2003). Scale Development:

Theory and Applications. Second Edition. Thousand Oaks, CA: Sage.

Gandek, B., & Ware, J.E., Jr., for the IQOLA Project (1998). Methods for validating and norming translations of health status questionnaires: The IQOLA Project Approach. J. Clinical Epidemiology, 51 (11), 953-959.

McHorney, C.A., Ware, J.E., Jr., & Raczek, A.E. (1993). The MOS 36-item short-form health survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), 247-263. [Note: Tables 1 and 3 adapted and reprinted with the permission of Lippincott, Williams & Wilkins, Publishers.]