Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics 5 th Annual...

Methods to Detect and Control Bias Methods to Detect and Control Bias in Comparative Analyses of in Comparative Analyses of

Disability StatisticsDisability Statistics

55thth Annual Meeting of the Washington Group on Annual Meeting of the Washington Group on

Disability StatisticsDisability Statistics Rio de Janeiro, Sept 21-23 2005Rio de Janeiro, Sept 21-23 2005

Bedirhan Bedirhan ÜÜststüün & Nenad Kostanjsekn & Nenad KostanjsekClassifications Assessment TerminologyClassifications Assessment Terminology

Measurement and Health Information SystemsMeasurement and Health Information Systemswww.who.int/classifications/icf

CLASSIFICATIONS … … BUILDING BLOCKS OF HEALTH & Disability INFORMATION …BUILDING BLOCKS OF HEALTH & Disability INFORMATION …


Temperature: Hot, Mild or Cold ???

Measured Real temperature 20 C:

• Iceland - Hot • Germany- Mild • Brazil - Cold


Big Mac Index

• University Professor in Turkey gets 3000 USD.• University Professor in the Switzerland gets 8000 USD.

• Big Mac in TR = 1.5 USD• Big Mac in CH = 4 USD

• Salary = 2000 Big Macs

in both


Health & Disability Survey Data

• prove useful to complement National Health & Disability Information Systems– more representative of entire populations– direct information from individuals on their own experiences

rather than professionals or surveillance information

• can be tailored to specific management and evaluation purposes in mind:– provide empirical data for system performance assessment

• More like to be standard and accurate – gathered more carefully


Numerous

National and International Health & Disability Surveys exist and they differ in:

• Content• Questions• Response scales• Samples• Timing

DOWNSIDE of Surveys


Problem of Comparability in International Health Data

• Mortality statistics 3 x

• Health Survey data 6 – 8 x

• Disability Survey data 30 – 60 x

• Health Interventions 40 – 300 x


Methods for Comparability

1. Same ConceptsExamples:

– Disability = Impairment, Activity limitation & Participation restriction (ICF definition)

– Disability = Impairments & simple activities

– Hearing = body function– Communicating = activity and participation domain

– The self-care category includes the sub-categories: washing, dressing, toileting….



2. Same Domains & Questions

Ability to See

0

10

20

30

40

50

60

1 2 3 4 5

Scale: None - Mild - Moderate - Severe - Complete

Pe

rce

nta

ge



2. Same Domains & Questions (cont.)

Seeing and recognizing Person Across Road

0

20

40

60

80

100

1 2 3 4 5

Scale: None - Mild - Moderate - Severe - Complete

Pe

rce

nta

ge



3. Same response scale (cont.)

• Different response categories for severity scale

– none, mild, moderate, severe…

– Numerical scales (0,1,2,3,4) (1,2,3,4,)

– Likert visual

– Percentage visual scale

• Different response categories for frequency scales

– always, usually, sometimes, never or all the time,

– once, a few times, sometimes, many times



3. Same response scale (cont.)



4. Translation protocols• Translation/interpretation of questions

• Translation/interpretation of response scales

• The meaning of the underlying construct may vary

• The relation between the questions and the underlying construct may vary



4. Translation protocols (cont.)

• The item is difficult or impossible to translate because there is no equivalent term in the local language.

• The item, when translated, has a meaning that is more narrow than the original term. Only part of the connotations of the original term carries over into the local language.

• The item, when translated, has a meaning that is expanded beyond the meaning found in the original. The translated items has additional meanings that would change the interpretation of the item.

• The item is translated into exactly the same word that has been used for another item in the questionnaire. Words that are different concepts in the questionnaire become merged into the same concept in the local language.

•

• The item can be translated, but there is a cultural applicability problem with the item, or with the definition, or with the examples used for the item.



5. Same application mode – same stimulus

RespondentInput

Output



6. Measuring and adjusting for shifts in cut points “Different Rulers”

Do response categories mean the same in different populations?

Very good

Very bad

Good

Bad

Moderate

True Mobility Response Scale A B C


Systematically inconsistent reporting in different surveys • men and women• age groups• education groups• economic level• education levels• cultural and ethnic groups• occupations• levels of health insurance.


Methods for adjusting for shifts in cut points

Calibration Tests: Vision used extensively in

international studies

Distant Visionstandard Snellen’s chart

Near vision standard card for use in

illiterate populations

used extensively in international studies

Distant Visionstandard Snellen’s chart

Near vision standard card for use in

illiterate populations


Methods for adjusting for shifts in cut points

Calibration Tests: Cognition

•Verbal fluency

•names of animals in one minute

•record correct number & errors

•Simple Memory

•Immediate recall

•delayed recall (20 min.) of word list

•Colour cancellation test

•time taken

•number of correct and wrong cancellationsselected from WHO international studies on

Culture-fair Cognitive Battery


Methods for adjusting for shifts in cut points Calibration Tests: Mobility

•Posturo-Locomotor-Manual test

•Time to stand up

•Time taken to complete task

•3 trialsstand up from sitting positionwalk one steplift up weight of 2 kgswalk 6 metresplace weight at shoulder weightlift up weightwalk back towards chairplace weight back return to chair and sit down


Methods for adjusting for shifts in cut points VignettesVignettes

Mobility [Mary] has no problems with walking, running or using her hands, arms and legs. She jogs 4 kilometres twice a week.

[Anton] does not exercise. He cannot climb stairs or do other physical activities because he is obese. He is able to carry the groceries and do some light household work.

[David] is paralyzed from the neck down. He is unable to move his arms and legs or to shift body position. He is confined to bed.


“bias”Differential Item Functioning

• a distortion (possibly systematic and non-random) of the results

• due to a measurement error affects all measurements in the same way

• violates the fair comparison of results across different groups– substantive implications?– statistically significant?– random or systematic?


Methods for Cross-Population Comparability

Same concepts

Same domains & questions

Same response scales

Translation protocols

Same application mode – same

stimulus

Correction for confounders (education,

income…) using known calibrators

Use modern Item Response Theory

(IRT) approaches

Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics 5 th Annual...

Documents

Transcript of Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics 5 th Annual...