Verbal Autopsy Modules in Surveys
______________________________________________________________
Henry Kalter, MD, MPHHenry Kalter, MD, MPHJohns Hopkins Bloomberg School of Public HealthJohns Hopkins Bloomberg School of Public Health
Baltimore, MD, USABaltimore, MD, USA
Sneak Preview
Some available VA questionnairesSome available VA questionnaires
Use of VA in national surveys (DHS)Use of VA in national surveys (DHS)
Misclassification error and possible solutionsMisclassification error and possible solutions
Available VA tools and ongoing developments WHO standard VA for infants and children (1999)WHO standard VA for infants and children (1999)
Developed by WHO/JHSPH/LSHTM
Tanzania Adult M&M Project (AMMP)Tanzania Adult M&M Project (AMMP)
INDEPTH Network standardized VA (2003)INDEPTH Network standardized VA (2003) Built on WHO infant/child and AMMP adult formats
SAVVY VA for neonates, children and adultsSAVVY VA for neonates, children and adults Developed by Measure Evaluation in collaboration with
HMN for use in a nationally representative sample or selected sentinel area
Also can be used in surveys or censuses Used by India SRS, which claims better ascertainment of laims better ascertainment of
births and deaths than single surveysbirths and deaths than single surveys
Available VA tools and ongoing developments
WHO consensus group VA for neonates, children WHO consensus group VA for neonates, children and adults (2003–2007)and adults (2003–2007) Sponsored by HMN: to replace SAVVY VA tool and be part
of ‘HMN’s Stepping Stones’ resource kit for strengthening national vital statistics systems
Modules: birth-27 days, 28 days-14 years, 15+ years
Ongoing Harvard/JHBSPH/Queensland neonatal, Ongoing Harvard/JHBSPH/Queensland neonatal, child and adult VA validation study (Gates GC13)child and adult VA validation study (Gates GC13) Tanzania, Philippines, India (2 sites) Modules: birth-27 days, 28 days-11 years, 12+ years Will compare results of three analytic methods
Individual causes, by algorithms and physician readers All causes of death at once, by symptom profiles
National surveys that use VA
MACRO’s DHS (16/187 surveys, 1987–2007)MACRO’s DHS (16/187 surveys, 1987–2007) Stillbirths, child (NN, 1-11 mo, 12-59 mo), maternal deaths No non-maternal adult deaths
UNICEF’s MICS (limited COD information)UNICEF’s MICS (limited COD information) AIDS: ‘anyone aged 18-59 years who died in past 12 months
and was seriously ill for 3/12 months before death’ MM: sisterhood method + ‘death during pregnancy,
childbirth or within 6 weeks after the end of pregnancy’
Other national health surveys, e.g., Turkey 2003Other national health surveys, e.g., Turkey 2003
Country DHS with VA Subsequent DHS without VA
Morocco 1987 1992, 2003-04
Egypt 1988 1992, 1995, 2000, 2005
Cameroon 1991 1998, 2004
Namibia 1992 2000, 2006
Bolivia 1994 1998, 2003
CAR 1994-95 --
Haiti 1994-95 2000, 2005
Chad 1996-97 2004
Nigeria 1999 2003
Bangladesh 20042007 (but 1993-94 & 1996-97 w/VA: showed declines in most causes)
Cambodia 2005 --
Honduras 2005 --
Nepal 2006 --
Pakistan 2006 --
Angola 2006 --
Uganda 2007 --
Country ModulesReference period Analysis
Morocco<6 years: identified death but not cause -- --
Egypt<5 years: accident, 8 Sxs, 2 Dxs Child: 5 years
NN, 1-11 mo, 12-59 mo: Sxs (sum >100%)
Cameroon<5 years: accident or illness type, 9 Sxs Child: 5 years
NN, 1-59 mo: combine mother’s opinion &/or algorithm, e.g., diarrhea=algorithm, malaria= mother or algorithm (sum>100%)
Namibia
NN: COD, 9 Sxs1-59 mo: COD, 19 SxsMAT: preg/deliver/6wks
Child: 5 years MAT: sisterhood
NN, 1-59 mo: non-specific algorithms (e.g., measles: age >4 mo + rash (sum>100%)MAT: direct (age & year of death)
Bolivia MAT: preg/deliver/6wks MAT: sisterhood MAT: direct
Central African Republic
NN: COD, 9 Sxs1-35 mo: COD, 19 SxsMAT: preg/deliver/6wks
Child: 3 yearsMAT: sisterhood
NN, 1-35 mo: combine mother’s opinion &/or algorithm (sum>100%)MAT: direct
HaitiNN: COD, 7 Sxs1-59 mo: COD, 16 Sxs Child: 5 years
NN, 1-11 mo, 12-59 mo: combine mother’s opinion &/or algorithm (sum>100%)
Country ModulesReference period Analysis
Chad
NN: COD, 9 Sxs1-59 mo: COD, 19 SxsMAT: preg/deliver/2mo
Child: 5 yearsMAT: sisterhood
NN, 12-59 mo: combine mother’s opinion &/or algorithm (sum>100%)MAT: direct
Nigeria MAT: preg/deliver/2mo MAT: sisterhood MAT: direct (but data quality prblm)
BangladeshNN, 1-59 mo: detailed format for each group Child: 5 years
NN, 1-11 mo, 12-59 mo: detailed algorithms, w/hierarchical assignment of cause(s) (sum=100%)
Cambodia MAT: preg/deliver/2moChild: 3 yearsMAT: sisterhood
NN, 1-59 mo: mother’s opinion & algorithm (separately); MAT: direct
HondurasNN: COD, 31 Sxs1-59 mo: COD, 28 Sxs Child: 5 years NN, 1-11 mo, 12-59 mo: ?
Nepal
Stillbirth, NN, 1-59 mo: detailed format for each groupMAT: preg/deliver/2mo MAT: sisterhood
Stillbirth, NN, 1-11 mo, 12-59 mo: detailed algorithms, w/hierarchical assignment of cause(s) + MD review of undetermined cases (sum=100%)MAT: direct
Pakistan
Stillbirth, NN, 1-59 mo: detailed module for eachMAT: detailed VA
Child: ?MAT: ?
Child: ?MAT: MD review to determine if maternal, direct/indirect, cause & CS
Country ModulesReference period Analysis
AngolaNN, 1-59 mo: 3-page format for each group Child: ? Child: ?
UgandaNN, 1-59 mo: 6-8-page format for each group Child: ? Child: ?
Verbal autopsy in DHS surveys
Survey design issues (‘Deaths of children born in Survey design issues (‘Deaths of children born in the last 3-5 years’)the last 3-5 years’) 1-year maximum recall recommended for child deaths Variable recall depending on age (shorter for older children) Age and cause distributions distorted
Disproportionately captures deaths of younger children• This also distorts the all-ages cause distribution
Nepal design does not distort:• All child deaths in past 5 years
Verbal autopsy in DHS surveys
Most use a sparse questionnaireMost use a sparse questionnaire No adult (non-maternal) deaths Based on validation studies, but perhaps too few items with
insufficient detail 21-60% (high end for NN deaths) of cases with
undetermined cause of death Bangladesh & Nepal VAs longer, based on standard formats
Require re-visit to administer 1.1-3.4% (Bangladesh) and 4.5-11.4% (Nepal) of cases
with undetermined cause of death by algorithm Consider developing a standard DHS VA questionnaire based
on current best practices and what’s practical
Verbal autopsy in DHS surveys
Unusual methods for coding VA diagnosesUnusual methods for coding VA diagnoses Usual methods: physician readers or one algorithm/COD Most DHSs examine multiple algorithms for each cause Most DHSs combine maternal opinion with algorithms
Unclear decision tree (when to use which method?) Diagnoses add to >100%, but usually unclear as to which
children have >1 diagnosis Consider developing a standard DHS coding method based
on current best practices (change may be on the horizon)
VA diagnosis misclassification
ALRI (sensitivity/specificity)ALRI (sensitivity/specificity) Cough >3 days and difficult breathing >3 days
Bangladesh: 64%/84% Uganda: 51%/68%
Malaria (sensitivity/specificity)Malaria (sensitivity/specificity) Fever and convulsions or loss of consciousness
Namibia: 45%/87% Uganda: 44%/77%
Measles (sensitivity/specificity)Measles (sensitivity/specificity) Age >120 days, rash and fever >3 days, rash on face (Nam.)
or rash anywhere except extremities (Phil.) Namibia: 83%/86% Uganda: 98%/93%
VA diagnosis misclassification
Methods of dealing with misclassificationMethods of dealing with misclassification Do nothing (usual method)
False positives and false negatives may counter-balance each other to produce an accurate cause-specific mortality estimate, but this is uncertain
Despite uncertainties, there is some evidence that VA can usefully measure changes in cause-specific mortality
Improve VA performance Accommodate for misclassification Adjust for misclassification Go around misclassification
VA diagnosis misclassification – Improve VA performance
Attempts to improve WHO standard neonatal VAAttempts to improve WHO standard neonatal VA Addition of stillbirth module Additional details on pregnancy and L&D complications New signs of NN illnesses strive for increased specificity
Breathed ‘immediately’ after birth Breathed ‘immediately’ after birth (was ‘able to breathe’)(was ‘able to breathe’) Sucked normally ‘during the first day’ Sucked normally ‘during the first day’ (was ‘after birth’)(was ‘after birth’)
Attempt to improve coding of VA diagnosesAttempt to improve coding of VA diagnoses Compare physician readers to algorithms
Improvements may be elusiveImprovements may be elusive Best sensitivity and specificity depend on disease prevalence Disease mix can affect specificity (and perhaps sensitivity)
VA diagnosis misclassification – Accommodate for misclassification
1 0 % M o r t a l i t y F r a c t i o n
S n/ Sp 60% / 7 0% 60 % / 90% 80 % / 90%D i s e a s e D i s e a s e D i s e a s e
VA dx yes no yes no yes no + 18 81 33% 18 27 15% 24 27 17%
- 12 189 12 243 6 243
30 270 300 30 270 300 30 270 300
D i s e a s e D i s e a s e D i s e a s eVA dx yes no yes no yes no + 18 81 33% 18 27 15% 24 27 17%
- 12 189 12 243 6 243
30 270 300 30 270 300 30 270 300
VA diagnosis misclassification – Accommodate for misclassification
40% Mortality Fraction
Sn/Sp 60%/70% 60%/90% 80%/90%
Disease Disease DiseaseVA dx yes no yes no yes no + 72 54 42% 72 18 30% 96 18 38%
- 48 126 48 162 24 162 120 180 300 120 180 300 120 180 300
Disease Disease DiseaseVA dx yes no yes no yes no + 72 54 42% 72 18 30% 96 18 38%
- 48 126 48 162 24 162 120 180 300 120 180 300 120 180 300
VA diagnosis misclassification – Accommodate for misclassification
Sensitivity/Specificity that can Estimate Cause-Specific
Mortality Fraction Within + 20% of the True Level
5% MF
70-100%/100%
10% MF
80-100%/100%
50-70%/95%
20% MF
80-100%/100% 60-100%/95% 50-80%/90% 50-60%/85%
30% MF
80-100%/100% 70-100%/95% 60-95%/90% 50-85%/85% 50-70%/80% 50-60%/75% 50%/70%
40% MF
80-100%/100% 80-100%/95%
70-100%/90% 60-95%/85% 50-90%/80% 50-80%/75% 50-70%/70% 50-60%/60%
VA diagnosis misclassification – Accommodate for misclassification
ObjectivesObjectives Determine how different (cultural and disease) settings affect VA
performance Identify algorithms with consistent and appropriate performance in similar
settings
Method: Conduct validation studies with standardized Method: Conduct validation studies with standardized methods in multiple settingsmethods in multiple settings Much of the apparent variability in VA performance may be due to
inconsistent study methods Determine the effects of site characteristics on performance
Different cultural settings Different disease mixes (e.g., with and without malaria) Malaria sites with different transmission intensities
VA diagnosis misclassification – Adjust for misclassification
‘‘Back-calculate’ to adjust for misclassificationBack-calculate’ to adjust for misclassification
Uses sensitivity/specificity estimates of VA algorithms from hospital-based validation studies
Very sensitive to inaccurate estimates caused by: Hospital-based study biases:
• Differences in hospital/community disease mix• Medical exposure• Cultural, SES, etc. differences
Differences between validation study and survey sites Basis of the problem: composite nature of specificity vs.
“yes/no” classification
CSMF = (VA + Sp – 1) / (Sn + Sp – 1)
VA diagnosis misclassification – Back-calculate to adjust for misclassification
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Uganda Bangladesh Nicaragua
Back-calcVA estimateTrue MF
Diarrhea-Specific True Fraction and Estimated Levels Determined by Verbal Autopsy and Back-Calculation
(using the average sn/sp from the other two sites)
VA diagnosis misclassification – Go around misclassification
Calculate disease probability from symptom profilesCalculate disease probability from symptom profilesP(S) = P(S|D) P(D) 2K x 1 2K x J J x 1
P(S|D)
Symptom profile COD_1 COD_2 COD_3 P(S)
000 0.09 0.08 0.04 0.04
001 0.37 0.27 0.11 0.32
010 0.14 0.12 0.04 0.11
011 0.00 0.00 0.00 0.00
100 0.12 0.29 0.38 0.27
101 0.00 0.18 0.15 0.09
110 0.10 0.06 0.00 0.07
111 0.18 0.00 0.28 0.10
All profiles 1.00 1.00 1.00 1.00
P(D) ? ? ?
VA diagnosis misclassification – Go around misclassification
Calculate disease probability from symptom profilesCalculate disease probability from symptom profiles
Does not require VA algorithms, sensitivity/specificity estimates or physician readers
Estimates mortality fractions of all CODs at once [P(D)] Eliminates biases caused by dichotomizing COD
Uses P(S|D) estimates from hospital-based study Less sensitive to inaccurate estimates than VA algorithms:
‘Symptom’ profiles can be manipulated at will to find differences in P(S|D) between diseases
Does not require big differences in P(S|D)s Allows multiple P(S|D)s for each disease (vs. one “yes/no”
algorithm) Still liable to bias due to inaccurate P(S|D) estimates
P(S) = P(S|D) P(D) 2K x 1 2K x J J x 1
VA diagnosis misclassification – Go around misclassification
Validation in Tanzania for adults (left graph), children (middle), and infants (right). In each graph, a direct estimate of cause-specific mortality is plotted horizontally by our verbal autopsy estimate plotted vertically (G. King, Y. Lu, 2006; data: Setel et al. 2006)
Summary and Conclusions –
DHS often uses sub-optimal VA methods DHS often uses sub-optimal VA methods Sparse modules, no adult (non-maternal) module, Sparse modules, no adult (non-maternal) module,
problematic analytic methodproblematic analytic method Convene a study group to improve modules and analytic Convene a study group to improve modules and analytic
methods based on current knowledge and practicesmethods based on current knowledge and practices
Ongoing research holds promise for improvementsOngoing research holds promise for improvements Identify algorithms with increased sensitivity/specificity Identify algorithms with increased sensitivity/specificity Gain better understanding of how cultural and diseases Gain better understanding of how cultural and diseases
settings affect VA performancesettings affect VA performance New (experimental) analytic method decreases bias in VA New (experimental) analytic method decreases bias in VA
estimates due to disease misclassificationestimates due to disease misclassification
THANK YOU
Top Related