Post on 22-Dec-2015
“To Ignore or Not to Ignore?”
Follow-up to Statistically Significant Signals"
Biosurveillance Information Exchange Working Group
Reflections from San Diego County
Jeffrey Johnson, MPHSan Diego County Health & Human Services Agency
2/23/06
SAN DIEGO COUNTY
• Nearly 3 million population
• International border
• Large military presence
• Biotechnology Hub
• 21 Emergency Departments
Early Event Detection in San Diego
• Evolving effort since pre - 9/11
• Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales
• Systems: Local SAS/Minitab system, ESSENCE, and BioSense
• Statistical Methods: Descriptive, time series, CUSUM, EWMA, process
control methods (P&U Charts)
• Multiple syndromes • Visualization and alerting• Incident Characterization• Follow-up to signals County of San Diego
Health & Human Services Agency
If We Ignore A Signal……
• We take no action or follow-up
• Save staff resources
• Avoid bothering hospital staff yet again
• Another data source may signal
• “The Feds may pick it up”
• Might lose an earlier start to a response
• We might be dead wrong to ignore
If We Do Not Ignore a Signal……
• Will it be another “false alarm”
• May detect an event earlier
• Earlier response
• Continued interaction with the medical community
• Gain experience with follow-up
• Increased situational awareness
Characterization of Detections
• Detection Method • Syndrome group• % Admitted• Deaths?• Geographic cluster?• Prior day’s level?• Recent level?• Age groups?• Severe syndrome?• Detections in other data sources?• Other epidemiological intelligence?• Other diagnostic information
Follow-up?
Action or
No Actionor
Watch
Detection Follow-up with Medical Community
What is the final diagnosis of Patients A, B, C?
Is there a common pattern among admitted patients?
Did any have lab test results that might suggest a larger event?
Among patients with a common zip code, was there a shared living setting or common exposure?
Can we send someone out to review medical charts?
What is your facility’s assessment of the situation?
County of San DiegoHealth & Human Services Agency
County of San DiegoHealth & Human Services Agency
Routine Surveillance Activities
Rule out system error
YES
NO
Preliminary evaluation
Describe initial results
True Positive
Aberrationdetected
Potential false positive
YES NO“False Positive”
Inform key divisional staff
Intensivemonitoring & surveillance
Evaluate otherdata sources
Cluster check
VERIFY
NOTIFYInform key departmental staff
IDENTIFY
Ignore?
Ignore?
Ignore?
Ignore?
0
10
20
30
40
50
60
70
80
90
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal 7 Day Moving Average
0
5
10
15
20
25
30
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal 7 Day Moving Average
0
5
10
15
20
25
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal/Genitourinary 7 Day Moving Average
GI Syndrome Over Time (10/31/04 – 8/24/05)
ED
911
Par
amed
ic
Ru
ns
The Significant Aspects of Syndromic Surveillance
• Statistical Significance
• Public Health Significance
• Significant Event
• Significant Public Awareness
• Significant Biological Agent Detection
HAZMAT FLAG – 12/04/2004
County of San DiegoHealth & Human Services Agency
Statistical significance vs. public health significance
County of San DiegoHealth & Human Services Agency
Statistical significance vs. public health significance
County of San DiegoHealth & Human Services Agency
Syndromic Surveillance for Natural Disasters
San Diego Wild Fires, 2003
Significant event with statistically significance outcomes
San Diego County
Syndromic surveillance for natural disasters
Significant event with statistically significance outcomes
San Diego County: Normalized Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
-3
-2
-1
0
1
2
3
4
5
12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04
No
rmal
ized
Co
un
t
ED Visits Ambulance Runs
San Diego County: Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
0
10
20
30
40
50
60
12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04
Co
un
t
ED Visits Ambulance Runs
“The Clinton Effect”
September 4, 2004
While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, 2004.
Significant Public Awareness
7/7/05 London Bombings
San Diego CountyParamedic Transports
for “Chest Pain”
Significant Public Awareness
Application of Syndromic Surveillance Agent:
Syndrome categoriesSpecific word search in CC or DX fields
Sensor site:Zip codes, population (schools)
Date:Temporal based surveillanceNew pre-detection baselines
Biowatch
BioWatch Detection• Tells us agent, sensor site and date• Plume plot may help us narrow surveillance on a geographic area
Significant BT Agent Detection
Anatomy of a Detection(a case example)
Daily Email Report
Attached Table
Feb 5, 2006 911 Call Data
911 Call Center - GI Syndrome Signal
Line listing for review
Non-specific call
complaints
21 Signals since 07/01/03
Various statistical signals
The count for the signals include a consistent range
911 Call Center - GI Syndrome Signal
What did we do?
• Magnitude of cases
• Which method(s) signaled?
• Check the other call centers
• Check the other data sources
(ED data, EMS transports)
• Review the line listing
• Our conclusion…..
…... 14 vs mean of 7.8
…... CUSUM (2), P-Chart, U-Chart
…… No signals
…… No Signals
……. No apparent pattern
>>>>> • Super Bowl Sunday • Fewer trauma calls • Smaller denominator (P-Chart)• Traditional increase in GI on
this day• Watch next day’s results
Case Example #2
Hospital 9 ED DataRespiratory Syndrome
Hospital 9 - Daily Results Table
0
5
10
15
20
25
30
35
40
1/1/2004
2/1/2004
3/1/2004
4/1/2004
5/1/2004
6/1/2004
7/1/2004
8/1/2004
9/1/2004
10/1/ 2004
11/1/2004
12/1/ 2004
1/1/2005
2/1/2005
3/1/2005
4/1/2005
5/1/2005
6/1/2005
7/1/2005
8/1/2005
9/1/2005
10/1/ 2005
11/1/2005
12/1/ 2005
1/1/2006
2/1/2006
Count Signal
Hospital 9 Respiratory Syndrome
01/01/04 - 02/03/06
Many signals….
So what’s the context?
Do we ever ignore the signals?
Hospital 9 Respiratory Syndrome
• 24 signals over a 37 day period• Count range: 11 – 34• Over time an increasing mean
Hospital 9 Influenza-like-illness (ILI) Syndrome
• “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome”
• 16 signals over a 37 day period
Greater Syndrome Specificity……
• S$gn&ls Happen!
• Make sure you see flames before yelling “Fire”
• CUSUM 2 & 3 STD may be too sensitive
• We lose precision with non-specific syndromes
• Everyone wants to know what’s going on all the time
• Increasing focus on situational awareness
• Further evaluation and testing required
What We Have Learned
Event or
Technology Trigger
Peak of
Inflated Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Hype Cycle of Emerging “Syndromic Surveillance” Technologies
Adapted from the Gartner Hype Cycle
Too many signals?, IT Costs, poor syndrome specificity, evaluation
results
Prioritized data sets, protocols in place,
The “magic bullet”
9/11, Anthrax attacks
Dual use, situational awareness, appropriate
signals
• More work in all areas of syndromic surveillance is needed
• Knowledge requires responsibility
• The enemy is studying our efforts
• Current/future funding levels require reliability, efficiency and sustainability of systems and approaches
• The Future:Neural networks and Artificial Intelligence (AI)?
• Are we ready?
Considerations
Contact InformationContact Information
Jeffrey Johnson619-531-4945
jeffrey.johnson@sdcounty.ca.gov
Thank You