Introduction to baselines & significance for interpretation

Sheena Sullivan, MPH, PhD

2

Baselines and thresholds Baseline

The usual or average level of influenza activity that occurs during a typical year

Threshold The level of influenza activity that signals the

occurrence of a specific activity Seasonal threshold

The level of influenza activity that signals the start and end of the annual influenza season

Alert threshold A level above which influenza activity is higher than

most years.

3

Baselines and thresholds Used as a point of reference to detect

Start/end of the season Severity of season Outbreaks

Useful to inform public health actions improve clinical diagnosis stimulate diagnosis encourage early prescription of antivirals Indicate uptake and timeliness of vaccine

Used retrospectively and prospectively

Example: Victorian sentinel data

Tay et al (submitted) 4

0

10

20

30

40

50

60

2005 2006 2007 2008 2009 2010 2011

Year

Admission Proportion per 100000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

ILI Proportion per 100000

VAED GPSS MMDS

worse

outbreakstart

5

Data considerations Baselines are difficult to establish

Rely on very stable data collected over a long period of time

Multiple years of data are needed to account for fluctuations in activity from year to year Severity of virus types Changes in data collection methods, diagnosis coding,

participating clinical facilities, insurance eligibility, testing practises

Most surveillance systems cannot adjust quickly to changing baselines, so that utilization shifts may trigger false alarms – public health crises and major public events may undermine health surveillance systems at the very times they are needed most

6

Data considerations Five years of surveillance data collected

consistently is considered standard for a reliable baseline 3 years minimum Several months may be sufficient with

sophisticated modelling (e.g. Cowling 2006) Difficult in tropical areas or areas with many

different causes of ILI

7

Sources of data ILI

E.g. number of consultations at sentinel GPs among all patients seen

E.g. proportion of total outpatient visits SARI

E.g. hospital admissions Deaths

E.g. number or rate of influenza or pneumonia deaths Laboratory notifications

E.g. percentage of influenza-positive specimens among respiratory tests

Over the counter pharmaceutical sales Call volume to information /advice lines Frequency of online searches (e.g. Google flu trends,

Ginsberg et al 2009)

8

Sources of data Multiple sources of data can be combined to

develop composite indicators of baseline and thresholds E.g. start of season = week in which ILI crosses a

certain value and the percentage of specimens testing positive reaches a certain value

9

Types of data Count

May be used where denominators are not known, or cases are rare

Require very stable data An average number of cases may be used where

denominators are unknown Proportion

Better parameter to use Can correct for shifts both in numerator and

denominator activity (e.g. health care utilization and in disease activity)

Help to clarify/magnify an outbreak signal

10

Example: counts v proportions------- total number of visits------- counts of ILI

Burkom et al. 2008

11

Individual site vs. aggregated site Baselines may be calculated for individual

sites, to monitor activity in particular locations, or may use data compiled from all surveillance sites In locations with few sites, individual baselines

help control for non-reporting sites, regional variation within a country, or relative representativeness of sites of the national or regional populations

In locations with many sites, data aggregated from all sites might be the best way to set a baseline

12

Data represented The type of threshold calculated depends on

data available and usage Indicate start of season Comparison of this week’s value with the

expected value Outbreak indicator

13

Timeliness of data Depends on needs

Early-warning systems require near-real-time data

If wanting to report what happened in a season, timeliness less important

Real-time E.g. China

Weekly or fortnightly reports E.g. US

Death certificates May take months to verify

14

Methods for determining thresholdsMethod description

Examples Advantages Disadvantages

Visual Based on a visual analysis of past data, define baseline, off-seasonal baseline, threshold and seasonal threshold values

Graphically based19 Model based3

Very simple to implement and understand

Overly simplified, will not capture any trend changes over time

Averaging Usually involves calculating a median or mean of data

Methods of moments (MEM)18

Simple to implement

Can allow a past season's or week's aberrant values to influence current time prediction of baseline and threshold values

15

Methods for determining thresholdsMethod description

Examples Advantages Disadvantages

Process control Based on similar processes to those used in detecting anomalies in industrial production processes. Most methods rely on some method of setting an upper control limit. Some methods also involve looking at the rate of change in the data series.

Shewhart charts9,16 CUSUM charts4,13,14

Exponentially weighted moving average charts13,16

Best for detection of start of season and unusual patterns. Works well in situations where rates are low. May be best method to use in tropical climates. Good at detecting the start of the season when the start is slow.

Not as accurate as time series methods. May be sensitive to small changes in reporting efficiency.

Other Regression3,4,15

Time series4Account well for seasonal variations. Good for assessing severity. Useful for predicting activity

Difficult for the non-statistician

16

Types of baselines

Figure 2. ILI presentation rates at metropolitan and rural general practice sentinel sites, 1997 to 2012

The 2012 Victorian Influenza VaccineEffectiveness Audit Report: http://www.victorianflusurveillance.com.au

Static (flat) Does not change with

changes in seasonal patterns found in data

Uses data for a specified period of time (may be for a whole year, may be for only period when surveillance is conducted)

Baseline - defines the start and end of an influenza season

Average and above average - describe the intensity of a season

17

Visual inspection

Jan-94

Mar-94May-

94Jul

-94Sep-

94Nov-

94Jan

-95Mar-9

5May-

95Jul

-95Sep-

95Nov-

95Jan

-96Mar-9

6May-

96Jul

-96Sep-

96Nov-

96Jan

-97Mar-9

7May-

97Jul

-97Sep-

97Nov-

97Jan

-98Mar-9

8May-

98Jul

-98Sep-

98Nov-

98Jan

-99Mar-9

9May-

99Jul

-99Sep-

99Nov-

99Jan

-00Mar-0

0May-

00Jul

-00Sep-

00Nov-

00

0

50

100

150

200

250

300

350

400

450

500

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Hospital admission

ILI consultations

Adm

issi

ons

ILI p

er 1

00 p

atie

nts

seen

>3.5 epidemic

1.5-3.5 higher than expected seasonal activity

0.25-1.5 normal seasonal activ-ity

<0.25 baseline activity

Watts et al., 2003

18

Shewart chartsFigure 1. Victorian weekly laboratory notifications of influenza 2002-2008 with Shewhart Chart threshold of 6.5.

Simplest control chart Developed by industry for QA/QC Assumes normal distribution

Binomial (Bernoulli) & Poisson variants Mean (w) based on previous data Control limits

Upper limit = w + kw K either predetermined, usually 2 or 3,

upper limit of 95%CI or set based on recent observations

Alert declared when obervations exceed this upper limit i.e. |yt − µ| > kσ

False alarms mean and sd should be estimated

from large dataset to avoid false alarms

Steiner et al 2010

19

Example: US ILI Baseline Plots % of patient visits to healthcare providers for ILI

reported each week weighted by state population and compared with a baseline % visits = n(ILI) / N N=total patients for that week y = % visits * w w=weight for the state

Baseline = mean % of patient visits for ILI during non-influenza weeks for the previous three seasons plus 2 standard deviations (+2) Non-influenza week = two or more consecutive weeks in

which each week accounted for less than 2% of the season’s total number of specimens that tested positive for influenza

National baseline = 2.2%, but each region has its own baseline also

Does not include summer data; do not know if activity is outside the norm for the summer months

http://www.cdc.gov/flu/weekly/overview.htm#Outpatient

20

National Baseline Outbreak/Epidemic Activity

Australia Influenza Surveillance Report, Oct 2012 21

Types of baselinesFigure. Rate of deaths classified as influenza and pneumonia from the NSW Registered Death Certificates, 1 January 2007 to 21 September 2012

Cyclical (seasonal) Good for data with regular

seasonality May be inappropriate in

regions with unclear seasons (e.g. tropics)

Cyclical changes in baseline reflect seasonal pattern of disease activity Distinguishes disease related

increases with normal seasonal increases in a syndrome (e.g. pneumonia)

Different methods used: Regression models, moving

averages, time series

Unusual activity

22

Example: 122 Cities Mortality Baseline Weekly report of total death certificates & total for

which pneumonia or influenza (P&I) was listed as underlying cause of death, by age group

Percentage of deaths due to P&I are compared with seasonal baseline and epidemic threshold values calculated for each week

Seasonal baseline is calculated using a periodic regression model that incorporates a robust regression procedure applied to data from the previous five years

An increase of 1.645 standard deviations above the seasonal baseline is considered the “epidemic threshold”

23

Example: 122 Cities Mortality Baseline

http://www.cdc.gov/flu/weekly/

24

Interpreting data: which method is best? Depends on the application

Methods which use a static baseline may be better for defining the beginning/end of a season

Methods that rely on seasonality may be inappropriate for the tropics

Can formally evaluate based on: Sensitivity – true alarm rate Specificity – false alarm rate Positive predictive value – ratio of true positive

epidemic alarms over the total number of alerts Timeliness – how quickly the method signals an

outbreak (run-length)

25

Interpreting data Knowing your data is key to interpreting

Time series data may show anomalies associated with holidays, long weekends, etc. Interpretation of those anomalies is dependent on the

data analyst’s knowledge of trends not accounted for in the detection algorithm

Baselines may be influenced by changes in Data collection methods Provider participation Changes in case definitions Changes in population/health care use

26

Example: understanding and interpreting data

Ungchusak et al. 2012

27

Example: Categories of influenza season in Victoria for six surveillance datasets, 2002 - 2011

0

5

10

15

20

25

30

Year

GPSS

Alert

Above Average

Average

Alert

Above Average

Average

Alert

Above Average

Average

Alert

Baseline

Above Average

Average

Alert

Above Average

Average

Baseline

Alert

Above Average

Average

Baseline0

20

40

60

80

100

120

Year

MMDS

Alert

Above Average

Average

Baseline0%

5%

10%

15%

20%

25%

30%

35%

40%

Percentage

Year

Test Positive Influenza

Alert

Above Average

Average

Baseline

0

2

4

6

8

10

12

14

16

Proportion per 1,000

Year

GPSS Composite

Alert

Above Average

Average

Baseline 0

5

10

15

20

25

30

Proportion per 1,000

Year

MMDS Composite

Alert

Above Average

Average

Baseline0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Proportion per 100,000

Year

VAED

Alert

Above Average

Average

Baseline

Tay et al (submitted)

28

Understanding source data Lab-confirmed influenza

May not be available Mortality data

Lag time between influenza circulation and death Hospitalisation

Issues with coding Lag time between infection and rise in

presentations to hospital Weekly versus daily data

Greater variation with daily reports

Example: understanding and interpreting data

30

Summary Baselines and epidemic thresholds help to

understand the significance of increased influenza activity When to know a flu season has begun When to know if a spike in activity is a real spike An indicator that activity is unusual or outside

the norm An indicator for public health action

Graphical representation of current surveillance data compared with baseline data and previous years’ data provides a meaningful snapshot to public health practitioners, policy makers, and others regarding current activity

31

References1. Burkom, et al. Developments in the Roles, Features, and Evaluation of Alerting Algorithms for Disease Outbreak

Monitoring, Johns Hopkins APL Technical Digest, 20082. Clothier HJ, et al. A comparison of data sources for the surveillance of seasonal and pandemic influenza in

Victoria. Commun Dis Intell. 2006;30(3):345-9. 3. Cooper DL, et al. Can syndromic thresholds provide early warning of national influenza outbreaks? J Public

Health. 2009;31(1):17-25.4. Cowling BJ, et al. Methods for monitoring influenza surveillance data. Int J Epidemiol. 2006;35(5):1314-21.5. Dedman D, Watson J. The use of thresholds to describe levels of influenza activity. PHLS Microbiol Dig.

1997;14:206-8.6. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using

search engine query data. Nature. 2009 Feb 19;457(7232):1012-4.7. Goldstein E, et al. Improving the estimation of influenza-related mortality over a seasonal baseline. Epidemiology.

2012;23(6):829-38.8. Goldstein E, et al. Predicting the epidemic sizes of influenza A/H1N1, A/H3N2, and B: a statistical method. PLoS

Med. 2011;8(7):e1001051. 9. Hashimoto S, et al. Detection of epidemics in their early stage through infectious disease surveillance. Int J

Epidemiol. 2000;29(5):905-10.10. Health Protection Agency (HPA). Surveillance of influenza and other respiratory viruses in the UK: 2010-2011.

London: HPA; May 2011. Available from: http://www.hpa.org.uk/Publications/InfectiousDiseases/Influenza/1105influenzareport/

11. Hutwagner LC, Maloney EK, Bean NH, Slutsker L, Martin SM: Using laboratory-based surveillance data for prevention: an algorithm for detecting salmonella outbreaks. Emerg Infect Dis 1997, 3:395–400.

12. Kelly HA, et al. The significance of increased influenza notifications during spring and summer of 2010-11 in Australia. Influenza Other Respi Viruses. 2012.

13. Kuang J, et al. Epidemic features affecting the performance of outbreak detection algorithms. BMC Public Health. 2012;12(1):418.

14. O'Brien SJ, Christie P. Do CuSums have a role in routine communicable disease surveillance? Public Health. 1997;111(4):255-8.

15. Serfling RE. Methods for current statistical analysis of excess pneumonia-influenza deaths. Public health reports 1963; 78:494-506.

16. Steiner SH, et al. Detecting the start of an influenza outbreak using exponentially weighted moving average charts. BMC Med Inform Decis. 2010;10.

17. Ungchusak et al. Lessons Learned from Influenza A(H1N1)pdm09 Pandemic Response in Thailand. EID 2012:18, 18. Vega T, et al. Influenza surveillance in Europe: establishing epidemic thresholds by the Moving Epidemic Method.

Influenza Other Respi Viruses. 2012. & http://cran.r-project.org/web/packages/mem/mem.pdf 19. Watts CG, et al. Establishing thresholds for influenza surveillance in Victoria. Aust N Z J Public Health.

2003;27(4):409-12.20. World Health Organization. WHO Interim Global Epidemiological Surveillance Standards for Influenzaa. Geneva:

Global Influenza Programme, Surveillance and Monitoring team, World Health Organization, 2012. http://www.wpro.who.int/emerging_diseases/documents/docs/GuideforDesigningandConductingInfluenzaStudies.pdf

32

Exercise - Defining baseline curves and alert threshold Determining the baseline1. Align the transmission peaks of several

years data around the median week of peak reporting

2. Calculate an average weekly number for each week centred on the median peak week of transmission

33

34

Defining alert threshold Display the lowest and highest season,

excluding exceptional events (e.g. pandemic)

35

Defining alert threshold Calculate the standard deviation of the mean

for each week and create a curve based on those values

36

Examining data Plot the current year’s data on the curve