Funnel Plots: Visualizing Small Area Estimates

eSUG May 2010

Doug Dover Surveillance and Assessment Branch

Community and Population Health Division Alberta Health and Wellness

Funnel Plots: Visualizing Small Area Estimates

eSUG May 2010

Doug Dover Surveillance and Assessment Branch

Community and Population Health Division Alberta Health and Wellness

Funnel Plots

The need to visualize small area rates. The need to visualize small area rates.

Small Area Rates

Issues:

•large variability in small populations

•ranking and rates •ranking and rates

•identification of anomalies

Small Area Rates

large variability in small populations

ranking and rates ranking and rates

identification of anomalies

Funnel Plots ­

Developed as alternatives to caterpillar or league plots.

­ History

Developed as alternatives to caterpillar or league plots.

Funnel Plots

Funnel plots show point estimates for small areas or domains against a background “expected” confidence interval. interval.

Funnel plots show point estimates for small areas or domains against a background “expected” confidence

Funnel Plots Expected Standard Errors

The “expected” standard error in the binomial case is given by

ˆ ˆ (1 ) / p p n − where is the overall rate

and n varies freely.

ˆ ˆ (1 ) / p p n − ˆ p

Expected Standard Errors

The “expected” standard error in the binomial case is given by

ˆ ˆ (1 ) / p p n − where is the overall rate

varies freely.

ˆ ˆ (1 ) / p p n −

Funnel Plots Expected Standard Errors Expected Standard Errors

Funnel Plots Expected Standard Errors

small n, large CI

Expected Standard Errors

, large CI

Funnel Plots Expected Standard Errors

small n, large CI

Expected Standard Errors

, large CI

large n, small CI

Funnel Plots Expected Standard Errors

small n, large CI

Expected Standard Errors

, large CI Limits centered at a fixed rate

large n, small CI

Funnel Plots Nice Example

Funnel Plots Nice Example

Same disease,

a few years later…

Funnel Plots in SAS

The funnel plots presented so far were created in SAS, but why not just use Excel for such simple graphs?

The funnel plots presented so far were created in SAS, but why not just use Excel for such simple graphs?

Funnel Plots in SAS

Because SAS is cool! Because SAS is cool!

Funnel Plots Which point?? Which point??

Funnel Plots Mouse over!

Funnel Plots SAS mouse over

*make funnel plot*;

goptions reset=symbol device=activex; ods html body="&htmlfile" path="&htmlpath" parameters=("ZoomControlEnabled"="false" "DataTipStyle"="stick_fixed");

Proc GPlot Data=_Funnel_; Plot &rate*&pop lfunnelLimit*&pop ufunnelLimit*&pop

/ overlay / overlay hminor = 0 vminor = 0 vaxis = axis1 vref=&ABvalue cvref=purple lvref=3 frame anno=anno

html=popup ;

format &pop comma12.; run; quit; ods html close;

SAS mouse over

device=activex; ods html body="&htmlfile" path="&htmlpath" parameters=("ZoomControlEnabled"="false" "DataTipStyle"="stick_fixed");

Plot &rate*&pop lfunnelLimit*&pop ufunnelLimit*&pop

vref=&ABvalue cvref=purple lvref=3

Funnel Plots SAS mouse over

*make popup text*; Data _Funnel_; Set _Funnel_; popup='ALT="'||compress(&PopupText)||'"';

run;

SAS mouse over

popup='ALT="'||compress(&PopupText)||'"';

Funnel Plots Are there commonalities? Are there commonalities?

Funnel Plots Text!

Funnel Plots Colour!

Funnel Plots SAS annotate

*make funnel plot*; goptions reset=symbol device=activex; ods html body="&htmlfile" path="&htmlpath" parameters=("ZoomControlEnabled"="false" "DataTipStyle"="stick_fixed");

Proc GPlot Data=_Funnel_; Plot &rate*&pop lfunnelLimit*&pop ufunnelLimit*&pop

/ overlay / overlay hminor = 0 vminor = 0 vaxis = axis1 vref=&ABvalue cvref=purple lvref=3 frame

anno=annods html=popup ;

format &pop comma12.; run; quit; ods html close;

goptions reset=symbol device=activex; ods html body="&htmlfile" path="&htmlpath" parameters=("ZoomControlEnabled"="false" "DataTipStyle"="stick_fixed");

Plot &rate*&pop lfunnelLimit*&pop ufunnelLimit*&pop

vref=&ABvalue cvref=purple lvref=3

Funnel Plots SAS annotate

*annotations*; Data anno_label; Set anno_label; format function $8. text $12. style $12.; retain xsys '2' ysys '2'; retain when 'A';

*move to point*; function='move'; x=&pop; y=&rate; output;

*decide where to draw line to*; offset_xpct = 0.03*(&maxpop ‐ offset_ypct = 0.02*(&maxrate ‐ *min() and max() keep x,y coords within plot*; if _OOL_=+1 then do; x=min(x+offset_xpct,&maxpop);

y=min(y+offset_ypct,&maxrate); end;

if _OOL_= 0 then do; x=min(x+offset_xpct,&maxpop); end;

if _OOL_=‐1 then do; x=min(x+offset_xpct,&maxpop); y=max(y‐offset_ypct,&minrate);

end; *draw line*; function='draw'; line=1; size=1; output;

format function $8. text $12. style $12.;

function='move'; x=&pop; y=&rate;

*decide where to draw line to*; ‐ &minpop ); ‐ &minrate);

*min() and max() keep x,y coords within plot*; if _OOL_=+1 then do; x=min(x+offset_xpct,&maxpop);

y=min(y+offset_ypct,&maxrate);

if _OOL_= 0 then do; x=min(x+offset_xpct,&maxpop);

1 then do; x=min(x+offset_xpct,&maxpop); offset_ypct,&minrate);

Funnel Plots SAS annotate

*add text*; function='label'; if _OOL_=+1 then position='3'; if _OOL_=0 then position='6'; if _OOL_=‐1 then position='9'; text=&AnnoVar; size=2; hsys='3'; style="Calibri"; %if &explore ne %then %do; *add explore category to label*; text=trim(compress(text))||" "||trim(compress(&explore)); %end; %end; output;

drop offset_xpct offset_ypct; run;

text=&AnnoVar; size=2; hsys='3'; style="Calibri"; %if &explore ne %then %do; *add explore category to label*; text=trim(compress(text))||" "||trim(compress(&explore));

Funnel Plots SAS colour

*draw symbol*; function='symbol'; hsys='3'; size=2 Select(&explore); When(0) do; color="blue"; text="star"; end; When(1) do; color="red"; text="dot"; end; When(2) do; color="yellow"; text="plus"; end; When(3) do; color="green"; text="diamond"; end; When(4) do; color="orange"; text="triangle"; end; When(4) do; color="orange"; text="triangle"; end; When(5) do; color="brown"; text="#"; end; When(6) do; color="blue"; text="hash"; end; When(7) do; color="red"; text=")"; end; When(8) do; color="yellow"; text=":"; end; When(9) do; color="green"; text="'"; end; Otherwise;

end; output;

function='symbol'; hsys='3'; size=2; x=&pop; y=&rate;

) do; color="blue"; text="star"; end; ) do; color="red"; text="dot"; end; ) do; color="yellow"; text="plus"; end; ) do; color="green"; text="diamond"; end; ) do; color="orange"; text="triangle"; end; ) do; color="orange"; text="triangle"; end; ) do; color="brown"; text="#"; end; ) do; color="blue"; text="hash"; end; ) do; color="red"; text=")"; end; ) do; color="yellow"; text=":"; end; ) do; color="green"; text="'"; end;

Funnel Plots Automation! %macro funnel

( Data=, Out=FunnelOut,

Rate=, Pop=,

ABvar=, ABrate=, RatePer=100000, RatePer=100000,

CIpct=95, CIpct2=, CItrans=NORMAL, OD=1, minpop=, maxpop=, minrate=, maxrate=,

htmlfile=FunnelPlot.html, htmlpath=,

AnnoWhen=OUTSIDE, AnnoVar=,

Explore=, League=NO );

( Data=, Out=FunnelOut,

Rate=, Pop=,

ABvar=, ABrate=, RatePer=100000, RatePer=100000,

CIpct=95, CIpct2=, CItrans=NORMAL, OD=1, minpop=, maxpop=, minrate=, maxrate=,

htmlfile=FunnelPlot.html, htmlpath=,

AnnoWhen=OUTSIDE, AnnoVar=,

Explore=, League=NO );

Funnel Plots in SAS

Data + Creation + Outputs + Automation Data + Creation + Outputs + Automation

Funnel Plots Analytics

Anomaly detection Anomaly detection

Over dispersion

Funnel Plots Modeling

adjusted rate p

ˆ i

where p is the overall rate, O is the observ and E s the expected count from a Poisson i using small area data and the covariates

adjusted rate p ˆ O i adjusted rate p i E = ⋅

where p is the overall rate, O is the observed count, i and E s the expected count from a Poisson regression using small area data and the covariates of interest.

ˆ i adjusted rate p i E i = ⋅

Funnel Plots Analytics

***adjustment time***;

Proc Genmod Data=rate_data;

Model count = risk_factor_1 risk_factor_2 Model count = risk_factor_1 risk_factor_2 / dist=Poisson offset=logpop;

Output out=adj predicted=expected; run;

**and then merge back expected counts**;

***adjustment time***;

Proc Genmod Data=rate_data;

Model count = risk_factor_1 risk_factor_2 Model count = risk_factor_1 risk_factor_2 / dist=Poisson offset=logpop;

Output out=adj predicted=expected;

**and then merge back expected counts**;

Funnel Plots Modeling

unadjusted adjusted

Funnel Plots Modeling

unadjusted adjusted

Funnel Plots

•Utility in analyzing small areas

•Regression diagnostics

•SAS macro for Automation

•Proc GPlot with Annotate and Pop html output

Utility in analyzing small areas

Regression diagnostics

SAS macro for Automation

Proc GPlot with Annotate and Pop­up in

Work on Funnel Plots is part of an on­going collaboration between

Doug Dover, AHW Doug Dover, AHW

Don Schopflocher, UofA

Work on Funnel Plots is part of an going collaboration between

Doug Dover, AHW Doug Dover, AHW and

Don Schopflocher, UofA

Thank you Thank you

References:

Funnel Plot Theory “Funnel plots for comparing institutional performance” Spiegelhalter 2005 ­everything to do with funnel plots and more! ­quite accessible ­addresses many issues with use ­excellent, readable paper

“Bristol, Shipman, and Clinical Governance: Shewhart’s Forgotten Lesson” Mohammed, Cheng, Rouse, Marshall 2001 Mohammed, Cheng, Rouse, Marshall 2001 ­commentary on use of control plots ­related to funnel plots

“Defining poor and optimum performance in an IVF programme” Castilla et al 2008 ­illustrates league plots, control plots and funnel plots

“Funnel plots for comparing institutional performance”

everything to do with funnel plots and more!

addresses many issues with use

“Bristol, Shipman, and Clinical Governance: Shewhart’s Forgotten Lesson” Mohammed, Cheng, Rouse, Marshall 2001 Mohammed, Cheng, Rouse, Marshall 2001 commentary on use of control plots

“Defining poor and optimum performance in an IVF programme”

illustrates league plots, control plots and funnel plots

References:

Examples of Funnel Plots “Regional differences in population prefectures in Japan: Application of relative survival models with funnel plots” Ito et al 2009 ­excellent example of what can go wrong using funnel plots ­has large and small regions similar to Alberta ­use of funnel plots to show regional survival rates

“Using Funnel Plots to Explore Variation in Cancer Mortality across Primary Care Trusts in South Primary Care Trusts in South Davies et al 2008 ­cancer mortality rates in 76 PCTs

“An alternative approach to quantifying and addressing inequity in healthcare provision” Battersby, Flowers, Harvey 2004 ­they used control plots

Examples of Funnel Plots “Regional differences in population­based cancer survival between six prefectures in Japan: Application of relative survival models with

excellent example of what can go wrong using funnel plots has large and small regions similar to Alberta use of funnel plots to show regional survival rates

“Using Funnel Plots to Explore Variation in Cancer Mortality across Primary Care Trusts in South­East England” Primary Care Trusts in South­East England”

cancer mortality rates in 76 PCTs

“An alternative approach to quantifying and addressing inequity in

Battersby, Flowers, Harvey 2004

Author: Doug Dover Surveillance and Assessment Branch

Visualizing Small Area Estimates

Surveillance and Assessment Branch Community and Population Health Division Alberta Health & Wellness

Version History: Based substantially on the work of Doug Dover and Don Schopflocher presented at AH&W on Tuesday, April 6, 2010, in the Statistics in Surveillance Series, under the same title. “Funnel Plots as a Vehicle for Policy Relevant Analysis: Gaining Focus” Presented at ICHPS, Washington, 2009.

Funnel Plots: Visualizing Small Area Estimates

Community and Population Health Division

Based substantially on the work of Doug Dover and Don Schopflocher presented at AH&W on Tuesday, April 6, 2010, in the Statistics in Surveillance Series, under the

“Funnel Plots as a Vehicle for Policy Relevant Analysis: Gaining Focus” Presented at ICHPS, Washington, 2009.