A Framework for Understanding Statistical Performance

Paul Askew

CONFERENCE2-5 SEPTEMBER 2013

NEWCASTLE

1. Introduction

2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers

3. Framework – the “How”

Macro level

Analytical level

Outline

1. Scope….A framework for

Managing Statistics about performance

(rather than performance of statistical techniques)

2. Operational Origins

• More about practical drivers and process

• Utility….target setting, performance improvement

3. Distilling application and development across sectors….

• Criminal justice, regulation, education, health

• It really matters….safety, housing, education….

1. Introduction

1. Introduction

Operational Delivery

Methodological Leadership

1. Scope….A framework for

Managing Statistics about performance

(rather than performance of statistical techniques)

2. Operational Origins

• More about practical drivers and process

• Utility….target setting, performance improvement

3. Distilling application and development across sectors….

• Criminal justice, regulation, education, health

• It really matters….safety, housing, education….

1. Introduction

1. Introduction

2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers

3. Framework – the “How”

Macro level

Analytical level

Outline

Data

Eviden

ce

Decisi

ons

1. It actually matters to people – safety, home, education

2. Performance Regime – broad scope, high profile, deep drill down

3. “Multi-multi” dimensional – both of measures and assessments

4. Statistics meaning – datum, summary, technique

5. Targets - legal, audited, collaborative!

6. Performance Pantomime

7. Less about techniques, more about process

8. Operational Delivery – police, health, regulation…

2. Why - Operational Drivers

“Burglary is down compared to last month”

“Yes but it’s up compared the same month last year”

“Yes but it’s down overall for the financial year to date”

“Yes but its’ up for the calendar year so far”

“Yes but we’re still less better than our neighbours”

“Yes but they are reducing faster than we are this year”

“Yes but we’re still under (over) target”.

etc………….

1. It actually matters to people – safety, home, education

2. Performance Regime – broad scope, high profile, deep drill down

3. “Multi-multi” dimensional – both of measures and assessments

4. Statistics meaning – datum, summary, technique,

5. Targets - legal, audited, collaborative!

6. Performance Pantomime

7. Less about techniques, more about process

8. Operational Delivery – police, health, regulation…

2. Why - Operational Drivers

Smoothed Data or Real Data

Highs and lowsHigh and low

DecreasingIncreasing Increasing convergence

Decreasing convergence

Three month step Six month stepTwo month step

Smoothed Data – 12 month rolling average

Smoothed Data

Example Real Data

Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).

This smoothed data is derived from any of these underlying

raw data examples.

1. Data, Evidence, Decisions… Impact, Value.

2. Big & Open & Now data

3. Tactical vs. Strategic focus

4. Key Strategies…Communication emphasis - ONS, RSS…

5. Underlying Numeracy and statistical literacy

6. Policy Perception Gap

7. Data Science – Shakespeare review, Open Data, UKSA…

8. Austerity World - Effective (right thing) & Efficient (right way)

2. Why - Current and Strategic Drivers

Scope - Detail - Volume

Data.gov…10K

1. Data, Evidence, Decisions… Impact, Value.

2. Big & Open & Now data

3. Tactical vs. Strategic focus

4. Key Strategies…Communication emphasis - ONS, RSS…

5. Underlying numeracy and statistical literacy

6. Policy Perception Gap

7. Data Science – Shakespeare review, Open Data, UKSA…

8. Austerity World - Effective (right thing) & Efficient (right way)

2. Why - Current and Strategic Drivers

Words

Numbers

Data

Eviden

ce

Decisi

ons

1. Data, Evidence, Decisions… Impact, Value.

2. Big & Open & Now data

3. Tactical vs. Strategic focus

4. Key Strategies…Communication emphasis - ONS, RSS…

5. Underlying Numeracy and statistical literacy

6. Policy Perception Gap

7. Data Science – Shakespeare review, Open Data, UKSA…

8. Austerity World - Effective (right thing) & Efficient (right way)

2. Why - Current and Strategic Drivers

Skills for Life Survey 2011 (England)Department for Business Innovation and Skills

• Literacy Improving while Numeracy declining

Numeracy • 26% to 22% (7.5m

adults) with GCSE+

• 17m adults at primary school level

The numeracy challenge is big and getting bigger…%

Adu

lts a

t GC

SE

+ L

evel

s

A Framework for Understanding Statistical Performance

Paul Askew

1. Data, Evidence, Decisions… Impact, Value.

2. Big & Open & Now data

3. Tactical vs. Strategic focus

4. Key Strategies…Communication emphasis - ONS, RSS…

5. Underlying Numeracy and statistical literacy

6. Policy Perception Gap

7. Data Science – Shakespeare review, Open Data, UKSA…

8. Austerity World - Effective (right thing) & Efficient (right way)

2. Why - Current and Strategic Drivers

1. Introduction

2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers

3. Framework – the “How”

Macro level

Analytical level

Outline

3. How - Macro

DATA - inputs -

ANALYSIS- process -

PRODUCTS- outputs -

INSIGHT- outcomes -

1. Purpose

2. Require-

ments

4. Design

3. Constr-

aints

5. Defiine

6. Specify

8. Record

7. Collect

9. Entering

10.Process

12. Storage

11. Validate

2.Tools

1. Data

3.Skills

4.Capacity

5.Question

6.Inclinat-

ion

Plan

Implement

Manage

3.Target

2. Bench-mark

1. Trend

ANALYSISComp-

aritorsTime

Periods

DATA

Time Periods

Keys Message

INSI

GH

T

PRODUCTS

Cover the

angles

Lift Pitch EvidenceSummary

Comms

Stake-holders

Synthesis

- inputs -

- process --

- outputs -

- out

com

es - Analysis Strategy

Words

Graphics

Numbers

1. Purpose

2. Require-

ments

4. Design

3. Constr-

aints

5. Defiine

6. Specify

8. Record

7. Collect

9. Entering

10.Process

12. Storage

11. Validate

2.Tools

1. Data

3.Skills

4.Capacity

5.Question

6.Inclinat-

ion

Plan

Implement

Manage

3.Target

2. Bench-mark

1. Trend

ANALYSISComp-

aritorsTime

Periods

DATA

Time Periods

Keys Message

INSI

GH

T

PRODUCTS

Cover the

angles

Lift Pitch EvidenceSummary

Comms

Stake-holders

Synthesis

- inputs -

- process --

- outputs -

- out

com

es - Analysis Strategy

Words

Graphics

Numbers

OPE

NOPEN

OPEN

OPEN

II = (d)D x (t)T x s(S) x (c)C x (q)Q x (i)IThe factors:

D Data: Right data? Enough of it? Good enough?T Tools: Have any? Right ones?S Skills: Have any? Right ones?C Capacity: How much? Realistic?Q Question: Specific question to answer or issues to address I Inclination: Desire and drive to want to address the issues

d,t,s,c,q,i Relative weights

Askew Analytical Insight Index

1. Purpose

2. Require-

ments

4. Design

3. Constr-

aints

5. Defiine

6. Specify

8. Record

7. Collect

9. Entering

10.Process

12. Storage

11. Validate

2.Tools

1. Data

3.Skills

4.Capacity

5.Question

6.Inclinat-

ion

Plan

Implement

Manage

3.Target

2. Bench-mark

1. Trend

ANALYSISComp-

aritorsTime

Periods

DATA

Time Periods

Keys Message

INSI

GH

T

PRODUCTS

Cover the

angles

Lift Pitch EvidenceSummary

Comms

Stake-holders

Synthesis

- inputs -

- process --

- outputs -

- out

com

es - Analysis Strategy

Words

Graphics

Numbers

OPE

NOPEN

OPEN

OPEN

3.Target

2. Bench-mark

1. Trend

Comp-aritors

Time Periods

Time Periods

3. How – Analytical Level

0. Snapshot

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

0. Snapshot - we have a number which is important to us

1. Trend - what’s happening to our measure over time

2. Benchmark – how this compares to others

2a. Trend for the comparison to others

3. Target – the trajectory for our measure

3a. – the comparison trajectory 3.2. 1. 0.

3. How – Analytical Level

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

0. Snapshot – we have a number which is important to us

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

1. Trend – what’s happening over time

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

2. Benchmark – how this measures compares to others

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

2a. Trend for the comparison to others

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

3. Target - the trajectory for our measure

t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40

20

40

60

80

100

120

140

160

Value

Time

3a. Target - Trajectory for the comparison to others

1. Introduction

2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers

3. Framework – the “How”

Macro

Analytical

Outline

A Framework for Understanding Statistical Performance

Paul Askew

CONFERENCE2-5 SEPTEMBER 2013

NEWCASTLE

Thank You