Savannah River Nuclear Solutions Fluor Government Group · 2010 ISM Workshop, August GA 3 Overview...

Steve PrevetteSenior Statistician

Tony UmekVice President, ESH&Q

Savannah River Nuclear SolutionsFluor Government Group

September 2010

DETECTING CHANGES IN SAFETY CULTURE USING TRENDING AND LEADING INDICATORS

Steven Prevette is a Senior Statistician for Fluor Corporation, and is assigned to Savannah River Nuclear Solutions LLC. Steve holds a M.S. degree in Operations Research, and is an American Society for Quality Certified Quality Engineer. Steve was previously a US Navy submarine officer, and then spent 16 years at the Department of Energy Hanford Site developing performance metrics and trending methodology using Dr. Shewhart’s Statistical Process Control and Dr. Deming’s System of Profound Knowledge methods. Steve has authored more than 75 articles and presentations for groups such as the American Society of Quality, American Society of Safety Engineers, past ISM workshops, Health Physics Society and EFCOG working groups on the subject of performance metrics, leading indicators, and trending.

2010 ISM Workshop, August GA 3

Overview

•

Leading Indicators and Safety Culture continue to gain interest, but few excel

•

Why Not?–

Must understand Systems

–

How you use Leading Indicator data is more important than what they are

–

Trends are more important than Level–

Numerical Goals for level destroy Systems and usefulness of Leading Indicators

–

Trend = Culture Change = Risk Change = Disaster?


Not a Clockwork Mechanism

Humans, and systems consisting of humans are NOT deterministic (clockwork mechanism) or direct cause and effect

UNSAFE

SAFE

By http://www.photos8.com

http://www.photos8.com/


Systems Thinking

Systems (and “safety cultures”) consisting of equipment and humans are subject to many influences, some of which are apparently random.

http://images.google.com/imgres?imgurl=http://www.oitc.com/Dice/images/Dice.gif&imgrefurl=http://www.oitc.com/Dice/&usg=__MByKK6MM6SQQP2iHtv8nPkm5Hik=&h=400&w=400&sz=17&hl=en&start=2&um=1&tbnid=_CEudvuYRBls1M:&tbnh=124&tbnw=124&prev=/images%3Fq%3Ddice%26hl%3Den%26sa%3DN%26um%3D1





Systems Implications

•

A system is a set of interrelated parts•

Any attempt to dissect a system will destroy it

•

The output of an individual is X + f(X), their individual efforts, plus (or minus) interaction

•

No individual has an independent effect on the system

Concepts from Dr. Russ Ackoff

and Dr. W Edwards Deming


Understanding Variation

We need to be able to tell the difference between random noise and a signal of changing conditions. Trend = Δ Risk

Injuries per Month

0

5

10

15

20

Jan-

03

Mar

-03

May

-03

Jul-0

3

Sep

-03

Nov

-03

Jan-

04

Mar

-04

May

-04

Jul-0

4

Sep

-04

Nov

-04

Jan-

05

Mar

-05

May

-05

Injuries per Month - as a Control Chart

0

5

10

15

20

25Ja

n-03

Mar

-03

May

-03

Jul-0

3

Sep-

03

Nov

-03

Jan-

04

Mar

-04

May

-04

Jul-0

4

Sep-

04

Nov

-04

Jan-

05

Mar

-05

May

-05


Trend = Change in Risk•

Dr. Winokur*: “Changes in Culture often Precede Major Accidents”

•

A trend in a leading indicator represents a change in culture, implying Δ risk, but do not immediately leap to is it “good” or “bad”

•

Example –

a decrease in First Aid Case Rate may be “bad” –

if it occurred because workers

were told –

the next person to have a first aid case will be fired!

(pink slip)* Peter S. Winokur, Ph.D.,Chemical Safety and Lifecycle Management Workshop and ESH Group Spring Meeting, March 16, 2010


Implications for Leading Indicators

•

We cannot predict the future•

Even if we could predict an increase in injuries, we would not allow it to happen!

•

Nor do we simply want to live with the predicted future

•

We want to influence the future –build a better future


•

In Dr. Deming’s Red Bead Experiment, we react to the random noise from result to result.

•

Numerical Goals and other incentives had no effect on the results of the process.

•

The process was stable and needed to be changed!

The Red Bead Experiment


Common Trending Errors

Error Typical Behavior

Reaction Action

Reacting to ups and downs (false alarms)

Comparisons point to point, to average, to last year

Tampering and knee jerk reactions, frustration

When stable, work on long-

term history, fix the system

Failure to detect trend

No criteria to separate trend from noise

Molehill grows into mountain

Use SPC to detect trends accurately and in time


•

Developed in 1930 by Dr. Shewhart

Statistical Process Control

Injuries per Month - as a Control Chart

0

5

10

15

20

25

Jan-

03

Mar

-03

May

-03

Jul-0

3

Sep-

03

Nov

-03

Jan-

04

Mar

-04

May

-04

Jul-0

4

Sep-

04

Nov

-04

Jan-

05

Mar

-05

May

-05

Injuries per month – as a control chart

Upper Control Limit

Lower Control Limit

“Baseline” Average

Data being TrendedRed is used for the control limit in the “bad” direction

Green or Blue is used for the control limit in the “good” direction


Trend Rules = Action LimitsThese are “triggers” that alarm when

something is changing:•

One point outside the control limits

•

Two out of Three points two standard deviations above/below average

•

Four out of Five points one standard deviation above/below average

•

Seven points in a row all above/below average•

Ten out of Eleven points in a row all above/below average

•

Seven points in a row all increasing/decreasing


Stable and Predictable•

CAUTION:

Being stable and predictable

(no trends) is not necessarily good!•

It just implies that the risk has been there, is still there, and is predicted to remain there.

•

Should we accept this risk or change?


Leading and Lagging Indicators•

Lagging Indicators dominate at the higher levels, reflecting outcomes. Need to be standardized and dictated from above.

•

Leading Indicators dominate at the lower levels, reflecting processes that achieve the outcomes, improvement methods. Need to be customized, and driven from the bottom-

up. Need to demonstrate correlation and effect upon Lagging Indicators.


Lagging and Leading Indicators

LEADING

LAGGINGDOE HQDOE SiteContractorProjectFacilityTeam

Lagging indicators dominate at high levels, leading at lower levels. Leading indicators are developed from the grass roots, and are customized to the local organization


1stWork down from the Aim of the Organization (Lagging)

2ndWork up from the data on hand (Leading)

3rd Meet in the middle, identify gaps

Developing Indicators


Developing Leading Indicators

•

Central committees for metrics and laundry lists of potential measures are not effective

•

Use flowcharts, observations, and talking to the workers in order to understand the systems in the field–

Worker survey data may help–

Oversight results may help

•

Rapid Prototyping! Only going out and getting data and looking at it will give information

•

Apply SPC and find trends (Δ Risk)•

Refine through trial and error and keep learning!


SRNS Experience•

Significant increase in injuries and sentinel events July –

August 2009

–

Acid Burn, Arc Flash•

Analysis revealed many indicators with trends prior to July 2009–

Interestingly, DART showed a change before TRC

•

These indicators became the basis for the current set of Leading Indicators and SICAM

•

Implemented SPC for both Leading and Lagging Indicators (FY 2010 POMC)


The Lagging Indicators

SRNS Total (Ops, Construction, Subs) DARTper 200,000 hours

0.000.200.400.600.801.00

Oct

-06

Jan-

07

Apr

-07

Jul-0

7

Oct

-07

Jan-

08

Apr

-08

Jul-0

8

Oct

-08

Jan-

09

Apr

-09

Jul-0

9

Oct

-09

SRNS Total (Ops, Construction, Subs) TRCper 200,000 hours

0.000.200.400.600.801.001.201.401.60

Oct

-06

Jan-

07

Apr

-07

Jul-0

7

Oct

-07

Jan-

08

Apr

-08

Jul-0

8

Oct

-08

Jan-

09

Apr

-09

Jul-0

9

Oct

-09


The Early Indicators

0

1000

2000

3000

4000

5000

6000

7000Fe

b-08

Mar

-08

Apr-

08

May

-08

Jun-

08

Jul-0

8

Aug

-08

Sep

-08

Oct

-08

Nov

-08

Dec

-08

Jan-

09

Feb-

09

Mar

-09

Apr-

09

May

-09

Jun-

09

Jul-0

9

Aug

-09

Sep

-09

Oct

-09

Number of Personnel Observed by BBS

Decreasing Trend

98.0%

98.5%

99.0%

99.5%

100.0%

Feb-

08

Mar

-08

Apr

-08

May

-08

Jun-

08

Jul-0

8

Aug

-08

Sep

-08

Oct

-08

Nov

-08

Dec

-08

Jan-

09

Feb-

09

Mar

-09

Apr

-09

May

-09

Jun-

09

Jul-0

9

Aug

-09

Sep

-09

Oct

-09

Percent Safe Behaviors BBS

Beginning ofDownward Trend

Number of Management Field Observations Conducted

0

200

400

600

800

1000

1200

1400

Jan-

07

Mar

-07

May

-07

Jul-0

7

Sep

-07

Nov

-07

Jan-

08

Mar

-08

May

-08

Jul-0

8

Sep

-08

Nov

-08

Jan-

09

Mar

-09

May

-09

Jul-0

9

Sep

-09

Decreasing Trend

SICAM

0

2

4

6

8

10

12

14

16

Aug-

08

Sep-

08

Oct

-08

Nov

-08

Dec

-08

Jan-

09

Feb-

09

Mar

-09

Apr

-09

May

-09

Jun-

09

Jul-0

9

Aug-

09

Sep-

09

Oct

-09

SCD-4 Corrective Actions From ERROR and FACT FINDING

Growth in Corrective Actions

(and well correlated to TRC)


SICAM•

Safety Improvement Compensatory Actions and Measures implemented October 2010

•

Rolling Time Outs•

Management Field Observations and Behavior Based Safety Observations re-invigorated–

These became Leading Indicators

•

Significant improvements in the Leading Indicators and then significant improvement in Operations TRC and DART

•

Minor warning in June 2010 heeded


Industry Best Practice


Path to Success•

Reviews and acceptance by professional societies, INPO, DuPont, and DOE

•

Methods built upon valid theory and experience (Shewhart, Deming, Ackoff)

•

Rigor, standardization, and low expense•

Statistical Process Control for trend alarms

•

Trends = change in risk•

Trend is more important than level

•

Leading Indicators Build a Better Future

Savannah River Nuclear Solutions Fluor Government Group · 2010 ISM Workshop, August GA 3 Overview...

Documents

Transcript of Savannah River Nuclear Solutions Fluor Government Group · 2010 ISM Workshop, August GA 3 Overview...