VPP Annual Self-Assessments A Process for

Continuous Improvement

STAR Consultants, Inc. www.starconsultants.net

Paul A. Esposito, CIH, CSP Annapolis, MD 21012

410-218-8451

paul.esposito@starconsultants.net

Prepared for: VPPPA

Reg. 3, 2013

2

WHO IS STAR?

In business since 1997

Founded by Paul Esposito, CIH, CSP

33 years of experience

Over 300 VPP Assessments performed since

1992

Located in Annapolis MD

Safety Through Accountability and Recognition

3

Workshop Outline

Drivers and Requirements

Evaluation Process Document Review

Interviews

Visual Observations

Report Format

Goals, Objectives and Action Plans

References/Resources

4

What Are The Drivers?

http://www.osha.gov/dcsp/vpp/index.html

OSHA’s S&H Program Management Guidelines 1/26/1989

OSHA Federal Register Notice 51:33669-77 (1986, September 22).

Revisions: (2000, July 24)

CSP-03-01-003. April 18th 2008

VPP Federal Register Notice January 9, 2009

5

When do you do Evaluations?

As part of the application process

At least one year prior to the application and

Re-survey, to demonstrate ability to actually

evaluate and improve

Annually once you are in the VPP

6

• Annual Self-Evaluations

WORKSITE

HAZARD

ANALYSIS

HAZARD

PREVENTION

& CONTROL

SAFETY

& HEALTH

TRAINING

• Management Commitment

• Policy • Goals, Objectives and

Planning • Visible Top Management

Leadership • Responsibility and

Authority • Line Accountability • Resources • Contract Employees • Written S&H

Management System

• Routine Hazard Analysis

• Significant Change Hazard Analysis

• Pre-use • Baselines

• Routine Self-Inspections

• Hazard Reporting • Industrial Hygiene • Investigations • Trend Analysis

• Certified Professional Resources

• Hazard Elimination and Control Methods

• Hierarchy of Controls • Engineering • Admin • Work Practices/Rules • PPE • Discipline

• Process Safety Management

• Occupational Health Care

• Preventive Maintenance

• Hazard Tracking • Emergency

Preparedness

• Managers • Supervisors • Employees • (Contractors) • Emergencies • PPE

VPP Criteria – Evaluation Report

Management

Commitment

Employee

Involvement

• (Encourage-ment

• Participation • Committees)

2008: Appx C, Section D: CSP 03-01-003!

MANAGEMENT

LEADERSHIP

7

VPP Annual Self-Evaluation

Is: Objectives and Action Plan Review

A systematic evaluation of your company to each VPP element.

An examination and verification of management records, procedures, training and the workplace environment.

Interviews with employees and supervisors in all departments

Is not: Plant audit or inspection looking for physical hazards and violations.

Corporate Compliance Audit

8

An Assessment

Document

Review

Interviews Verify/Observe

Complete the

Protocol

Preparation, Protocols and Document Review

9

Who Can Perform a Self-

Evaluation? Site Officials

Safety Committee

Quality Auditors

Corporate Personnel

Third Party

Insurance Companies

Consultant

Qualifications

Knowledge of S&H programs

Knowledge of site operations

Independent

Or any mix of these.

10

Creating the Assessment Protocol

Use public or commercial criteria lists, create a

listing of program expectations for each of the

program elements. • Site Based Participation Evaluation Report

(note: this evaluation does NOT match up to the annual

report elements)

Or, Self-Assessment Guides

are commercially available

11

Step 1: Modify or Develop

Criteria Use OSHA Form 33 (SHARP)

Use OSHA’s Program Management Guidelines (1/26/89)

Use Checklist in OSHA’s “Managing Worker Safety and Health”, Chapter 12

Department of Energy (VPP Part IV), http://tis.eh.doe.gov/vpp/policy/policy.html or

Site-based Evaluation Report

Commercially available protocols (e.g., VPPProfile™ Screen from STAR)

12

Program Evaluation - VPP

Requirements An Element of Management Leadership

Requirements:

Annual Review – conducted by qualified personnel

Evaluate their success in meeting the goal and

objectives,

The written annual evaluation must identify the

strengths and weaknesses

Must contain specific recommendations, time lines,

and assignment of responsibility for making

improvements.

And address the status of last year’s actions, goals and

objectives

Chapter III C.1.d

13

Program Evaluation - VPP

Requirements An Element of Management Leadership

Requirements: It must also document actions taken to satisfy the recommendations.

The annual self-assessment must be in place at least one year prior to VPP application submittal

Submitted as a Narrative Report, Feb. 15th to OSHA VPP (2 copies)

Injury rates and applicable contractor rates. With explanation of potential problems

Construction: must also perform a final evaluation immediately prior to completion of construction

Chapter III C.1.d

14

Malcolm Baldrige (Form 33) Approach LEVEL/

CATEGORY: 1

A. Ineffective 2

B. Developmental 3

C. Basic 4

D. VPP 5

E. World Class

1.2.1* Goals & Objectives (Action plans)

1. Site H&S goal is established for injury and illness rates.

1. Some descriptive goals exist, and are in part based on a review of injury and illness analysis and the ESP Work Plan.

2. Some objectives to

support the goals are developed and documented.

3. Some goals are owned by

Management (not H&S).

1. Objectives have been established with input and support from all levels in the organization.

2. Supporting objectives

aim at a specific area of performance that can be measured or verified.

3. Goals and objectives are

communicated to workers.

4. Goals and objectives are

developed for most teams and committees.

5. Some goals and

objectives are owned by line management.

1. Some goals and objectives also stem from an analysis of incident root causes, inspection findings, worker concerns, maintenance safety issues, and other safety data.

2. Departments each have

established one or more objectives annually as part of the H&S program.

3. The majority of

managers, supervisors and the workforce can explain program goals and objectives and the measures to achieve them.

1. Objectives are developed as part of a grass roots process, with agreement from the majority of the workforce.

2. Personnel at all levels

can explain the desired results and measures for achieving H&S program objectives.

Step-by-Step, 5 Level Approach

15

Evaluation Example

From CSP 03-01-003, Appx D:

And add “Partially Meets”

16

Response Scoring / Weighting

ResponsePoints

receivedPoints

available

Evaluation Criteria

1.1.1 Meets 3 3

1.1.2 DNM 0 3

1.1.3 PM 2 3

1.1.4 N/A - -

1.1.5 UD 1 3

Raw Score* 6/12 12

Score 50%

*Points Achieved/Points Available

Red = 0 – 50

Yellow = 51 – 79

Green = 80 - 100

17

Management

Employee Involvement

Worksite Analysis

Hazard Prevention

Training

Using a scoring system,

measure and compare

program elements each year

to truly determine of your

programs have improved...

> 50%

51 to 79%

< 80%

Red

Yellow

Green

Previous Evaluation Current Evaluation

ID Description # of

Priority

1’s

Section

Score

# of

Priority

1’s

Section

Score

1.1 Safety & Health Policy 3 56 2 78

1.2 Goals & Objectives 2 44 1 59

1.3 Planning 2 76 2 78

1.4 Top Management Involvement 1 79 - 85

1.5 Responsibility & Authority - 75 - 78

1.6 Line Accountability 1 62 1 62

1.7 Resources - 92 - 100

1.8 Contract Workers - 88 - 88

1.9 Written Safety & Health Management Systems 5 47 3 72

1.10 Program Evaluation - - 2 80

1.0 Management Commitment 12 59 11 69

2.1 Encouragement 5 55 3 76

2.2 Participation 4 46 2 83

2.3 Committees 3 76 2 84

2.0 Employee Involvement 12 59 7 81

3.1 Baseline Assessments 4 33 3 50

3.2 Routine Hazard Analysis 3 76 2 81

3.3 Change Hazard Analysis 3 33 2 64

3.4 Inspections 9 25 5 59

3.5 Reporting System 2 76 1 89

3.6 Industrial Hygiene Program 7 17 5 33

3.7 Investigations 3 58 1 76

3.8 Trend/Pattern Analysis 8 46 3 79

3.0 Worksite Hazard Analysis 39 46 22 66

Comparative Scores

18

Assessment Activities

Pre-Assessment Activities

Select and Schedule

Facilities/Department to

Assess

Select Team Members and

Confirm their Availability

Gather Background Info.

• Discuss program • Advance visit (optional) • Pre-survey questionnaire

Post-Assessment Activities

Prepare Draft Report

• Send to cognizant parties

Issue Final Report

Action Planning

• Proposed action(s) to address

each finding

• Responsibility/timetable

actions

• CO Endorsement

Schedule Follow-Up

Key Activities at Activity

STEP 1- Understand Management Systems

STEP 2- Assess Strengths and Weaknesses

• Internal risk • Internal controls

STEP 3 Gather Evidence

• Assessment • Verification (Interviews) • Sampling strategies

STEP 4 Evaluate Findings

• Review operations • Perform Visual Observations • Evaluate controls

STEP 5 Report Findings

•Score the Assessment

• Exit meeting

• Review Policies and Procedures • Interview responsible parties

Preparation, Assessment, Report, and Follow-up

• Define scope • Identify priority hazards • Prepare/modify protocols • Allocate resources

Plan the Assessment

19

WORKSITE

HAZARD

ANALYSIS

HAZARD

PREVENTION

& CONTROL

SAFETY

& HEALTH

TRAINING

• Management Commitment

• Policy • Goals, Objectives and Planning • Visible Top Management Leadership • Responsibility and

Authority • Line Accountability • Resources • Contract Worker

Coverage • Written S&H

Management System • Annual Self-Evaluations

30 Elements OSHA’s Program Management

Guidelines - VPP Criteria

Management

Commitment

Employee

Involvement

1/26/89 & 3/25/03

MANAGEMENT

LEADERSHIP

20

Document Review

Documents are reviewed to determine the depth of understanding about the requirements and programs. Match the records to those required by the procedures or documentation

Interview Authors and Responsible Parties to verify your understanding about the process

Identify a hypothesis about where gaps may exist

Verify program strengths by planning additional interviews (I) and visual observations (V)

Challenge the gaps to determine of they are systematic or random

Written programs by themselves have limitations, such as:

who reads them/knows them

how are they used

are they really implemented and to what extent

Gaps at this level will almost guarantee systematic failures at the

implementation stage.

21

List of Documents Completed by Site Completed by Evaluator

Document Description Owner

(Individual)

Location

(Organization)

Completed

Yes / No Comments

1.1 Policy

1) Safety and Health Policy 1) 1)

1,2 Goals & Objectives

1) Safety and Health Program

a). Goals

b). Supporting Objectives

c). Action Plans

(previous and current year’s)

1)

a)

b)

c)

1)

a)

b)

c)

1.3 Planning

1) Management Meeting Minutes 1) 1)

1.4 Top Management Involvement

Section 1.10 Program Evaluation & 2.3 Committees

1.5 Responsibility and Authority

1) Job Descriptions: (sample three or four of each)

a) Management

b) Salaried

c) Hourly

2) See 1.9 – Written Programs

1)

a)

b)

c)

2)

1)

a)

b)

c)

2)

Where do we find these?

Who owns them?

22

Goals and Objectives - Examples

Goal: Improve our Hazard Identification & Hazard Reduction.

Objective: "Conduct weekly inspections with emphasis on good housekeeping, proper use of protective equipment, condition of critical parts of equipment, and preventive maintenance. "

Goal: Improve our Accident Investigation Process.

Objective: "Determine the cause(s) of any accident within 24 hours. "

Goal: Implement a Job Safety Analysis Program.

Objective: "Complete one job safety analysis each month in each department, with follow-up revision of safe work procedures and employee training by the following month."

Goal: Improve our Emergency Preparedness Capability.

Objective: "Hold and evaluate emergency drills for tornadoes every six months and a joint fire drill/evacuation with local emergency organizations every year.”

Still need to ID Action plans, owners, deadlines

23

Worker Interviews

What questions do you ask? To start with?

Follow-up?

To avoid?

How many employees do you interview? One-on-one

Focus Groups

Who do you interview Workers

Supervisors

Committees

Contractors

Get specific examples of both strengths and weaknesses for follow-up!!!

Interviews

24

Personnel Interviews

… Are typically performed after you review documentation to determine if the information is appropriate, communicated, understood and practiced.

… Employee interviews are typically the initial step in verifying S&H program implementation and effectiveness.

… Perform personnel interviews after your document review and responsible party examinations. It is helpful to know some of the answers or problem areas before the interviews. Use the interviews to get specific examples.

Interviews

25

Interview Qs from Appx F.

1. Can you tell me what the safety and health policy is at this worksite?

2. Can you tell me what the overall goal(s) for safety and health is at this worksite? Where are we now? Ahead or behind?

3.What is one objective in your department to achieve the goals?

4. What is one method of reporting a safety or health concern?

What was the last unsafe practice you reported and/or

corrected?

5. How do your supervisors demonstrate their involvement in

safety and health?

Interviews

26

Personnel Interviews

Employee Interview Table

Employee # 1 2 3 Max - allMin - allAvg - allStd Dev - all % 4 & 5

B. Orientation and Training 67.00%

1. Did you receive safety and health training when you began

to work here? (If so, please describe). How soon after you

begin to work did you receive training? How long did it last? 2 4 5 5 2 3.67 1.25

2. If you did not get training when you were first hired (or

transferred to a new job), have you received any basic safety

and health training since that time? If so, please describe. 3 4 5 5 3 4.00 0.82

Maximum 3 4 5

Minimum 2 4 5

Average ### ### ###

Standard Deviation 0.5 0 0

1- Weakness ……………………………………………. Strength- 5

27

Element Scoring: Perception Survey

Management strengths may be considered by the % of 4’s and 5’s for each category ( # interviewed). Gaps = % scores less than 50%, Average responses less than 3.5 or Standard Deviation greater than 1.0.

Management Commitment 62% 3.2 0.87

Gaps: Goals and objectives Top mgmt involvement Communication

Employee Involvement 45% 2.3 1.33 Gaps

VPP knowledge Committee/Team members Committee/Team Activities

Worksite Analysis 70% 3.6 0.54 Gaps:

Reporting hazardous conditions Inspection results?

Hazard Prevention and Control 75% 3.7 0.32 Gaps

Training 64% 3.6 1.22 Gaps

Specific work procedures/JSAs

1 weakness ------ ------ ------ strength 5

% Score Standard Deviation Avg Score

GAPS

28

Visual Observation

Document

Review

Interviews Verify/Observe

Complete the

Protocol

Final Verification

29

Visual Observations

Take records (both accurate and suspect) and Re-survey.

Action Plans

Mishaps

Inspections

Meeting Minutes

JSA/Hazard Analysis

Each Program Element

Informally discuss process and closure with cognizant workers and supervisors

Visit areas and people: issues from interviews

Confirm or deny strengths and weaknesses.

30

Answering the Criteria

Document

Review

Interviews Verify/Observe

Answer the

Criteria

Final Assessment Step

31

Systemic or Random?

Verify if a process ensures safe behavior

Does the Process/Procedure:

Exist

Conflict with other processes or motivations

Lack documentation or communication

Lack motivation (metric)

Lack QC

Report in statistics, if possible:

“8 out of 10 reports were missing assignment of

responsibility”

32

Final Verification

To answer the criteria an evaluator:

Reviews Documents (D)

Performs Interviews (I)

Conducts Visual Observations (V)

Perform at least two of the three in

combination to fully verify your finding or

rating.

Compare your findings and evidence to the

Criteria to verify that the assessment is

complete and in enough detail.

33

Evaluation

The most comprehensive evaluation includes: reviewing written programs,

walking through the workplace, and

interviewing employees.

During this process participants should be answering the following questions relating to each element and sub-element of their safety and health management system:

Is it comprehensive?

Is it operating effectively and meeting established goals and objectives?

What improvements can be made to make it even more effective?

What goal modifications should be made for the upcoming year?

34

Annual Report

The Report will typically examine each program element from the following viewpoints:

A description of past activities and action plans from the prior year

A statement describing the current condition/status

An evaluation discussing strength and weaknesses to each program element

Corrective actions plans to resolve issues

Specific actions, responsible parties and target completion dates

Report in statistics or scores as a better way to communicate with Management

Appendix D

Rates, Significant Changes, Narrative, Success Stories

35

Calculating Your Rates

Two Tables

Site (employees and supervised temps and

contractors)

Applicable Contractors

Hours

36

Annual Report Outline

A: Summary

Site info, contractors, union (if applicable)

B: Injury and Illness Rate info

C: Significant Changes or Events

D: Narrative, each VPP Element

Ex: Incident investigation program

Narrative (strengths and weaknesses)

Status of last year’s improvement initiatives

Current next step improvements: assigned to. Target date

E: Summary Goals (last yr’s status and this yr)

F: Success Stories

37

Additional References/Resources

OSHA’s Draft Managing Worker Safety and Health, Chapter 12 Evaluating Your

Safety and Health Program (1991-1993)

http://www.labor.mo.gov/DLS/WorkplaceSafety/managing_safety.asp

Perception Surveys

National Safety Council

www.NSC.com

STAR Consultants

www.starconsultants.net

Assessment Tools

Preparing for the Voluntary Protection Programs

Margaret Richardson, Wiley Press @ 1999

STAR Consultants

VPP Self-Evaluation Manual

Esposito, P. “Building on Your VPP Success” Autumn 2010. The Leader,

Voluntary Protection Program Participants Association

Esposito, P. “Annual Self-Evaluations” published in the VPPPA Leader, Fall

2001

The Strategic Partner to help your

safety and health programs and risk

management!!!

Paul Esposito, CIH, CSP

Paul.esposito@starconsultants.net

410-218-8451

Questions?