Can we predict safety culture? Safety modelling and analysis of organizational ... ·...

Can we predict safety culture?Safety modelling and analysis of

organizational processes in air traffic

Can we predict safety culture?Safety modelling and analysis of

organizational processes in air traffic

Sybert Stroeve, Alexei Sharpanskykh, Barry Kirwan

EUROCONTROL CARE Innovative III Project

vrije Universiteit amsterdam

EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 2

ContentsContents

Safety culture

Organizational modelling

Model results and validation

Conclusions


Safety in complex organizations Safety in complex organizations

Human1

Human3

Human2

System1

System4System2

System3

Accident

Human6Human4 Human5System5


Organizational & Safety culture Organizational & Safety culture

Values: What is important

Beliefs: How things work

Norms: The way we do things around here

Organizational culture

Safety of operations

Safety cultureOrganizational culture aspects that impact safety


Safety culture measurement Safety culture measurement

What is important?

How do things work?

What is the way you do things around here?

Safety culture survey

Safety culturesurvey data

Causal relations with organizational structures and processes?

How to (structurally) improve safety culture?

Effect on safety risk?

Remaining questions / Research aim

?


Case & SourcesCase & Sources

Case� Safety culture issues in safety occurrence reporting at a specific Air Navigation Service Provider (ANSP-3)

Sources for model development� Safety culture issues identified by EUROCONTROL at ANSP-1&2� Formal organizational documentation of ANSP-3� Interviews at ANSP-3 and Eurocontrol HQ� Agent-based organizational modelling method� Sociological & psychological literature

Sources for model validation� Results of EUROCONTROL safety culture questionnaires at ANSP-3� Results of EUROCONTROL safety culture workshops at ANSP-3


ContentsContents

Safety culture



Conclusions


Organizational modelling framework(Sharpanskykh, 2008)Organizational modelling framework(Sharpanskykh, 2008)


Organization-oriented view (example)Organization-oriented view (example)

Safety

Investigator

Safety

Manager

Safety Investigation Unit

Level 1

Level 2


Performance oriented view (example)Performance oriented view (example)

………

Level of details of (incident/ hazard) notification reports

It is required to maintain a sufficient level of details of (incident/hazard) notification reports

18.2

Proficiency level of incident investigators

It is required to maintain a high proficiency level of incident investigators

18.1

Timeliness and quality of the incident investigation

It is required to maintain timeliness and a high quality of the incident investigation

18

Performance indicatorGoalN


Process-oriented view (example)Process-oriented view (example)

Create a

notification report

Investigation of

an occurrence

Begin

begin_or(or1)

Report

occurrence?

Yes

Preliminary processing

of a notification report

begin_or(or3)

Decision

positive?

Making decision about

the investigation

necessity

Discussion of the

intermediate occurrence

investigation results

Distribute the final

safety occurrence

assessment report

end_or

(or4)

No

Yes

end_or

(or3)

No

End

Making decision

on the occurrence

reporting

Update an interim safety

occurrence assessment

report

begin_or(or2)

Report provided

to SIU?

Making decision about

the notification report

assignment

Preliminary

assessment of

an occurrence

Yes

Implementation of

safety

recommendations

end_or

(or2)

No

begin_or(or4)

Recommenda-

tions endorsed?

Yesend_or

(or1)

No

Responsibilities for task Investigation of an occurrence

Execution and making technological decisions: Safety Investigator roleMonitoring, consulting and making managerial decisions: Safety Manager role


Agent-oriented viewAgent-oriented view

Controller

Agent 1

Controller

Agent 4

Controller

Agent 3

Controller

Agent 2Supervisor

Agent

Manager

Agent

Internal

model

Environment

(external world)

observation

action

communication

actionobservationMulti-agent organization


Internal agent perspectiveInternal agent perspective

Communication

Observation

Communication

Action

beliefs

knowledge

Information attitudes

desires

commitments

intentions

Pro-attitudes

Characteristics

needs

decisionmaking

skillspersonality


Agent’s information and pro-attitudesAgent’s information and pro-attitudes

Dynamic internal states: Temporal Trace Language

Relations between internal states: Causal Networks

Commitment to safety

of a controller

Priority of safety-related

goals in the role

description

Perception of

commitment to safety of

management

Perception of

commitment to safety of

team

Influence of a controller

on safety activities

Maturity level w.r.t. ATC

task

e1

e2

e3

e4

e5

e6

w1

w2

w3

w4

w5


Agent’s decision making: Vroom’s expectancy theoryAgent’s decision making: Vroom’s expectancy theory

Report an

occurrence Own group approval

Contribution to

organizational safety-

related goals

Self-esteem, self-

confidence, and self-

actualization needs

Administrative reprimand

Improvement of safety

Material reward Extrinsic needs

E12

E13

E14

E15

I21

I32

V1

V2

V5

V4

Social interaction needs

Own group appreciation of

the action

Management approval V3

Management appreciation of

the action

E16

I43

I51

Intrinsic needs

I64

I65

I35

I45

Decrease of own professional

status in own group

Decrease of own professional

status in management’s

opinion

E17I72

I75

E18

I83

I85


Model input (examples) Model input (examples)

Evidence input variables (23)� e1: Priority of safety-related goals in the role description � e4: Influence of a controller on safety activities � e7: Sufficiency of the amount of safety investigators � e10: Sufficiency and timeliness of training for changes � e19: Self-confidence for ATC task � e35: Intensity of informal interactions in the team of controllers� e61: Individualism index of a controller

Weights (60)� Couplings between evidence variables in causal networks

Other parameters (115)� Task durations� Decision making parameters� Environment


Model outputModel output

Perceived management commitment to safetyI7

Perceived supervisor commitment to safetyI6

Perceived team commitment to safety I5.2

Commitment to safety of controllersI5.1

Quality monthly safety overview reportsI4.1

Quality of safety occurrence assessment reportsI3.1

Quality of processed notification reports I2.1

Reporting quality by controllersI1.1

Safety culture indicatorIndex


ContentsContents

Safety culture



Conclusions


Validation PhasesValidation Phases

Model

Questionnaire

1 Safety culture indicators

Organizational information

Model

Questionnaire

2 Safety cultureindicators

Organizationalinformation

Model

Questionnaire3

Organizationalinformation Sensitivity analysis

Workshops

Factors &Recommendations


Results of Phase 1Results of Phase 1

MediumHighPerceived management commitment to safetyI7

LowHighPerceived supervisor commitment to safetyI6

MediumHighPerceived team commitment to safety I5.2

MediumHighCommitment to safety of controllersI5.1

MediumHighQuality monthly safety overview reportsI4.1

MediumMediumQuality of safety occurrence assessment reportsI3.1

-HighQuality of processed notification reports I2.1

MediumHighReporting quality by controllersI1.1

SurveyModelSafety culture indicatorIndex

Model

Questionnaire

Safety culture indicators

Organizational information


Updates of model input valuesin Phase 2 (examples)Updates of model input valuesin Phase 2 (examples)

I have the opportunity to provide input in the ATC systems development or acquisition process.

We are consulted about changes to the technical/engineering system that impact on the

way we do our work0.500.70

Our opinion and input into safety assessments are actively sought after.

Influence of a controller on safety activitiese4

The other people in the organization do not understand the safety roles we fulfil

Safety is a responsibility shared throughout the organization 0.700.90

Balancing safety against the other requirements of my job is a challenge

Priority of safety-related goals in the role

descriptione1

NewOld

Value of variableRelated survey questionsDescriptionVariable

Model

Questionnaire

Safety cultureindicators



Results of Phase 2Results of Phase 2

MediumMediumPerceived management commitment to safetyI7

LowMediumPerceived supervisor commitment to safetyI6

MediumMediumPerceived team commitment to safety I5.2

MediumMediumCommitment to safety of controllersI5.1

MediumMediumQuality monthly safety overview reportsI4.1

MediumMediumQuality of safety occurrence assessment reportsI3.1

-HighQuality of processed notification reports I2.1

MediumMediumReporting quality by controllersI1.1

SurveyModelSafety culture indicatorIndex

Model

Questionnaire

Safety cultureindicators



Major Organizational Factors (Phase 3)Major Organizational Factors (Phase 3)

Sufficiency and timeliness of training for changes8

Developed and implemented Safety Management System (SMS)7

Influence of a controller on safety activities6

Availability of reliable and ergonomic technical systems for controllers

5

Priority of safety-related goals in the role description4

Sufficiency of the number of safety investigators3

Level of development of managerial skills of supervisors2

Sufficiency of the number of controllers1

Major Organizational Factors

Model

Questionnaire


Sensitivity analysis Major Organizational

Factors


Major Organizational Factors (Phase 3)Major Organizational Factors (Phase 3)

GoodSufficiency and timeliness of training for changes8

GoodDeveloped and implemented Safety Management System (SMS)7

GoodInfluence of a controller on safety activities6

GoodAvailability of reliable and ergonomic technical systems for

controllers5

SomePriority of safety-related goals in the role description4

GoodSufficiency of the number of safety investigators3

GoodLevel of development of managerial skills of supervisors2

SomeSufficiency of the number of controllers1

AgreementMajor Organizational Factors

Model

Questionnaire


Sensitivity analysis

Workshops

Major Organizational

Factors


Recommendations (Phase 3)Recommendations (Phase 3)

1. More involvement of controllers in safety assessment for development of new systems and procedures

2. Improve workload of controllers by developing explicit rules for balancing safety and capacity in nominal and non-nominal conditions

3. Improve the quality of management by supervisors

4. Improve coherence and communication in the safety management system

5. Improve the communication about and training for changes

Model

Questionnaire

Organizationalinformation Sensitivity analysis Factors &

Recommendations


Comparison of model-based and survey-based recommendationsComparison of model-based and survey-based recommendations

The model-based recommendations are all consistent with those of the survey study

The survey-based recommendations reflect the larger detail in the organizational context as emerged in the survey workshops and are thus more specific

Additional recommendations identified in the survey study for aspects out-of-the-scope of the model, e.g.� On-the-job-training� Learning processes at the Engineering department

Model

Questionnaire

Organizationalinformation Sensitivity analysis

Workshops

Factors &Recommendations


ContentsContents

Safety culture



Conclusions


Can we predict safety culture?Can we predict safety culture?

Valid predictions of safety culture indicators were achieved on the basis of the model and the survey questionnaire input data

Valid predictions of major organizational factors influencing safety culture and consistent recommendations were achieved on the basis of a sensitivity analysis

The agent-based organizational modelling approach can be an added value in the safety culture survey method� The model provides a structured and formally grounded means for analysis and improvement of safety culture

� Sensitivity analysis helps identifying key factors� The modelling approach is expected to support enhancing questionnaires and preparing safety culture survey workshops

Yes we can!


Beyond our validated caseBeyond our validated case

The value of organizational modelling� Systematic analysis of organizational structures and processes � Agent-based versus traditional methods (system dynamics, operation research)

� Emergent behaviour at various aggregation levels (individuals, units)

Enhance application and scope for safety culture modelling� Modelling and assessment of other ANSPs� Broadening the scope of the organizational model� Modelling the effect of national culture on safety culture � Modelling goal-directed organizational change processes

Relation between safety culture and safety performance� Agent-based model of safety culture influencing operational performance

� Risk assessment by agent-based model of operational performance


Questions / DiscussionQuestions / Discussion

Can we predict safety culture? Safety modelling and analysis of organizational ... ·...

Documents

Transcript of Can we predict safety culture? Safety modelling and analysis of organizational ... ·...