Lean Six Sigma and Assurance of Learning: Challenges and ...
The challenges of so s assurance
description
Transcript of The challenges of so s assurance
© The University of York© The University of York
The Challenges of SoS Assurance:
A Safety Case Perspective
George Despotou
© The University of York
Overview
• Systems of Systems
– Example
• Safety lifecycle
• (Top down) safety case
• SoS hazards and related arguments
• Overview of the safety case
• Related challenges
© The University of York
Systems of Systems
• Distinct class of systems
demonstrating certain attributes
• Geographical dispersion
• Overall objectives
• Complexity
• Multiple elements
• Autonomy
• Collaboration
• Communication
• Heterogeneity
• Common methods often
fall short for these type of
systems
© The University of York
Challenges in SoS Analysis
• Evolving systems
– Constantly evolving and adapting
• Emergent behaviour
– Behaviour affected by this of its components
• Functional composition
– SoS functions composed of elements’ functions
• Network centred
– Heavy reliance on communications
• Multi-attribute failures
– Element failures propagate and manifest as different types
– Trade-offs
© The University of York
SoS Safety
Lifecycle
• Extends ARP
lifecycle
• Incorporates SoS
modelling
approaches
(MODAF/DODAF)
ABM Stages Preliminary
Scenario-based
Hazard
Identification
Activity-based
Hazard
Analysis
Preliminary
SoS Safety
Assessment
Co
mm
on
Mo
de
& E
mer
gen
t (S
oS
) H
aza
rds
Overall
Scenario and
Capabilities
Activities,
Actors and
Information
System
Functions,
Data and
connectivity
System
Realisation and
Infrastructure
System
Safety
Assessment
often black box
SoS Safety
Assessment
Safety Synthesis
© The University of York
SoS Safety Case
• Safety case principles are the same
• Argument: series of statements (claim) about properties (safety) of the system
• Evidence supporting the claims
SoS Safety Argument
Evidence
© The University of York
Top down start
• Definition of safety always
requires
consideration of operation
(relevance)
• Capability based in SoS
• Scope not always
clear
SoS Safety Argument
Evidence Evidence
Elaboration
Definition of Safety
(argument)
© The University of York
Definition of safety context
• Scenario based approaches
• Different acceptability criteria for each
• Safety tolerance (risk)
• Operational fitness
– Performance, security,
availability etc.
• Identification of stakeholders goals
MODAF Ov1
•Hidden sub-scenarios may be
present (roles)•e.g. S&R, intelligence, suppression, maintenance etc.
•Understanding the variations
of requirements in each•Crucial for later stages (ALARP)
© The University of York
How is the capability offered?
• Many systems
collaborating
• Different vendors
• Different technologies, standards, methods
• Often in isolation
(competition?) with
each other
• However, common
operator/customer
– Customer structure
Supplier A Supplier B Supplier C
Supplier D Supplier E
Army
Air Force
Comms
Command
Allies Opera
tor
Facets
© The University of York
Multiple supplier safety caseDefinition of Safety
(argument)
Argument for
System
X
Argument for
System
Y
Argument for
System
Z
Evidence for
system X
Evidence for
system Y
Evidence for
system Z
Assura
nce C
ase f
or
Sys
tem
X
Assura
nce C
ase f
or
Sys
tem
Y
Assura
nce C
ase f
or
Sys
tem
Z
Supplier A Supplier B Supplier C
© The University of York
Multiple supplier safety caseDefinition of Safety
(argument)
Argument for
System
X
Argument for
System
Y
Argument for
System
Z
Evidence for
system X
Evidence for
system Y
Evidence for
system Z
Assura
nce C
ase f
or
Sys
tem
X
Assura
nce C
ase f
or
Sys
tem
Y
Assura
nce C
ase f
or
Sys
tem
Z
Supplier A Supplier B Supplier C
This is a
challenging step
© The University of York
SoS Hazard Anlysis
• Focusing on individual systems is a starting point (e.g.
DDA)
– But not sufficient
• Obscure (complex) causal chain to hazards
– Combination, propagation and transformation of failures
• The degree of this phenomenon can be profound
– Controls for a single hazard may need to be implemented by multiple suppliers
– Often difficult to predict behaviour (e.g. autonomy)
• Not always deviant behaviour
– An SoS can operate as designed
• Yet resulting in (emergent) hazards
© The University of York
The role of simulation
• Exhaustive
– Cannot claim that
– Only to a degree of confidence
• (Exhaustive) manual analysis almost impossible
– Scale
– Configurations
• Known and unknown (adaptability)
• Simulation offers a ‘brute force’ approach to identifying hazardous conditions
– Fault injection
© The University of York
Challenges for simulation
• Two well known challenges
– Valid
• Do the simulated models represent the right system
– Verified
• Has the simulation been implemented correctly?
• Both may undermine our confidence in simulation
• Effectiveness of simulation is unknown
– Cannot argue whether the potential risk reduction from
simulation justifies the cost
• Need for more empirical evidence about simulation
• And tactics to argue/achieve validation and verification
© The University of York
Safety Requirements
• System AC will support the safety argument
– i.e. hazard oriented argument
• Follows identification of SoShazards
– And the contribution of each system
to them
• Assumes ‘straight forward’
contribution of system operation to
hazards
– Discharging safety requirements to
suppliers
Ass
ura
nce
Ca
se X
Definition of Safety
(argument)
Ass
ura
nce
Ca
se Y
Ass
ura
nce
Ca
se Z
Dependency Definition
(hazard controls argument)
© The University of York
(Addressing) Cross supplier
hazards
• Hazards occurring from
combination of failures
• Addressing the hazard may
result in depending on another
suppliers assurance case
– Challenges during acquisition
• Cannot anticipate specific
dependencies in advance
– Impact on acquisition process
(and contracts)
Assurance
Case X
Definition of Safety
(argument)
Dependency Definition
(hazard controls argument)
Assurance
Case Y
Evidence
Evidence
© The University of York
Hazard controls
• Discharging (safety related) requirements
– Potential dependencies among suppliers
• Coordination throughout acquisition is crucial
• Safety policy
– Implemented on top of the SoS elements
– Who will be responsible to monitor/enforce it
• Controller vs. distributed policy
• Any additional requirements to suppliers?
– Onus on the operator to analyse hazards and argue safety
• May result in the operator having to produce evidence
© The University of York
SoS Safety
Case
Definition of Safety
(argument)
Hazard Controls Argument
Ass
ura
nce
Ca
se X
Opera
tional S
C X
Ass
ura
nce
Ca
se Y
Opera
tional S
C Y
Ass
ura
nce
Ca
se Z
Opera
tional S
C Z
SoS Safety
Policy
Argument
SDR
Argument
Other means
(e.g.
controller)
AC
ab
ou
t oth
er
safe
ty r
elate
d
syst
ems
© The University of York
Safety Case - Responsibility
• Not clear
– Except for assurance cases for individual systems
– Interweaved interest
• Supplier needs to be aware of hazard controls
– Dependencies on their system
» SDR and policy
– Claims made, which may depend on other supplier
• Who provides the contextual information?
– E.g. SoS hazard analysis
• operator: ability to do so?
• supplier: access to information?
– Inevitable assumptions
© The University of York
SoS Acquisition Challenges
• Increasingly apparent that collaboration amongst stakeholders is inevitable
• Commercial/competition issues
• Customer/operator driven process
– Clear identification of stakeholder interfaces
– Cannot be a single demonstration
• Continuous process and support
• Not zero sum (?)
– It is in the interests of the suppliers to increase
collaboration (?)
© The University of York
Synopsis
• SoS demonstrate a distinct combination of
characteristics
• Challenged to safety analysis
• Safety case principles are the same
• SoS level hazards are difficult to identify
• Allocation of requirements can involve multiple
stakeholders
• Often, dependencies between suppliers
• Responsibility of the safety case is unclear
• Some of these issues can be incorporated in the
acquisition process
© The University of York
Further Info
• SoS community
– www.dependablesos.org
• Informal; contributions welcome
• www-users.cs.york.ac.uk/~george
• Assurance cases
– GSN: www.goalstructuringnotation.info
– AC editor: http://code.google.com/p/acedit/