Post on 30-Mar-2015
Mapping Assurance to the Software Engineering Process
Alfred H. Kromholz, Ph.D.
The MITRE Corporation
kromholz@ mitre.org +1-703.883.7331
Copyright © 2004 The MITRE Corporation
Copyright © 2004 The MITRE Corporation2
Relation between Waterfall Process & V-Chart
Plans & RqmtsProd. Design
Detail Design
Unit TestInteg. Test
Syst. Test
Code
Concept
Accep. TestConcept
Plans and Requirements
Product Design
Code
Detail Design Unit Test
Integration Test
System Test
Acceptance Test
Change the shape from waterfallto fall and rise
Traditional Waterfall
The obvious question: Why bother?
Copyright © 2004 The MITRE Corporation3
Conceptual
Plans and Requirements
Product Design
Code
Detail Design Unit Test
Integration Test
System Test
Acceptance Test
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements Element
Test
Subsystem Test
Development V-ChartStarts with a Product Focus to Process Abstraction
Analysis phase(Decomposition)
Synthesis phase(Verification)
Development V-ChartShift from Product Focus to Process Abstraction
Copyright © 2004 The MITRE Corporation4
For efficiency, focus on
interfaces & interactionsSystem
Requirements
Subsystem Requirements
Element Requirements
Subsystem Test
Development V-ChartProcess and Product
Needs
Elements
Element Test
System Test
Acceptance Test
Analysis phase(Product Explosion)
Synthesis phase(Product Implosion)
The structure of the decomposition is the product architecture.
Analysis phase Synthesis phase
Copyright © 2004 The MITRE Corporation5
Development V-ChartTypical Generic Products of the Process
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
NeedsStmt
System DesignSpec
Sub-syst DesignSpec
Detail DesignSpec
Test Report
Test Report
Test Report
Test Report
+
Copyright © 2004 The MITRE Corporation6
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
Development V-ChartWhen Do We Establish Test Criteria?
Test Report
Test Report
Test Report
Test Report
+
NeedsStmt
DesignSpec
DesignSpec
DesignSpec
AcceptCriteria
TestSpec
TestSpec
TestSpec
Development V-ChartTypically, We Establish Test Criteria on the Way Up
Same products on this side
Earlier products on this side
AcceptCriteria
TestSpec
TestSpec
TestSpec
Development V-ChartWe Should Establish Test Criteria on the Way Down
Generally, just barely before we test
Copyright © 2004 The MITRE Corporation7
Validate / Verify
V&V
V&V
V&V
Development V-ChartVerify and Validate Before Staring Upward
NeedsStmt
DesignSpec
DesignSpec
DesignSpec
AcceptCriteria
TestSpec
TestSpec
TestSpec
Test Report
+
Test Report
Test Report
Test Report
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance Test
Copyright © 2004 The MITRE Corporation8
Validate / Verify
V&V
V&V
V&V
Development V-ChartValidate, Plan for Verify & DFT/DFM Early
NeedsStmt
DesignSpec
DesignSpec
DesignSpec
AcceptCriteria
TestSpec
TestSpec
TestSpec
Test Report
+
Test Report
Test Report
Test Report
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Acceptance TestDFT/DFM Feedback
DFT/DFM
DFT/DFM
Copyright © 2004 The MITRE Corporation9
Needs
System Requirements
Subsystem Requirements
Element Requirements
NeedsStmt
System DesignSpec
Sub-syst DesignSpec
Detail DesignSpec
Element Test
Subsystem Test
System Test
Acceptance Test
Let’s Go Back to the Original Development V-Chart
Rationale
Rationale
Rationale
Rationale
Test Report
+
Test Report
Test Report
Test Report
Elements
The “Missing” Rationale
How did we get
from here
How did we get
from here to here?
Or here?
Or here?
Or here?
Does it need to be captured?
Recording the rationale for
decomposition can help us
Copyright © 2004 The MITRE Corporation10
Where Does Assurance Fit In?
Two broad customer circumstances
• Functional expectations but no assurance requirements– Expectations may be informal – mass-market, “shrink-wrapped” products
– Expectations may be formal – supplier requirements documents & specifications– Customer says, “Convince me that the product meets expectations.”
– Customer may say, “Why should I have confidence in your claims?”
• Both functional requirements and assurance requirements– Customer tells supplier what it takes to convince
– Assurance requirements may be direct – i.e., included in the specifications
– Assurance requirements may be indirect – i.e, specifications identify product and/or process standards to be followed
Copyright © 2004 The MITRE Corporation11
Customer ActivitiesSupplier ActivitiesThese activities take place during the
a posteriori phase
after the product is produced
Sub-claim evaluation
Sub-claim evaluation
Claim evaluation
Acceptance
Assurance V-Chart
AssuranceClaim
Evidence
Evidence
Evidence
Evidence
Body of Evidence
Sub-Claims
Sub-Claims
Sub-Claims
No customer-specified assurance requirements
Two groups of assurance-related
activities
Copyright © 2004 The MITRE Corporation12
Expectations, standards, desiderata
Claims re Expectations
Sub-claims re Expectations
Sub-claims re Expectations
a priori phase
a posteriori phase
Body of Evidence
Sub-claim evaluation
Sub-claim evaluation
Claim evaluation
Acceptance
Argument
Argument
Argument
Argument
The specific problem domain implies both the applicable
standards and the Assurance Level required
Assurance V-Chart (Augmented)Customer-specified assurance requirements
But how do we know these are necessary and sufficient with respect to the previous level?
Evidence-Item Classes
We add an a priori phase that sets up an assurance case before a
product is designed and producedAs in the previous
chart
Tells the supplier what evidence is expected (and maybe in what form to present it)
are decomposed into
We add an “argument” that justifies how sub-claims support a claim and evidence supports
the subclaims –
are decomposed into
are decomposed into
are decomposed into
for each decomposition
activity
Copyright © 2004 The MITRE Corporation13
DFT/DFM Feedback
Evidence
DFT/DFM
Evidence
DFT/DFM
Evidence
Needs
System Requirements
Subsystem Requirements
Elements
Element Requirements
Element Test
Subsystem Test
System Test
Evidence
Evidence
Evidence
Evidence
Evidence
Evidence
Evidence
Evidence
Evidence
Acceptance Test
Development V-ChartEvidence for Assurance Comes from All Development Activities
Validate&Verify
Evidence
V&V
Evidence
V&V
Evidence
V&V
Evidence
Evidence is gathered in accordance with the
evidence classes identified in the a priori phase.
Remember the “rationale” for
decomposition?This is where it comes in handy.
Remember the “rationale” for
decomposition?This is where it comes in handy.
Remember the “rationale” for
decomposition?This is where it comes in handy.
Remember the “rationale” for
decomposition?This is where it comes in handy.
Copyright © 2004 The MITRE Corporation14
Assuranceprocess
Development process
Minimum SeparationDevelopment starts after 1 or 2 levels of Assurance
decomposition
Ideal SeparationAll necessary evidence
identified before Development starts
Assurance CompletionOccurs after all evidence
from Development is received
Assurance SynthesisStarts as early as evidence classes
are identified and evidence instantiations are available
Time Relationship of Assurance & Development V-Charts
How much time?
Clearly, assurance requirements should be
identified before development begins.
Current Work & Future Directions
Copyright © 2004 The MITRE Corporation16
Using Adelard Safety Case Editor to Argue a Safety Case
This illustrates the idea of a supplier making a claim to a purchaser that a product is safe.Thus the arrows go upward from evidence to argument to claim.
Copyright © 2004 The MITRE Corporation17
Arguing a Safety Case: Basic Supplier Approach
CLAIMArchitecture
is safe
ARGUMENTPartitioning is
proper
ARGUMENTMulti-versioning Dissimilarity is
properly provided
ARGUMENTSafety
Monitoring is provided
EVIDENCEPartitioning
Documentation
EVIDENCEMulti-versioning Documentation
EVIDENCESafety-Monitoring
Documentation
Note that there is no justification provided to support the assertion that the given arguments support the claim.
Argument is made on the basis of the
evidence.
Evidence is produced prior to making the claim.
Collection of arguments is used to
support the claim.
Copyright © 2004 The MITRE Corporation18
Arguing a Safety Case: Standards-Based Supplier Approach
CLAIM2.3
Architecture is safe
ARGUMENT2.3.1
Partitioning is proper
ARGUMENT2.3.2
Multi-versioning Dissimilarity is
properly provided
ARGUMENT2.3.3
Safety Monitoring is
provided
EVIDENCE2.3.1
Partitioning Documentation
EVIDENCE2.3.2
Multi-versioning Documentation
EVIDENCE2.3.3
Safety-Monitoring Documentation
However, there is still no justification provided. Indeed, many standards documents not only omit but deliberately exclude rationale.
The supplier can include references to governing documents
as part of the assurance package to help
support assertions.
(Numbering here is adapted from DO-178B, Software Considerations in Airborne Systems and Equipment Certification)
Copyright © 2004 The MITRE Corporation19
CLAIM2.3
Architecture is safe
CLAIM2.3.1
Partitioning is proper
CLAIM2.3.2
Multi-versioning Dissimilarity is
properly provided
CLAIM2.3.3
Safety Monitoring is provided
EVIDENCE2.3.1
Partitioning Documentation
EVIDENCE2.3.2
Multi-versioning Documentation
EVIDENCE2.3.3
Safety-Monitoring Documentation
ARGUMENTPartitioning, multi-versioning,
and safety monitoring are necessary and sufficient for a
safe architecture
Arguing a Safety Case: Justified Supplier Approach
Justification can be provided by the supplier to support the claim ...
but there is no guarantee that this will satisfy the
customer.
Copyright © 2004 The MITRE Corporation20
REQUIREMENT2.3
Architecture is safe
REQUIREMENT2.3.1
Partitioning is proper
REQUIREMENT2.3.2
Multi-versioning Dissimilarity is
properly provided
REQUIREMENT2.3.3
Safety Monitoring is
provided
EVIDENCE class2.3.1
Partitioning Documentation
EVIDENCE class2.3.2
Multi-versioning Documentation
EVIDENCE class2.3.3
Safety-Monitoring Documentation
ARGUMENT 2.3Partitioning, multi-versioning,
and safety monitoring are necessary and sufficient for a
safe architecture
Arguing a Safety Case: Requirements-Based Approach
In the Requirements-Based approach, the arrows are top-down
Customer defines what “safe architecture” means
(at least internally to the acquiring organization)
Customer identifies
expectations regarding proof that architecture
is safeNote:The bottom level calls for classes of evidence, not specific documents.
Customer defines that architecture must be safe
Customer establishes
requirements based on
decomposed definition
Copyright © 2004 The MITRE Corporation21
REQ’MENT2.3
Architecture is safe
REQ’MENT2.3.1
Partitioning is proper
REQ’MENT2.3.2
Multi-versioning Dissimilarity is
properly provided
REQ’MENT2.3.3
Safety Monitoring is provided
EVIDENCE Class2.3.1
Partitioning Documentation
EVIDENCE Class2.3.2
Multi-versioning Documentation
EVIDENCE Class2.3.3
Safety-Monitoring
Documentation
ARGUMENT 2.3Partitioning, multi-
versioning, and safety monitoring are necessary and sufficient for a safe
architecture
Arguing a Safety Case: Consolidated Flow
CLAIM2.3
Architecture is safe
EVIDENCE2.3.1
Partitioning Documentation
EVIDENCE2.3.2
Multi-versioning Documentation
EVIDENCE2.3.3
Safety-Monitoring
Documentation
CLAIM2.3.1
Partitioning is proper
CLAIM2.3.2
Multi-versioning Dissimilarity is
properly provided
CLAIM2.3.3
Safety Monitoring is provided
ARGUMENT 2.3Partitioning, multi-
versioning, and safety monitoring are necessary and sufficient for a safe
architecture
Customer specifications
(a priori)
Supplier evidence
(a posteriori)Works remarkably like
a V-Chart, doesn’t it?
Copyright © 2004 The MITRE Corporation22