Technological infrastructural needs to support third party certification

Certification of Safety-Critical Software-Intensive SystemsFirst Public Workshop

November 11, 2011

Sushil BirlaDivision of Engineering

Office of Nuclear Regulatory Research(301-251-7660, Sushil.Birla@nrc.gov)

2

Background & source of vision

Context: U.S. Govt. Inter-agency coordination activities

– NITRD (Networking & IT R&D)

• HCSS (High Confidence Software & Systems)

– Cyber-physical systems

» Focus area: Safety critical systems

SectorsHealthEnergyDefenseTransportationNational Security

Commonalities

3

Current state – some commonalities

• Safety-critical CPSs are typically too complex to be completely verified and validated. Remaining uncertainties are significant, but not well understood.

• Safety analysis and evaluation require high competence and judgment, but these capabilities are very scarce.

• Cyber adversaries’ ability to develop and launch new attack tools and techniques outpaces the ability to develop and deploy countermeasures.

• The competencecomplexity gap is widening rapidly.

• Similar problems exist in most safety-critical, mission-critical application domains, but there is little synergy to find a common core set of underlying solution capabilities.

• The requisite knowledge is not well-systematized

• Commercially available tools, driven by non-critical consumer applications, are being used in critical applications, but their commensurate verification Is not feasible economically.

4

Current state: Some complexity issues

• A single defect can make logic wrong, potentially leading to serious consequences, but the capability to engineer defect-free systems does not exist.

• Networking (wired or wireless) introduces new vulnerabilities that are not well understood

– Hidden dependencies and couplings• Latent defects could combine in many scenarios

• Latent defects could cause a high consequence failure

• The more complex a system the more exposure to defects

• Verification of a high-integrity system or component, e.g. operating system, takes more effort and time than its initial development.

5

Vision state: Some commonalities

• Systems can be routinely developed with built-in assurance of safety and security

– “Do it right the first time” becomes the cheapest and fastest way to realize a system

• Accredited third party services are commercially available for verification & validation (V&V)

• Accredited third party services are commercially available for review, attestation, and certification

• Requisite tools are certified• Requisite competence (knowledge, skills) is certified• Requisite competence becomes readily available• Requisite body of knowledge is mature and readily accessible• Educational and training institutions have mature curricula to

produce and certify the requisite competence

6

ISO 17000 definitions - 1

Third-party attestation related to products, processes, systems or persons

5.5 certification

Issue of a statement, based on a decision following review, that fulfillment of specified requirements has been demonstrated

5.2 attestation

Verification of the suitability, adequacy and effectiveness of selection and determination activities, and the results of these activities, with regard to fulfillment of specified requirements by an object of conformity assessment

5.1 review

7

ISO 17000 definitions - 2

Need or expectation that is stated. NOTE: Specified requirements may be stated in normative documents such as regulations....

3.1 specified requirement

Demonstration that specified requirements relating to a product, process, system, person or body are fulfilled

2.1 conformity assessment

A person or body that is independent of the person or organization that provides the object, and of user interests in that object

2.4 third party

8

ISO 17000 definitions - 3

Third-party attestation related to a conformity assessment body conveying formal demonstration of its competence to carry out specific conformity assessment tasks

5.6 accreditation

Body that performs conformity assessment services

2.5 conformity assessment body

Authoritative body that performs accreditationNOTE … authority … generally derived from government

2.6 accreditation body

9

Some expectations & gaps

Accreditation bodies

3rd party conformity assessment bodies

Competence criteriaFormally demonstrate competence

Enable certification of safety-critical software

10

Some more gaps

Regulatory requirementsare abstract

SW in safetysystem

Concretederived requirementsmissing/incomplete

Review

Interpret Regulatory guides

Standards

ExpertJudgmentneeded

Scarce!

11

Research needs identified

Questions posed to expert group

• What are sources of uncertainties?

• What evidence do we need to reduce these uncertainties?

• What are the areas that need more research?

Uncertainties even after best practices

12

ResidualUncertainties?

“Good” design practice

NRC’s regulatory guidance framework

Appendix A in RIL-1001

Focus of group

Assumeconformity

Uncertainties and resulting size of potential fault space

13

Some sources of uncertainties

• Validation of Requirements

• Architecture: Complexity

• Verification: Adequacy of coverage

• Impact of change: Hidden/obscure dependencies

• Transformation tools

• Integrating/Combining evidence

14

Current review practice

Perform thread audits of several requirements• Check for conformance clause-by-clause

Is clause-by-clause review enough?

15

Combined effects of deviations

Charles Perrow in “Normal Accidents- Living with High Risk technologies” 1984: – A major failure of a complex system is

typically caused by a combination of relatively small incidents: Three Mile Island

16

Combined effects in SW

• A single defect can make logic wrong

• Hidden dependencies and couplings

• Latent defects

– Could combine in many scenarios

– Could cause a high consequence failure-

– The more complex a system the more exposure to defects

17

Combined effects of seemingly insignificant deviations

High consequence failure of a complex system

Operators’ Action

Faulty Equipment

Incorrect indicator

Inadequate Procedures

Inadequate Design

18

Example of evidence gaps

Uncertainties cannot combine to produce

more complex uncertainties

Independence and decoupling

Compliance with architecture principles & constraints

Demonstrate

Demonstrate

Demonstrate

Inadequate criteria

19

Architecture: Complexity issues

New I&C architectures overly complex

1. High degree of connectivity between two systems which are suppose to be independent

2. Safety to non-safety interconnectivity

http://www.hse.gov.uk/newreactors/ri-ukepr-0002.pdf

20

Transformation tool issues

• New (unknown) ways of introducing defects

• Preservation of semantics

21

Tools

Integrating effect of uncertainties in software assurance

Reqmts Arch Reqmts Arch D IUnitTest

IntegrTest

FAT

Auto code gen

Auto test gen

Safety demonstration in the presence of

uncertainties

Change ImpactAnalysis

? ? ? ? ? ? ? ? ?

system software system

V&V results

?Each anomaly or uncertainty by itself seems to be small

Coverage evidence (Diverse complementary)

22

V&V: Adequacy of coverage

Environment• Assumptions• Input validity

Requirements• Correct?• Complete?• Consistent?

Incompletecoverage

Interference

Analysis

Model checking

Testing- Coverage based

…

Proof of non-interference

Some major sources of uncertainties

Safety Demonstration(e.g. assurance case)

Evidence about other uncertainties

Safety demonstration: Adaptation of Toulmin’s model

23

Backing, e.g., theoretical or causal model

Inference rule

Evidence/ Grounds Assertion/

Belief/Claim

Factors influencing validity of argument

basis for

Qualifiers (Strength; Condition)

Challenges; rebuttals; inconsistencies

Argument

used in

affects

24

Recap: NRC areas of interest

Certification infrastructure needed• Accreditation bodies• Competence criteria• 3rd party conformity assessment bodies

Some gaps in assurance technology infrastructure• Validation of Requirements• Architecture: Complexity • Verification: Adequacy of coverage• Impact of change: Hidden/obscure dependencies• Transformation tools • Integrating/Combining evidence