Experiences with Third Party Qualification of Critical Software Presenter: David Tremaine, SWI.
20
Experiences with Third Party Qualification of Critical Software Presenter: David Tremaine, SWI
-
Upload
trevor-murphy -
Category
Documents
-
view
218 -
download
0
Transcript of Experiences with Third Party Qualification of Critical Software Presenter: David Tremaine, SWI.
Experiences with Third Party Qualification of Critical Software
Presenter: David Tremaine, SWI
Overview
• Evolution of Qualification (CANDU)
• Lessons / Challenges
• Future Directions
© SWI 2011
CANDU and Software
© SWI 2011
Analog Trip MetersAll Emergency Trip Parameters
CANDU and Software
© SWI 2011
Analog Panel MetersAll Plant InformationManual Control
Analog Trip MetersAll Emergency Trip Parameters
CANDU and Software
© SWI 2011
Engineering vs. Software EngineeringDesk checkingUnit TestingCommissioning tests
Analog Panel MetersAll Plant InformationManual Control
Analog Trip MetersAll Emergency Trip Parameters
DCCMonitoring and DisplayAutomatic Control
Reactor Regulating SystemBoiler Pressure ControlBoiler Level ControlHeat Transport System Pressure…
Evolution of Qualification
• First a crisis Darlington NGS• Digital Shutdown Systems
• Is the software is safe?
• Trial and Error: A convergence of approaches
© SWI 2011
Evolution of Qualification
• Next standardization• Categorization: Cat I, II and III
• System and software engineering standards
• Defense-In-Depth
• Category I highlights• Formal requirements• Active Requirements Review• Systematic design & code verification• Hazards analysis• Reliability qualification
© SWI 2011
Evolution of Qualification
• Then a focus on third-party software
© SWI 2011
COTS
MOTS
PE
Evolution of Qualification
• COG-95-179-I Guide for the Qualification of Software Products
• The Context: System Engineering
© SWI 2011
Engineering Team
ComponentSelection
• Form• Fit• Function• Safety• Category
Qualifier• Reliability• Maintainability• Reviewability• Safety*
• Implementation• System Integration• V&V• Commissioning
System Desig
nSystemSpec
Evolution of Qualification
• Qualification result one of:• Qualified for use in the application
• Not qualified
• Qualified with restrictions
• Qualified with project or operational obligations
© SWI 2011
Evolution of Qualification
• Qualification approach• Preponderance of evidence argument
• Derived using a combination of various methods
© SWI 2011
INDIRECT EVIDENCE1. Vendor QA Assessment2. Operating History3. Reference Sites4. Maintainability Review5. Certifications6. Maintenance Capability7. Anecdotal Evidence
DIRECT EVIDENCE8. Failure Modes Analysis9. Goodness of Design10. Safety Net Review11. Certifications12. Fault Tree Analysis13. Code Review14. Functional Testing15. Failure Modes Testing16. Reliability Qualification17. Proof of Correctness
Evolution of Qualification
• CSA N290.14-07• Addresses Category I
• Focus on software concerns• Hidden flaw
1. Recognized Program
2. Mature Product
3. Preponderance of Evidence
• Other common concerns
1. Security
2. Flooding
3. Modal Behaviour
4. …
© SWI 2011
Lessons Learned
• 200+ components
• Not an appreciable improvement in SQA• Even with IEC 61508 certification SQA not a factor
• Proven-in-use data & re-testing
• Configuration management / Six Sigma-Lean
• I&C Software is Surprisingly Good• Choosing “qualifiable” software + market forces
• I&C software tends to be relatively simple
• Commercial market scaling substantial burn-in
• Quality without strong SQA?© SWI 2011
Lessons Learned
• What is missing in SQA standards?• People and project structure
• Small teams
• Subject matter experts
• Technical leadership/mentorship
• Engineering culture (if not software-engineering)
• Serious about achievement / peer pressure
• Professional attitude
• Focus on software concerns (N290.14) helps• Points to important issues
© SWI 2011
Challenges
• Complexity is the big foe• The many obvious impacts …
• Understanding the system to sufficient depth
• Increased dormant code
• Vendor behaviour• Obtaining adequate cooperation for a specialty market
• Higher consolidation + higher specialization
• Earlier version retirement
© SWI 2011
Challenges
• The march of progress• Replacement of analog meters
• Multi-core processors and determinism
• Freeware / Shareware
• Qualification is not a rubber stamp!• Part of the engineering process
• All qualification “obligations” must be resolved
© SWI 2011
Future Directions
• McSCert-SWI Research• Tom Maibaum and Marc Bender
• Preparation for CSA N290.14-07 revision
• Other software concerns?• Various avenues pursued• Some joy with Peter Neumann’s List• Search continues
• Application of assurance cases to qualification• Looking at the rationale behind the qualification• Re-writing sample reports using Goal Structured Notation• Need an obligation box• Need some rules
© SWI 2011
OB: ………
Future Directions
• SDLC Utility Survey• What SQA process provide the most value?
• Internal software survey
• Highlights …
© SWI 2011
Active Requirements Review
Scenario BasedDesign Presentation
Desk Checking&
Peer ReviewUnit and StressTesting
Questions
© SWI 2011
