Air Force Systems Engineering Assessment Model (AF … · Air Force Systems Engineering Assessment...
Transcript of Air Force Systems Engineering Assessment Model (AF … · Air Force Systems Engineering Assessment...
1
Air Force
Systems Engineering Assessment Model
(AF SEAM)
Randy BullardAF Center for Systems Engineering
2
Outline
1. Background
• AF SEAM Pedigree
• AF SEAM Goals
2. Model Contents (What is Included)
• Process Areas (PAs)
• Practices (Specific)
• Practices (Generic)
• References (What)
• Other Information/Elaboration
• Typical Work Products
• Methodology
3
AF SEAM Background
• In 2006, AFMC Engineering Council Action Item to:
• Provide an AF-wide SE Assessment Model
• Involve AF Centers (product and logistics)
• Leverage current CMMI®-based models in use at AF Centers
• Baseline Process capability & usage
• Definition of AF Systems Engineering Assessment
Model:
• A single AF-wide tool which can be used for the
assessment and improvement of systems engineering
processes in a program/project.
4
AF SEAM Goals
• Goals
• Ensure a Consistent Understanding of SE
• Ensure Core SE Processes are in Place and
Being Practiced
• Document repeatable SE “Best Practices”
across AF
• Identify Opportunities for Continuous
Improvement
• Clarify Roles and Responsibilities
• Improve Program Performance & Reduce
Technical Risk
5
Why We Need SE Assessment
• Lack of Disciplined System Engineering (SE) has been a major contributor to poor program performance
• Many Problems Have Surfaced Repeatedly with AF Programs
• Missed or Poorly Validated Requirements
• Poor Planning Fundamentals
• Lack of Integrated Risk Management
• Lack of Rigorous Process
• Lack of Process Flow Down
• Restoring SE Discipline in AF Programs Is Key to Improved Performance and Credibility
6
Benefits
• Restoring Disciplined SE
• Clear Definition of Expectations
• Well Aligned with Policy
• Established Assessment Methods & Tools
• Best Practices Baseline
• Driving Improvement
• Moving towards
• Deeper Understanding of SE Processes
• More Efficient Programs
7
Why AF SEAM
• AF SEAM is a composite of Industry & DoD SE best practices• Maps to CMMI -ACQ 1.2 & -DEV 1.2
• Consistent w/ Industry and DoD guidance
• Advantages to using AF SEAM• Streamlining of CMMI process areas to AF programs
• AF-centric w/ end-to-end life cycle coverage
• More focused document requires less program overhead
• Does not require SEI certified assessors
• Impact to AF programs
• Assure programs are achieving desired outcomes
• Ensure program teams have adequate resources
• Qualified People, Process Discipline, Tools/Technology
8
AF SEAM Pedigree
• All AF product Centers selected and tailored some version of the Software Engineering Institute (SEI) Capability Maturity Model Integration (CMMI®) to baseline process institutionalization
• SEI CMMI® is the Defense Industry-wide accepted method for process appraisal and improvement
• The SEI CMMI® incorporates principles and practices from recognized industry and US Government system engineering and related standards such as:
• AFI 63-1201 Life Cycle Systems Engineering
• Defense Acquisition Guidebook, Chapter 4
• MIL-STD 499B System Engineering
• ANSI/EIA 632 Processes for Engineering a System
• IEEE/EIA 731 Systems Engineering Capability Model
• ISO/IEEE 15288 Systems Engineering-System Life Cycle Processes
• INCOSE System Engineering Standard
• IEEE 1220 Application and Management of the Systems Engineering Process
9
AF SEAM Content
• Process Areas (PAs)
• Goals
• Practices
• Informative Material
• Description
• Typical Work Products
• Reference Material
• Other Considerations
Level of
Specificity
Broadest
Most Detailed
10
Principles & Objectives
Tools &
Technology
the Means
to Execute
Process &
Procedures
the Glue that
Holds it Together
People
with Skills,
Training & Motivation
Baseline
Practice of
Systems
Engineering
Kaizen or
Continuous
Improvement
Best Practices
from
Government
& Industry
Lean
Assessment of
Integrated
Team
Continuous
Process
Improvement
11
Why Focus on Process?
"If you can't describe what you
are doing as a process, you
don't know what you are
doing."
- W. Edwards Deming
12
AF SEAM Elements
• 10 Process Areas (PAs)– Based in CMMI process area construct– Conforms with AFI 63-1201 & DAG Chapter 4
• 34 Goals - Are Accomplished through the Specific Practices• 120 Specific Practices • 7 Generic Practices (Apply to each Process Area)
Process Areas (PAs)
• Configuration Mgmt (CM)
• Decision Analysis (DA)
• Design (D)
• Manufacturing (M)
• Project Planning (PP)
• Requirements (R)
• Risk Mgmt (RM)
• Sustainment (S)
• Tech Mgmt & Ctrl (TMC)
• Verification &Validation (V)
13
AF SEAM Practices
• Specific Practices – Each one applies to only
one Process Area
• Each Practice has Informative Material
• Description
• References
• Typical Work Products
• Other Considerations
• Generic Practices
• Must be accomplished for each Process Area
• Ensures specific practices are executed
• Involves stakeholders
14
AF SEAM Practices
Process Area Goals
Specific
Practices
Generic
Practices
Total
Practices
Configuration Mgmt 3 8 7 15
Decision Analysis 1 5 7 12
Design 3 14 7 21
Manufacturing 4 12 7 19
Project Planning 3 15 7 22
Requirements 4 13 7 20
Risk Mgmt 3 7 7 14
Sustainment 4 15 7 22
Tech Mgmt & Control 4 15 7 22
V & V 5 16 7 23
Total 34 120 70 190
15
Sample Specific Practice
• RMG1P1 Determine risk sources and categories• Description: Establish categories of risks and risk sources for the project
initially and refine the risk structure over time (e.g., schedule, cost, supplier
execution, technology readiness, manufacturing readiness, product safety, and
issues outside control of team), using Integrated Product Teams. Quantify the
risk probability and consequence in terms of cost and schedule.
• Typical Work Products:
• Risk matrix
• Risk management plan
• Reference Material: USAF Operational Risk Management, AFI 90-901
• Other Considerations: Consider using Acquisition Center of Excellence Risk
Management Workshops when needed. For manufacturing risks consider the
capability of planned production processes to meet anticipated design
tolerances. Include the supplier’s capacity and capabilities in the analysis.
Generic Practices
1. Establish and maintain the description of a defined process
2. Establish and maintain plans for performing the process
3. Provide adequate resources for performing the process,
developing the work products, and providing the services of
the process
4. Assign responsibility and authority for performing the
process, developing the work products, and providing the
services of the process
5. Train the people performing or supporting the processes
needed
6. Monitor and control the process against the process plan
and take appropriate corrective action
7. Review the activities, status, and results of the process with
higher level management and resolve issues
16
Process Detail Outline
17
STARTSELF
ASSESSMENT
VALIDAT
ION
REQUIR
ED?
CONDUCT
VALIDATION
POST
RESULTS
NO
YES
A B C D
E
A – B• Roles/Responsibilities
• Training
- Leadership
- Self Assessment
• Leadership
identifies “area(s)”
of self assessment
• Describes self
assessment activity
• What needs to be
accomplished
• Capture data
• Presentation of
results
• In brief
• Conduct interviews
• Analysis
• Presentation of
results
C – D• Build Team
• Train team
• Logistics support
• Set schedule
Feedback
18
Criteria for Methodology
• Facilitate Self Assessment
• Facilitate Continuous Improvement
• Provide insight into Program/Project Processes & Capability
• Objective Assessment
• Consistent Near and Far Term Approach
• Provide Results that are meaningful for leadership• Relevant to PM/PEO/CC
• Simple
• Understandable
• Graphical
• Support Multi-level Measurement & Reporting• Program/Project, Squadron, Group, Wing, Center
• Resource Allocation
• SE Process Improvement
19
Defining the Methodology
• Hands Off
• Promulgate Policy
• Directives
• Instructions
• Checklists
• Guidance
• Expect Compliance
• Hands On
• Comprehensive Continuous Process Improvement
• Highly Detailed Process Bibles
• Training
• Validation Assessment
• Deep Dives
Assessment Methods that Balance Time and Effectiveness
• AF SEAM• Collaborative
& inclusive
• Leanest Possible Best Practices “Must Dos”
• Clearly Stated Expectations
• Program Team & Assessor Team
• Training
• Self Assessment of Program with Validation Assessment
Assessment ContinuumLow High
SE Assessment Activities
Phase I
Planning
• Read Ahead Package
• Logistics Planning
• Training
Phase II
Self-Assessment
• Self Assessment
Training
• Project performs
self-assessment
• Provide self -
assessment to review
team
Phase IV
Report Results
• Consolidate
Results
• Prepare final
report / outbrief
• Deliver Final
Results
• Team In-Brief
• Project Brief
• Review Self-
Assessment
• Collaborative
Interviews
• Document
Reviews
20
Phase III
Independent
Validation
21
Assessment Outputs
• Feedback
• Lessons learned from assessment tool
• Collaborative review
• Findings
• Completed assessment tool
• Strengths
• Improvement opportunities
• Output metrics
• Recommendations
• Final outbrief
PA/GP GP1 GP2 GP3 GP4 GP5 GP6 GP7 GP Overall
CM 1 1 1 1 1 1 1 7
DA 0 0 1 1 1 1 1 5
D 1 1 1 1 1 1 1 7
M 1 1 1 1 1 1 1 7
PP 0 1 1 0 1 0 1 4
R 1 1 1 1 1 1 1 7
RM 1 1 1 1 1 1 0 6
S 1 1 1 1 1 1 1 7
TMC 0 1 1 1 1 1 1 6
V 1 1 1 1 1 1 1 7
Generic Practices Summary
24
Summary
• Goal is to Continue to Improve Program
Performance
• Too many examples of program performance/issues
being tracked back to lack of SE discipline
• Long Term Goal – Revitalize & Institutionalize
Systems Engineering
• Use SE “Best Practices”
• Assist programs in achieving desired outcomes
• Assist program teams in resource planning
• Qualified People
• Disciplined Processes
• Tools/Technology
Team Members
Center Members
AAC Ian Talbot
AEDC Neil Peery, Maj Mark Jenks
ASC Gary Bailey
AF CSE Rich Freeman, Randy Bullard
HQ AFMC Caroline Buckey
ESC Bob Swarz, Bruce Allgood
OC-ALC Cal Underwood, Bill Raphael
OO-ALC Jim Belford, Mahnaz Maung
SMC Linda Taylor
WR-ALC Jim Jeter, Ronnie Rogers
Spiral 2
• Capability Enhancement• Re-look process areas for improvements
• Further refine assessment methodology
• Strengthen inclusion of software
• Capture and promulgate best practices/lessons learned
• Review scoring
• Examine potential use for SE health assessment
• Migrate to web-based platform
• Resources• Funding
• People
• Computer Based Training
• Schedule• Estimated 1-year effort
• One member from each Center
• Working Group meetings held approximately bi-monthly
• Lead POC/Steering Group• Staff support
• Community of Interest
• Model sustainment (continuous improvement)
Scoring Roll-Up
0
1
2
3
4
5
6
7
CM DA D M PP R RM S TMC V
Nu
mb
er
of
Pro
gra
ms
Process Area
Specific Practice Assessment ResultsXXX Center
Scoring Roll-Up
0
1
2
3
4
5
6
7
GP1 GP2 GP3 GP4 GP5 GP6 GP7
Nu
mb
er
of
Pro
gra
ms
Practice Area
Generic Practice Assessment ResultsXXX Center
Implementation By CenterCENTER 5 AUG 08 - FEEDBACK
AAC
"AAC began integrating AF SEAM in our established program assessment
process in January 2008 and expects to complete this integration in FY09."
AEDC "We will begin implementing AF SEAM in October."
ASC
"We are creating a plan to migrate from our current tool to SEAM, tailored
with AFMC and ASC specific areas of interest."
ESC
"We have initiated tailoring efforts to implement AF SEAM by the end of the
calendar year. We will be working closely with SMC, our acquisition partner,
on the tailoring and implementation effort."
OC-ALC "Strongly support, have plans in place, ready to go!"
OO-ALC "We are implementing now."
SMC "SMC plans to adopt AF SEAM and comply with related policies."
WR-ALC
"We'll begin implementation at Robins with pilot assessments in F-15 and
Avionics."
Development process yielded 100% buy-in