Using Qualitative & Quantitative Techniques to Improve Service Delivery

Joanna PattersonCACI, Operational ExcellenceQuality Manager

Maggie GloverExcellence in MeasurementHigh Maturity Lead Appraiser

Presentation Information

This presentation was presented at the 2015 CMMI Institute Global Congress in Seattle. The presentation is from two perspectives.

The first is that of the CACI Quality Manager who led an effort to improve software code security using qualitative and quantitative techniques.

The second is that of the high maturity lead appraiser who facilitated the CACI CMMI for Development Maturity Level 5 appraisal related to this effort.

Joanna Patterson

Joanna Patterson is a Quality Manager for CACI International. Joanna holds a BS in Adult Education, and an M.B.A. with a minor in Information Security Management. In addition, Ms. Patterson is in the process of completing her Doctorate in Information Technology Management. Ms. Patterson is a recognized Golden Key Honor Society scholar and completed her doctorate course work with a 4.0 GPA. Ms. Patterson has over 15 years of relevant industry experience and has worked on deploying emerging technologies or large scale network efforts.

CACI

Key statistics about CACI International:• Founded in 1962• Over 16,000 employees worldwide• CACI provides information solutions and services in

support of national security missions and government transformation for Intelligence, Defense, and Federal Civilian customers.

Commitment to Quality

Long standing commitment to quality• Deploy standards as they are needed

• ISO 9001 – Shipyards, Help Desks, SD• ISO 20000 – Help Desks, Network Support• ISO 27001 – Help Desks, Medical, PII, HIPPA• ISO 28000 – Logistics, Supply Chain• CMMI for Development ML 3 & 5 – Solutions Development• CMMI for Services ML 5 – Help Desks, Medical

Defining IA Activities

• Information Assurance (IA), for this presentation, is defined as the activities related to securing the code developed by CACI for the government.

• The project, specifically the developers, are required to develop secure code

The Challenge

CACI recognized that information security was, and is, a risk that should be proactively address.• Information Security / Cyber-Attacks• Hackers becoming more

sophisticated • Increasing government

oversight/regulations

The Problem

• It is commonplace to measure the number of defects found during the software development lifecycle in hopes of reducing them.

• Information assurance “vulnerabilities” are considered defects

• An IA defect is a vulnerability which can lead to an exploit

The Problem

• IA defects are most often found when the government Information Assurance Manager runs a static code scan

• The scan finds potential vulnerabilities based on the Software Technical Implementation Guide (STIG)

• Example defects: SQL Injection Error, Cross-Site Request Forgery, Hidden Field, Empty Catch Block

Further Explanation

What can you do with a SQL Injection vulnerability?

SQL injection is a code injection technique, used to attack data-driven applications, in which malicious SQL statements are inserted into an entry field for execution (e.g. to dump the database contents to the attacker).

Real Headlines!“Up to 100K Archos customers compromised by SQL Injection attack”“Hackers Stole 100K from California ISP using SQL Injection”“Hackers discloses vulnerabilities in dozens of Military & Pentagon websites attributed to SQL Injection vulnerability”

Doors to Destruction

Government scans found, on average, 3000 errors

Every defect introduced is a potential door for a hacker

This is War

Every battle is won before it is fought.

Sun Tzu

Know your weaknessKnow your EnemyCreate a battle planChange - Adapt

Qualitative Research

What is Qualitative Research?• Exploring issues• Enhances the reliability of the statistical data• Increases your knowledge on the topic• Understand behavior• Descriptive versus predictive

Qualitative Research

Why did CACI use qualitative research methods?• Constantly changing IA climate• Need to determine all the variables involved• Immense influence from outside factors

Qualitative Research

What is Qualitative Research?• Exploring issues• Enhances the reliability of the statistical data• Increases your knowledge on the topic• Understand behavior• Descriptive versus predictive

Qualitative Research

The primary misconception regarding high maturity efforts is that you can baseline, measure, and build predictive models without quantifying or qualifying the problem.

Yes there are IA defects…

Why?

Yes the developers are involved..

How?

Yes the customer has requirements…

What?

Understand your problem before you try and fix it

Qualitative Research

Do your research and research includes talking to your people:

To find out the true origins of this problem CACI:

• Interviewed the developers

• Surveyed the customer

• Reviewed industry IA literature

Data Collection

• Keep it:• Standardized • Open-ended• Let the people on the front line be heard

• Code the data• Open Coding – conceptualize the data• Axial Coding – put it back together, make connections• Selective Coding – look at the core variables

The following slides give a high level overview of coding techniques used and should not be considered all inclusive.

Interview Question / Open Coding

Question: What is your current IA scanning procedure?

Open Code Actual participant words

Inconsistent procedures The way I do it…The way the government does it…Need centralized processNeed communication from governmentNo training

Government ScansCACI does not ScanRetina scansGold Disk

The government scans during validationWe perform Retina scansWe do gold disk scans

Lack of SoftwareOutdated Software

We do not own HP FortifyFortify is expensiveOur version is outdated

STIG requirements What is a STIGNot synced with current STIG template

Data Concepts & Categories

I. As data is collected and coded, codes of similar content appear and allow the data to be grouped into concepts.

II. As concepts emerge, they are grouped to form categories that are then used to generate a theory.

III. As QA collected data and coded it, several categories of concepts were identified.

Codes

Concepts

Categories

Concepts & Categories

Open Code Concepts CategoriesInconsistent procedures IA processes inconsistent

Government ScansCACI does not ScanGovernment

Heavy reliance on government intervention

Retina scansLack of SoftwareOutdated SoftwareDevelopers

Inconsistent resource utilization

Gold DiskSTIG requirementsNot familiarNeed trainingDesire more immediate feedbackValidity of existing errors

Inconsistent and subpar training

STIG requirementsOn loan IA personNo IA SupportNo external or internal resources

Improper planning or IA defect mitigation

Axial Coding Results

The end result are the areas CACI focused on fixing

Time to Quantify the data

CACI now moved on to creating baselines in order to quantitatively manage the IA process areas. Additionally, there were simultaneous efforts to correct issues in areas that were not quantifiable (in this instance). For instance, training, or requirements.

Quantitative

Created a baseline of IA defects by category

Initial Baseline:

Mean of 437 CAT I errors

Range:

Over 1000 errors

What this means?

Wide variation in the data with no outliers means the process is in statistical control but overall

performance of the process is not yet predictable.

Note: Just because the process was in statistical control, doesn’t necessarily mean it was cost efficient or time efficient for resource planning.

IA Cost Baseline

Created a IA cost baseline

Average of $27,043 IA cost per month

IA costs incurred to correct CAT I or CAT II defects initially only consisted of the cost of the developer to correct the code.

Made Improvements

Based on the root causes identified during the qualitative analysis, we identified where opportunities for improvement existed in the IA process.

• These are identified as the “pain points” of the process.

• These pain points were not specific to one project.

• There were multiple root causes therefore multiple pain points in the process = multiple changes

Decreased Errors

CAT I errors before and after process improvement (mean decrease from 437 to 57)

Mean: 57

Range:

264

What this means?

The decrease in range and mean indicate the process is not only

stable but is now in a predictable state. Projects can anticipate the number of defects that will need

to be remediated per sprint.

Decreased Cost

Average monthly cost decreased from $27,043 to $16,591; range decreased from

$50,000 to $10,000

Predictive Models

Model Type Predictor When to use?

Linear Regression Cost to fix CAT I defects

No dedicated IA story or hours during sprint; after adhoc scan performed by IA Analyst or Government; PM wants to determine ability of project to meet ORG objective

Multiple Regression Cost to fix CAT I and CAT II defects

No dedicated IA story or hours during sprint; after scan is performed by IA Analyst or Government; PM wants to determine ability of project to meet ORG objective

Linear Regression Hours to fix CAT I defects

After adhoc scan performed by IA Analyst or Government

The Results!

Higher Customer Satisfaction

- Delivery of Secure Code

- On Time Releases

- Decreased IA Costs

IA Cost and Predictability

- Answer “How Long” & “How Much”

- Proactive vs Reactive

Immediate Adoption of IA Process

- Easy to Adopt and Implement

- Minimal interference with project management activities

Through the eyes of a CMMI High Maturity Lead Appraiser, Six Sigma Black belt and Scaled Agile Framework (SAFe)

Product Consultant

Ms. Glover is a High Maturity Lead Appraiser for CMMI-Services, Development and Acquisition. She also is an Intro to CMMI Instructor certified by the CMMI Institute.

Ms. Glover is a former Air Force Captain who served as a Satellite Office in SPACECMD in Cheyenne Mt. CO. She has a graduate degree in IT Systems. She is also and ISO 9000, 20000 and 27000 Lead Auditor as well as a Six Sigma Black belt.

Ms. Glover is currently working at the Cigna Agile Center of Excellence as a Scaled Agile Framework (SAFe) Product Manager. She is currently working on developing their SDLC for their Agile Development Lifecycle.

Margaret Tanner Glover

How CMMI HM Helped

• Its all about performance!

• Maturity Level 5 is built on the understanding of the quantitative measures that are defined in order to lead to process improvement objectives which enable the organization to better meet their business objectives and associated quality and process performance objectives.

Remember The Problem?

• It is commonplace to measure the number of defects found during the software development lifecycle in hopes of reducing them.

• Information assurance “vulnerabilities” are considered defects

• An IA defect is a vulnerability which can lead to an exploit

Defining Quantitative Objectives

WHAT DO YOUR CUSTOMERS WANT?• How do you measure and model your processes to that end

result of customer satisfaction?

We used some Six Sigma objectives:• The D in DMAIC is Define.• Define Voice of the Customer. What does the customer want?• The customer wants defect “free” code as to not have any

vulnerability for exploitation and unethical hacking of their code • In Agile Development, the Product Owner is responsible for

getting the User Stories from the customer to determine the requirements of the system.

Six Sigma for Voice of the Customer

WHAT DO YOUR CUSTOMERS WANT?

Determine Quantitative Objectives

Once you have determined the Voice of the Customer, then:

What is the process capable of (Determine Voice of the Process)?

What can be controlled (Controllable Factors)?• CACI could institute the use of software development tools

for security• CACI could institute good software security best practices

.

Process Stability and Process Capability

Determine Quantitative Objectives

Defect free code is the goal. Is it a controllable factor?• Controllable factor is something in your process that you have

control over. Examples include:• Training the organization provides • Coding standards the organization institutes• Tools that required for estimating• Encryption and firewalls• Software Defined Lifecycle (Agile)

• Non controllable factors is something that you do not have control over. Examples include:

• Weather• Employees sick time• Hackers trying to break through your firewall

Determine Quantitative Objectives

Defect free code is the goal. Is it a controllable factor?• Maturity Level 4/ calls for the determination of “Controllable

Factors” which aid in the implementation of Quantitative Project Management (QPM) especially the following practices:

• QPM SP 1.4 Select measures and analytical techniques to be used in quantitative management.

• QPM SP 2.3 Perform root cause analysis of selected issues to address deficiencies in achieving the work’s quality and process performance objectives.

.

Determine Quantitative Objectives

• CMMI helped define the Quantitative Process Performance Objectives which traced to the VOC and hence CACI’s business objectives (OPP).

• Can you then satisfy your customer with the current process?

• One more requirement:• A Stable process has to be in place which can be

measured by the institutionalization of ML3 practices.• Before the process can be “Capable” it has to be “Stable”.

Determine Quantitative Objectives

Determine Quantitative Objectives

• One more requirement:• A process cannot be released to production until it has

been proven to be stable. • We cannot begin to talk about process capability until we

have demonstrated stability in our process. • A process is said to be stable when all of the response

parameters that we use to measure the process have both constant means and constant variances over time, and also have a constant distribution.

• This is equivalent to our earlier definition of controlled variation or a Stable Process.

Quantitative Management for ML4

44

Process Area: OPP

The IA Process was stable and predictable

Now What?

First – CACI listened to the “voice of the customer”. CACI created models based on project management feedback and

customer input

Second – CACI listened to the “voice of management”. Models were created to predict the projects ability to meet the org measure

Determine Quantitative Business Objectives

Business Objectives at CACI are to increase customer satisfaction• Maturity Level 5 required the organization to proactively

manage the organization's performance to meet its business objectives.

• Using Organizational Performance Management (OPP)• OPP SP 1.1 Maintain business objectives based on an

understanding of business strategies and actual performance results.

.

Traceability

46

Business Objectiv

e

Increase business for the division by achieving a X% success rate on re-competes, and X% success rate on new bids by June 30, 2014 (for all

performance based awards).

Sub Goals Exceed Customer Expectations Maintain a competitive advantage through

quality and process improvementReduce physical and information system

weaknesses

CMMI requires the linkage of the business objectives to the QPPO(Goal provided is an example for the purposes of confidentiality)

Process Area: OPP

How CMMI HM Helped

• When CACI determined the quantitative objectives of the IA efforts, they were able to determine the baselines and models and ensure the business objectives were being satisfied.

• CMMI Requirements for ML5 in Causal Analysis and Resolution (CAR) was used to determine where to implement the selected action proposals and evaluate the effect of those implement actions.

• Example: “using the root causes identified during qualitative analysis, we identified where opportunities for improvement existed in the IA process”

How CMMI HM Helped

• The requirements at ML5, asking an organization to use their business performance and manage it using statistical techniques which lead to identify potential areas for improvement that could contribute to meeting those objectives were realized when the physical and information system weaknesses were found and eliminated.

• This led to a decrease in dollars saved, allowing CACI to meet their business objective of “Exceeding Customer Expectations”.

49OPM

Questions?

You have questions?

We have answers!