Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl

Operated by Los Alamos National Security, LLC for NNSA

U N C L A S S I F I E D

Adaptive Metrics Develop metrics that determine how well

we are adapting to our ever-changing environment.

Fitness Functions Identify dependencies and requirements

for optimum productivity around the Laboratory.

Measure the impact of a localized failure of one entity across the entire organization.

Balanced Score Card Review our program from a balanced

perspective. Provide metrics by which we can manage.

Optimizing Performance Management

Gina Fisk, LANL Senior Cyber Security [email protected]



Starting Point – Remove the Clutter

Remove metrics that we can’t use to manage our information security program.

How many customers called our help desk. How many connections were deflected by our firewall. How many times our network was scanned, etc.

Bin the remaining metrics into the BSC framework for a Phase I BSC.

Financial. Customer. Internal Processes. Learning and Growth.

2



Determine Impacts of Failure Conduct IT Impact Analysis

Determine the cost to an organization if various IT services failed for variable lengths of time.

Network, Email, local storage, etc.

Calculate Impact Rating for each IT Service. 1/n, where n is the average number of days until an

organization has lost 100% of productivity. Calculate the Daily Monetary Impact of the

Loss of that IT Service for an organization. Calculate the overall productivity cost for the

Laboratory as a whole based on that loss.

3



Focus Areas From IT Impact Analysis results, identify IT

Services with largest impacts to productivity. Loss of Accreditation of systems Loss of local network access Loss of Email Loss of Oracle Loss of Internet access

Goals that the CIO and CISO had set for the organization in the Strategy Map.

Develop metrics based on these focus areas and develop Phase II of the BSC.

4



Information Security Strategy Map

5

INTERNAL AND PROCESS PERSPECTIVE

LEARNING AND GROWTH PERSPECTIVE

LG1. Attract, develop, and retain highly skilled security

professionals

LG2. Develop risk-focused and customer-centric culture

LG3. Align employee training with strategic

initiatives

CUSTOMER PERSPECTIVE

Competency Contribution

IP1. Streamline compliance program to achieve 100% of

scheduled accreditationsIP2. Optimize operations to

reduce KTLO by 10% per BUIP3. Enhance performance through implementation and

management of service agreements

Achieve Operational Excellence

IP4. Mature IT governance processes and increase partner

participationIP5. Build a structured, transparent

and collaborative regulator relationship

IP6. Promote transparency and performance through holistic

metrics program

Create and Support Internal Programs and External Partners

IP7. Propose and deliver business-enabling information

security solutionsIP8. Mature IT risk program to drive security, portfolio, and

governance decisionsIP9. Enhance red network

monitoring and vulnerability management

Deliver Innovative Security Solutions

FINANCIAL PERSPECTIVE

Competitive Advantage Operational Excellence

C1. “Understand and consistently deliver

what I need”

C2. “Keep me out of security and

compliance trouble”

C3. “Establish a positive reputation

which will help me with my customers”

C4. “Become a trusted partner by helping me solve my challenging

problems”

F2. Maximize operational efficiency

F4. Facilitating acquisition of new business through best-in-class IT security execution

F3. Minimize IT enterprise risk

Maximize mission enablementby balancing risk and value (F1)



Balanced Score Card

Financial (F1-F4)

Security unit costs

On-time rate of accreditations

Enterprise risk rating

Business impact of incidents

Projects on-time/budget

Cyber PBI ratings

Lower unit costs 100% on time Maintain .3

rating <25hrs/Q <10% variance >95% green

Target

Initiative

Customer (C1-C4)

Communication ComplianceCustomer Support

Program InputTime per

accreditationCustomer

Satisfaction

>80% survey scores

>70% survey scores

>80% survey scores

>90% governance participation

>95% CA/avg times

>80% survey scores

Target

Initiative

Internal Processes (IP1-IP7)

AOE: Opex reduction

AOE: SLA performance

CSIPP: unplanned

work

DISS: AOP risk mapping

DISS: BP tied to risk

DISS: Red capabilities

>=2.5% Q/Q <10% variance <=3/Q >=80%>=30% key processes

Positive trend

Target

Initiative

Hits target. Initiative on track

Short of target. Initiative recoverable

Failed process. Initiative not recoverable

Target not defined. No initiative

Learning and Growth (LG1-LG3)

Training roadmap

Planned role rotations

Attrition reduction

Strategic training

X X

<10% schedule variance

>=1/QReduced

attrition rate

>50% training mapped to initiatives

X X

Target

Initiative

Note: BSC target performance scores are represented here for explanatory purposes only

6



Fitness Functions Fitness functions measure the

overall health of an organization by measuring not only performance, but also the performance of those organizations on which we are dependent to achieve our goals. If the performance of one of the dependencies fails, there are ramifications throughout the entire organization.

Using the fitness scores of dependent organizations, we can measure the impact of a localized failure of one entity across the entire organization, providing valuable measurements of the actual cost of security incidents, network outages, etc.

We can trend these scores to evaluate performance at various levels of the organization.

SystemAdministrati

on

SystemAdministrati

on

NetworkServicesNetworkServices

IdentityManagemen

t

IdentityManagemen

t

Scientific Computing

Core Services

Production Cycles

Production Cycles

Visualization Services

Visualization Services

Backups and Storage

Backups and Storage

Security Infrastructur

e

Science and Engineering

C&A Physical Infrastructure

Publications

ContractsPatents

Dep

en

den

cies

Relia

nt

Org

an

izati

on

s

7



Example Fitness Function Framework Fiscal Responsibility (weight: 20%)

Milestones and deliverables (quality, timeliness) Expenditures (percentage over budget)

Customer Productivity (weight: 15%) Services maximize productivity around organization (uptime, etc)

Customer Orientation (weight: 15%) Responsiveness to the customer (SLAs, etc)

Improving Security (weight: 15%) Progress made toward improving security against our current threat

environment (hardening tools, etc) Institutional Responsibilities (Weight: 20%)

PBI deliverables and reporting (quality, timeliness) CAP deliverables and reporting (quality, timeliness) Metrics reporting (quality, accuracy)

Goal-Based Initiatives (weight: 10%) Progress made against organizational goals.

8



Fitness Function Example

Fiscal Responsibility (weight: 20%) .89 Timeliness of deliverables and milestones: .83 % of projects +/- 10% of budget allocation: .95

Laboratory Productivity (weight: 15%) .98 Uptime of service: .98

Customer Orientation (weight: 15%) .89 Customer Satisfaction Rating .89

Improving Security (weight: 15%) ** .56** Progress made toward improving security against our current threat environment

(hardening tools, etc) .56 Institutional Responsibilities (Weight: 25%) .68

PBI deliverables and reporting (quality, timeliness) .90 CAP deliverables and reporting (quality, timeliness) .75 Metrics reporting (quality, accuracy) .40

Goal-Based Initiatives (weight: 10%) .98 Progress made against initiatives. .98

FITNESS SCORE: .806

9

Note: Fitness scores are represented here for explanatory purposes only



Fitness Score Trends Fitness scores allow us to watch for trends and to manage by our metrics.

See how major changes affect our performance from month to month. Change in Management Change of Platform Change of Vendor, etc.

10



Adaptive Metrics IT Impact Analysis provide us with costs of the

failures of IT Services. We have the data on our ever changing threat

environment. The fitness functions allow us include “moving

target” metrics, which change each month, to measure our performance against our current threat environment.

11



**Adaptive Metric Example

“Improving Security (weight: 15%)” Identify top threats for the month.

Phishing, Windows vulnerability, Oracle vulnerability. Calculate cost of failure of these services across the

organization per month. Email: $200K, Windows: $500K, Oracle: 800K Overall Budget: 10% in jeopardy

Review % of security effort we are placing on these areas ($$$ spent).

Email: 5%, Windows: 31%, Oracle: 20% Weight the fitness function by how responsive we are to

these areas. 56% of our budget is spent on our top threat areas.

12

Note: Threats and budgets represented here for explanatory purposes only



Managing by the Metrics

Our budget, metrics, and initiatives are actionable and directly tied to our goals.

Our use of the Balanced Score Card helps us ensure uniform management of our business.

Our use of the Fitness Functions help us trend our metrics effectively and monitor the major changes.

We can trend our components individually or as a whole, organizationally or institutionally.

Our use of Adaptive Metrics keep our outlook fresh and defendable.

13

Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl

Documents

Transcript of Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl