Michael S. McCalla

An Outcome Measurement Model:

Is your Agile Adoption Moving the Needle?

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

National Champs

Title Town!

Michael S. McCalla CSP, PSM II, SA, PMI-ACP

• Enterprise Agile Coach

• 10 Years of Experience Assisting Organizations with Agile Transformation

• Broad Range of Experience: Coaching on Team, Program, Portfolio, and Enterprise Levels

• Activist in the Agile Community:

• Named to Reviewer Panel of the Agile Practice Guide, a collaborative effort between Agile

Alliance & PMI

• One of the original board members of the Agile Uprising, online community dedicated to

advancing the Agile Mindset

• Other community involvement includes Social Media Enthusiast, Author, Agile Philly

Coordinator/Mentor

• Data Nerd

• Problem Solving and Product Development is My Passion, Coaching is a Close Second

• President of Lean Agile Intelligence, an Agile Assessment, Improvement, and Measurement

Platform Assisting Organizations with Agile Adoption given their Desired Outcomes

The Evolution: How Did We Get Here?

A Positive Upbringing

Are you saying………..

Confident, Ambitious, Passionate………..

July 15, 2008

“Embarking on a Journey”

Fundamentals

Source: Ahmed-Sidky

Coaching Ineffectiveness

“That is not Agile”

“The Agile Principle

Says……”

“Lets Check the

Scrum Guide”

“You need to develop more

of an Agile Mindset” “Let them Self-Organize”

“Kanban vs. Scrum”REPEATED

FAILURE!

The Daily Bashing

What I Wasn’t Considering

Enterprise CultureCommand & Control

Leadership

Continuously Learning from the Failure

The Epiphany

The Lean Startup

“A true experiment

follows the scientific

method. It begins

with a clear

hypothesis.“

“Focuses on

working

backwards from

desired business

results instead of

forward from the

technology stack

Validated Learning

Fundamentals

Source: Ahmed-Sidky

Taking It Further

Taking It Further

Based of Context

Traditional Approach

Based of Context

I Wasn’t Alone

Source: Version One State of Agile 2016 Survey

A Different Approach

Based of Context

Changing the Conversation…….

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Outcome Driven Improvement & Measurement Model

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Outcome Driven Improvement & Measurement Model

Enterprise

Unit

Program

Team

Self

For Every Level of Today’s Agile Organization……

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Outcome Driven Improvement & Measurement Model

Enterprise

Unit

Program

Team

Self

For Every Level of Today’s Agile Organization……

Case Study:

The Payments unit of a Credit Card Company has ten cross functional teams just

adopting scrum. They are responsible for all payments applications in the

organization. Six months ago, they lost a number of key senior developers to an early

retirement package. They were immediately replaced by developers fresh out of

college. Since then there has been several priority one bugs which has led to missed

payments and manual intervention. The bugs were created as a result of new

enhancements introduced, and there new feature backlog is massive.

Management is starting to question the hires. The teams have been vocal that the

products lack documentation and are very fragile. They are unaware of the impacts

of their changes both upstream and downstream. QA has advised that the bugs

found were edge cases, and that is why they were not caught in manual testing.

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

• What outcome are we looking to achieve given

our context?

- Improved Reliability

• Do we have clear priorities and does it align to the

organization's goals for the year?

- Less Interruptions to the Customer

• What do we need to focus on to help us get

there?

- High Quality

• What are some of the team challenges?

- Lack of Experience and Documentation

• How are we going to measure that outcome?

- Escaped Defects

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

• What practices will help us achieve our desired outcomes?

1. Pair Programming

2. Continuous Integration Pipeline

3. Automated Tested

4. Slack

5. Agile Testing

• Who is best to tell us how the team is performing in these practices?

1. The Team!

• How are we performing in the practices that will lead to better

outcomes?

1. No Pairing

2. CI Pipeline only executed unit tests

3. Integration & Acceptance Tests were only partially automated

4. 100% utilized every sprint

5. Developers only responsible for development not testing

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Organization

Unit

Program

Team

Self

• What short term tactical goals can we set to help achieve

our outcome?

1. Identify Pairs, and pair 50% of the Time

2. Increase Number of Integration Tests and Automate

3. Add Integration Tests to CI Pipeline

• Do we have any impediments that are out of control

preventing us from improving in the practice?

1. Full Backlog of Features for Company wide projects

• What metrics can we use to measure our progress?

1. Leading: # Integration Tests, Integration Tests

Automated %, WIP, Acceptance Tests Automated %, Pair %

2. Lagging: Escaped Defects & Throughput

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

• Did we meet our tactical goals and do what we said we

were going to do?

- Look at Leading Indicators

• Is it helping us move the needle?

- Look at Lagging Indicators

• Do we have organization constraints that are slowing us

down?

- This is an opportunity to gather insights and determine

new objectives for another cycle of improvement

• Do we need to continue improving in these practices, or

should we try some others?

- Has our hypothesis proved to be true?

Measure: Tracking Progress of Leading IndicatorsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Track leading indicators to see if the team is progressing towards the goal

Measure: Track Progress of Lagging IndicatorsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Track lagging indicators to see if the tactical improvement is leading to desired outcomes

Measure: Track Organization PatternsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Look at patterns across the organization to identify possible organization constraints

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

Outcome Driven Improvement & Measurement Model: Enterprise Level

Real Life Case Study

Just Do It Advisors is mid-tier (5000 people) asset management boutique that has

been in existence for 85 years. They pride themselves on establishing personal

relationships with their clients and utilizing an active money management approach.

This strategy has enabled them to charge higher fees. However, over the past 6

years, market share and revenue has declined significantly, because many of their

clients have become deceased and their children have chosen to go elsewhere.

Feedback has been competitors offer a comparable service at a cheaper price, and

provide more transparency of funds and reporting via online portals.

They have a traditional silo-ed organization structure in which the IT organization has

always been seen as a cost center who keeps the lights on, and serves multiple

business units. Recently, they have been asked to deliver new applications, but

feedback was the deliveries did not meet the need or it took to long. They have

experienced with Scrum, but it is the typical “Scrum-But”

Taking It Further

Frameworks & Practices

Tactical Goals: Determine Leading Indicators

• Team and groups should proactively pick the metrics that will help them track their progress

towards their desired outcomes so they can baseline and incrementally measure progress

• Identify Leading Indicators that will help measure if you are on track to achieve your tactical goal

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Time to Market Customer

Satisfaction

Employee

Satisfaction

Innovation Responsiveness Reliability Predictability

Throughput

(Velocity)

NPS Happiness Index # of New Products

Released

Product/ Project

Lead Time

Test Coverage Story Point

Attainment

Build Time Value Points Team Health

Assessment

Bug Fixing vs New

Product

Development %

Feature Lead

Time

Builds Per Day Velocity Variation

Deployment Lead

Time

Cost of Delay Code Complexity

New Environment

Automation

Failed

Deployments

Measure: Determine Lagging Indicators

• Team and groups should proactively pick the metrics that will help them track their progress

towards their desired outcomes so they can baseline and incrementally measure progress

• Identify Lagging Indicators to see if your tactical improvements are moving the needle

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Time to Market Customer

Satisfaction

Employee

Satisfaction

Innovation Responsiveness Reliability Predictability

Number of

Releases

ROI Attrition New Customer % New

Project/Product

Cycle Time

Escaped Defects

in Prod

On Time Delivery

Rate

Project/Product

Cycle Time

Revenue Yearly Surveys New Markets

Entered

New Feature

Cycle Time

Availability

Feature Cycle

Time

Profit Number of

Production

Deployments

# of Prod

Deployments

Takeaway

Start with Outcomes and Context…..

And Gain Alignment…..

Contact Me

Website: www.leanagileintelligence.com

Email: mmccalla@leanagileintelligence.com

Twitter: @MSMcCalla_1

LinkedIn: https://www.linkedin.com/in/michaelsmccalla/

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

• What is important to this organization right now?

Transitioning it’s Business Model to be more Digital

• What outcome are we looking to achieve?

Increased Customer Satisfaction with Innovative Products

• What are some of the team challenges?

Siloed Environment & Lack of Collaboration

• Do we have clear priorities?

New Products Trump Everything else

Enterprise

Unit

Program

Team

Self

• What practices will help us achieve our desired outcomes?

Practices that ensure we deliver the right things:

1. Stable Cross Functional Teams

2. Dedicated “True” Product Owners

3. Customer Engagement

4. Experiments and Prototyping

5. Lean Budgets

• How are we performing in the practices that will lead to

better outcomes?

Initially, really bad

• Who is best to tell us this information?

The Team; in this case Senior Leadership

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

• What short term tactical goals can we set to help achieve

our outcome?

1. Identify Product Owners from the Business

2. Get New Product Owners Training

3. Create 2 Cross Functional Teams for New Prod Dev

• Do we have any impediments that are out of control

preventing us from improving in the practice?

1. Product Owner Empowerment

• What metrics can we use to measure our progress?

1. Leading: # of Dependencies, % of Trained POs, New vs.

Existing Prod Dev %, Net Promoter Score (NPS)

2. Lagging: Retention, Referral, Revenue

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

• Did we meet our tactical goals and do what we said we

were going to do?

Look at Leading Indicators

• Is it helping us move the needle?

Look at Lagging Indicators

• Do we have organization constraints that are slowing

us down?

This is an opportunity to gather insights and determine

new objectives for another cycle of improvement

• Do we need to continue improving in these practices,

or should we try some others?

Has our hypothesis proved to be true?

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Enterprise

Unit

Program

Team

Self

Measure: Tracking Progress of Leading IndicatorsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Track leading indicators to see if the team is progressing towards the goal

Measure: Track Progress of Lagging IndicatorsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Track lagging indicators to see if the tactical improvement is leading to desired outcomes

Measure: Track Organization PatternsMeasure

Strategic Objectives

Self-Assess

Tactical Goals

Look at patterns across the organization to identify possible organization constraints

Contact Me

Website: www.leanagileintelligence.com

Email: mmccalla@leanagileintelligence.com

Twitter: @MSMcCalla_1

LinkedIn: https://www.linkedin.com/in/michaelsmccalla/

Appendix

Did we meet our tactical goals and do what we said?

Is it helping us achieve the desired outcome?

Do we have organization constraints slowing

us down?

Outcome Driven Improvement and Measurement Model Canvas

What is important to this team right now?

What outcome are we looking to achieve?

What are some of the teams' challenges?

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

What short term tactical goals can we set in

those practices to help us achieve our desired

outcomes?

Do we have any impediments out of our control

preventing us from achieving those tactical

goals?

What metrics, both leading and lagging,

can we use to measure our progress?

What Agile and Lean practices will help us

achieve those outcomes?

Where could we go to better understand the practices

and if we are doing them correctly?

Who is best to assess our performance on them?

Tactical Goals: Determine Leading Indicators

• Team and groups should proactively pick the metrics that will help them track their progress

towards their desired outcomes so they can baseline and incrementally measure progress

• Identify Leading Indicators that will help measure if you are on track to achieve your tactical goal

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Time to Market Customer

Satisfaction

Employee

Satisfaction

Innovation Responsiveness Reliability Predictability

Throughput

(Velocity)

NPS Happiness Index # of New Products

Released

Product/ Project

Lead Time

Test Coverage Story Point

Attainment

Build Time Value Points Team Health

Assessment

Bug Fixing vs New

Product

Development %

Feature Lead

Time

Builds Per Day Velocity Variation

Deployment Lead

Time

Cost of Delay Code Complexity

New Environment

Automation

Failed

Deployments

Measure: Determine Lagging Indicators

• Team and groups should proactively pick the metrics that will help them track their progress

towards their desired outcomes so they can baseline and incrementally measure progress

• Identify Lagging Indicators to see if your tactical improvements are moving the needle

MeasureStrategic

Objectives

Self-Assess

Tactical Goals

Time to Market Customer

Satisfaction

Employee

Satisfaction

Innovation Responsiveness Reliability Predictability

Number of

Releases

ROI Attrition New Customer % New

Project/Product

Cycle Time

Escaped Defects

in Prod

On Time Delivery

Rate

Project/Product

Cycle Time

Revenue Yearly Surveys New Markets

Entered

New Feature

Cycle Time

Availability

Feature Cycle

Time

Profit Number of

Production

Deployments

# of Prod

Deployments

Agile Assessments

They Are Not……….• Maturity Models• Audits• Absolute

They Are • Another vehicle to help teams identify improvement

opportunities• An opportunity to have rich conversation with teammates• An opportunity to learn

3.45Agile Team Score

Why Assessments?

• Easy and Tangible Improvement Model for Teams to Grasp • Self Sustaining – Something that can be done by themselves when coaches

aren’t around or consultants walk out the door

• Self-Assessments• Invitation to Improve!• Everyone has a voice, and stake in the game• Empowerment to self organize and get to the next level of team

performance

• Provides Holistic View of Agility Across the Enterprise • Aggregated Results all the way up to the Enterprise Level• Data Analytics to identify patterns preventing Business Agility such as

coaching needs, investment opportunities, and organizations constraints