Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Leveraging cloud data to optimize your Agile processes Ronen Aseo, Product Manager Efrat Egozi-Levi, Senior Data Scientist, CTO office


HP Agile Manager defined

Agile project management

Release & User Story mgmt.

Defect management

ALM/PPM synchronization

ALI integrations

Key Capabilities

A SaaS solution for Agile

Instant-on service

Regular, incremental releases driven by user feedback

Web-based, intuitive UI

Complements HP ALM/QC

Key Characteristics


Ø  3x number of users within the last year Ø Monthly drops to production Ø Vast internal adoption throughout HP

Our Story

http://www.tonycooke.org/


HP Agile Manager Development Teams

Agile Manager Product characteristics •  3 applications

•  Teams are co-located

•  On premise & SaaS product

Products

ALM Synchronizer

Application Lifecycle Intelligence

Agile Manager

Sunnyvale, CA Product

Management

Prague, CZ Development

Yehud, IL Development

Ho-chi minh, Vi Development


AgM Delivery Model SaaS Production Farms (External)

Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6

SaaS Production Farm (Internal – early adapters)

Test Farm (trunk)

Nightly deployment

trunk

Deploy after two weeks on internal farm, including required bug fixes

RELEASE/PSI

Internal branch

Bug fixes

Test Farm – internal branch

Bug fixes

Deploy after two weeks on internal farm, including required bug fixes

Drop 1 Drop 2

Drop 1 Drop 2

Sprint length: 2 weeks


Source: http://www.redeemthecommute.com/


Challenges Roadmap Features

Coverage

Resources

Customer Requests


BIG DATA - TO THE RESCUE

Source: http://coffeebreakwithgamechangers.tumblr.com/


Data Science Office @ HP Software

Add data-science and

analytic capabilities in HP products

machine learning

graph analysi

s

static / dynamic

code analysis

other technique

s


•  We want to make informed product/dev decision based on real users behavior

•  We have real user monitoring today e.g. Google Analytics •  Information is event based not flow based •  Flow based is hard to understand •  There is code coverage but this doesn’t reflect real user usage •  What we need is real user coverage and insights

Problem scope Event-based analysis in Google analytics


Problem scope External Flows

Open

Chat

Write Msg

Attach

Send

Send Login

Test Flows

Feature: Use Chat

Login

Open Chat

Write Msg

Send

Open

Chat

Write Msg

Attach

Send

Send Login Ope

n Chat

Write Msg

Attach

Send

Send Login


Real user test coverage algorithm •  Difference between two flows based on their similarity score

definition •  The similarity score combines 2 criteria (Law & Order):

–  the identical steps, which exist in both flows –  the similarity in the step order in each flow

•  Match flows based on the difference function with an adaptive sensitivity threshold

•  Compute the weighted similarity for a set of matched production flows per test

Production Flows

Internal / Test Flows 1. Login > 2. Open Chat > 3. Write Msg > 4. Send

1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5. Send

1. Login > 2. Open Chat > 3. Send

1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5. Write Msg > 6. Write Msg > 7. Attach > 8. Write Msg > 9. Send

1. Login > 2. Open Chat > 3. Write Msg > 4. Send > 5. Attach

Usage: 50%

Coverage: 78%

Feature X 84

78

70 75


Key Findings

Ø  The automated tests cover 35% of production flows

Ø  50% of production events are not covered by any automatic test

Ø  Interesting insights from comparing internal vs. external user flows

From the motion picture “Moneyball”,2011


Significant increase in real user test coverage after gathering insights

0

10

20

30

40

50

60

70

80

100 150 200 250 300 350 400 450

Drop 1 stabilization Drop 2

New Content

#tests

Rea

l use

r cov

erag

e in

test

s


Example: Dev Summary

External Flows

Internal Flows Release mngmnt > Taskboard > Zoom > ali summary > all > builds

Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > Taskboard > Zoom > ali summary > all > shared items

•  Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > tab: Storyboard

•  Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > Taskboard > Zoom > ali summary > all > shared items

•  Release mngmnt > Taskboard > Zoom > ali summary > all > builds •  Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective

•  Compared: external customers to internal customers

•  Surprise: page is used similarly, regardless of dev configuration

•  => Cancel “prompt to config” plans

Feature: Dev

Summary Same usage in external

and Internal


Example: Planning board

•  Planning board event is rarely used internally – Should we invest?

•  Surprise: Planning board is used as the 1st step in many flows

Planning board Kanban Release backlog

[?] Sprint Backlog …


•  Surprise: “Sprint Closure” flows have high usage (>20%) but low coverage (3%)

•  Sprint closure should have high priority and additional test automation

Example: Sprint closure


Benefits - Recap

Data driven decision making

Better understanding & evaluation of customer needs

Improved real user test coverage

Optimize resource allocations


Thank You Contact: [email protected] [email protected]


Release Planning

•  Feature Ranking •  Feature Scoping - MMF •  Initial Feature Sizing •  Define Feature Lead •  Define Release Scope


Definition Of Done - DoD

ü  Unit tests are written and green ü  Acceptance tests are defined by QA and passed on top of a build ü  Main functionality of the user story is covered by automated acceptance tests ü  Sanity and Regression tests are passed green after the user story is checked in

0 C/H open defects ü  Majority of Medium defects are fixed ü  All fixed defects are validated

ü  The user story is pushed to production, monitored and green


Delivery Model to Production Release Criteria

Progress Status Target Current Status

Test Coverage On Track 100% •  100%

Quality On Track No Critical defects No C/H New features defects No C/H customer encountered defects No Functional Regression defects <10 High defects 70% M defects fixed

•  0 Critical defects •  0 C/H New features defects •  1 C/H customer encountered defects •  3 Functional regression •  10 open High defects •  61% of the M defects were fixed

Performance On Track No C/H issues No C/H issues

Security On Track No C/H issues No C/H issues


Product Owner Challenges

•  How to prioritize features and enhancement requests? •  Can I cut a story from a feature and it will still have value for the user?

(MMF) •  How will a new feature impact current usage? •  Does my acceptance criteria cover the end user’s flow? •  Are features being used the way they are “supposed” to? •  Is there a difference between how features are used internally vs.

externally?


Team Challenges

•  How much to invest in automation? •  What areas to automate and how to automate? •  What to include in our Regression testing?

Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat...

Documents

Transcript of Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat...