How Agile Ended Our Defect Report-Fix-Check-Rework Cycle

Copyright 2003-2005, Rally Software Development Corp

How Agile Ended Our Defect Report-Fix-Check-Rework Cycle

Richard LeavittRally Software [email protected]


Objectives for Today

Observation: ∙ Traditional workflows and project metrics evolved to

manage large defect inflows during long testing phases

Question: ∙ How relevant are these management processes in

dramatically shorter development cycles?

Agenda:∙ Defect Workflow & Metrics in Test-Last∙ Our Agile Transition∙ New Dev & Test Processes, Workflows and Reports∙ Benefits and Bruises


Agile Doesn’t Change Our Goals

∙ Customers judge our software as high value and high quality

∙ Our business judges our efforts as timely and providing high ROI

∙ We like our jobs


Agile Doesn’t Change Our Questions

We desire:∙ A small set of useful and valid measures,

provided by ∙ Standard reports that everyone anticipates

and understands, with∙ The ability to drill down and roll up

ReadinessWhen can

we release?

StatusHow’s feature X

doing?ProgressWhat’s

blocking?


Before: Phased Development Process

Advantages:• Requirements and design

“up front”• Logical• Prescriptive• Plan based

Disadvantages:• Requirements “freeze” paradigm• Dreaded system integration phase• High defect counts and “surprises”• Hard to adapt to changing requirements• Plan based


One of My Test-Last Environments

∙ Release size: 16 FTEs x 14 Months ∙ Detailed work breakdown with task complete

dates∙ Unit testing? What’s that? ∙ QA drops with high bug counts (100’s)∙ Islands of information

– Project plan– Requirement docs

(MRD->PRD->SRS)– Test cases in Excel– Build results

– Test effort & results– Traceability matrix– Defect/Issue/Request

lists and metrics


High Defect Counts Add Weight

∙ “Push” workflow systems with complex rule engines for routing & signoffs

∙ Fine-grained permissions to control ownership and changes

∙ Lengthy bug reviews & management burdens

∙ DELAY!


But Defects are Easy to Count & Graph!Statistical gymnastics∙ Defect rates∙ Densities, cycle times∙ By phase∙ By owner, type,

status, severity, module, how found…

∙ In general, lots of totals, percentages, averages and ratios


And how else can we answer our questions?

After all, our typical experience goes like this:

“I’ve never heard of any direct measure of project status, program readiness for release, or progress toward meeting milestones, etc.

Instead, we’ve used surrogate measures and metrics.”

The Darker Side of MetricsDouglas Hoffman, Software Quality Methods, LLC


Problems with Defect Metrics

QA Drops

ReadinessWhen can

we release?


Problems with Defect Metrics

Bug Review Meetings

Close Rate1

Close Rate2Defects “closed” as• Deferred• Duplicate• Low priority• Can’t Reproduce• Enhancement Request

ReadinessWhen can

we release?


So, Phased Development Processes Encourage a Defect-Driven Approach

And its accompanying problems:

∙ Large defect counts with heavy process burdens, high management effort and delay

∙ Defect-based metrics that don’t relate well to the real questions we’re trying to answer₋All metrics programs cause unintended side

effects on our behaviors, but can we do better?


Why Agile Development?

∙ Agile promotes rapid delivery and feedbackover elaborate upfront planning

Short iterations give all stakeholders timely and regular visibility into readiness, status and progress

∙ Shared commitments simplify workflows…but everyone needs real-time status information!

∙ Agile handles change better ₋ Accepts late-binding requirements

₋ Continuous integration and early testing means less “integration hell” and associated defects & surprises

In ShortMinimum Process, Maximum Value


The Agile Paradigm Shift

Constraints

Estimates

Schedule

ScheduleCost

Features

Value /VisionDriven

The Plan drives cost & schedule The Vision drives features

Requirements

Cost

PlanDriven

Waterfall Agile


Moving to Iterative & Incremental

ProjectMgmt

DevelopmentProcess

Requirements& Design

Test & FixProcess

Critical Path through Phases

Critical Drop Schedule

Time Boxes

Freeze & Signoff

Last Phase Only

“Test What’s Working”

Acceptance Test Inside the Iteration

Manage Scope Creep

Detailed Design Inside Iteration

All Features in Parallel

Multiple Drops to QA

Increment at a time

Waterfall IterativeIterative andIncremental

ParallelAcceptanceTest Driven

Agile Development


Agile Team Practices

∙ Time box everything∙ Two levels of planning: rough for releases,

fine-grained for iterations∙ Just-in-time requirements elaboration∙ Focus on the highest priorities∙ Early and continuous testing∙ Try to remove roadblocks and defects

immediately

Slide 2


The New Planning Process

∙ Release cycle: 8 or 10 weeks (4 or 5 iterations)₋Prod Mgmt defines themes & features/stories

₋Dev provides rough “planning” estimates

₋Release Backlog is then negotiated & prioritized

₋“Hardening” iteration at the end tackles technical debt

∙ Iteration cycle: 2 weeks with one ½ day planning meeting₋Recap previous iteration

₋PM presents new and elaborated stories & use cases

₋Dev details tasks & estimates just-in-time

₋Priorities are set and the team commits


Story Acceptance is Common Goal

∙ One schedule, one budget, one team, but independent dev & test people and tasks

∙ Features are squeezed, NOT testing and schedule

∙ Dev codes highest priority requirements all the way to acceptance

∙ QA elaborates & automates acceptance tests in parallel with coding₋Dev-QA collaboration versus separation improves

code robustness and testability right away.

₋Quality no longer “tested in”


Early & Continuous Testing Drives Down Defect Counts

∙ Unit tests run on code check-in ∙ Nightly build runs automated acceptance

tests∙ Team tries to resolve failures immediately

∙ When combined with short iterations, we get∙ Tiny defect counts (dozens, max) with little

management burden and team overhead

In Short, We Stay Close To Shippable Code


Iteration Decisions & Metrics

Meeting iteration intent and schedule means responding to changing realities∙ Should we cut or adjust any features?

₋What progress have we made on each story?

₋How much is there left todo?

₋What’s our remaining budget?

∙ What’s standing in our way of acceptance?₋Blocks?

₋Which story card tests are failing?

₋Are any defects open against our iteration stories?


New Iteration Status ViewsReadinessWhen can

we release?

StatusHow’s feature X

doing?ProgressWhat’s

blocking?

24


Tie Acceptance Test Results To Story

ProgressWhat’s

blocking?


So, Failing Tests Replace Many Defects for Simple, Pull-based Workflow

∙ Dev and Test now communicate via failing tests instead of routing defects

Test-driven Advantages₋Failures are easy to reproduce

₋Failures are transient, less management overhead

₋We’re done when the tests pass

₋Lighter process with less delay


Summary: Instead of Focus on Defects


We focus on Acceptance


Summary: Instead of Complex, Push-based Workflow


We use Simpler, Pull-based Workflows


Benefits and Bruises

∙ Agile has dramatically changed our project planning, tracking and workflow. We are more responsive to new requests and late changes.

>Everyone needed education on paradigm shift

∙ Everyone focused on incremental, meaningful functionality, not defect find/assign/fix/regress

>Demands high collaboration and real-time status. >Upstream analysts must provide steady flow of

business needs and priorities


More Benefits and Bruises

∙ Time boxes mean we always know when we’re going to release

>Though the scope may be slightly different!

∙ QA plays large role in requirement elaboration and defect prevention.

>Early resistance to TDD – required a change agent (Mike Cohn)

>Struggles to automate tests inside iteration>New skills required in test design and automation.


More Benefits and Bruises

∙ Tiny bug lists (dozens) with little management overhead>Most “bugs” found during exploratory testing in

hardening iterations. >Re-factoring tests is critical to keep up velocity.

∙ Simple, pull-based workflow with team members actively grabbing the next most important item off the stack

>Need culture of shared ownership that “takes responsibility”


Suggestions for Getting Ready

∙ Suggested Process₋ Brown bag discussions

₋ Guest lectures & webinars

₋ Local Agile/XP user groups

₋ Agile Conferences∙ Agile2005 – Denver, July

05

∙ Where to start on the web₋ Agile Alliance Web Site

www.agilealliance.org

₋ Rally Ecosystem Pages www.rallydev.com/ecosystem

∙ Suggested Reading₋ Agile Project Management

w/ SCRUM www.controlchaos.com/

₋ Agile & Iterative Development www.craiglarman.com

₋ Lean Software Development www.poppendieck.com/

₋ Agile Project Management www.jimhighsmith.com/

₋ Tactical Management of Iterative Development www.rallydev.com


Questions?


Acknowledgments

∙ The Darker Side of Metrics, Douglas Hoffman, PNSQC 2000

∙ It’s Not About the Bugs, Harry Robinson, Stickyminds 2003

∙ Managing Software Requirements: A Use Case Approach, Dean Leffingwell, 2003

∙ The Test Manager at the Project Status Meeting, Brian Marick, Steve Stukenborg, 1997

How Agile Ended Our Defect Report-Fix-Check-Rework Cycle

Documents

Transcript of How Agile Ended Our Defect Report-Fix-Check-Rework Cycle