Agility and Quality

04/20/23 (c) USC-CSSE 1

Agility and Quality

Barry Boehm, USC

04/20/23 (c) USC-CSSE 2

Outline

• Increasing importance of both agility and quality

• Challenges of achieving both agility and quality

• Approaches for achieving both agility and quality

• Case studies and critical success factors

• Conclusions

04/20/23 (c) USC-CSSE 3

Need for Agility: Increasing Pace of Change

•Technology change

•Related infrastructure and services

•Marketplace dynamics

•Competition dynamics

•Organizational change

•Software is critical

•User agility aids are even more critical

04/20/23 (c) USC-CSSE 4

Agile Methods Examples

• Scrum– 30-day sprints; daily standup Scrums; prioritized backlog

• eXtreme Programming (XP)– Programming: simple design; refactoring; test first;

continuous integration– Communications: on-site customer; metaphor; stories; pair

programming; collective ownership; code standards– Management: small releases; planning game; 40-hour week

• Others– Adaptive Software Development; Crystal; Dynamic Systems

Development; Feature-Driven Development

04/20/23 (c) USC-CSSE 5

The Agile Manifesto

• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

That is, while there is value in the items on the right, we value the items on the left more.

04/20/23 (c) USC-CSSE 6

The Need for Software Quality

• “The world runs on software” – Stroustrup• “With C, you can easily shoot yourself in the foot.

With C++, you can easily blow off your leg” – Stroustrup

• Critical global infrastructure: finance, energy, transportation, communications, trade

• Dependability: everything you depend on– Accuracy, adaptability, affordability, availability, …– Complex attribute conflicts and tradeoffs

04/20/23 (c) USC-CSSE 7

Traditional Quality Approach

• Complete, consistent, testable requirements • Traceable to design, code, test cases• Heavyweight documentation

• COCOMO documentation rates, Very High Reliability projects– Average 120 pp/KSLOC; median 83; range 32-241

• Rewriting needed for 1000 KSLOC project– 160 people; 120,000 pages of documentation– 1% change/month: 1200 pages (7.5 pages/person)– 10% change/month: 12,000 pages (75 pages/person)

04/20/23 (c) USC-CSSE 8

Sarbanes-Oxley• A new US Law

– Congress’ response to Enron, WorldCom, et al– Internal Controls: evaluate and disclose effectiveness– Disclose fraud– Affects public companies and “significant” vendors

• Development process must include internal controls for– Fraud– Asset Management and Safeguarding– Financial Reporting

• Why is this important to executive management?– Executives can go to jail.– IT management can be held grossly negligent and sued by a

company or shareholders.

• In effect since 2004

04/20/23 (c) USC-CSSE 9

What an Auditor Looks for…

Processes and tools over individuals and interactionsComprehensive documentation over working software

Contract negotiation over customer collaborationFollowing a plan over responding to change

An Auditor Manifesto?

04/20/23 (c) USC-CSSE 10

Agile Methods and Quality

• Responding to change over following a plan– Major source of software-induced rocket failures

• Small releases: It’ll be fixed by next month– OK for discomfort; not for safety

• Test-driven development helps, but often leads to patching– Example: Ada compiler validation suite

04/20/23 (c) USC-CSSE 11

Agile and Plan-Driven Home Grounds: Five Critical Decision Factors

• Size, Criticality, Dynamism, Personnel, Culture

Personnel

Dynamism (% Requirements- change/month)

Culture (% thriving on chaos vs. order)

Size (# of personnel)

Criticality (Loss due to impact of defects)

3010

3.01.0

0.3

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential Funds Discretionary

Funds Comfort

Single Life

Many Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Agile

Plan- driven

Personnel

Dynamism (% Requirements- change/month)

Culture (% thriving on chaos vs. order)

Size (# of personnel)

Criticality (Loss due to impact of defects)

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential Funds Discretionary

Funds Comfort

Single Life

Many Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Agile

Plan- driven

04/20/23 (c) USC-CSSE 12

Outline

• Increasing importance of both agility and quality

• Challenges of achieving both agility and quality

• Approaches for achieving both agility and quality

• Case studies and critical success factors

• Conclusions

04/20/23 (c) USC-CSSE 13

Using Risk to Balance Discipline and Agility - Overview

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

04/20/23 (c) USC-CSSE 14

Hybrid Agile/Plan-Driven Strategy– CRACK: collaborative, representative, authorized, committed, knowledgeable

LCALCOStartup Teambuilding DevelopmentSystems Architecting

Sta

ke

ho

lde

rsP

roje

ct

Le

ad

ers

hip

, R

isk

M

an

ag

em

en

t T

ea

ms

Ag

ile

, P

lan

D

riv

en

D

ev

elo

pe

rs

Furnish CRACK representatives and alternates

•Staff and organize to cover major risk areas

•Develop shared vision

•Negotiate top-level system objectives, architecture, plans, feasibility rationales.

•Prepare for/select developers

•Formulate/negotiate definitive requirements, architecture, plans, feasibility rationales.

•Ensure representative exercise of incremental capabilities

•Monitor, adapt to new developments

•Monitor and manage project progress, risk resolution, and new technology developments

•Continuously integrate/test growing software infrastructure and components

•Develop compatible architectures, plans, feasibility rationales •Develop system

components

Encapsulate agile portions

•

04/20/23 (c) USC-CSSE 15

Incremental Commitment Model:Single Increment View

Increment N Baseline

Rapid Change

High Assurance

Short, Stabilized Development of Increment N

Increment N Transition/O&M

Short Development Increments

Stable Development Increments

Foreseeable Change (Plan)


Rapid Change

High Assurance





Foreseeable Change (Plan)

04/20/23 (c) USC-CSSE 16

Incremental Commitment Model:Single Increment View


Future Increment Baselines Rapid Change

High Assurance

Agile Rebaselining for Future Increments


V&V of Increment N


Current V&V


Future V&V


Continuous V&V

Concerns Artifacts

Deferrals Foreseeable Change (Plan)

Resources Resources


Future Increment Baselines Rapid Change

High Assurance

Agile Rebaselining for


V&V of Increment N


Current V&V


Future V&V


Continuous V&V

Concerns Artifacts

Deferrals Foreseeable Change (Plan)

Resources Resources

Unforseeable Change (Adapt)

04/20/23 (c) USC-CSSE 17

Different Risk Patterns Yield Different Processes

04/20/23 (c) USC-CSSE 18

Pass/Fail Feasibility Rationales

• Evidence provided by developer and validated by independent experts that:If the system is built to the specified architecture, it will– Satisfy the requirements: capability, interfaces, level of service, and

evolution– Support the operational concept– Be buildable within the budgets and schedules in the plan– Generate a viable return on investment– Generate satisfactory outcomes for all of the success-critical

stakeholders

• All major risks resolved or covered by risk management plans

• Serves as basis for stakeholders’ commitment to proceed

04/20/23 (c) USC-CSSE 19

Case Studies andCritical Success Factors

• Diversified, USA (AgileTek)

• Aerospace, USA (Lockheed Martin)

• Medical, USA

• Enterprise Resource Planning, Germany

• Enterprise Infrastructure, Germany

04/20/23 (c) USC-CSSE 20

Agile Tek and Agile+

• Agile+ is XP + … + Business Process Analyses (BPAs) + Story “Actors” + Delphi-STE Estimation + Risk-Based Situation Audits (RBSAs) + Componentized Architecture + Wall Gantts and Instrument Panels + Automated Contract and Regression Testing + Automatic Document Generation Strict Pair Programming 40-Hour Work Week Restriction + Flexibility to meet special needs

04/20/23 (c) USC-CSSE 21

Agile Tek: Solutions to quality issues

• Scaling– Componentized Architecture/Interface Definitions– Automated Build and Test Processes – (Virtual) Team Rooms

• Unpredictability at Macro Scale– Delphi Estimation– STE usage for larger projects

• Vulnerability to changes at system level– Componentized Architecture

• Vague about system testing– Automated Contract and Regression Testing

• Inflexible to special needs– Treat the Special Need as a User Story and prioritize it accordingly

• Some ADM Practices Are Impractical– Use practices that make sense and work in real-world situations– Abandon or modify those that don’t

• Do not Manage Risks Explicitly– Use Risk Based Situation Audits– Establish a risk management philosophy

04/20/23 (c) USC-CSSE 22

Lockheed Martin Experiences

• 5 programs that used Agile to some degree:– Maritime Systems and Sensors– Aeronautics– Space Systems– Up to 4-team Scrum of Scrums

• Experience summary– Agile processes/practices used– What went well– What didn’t go quite so well

04/20/23 (c) USC-CSSE 23

Lockheed Martin: Agile Practices Used

• Project Planning– Sprint Planning– Backlog Lists– Risk Management– Manage scope, not cost/schedule/quality

• Design– Use agile modeling techniques– Keep it simple– Document just enough to keep you going

• Implementation and Test– Pair Programming – Refactoring – Test driven design – Continuous integration– Development on the target system

04/20/23 (c) USC-CSSE 24

Lockheed Martin: What Went Well

• Team empowerment/group ownership

• Plan for and embrace change

• Short cycle times allowed for prompt and frequent feedback

• Continuous integration

• Customer involvement

• Pair Programming

04/20/23 (c) USC-CSSE 25

Lockheed Martin: Needed Improvement

• Increased levels of stakeholder involvement

• Manage expectations

• Agile development processes require agile organizational processes

04/20/23 (c) USC-CSSE 26

Average Change Processing Time: 2 Systems of Systems

• Average number of days to process changes

0

20

40

60

80

100

120

140

160

WithinGroups

AcrossGroups

ContractMods

04/20/23 (c) USC-CSSE 27

Medical Case Study -- USA

• 1400 software people; 7M SLOC; 7 sites– 4 in Europe, 2 in India

• 500 medical applications; 500 financial; others• Survivability-critical software problems

– Reliability, productivity, performance, interoperability– Sarbanes-Oxley requirements– Management receptive to radical change

• Some limited experimental use of agile methods– Led by top software technologist/manager

• Committed to total change around Scrum and XP

04/20/23 (c) USC-CSSE 28

Medical-USA Adoption Profile

0102030405060708090

100

2004Jul

2005Jul

2006Jul

2007Mar

ScrumTeams

• July 2004 - July 2005– Recruit top people from all

sites into core team(s)

– Get external expert help

– Develop architecture

– Early Scrum successes with infrastructure

– Revise policies and practices

– Train, reculture everyone

– Manage expectations

• July 2005 – July 2006– Begin full-scale development

– Core teams as mentors

04/20/23 (c) USC-CSSE 29

Key Practices – USA Medical• Include customers and marketers

– New roles; do’s/don’ts/opportunities; CRACK personnel; full collaboration and teamwork; expectations management

• Scrum; most XP practices; added company practices– 6-12 person teams with team rooms, dedicated servers– Hourly smoke test; nightly build and regression test– Just-in-time analysis; story-point estimates; fail fast; detailed short-term plans;

company architecture compliance– Embrace change in applications and practices– Global teams: wikis, daily virtual meetings, act as if next-door

• Release management– 2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint; 1-6 month

beta test– Next Sprint Zero concurrent with Release Sprint

• Initiative manager and team– Define practices; evolve infrastructure; provide training; guide implementation;

evaluate compliance/usage; continuous improvement

04/20/23 (c) USC-CSSE 30

ERP Case Study -- Germany

• 35,000 employees; 32,000 customers worldwide– Major development centers in India, Israel

• Need to improve software development productivity, adaptability of Product Innovation Life Cycle– Invent, Define, Develop, Deploy, Optimize

• Proposed agile development cycle– Scrum; tailored corporate practices; strong, >70%-time

Product Owner and Scrum Master– 10-person teams best; up to 3-team Scrum of Scrums– Release cycle similar to Medical-USA

• Strong, highly experienced agile initiative leader– With top management support

04/20/23 (c) USC-CSSE 31

ERP-Germany Adoption Profile

050

100150200250300350400450500

2005Nov

2006Apr

2006Nov

Scrum-trainedScrumprojects

• Two large projects– 8 teams, 2 countries

– 5 teams, 5 Product Owners, 3 countries

• Mentoring new teams– Pre-training

– Experienced Scrum Master

– Mentor first sprint

– Coach second sprint

• Benefits: visibility, communication, productivity

04/20/23 (c) USC-CSSE 32

Challenges and Lessons Learned

• Challenges– Keeping roles straight– Setup for large projects– Rebaselining, reprioritizing backlog– Cross-cultural training, jelling– Team learning culture

• Lessons learned– Need strong Product Owner and Scrum Master

• Training for Product Owner, other stakeholders• Especially for scaling up to multiple teams

– Reinforce, evolve common corporate Scrum process– Can’t neglect CM, version control, change management

• Of applications and process

04/20/23 (c) USC-CSSE 33

Corporate Infrastructure -- Germany

• Fortune World 100 company• Global development

– Especially US, India, China

• Need to rebaseline corporate infrastructure– Business processes, services, IT infrastructure– Multi-platform portability, always on– With worldwide participation and buy-in– Strategy: RUP/Spiral architecting; Scrum-based development

• Began with multi-site core team of top personnel– Led by strong, results-oriented project/technology leader– With top management support and backing

• Grown to 4 sites, 250 people, 24 teams in 2.5 years– Largest project: 10 teams of 10-person Scrums

04/20/23 (c) USC-CSSE 34

Corporate Infrastructure: Principles

• Service-oriented architecture– Business processing: IBM, SAP, Microsoft– Generic applications: phone, Web, user interface– Infrastructure: servers, gateways, networks

• Learn form others• Select, protect best team• Inclusive stakeholder communication and

involvement• Plan for early wins• Corporate product and process framework with

explicit tailoring areas, continuous improvement

04/20/23 (c) USC-CSSE 35

Development Practices

• RUP/Spiral startup– 2 Inception, 3 Elaboration, 4 Construction cycles– Proposal bay with wall stickies for risk prioritization– 30 top people from all 4 sites

• NetMeeting for remote office involvement

– SharePoint vs. heavy documentation– Dedicated specialists (architecture, performance, test)

• Initial Operational Capability development– Timeboxed sprints with prioritized requirements

• 30% of initial requirements dropped to admit new features

• Use backlog as management instrument

• Post-IOC release sprint for documentation, installation, traning

– Followed by field test, concurrent detailed expansion planning, return of offsite core team members to lead distributed-development scaleup

04/20/23 (c) USC-CSSE 36

Critical Success Factors

• Management commitment, with incremental feasibility checkpoints– Clear message about objectives, scope, and strategy– Involve top people from stakeholder organizations– Build in growth to expansion sites– Lead through early successes

• Thoroughly prepare the ground– Infrastructure, policies, practices, roles, training– Customer buy-in and expectations management– Get help from experts

• Make clear what’s essential, optional– Most frequently, Scrum plus organizational essentials– Precede Development Sprints by Architecting Sprint

• Follow by Release Sprint, beta testing

– Where needed, work compliant mandate interpretations

• Monitor, reflect, learn, evolve

04/20/23 (c) USC-CSSE 37

Conclusions

• Success-critical to achieve both agility and quality• Hybrid agile/plan-driven methods emerging

– Incremental commitment framework– Concurrent engineering with synchronization milestones– Scrum plus organizational essentials

• Success stories emerging– Management commitment to objectives and strategy

• With incremental feasibility checkpoints

– Strong core team of technical and management leaders– Thorough preparation of organizations, people, infrastructure

• Involvement, architecture, policies, practices, plans, training

– Continuous change monitoring and adaptation

04/20/23 (c) USC-CSSE 38

Backup Charts

04/20/23 (c) USC-CSSE 39

Incremental Commitment In Systems and Life:

• Common System/Software stakeholder commitment points– Defined in concert with Government, industry organizations – Initially coordinated with Rational’s Unified Software Development

Process• Exploration Commitment Review (ECR)

– Stakeholders’ commitment to support initial system scoping– Like dating

• Validation Commitment Review (VCR)– Stakeholders’ commitment to support system concept definition and

investment analysis– Like going steady

• Architecting Commitment Review (ACR)– Stakeholders’ commitment to support system architecting – Like getting engaged

• Development Commitment Review (DCR)– Stakeholders’ commitment to support system development– Like getting married

• Incremental Operational Capabilities (OCs)– Stakeholders’ commitment to support operations– Like having children

Anchor Point Milestones

04/20/23 (c) USC-CSSE 40

The Incremental Commitment Life Cycle Process: Overview

04/20/23 (c) USC-CSSE 41

ICM HSI Levels of Activity for Complex Systems

04/20/23 (c) USC-CSSE 42

Process Models

Principles

Stakeholder Satisficing

Incremental Growth

Concurrency Iteration Risk Management

Sequential Waterfall, V

Assumed via initial requirements; no specifics

Sequential No NoOnce at the beginning

Iterative, Risk-Driven Waterfall, V

Assumed via initial requirements; no specifics

Risk-driven; missing specifics

Risky parts Yes Yes

Risk-Driven Evolutionary Development

Revisited for each iteration


Risky parts Yes Yes

Concurrent Engineering

Implicit; no specifics

Yes; missing specifics

Yes Yes Implicit; no specifics

Agile Fix shortfalls in next phase

Iterations Yes Yes Some

Spiral Process 2001

Driven by stakeholder commitment milestones


Yes Risk-driven Yes

Incremental Commitment

Stakeholder-driven; stronger human factors support

Risk-driven; more specifics

Yes Yes Yes

Process Model Comparison with respect to Process Principles

Agility and Quality

Documents

Transcript of Agility and Quality