USC-CSE Annual Research Review Los Angeles, CA March 17, 2004.

USC-CSE Annual

Research Review

Los Angeles, CAMarch 17, 2004

2

Outline

• Background• Impact of 2003 Workshop on book• Definitions and characterizations• Home grounds and the five critical

dimensions• A risk-based approach to balancing

agility and discipline• Observations

3

Background (Rationale for book)

• Two approaches to software development – Plan-driven (SW-CMM, document-based, heavy

process)– Agile (XP, tacit knowledge, light process)

• Both have strengths and weaknesses• Agile and plan-driven proponents are believers• Many of us are perplexed • We still perceive perplexity• More voices being heard:

– Larman– Anderson – Rosenberg/Stevens– Poppendieck– Others???

4

Impact of 2003 Workshop - 1

• “Disciplined” changed to “plan-driven”– Driven by process and product plans

• Both agile and plan-driven methods need to balance agility and discipline– XP: 12 highly-disciplined practices– RUP: Flexible mix of requirements, design,

code

5

Impact of 2003 Workshop - 2

• More examples of hybrid methods– AgileTek: Agile with top-level plans,

architecture– Lightweight versions of RUP and TSP– XP variants:

• Part-time customers• Longer workweeks• Less collective ownership and metaphor

• More examples of empirical data• Balancing illustration and need for

transition guidance– Collins’ – Good to Great

6

Collins’ Good to Great

Bureaucratic Organization

Start-up Organization

Hierarchical Organization

Great Organization

Low

Low

High

High

Ethic of Entrepreneurship

Culture of Discipline

7

Trends since last year

• Increasing demands for agility, velocity, quality

• Alternative manifestos• Push toward evolutionary acquisition in

traditional communities• More agile approaches to CMMI

improvement, assessment• More use of agile methods in large

companies• More hybrid agile/plan-driven

approaches

8

Outline




9

Sources of Perplexity

• Distinguishing method use from method misuse– Claiming XP use when simply “not documenting”– “CMM Level 4 Memorial Library” of 99 2-inch binders

• Overgeneralization based on the most visible instances– XP is Agile; CMM is Plan-driven

• Multiple definitions– Quality: customer satisfaction or compliance?

• Claims of universality– The pace of IT change is accelerating and Agile

methods adapt to change better than disciplined methods therefore Agile methods will take over the IT world.

– Software development is uncertain and the SW-CMM improves predictability therefore all software developers should use the SW-CMM

10

Sources of Perplexity (2)

• Early success stories– Chrysler project that successfully birthed XP was

later cancelled– Cleanroom has never made it into the mainstream

• Purist interpretations (and internal disagreements)– “Don't start by incrementally adopting parts of

XP. Its pieces fit together like a fine Swiss watch“– "An advantage of agile methods is that you can

apply them selectively and generatively."– "If you aren't 100% compliant with SW CMM Level

3, don't bother to bid" – "With the CMMI continuous interpretation, you can

improve your processes in any order you feel is best."

11

Key Definitions

• Agile method – – one which fully adopts the four value propositions

and twelve principles in the Agile Manifesto.

• Discipline – (per Webster)– control gained by enforcing obedience or order; – orderly or prescribed conduct or pattern of behavior; – self-control.

• Plan-driven – – Processes driven by documented plans (order is

often defined in plans)

• Plan – (per Webster)– a method for achieving an end (a process plan); – an orderly arrangements of parts of an overall

design (a product plan). – We assume that such plans are documented.

12

Finding Middle Ground

• A pragmatic view• Both approaches have “home grounds”• A broad range of environments and needs• Trends require better handling of rapid

change• Processes should be the right weight for the

job• We believe risk-based analysis is the key to

finding middle ground

13

Contrasts and Home Grounds

• Comparison is difficult, but we found 4 areas where there are clear differences

• Application– Project goals, size, environment; velocity

• Management– Customer relations, planning and control,

communications

• Technical– How the software is developed

• Personnel– The type and competency of developers and

stakeholders

14

Application Characteristics

• Primary goals– A: Rapid value, responsiveness to change– P: Predictability, stability, high assurance

• Size– A: Small to medium– P: Large

• Environment– A: Turbulent, high change– P: Stable, requirements defined

• Project velocity– A: Code, learn, and fix– P: Architect for parallel, incremental

development

15

Management Characteristics

• Customer Relations– A: Dedicated, collocated; where feasible– P: Contractual; increasingly evolutionary– A: Trust through working software and

participation– P: Trust through process maturity evaluations

• Planning and Control– A: Means to an end– P: Anchor processes, communication

• Project Communications– A: Tacit, interpersonal knowledge– P: Explicit, documented knowledge

16

Technical Characteristics

• Requirements– A: Adjustable, informal stories– P: Formally baselined, complete, consistent

specifications

• Development– A: Simple Design– P: Architecture-based design

• Testing– A: Automated, test-driven– P: Planned, requirements-driven

17

Technical Characteristics (2)Cost of Change: Beck, Li

Cos

t of

Cha

nge

Time

Beck

LiLi

18

Technical Characteristics (3) Cost of change: Poppendieck

• At least two cost-escalation curves• Lean development: Architect to defer

high-stakes decisions – e.g. COTS• Easier to change an unmade decision

Most Changes

Changes in High-stakes Constraints

Time

Co

st o

f C

han

ge

19

Technical Characteristics (4)Cost of Change: TRW

Beck

LiLi

Steeper Cost-to-fix for High-Risk Elements

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

% of Software Problem Reports (SPR’s)

TRW Project A373 SPR’s

TRW Project B1005 SPR’s

% ofCosttoFixSPR’s

Major Rework Sources:Off-Nominal Architecture-BreakersA - Network FailoverB - Extra-Long Messages

Steeper Cost-to-fix for High-Risk Elements

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

% of Software Problem Reports (SPR’s)

TRW Project A373 SPR’s

TRW Project B1005 SPR’s

% ofCosttoFixSPR’s

Major Rework Sources:Off-Nominal Architecture-BreakersA - Network FailoverB - Extra-Long Messages

20

Technical Characteristics (5)Cost of Change: CCPDS-R

Beck

RATIONALS o f t w a r e C o r p o r a t I o n

Project Development Schedule15 20 25 30 35 40

40

30

20

10

Design Changes

Implementation Changes

MaintenanceChanges and ECP’s

HoursChange

RATIONALS o f t w a r e C o r p o r a t I o n

Project Development Schedule15 20 25 30 35 40

40

30

20

10

Design Changes

Implementation Changes

MaintenanceChanges and ECP’s

HoursChangeHours

Change

21

Technical Characteristics (6)How Much Architecting?

Beck

LiuLiu

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

Percent of Time Added for Architecture and Risk Resolution

Per

cen

t of T

ime

Ad

ded

to O

vera

ll S

ched

ule

Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution

Added Schedule Devoted to Rework(COCOMO II RESL factor)

Total % Added Schedule

Sweet Spot Drivers:

Rapid Change: leftward

High Assurance: rightward

Sweet spot

10 KSLOC

100 KSLOC

10,000 KSLOC

22

Personnel Characteristics

• Customers– A: CRACK customers throughout development– P: CRACK customers early

• CRACK: Collaborative, Representative, Authorized, Committed, and Knowledgeable

• Developers– A: Heavy mix of high caliber throughout– P: Heavy mix early with lower capability later

• Culture– A: Many degrees of freedom (Thrives on

chaos)– P: Clear policies and procedures (Thrives on

order)• Collocated customers are difficult to

find/keep/keep current on either side

23

Personnel Characteristics (2)Modified Cockburn Levels

(Ski

ll, U

nders

tandin

g)

(Formality, Documentation)

(Ski

ll, U

nders

tandin

g)

(Formality, Documentation)

Level Characteristics

3 Able to revise a method (break its rules) to fit an unprecedented new situation

2 Able to tailor a method to fit a precedented new situation

1A With training, able to perform discretionary method steps (e.g., sizing stories to fit increments, composing patterns, compound refactoring, complex COTS integration). With experience can become Level 2.

1B With training, able to perform procedural method steps (e.g. coding a simple method, simple refactoring, following coding standards and CM procedures, running tests). With experience can master some Level 1A skills.

-1 May have technical skills, but unable or unwilling to collaborate or follow shared methods.

24

Outline




25

Summary of Home GroundsCharacteristics Agile Plan-driven

Application

Primary Goals Rapid value; responding to change Predictability, stability, high assurance

Size Smaller teams and projects Larger teams and projects

Environment Turbulent; high change; project-focused Stable; low-change; project/organization focused

Management

Customer Relations

Dedicated on-site customers, where feasible; focused on prioritized increments

As-needed customer interactions; focused on contract provisions; increasingly evolutionary

Planning/Control

Internalized plans; qualitative control Documented plans, quantitative control

Communications

Tacit interpersonal knowledge Explicit documented knowledge

Technical

Requirements Prioritized informal stories and test cases; undergoing unforseeable change

Formalized project, capability, interface, quality, forseeable evolution requirements

Development Simple design; short increments; refactoring assumed inexpensive

Architect for parallel development; longer increments; refactoring assumed expensive

Test Executable test cases define requirements

Documented test plans and procedures

Personnel

Customers Dedicated, collocated CRACK* performers CRACK* performers, not always collocated

Developers At least 30% full-time Cockburn level 2 and 3 experts; no Level 1B or -1 personnel**

50% Cockburn Level 3s early; 10% throughout; 30% Level 1B’s workable; no Level -1s**

Culture Comfort and empowerment via many degrees of freedom (thriving on chaos)

Comfort and empowerment via framework of policies and procedures (thriving on order)

* Collaborative, Representative, Authorized, Committed, Knowledgeable** These numbers will particularly vary with the complexity of the application

26

Five Critical Decision Factors

• Represent five dimensions• Size, Criticality, Dynamism, Personnel,

Culture Personnel

Dynamism (% Requirements-change/month)

Culture (% thriving on chaos vs. order)

Size (# of personnel)

Criticality (Loss due to impact of defects)

5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential Funds Discretionary

Funds Comfort

Single Life

Many Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Agile

Plan-driven

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Agile

Plan-driven

Figure 6-1

27

Outline




28

Using Risk to Balance Discipline and Agility - Overview

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

29

Risks Used in Risk Comparison

• Environmental risks– Technology uncertainties– Many stakeholders– Complex system-of-systems– Legacy compatibility

• Agility risks– Scalability– Use of simple design– Personnel turnover– Too-frequent releases– Not enough agile-capable people

• Plan-driven risks– Rapid change– Need for rapid results– Emergent requirements– Not enough discipline-capable people

30

Three Examples

• Agent-based systems• Small – Event Managers application• Medium – SupplyChain.com application• Large – National Information System for

Crisis Management (NISCM) application• Each example results in a development

strategy based on risk analyses

31

Event Managers Project

• Small startup company • Diverse set of smaller agent-based

planning applications • This project: support for managing the

infrastructure and operations of conferences and conventions

• Widening variety of options and interdependencies, makes an agent-based approach highly attractive

32

Event Managers Profile

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Risk Items Risk Ratings Event Managers

Environmental Risks E1. Technology uncertainties E2. Many stakeholders E3. Complex system of systems Risks of using Agile Methods A1. Scalability A2. Use of simple design A3. Personnel turnover A4. Too-frequent releases A5. Not enough agile-capable people Risks of using disciplined methods D1. Rapid change D2. Need for rapid results D3. Emergent requirements D4. Not enough discipline-capable people

Risk rating scale: - minimal risk - moderate risk

- Serious but manageable risk - Very serious but manageable risk - Show-stopper risk

33

Event Managers Strategy

1. StartupTeambuilding and Shared

Vision

2. Design, Development and Deployment

• Establish CRACK joint management team

• Develop shared vision, results chain, life cycle strategy, infrastructure choices

• Establish commitment to proceed, incremental capabilities, backlog

• Develop next-increment features

• Test, exercise, deploy new increment• Re-prioritize next-increment features• Analyze lessons learned; prepare strategy for next increment

Customer Management; Representative Users

Joint Developer-Customer Agile Team

34

SupplyChain.com Profile

• Turnkey agent-based supply chain management systems

• Distributed, multi-organization teams of about 50 people

• Parts of applications are relatively stable, while others are highly volatile

• Architectures are driven by a few key COTS packages that are also continually evolving

35

SupplyChain.com Profile

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Risk Items Risk Ratings SupplyChain.com

Environmental Risks E1. Technology uncertainties E2. Many stakeholders E3. Complex system of systems Risks of using Agile Methods A1. Scalability A2. Use of simple design A3. Personnel turnover A4. Too-frequent releases A5. Not enough agile-capable people Risks of using disciplined methods D1. Rapid change D2. Need for rapid results D3. Emergent requirements D4. Not enough discipline-capable people



36

SupplyChain.com Strategy

Startup 1. Teambuilding and Shared

Vision

3. Design, Development and Deployment2. Systems Definition and Architecting

Stakeholders

Furnish CRACK representatives and alternates

Staff and organize to cover all success-critical risk areas

• Develop benefits-realization results chains

• Identify missing stakeholders

• Enhance mutual understanding

• Explore goals, options, technology, prototypes, risks

Review; Commit

to proceed

• Elaborate supply chain operational concept, prototypes, transition strategy

• Evaluate and determine COTS, reuse, and architecture choices

• Prioritize and sequence desired capabilities, outstanding risks

• Establish project organization, overall process, and feature teams

Review; Commit

to proceed

• Ensure representative exercise of incremental capabilities

• Monitor progress and new operational development

• Monitor project progress, risk resolution, and new technology developments

• Identify and work critical project-level actions

• Develop and integrate new feature increments

• Communicate proposed redirections

• Prepare for and execute acceptance tests, installation, operational evolution

• Analyze feedback on current version

• Reprioritize features

• Prepare strategy for next increment

• Consolidate process lessons learned

• Use risk to determine content of artifacts

Project Manager andRisk Management Team

Agile Feature Team

37

NISCM Profile

• Broad coalition of government agencies and critical private-sector organizations

• Support cross-agency and public-private sector coordination of crisis management activities

• Adaptive mobile network, virtual collaboration, information assurance, and information integration technology

• Private-sector system-of-systems integration contractor

38

Risk Items Risk Ratings NISCM

Environmental Risks E1. Technology uncertainties E2. Many stakeholders E3. Complex system of systems Risks of using Agile Methods A1. Scalability — A2. Use of simple design — A3. Personnel turnover A4. Too-frequent releases A5. Not enough agile-capable people — Risks of using disciplined methods D1. Rapid change D2. Need for rapid results D3. Emergent requirements D4. Not enough discipline-capable people



NISCM Profile

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

39

NISCM Strategy

Startup Teambuilding and Shared

Vision

Design, Development and Deployment

Systems Definition and Architecting

Stable, Higher-criticality Module Teams

Stakeholders

Furnish CRACK representatives and alternates

• Staff and organize to cover all success-critical risk areas

• Prepare for and select competitors for concept definition

• Develop results chains• Identify missing

stakeholders• Enhance understanding• Explore goals, options,

technology, prototypes, risks

• Negotiate mutually satisfactory objectives and priorities

• Formulate top-level operational concept, requirements, architecture, life cycle plan, feasibility rational

• Select winning system integration contract

Hold Life Cycle

Objective Review. If feasibility

rationale is valid,

commit to proceed

• Prepare for/select competitors for component subcontract definition

• Elaborate operational concept, prototypes, transition strategy

• Evaluate/determine COTS, reuse, and architecture choices

• Develop/ validate critical-infrastructure software

• Prioritize/sequence desired capabilities, outstanding risks

• Formulate definitive operational concept, requirements, architecture, life cycle plan, feasibility rationale

• Use to solicit/select component subcontractors

Hold Life Cycle

Architecture Review. If feasibility

rationale is valid,

commit to proceed

• Ensure representative exercise of incremental capabilities

• Monitor progress and new operational development

• Monitor project progress, risk resolution, and new technology developments

• Identify/work critical project-level actions

• Continuously integrate/ test growing software infrastructure and components

Dynamic, Lower-criticality Module Teams

• Develop compatible component-level prototypes, requirements, architectures, plans, critical software, feasibility rationale

• Classify modules by likely stability and criticality

Organize into disciplined and agile module teams

Plan, specify, develop stable, higher-criticality module increments

Plan, specify, develop dynamic, lower-criticality module increments

• Communicate proposed redirections

• Prepare for and execute acceptance tests, installation, operational evolution

• Analyze feedback on current version

• Reprioritize features

• Prepare strategy for next increment

• Use risk to determine content of artifacts

NSO and I3-led ProjectRisk Management Team

40

Risk Profiles Across Examples

Risk Items Risk Ratings Event Managers SupplyChain.com NISCM

Environmental Risks E1. Technology uncertainties E2. Many stakeholders E3. Complex system of systems Risks of using Agile Methods A1. Scalability — A2. Use of simple design — A3. Personnel turnover A4. Too-frequent releases A5. Not enough agile-capable people — Risks of using disciplined methods D1. Rapid change D2. Need for rapid results D3. Emergent requirements D4. Not enough discipline-capable people Risk rating scale: - minimal risk - moderate risk


41

Outline




42

Summary of Observations

• No silver bullet• Clear agile and

plan-driven home grounds

• Increased demands for agility, velocity, quality

• Balanced methods are emerging

• Build methods up vs. tailor down

• Methods are important; people more so – Values– Communications– Expectations

management

43

No Silver Bullet

• Brooks’ werewolf concerns– Complexity,

conformity, changeability, invisibility

• Agile methods – Handle changeability

and invisibility– Miss on complexity

and conformity

• Plan-driven methods– Handle conformity and

invisibility– Miss on changeability

and complexity

44

Future Applications Need Both

• Historically– Many small, non-critical,

well-skilled, agile culture, rapidly evolving projects

– Many large, critical, mixed-skill, ordered culture, stable projects

• In the future– Large projects are no

longer stable– Maintenance of

extensive process and product plans will become too expensive

– Complexity and conformity werewolves are waiting for agile projects

– Attributes of both approaches will be needed

45

Balanced Methods are Emerging

• Agile methods– Crystal Orange– DSDM– FDD– Lean Development

• Plan-Driven methods– Rational Unified

Process– CMMI

• Hybrid– Boehm-Turner Risk-

based– Code Science/Agile

Plus

46

Build up – Don’t Tailor Down

• Plan-driven methods traditionally– Define heavyweight process guidelines– Advocate tailoring– Treat mass of processes as security blanket

• Agilists traditionally– Begin with the minimum – Add as needed (and justified by cost-benefit)– Start small; extend only where necessary

47

Maybe its not the methods!

• Agile movement has echoed a long line of warning calls

• Success of agile may be due as much to people factors as to technology

• Valuing people over processes is the most important factor in the manifesto

I know I saw something about that in the process somewhere…

48

People

• Of the people, by the people, for the people

• Separations of concerns is increasingly harmful

• A few quotes…– The notion of “user” cannot be

precisely defined, and therefore it has no place in computer science or software engineering. Dijkstra

– Analysis and allocation of the system requirements is not the responsibility of the software engineering group, but it is a prerequisite for their work. The Capability Maturity Model for Software.

– Software engineering is not project management. Tucker

49

Values

• Reconciling values is a critical people-oriented task

• Stakeholders value different things• Software engineering is usually value-

neutral– Every requirement, object, test case, defect

equally important• Process improvement and plan-driven

methods are inwardly-focused – Aimed at productivity improvement– Not higher value to customer

• Agilists attention to prioritization and negotiation are promising

50

Communications

• Face it, most engineers can’t talk, much less write

• Need to bridge the customer-developer communications gap

• IKIWISI* reigns• Rapid change increases

need for solid communication

• Few sources of guidance– Alistair’s Agile Software

Development is a good starting place

*I’ll know It When I See It

51

Expectations Management

• Differences between successful and troubled projects is often expectations management

• SW developers have problems with EM – Strong desire to please– Little confidence in prediction– Over confidence in abilities

• Most significant common factor in highly effective plan-driven or agile teams

• EM means not setting up unrealistic expectations– Process mastery– Preparation– Courage

Developers seem to like Sisyphean tasks…

52

Conclusions

• Plan-driven and agile methods both aim to– Satisfy customers– Meet cost and schedule arameters

• Home grounds exist, but the opportunity for integration is expanding

• We recommend a self-assessment of your organization and business – Locate your place in the home ground space– Identify areas of change and how they might

move your location– Look for ways to integrate method to better

repond to change

USC-CSE Annual Research Review Los Angeles, CA March 17, 2004.

Documents

Transcript of USC-CSE Annual Research Review Los Angeles, CA March 17, 2004.