Requirements Volatility Workshop Rick Selby USC Center for Systems & Software Engineering February...

Requirements Volatility Workshop

Rick Selby

USC Center for Systems & Software Engineering

http://csse.usc.eduFebruary 15, 2007

Rick Selby NGC [email protected]

JoAnn Lane USC CSSE [email protected]

Sue Koolmanojwong USC CSSE [email protected]

Scott Rigby Raytheon [email protected]

Mike Barker NAIST [email protected]

Jim Lambert CISCO [email protected]

Tom Schroeder BAE Systems [email protected]

Robert Culbertson CISCO [email protected]

Gary DeGregoro Motorola Labs [email protected]

Ray Hunnicutt Lockheed Martin [email protected]

Karen Owens The Aerospace Corp. [email protected]

Da Yang Institute of Software CAS [email protected]

Attendee List

Dan Ligett Softstar [email protected]

Ray Madachy USC CSSE [email protected]

Anna Warner Boeing [email protected]

Barbara Park Boein [email protected]

A Winsor Brown USC CSSE [email protected]

George Hulin DSC [email protected]

Rosaline Lewis The Aerospace Corp. [email protected]

Barry Boehm USC CSSE [email protected]

Di Wu USC CSSE [email protected]

Vu Nguyen USC CSSE [email protected]

Rod Robertson Boeing [email protected]

Attendee List

4

Workshop Guidelines

Product: briefing, preferably with notes

Topics should include:– Most critical success factors in each area– Current best practices for addressing them– Areas for further research

Rated 0-10 on value and difficulty of research

5

Requirements Critical Success Factors

Technical– (tech)Investigate req interdependencies and dependencies– (tech)Demistify the notion that requirements are independent– (tech)Requirements gathering-what are the important attributes

that should be captured – (tech)Want virtual model and understand attributes– (tech and people)Better ways to prioritize requirements,

considering risk, cost factors, stakeholder utility function and how to reconcile stakeholders interest

– (tech and people)Same as above with greater emphasis on change

– (tech)Specify requirements/not implementation; users often want to specify implementation

– (tech)What are the critical/driving requirements

6

Requirements Critical Success Factors (continued)

Technical– (tech)Relate requirements work back to COSOSIMO and

COSYSMO– (tech)Missing rich enough vocabulary to model “this whole thing”

(wants, needs, problems, issues, business drivers that are referred to as requirements)

– (tech)Need modeling approach other than English– (tech/tool)How do we automatically measure requirements,

requirement volatility, progress, scope– (tech)Accurately assess/predict impact of changes– (tech)Standard classification method to capture types of

requirements and level of requirements– (tech)Classify requirements volatility types and define metrics to

measure volatility– (tech)Address differences in req vol across market segments

7


Technical– (tech)Function points necessary but not sufficient—

miss technical performance and constraints– (tech)Capabilities versus requirements– (tools/tech)Tools: relate estimation inputs to the

outputs (for example, inputs req lead to sloc output instead of sloc to effort)

– (tech)Global collaboration—state requirements in a way that they can be understood globally and implemented locally

– (tech)Specify the behavior for off-nominal conditions and events

– (tech)Detect the confliction between requirements

8


Tool– (tech/tool)How do we automatically measure

requirements, requirement volatility, progress, scope– (process/tool)Requirements traceability from the

beginning (who initiated and why, which application, which test)

– (tools)Investigate tools available for requirements process (measurement, tracking)

– (tools/tech)Tools: relate estimation inputs to the outputs (for example, inputs req lead to sloc output instead of sloc to effort)

– (process/tools)Need a good way to move back and forth between reqs and models

9


Product– (prod)Define requirements in a sufficient enough way

to cost them, but high level enough to provide flexibility (R. Turner comment)

10


Process– (process)Capture requirement up front using elicitation in a way

that avoids future change– (process)Plan for requirements to change

(evolutionary/incremental development)– (process)Late binding of requirements-need to specify in a way

to promote late binding– (process)Involve all key stakeholders and ensure their

accountability for the requirements– (people/process)Treat requirements process as a negotiation

and learning process– (management/process)Minimize requirements volatility by

managing decision volatility– (management/process)Link decisions and requirements– (process)Need to identify assumptions as well as requirements

11


Process– (process)Be clear what is an assumption and what is a

requirement– (people/process)Treat requirements process as a negotiation

and learning process– (process)Standard on requirements volatility-how to measure,

what to measure– (process/methodology)Look at process used to development

requirements, especially across enterprises Modeling and sim Environment (process, tools, people) Instrumentation Increase propensity to accept changes in the requirements

process– (process)Steal from agile, tests are one of the better forms for

capturing requirements

12


Process– (process/management)Get immediate feedback from

development and test teams that requirements can be implemented and tested

– (process/management)Write requirements in alignment/context with business drivers

– (process/tool)Requirements traceability from the beginning (who initiated and why, which application, which test)

– (process/management)Keeping the multiple stakeholders engaged across the life cycle

– (process/management)Integrate measurement and modeling of requirements changes

13


Process– (management/process)Improve/investigate

methodology for requirements development by incorporation of meta models, semantics, and ontology

– (process)Using value based systems engineering we need to identify the 10-20 key driving requirements for large scale systems

– (people/process)Collaboration with customer to prevent later volatility (using analog of collaborative design in the requirements space)

14


Process– (process)Moving from requirements to models too

early leads to premature codification of solution(s)– (process/tools)Need a good way to move back and

forth between reqs and models– (process)25% req change results in 100% change in

the solution space– (process)Requirements completeness for context at

various points in time

15


People– (tech and people)Same as above with greater

emphasis on change– (people) Committed, representative, available,

collaborative, knowledeable (CRACK) stakeholders – (people/process)Treat requirements process as a

negotiation and learning process– (people/process)Treat requirements process as a

negotiation and learning process– (people)Make stakeholder incentives explicit– (people)Function point modeling to provide another

way to look at requirements

16


People– (people)Data modeling (part of FP modeling) to

analyze requirements– (people/management)Change management and

expectation management– (people/org)How do we convince customers to think in

smaller steps– (people/org)Customers tend to think 5-10 yrs in the

future, others tend to think 9 months ahead– (people/process)Collaboration with customer to

prevent later volatility (using analog of collaborative design in the requirements space)

– (people)Experience of team

17

People– (people)Diverse representation in the systems engineering

organization: role, function, people, cultures– (people/org)Managing stakeholder continuity/volatility/high

variance of req interpretation– (people)Ensure that you understand your customer (familiar with

customer)– (people)Impact of cognitive makeup of req candidates using

standard psych tests (Toyota evaluates critical thinking, problem solving, decision making—40 hours of testing prior to interviewing)

– (people)UML never represent customer space, they could, but they don’t—the first one you see is taken as a design model (any time you make an abstract model, it is assumed by others that it is how you plan to design the solution)—how do you convince people that there are abstract models that are useful to define the problem without defining the solution and to treat them as such

18


People– (people)People involved in eliciting and analyzing

requirements Well versed in domain and the trade space for

solutions leads to more effectiveness in eliciting and analyzing requirements

– (people)Having domain knowledge and experience in extracting requirements leads to

19


Management– (management/process)Minimize requirements volatility

by managing decision volatility– (management/process)Link decisions and

requirements– (management)Use models to understand effect of

requirements change on the process– (people/management)Change management and

expectation management– (process/management)Get immediate feedback from

development and test teams that requirements can be implemented and tested

20


Management– (process/management)Write requirements in

alignment/context with business drivers– (process/management)Keeping the multiple

stakeholders engaged across the life cycle– (process/management)Integrate measurement and

modeling of requirements changes– (management/process)Improve/investigate

methodology for requirements development by incorporation of meta models, semantics, and ontology

21


Methodology– (process/methodology)Look at process used to

development requirements, especially across enterprises Modeling and sim Environment (process, tools, people) Instrumentation Increase propensity to accept changes in the

requirements process

22


Organizational– (people/org)How do we convince customers to think in

smaller steps– (people/org)Customers tend to think 5-10 yrs in the

future, others tend to think 9 months ahead– (people/org)Managing stakeholder

continuity/volatility/high variance of req interpretation

23


(citation)Software requirements: the Requirements Set (Ferdinandi) (about categories of requirements)

24

Research ProjectsVc

V: 1,7,2 D:2,7,2 Establish relationships between capabilities and requirements 3

V: 5,6,2 D: 0,4,5 How to support client/customer in requirements elicitation 4– for naïve/non-expert clients– when the elicitor does not have domain knowledge

V: 2,5,5 D: 0,10,0 Define and evaluation a taxonomy of knowledge needed in req elicitation 2

A) V: 11,3,0 D: 0,8,6 Develop improved/standardized methods for measuring and evaluating reqs/reqs changes and evaluate the benefits on experimental projects 7, 22/14=1.57

V: 7,7,0 D: 1,8,4 Investigate use of boundary objects for modeling requirements in collaborative teams 5

V: 5,6,4 D: 0,5,9 Build tools to visualize and investigate the relationships between decisions and requirements, with emphasis on managing changes that are resulting from decision change 5

25

Research Projects (continued)

B)Define set of rules/metrics that can accurately predict/reflect 13 ; D: 0,4,13 21/17=1.24– requirements quality– decision quality– Req traceability with code and design (bi-directional)– Requirements completion– Requirements omissions

Develop checklists/tools help one identify missing reqs 4 Translate English requirements set into a virtual

representation 4 C)Develop knowledge management tool to categorize

requirements and based on this, develop cost estimate/risk predictions 6; D: 0,4,11 19/15=1.27

Make function point models into a useful form of boundary objects 1

26


D) Define a representation and accompanying new tools for reqs that enables automated measurement and analysis 6; D: 0,3,11 17/14=1.21

Investigate team org and roles that are effective at reqs elicitation, analysis, pruning, and prioritization 5

E) Extend USC COCOMO research to relate reqs as inputs to SLOC as output and verify on real projects 6; D:0,7,8 22/15=1.47

Investigate levels of requirements and develop factors to estimate number of reqs at various levels (requirements normalization to get consistent count of reqs at a given level) 4

Investigate the effects of communicating reqs across cultures, especially globally 3

27


Small to medium projects: does a very structured process have an impact on 5– Req quality and req volatility– quality and cost of end product

Define and develop helper for ensuring that behavior for undesirable events and off-nominal conditions are captured in the reqs 1

Tool to identify potential emergent behaviors resulting from requirements 3

F) Develop an approach for costing by capabilities instead of requirements 7; D:0,6,8 20/14=1.42– Want to build a system “like that one” – reqs def by analogy

Investigate feasibility of function points as an alternative to COSYSMO reqs input 2

Define/develop Tools to identify “similar requirements” to enable detect redundancy, conflicts, and opportunities for reuse 5

28


Compare contribution of people vs. process to the quality of the developed requirements 3

Define and develop a way to measure the quality of developed requirements 3

G) Develop meta model, semantics (categorization of terms, naming conventions, nomenclature) and ontologies of requirements 7; D:0,8,6 22/14=1.57

Investigate the efficiencies of various requirements derivation techniques for deriving high quality/complete requirements 3

H) Define the minimum required steps for lean requirements-related processes 7; D:4,9,2 32/15=2.13– Small projects– Large projects

29


Develop tool to capture requirements associated with selected/reused/COTS components 1

Identify preferred methodology for incorporation of legacy requirements into new projects 2

Investigate the extent to which we can simulate the user’s problem space (virtual model)—executable model 3

I) Define a mechanism for capturing reqs using test procedures (test-centric requirements development) 6; D: 0,12,1 25/13=1.92

Define more specific completeness criteria for passing LCO/LCA milestones 4– Investigate continuous milestones with continuous/tightened

criteria Identify and do good write-ups of case studies on outstanding

requirements and processes in industry. 5 Identify best practices for reqs development 4 Identify preferred approach for seamless incorporation of se and sw

eng life cycles, with emphasis on software life cycle within se life cycles 3

30


Define and develop an incremental sys eng life cycle model 3

J) Develop a model to predict reqs volatility 14; D: 0,3,13 19/16=1.19– Based on a given set of reqs, problem domain, and

stakeholders– With emphasis on the velocity with which reqs were

developed, changes made, defects injected and defected

Capture the types and provide predictors for latent req defects 1

31

Top 10 Research Project

Develop a model to predict reqs volatility 14– Based on a given set of reqs, problem domain, and stakeholders– With emphasis on the velocity with which reqs were developed,

changes made, defects injected and defected Define set of rules/metrics that can accurately predict/reflect 13

– requirements quality– decision quality– Req traceability with code and design (bi-directional)– Requirements completion– Requirements omissions

V: 11,3,0 D: 0,8,6 Develop improved/standardized methods for measuring and evaluating reqs/reqs changes and evaluate the benefits on experimental projects 7

Develop knowledge management tool to categorize requirements and based on this, develop cost estimate/risk predictions 6

32

Best Practices:

What are the current best practices for addressing risk related to requirements volatility– Postponement of requirements– Iterative requirements development– Clarity of requirements– Prioritize requirements– Prioritize by feasibility– More agile process– Capture justification and history of requirements– Architect product to withstand change– Better standards for requirements documentation– Agile prototyping– Involve many stakeholders, especially downstream

customers– Hierarchical classifications (Layered—577 MBASE)

33

Best Practices (continued)

Prioritize early and at high levels on the requirements hierarchy Change propagation and traceability Requirements traceability Over-engineer (just in case) and create a resilient architecture that

anticipates change Visible and understandable reqs Measure requirements and volatility Assess and communicate impact to all stakeholders Training

– Processes– Domain

Checklists Make explicit win conditions and incentives of stakeholders Triage requirements—if some reqs change, it does not matter, if

others change, it matters a lot

34

Best Practices

ROI analysis for alternative requirements Use tools Rely on standards to a larger extent Conduct sensitivity analysis up front Categorize requirements, capabilities, project,

performance, interface, evolution, future

35

Ease / Value

Project Letter Difficulty EASE Value:

Research Project desc. main Key Words, Phrases and Concept 3 2 1 TopTen

(starts with) Easy Medium HardWeighted NormalizedVotes

aDevelop improved ... Measure & evaluate; reqts/changes; assess benefits 0 8 6 22 1.57 7

bDefine set of … Rules/Metrics related to reqts. Qualities 0 4 13 21 1.24 13

cDevelop KM tool … Categorize reqts.; cost est.; risk prediction 0 4 11 19 1.27 6

dDefine a representation … Tools; Automated measurement & analysis 0 3 11 17 1.21 6

eExtend USC COCOMO … Reqts to SLOC 0 7 8 22 1.47 6

fDevelop an approach … Costing by capabilities; reqts. definition by analogy 0 6 8 20 1.43 7

gDevelop meta-model, … Meta-models, semantics & ontologies 0 8 6 22 1.57 7

hDefine the minimum … Steps for lean requirements processes 4 9 2 32 2.13 7

iDefine a mechanism … Test-centric reqts. development 0 12 1 25 1.92 6

jDevelop a model … Predict reqts. volatility 0 3 13 19 1.19 14

36

Ease / Value Chart

Requirements Volatility Workshop Rick Selby USC Center for Systems & Software Engineering February...

Documents

Transcript of Requirements Volatility Workshop Rick Selby USC Center for Systems & Software Engineering February...