Project Estimates Are probabilistic Statements
description
Transcript of Project Estimates Are probabilistic Statements
Project Estimates Are probabilisticProject Estimates Are probabilisticStatementsStatements
Martin Shepperd
Brunel Software Engineering Research Centre (B-SERC)
Brunel [email protected]
people.brunel.ac.uk/~csstmms/
Slide Slide 22: : September 27, 2005September 27, 2005
OverviewOverview
1. Background
2. Expert judgement
3. Group techniques
4. Observing groups
5. Discussion
Slide Slide 33: : September 27, 2005September 27, 2005
1. Background1. Background
Software developers need to predict, e.g.
effort, duration, number of features
defects and reliability
But ...
little systematic data
noise and change
complex interactions between variables
poorly understood phenomena
Slide Slide 44: : September 27, 2005September 27, 2005
Current ApproachesCurrent Approaches
Expert judgement
Off-the-shelf parametric models e.g. COCOMO, etc.
Develop a (simple) local model e.g. LSR
“Zingy” methods
Combination
Slide Slide 55: : September 27, 2005September 27, 2005
Some Causes of Poor EstimationSome Causes of Poor Estimation
We don’t cope with political
problems that hamper the
process.
We don’t develop estimating
expertise.
We don’t systematically use
past experience. Tom DeMarcoControlling Software Projects. Management, Measurement and Estimation. Yourdon Press: NY, 1982.
Slide Slide 66: : September 27, 2005September 27, 2005
So What is an Estimate?So What is an Estimate?
An estimate is a prediction based upon probabilistic assessment.
p
effort0
most likely
equal probability of under / over estimate
Slide Slide 77: : September 27, 2005September 27, 2005
Points or Distributions?Points or Distributions?
Alternatives to point values include:
– 3 point estimates
– range and confidence level e.g. 90% confident that the duration will be between 10 and 15 weeks.
Slide Slide 88: : September 27, 2005September 27, 2005
Over confidenceOver confidence
Studies report that many experts over estimate their confidence levels / select too narrow a range.
Experience can exacerbate the problem.
Also asymmetric distribution of errors.
Slide Slide 99: : September 27, 2005September 27, 2005
Useful ResponsesUseful Responses
Training (including probability) and feedback.
Experience can exacerbate the problem.
Also asymmetric distribution of errors.
Slide Slide 1010: : September 27, 2005September 27, 2005
( Defining “Good” Prediction )( Defining “Good” Prediction )
Many criteria to judge a prediction system:
- accuracy (centre, bias, spread)
- ease of use
- explanatory value
- sensitivity
- demand for training data
- adaptability
- etc.
Slide Slide 1111: : September 27, 2005September 27, 2005
2. “Expert Judgement”2. “Expert Judgement”
Most widely used estimation technique
No consistently “best” prediction system
Lack of historical data
Need to “own” the estimate
Experts plus … ?
Slide Slide 1212: : September 27, 2005September 27, 2005
Expert Judgement Expert Judgement DrawbacksDrawbacks
BUT Lack of objectivity
Lack of repeatability
Lack of recall /awareness
Lack of experts!
Preferable to use more than one expert.
Preferable to use more than one expert.
Slide Slide 1313: : September 27, 2005September 27, 2005
What do We Know About Experts?What do We Know About Experts?
Most commonly practised technique.
Dutch survey revealed 62% of estimators used intuition supplemented by remembered analogies.
UK survey - time to estimate ranged from 5 minutes to 4 weeks.
US survey found that the only factor with a significant positive relationship with accuracy was responsibility.
Slide Slide 1414: : September 27, 2005September 27, 2005
Information UsedInformation Used
Design requirements
Resources available
Base product/source code (enhancement projects)
Software tools available
Previous history of product
... Bob Hughes, Information & Software Techmology 38(2) pp67-75
Slide Slide 1515: : September 27, 2005September 27, 2005
Information NeededInformation Needed
Rules of thumb
Available resources
Data on past projects
Feedback on past estimates
...
Slide Slide 1616: : September 27, 2005September 27, 2005
FeedbackFeedback
fr lm none reviews vl
2
4
6
FEEDBACK
Key:
fr - final report
lm - line management
none
reviews
vl - very little
Slide Slide 1717: : September 27, 2005September 27, 2005
3. Group Techniques3. Group Techniques
“Two” experts are better than one synergy coverage transfer expertise group “ownership”
Slide Slide 1818: : September 27, 2005September 27, 2005
Other Group TechniquesOther Group Techniques
Brainstorming - to identify full list of possibilities e.g. opportunities / threats
Nominal group techniques - silent generation of ideas, round robin sharing of one idea at a time. Each idea then discussed. Spontaneous evaluation. Silent voting.
Slide Slide 1919: : September 27, 2005September 27, 2005
Delphi TechniquesDelphi Techniques
Methods forstructuring group communication processes
tosolve complex problems.
Characterised byiterationanonymity
Devised by Rand Corporation (1948). Refined by Boehm (1981).
Slide Slide 2020: : September 27, 2005September 27, 2005
Stages for Delphi ApproachStages for Delphi Approach
1. Experts receive spec + estimation form
2. Discussion of product + estimation issues
3. Experts produce individual estimate
4. Estimates tabulated and returned to experts
5. Only expert's personal estimate identified
6. Experts meet to discuss results
7. Estimates are revised
8. Cycle continues until an acceptable degree of convergence is obtained
Slide Slide 2121: : September 27, 2005September 27, 2005
Wideband Delphi FormWideband Delphi Form
Project: X134 Date: 3/10/05
Estimator: MJS
Estimation round: 1
0 10 20 30 40 50
x x* x x! x x x
Key: x = estimate; x* = your estimate; x! = median estimate
Slide Slide 2222: : September 27, 2005September 27, 2005
Delphi ObjectivesDelphi Objectives
Obtain informed judgement
Consensus judgement (not essential)
Educate participants, especially if multi-disciplinary / diverse inputs
Devised by Rand Corporation (1948). Refined by Boehm (1981).
Slide Slide 2323: : September 27, 2005September 27, 2005
4. Observational Study4. Observational Study
Many text books refer to Delphi as a prediction technique.
Unaware of any published studies.
So, an empirical study but not a formal experiment.
Quantitative and qualitative data.
Slide Slide 2424: : September 27, 2005September 27, 2005
Observing Delphi GroupsObserving Delphi Groups
Four groups of MSc student (replicated at UMD - thank you Shari Pfleeger)
Developing a C++ prototype for some simple scenarios
Requested to estimate size of prototype (number of delimiters)
Initial estimates followed by 2 group discussions
Recorded group discussions plus scribes
Slide Slide 2525: : September 27, 2005September 27, 2005
Delphi Size Estimation Delphi Size Estimation ResultsResults
Estimation Mean Median Min Max
Initial 371 160.5 23 2249Round 1 219 40 23 749Round 2 271 40 3 949
Absolute residuals
Slide Slide 2626: : September 27, 2005September 27, 2005
Converging Converging GroupGroup
0
50
100
150
200
250
300
350
400
450
Initial Size Round1 Size Round2 Size
Series1Series2Series3
Group A
Slide Slide 2727: : September 27, 2005September 27, 2005
A Dominant A Dominant IndividualIndividual
0
500
1000
1500
2000
2500
3000
Initial Size Round1 Size Round2 Size
Series1Series2Series3
Group D
Slide Slide 2828: : September 27, 2005September 27, 2005
Qualitative AnalysisQualitative Analysis
What, how and when
Group behaviour and attitudes
Differences between “successful” and “unsuccessful” groups
Slide Slide 2929: : September 27, 2005September 27, 2005
AssumptionsAssumptions
Exploring assumptions was a major preoccupation for 3 out of 4 groups (A, B and D)
Most frequent as scoping (early in discussion) or triggered by comparison with other estimates
J: “Can we reuse stuff?”
J: “Assume we have access to a dbms.”
N: “We've made different assumptions. Lots of estimates are 1500.”
Slide Slide 3030: : September 27, 2005September 27, 2005
ComparingComparing
Unsurprisingly comparing estimates was also a major preoccupation for all groups.
Range of reactions - defensive / open
I: “How is the median 275? What are the others thinking?”
Discussion as to whether to use the estimates from outside the group.
All a little surprised by the other estimates … J begins to question some of his assumptions.
Slide Slide 3131: : September 27, 2005September 27, 2005
Estimation HeuristicsEstimation Heuristics
Decide to start with code size estimation task as easier.
However, AJ then began to try to justify his own estimate, using an essentially bottom-up approach.
N: “What about the limits of the time scale, we only have 3 days.”
J: “TD [lecturer] says to add 20% to any estimate”
Slide Slide 3232: : September 27, 2005September 27, 2005
SizingSizing
All groups tried to estimate by using different units of size to LOC.
N: “We could use count of button/text boxes to generate estimates.”
AJ then tried to estimate the size of product in terms of classes and variables. He estimated 30 variables and 8 to 9 classes.
The technique used was again to think in terms of an estimated number of functions and then use this to gain a figure for lines of code.
Slide Slide 3333: : September 27, 2005September 27, 2005
AnalogiesAnalogies
Groups A and B made extensive use of a range of analogies
Group identify assignment from M [lecturer] as another source of data. Main driver was about 80 LOC.
B: “it was a bit like the linked list assignment and a bit more”.
AJ: ... in his experience the rate at which he could write code (i.e., code that worked), was about ten statements in 5 hours.
Slide Slide 3434: : September 27, 2005September 27, 2005
Information NeedsInformation Needs
Relatively few comments about the need for additional information. Perhaps resigned or perhaps due to the teaching context.
What would be really useful would be to know how the other groups had arrived at their figures. In other words they wanted to gauge the strength of their arguments.
J: can't get a very detailed breakdown even for low complexity project.
Slide Slide 3535: : September 27, 2005September 27, 2005
FollowingFollowing
Groups B and D exhibited “following” behaviour.
J: acknowledges they are far from expert, and so could be unduly influenced by other people's figures/suggestions.
AJ appeared completely unconvinced by this, but said that he was “not prepared to stand by the [his] figures”.
They both seemed to be happy to be persuaded by others, allowing for their own inexperience (as if the others were somehow collectively more experienced).
Slide Slide 3636: : September 27, 2005September 27, 2005
DominatingDominating
Groups C and D exhibited “dominating” behaviour.
I: gave a figure ... other stated whether they were higher or lower. He then said how he had arrived at the figure, the others approved his method.
J: gave his methods for working out the estimate, again general approval (did not discuss why their figures were different).
The experienced team member [E] said he has never worked on something that took longer than three months. The non-experienced didn’t even ask him if he worked on a group project.
Slide Slide 3737: : September 27, 2005September 27, 2005
Co-operatingCo-operating
Groups A and C most obviously exhibited co-operation.
N: Don't feel pressurised [to change estimate]
Slide Slide 3838: : September 27, 2005September 27, 2005
Differences Between Differences Between GroupsGroups
Groups A, B and C improved their estimates but Group D diverged.
A, C and D all had “natural” leaders but D was allowed to dominate.
Other members of D tended towards apathy.
Group D make least overt use of analogies, sizing or estimation heuristics other than contingency.
Group D appeared least confident.
Slide Slide 3939: : September 27, 2005September 27, 2005
5. Discussion5. Discussion
In general, group estimating (a variant of Delphi) improved the accuracy of size estimates.
Subjects tended to gain in confidence (pre and post questionnaires).
But, a small study using students and a relatively constrained estimation task.
Slide Slide 4040: : September 27, 2005September 27, 2005
Negative FactorsNegative Factors
One group (D), were led away from the true figure (repeated at UMD).
The other members allowed one individual to dominate.
The poorest predictions were characterised by least overt use of heuristics and sizing and, encouragingly least, confidence.
Group D were defensive when comparing their estimates with those of others.
Slide Slide 4141: : September 27, 2005September 27, 2005
Experts plus … ?Experts plus … ?
Experiment by Myrtveit and Stensrud using project managers at Andersen Consulting
Asked subjects to make predictions
Found expert+tool significantly better than either expert or tool alone.
?What type of prediction systems are easiest to collaborate with?
Slide Slide 4242: : September 27, 2005September 27, 2005
Future AvenuesFuture Avenues
Great need for useful prediction systems
Consider the nature of the prediction problem
Combining prediction systems
Collaboration with experts
Managing with little or no systematic data