Test Effort Estimation What is Estimation? 6th Sep, 2011 Chemuturi Consultants, 1.
-
Upload
janel-jefferson -
Category
Documents
-
view
217 -
download
0
Transcript of Test Effort Estimation What is Estimation? 6th Sep, 2011 Chemuturi Consultants, 1.
Test Effort Estimation
• What is Estimation?
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com1
Test Effort Estimation
• Definition of Estimation– Estimation is defined as the intelligent anticipation of
the quantum or work that needs to be performed and the resources (namely, human resources, monetary resources, equipment resources and time resources) required to perform the work at a future date in a defined environment using specified methods.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com2
Test Effort Estimation
• Definition of Test Effort Estimation– Test Effort Estimation is that estimation of the testing size,
testing effort, testing cost and testing schedule for a specified software test project in a specified environment using defined methods, tools and techniques.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com3
Test Effort Estimation
• Now the key words are – – Estimation – defined earlier– Testing Size – the amount (quantity) of testing that
needs to be carried out– Testing Effort – the amount of effort in either person
days or person hours necessary for conducting the tests – Testing Cost – the expenses necessary for testing,
including the expense towards human effort– Testing Schedule – the duration in calendar days or
months that is necessary for conducting the tests
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com4
Test Effort Estimation
• EstimationStandard Time – is defined as the amount of time taken to accomplish
a unit of work in a defined environment by a qualified worker, after adjustment, at a pace that can be maintained day after day without any harmful physical effects
Estimated Time - the time / effort that is expected to be consumed for accomplishing a defined amount of work in a defined environment
Actual Time - the time / effort that is actually consumed for actual amount of work in the actual environment
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com5
Test Effort Estimation
• Remember, an “exact/accurate estimate” is an oxymoron (“credible estimate” is a better phrase)
• Estimate how long will it take you to place to another place– On what basis did you do that?– Experience right?– Likely as an “average” probability– For most software projects there is no such ‘average’
• Most estimations are off by 25-100%
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com6
Test Effort Estimation
• Why testing?– Uncover all defects– Assure the customer that the product is built
conforming to his/her specs/requirements– Certify the product – IV&V
• No virus / No spyware / No malware • Works as specified• Safe & secure• Any other objective
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com7
Test Effort Estimation
• Test Effort Estimation– Issues in Test Effort Estimation– Estimate testing project size as a percentage of
Software size to be tested and derive effort– Convert Software Size into Testing Project size and
then convert it into testing effort– Test Case Enumeration– Task Based Estimation – STU Estimation– Delphi– Analogy
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com8
Test Effort Estimation
• Issues in Test Effort Estimation– Who Needs Test Effort Estimation? - There
are four categories of people who would like to carry out test estimation –
• The project Team who obviously have to carry out testing to ensure that their work is satisfactory and they can certify it so and pass it on to professional independent testing team. This set would carry out all the tests necessary
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com9
Test Effort Estimation
• Issues in Test Effort Estimation– Who Needs Test Effort Estimation?
The in-house Software Quality Assurance (SQA) Team that would test and certify the product for customers. Their objective is to unearth all defects and make the product as defect proof as possible while ensuring that the product meets all the specified functionality.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com10
Test Effort Estimation
• Issues in Test Effort Estimation– Who Needs Test Effort Estimation?
Testing Organizations, whose mainline of business isto test other’s software and certify the products.
Their objective is to ensure that the product meets the end-user expectations. This set would carry out –
a. Mainly Black Box testingb. Functional Testingc. System Testingd. Negative Testinge. Regression Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com11
Test Effort Estimation
• Issues in Test Effort Estimation– Who Needs Test Effort Estimation?
Customers who entrusted their software development to a vendor. Their objective is to ensure that they are getting what they are paying for. This set would carry out
a. Acceptance Testing / Positive Testingb. Regression Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com12
Test Effort Estimation
• Issues in Test Effort Estimation– The type of test effort estimation carried out by
sets would be different.1. Project Team – since they carry out testing as part of
development work – it may be taken as a percentage of overall development work. Their purpose would be mainly to estimate resource requirement and meeting the delivery schedule
2. In-house SQA Team – they need to estimate mainly for the purpose of resource estimation and meet the delivery schedule
3. Testing Organizations – they need to estimate so that they can offer a fixed price to the customers and manage a healthy profit for their organizations besides resource estimation and meeting delivery schedule. For them “Size” of testing project would have significant appeal.
4. Customers – They really are supported in this activity of acceptance testing by the vendor organization.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com13
Test Effort Estimation
• Issues in Test Effort Estimation– Types of Tests conducted
1. Unit Testing2. Integration Testing & Incremental Integration
Testing3. System Testing4. User Acceptance Testing5. Volume Testing6. Load Testing7. Functional Testing8. End to end Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com14
Test Effort Estimation
• Issues in Test Effort Estimation– Types of Tests conducted
9. Parallel Testing10. Concurrent Testing11. Stress Testing12. Positive Testing13. Negative Testing14. Deployment Testing15. Sanity Testing16. Regression Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com15
Test Effort Estimation
• Issues in Test Effort Estimation– Types of Tests conducted
17. Security Testing18. Performance Testing19. Usability Testing20. Install / Uninstall Testing21. Compatibility Testing22. Intuitive Testing23. Comparison Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com16
Test Effort Estimation
• Issues in Test Effort Estimation– Types of Tests conducted
24. Alpha Testing25. Beta Testing26. GUI Testing / navigation Testing
• How of Testing• White Box Testing• Black Box testing• Product Testing• Project Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com17
Test Effort Estimation
• Testing Project– Testing as part of Development projects
• Embedded Continuous Testing• Testing at the end of the development
– Independent Verification and Validation Testing
• What tests are included in Testing project?– Often specified– Often not specified !!– Quantum of work depends upon the tests included
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com18
Test Effort Estimation
• Test Strategy– First Step in Test Effort Estimation– Includes
• Testing Objectives– Unearth all defects– Unearth all possible defects within the given time– Obtain Customer acceptance sign-off– Certify the product
» Virus & Spy-ware free» Function, security, and so on
– So on
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com19
Test Effort Estimation
• Test Strategy– Includes
• Types of testing to be carried out• How testing is done
– Intuitive– Test Plan and Test Case based– White box or Black Box– Manual Testing or Tool based Testing
• Testing Environment– How many?
• Criteria for completing Testing• Regression Testing
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com20
Test Effort Estimation
• Test Strategy– Includes
• Defect Closure and Escalation mechanism• Progress Reporting• Defect Analysis
– ABC analysis– Defect Category Analysis
• Now we are ready for test effort estimation !
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com21
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– One way is to compute software size to Test Effort
• Let us say one FP needs 2 hours of testing– If the project size is 100 FP then Testing Effort is 200 Person Hours
• Let us say one UCP needs 4 hours of testing– If the project size is 100 UCP then Testing Effort is 400 Person Hours
– Simple to use– There are no benchmarked data available – therefore,
organizational data needs to be developed and maintained– Norms to be developed and maintained–
• No of FP (or UCP) for one hour of User Acceptance Testing• No of FP (or UCP) for one hour of Integration Testing• No of FP (or UCP) for one hour of System Testing• Etc.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com22
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– Demerits
• Too simplistic• Testing Size is not available in Software Test Units – hence deriving
productivity is based on software size• Amount of testing differs from project to project – the variance can be very
high• Organization has to keep meticulous records and strictly adhere to process
to obtain reliable figures• Time sheet and data collection have to be rigorous
– Merits• Facilitates arriving testing effort for any individual or combination of tests• Very quick• Can be surprisingly accurate – if the norms are well derived and maintained
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com23
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– One way is to convert software size to Software Test Units (STU)
• Let us say one FP results in 5 STU– If the project size is 100 FP then Testing Size is 500 STU
• Let us say one UCP results in 8 STU– If the project size is 100 UCP then Testing Size is 800 STU
• Now convert the STU into effort– Let us say each STU consumes 30 minutes then the project consumes -
» 500 * 0.5 = 250 Person Hours» 800 * 0.5 = 400 Person Hours
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com24
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– We need to define a STU– Maintain meticulous records to derive and establish relationship
between Project in FP / UCP and TP
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com25
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– Demerits
• Too simplistic• Need to compute the software size first (for IV & V projects) in FP
or UCP• If the software size is not computed in-house – then the software
size itself may be in question• Organization has to keep meticulous records and strictly adhere to
process for deriving relationship• Time sheet and data collection have to be rigorous
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com26
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size– Merits
• Testing Size is defined in STU – hence facilitates deriving Testing Productivity
• Very quick• Can be surprisingly accurate – if the norms are well derived and
maintained
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com27
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration– Enumerate the test cases– Estimate testing effort for each test case– Use Best Case, Normal Case and Worst Case scenarios for
estimating effort needed for each test case– Compute Expected Effort for each case using Beta Distribution
• Best Case + Worst Case + (4 * Normal Case) / 6
– Sum up the –• Expected times to get Expected effort estimate for the project• Best Case times to obtain best case effort estimate• Worst Case times to obtain worst case effort estimate• Normal Case times to obtain normal case effort estimate
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com28
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration– Best Case estimate and the worst case estimate give the range
of effort estimate for the project– We can use Excel / MS Project to define predecessors and
successors, and build a network diagram – this facilitates arriving at the project duration and computation of probability
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com29
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration– Demerits
• No Testing size – hence productivity can not be derived• All test cases and attendant overheads need to be enumerated
– Merits• Auditable estimate• Fairly accurate• Probability can be computed• Progress monitoring is facilitated
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com30
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration– Best Case estimate and the worst case estimate give the range
of effort estimate for the project• Standard Deviation for a test case is –
– (Worst Case - Best Case)/ 6
• Variance for an a test case is the square of its standard deviation
• Standard deviation (ST) for the entire project is –– Square Root of (sum of variances of all critical a test case)
– We can use these for computing probability of success of project
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com31
Test Effort Estimation
• Estimation – Activity/Task -based Estimation– This is concerned with identifying the tasks (Activities) to be
performed to complete a task and estimating effort required to complete each of them
– Steps are –• Identifying the component tasks in each phase• Estimating the effort for each task• Calculating the effort for each phase• Consolidating the effort for the entire project
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com32
Test Effort Estimation
• Estimation – Activity/Task -based Estimation• Phases in a Testing project
– Project Initiation– Project Planning– Specifications Analysis– Test Strategy Design– Test Planning– Test Case Design– Testing and Defect reporting – Regression Testing and defect closure– Documentation– Project closure
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com33
Test Effort Estimation
• Estimation – Activity/Task -based Estimation• List out all the tasks in each of these phases• Eg – Specifications Analysis
• Study proposal and Order• Document understanding gaps• Obtain Clarifications
– Test Strategy Design• Define Testing Objectives• Decide the type and number of tests• Decide the Test Environment• Define Testing Completion Criteria• Define defect reporting and closure mechanism• Define Regression testing mechanism• Define Tools usage
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com34
Test Effort Estimation
• Estimation – Activity / Task-based Estimation– Assign durations for each of the Tasks– Use three time estimates – Best Case, Worst Case and Normal Case– Compute the expected time using formula – (Best Case + Worst Case + 4* Normal Case)/6– Make adjustments for project complexity, familiarity with the platform, skill of
the developers, tools usage– Make adjustments using heuristics – (e.g. Requirements 10%, Design 25%,
Construction 40%, Testing 25%)– Use Delphi technique to validate the estimate– Sum up the total effort estimate of the project
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com35
Test Effort Estimation
• Testing Size Estimation– Why is size necessary?
• Productivity computation• Measure performance• Agree with Client about the quantity of work being performed• Monitor progress
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com36
Test Effort Estimation
• Testing Size Estimation– What are the units of measure available for measuring testing
size?• STU are being discussed• But no universally accepted definition is available on what
Constitutes a STU• The definition is just escaping STU !!
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com37
Test Effort Estimation
• What is my definition of STU?– STU should cater to all combinations of Tests– Be it a unit test or be it acceptance test, be it system test – or
any other test – STU should be able to measure its size– What is common to all types of tests is – the Test Case
• STU is a size measure for measuring the size of a software-testing project and that a STU is equivalent to a normalized test case.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com38
Test Effort Estimation
• What is my definition of STU?– What is a test Case?
• Test Case provides an input to the system which would produce a response from the system. The response could be –
– Store the data– Produce derived / un-derived data– A message
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com39
Test Effort Estimation
• STU Estimation approaches– Derive from the size Estimate– Bottoms Up approach
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com40
Test Effort Estimation
• STU Estimation approaches – EstimatorPal Approach– The Method of computing STUs
1. This method uses an existing software size in FP/UCP etc2. It converts development project size into Unadjusted STUs (UTP)
using a conversion factor which can be modified by the user to suit his environment
3. It computes a Composite Weight Factor (CWF)a. Sum up all individual weights of included testsb. Multiply it by the weight of the application weightc. Multiply it by the language weight if Unit Testing is selectedd. Multiply it by Tools Weight if Tools Usage is selected
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com41
Test Effort Estimation
• STU Estimation4. UTP are multiplied by CWF to obtain the testing size in STUs
size5. The Productivity Factor indicates the number of person hours
per one STU6. Testing Effort in Person hours is computed by multiplying
STU Size by the Productivity Factor.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com42
Common Pitfalls in Estimation
• Inexperienced people carrying out estimation activity• Lack of training• Lack of historical data• Absence of tools• Over-estimation and under-estimation• Inadequate time for estimation• Lack of review
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com43
Common Pitfalls in Estimation
• Lack of causal analysis for deviations in actual Vs. estimates
• Productivity Figures - no focus• Application of one productivity figure for conversion
from size to effort
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com44
Over and Under Estimation
• Over estimation issues– The project will not be awarded
• Conservative estimates guaranteeing 100% success may mean winning probability of zero.
– Parkinson’s Law: Work expands to take the time available for its completion
– Danger of feature and scope creep– Be aware of “double-padding”: team member + manager
• Under estimation issues– Quality issues (short changing key phases like testing)– Inability to meet deadlines– Morale and other team motivation issues
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com45
Estimation Good Practices
• Experienced people carry out estimation• Process is defined for estimation• Estimation guidelines are defined and available• Productivity figures are derived and maintained up-
to-date• Estimators are trained• Estimation tools are available
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com46
Estimation Good Practices
• Repository of estimates are maintained and are made available to estimators when needed
• Expert guidance is available• All estimates are subjected to peer review as well as
managerial review• Actual Vs. Estimate analysis (postmortem) is always
carried out and causal analysis is carried out and norms of estimation updated
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com47
Estimation Guidelines
• Estimate iteratively!– Review two heads are better than one– Make your best estimates at each planning stage– Refine estimates and adjust plans iteratively– Plans and decisions can be refined in response– Balance: too many revisions vs. too few
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com48
Estimation “Presentation”• How you present the estimation can have huge
impact• Techniques
• Plus-or-minus qualifiers• Ranges• Risk Quantification
– +/- with added information– +1 month of new tools not working as expected– -2 weeks for less delay in hiring new developers
• Cases– Best / normal / Worst cases
• Confidence Factors– April 1 – 10% probability, July 1 – 50%, etc.
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com49
Other Estimation Factors
– Account for resource experience or skill• Up to a point• Often needed more on the “low” end, such as for a
new or junior person
– Allow for “non-project” time & common tasks• Meetings, phone calls, web surfing, sick days
– There are commercial ‘estimation tools’ available
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com50
Other Estimation Factors
– Please download our tool TestPal• Test Case Design assistance• Test Result logging• Test Effort Estimation
– STU– Task Based– Test Case Enumeration
– www.chemuturi.com/TestPal-Setup.exe
–Thank you Very much
6th Sep, 2011Chemuturi Consultants,
www.chemuturi.com51