Iwsm2014 manage the automotive embedded software (alexandre oriou)
Iwsm2014 the effect of highlighting error categories in fsm training on the accuracy of...
description
Transcript of Iwsm2014 the effect of highlighting error categories in fsm training on the accuracy of...
The Effect of Highlighting Error Categories in FSM Training on the Accuracy of Measurement
ALI MERT ERTUGRUL*, GOKCEN YILMAZ*, MURAT SALMANOGLU*, ONUR DEMIRORS*
* Department of Information Systems, Middle East Technical University – Ankara / TURKEY
October 6-8, 2014
International Workshop on Software Measurement and International Conference on Software Process and Product Measurement – IWSM / MENSURA 2014Rotterdam, the NETHERLANDS
3
OUTLINE
• Introduction to FSM• Related Work on FSM accuracy• Motivation• Research Methodology and Implementation• Results & Discussion• Conclusion & Future Work
07/10/2014
4
Functional Size Measurement
• Functional Size Measurement (FSM) is a technique for measuring software in terms of the functionality it delivers.• IFPUG, MkII, COSMIC, etc.• FSM accuracy is crucial since based on the size of
the software, the cost and effort estimation are done.
07/10/2014
5
Variance in FSM
• Two main reasons:• Different assumptions and interpretations
• Misunderstanding of measurement standards• Solution: COSMIC Training
Turetken et al. 2008
Ungan et al. 2009
07/10/2014
6
FSM Training on FSM Accuracy
• Educational framework for the teaching and learning of software measurement topics at the undergraduate level.• Guidelines to teachers and instructors for
promoting the learning of software measurement topics.• Besides FSM, productivity, customer satisfaction,
etc.
Villavicencio et al. 2013
07/10/2014
7
FSM Training on FSM Accuracy
• Conducted two experiments:• Standard COSMIC training• Standard COSMIC training with practical cases
• The latter increased the reliability and accuracy of FSM more.
• Errors were not the prior concern in practical examples.
Ungan et al. 2010
07/10/2014
8
Definition of Error Categories
• Errors made during COSMIC FSM are categorized.
• Proposed tool detects the measurement errors automatically.
• Does not aim to increase FSM accuracy via training.
Yilmaz et al. 2013
07/10/2014
9
Motivation
• COSMIC FSM accuracy increases with training.
• Frequent error categories are defined.
• We argued, whether highlighting the error categories during FSM training increases the accuracy of FSM.
07/10/2014
10
Error Categories (EC) Name
EC1 Duplicate Functional Process (FP)
EC2 Lack of List FP before Update FP
EC3 Lack of List FP before Delete FP
EC4 Lack of Retrieve FP before Update FP
EC5 Lack of Data Movement (DM) type Write (W) in Add, Delete and Update FPs
EC6 Redundant DM type W in List FPs
EC7 Multiple occurrences of the same DM within the same FP
EC8 Each FP should be composed of at least 2 DMs
EC9 Each FP should contain at least 1 Write (W) / Exit (X) DM
EC10 Each FP should contain at least 1 Entry (E) DM
EC11 List FP might be included in Update/Delete FPs
EC12 Create/Delete/Update operations might be combined
EC13 Data Group (DG) Duplication
EC14 User interface components and System users are considered as DG / Object Of Interest (OOI)
EC15 OOIs are named wrong
07/10/2014
11
Data Acquisition
• We analyzed COSMIC FSM assignment data:• Obtained from two courses (SM502 and IS529)• Four consequent years (2010, 2011, 2012 and 2013)• Given by the same instructor for four years and two
courses• Similar MIS projects• 44 measurement results (10, 11, 11 and 12)
07/10/2014
12
Highlighting Error Categories
• Measurer teams were:• Unexperienced about COSMIC FSM before the given
courses.• Given COSMIC FSM training without being warned about
Error Categories in 2010 and 2011.• Given COSMIC FSM training in which Error Categories
were highlighted with detailed examples in 2012 and 2013.
07/10/2014
13
Expert Review Process
• Reference keys were prepared for each project.
• Two experienced measurers evaluated the results.
• An iterative review process was followed.
07/10/2014
14
Total and Average # of Errors 2010 – 2013EC
Total number of errors for years
Average number of errors for years
2010 2011 2012 2013 2010 2011 2012 2013EC1 60 18 19 14 6 1,6 1,8 1,2EC2 17 57 5 1 1,7 5,2 0,5 0,1EC3 23 69 6 5 2,3 6,3 0,5 0,4EC4 20 20 6 10 2 1,8 0,5 0,8EC5 26 11 3 6 2,6 1 0,3 0,5EC6 3 0 0 0 0,3 0 0 0EC7 19 81 4 6 1,9 7,4 0,4 0,5EC8 0 0 0 1 0 0 0 0,1EC9 19 2 1 0 1,9 0,2 0,1 0
EC10 7 4 2 2 0,7 0,4 0,2 0,2EC11 25 38 4 9 2,5 3,5 0,4 0,8EC12 3 1 0 7 0,3 0,1 0 0,6EC13 N/A N/A N/A N/A N/A N/A N/A N/AEC14 330 63 13 7 33 5,7 1,2 0,6EC15 66 33 7 10 6,6 3 0,6 0,8
07/10/2014
1507/10/2014
16
Results & Discussion
• Majority of the ECs has a significant decrease in total and average number of errors.• Related to List FP:• EC2: Lack of List FP before Update FP• EC3: Lack of List FP before Delete FP• EC11: List FP might be included in Update/Delete FPs
• Related to Naming:• EC14: User interface components and System users
are considered as DG / Object Of Interest (OOI)• EC15: OOIs are named wrong
• Also, EC5 and EC7
07/10/2014
17
Results & Discussion• Some of the ECs do not have a significant decrease in
total and average number of errors.• Core measurement rules:
• EC8: Each FP should be composed of at least 2 DMs• EC9: Each FP should contain at least 1 Write (W) / Exit
(X) DM• EC10: Each FP should contain at least 1 Entry (E) DM
• Naturally correct:• EC6: Redundant DM type W in List FPs• EC12: Create/Delete/Update operations might be
combined• Analysis on the errors does not contain EC13.
• Students were not responsible for EC13 during measurements
07/10/2014
18
Conclusion
• Frequent errors during COSMIC FSM were analyzed.• In the first two years COSMIC FSM training did not
include Error Categories.• Error categories were highlighted in detail during
COSMIC FSM training in the second two years.• Sharp decrease was observed in ECs in the second two
years.• Proposed study shows that highlighting error categories
in the course curriculum increases the FSM accuracy remarkably.
07/10/2014
19
Future Work
• Increase sample size• Apply enhanced statistical analysis (e.g. variance)• Combine with error prevention and identification
methods and tools• Categorize errors to be prevented via training or
tools
07/10/2014
20
References• O. Turetken, O. O. Top, B. Ozkan, and O. Demirors, “The Impact of Individual Assumptions on
Functional Size Measurement,” IWSM/Metrikon/Mensura, volume 5338 of Lecture Notes in Computer Science, pp. 155–169, 2008.
• E. Ungan, O. Demirörs, and Ö. Ö. Top, “An Experimental Study on the Reliability of COSMIC Measurement Results.”, IWSM '09 /Mensura '09 Proceedings of the International Conferences on Software Process and Product Measurement, pp. 321-336, 2009
• O. O. Top, O. Demirors, and B. Ozkan, “Reliability of COSMIC Functional Size Measurement Results: A Multiple Case Study on Industry Cases,” 35th Euromicro Conf. Softw. Eng. Adv. Appl., pp. 327–334, 2009.
• E. Ungan, Ö. Ö. Top, B. Özkan, and O. Demirörs, “Evaluation of Reliability Improvements for COSMIC Size Measurement Results,” IWSM/MetriKon/Mensura, 2010.
• The Common Software Measurement International Consortium (COSMIC): 2011 Guideline for Assuring the Accuracy of Measurements, Version 1.0.
• M. Villavicencio and A. Abran, “Towards the Development of a Framework for Education in Software Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int. Conf. Softw. Process Prod. Meas., pp. 113–119, Oct. 2013.
• G. Yilmaz, S. Tunalilar, and O. Demirors, “Towards the Development of a Defect Detection Tool for COSMIC Functional Size Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int. Conf. Softw. Process Prod. Meas., pp. 9–16, Oct. 2013.
07/10/2014
21
Thank You!
07/10/2014