Validation of the Method Adoption Model for Functional Size Measurement of Web Applications

Validation of the Method Adoption Validation of the Method Adoption Model for Functional Size Model for Functional Size Measurement of Web ApplicationsMeasurement of Web Applications

Silvia AbrahãoValencia University of Technology,

Spain

[email protected]

Geert PoelsGhent University, Belgium

[email protected]

2nd Workshop on Software Metrics ValidationGhent, July 06, 2004

Ghent, July 06, 2004QAOOSE 2004

Contents

Functional Size Measurement (FSM) How to evaluate FSM methods?

A Process Model for FSM

A Theoretical Model for Evaluating FSM Methods

A Laboratory Experiment to test the Theoretical

Model

Summary of Findings

Next Steps


Functional Size Measurement

Functional size measurement (FSM) methodsDefine functional size measuresDescribe how to apply functional size measurement

New FSM methods for new technologies E.g. OO-Method Function Points (OOmFP) and its extension to

the Web (OOmFPWeb)

No methodology for systematic evaluation of FSM methods

Little evidence of validity of functional size measures


How to evaluate FSM methods?OOmFPWeb: A FSM method for Web Applications

OOmFPWeb Measurement Procedure


Possibly existing solutions

Theoretical validation Verification of empirical and numerical properties

using Measurement Theory Conformity evaluation to ISO/IEC 14143-1:1998

International Standard for FSM concepts

Empirical validation Demonstrate the relationship with relevant

software process and management variables Evaluate performance properties of FSM method

(e.g. using ISO/IEC TR 14143-3:2003) Pragmatic approach: method’s success in practice


Our position

Comprehensive evaluation framework neededMulti-perspective evaluationSystematic evaluation

Based on software measurement process modelDesign => Application => Analysis => Use

Based on multi-disciplinary approachMeasurement Theory IT / IS method evaluation modelsEmpirical software engineering


A Process Model for FSM

Validation of the design of the FSM method

Validation of the application of the FSM method

Validation of the models thatuse the FSM method results

Includes measurevalidation

Includes evaluationof performance-basedand perception-basedproperties of the method

Includes evaluationof model properties(e.g. predictionaccuracy, causality)

MeasurementTheoretic approach

Evaluation model forFSM methods

Empirical softwareengineering approach

Application of the measurement method rules

Design of the Measurement

method

Measurement result

Explotation of the measurement method result

Step 1 Step 2 Step 3 Step 4

(Jacquet and Abran, 1997)


Method should enable task to be performed faster, more cheaply or with less effort (efficiency) and/or improve the quality of the result (effectiveness)

Efficiency is defined by the effort required to apply the method.

Effectiveness is defined by how well the method achieves its objectives.

Efficacy is the combination of efficiency and effectiveness.

(Moody, 2001)

The Method Evaluation ModelAn Overview

TaskInputs Outputs

Method

reduce inputs (increase efficiency)

improve outputs (increase effectiveness)


Validation of the application of OOmFPWebMethod Evaluation Model (MEM): a theoretical model for evaluating IS design methods, which incorporates both aspects of method “success”: efficacy and adoption in practice.

(Moody, 2001)


Theoretical Model for Evaluating FSM Methods

Model’s variables: Measurement Time: the time taken by

a subject to size a OOWS conceptual schema.

Reproducibility: the agreement between the measurement results of different subjects using OOmFPWeb.

Perceived Ease of Use: the degree to which a subject believes that using OOmFPWeb would be free of effort.

Perceived Usefulness: the degree to which a subject believes that OOmFPWeb will be effective in achieving its intended objectives.

Intention to Use: the degree to which an individual intends to use OOmFPWeb as a result of her perception of the method’s efficacy.


Laboratory experiment to test the Theoretical Model

Analyze functional size measurements

For the purpose of evaluating OOmFPWeb

With respect to its efficacy and likely adoption in practice

From the point of view of the researcher.

The context of the experiment was an OOWS conceptual schema of

a Web application that is measured by PhD students in the

Department of Computer Science at the Valencia University of

Technology.

Validation of the application of OOmFPWeb


Research Questions

The research questions addressed were:

RQ1: Is OOmFPWeb efficacious?

RQ2: Is OOmFPWeb likely to be adopted in

practice?

RQ3: Is Method Adoption Model a valid

theoretical model for evaluating OOmFPWeb?


Hypothesis Formulation

We therefore test the following hypotheses:

Hypothesis 1: OOmFPWeb is efficient when compared

to current industry practices.

Hypothesis 2: OOmFPWeb is effective when compared

to similar studies reported in the literature.

Hypothesis 3: OOmFPWeb is perceived as easy to use.

Hypothesis 4: OOmFPWeb is perceived as useful.

Hypothesis 5: There is an intention to use OOmFPWeb.


H1:Efficiency: Productivity of subjects X Reported industry averages

The mean measurement productivity that was observed (108.79 FP/hour) is about three times the size of the industry benchmark.

H2:Effectiveness: Reproducibility X Previous studies

Compared to the results reported by Kemerer the consistency of measurements was high (mean REPi was 6%). The variation around the mean subject assessment (i.e. range of values divided by mean value) was 24.5%, comparing well with Rudolph’s study.

Results Obtained

sAssessmentOtherAverage

AssessmentSubjectsassessmentOtherAverageREP i

i

i

ii TimeSubject

AssessmentSubjectPROD


H3, H4 and H5: PEOU, PU and ITU: survey instrument included 14 closed questions

Example of a question to measure PEOU:

These hypotheses was tested by verifying whether the scores that

students assign to the constructs of the MAM are significantly better than the middle score (i.e. the score 3).

All hypothesis were confirmed

Results Obtained


Analyzing the Validity and Reliability of the MAM Constructs

Validity Analysis: inter-item correlation analysis was carried out.

Reliability Analysis: inter-rater reliabilities .7 considered to be

acceptable.

CONSTRUCT CRONBACH’S α

Perceived Ease of Use 0,70

Perceived Usefulness 0,75

Intention to Use 0,80


Regression Models


Research Question Result

RQ1: Efficiency

1.1 OOmFPWeb is efficient when compared to current industry practices

Yes

RQ1: Effectiveness

1.2 OOmFP is effective when compared to similar studies reported in the literature.

Yes

RQ2: Perception of Efficacy and Intention to Use

2.1 OOmFPWeb is perceived as easy to use Yes

2.2 OOmFPWeb is perceived as useful Yes

2.3 There is an intention to use OOmFPWeb Yes

RQ3: Validation of the Method Adoption Model

3.1 Perceived ease of use is determined by productivity No

3.2 Perceived usefulness is determined by reproducibility Si

3.3 Perceived usefulness is determined by perceived ease of use

No

3.4 Intention to use is determined by perceived ease of use and perceived usefulness

Partially.

Summary of Findings


Work done so far..

Developing and validating measurement instruments for perception-based variables

Three replications of the study presented here:

1. Valencia University of Technology, Spain 46 students in the last year of Computer Science

degree

2. Valencia University of Technology, Spain 18 PhD students

3. University of Klagenfurt, Austria. 28 Master’s students


Next Steps

Analyzing the collected data to verify whether the proposed theoretical model is a valid model and measurement instrument for evaluating FSM methods.

A new experiment using practitioners of OOWS in the context of the Spanish Association of Software Metrics.

Validation of the Method Adoption Model for Functional Size Measurement of Web Applications

Documents

Transcript of Validation of the Method Adoption Model for Functional Size Measurement of Web Applications