To Evaluate Benefits Realization to SPI Using CBD Architecture

23年 4月 19日 Chung-Ang Univ. Software Engineering Lab.Chung-Ang Univ. Software Engineering Lab.

To Evaluate Benefits Realization to SPI Using CBD

Architecture

Kyung whan Lee, professorComputer Science and Engineering Division,

Chung-Ang University([email protected])

Nov. 5. 2003


Table of Contents

1. Introduction

2. Performance Measures and GQM Approach

3. Data Collection and calibration for the process performance

4. GQM-based Questionnaire for the process capability

5. Process Capability Model to the Earned Value

6. Benefits Realization Model and Conclusions

7. References


1. Introduction (1)

Lynch and Cross(1) suggested that customer satisfaction, flexibility, and productivity are the driving process upon which company objectives are based. Business goals and strategies, together with factual data about attributes of product quality and process performance, are the key drivers that lead to actions that improve a software process. We must also look at the benefit that justify the costs associated with changing the process.


1. Introduction (2)

MeasureGoals and Strategies

Vision

External Effectiveness Internal Efficiency

Market Financial

Customer

satisfactionFlexibility Productivity

Quality DeliveryCycle time Waste

Bussiness Unit

Bussiness Operating Systems

Departments and Work Centers

Figure1. Performance Pyramid


2. Performance Measures and GQM Approach (1)

The new measurement is formed under a simple assumption: fast cycle time and elimination of waste improve productivity and enable lower prices and increased sales, which in turn improve financial performance.

On the figure 1, Performance Pyramid, we can make the relationships between the local operating measures and the financial results. For example, companies that achieve fast cycle times are reporting triple the revenue growth and double the profits over industry-average competitors. Fast cycle times and the elimination of waste will equal better financial performance.

The GQM approach (3) is a systematic way to tailor and integrate an organization’s objectives into measurement goals and refine them into measurable values. The approach assumes that software development is still an immature engineering discipline and thus much is unknown about its characteristics and performance.



The GQM-based measurement approach has been implemented as part of an organization-wide process improvement program. As a direct result of the measurement program, interest in software reuse has increased and several other projects have renewed their efforts to develop reusable models. The reuse package described earlier is still frequently used within the organization to show the benefits of reuse.



The new measurement and the bottom line

Increased salesMore customers

Increased revenuesImproved benefit realizationIncreased earned valueImproved return on investment (ROI)

Improved

Quality

On-timedelivery

Fastercycle time

Less waste(e.g. carrying reuse)

Reduced price

Lower General and Administrative expenseLower operating expense


3. Data Collection and calibration for the process

performance (1)

The operational activities of measurement begin collecting data. The procedures that we define for collecting and refining data need to be integrated into our software processes and made operation.

Collecting data is more than just making the measurements. It consists of the implementing our plans, ensuring the work, and sustaining for the measurement activities of the results. The documentations of our procedure include follows:

• Identifying the responsible persons and organizations• Specifying where, when, and how measurements will be done• Defining the procedure to be used for recording and reporting

the results



performance (2)

We collected the data by performing the assessments with SPICE and marked GQM-based process maturity questionnaire. The requirements help to ensure that the assessment output is self-consistent and provide the evidence for the substantiation of markings. The assessment has to be conducted in accordant to the documented process that in capable of meeting the assessment process. The assessment process shall contain minimally the following activities such as planning, data collection, data validation, process rating, and reporting. The qualified assessors are educated and certified with ISO Guide62 and TR15504, and I identify the responsible persons to be rating on the maturity questionnaire.

For the calibration of the isolate data, need to verify the calibration

model. These problems are caused by two sources. First, the capability of respondent to the questionnaire is low. Second, it is the case of confusion for the respondent by the item of questionnaire itself.



performance (3) Now, we make a decision to exclude the first source because the

reviewer is the certified assessor. If, however, there is a wild deviation between assessors and is too large, there will be some problem in the product of work on the process. And there are some ambiguities in the process definition and its questionnaire. Therefore, we want to verify the confidence of the calibrated model with the hypothesis and assumptions as shown in <Table 1>



performance (4)<Table 1> Assumption and hypothesis to verify the calibrated model

Assumption 1 : Current maturing questionnaire is not trusted

Assumption 2 : If population follows the normal distribution (Z) by the science of statistics theory, it will be in the administration section (believability) within more than 98%.

Assumption 3 : The same (n) of population, which follows the normal distribution by the science of statistics theory, forms T-distribution.

Assumption 4 : In case of the distribution sample group (N) comes to population and is to follow the normal distribution (Z), if the sample is followed to the T-distribution by the science of statistics theory.

Null hypothesis (H0) : if the sample group (N) is the significant leve (P) = 0.2, that is in reliable if it showed within the confidence interval 99.8%.

Alternative hypothesis (H1) : Cannot be trusted sample group (N) is not existed within the confidence interval 99.8%.



performance (5)Let us <Table 1> put into a diagram as follows;

Population Data(Inference)

Sample space=>(N)

n1 n2

n3 n4

Defect Defect DataData

<Figure 3> Process data verification



performance (7)

Therefore, the calibration model established as presented in <Table 2>

Criteria

DeletionDelete T-item that is out of the range of critical region of distribution in the sample group (N) commonly.

CalibrationCorrect T-item, which is out of the range of critical region of distribution, within the confidence interval 99.8% in each sample (n).

<Table 2> calibration model of process data


4. GQM-based Questionnaire for the process capability (1)

Project practical experiences1) Size of work product2) Defect opportunity3) Capability of defect operation4) Defect introduction by Software quality assurance activity5) Development maturity capability using CBD Architecture

Capability metric subscript with the GQM-based approach

I can get the meta data using by the Goal-Question-Metrics approach as follows:1) Quality:

– Defect density– Impact of requirement change rate– Effectiveness using CBD architecture


4. GQM-based Questionnaire for the process capability (2)

2) Delivery:– Delivery time

– Reduced time-to-market using CBD architecture

3) Cycle time:– Classification of process, activity, and task time

– Man-Months used(effort distribution)

– Size estimation

– Efficiency using CBD architecture

4) Waste:– Rework

– Productivity

– Reuse

– Efficiency using CBD architecture


5. Process Capability Model to the Earned Value (1)

Earned value is measure of budgetary performance that relates actual expenditure to technical achievements as determined by milestone completions. Trend analysis use in conjunction with earned value. In business case, trends are heavily used in forecasting. For examples, managers can consider future raises when estimating labor costs by inflating salaries based on trends in economic index data. They can plot buying trends as a function of their sales forecast.

In software organizations during testing, software engineers can examine defect find-and-fix rates to determine whether they are removing defects more quickly than they are finding them. Software managers can plot their rate of progress in achieving milestones as a means to assess their programmatic performance to supplement concept like earned value.



6 Sigma typically monitors the process using control charts, which compare control limits with specification limits to determine process capability. We measure process capability in terms of capability indices which are simplified measures that quickly describe the relationship between process variability and the specification limits’ spread.

Process capability is the number of standard deviations that fit between the mean and specification limit; this unit corresponds to defect probability. The essential value of process capability analysis is its ability to predict the percentage of process characteristics or products that the process will produce within specification limits.

When analyzing process performance data, we must be constantly concerned that we have identified all sources of variation in the process. When data are aggregated, our process will be particularly susceptible to overlooked or hidden sources of variation.



The example that follows shows some of the advantages to calculate the process capability.

1) On the quality of performance pyramid;

- work product defect density

- technical documentation defect density

- code defect density

- defect management density

- requirement change rates

- typical costs : appraisal costs



2) On the delivery of performance pyramid;– differences between the budget and the actual progress

– differences between the plan and the actual progress of planning and analysis activity

– differences between the plan and the actual progress of design activity

– differences between the plan and the actual progress of construction activity

– differences between the plan and the actual progress of testing activity

– typical costs : document, reports, records, memos, indoctrination costs of custom process



3) On the cycle time of performance pyramid;– resources allocation for the planning and analysis activity

– resources allocation for the design activity

– resources allocation for the construction activity

– resources allocation for the testing activity

– differences between the budget and the actual human resources

– typical costs : policies, procedures, processes, activity

4) On the waste of performance pyramid;– productivity

– documentation size to SLOC

– work product size to a man-month

– reuse rates of code

– accuracy rates of size estimation

– typical costs : training, labor hours, travel costs, project costs, administration



Now, we can design the earned value system monitors benefis realized;

- financial benefits realized vs. cost (ROI)

- benefits realized vs. schedule

- actual cost vs. schedule

and propose classical earned value process as follows (4);

BCWS : Budgeted Cost of Work Scheduled,

BCWP : Budgeted Cost of Work Performed, Cost : Actual Cost of Work Performed

yes

Develop/update

Plans, BCWS

Performance

to plans

BCWP >=

BCWS?

BCWP >=

Cost?

yes

Determine Corrective Action

no no

yes



Benefit Realization Results Chain to Calculate Earned Value

INITIATE

Implement

SPI system

OUTCOME

• productivity

• increased customer

satisfaction

• flexibility

INTER MEDIATE OUTCOME

• reduce defect • time to market • cycle efficiency • reduce rework

Assumption

•

•

∂F

∂Rq * ∂Rd= 0

∂F

∂Rc * ∂Rw= 0

contribution contribution

: reduce time to process

: reduce time to deliver product-delivery effectiveness

: ① elaboration of results chain assumption

- base of estimation

② value management

-assumption monitoring

- progress/risk monitoring

: earned value extension

- change management


6. Benefits Realization Model and Conclusions (1)

On GQM-Based questionnaire we defined quality and delivery for the external effectiveness, and cycle time and waste for the internal efficiency to performance pyramid as follows;

quality ::= Rq (defect management, process capability, SPI cost,

component asset),

delivery ::= Rd (schedule, progress, marketing, time-to-market)

and

cycle ::= Rc (process capability, process improvement,

resource management)

waste ::= Rw (productivity, reduce rework, reuse capability,

expenditure rate)



Now we propose the benefit maximize function as follows; F (Rq, Rd, Rc, Rw) = 0.

To maximize benefit function for the external effectiveness,

∂F = 0, ∂Rq* ∂Rd

and to maximize benefit function for the internal efficiency,

∂F = 0, ∂Rc* ∂Rw



These condition links to assumptions, which condition the realization of outcomes on the benefit-realization approach chain.

For the assumptions we can propose the procedures for benefit realization as follows;



I. To evaluate process capability and estimate project size,

II. To calculate investment cost

A0 : the existence project cost

A1 : cost for SPI using CBD Architecture

A1j : j = 1, n (SPI items), aij : i, j = 1, n (weight of each item)

And then

A1 = Σ

III. To make cause-effect graph for the weighted cost

IV. To make cause-effect graph for the weighted external effectiveness and internal efficiency

V. Now calculate the following benefits;

i,j =1

naijA1j



benefit realization = (%)

Benefit / cost ratio =

ROI = * 100(%)

GQM-Based questionnaire gives us potential indicators of customer satisfaction, process flexibility, and productivity. Using them we can evaluate the benefit realization to software process improvement with CBD architecture which identify process measure to the business indicators and relate the costs and benefits of SPI also to the business indicators.

External Effectiveness + Internal efficiency

A0+A1

project benefits

project costs

project cost

project benefits – project cost


요약문

GQM 기반 설문서는 고객에 대한 만족도 , 프로세스의 유연성 , 및 제품의 생산성을 측정하고 이를 위해서 투자된 비용의 효율을 평가 할 수 있는 데이터를 수집해 준다 .

CBD 아키텍쳐에 의해서 GQM 기반 설문서에 의해서 수집할 데이터를 조정하여 신뢰성 구간을 설정하고 SPI 에 대한 현실적인 이익을 계산하는 것은 CBD 아키텍쳐를 사용한 소프트웨어 시스템의 효율을 평가하는 기초가 된다

비즈니스 버전에 맞추어서 CBD 아키텍쳐를 사용하여 개발한 소프트웨어 시스템의 Benefit Realization, Benefit/Cost Ratio, 및 ROI 를 계산할 수 있는 모델은 CBD 아키텍쳐의 재사용을 권장할 수 있는 기준과 지침이 될수 있다 .


References

1. William A. Florac, Anita D. Carleton, Measuring the software process, 1999. SEI Series, Addison Wesley.

2. Richard L. Lynch, Kelvin F. Cross, Measure up!, 1995, Blackwell.

3. Frank Van Latum, Rini Van Soligen, Adopting GQM-Based Measurement in an Industrial Environment, 1998, IEEE software.

4. Value-Based software Engineering : A Case Study, Barry Boehm, Computer, March 2003, IEEE

To Evaluate Benefits Realization to SPI Using CBD Architecture

Documents

Transcript of To Evaluate Benefits Realization to SPI Using CBD Architecture