Process Improvement and Risk Management in Off-The · PDF fileProcess Improvement and Risk...

Process Improvement and Risk Management in Off-The-Shelf

Component-Based Development

Jingyue Li

Doctoral Thesis

Submitted for the Partial Fulfilment of the Requirements for the Degree of

Philosophiae Doctor

Department of Computer and Information Science Faculty of Information Technology, Mathematics and Electrical Engineering Norwegian University of Science and Technology

June 2006

TO MY PARENTS ZHONGMIN LI AND YUSHU SHEN, MY WIFE FUFEN JIN AND MY SON LELE

i

Abstract Reusing software components from third-party vendors is one key technology to gain shorter time-to-market and better quality of the software system. These components, also known as OTS (Off-the-Shelf) components, come in two types: COTS (Commercial-Off-The-Shelf) and OSS (Open–Source-Software) components. To use OTS components successfully, it is necessary to know how the development processes and methods have to be adapted. Most current studies are either theoretical proposals without empirical assessment or case studies in similar project contexts. It is therefore necessary to conduct more empirical studies on how process improvement and risk management were performed and what were the results in various project contexts.

The overall research design of the work was a sequential, mixed method design that consists of several empirical studies. The thesis contains five novel main contributions (C1 to C5) and one minor contribution (C6):

C1. Better understanding on reusing in-house built components. A quantitative preliminary study shows that reusing in-house built components has the same challenges as OTS components related to requirements (re)negotiation, component documentation, and the specification of quality attributes. The informal communication between developers supplements component documentation. The component repository is not a key factor to successful component reuse.

C2: Summarizing the state-of-the-practice of development and selection processes in OTS component-based projects. A qualitative pre-study, a quantitative main study, and a follow-up study with industrial seminar show that OTS component-based development processes are typically variations of well-known process models, mixed with OTS specific activities. Concerning the OTS component selection, we find that mainly familiarity-based and Internet searches with hands-on-trials processes are applied. We summarize the state-of-the-practice of OTS processes into seven scenarios and propose how to customize the processes in OTS component-based projects.

C3: Empirical verification of the risk management proposals in OTS component-based development. The results of the main study reveal that allocating more effort into learning OTS components, performing the integration testing early, evaluating the quality of OTS components thoroughly, and following the update of OTS components have helped to mitigate corresponding risks. However, some problems, such as wrong estimation of the integration efforts and inefficient debugging, still happen frequently in practice and need further investigations.

C4: Better understanding of decision-making in OTS component-based development. By comparing projects using COTS components with those using OSS components, we reveal who, why, and the results of using different OTS components.

C5: Empirical validation of six newly published theses from the literature, which are related to process and risk management in OTS component-based development. The results of the main study support four theses and contradict the two others.

C6. Improved understanding about performing an international survey in the ICT industry (minor contribution). The main study is probably the first software engineering survey using a representative subset (sample) of industrial companies. Lessons learned from performing such as study reveal that, at best, we can achieve a stratified-random sample of ICT companies, followed by a convenient sample of relevant projects.

iii

Preface This thesis is submitted to the Norwegian University of Science and Technology (NTNU) for partial fulfilment of the requirements for the degree of Philosophiae Doctor.

The work referred to has been performed at the Department of Computer and Information Science, NTNU, Trondheim, under the supervision of Professor Reidar Conradi.

The doctoral work was financed by NTNU for four years. This thesis has also been affiliated with the INCO (INcremental and COmponent-based development) done jointly by the University of Oslo and NTNU.

v

Acknowledgements During the work on this thesis, I have benefited from a lot of communication with many people, who have provided valuable input to my studies. First of all, I would like to thank my supervisor Prof. Reidar Conradi for giving in valuable feedback and advice during my work. His engagement and knowledge have inspired me a lot. I would also thank Prof. Maria Letizia Jeccheri, Associate Prof. Dag Svanæs, Prof. Tor Stålhane, Dr. Parastoo Mohagheghi, Dr. Carl Fredrik Sørensen, Odd Petter N. Slyngstad, Prof. Kristen Ringdal, and Prof. Ola Listhaug at NTNU. They have all helped me with their insights and advice. Furthermore, I acknowledge Assistant Prof. Marco Torchiano and Associate Prof. Maurizio Morisio at Politecnico di Torino and Dr. Christian Bunse at Fraunhofer IESE for their support and cooperation during the international survey. Stewart Clark has been especially helpful by proof-reading this thesis. I also acknowledge all the industrial partners that participated in this work.

Several master’s students at NTNU have contributed to studies in this thesis. I would like to thank Axel Anders Kvale, Odd Are Sæhle, and Øivind Wang for their contributions. Also, thanks to present and former colleagues at the software engineering group at the Department of Computer and Information Science, NTNU, for providing a good work environment.

I would like to thank Prof. Anthony Finkelstein at University College London, for his kind hospitality and cooperation during my stay there in spring 2005.

I would also like to thank my family. A specific appreciation goes to my parents for supporting and believing in me during all these years. Their effort in taking care of my child provided me enough time to finish this work. I would especially express my thanks to my wife, Fufen Jin and my son Lele for their love and support.

vii

Contents

Abstract........................................................................................................................ i

Preface........................................................................................................................ iii

Acknowledgements .................................................................................................... v

Contents .................................................................................................................... vii

List of Figures............................................................................................................. x

List of Tables .............................................................................................................. x

Abbreviations ............................................................................................................ xi

1 Introduction ........................................................................................................ 1

1.1 Problem Outline 1 1.2 Research Questions 2 1.3 Research Design 3 1.4 Contributions 4 1.5 Selected Papers 9 1.6 Thesis Structure 11

2 Software Reuse and Component-Based Development .................................. 13

2.1 Software Engineering Definitions and Challenges 13 2.2 Software Reuse 14 2.3 Component-Based Development 15 2.4 COTS-Based Development 18 2.5 OSS-Based Development 19 2.6 OTS Component-Based Development 21 2.7 Summary and Discussion 22

3 Process Improvement in OTS Component-Based Development.................. 25

3.1 Software Process Improvement (SPI) 25 3.2 SPI of OTS Component-Based Development 27

3.2.1 Improvement of the Whole Life Cycle............................................... 27 3.2.2 Improvement of the OTS Component Selection ................................ 28

3.3 Summary and Discussion 29

4 Risk Management in OTS Component-Based Development........................ 31

4.1 Risk Management in Software Development 31

viii

4.2 Risk Management in OTS Component-Based Development 33 4.3 Summary and Discussion 35

5 Research Methods............................................................................................. 37

5.1 Research Strategies in Empirical Software Engineering 37 5.2 Survey Methods 39 5.3 Research Design 40

5.3.1 Research Design of the Preliminary Study......................................... 40 5.3.2 Research Design of the Pre-study....................................................... 43 5.3.3 Research Design of the Main Study ................................................... 45 5.3.4 Research Design of the Follow-up Study ........................................... 52

5.4 Validity Issues 53 5.5 Summary and Discussion 54

6 Results................................................................................................................ 55

6.1 Collected Samples in Each Study 55 6.1.1 Collected Samples in the Preliminary Study ...................................... 55 6.1.2 Collected Samples in the Pre-study.................................................... 55 6.1.3 Collected Samples in the Main Study................................................. 56 6.1.4 Collected Samples in the Follow-up Study ........................................ 57

6.2 Answers to Research Questions 58 6.2.1 Reusing In-House Built Components – RQ1 ..................................... 58 6.2.2 Software Process Improvement – RQ2 .............................................. 59 6.2.3 Risk Management – RQ3 ................................................................... 65 6.2.4 Decision Making – RQ4..................................................................... 69

6.3 Our Lessons Learned on Performing the Studies 73 6.4 Summary and Discussion 74

7 Evaluation and Discussion ............................................................................... 75

7.1 RQ1: How to Improve the Reuse of In-house Built Components? 75 7.1.1 Component-Related Requirements (Re) negotiation.......................... 75 7.1.2 Component Repository ....................................................................... 76 7.1.3 Component Understanding ................................................................. 76 7.1.4 Quality Attributes of Components...................................................... 76

7.2 RQ2: How to Do SPI in OTS Component-Based Development? 76 7.2.1 How to Improve the Whole Life Cycle .............................................. 77 7.2.2 How to Select the OTS Components.................................................. 80

7.3 RQ3: How to Mitigate Risks in OTS Component-Based Development? 81 7.3.1 How to Mitigate the Cost Estimation Risks ....................................... 81 7.3.2 How to Mitigate the Quality Risks ..................................................... 82 7.3.3 How to Mitigate the Requirements Risks........................................... 83 7.3.4 How to Mitigate the Debugging and Deployment Risks.................... 83 7.3.5 How to Mitigate the Maintenance Risks ............................................ 84 7.3.6 How to Mitigate the Provider Support Risks...................................... 85

7.4 RQ4: How to Make the Decision between COTS or OSS Components? 85 7.4.1 Why Did Developers Decide to Use OTS Components..................... 86

ix

7.4.2 How Were the OTS Components Used in Practice? .......................... 86 7.4.3 COTS or OSS? Our Suggestions ........................................................ 87

7.5 Evaluation of Validity Threats 87 7.5.1 Possible Validity Threats in the Preliminary Study............................ 87 7.5.2 Possible Validity Threats in the Pre-study ......................................... 88 7.5.3 Possible Validity Threats in the Main Study ...................................... 89 7.5.4 Possible Validity Threats in the Follow-up Study.............................. 90

7.6 Summary 90

8 Conclusions and Directions for Future Work................................................ 91

8.1 Conclusions 91 8.1.1 Describing Different Aspects of Reusing Component ....................... 91 8.1.2 Verifying Existing Theories and Assessing Existing Methods .......... 92 8.1.3 Generating New Theories from New Perspectives............................. 93

8.2 Future Work 94 8.2.1 Need to Study the Cause-Effects of the SPI Issues ............................ 94 8.2.2 Risk Management Issues Need a Deeper Investigation...................... 94 8.2.3 Need to Better Understand OSS Projects and Products ..................... 94

References................................................................................................................. 95

Appendix A: Selected papers ................................................................................ 107

P1. Developer Attitude to Component Reuse 107 P2. Variations in COTS-Based Development Process 123 P3. Validation of Theses on OTS-Based Development 153 P4. SPI on OTS-Based Development Process 171 P5. Risk Management in OTS-Based Development 185 P6. Decision Making in OTS Component-Based Development 225 P7. Barriers to Disseminating Theory 233 P8. Reflections on Conducting a Survey 241

Appendix B: Other Papers.................................................................................... 259

Appendix C: Interview Guide for the Pre-study................................................. 265

Appendix D: Questionnaire in the Main Study................................................... 279

x

List of Figures Figure 1-1 Research questions and corresponding studies............................................... 3 Figure 1-2 Contributions C1 to C5 ................................................................................... 5 Figure 4-1 Risk management framework ....................................................................... 31 Figure 4-2 Risk management in OTS component-based development .......................... 35 Figure 5-1 Risk management before a project starts ...................................................... 46 Figure 5-2 Research framework for the risk management in the main study................. 46 Figure 6-1 Distribution of company size........................................................................ 56 Figure 6-2 Distribution of companies main business areas............................................ 57 Figure 6-3 Distribution of application domain of the systems ....................................... 57 Figure 6-4 The actual development process in the OTS-based project .......................... 60 Figure 6-5 What selection and evaluation actions were performed?.............................. 62 Figure 6-6 When was the OTS component selected?..................................................... 62 Figure 6-7 Result of thesis T5 ........................................................................................ 64 Figure 6-8 Occurrences of typical risks.......................................................................... 66 Figure 6-9 Risk management activities performed......................................................... 67 Figure 6-10 Project profiles of the COTS component-based system ............................. 69 Figure 6-11 Project profiles of the OSS component-based system................................ 70 Figure 6-12 General motivations of using COTS components ...................................... 70 Figure 6-13 General motivations of using OSS components ......................................... 71 Figure 6-14 Specific motivations of using COTS components...................................... 71 Figure 6-15 Specific motivations of using OSS components......................................... 72 Figure 6-16 Occurrences of risks in COTS component-based projects ......................... 72 Figure 6-17 Occurrences of risks in OSS component-based projects ............................ 73 Figure 7-1 Scenarios 1 to 3............................................................................................. 78 Figure 7-2 Scenarios 4 to 6............................................................................................. 79

List of Tables Table 1-1 Relations between research questions, contributions, and papers.................... 9 Table 2-1 Advantages and disadvantages of using COTS software............................... 19 Table 5-1 Qualitative vs. quantitative in empirical strategies ........................................ 38 Table 5-2 Questionnaire used in the preliminary study.................................................. 42 Table 5-3 Correspondence between questions and RQs in the preliminary study ......... 43 Table 5-4 Typical risks in OTS component-based projects ........................................... 48 Table 5-5 Typical risks management activities in OTS-based projects ......................... 49 Table 5-6 Revised research questions to investigate the newly published theses .......... 50 Table 6-1 Barriers to disseminating SPI theories to the IT industry .............................. 64 Table 6-2 Effective risk management activities ............................................................. 68 Table 6-3 Barriers to disseminating RM theories to the IT industry.............................. 68

xi

Abbreviations CBD Component-Based Development CBSE Component-Based Software Engineering CMM Capability Maturity Model CRM Customer Relationship Management COCOTS COnstructiveCOTS CORBA Common Object Request Broker Architecture COM Component Object Model (Microsoft) COTS Commercial-Off-The-Shelf DCOM Distributed Component Object Model (Microsoft) EJB Enterprise Java Beans (Sun) ERP Enterprise Resource Planning GUI Graphical User Interface HHM Hierarchical Holographic Modelling ICT Information and Communication Technology ISO International Standards Organization IEC International Electrotechnical Commission LOC Lines of Code NTNU Norwegian University of Science and Technology OO Object-Oriented OTS Off-The-Shelf OSS Open Source Software ROI Return On Investment RUP Rational Unified Process SEI the Software Engineering Institute (SEI) at Carnegie Mellon University SPI Software Process Improvement UP Unified Process XP eXtreme Programming

1

1 Introduction

In this chapter, the background to the research and the research context are briefly presented. The chapter also describes research questions, research design, and the claimed contributions. The list of papers, my specific contributions to each study, and the thesis outline are also presented.

1.1 Problem Outline

To gain competitive advantages, software organizations are forced to develop systems quickly and cost-efficiently. Reuse is therefore one key technology to reach their goals. Reuse is known since the beginning of computer science (e.g., reuse of command sequences) and has now be raised to the level of components (i.e., a reusable piece of software that can be easily integrated with other components with relatively little effort). The components can be built in-house, be acquired from the third-party providers, or be ordered from the sub-contractors. The components acquired from third-party providers, also known as OTS (Off-the-Shelf) components, come in two different types: COTS (Commercial-Off-The-Shelf) and OSS (Open–Source-Software) components. To integrate the OTS components successfully, one major research question is: how development processes or methods have to be adapted concerning the development of systems with such components?

Although researchers and practitioners have been dealing with OTS component-based development processes for quite a time, most studies are based on military or aerospace projects, or similar large projects. To propose and design cost-effective OTS-based development process, it is necessary to empirically investigate how OTS-based projects are performed in different application domains, especially in small or medium-sized projects and companies.

Although using OTS component may help to shorten time-to-market and save development effort, using such external components introduces many risks. Before project managers decide to acquire an external component instead of building it in-house, they must evaluate and compare possible risks and benefits. Although several risks and risk management activities in OTS component-based development have been identified from case studies, few empirical studies have subsequently verified their

Introduction

2

conclusions. As a result, software project managers have few effective and well-proven guidelines on using specific risk management activities to mitigate corresponding risks.

Although both COTS and OSS components can be acquired from third-party providers, they are different in their nature and appearance. COTS components are owned by commercial vendors that often provide specific support. Users of COTS components have limited access to the source code. On the other hand, OSS components are provided by open source communities with freely accessible source code, but with no promise about specific support. When planning a new software project, project decision-makers need to decide whether they should buy a COTS component or acquire an OSS one, if it was decided to use OTS components. To make such a decision, it is important to investigate previous projects using such components. Unfortunately, the relevant decision-making processes are often intuitive and only sparsely documented. Thus, further research is warranted to examine the current state-of-the-practice and to define guidelines for making such decisions.

1.2 Research Questions

The goal of this research is to investigate state-of-the-practice of COTS and OSS component-based development from the process and organizational viewpoints. The motivation is to give guidelines on how to select and integrate COTS or OSS components based on the experiences and lessons learned from finished projects. There are four main research questions in this study. The research questions are designed and refined step by step.

The process and risk management issues of component-based development were firstly investigated through a preliminary study on reusing in-house built components. This study examined research question RQ1:

RQ1: What is the status of component-based development based on components built in-house?

Results of the preliminary study gave valuable insights on designing research questions for later studies. We are aware that there are many common issues between reusing in-house built components and using OTS components. Therefore, we decided to focus more on using OTS components, because it will cover broader issues and some of these issues are also relevant to in-house built components.

Based on results of the preliminary study, it was decided to investigate the software

process improvement (SPI) and risk management in COTS components-based development with a pre-study. This study investigated research questions RQ2 and RQ3:

RQ2: What are the development processes and OTS component-selection processes used in OTS component-based development projects? RQ3: Which risk management activities have been performed and what are the results of performing them?

Introduction

3

By talking with the respondents face-to-face in the pre-study, we got insights on SPI and risk management in OTS component-based development. We also found that OSS components should be involved in future studies, because OSS components represent an important type of OTS components. In addition to know their commonalities, it is interesting to know the differences in using these two different types of OTS components. Therefore, research question RQ4 is defined as:

RQ4: How do project managers make their acquire decisions in using COTS instead of OSS components, or vice versa?

1.3 Research Design

Figure 1-1 shows the studies performed, their date and sequence, type of studies, and their relations with the research questions.

Figure 1-1 Research questions and corresponding studies

Introduction

4

Empirical studies may be performed quantitatively, qualitatively or in combination. The studies reported in this thesis have been a combination of qualitative and quantitative studies. The choices of approach are decided by the research questions.

The research question RQ1 was studied by an explorative quantitative preliminary study. Data from 26 people in three Norwegian IT companies were collected by interviews using a structured questionnaire.

Research questions RQ2 and RQ3 were first investigated by an explorative qualitative pre-study. The pre-study was performed through semi-structured interviews on COTS component-based development. In this pre-study, we interviewed 16 COTS component-based development projects in 13 Norwegian IT companies. In addition, RQ4 was initiated based on the results of the pre-study.

To validate the conclusions of the pre-study, a descriptive quantitative main study was performed. Data was collected with a structured questionnaire. Results from 133 OTS component-based projects in Norway, Italy, and Germany have been collected. Results of the main study verified the conclusions of the pre-study.

In the late phase of the main study, we performed an explorative qualitative follow-up study with an industrial seminar to present our conclusions, to discuss with industrial partners, and to get feedback from them. The intention was to briefly understand some cause-effects of the phenomena we discovered from the previous studies.

1.4 Contributions

To manage the OTS component-based project successfully, project members need to make several decisions. These decisions may not be sequential and may need several iterations. We mainly contributed on summarizing the state-of-the-practice and giving guidelines on making several key decisions, as shown in Figure 1-2.

Introduction

5

Figure 1-2 Contributions C1 to C5 Contribution C6 is a secondary contribution, which comes from the lessons learned

in performing the survey. It gives guidelines on how to perform an international survey with stratified-random sample selection strategy.

Detailed descriptions of each contribution:

C1: Increased understanding on risks of reusing in-house built components In a preliminary quantitative study, we have investigated challenges related to four key factors for development based on in-house built components, especially in development-with-reuse. These factors are component-related requirements (re)negotiation, component repository, component understanding, and components’ quality attribute specification. Another contribution is that we have compared three IT companies with different reuse levels to study the possible trend and challenges in these factors when more and more code is encapsulated as reusable components.

For component-related requirements (re)negotiation, we conclude that requirements (re)negotiation for in-house built components is important but not efficient. The

Introduction

6

efficiency will probably not increase with a higher reuse level (i.e., more reusable components are available in the organization).

For a component repository, we confirm that this is not a key factor for successful reuse. Furthermore, the potential value of a component repository will probably not increase with higher reuse levels.

For component understanding, we reveal that most developers are not satisfied with component documentation, and that developers’ satisfaction with component documentation will decrease with higher reuse levels. The results also show that informal communication channels (i.e., tacit knowledge), through which developers may get necessary information about the components, should be given more attention.

For components’ quality attribute specification, we discover that developers still need to spend much effort on testing the components before they start to use them, as they cannot get relevant information from component specifications. C2: Summarizing the state-of-the-practice of development and selection processes in OTS component-based projects The contributions to the OTS component-based development process focus on two dimensions:

- The whole software development life cycle. - OTS component selection and evaluation.

A qualitative pre-study with semi-structured interviews was first performed to

examine the state-of-the-practice of the development processes in COTS component-based development. We found:

- The use of COTS components can be done as part of traditional development

processes (e.g., waterfall and evolutionary), i.e., there is no special “COTS-based development process”.

- Successful use of COTS components in such processes, however, requires that some new activities and roles are introduced. Typical new activities are the make vs. acquire decision, COTS component selection, and COTS component integration. A new role is that of a knowledge keeper.

- Two of the new activities, the make vs. acquire decision and the COTS component selection, can be placed in different development phases (requirements, design, or implementation). This depends on project context, especially on the familiarity with possible COTS components and the flexibility of requirements.

- A set of explanatory scenarios and guidelines have been synthesized to assist in the customization of the traditional development processes with the new activities and roles.

Based on the results of the pre-study, a main study with a formal questionnaire was

launched. This studied 133 projects from Norway, Italy, and Germany. It had a broader focus than the pre-study and also included projects using OSS components. The results confirm our findings in the pre-study. We conclude:

Introduction

7

- The actual OTS-based development process was the traditional process with OTS-specific activities. The development process was dominated by the company/department rules, instead of the decision of using OTS components.

- The actual OTS component selection can be done in different phases. The main evaluation processes are familiarity-based or hands-on-trial-based. The phase to select OTS component has relationship with the project members’ familiarity with possible component candidates.

- The proposed software development process and OTS component selection process need further investigation to be suited to different project contexts.

By summarizing the state-of-the-practice of OTS component-based development

processes into seven scenarios, we gave systematic proposals about how to adopt the OTS-based development and selection processes based on the project contexts.

C3: Empirical validation of risk management proposals in OTS component-based development By summarizing the good practices and lessons learned in the projects investigated in the pre-study, we got new insights on managing the possible risks in OTS component-based development. In the main study, we further examined our findings. Our contributions answered the following three sub-questions:

- What are the most frequent problems occurred in OTS component-based development? Two of the most frequent problems in OTS component-based development are the incorrect estimation of the integration effort and inefficient debugging. The least frequent problems are relevant to the negative effect of the OTS components on the quality of the whole system.

- What are the most frequent risk management activities performed in OTS component-based development? Developers have seriously evaluated the quality of the OTS components in the selection phase. They have considered the possible learning effort in the effort estimation. They have done integration testing as early as possible and usually have internal experts to follow the update of the OTS components. However, developers have rarely involved their customers in the make vs. acquire decision and OTS component selection.

- What are the results of the risk management activities on corresponding risks?

To increase the accuracy of effort estimation, it is important to allocate enough effort to learn and understand the OTS components. To avoid the possible quality problems and to increase the efficiency of locating the defects, it is important to do black-box testing as part of OTS component selection and to do integration testing as early as possible. To mitigate the problems due to customers’ requirements changes, it is important to learn the OTS components thoroughly in the selection phases and to integrate the unfamiliar OTS components first. It is also important to be able to negotiate changed requirements with the customers. To increase the efficiency of debugging and

Introduction

8

system deployment, it is necessary to understand the components well and to do integration testing early. It is critical to keep and share the knowledge about OTS components and the component providers inside the organization. It shows to help mitigate the possible risks in system maintenance and evolution.

C4: Increased understanding of the decision-making practice in OTS component-based development We compared who, why, and the results of using COTS components instead of OSS components, or vice versa. This contributed to answer the three questions:

- Who were using OTS components? Both COTS and OSS components were used in projects with different application domains and different non-functional requirements. There was no significant difference between the profiles of COTS-based and OSS-based projects.

- Why was it decided to use OTS components? The main expectations of using either COTS or OSS components are to obtain shorter time-to-market and less development effort. COTS users have higher expectation on COTS component quality and COTS vendor support. Possible no-cost source code is the key motivation of OSS users. OSS users prefer to have access to the source code, so that they can revise it when necessary.

- What are the possible problems of using each type of OTS components? It is more difficult for COTS component users to follow requirement changes than OSS users. It is also more difficult for COTS users to estimate the selection effort.

C5: Empirical validation of the six newly published theses, which are related to the process and risk management in OTS component-based development The pre-study and main study also investigated six newly published theses in COTS component-based development proposed in [Torchiano04]. Our results support four of the theses and contradict the two others. The supported theses are: OSS components were mainly used without modification in practice; custom code mainly provided additional functionality; formal COTS selection processes were seldom used; COTS component users managed to get the required changes from vendors. The unsupported theses are: standard mismatches were more frequent than architecture mismatches; COTS components were mainly selected based on architecture compliance instead of functional completeness. C6: Improved understanding on performing an international survey in the ICT industry In the main study, a stratified-random sample selection strategy was used to collect data in three European countries. The lessons learned from performing the main study contributed to give guidelines on performing similar empirical studies in the IT industry. We found that:

Introduction

9

- A pre-study often clarifies both research questions and concrete questions in a later questionnaire. For instance, we “discovered” several variants of OTS-based development processes in our pre-study.

- A sample of national Information and Communication Technology (ICT) companies assumes access to public data sources, like company lists from census registers, but even such data may contain many errors. Close contact with people in such organs are needed to obtain stratified samples and expedite delivery of data in electronic form.

- For a variety of reasons it seems impossible to perform the same random sampling and data collection in different countries. The national differences may give an unknown mode effect in result interpretation between the countries.

- Random sampling and the ensuing contact process appear to be much more expensive (5 times?) than convenient sampling. In our case, this is because we got stuck with no previous contact person, and scant pre-knowledge and “goodwill” from such companies, especially the large ones.

- If the final sample size is under 100 respondents, use phone interviews. This applies particularly when phone contact is anyhow needed, e.g. due to a non-convenient sample. A written procedure for the contact process is anyhow needed.

- A project baseline can be useful for comparison purposes. In our case, we did not ask the companies about non-OTS projects, so we missed this.

- Interviewing on this level requires a sound background knowledge of the material; Company phone contact assumes at least PhD student qualifications.

1.5 Selected Papers

This thesis includes eight papers from P1 to P8 as follows. These papers show our main contributions to the research questions and are attached in Appendix A. Abstracts of other papers (from AP1 to AP7) related to work in this thesis are attached in Appendix B. All these papers can be downloaded from http://www.idi.ntnu.no/grupper/su/. The relations between the research questions, contribution, and the papers are shown in Table 1-1.

Table 1-1 Relations between research questions, contributions, and papers

Research questions Contributions Papers RQ1 C1 P1*, RQ2 C2 P2*, P4*, P7*, AP6, AP7 RQ3 C3 P2*, P5*, P7*, AP1, AP2, AP3, AP4 RQ4 C4 P6*, AP5 RQ2&RQ3 C5 P3* Method issues C6 P8*

* Papers attached in Appendix A [P1] [Li04a] Jingyue Li, Reidar Conradi, Parastoo Mohagheghi, Odd Are Sæhle, Øivind Wang, Erlend Naalsund, and Ole Anders Walseth: A Study of Developer Attitude to Component Reuse in Three IT Companies. Proceedings of 5th International Conf. on

Introduction

10

Product Focused Software Process Improvement (PROFES'2004), Apr. 2004, Nara area, Japan, Springer LNCS Vol. 3009, pp. 538-552.

Relevance to this thesis: This paper reports the result of the preliminary study, which covers RQ1 and shows our main contribution to C1.

My contribution: I was the leading author and contributed about 70% of the total work, including research design, questionnaire design, data analysis, and paper writing.

[P2] [Li06a] Jingyue Li, Finn Olav Bjørnson, Reidar Conradi, and Vigdis By Kampenes: An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT Industry. To appear in the Journal of Empirical Software Engineering (Special issue from Metrics 2004).

Relevance to this thesis: This paper reports the result of the pre-study, which covers RQ2 and RQ3 and partly shows our contribution to C2 and C3.

My contribution: I was the leading author and contributed about 80% of the total work, including research design, interview guide design, performing most interviews, data analysis, and paper writing.

[P3] [Li05a] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse, Umair Khan, Marco Torchiano and Maurizio Morisio: Validation of New Theses on Off-The-Shelf Component Based Development. Proceedings of the 11th IEEE International Metrics Symposium (Metrics'05), Sep. 2005, Como, Italy, IEEE Press, pp.26 (abstract), 10p.

Relevance to this thesis: This paper reports the result of the main study, which covers RQ2 and RQ3 and shows our contribution to C5.

My contribution: I was the leading author and contributed 70% of the total work, including research design, questionnaire design, and data collection in Norway, data analysis, and paper writing.

[P4] [Li06b] Jingyue Li, Marco Torchiano, Reidar Conradi, Odd Petter N. Slyngstad, and Christian Bunse: A State-of-the-Practice Survey of the Off-the-Shelf Component-Based Development Processes, Proceedings of the 9th International Conference on Software Reuse, Torino, Italy, June, 2006, Springer LNCS Vol. 4039, pp. 16-28.

Relevance to this thesis: This paper reports the result of the main study, which covers RQ2 and shows our main contribution to C2.


[P5] [Li06c] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse, Marco Torchiano, and Maurizio Morisio: A State-of-the-practice Study on Risk Management in OTS Component-Based Development. Submitted to IEEE Transactions on Software Engineering, June 2006.


Introduction

11

My contribution: I was the leading author and contributed 70% of the total work, including research design, questionnaire design, data collection in Norway, data analysis, and paper writing.

[P6] [Li06d] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse, Marco Torchiano, and Maurizio Morisio: An Empirical Study on the Decision Making Process in Off-The-Shelf Component Based Development. Proceedings of the 28th International Conference on Software Engineering (ICSE 2006), 20-28 May 2006, Shanghai, P.R. China, pp. 897-900, IEEE Press.



[P7] [Li05b] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse, Umair Khan, Marco Torchiano and Maurizio Morisio: Barriers to Disseminating Off-The-Shelf Based Development Theories to IT Industry. Proceedings of the International Workshop on Models and Processes for the Evaluation of COTS Components (MPEC'05 Arranged in co-location with ICSE'05), May, 2005, St. Louis, Missouri, USA, pp. 1-4. It is also published in the electronic version of ACM Software Engineering Notes, Vol. 30, No. 4, July 2005.

Relevance to this thesis: This paper reports the result of the follow-up study, which covers RQ2 and RQ3, and shows partly our contribution to C2 and C3.

My contribution: I was the leading author and contributed on 80% of the total work, including research design, seminar organization, data collection, data analysis and paper writing.

[P8] [Conradi05a] Reidar Conradi, Jingyue Li, Odd Petter N. Slyngstad, Christian Bunse, Marco Torchiano and Maurizio Morisio: Reflections on conducting an international CBSE survey in ICT industry. Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE 2005), Nov. 2005, Noosa Heads, Australia, pp. 214-223, IEEE Press.

Relevance to this thesis: This paper reports our lessons learned on the main study, and shows our main contribution to C6.

My contribution: I contributed on 40% of the total work, including sample selection design, performing the data collection, and major comments on the data analysis and draft paper.

1.6 Thesis Structure

Chapters 2 to 5 give the state-of-the-art of relevant fields of this thesis and introduce how the research questions are designed. Chapters 6 to 8 give the results and discuss our contributions to the research questions. Chapter 2: Software reuse and component-based development. This chapter introduces the state-of-the-art and challenges in software reuse and component-based

Introduction

12

development. First, we discuss the motivation of systematic reuse in software development and illustrate different kinds of reuse, such as reusing documentation, architecture, design, and test cases. Then, we introduce the intention and challenges of reusing software components. Finally, we focus on issues of reusing the third-party software components, such as COTS and OSS components. Chapter 3: Software process improvement in OTS component-based development. This chapter introduces the state-of-the-art of SPI in OTS component-based development. We describe not only proposed process models in COTS or OSS component-based development, but also OTS component selection processes. Chapter 4: Risk management in OTS component-based development. This chapter first introduces the theory of risk management. We then list the proposed specific risks and according risk management activities in OTS component-based development. Chapter 5: Research methods and metrics. In this chapter, the research goals and research questions are specified. We give our rationale for using different research methods to answer different research questions. Chapter 6: Empirical investigations and main results/findings. This chapter explains how we perform the preliminary study, the pre-study, the main study, and the follow-up study. We give detailed information on data collection methods, collected samples, and answers to research questions in each study. Chapter 7: Evaluation and discussion. The research questions are answered in this chapter. We summarize the results from the empirical investigation and compare them with the state-of-the-art. We also discuss possible threats to the validity of each study. Chapter 8: Conclusion and future work. This chapter sums up the main findings from the discussion ands outline possible future work. Appendix A contains 8 selected papers (P1 to P8) that provide detailed results and discussions of the individual studies. Appendix B contains the abstracts of other seven relevant papers (AP1 to AP7) which are not attached in this thesis. Appendix C includes the interview guide for the pre-study. Appendix D contains the questionnaire used in the main study. In both parts of this thesis, I have generally used the term “we” to present the work. The layout of papers in Appendix A has been changed to fit the format of the rest of the thesis.

Software reuse and component-based development

13

2 Software Reuse and Component-Based Development

This chapter describes the challenges in software engineering that are the motivations behind reuse. The state-of-the-art in Component-Based Development (CBD), COTS development, and OSS-based development are then described. Finally, the whole chapter is summarized and the research challenges related to this thesis are discussed.

2.1 Software Engineering Definitions and Challenges

Software engineering is the branch of system engineering concerned with the development of large and complex software intensive systems. It is concerned with the processes, methods, and tools for the development of software intensive systems in an economic and timely manner [Finkelstein00]. Software engineering activities or phases include managing, estimating, planning, modelling, analysing, specifying, designing, implementing, testing, and maintaining [Fenton97].

The term software crisis was first used in 1968 to describe the ever-increasing burden and frustration that software development and maintenance placed on otherwise happy and productive organizations [Griss93]. The much cited Standish report on software projects [Standish95] shows “a staggering 31.1% of projects will be cancelled before they ever get completed”. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg.

Philippe Kruchten [Kruchten01] discusses why software engineering differs from structural, mechanical, and electrical engineering due to the soft, but unkind nature of software. He suggests four key differentiating characteristics:

− Absence of fundamental theories or at least practically applicable theories makes is difficult to reason about software without building it.

− Ease of change encourages changes in software, but it is hard to predict the impact.

− Rapid evolution of technology does not allow proper assessment, and makes it difficult to maintain and evolve legacy systems.

− Very low manufacturing costs combined with ease of change have led the software industry into a fairly complex mess.


14

These characteristics have even become more extreme due to developments in Internet-speed. Internet-speed development involves rapid requirement changes and unpredictable product complexity [Baskerville03]. Software organizations have always been looking for effective strategies to develop software faster, cheaper, and better.

2.2 Software Reuse

The motivations for reuse are primarily economic: the potential of saving cost, time and effort of redundant work, increasing productivity, decreasing time to market and improving systems quality by reusing both artefacts, and the underlying engineering experience.

Software reuse means reusing the inputs, the processes, and the outputs of previous software development efforts. The idea of reuse was originally due to Doug McIlroy [McIlroy68] in his paper “Mass Produced Software Components”. Reuse is later named as a potential “silver bullet” [Brooks87]. Many software organizations around the world have reported successful reuse programs, such as at IBM, Hewlett-Packard, and Hitachi [Griss93]. Jones [Jones84] identified four types of reusable artifacts:

- Architecture reuse, which consists of standardizing a set of design and

programming conventions dealing with the logical organization of software. - (detailed) Design reuse: for some common business application. - Data reuse, involving a standardization of data formats. - Program reuse, which deals with reusing executable code.

The challenges facing reuse are structural, organizational, managerial, and technical

[Mili95]. Many organizations treat reuse as technology-acquisition problems instead of a technology-transition problem. However, just buying technology usually does not lead to extensive reuse. Morisio et al. [Morisio02b] found that:

− Top management commitment is the prerequisite for success. − Product family practice, common architecture, and domain engineering increase

reuse capability. − Size, development approach (object-oriented or not), rewards, repository, and

reuse measurement are not decisive factors, while training is. − The other three factors that are considered to be success factors are reuse process

introduced, non-reuse process modified, and human factors. − Successful cases tried to minimize change, to retain their existing development

approach, choosing reuse technology to fit that. Griss [Griss95] summarized the experience from industrial reuse and pointed out

that: − Reuse needs management support, since reuse involves more than one project. − Object technologies or libraries give no improvement in reuse. − Domain stability and experience are often more important for successful reuse

than general process maturity. − One way to adopt reuse incrementally is to increase your investment and

experience.


15

Frakes et al. [Frakes95] report that most software engineers prefer to reuse rather than to build from scratch. In addition, they did not find any evidence that use of certain programming languages, CASE tools, or software repositories promote reuse. On the other hand, reuse education and a software process that promotes reuse have a positive impact on reuse. They also found that the telecom industry has higher levels of reuse than some other fields.

Software reuse includes two interrelated parts: reuse as and reuse with. Reuse as means creating the asset for future reuse. Reuse with indicates using the reusable artefacts in the new system.

2.3 Component-Based Development

Component-based development (CBD) and Component-based Software Engineering (CBSE) facilitate reuse by providing logical units for assembly and make systematic reuse possible by demanding that components should adhere to a component model. CBD focuses on software architecture as a guideline to put pieces together, viewing components as independent units of development and deployment, and on component models. Developing components is one kind of developing for reuse, while developing systems of reusable components is developing with reuse [Karlsson95]. Many CBD concepts originate from reusability and Object-Oriented (OO) approaches.

Components, the heart of CBSE, are defined in various ways depending on the viewpoint. We introduce several theoretical definitions in this section. The component definition used in this thesis is shown in Section 2.6.

In the SEI’s report [Bachmann00] on technical aspects of CBD, a component is defined as:

− An opaque implementation of functionality. − Subject to third-party composition. − Conformant to component model.

Heineman and Council [Heineman01] define a software component as: A software element that conforms to a component model, which can be independently deployed and can be composed without modification according to a composition standard.

Szyperski’s [Szyperski02] definition is: A software component is an executable unit of independent production, acquisition, and deployment that can be composed into a functioning system. To enable composition, a software component adheres to a particular component model and targets a particular component platform.

What these three definitions have in common are: − Components are units of independent development and acquisition. − Components adhere to a component model that enables composition of

components. Composition is the term used for components, instead of integration.

None of these two aspects are found in OO approach. Some other differences

between component approach and OO approach are: − Instantiation: components may be instantiated or not, and if instantiated there are

usually not many instances of them [Atkinson02].


16

− Components may have state or not (in order to be replaceable they should not have state).

− Components are generally considerably larger than individual classes [Bosch00]. Bachman et al. [Bachman00] list the advantages of CBD as: − Reduced time to market: even if component families are not available in the

application domain, the uniform component abstractions will reduce overall development and maintenance costs.

− Independent extensions: components are units of extension and component models prescribe how extensions are made.

− Easy to acquire: there are component markets to acquire components. − Improved predictability: component frameworks can be designed to support the

quality attributes that are most important. Advantages added by Bass et al. [Bass00] are: − Improved productivity, which can lead to shorter time-to-market. − Separation of skills: complexity is packaged into the component framework and

new roles are added, such as developer, assembler, and deployer. − Components provide a base for reuse, since components are a convenient way to

package value. They have direct usability (may be used to build systems directly); while other approaches such as design patterns are more abstract.

Component-based system development is often promoted in the literature as a low

risk development strategy, which provides a simple and rapid mechanism for increasing the functionality and capability of a system. In reality, CBD carries significant risk throughout the system life cycle. Bass et al. [Bass00] mention inhibitors in CBD as being lack of available components, lack of standards for component technology, lack of certified components, and lack of engineering methods. Some challenges in each development phase and for a project as a whole are discussed here, using [Crnkovic02], [Ghosh02], [Jacobson97] and other various sources as:

− Management: decision-making on make vs. reuse vs. acquire, initiating product families, Return on Investment (ROI), vendor interactions for cost estimates. Although component markets have grown in the recent years, there are few empirical studies that can verify increased productivity or shorter time to market due to acquiring components.

− Component based software life cycle: The life cycle of component-based software is becoming more complex because many phases are separated in unsynchronized activities. For example, the development of components may be completely independent of the development of systems using those components. The process of engineering requirements is much more complex because the possible candidate components usually lack one of more features that the system requires.

− Composition predictability: Even if we assume that we can specify all the relevant attributes of components, we do not necessarily know how these


17

attributes will determine the corresponding attributes of systems of which they may become part.

− Trusted components and component certification: One way of classifying components is to certify them [Voas98a]. In spite of the component belief that certification means absolute trustworthiness, it in fact merely provides the results of tests performed and a description of the environment in which the tests were performed. Although certification is a standard procedure in many domains, it has not yet been established in software in general especially not for software components.

− Component configurations: As soon as we begin to work with complex structures, problems involving structural configurations appear.

− Tools support: The purpose of software engineering is to provide practical solutions to practical problems. Component evaluation tools, component repositories and tools for managing the repositories, component test tools, and component-based design tools, run-time analysis tools, and component configuration tools and so on.

− Dependable system and CBSE: The use of CBD in safety-critical domains, real-time systems and different process – control systems, in which reliability requirements are particularly rigorous, is particularly challenging.

Components can be built in-house, acquired from the third-companies, or just downloaded from the websites of the OSS communities.

− In-house development In-house development means that the provider and the consumer of the software reuse artefacts is the same person or organization. So, a component is a white-box for the person who may use it. Moreover, the component may be only used for special application and domain. Thus, it possible that the component is not based on uniform, standard coordination schemes.

− COTS-based development Practitioners using COTS software equate components to COTS products. According to the perspective of SEI [Brownsword00], “A COTS product is: sold, leased, or licensed to the general public; offered by a vendor trying to profit from it; supported and evolved by the vendor, who retains the intellectual property rights; available in multiple, identical copies; and used without source code modification.” Here, COTS components are mostly executable black-box components.

− Open Source Software Open Source Software (OSS) [Hissam01] makes the definition of component more complex. OSS means that the customers can get the source code of the software and they may change the source code and give back the modified artefacts. Thus, the open source component can also be regarded as a white-box.


18

2.4 COTS-Based Development

COTS software is generally a software that is not developed inside the project, but acquired from a vendor and used “as-is”, or with minor modifications.

Oberndorf [Oberndorf97] defines the term COTS product on the basis of the “Federal Acquisition Regulations”. It is defined as something that one can buy, ready-made, from some manufacturer’s virtual store shelf (e.g., through a catalogue or from a price list). It carries with a sense of getting, at a reasonable cost, something that already does the job. The main characteristics are 1) it exists a priori, 2) it is available to the general public, or 3) it can be bought (or leased or licenced).

Vidger et al. [Vidger96][Vidger97] define COTS as pre-existing software products; sold in many copies with minimal changes; whose customers have no control over specification, schedule, and evolution; access to source code as well as internal documentation is usually unavailable; complete and correct behavioral specifications are not available.

Basili and Boehm [Basili01] specify that COTS has the following characteristics 1) the buyer has no access to the source code, 2) the vendor controls its development, and 3) it has a nontrivial installed base.

Torchiano and Morisio [Torchiano04] give a more detailed, empirically based definition as: A COTS product is a commercially available or open source piece of software that other software projects can reuse and integrate into their own products. They also specify a COTS product:

− Is not produced in or exclusively for the project. − Can be closed source or open source. If it is open source software, it is usually

treated as if it were closed. − Is not a commodity. It is not shipped with the operating system, provided with

the development environment, or generally included in any pre-existing platform.

− Is integrated into the final delivered system. It is not a development tool. − Is not controllable, in terms of provided features and their evolution.

COTS software can be classified into different categories. Morisio et al.

[Morisio02a] summarized some of the previous work and gave a proposal on how to classify COTS. These classification attributes are supposed to be linked with software process (e.g., cost models, selection methods, architectures, testing, and validation techniques).

COTS usage promises faster time-to-market and increased productivity [Voas98c]. Developing software with as much COTS functionality as possible saves you from reinventing the wheel. With COTS software, the functionality you desire can be:

− Accessed immediately. − Obtained at significantly lower prices than otherwise. − Developed by someone who is an expert in that functionality.


19

At the same time, COTS-based software introduces many challenges, such as unknown quality properties of the chosen COTS components can be harmful for the final product and the economic instability of the COTS vendor may terminate the maintenance support of its COTS components [Voas98b]. The advantages and disadvantages of using COTS software are summarized in [Boehm99] as shown in the Table 2-1.

Table 2-1 Advantages and disadvantages of using COTS software

Advantages Disadvantages Immediately available; earlier payback Licensing, intellectual property

procurement delays Avoids expensive development Up-front licensing fees Avoids expensive maintenance Recurring maintenance fees Predictable, confirmable licence fees and performance

Reliability often unknown or inadequate; scale difficult to change

Rich functionality

Too-rich functionality compromises usability, performance

Broadly used, mature technologies Constraints on functionality, efficiency Frequent upgrades often anticipate organization’s needs

No control over upgrades and maintenance

Dedicated support organization Dependence on vendor Hardware/software independence Integration not always trivial;

incompatibilities among vendors Tracks technology needs Synchronizing multiple-vendor

upgrades

2.5 OSS-Based Development

The basic idea behind OSS is very simple: When programmers can read, redistribute, and modify the source code for a piece of software, the software evolves. People improve it, people adapt it, and people fix bugs. This can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing. However, open source does not just mean access to the source code. The distribution terms of OSS must comply with the following criteria, such as free redistribution, open source code, derived work and so on [OSS04].

Many people prefer the term free software to open source software. The term free software dates from 1984, when the idea and arguments for it were first published by Richard Stallman. The idea is not that software should be free “as in beer,” or available at no charge, but that it should be free “as in speech”, so that you can review it and change it as you need to [Freesoftware05]. The term open source dates from 1997, when a group of people, including Eric Raymond, Tim O’Reilly, and Bruce Perens, decided that the term free software and some of the arguments employed in support of it were making the idea less attractive to many businesses. They decided, as a marketing decision, to emphasize technical and practical advantages of open source software rather than arguments from principle [OSS04].


20

OSS software has risen to great prominence. There is excellent evidence that OSS has a significant market share in numerous markets, for example,

− Netcraft’s statistics on web servers shows that Apache is the number one web server with over three times the market share of its next ranked competitor [Netcraft05]

− A 2004 InformationWeek survey found that 67% of companies use OSS products with another 16% expecting to use it in 2005, and only 17% have no near-term plans to support OSS products [Infoweek04].

OSS software offers many choices from large suites to small components. Typical

OSS suites are Linux (operating system), Apache (web server), MySQL (database), PostgreSQL (database), Eclipse (development platform), and OpenOffice (office suites). Examples of software components are zlib (compression/decompression Library), SpamAssassin (spam-checker for email servers), and log4j (logging component). As of 27th April 2006, there are more than 118,615 OSS project registered at sourceforge.net and more than 40,590 project registered at freshmeat.net. The projects are classified according to the application domain of the software, such as clustering, games, networking, and security at sourceforge.net and freshmeat.net.

OSS has many proposed advantages [Fitzgerald04], [Ruffin04] [Madanmohan04]:

− OSS is usually freely available for public download. The collaborative, parallel efforts of globally distributed developers allow OSS to be developed more quickly than conventional software.

− Many OSS products are recognized for high reliability, efficiency, and robustness.

− When OSS is a de facto standard component and has a large user community, it is a lasting solution that can resist the commercial supplier uncertainties that can often abruptly end a product’s life

− Using mainstream OSS reduces cycle time for component updates and corrections. More and often free labour is available to localize and correct defects. Especially for embedded systems, OSS provides fast, new drivers and hardware-related features, even for rather exotic environments.

− Another attraction is that the marginal cost of scaling up is zero. OSS does not require additional licences as installation numbers grow.

Despite its wide appeal, OSS software faces a number of serious challenges and

constraints [Fitzgerald04] [Ruffin04]:

− The popularity of OSS increases the risk that so-called net-negative-producing programmers will become involved.

− Many software tasks, such as documentation, testing, and field support are lacking in OSS projects.


21

− If developers produce or sell a product, which integrates OSS as source code, they may need the licensor’s permission. Otherwise, the licensor might claim for damages or force them to end the product’s further development, delivery, and sale.

− Open source systems lack the comfort zone that a commercially acquired solution provides because support is in the form of bulletin boards and the like.

The studies on OSS focus on three dimensions:

− Since OSS projects have some typical characters, such as parallel development,

fast release, and peer review, some studies investigated how the OSS projects are organized and managed. The intentions of these studies are to summarize the good practice in OSS projects in order to improve the development of commercial software [Feller02].

− Some studies compared the OSS products with COTS products. For example, Paulson et al. [Paulson04] validated several hypotheses about the difference between OSS and COTS products. They concluded that there is still no empirical evidence that OSS fosters faster system growth and there is also no strong evidence that OSS is more modular than COTS software.

− Other studies concentrate on using OSS in the commercial software projects in order to facilitate the development in the IT industry [Madanmohan04].

2.6 OTS Component-Based Development

The granularity of the component in practice/marketplace is generally the same or larger than the theoretical component defined in Section 2.3. The components following COM, CORBA, and EJB models, or software libraries like those in C++ or Java, have all been regarded as components in industries and component marketplace [Componentsource04].

In our study, we investigated the components with the similar granularity as in industrial practice and component marketplace. However, we exclude platform software, such as OS and DB. The OTS components definition used in our studies are:

− Component: Software components are program units of independent

production, acquisition, and deployment that can be composed into a functioning system. We limit ourselves to components that have been explicitly decided either to be built from scratch or to be acquired externally as an OTS-component. That is, to components that are not shipped with the operating system, not provided by the development environment, and not included in any pre-existing platform. That is, platform (“commodity”) software is not considered, e.g., an OS like Linux, DBMSes, various servers, or similar software. Furthermore, components usually follow some component model, as expressed by the COM, CORBA, EJB, .Net, or Web Service standards, or they can be a C++/Java library.


22

− OTS (Off-The-Shelf) component is a component that: • is provided (by a so-called provider) from: - A commercial vendor (Commercial-Off-The-Shelf, COTS), or

- The Open Source community (Open Source Software, OSS). The components may come with certain obligations, e.g., payment or licensing terms. NB: we do not consider proprietary components developed by some other organization in the same company, i.e., internal reuse.

• Is not controllable, in terms of provided features and their evolution. • Is mainly used as closed source, i.e., no source code is usually modified,

even it is available. Typical COTS components are: Vector draw professional (3D modelling), Shamman

Reports for Excel (Database reporting), Dynamic Cube (OLAP). Typical OSS components are: Xerces (XML parser), Kiwi Toolkit (Class library), OpenEJB, Python (Interpreter).

2.7 Summary and Discussion

This chapter shows that software reuse is the systematic practice of developing software from a stock of building blocks. The components can either be an in-house built component, a COTS component, or an OSS component. Different kinds of components have different advantages and challenges. Moreover, the challenges facing reuse and CBD are structural, organizational, managerial, and technical. Focusing only on the technological issues usually does not bring the whole benefit of reuse and CBD. In this thesis, we focus on research challenges as follows:

− How to prepare for reusing more in-house built components? Component reuse is generally an incremental procedure. The company will build some reusable components in the beginning. In case of successful reuse, more and more code will be encapsulated into reusable components. The more reusable components are developed, the more complex the development process will be, and more support is required from the organization [Crnkovic01]. Most studies on software reuse focus on the final results of reusing the in-house built components without investigating how the reuse has been developed inside an organization. The challenge is to examine the relationship between companies’ reuse level and some key factors in component-based development, so that company with low reuse level can make necessary preparation when moving to a higher reuse level. This research challenge is formulated into research question RQ1 in this thesis. − How to manage OTS component-based projects? The use of COTS or OSS components has both advantages and challenges. Many researchers have summarized their experience on managing these risks through case studies in specific domains. One of the limitations of a case study is that the conclusion is difficult to be generalized if the studies are limited into a specific context. Therefore, it is necessary to do further empirical studies on managing the


23

challenges in reusing the OTS components in different project contexts, so that the project manager can efficiently customize their process, organization, and technology according to their needs. This research challenge is investigated by research questions RQ2 and RQ3 in this thesis. − What are the differences between using COTS and OSS components? Although both COTS and OSS components claim to save development effort, they are different in their nature and appearance. When project members want to use OTS components, they must select to either use COTS or OSS. It is therefore important for them to be able to compare the benefits and risks of using COTS or OSS components. This research challenge is investigated by research question RQ4 in this thesis.


24

Process improvement

25

3 Process Improvement in OTS Component-Based Development

This chapter starts by introducing some software development models and the challenges of process improvement in general. After that, it describes the proposed processes in OTS component-based development and OTS component selection. Finally, challenges facing our study are discussed.

3.1 Software Process Improvement (SPI)

A process is the glue that ties together people, technology, organizational structures, and management in a coherent whole, focusing on the business goals and objectives [Zahran98]. A process has three aspects. To start with, the process must be defined. Therefore, the first step is the process definition. Secondly, the process knowledge must pass to those who will perform it; hence the second aspect is process learning. The third aspect is the process results as manifested by the products produced as a result of executing the process activities. A software process is a set of activities that leads to the production of a software product [Sommerville04]. Although there are many software processes, some fundamental activities are common to all software processes:

- Software specification: the functionality of the software and constraints on its operation must be defined.

- Software design and implementation: the software to meet the specification must be produced.

- Software validation: the software must be validated to ensure that it does what the customer wants.

- Software evolution: the software must evolve to meet changing customer needs. A framework proposed for software process improvement comprises four parts

[Zahran98]: - Software process infrastructure, which includes organization, management, and

technical infrastructure. - Software process improvement roadmap, which specifies a model for

characterizing the software process and a logical step-by-step approach towards

Process improvement

26

the realization of effective software processes. It could be one of the standard public process roadmaps, for example the software capability maturity model (SW-CMM) [CMM06] or maturity levels of ISO/IEC 15504 [ISO06], or a tailored version to match the organizations’ own needs.

- Software process assessment method, which specify methods and techniques for assessing the organizations’ current software process, practice, and infrastructure.

- Software process improvement plan, which involves transforming the assessment findings into specific actions for software process improvement.

There are three main types of processes: management, engineering, and support.

Within a software project, key processes could be classified in three major process classes as follows:

- Management processes - Software engineering (life cycle) processes - Support processes (life cycle support)

A software process model is an abstract representation of a software process life

cycle [Sommerville04]. Each process life cycle represents a process from a particular perspective, and thus provides only partial information about that process. The traditional process models are: Code and Fix, Waterfall, V-Model, Spiral, Evolutionary Prototyping, and Adaptive Software Development.

- Code and Fix: Developers begin work without a set of good requirements or a

vision, writing code as long as there is time and money. This approach has no planning and therefore yields early results. High maintenance and rework effort are the result of this model.

- Waterfall: This model is a sequential, document-driven methodology. To advance from the current phase to the next phase, the project team must review and release an artefact in the form of a document [Royce87].

- V-model: This model is an extended waterfall model, adding more details on the validation and verification side. For each phase in development, there is an associated phase to verify and validate the result produced [V-Model05].

- Spiral: The spiral model breaks the project down into risk-oriented subprojects [Boehm88], [Boehm98]. The advantage of this model is the breakdown of development into small pieces, so that the areas of risk can be tackled at the beginning of the project.

- Evolutionary Prototyping: In this model, the system starts with an initial idea or proposal, which is then prototyped and released to the customer in incremental releases based on feedback.

- Adaptive Software Development: The adaptive software development model is an iterative, risk- and mission- driven, component-based, time-boxed, and change-tolerant process [Highsmith99]. It works well for small teams (four to eight people) when the requirements are uncertain or domain knowledge is missing.

Process improvement

27

The composed process models, which cover several aggregations, are: - Unified Process: The author of the Unified Modelling Language (UML) defined

a software process model called the Unified Process (UP) [Jacobson99]. The UP is a risk- and use-driven, architecture-centric, iterative and incremental software development model. The Rational Unified Process (RUP) [Kruchten00] is a commercial extension of the UP.

- Extreme Programming: The eXtreme Programming (XP) model [Kent99] is a flexible, lightweight, people- and result-oriented development process that allows for requirement changes at any time during development. XP defines 12 core practices that are essential to the success of an XP project, such as test-driven development, planning game, on-site customer, pair programming, and so on.

A software development process model should not be dumped blindly onto the

project but rather should be carefully chosen according to the project, the environment, and the goals. In addition, researchers and practitioners have realized that processes cannot be defined and frozen once for all. Processes need to continuously undergo changes and refinements to increase their ability to deal with the market expectations and requirements of the company stakeholders. Hence, process need to be continuously assessed and improved [Fuggetta00].

3.2 SPI of OTS Component-Based Development

Although an OTS component can be either COTS or OSS component, most process improvement studies focus on COTS-based development. Typically, a COTS-based process consists of four phases [Abts00], comprising:

- COTS component assessment and selection - COTS component tailoring - COTS component integration - Maintenance of COTS and non-COTS parts of the system

3.2.1 Improvement of the Whole Life Cycle There is consensus that the use of a COTS component implies changes in the software process [Brownsword00]. Most studies of a COTS-based development process focus on two dimensions: The whole software development life cycle and process of a specific phase, especially in COTS component selection and evaluation

Boehm and Abts [Boehm99] regard both the waterfall model and evolutionary development as unsuitable for COTS-based development because:

- In the sequential waterfall model, requirements are identified at an earlier stage

and the COTS components are chosen at a later stage. This increases the likelihood of the COTS components not offering the required features.

- Evolutionary development assumes that additional features can be added if required. However, COTS components cannot be upgraded for one particular

Process improvement

28

development team. The absence of source code bars the development team from adjusting the COTS components to their needs.

Based on the above arguments, Boehm and Abts [Boehm99] proposed that

development models which explicitly take risk into account are more suitable for COTS-based development than the traditional waterfall or evolutionary approaches.

The Software Engineering Institute has a large ongoing effort to address the development of COTS-based systems, and they have developed the Evolutionary Process for Integrating COTS-based Systems (EPIC) [Albert02]. The proposed process integrates COTS-related roles and activities into a RUP process. The iterative and evolutionary nature inherent in this process allows developers to adjust the architecture and system design, as more knowledge is gained about the operations of the COTS components.

The National Aeronautic and Space Administration (NASA) has been developing IT systems with COTS for many years. Their experience has been captured by Morisio et al. [Morisio00]. They have investigated the various processes that were used across 15 projects at NASA. Based on their insight, they have developed a COTS-based development process that was the most representative for the processes used in the projects.

3.2.2 Improvement of the OTS Component Selection To perform a successful OTS component-based project, one of the key issues is to select the right OTS component. Most previous studies focus on proposing selection processes and methods to select COTS components. One kind of such processes is based on direct assessment [Leung02], such as MBASE (Model Based Architecting and Software Engineering) [Boehm00], OTSO (Off-the-Shelf-Option) [Kontio96], CISD (COTS-based Integrated System Development) [Tran97a], PORE (Procurement-Oriented Requirement Engineering) [Maiden98], CAP (COTS Acquisition Process) [Ochs01], IIDA (Infrastructure Incremental Development Approach) [Fox97], CARE (COTS-Aware Requirements Engineering) [Chung02], RCPEP [Lawlis01], and CRE (COTS-Based Requirements Engineering) [Alves03]. In general, the direct assessment process includes three basic steps:

- Inspect all modules of each of the available COTS components to check whether they satisfy some or all of the functional requirements of the COTS-based system being developed.

- Check whether a COTS component also satisfies the non-functional requirements of the COTS-based system. Non-functional requirements may include properties such as the interoperability of the modules of the COTS product with other systems.

- Compare the COTS component candidates and select the most appropriate COTS component that satisfies both the functional and non-functional requirement of the COTS-based system.

Some of the direct assessment processes, such as OTSO [Kontio96], CAP [Ochs01],

and CISD [Tran97a], assume that the requirements are fixed and select the COTS components by comparing how well the COTS component candidates satisfy the

Process improvement

29

requirements. A formal decision-making process is usually used to select the best COTS component [Ncube02]. The formal decision-making process usually includes three basic elements: selecting evaluation criteria (factors), collecting and assigning values to these criteria, and applying formal decision-making algorithms such as MAUT [Maccrimmon73], MCDA [Morisio97], and AHP [Saaty90]. Other direct assessment processes, such as MBASE [Boehm00], PORE [Maiden98], IIDA [Fox97], CARE [Chung02], RCPEP [Lawlis01], and CRE [Alves03] emphasize the trade-offs between the requirements and COTS component functionalities. These processes do not assume that the requirements are fixed and should not be changed. On the other hand, these processes propose that the requirements in COTS-based development should be flexible enough so that they can be negotiated or changed according to the available functionalities of the COTS components.

Another kind of process is based on a domain-model [Leung02]. It includes a set-up phase and a selection phase. In the set-up phase, the vendors need to map their COTS modules to those modules of the domain that they find are applicable. In the selection phase, the corresponding modules in a domain model are identified for each of the modules of the COTS-based system in question. Then, the COTS modules that claim to be applicable are identified by the mapping from the domain models to the COTS modules. After that, the non-functional properties of the identified COTS modules are assessed. In the end, the most appropriate COTS modules are selected with reference to all of the assessment results.

After performing seven structured interviews of small and medium software companies in Norway and Italy, Torchiano and Morisio [Torchiano04] proposed two these T4 and T5 that reflect the state-of-the-practice of SPI in COTS-based development.

- T4: Developers seldom use formal selection procedures. Familiarity with either

the product or the generic architecture is the leading factor in selection. - T5: Architecture is more important than requirements for product selection.


This chapter has discussed the software development process improvement in general. It has also briefly introduced the proposed process improvement in COTS-based development. Finally, several COTS selection processes have been introduced. Since both the COTS and OSS components are acquired from third-party companies, the users usually have no control of the provided functionality and evolution of these components. Therefore, we generally believe that most proposed process improvement in COTS component-based development can also be used in OSS component-based development.

The limitations of current studies are that few of them have been empirically verified. In this thesis, we focus on answering:

− What are the state-of-the-practice of OTS component-based development processes and OTS component selection processes?

Since most proposed COTS-based development processes are either based on military projects or very large projects, it is difficult for small and medium-sized

Process improvement

30

projects to follow their proposals. It is therefore necessary to summarize the relationship between the results of the project and the process improvement activities performed. The intentions is to give guidelines about how to effectively improve the development process to get better project outputs, such as better quality products and shorter effort spent on development and maintenance. This research challenge is formulated into research question RQ2 in this thesis.

Risk management

31

4 Risk Management in OTS Component-Based Development

This chapter introduces risk management theory in software development projects. Then, it describes the particular risks and risk management activities in OTS component-based development. Finally, the challenges facing this thesis are discussed.

4.1 Risk Management in Software Development

Risk management is increasingly seen as one of the main jobs of project managers. It involves anticipating risks that might affect the project schedule or the quality of the software being developed and taking action to avoid these risks [Hall98] [Ould99].

Risk management includes two primary steps each with three subsidiary steps as described by Boehm [Boehm91] and shown in Figure 4-1.

Figure 4-1 Risk management framework

Risk identification produces lists of project-specific risk items likely to compromise

the success of a project. Risk analysis assesses the loss probability and loss magnitude

Risk management

32

for each identified risk item. Risk prioritization ranks the risk items identified and analysed. Risk management planning helps prepare to address each risk item. Risk resolution produces a situation in which the risk items are eliminated or otherwise resolved. Risk monitoring involves tracking the progress of a project towards resolving its risk items and taking corrective activities where appropriate.

In order to identify risks, a list of possible project risks must be produced. Boehm [Boehm89a] lists the top 10 risks by probing several large software projects and their common risks. Barki et al. [Barki93] identified 35 variables, which are subsequently used as a basis for creating scales in a project risk assessment instrument. The Software Engineering Institute (SEI) [Carr93] has contributed significantly to the literature with the SEI Taxonomy-Based risk identification instrument, which contains 194 questions. However, the notion of building single all-encompassing risk taxonomy to be globally used by all software developers is probably unrealistic. Thus, we may need different risk taxonomies in different project contexts [Moynihan97].

A few other studies have classified software risk items, i.e., consider software risks along several dimensions and have provided some empirically founded insights of typical software risks and their variation. The risks can be classified into technical, organizational, and environmental [Sherer95]. The possible risks can also be summarized into a hierarchical holographic modelling (HHM) framework [Chittister96]. Ropponen and Lyytinen [Ropponen00] empirically delineate six components of software development risks, such as scheduling and timing risks, system functionality risks, subcontracting risks, requirement management risks, resource usage and performance risks, and personnel management risks. Keil et al. [Keil98] extracted the experience of more than 40 software project managers and identified a universal set of risk factors, such as customer mandate, scope and requirements, environment, and execution. A following study [Wallace04] investigated the relationship between these factors and their effect on the result of the project.

During the risk analysis process, project managers have to consider each identified risk and make a judgment about its probability to occur and seriousness. The project managers can tabulate the results of this analysis process using a table ordered according to the serious of the risk [Sommerville04]. The estimation of the seriousness and the rank of the risk should be combined with cost estimation [Madachy97], [Kansala97]. Once the risks have been analysed and ranked, the project managers need to assess which are the most significant risks.

The purpose of identifying and analysing the risks is to manage these possible risks. The possible risk management activities have been proposed from processes [Boehm88], organization [Roppoen00], [Gemmer97], and technology [Hecht04] viewpoints. However, many researchers advocate rather than investigate. As a result, some advocated concepts are worth less than their advocates believe and there is a shortage of evaluative research to help determine the actual value of new tools and techniques [Glass01]. Although risk management proposals of the in-house software development have been verified by a survey-based research [Verner05], few studies have been performed to empirically investigate the risk management proposals in using the OTS components.

Risk management

33

4.2 Risk Management in OTS Component-Based Development

In addition to the classical risks associated with developing large systems, OTS components requires managers to modify their typical mitigation activities for some of the classic risks and to develop new mitigation activities for risks that are particular to the use of OTS components in a system.

Vigder et al. [Vigder96] investigated the specific risk issues in the COTS-based development and point out the fundamental differences between COTS-based development and building in-house are:

- COTS-based development may need infrastructure earlier to demonstrate and

validate the COTS component. - The COTS component may dictate standards, architecture, and design. - The COTS component may influence work flow. - Picking the wrong COTS component may be more expensive than fixing

problems in the in-house built software. - Issue resolution processes need to be in place earlier to resolve COTS component

issues. - Issue resolution processes may be more complicated because of the addition of

the vendor and possible incompatibilities between the vendor’s practices and the users.

Boehm et al. [Boehm03] summarized experience from their COTS-based university

projects. Examples of the summarized risks and corresponding risk management activities (in brackets) are:

- Requirement changes and mismatches (prototyping, win-win negotiation among

all stakeholders). - Overly optimistic expectation on COTS quality attributes (significant quality

features must be tested). - Overly optimistic COTS component learning curve (a most likely COTS

component learning curve must be accessed during planning and scheduling). - Version upgrade may result in retailoring of the COTS component (ensure the

new version still can implement the capability of the old version). Torchiano and Morisio [Torchiano04] discovered several phenomena about risks and

risk management in COTS component-based development and proposed theses T2, T3, and T6 as:

- T2: Integration problems result from lack of compliance with standards.

Architecture mismatches constitute a secondary issue. - T3: Custom code mainly provides additional functionalities. - T6: Integrators tend to influence the vendor on product evolution whenever

possible.

Risk management

34

The software engineering resource centre at the Federal Aviation Administration (FAA) [FAA03] summarized 10 risk factors in COTS-based development and proposed corresponding risk mitigation activities. The COTS lessons-learned in a repository made by Basili et al. [Basili02] summarized the possible risk management activities based on case studies. In these studies, COTS software can be components or very large packages, such as ERP (Enterprise Resource Planning), CRM (Customer Relationship Management), Data Bases, or Operating Systems.

A few other studies contribute to investigate risk management in OSS-based development. For example, Ruffin and Ebert [Ruffin04] point out the possible licensing problem of using OSS components. Fitzgerald [Fitzgerald04] mentioned that organizations should not expect maintenance and support to be available at a lower cost that would apply for commercial software, even though OSS may be available at little or no cost. Other researchers mention that freely available source code addresses two typical concerns with using COTS components: unknown implementation quality and long-term vendor support [Madanmohan04].

Other researchers have focused only on the component-based development without telling the differences between in-house built, COTS, or OSS components [Bass00], [Crnkovic02], [Ghosh02], [Jacobson97]. Kotonya and Awais [Kotonya01] put risk management in component-based development into the risk-weighting scheme of [Moynihan97] and listed several questions in each risk item to help project managers to quantitatively assess the risks.

Some studies focus only on COTS components. Rose [Rose03] classifies possible risks in COTS component-based development based on the different phases. Examples of possible risks and corresponding risk management activities (in brackets) are:

- Lack of skilled personnel (ensure that there is an appropriate number of senior

technical personnel assigned to the project). - Architecture mismatch (design the architecture first and select COTS component

that best fits the architecture). - Product obsolescence (maintain a continual watch on the marketplace looking for

substitute products as well as new technologies that might replace older ones). - COTS components have asynchronous update cycles with the COTS-based

system (involve senior analysts directly involved in the analysis of the upgrades and derive effective field upgrade procedure and schedules)

The CBSEnet project [CBSEnet04] investigated the state-of-the practice in COTS

component-based development and summarized the challenges from business, information technology, and legal perspectives.

Our pre-study (see paper P2 [Li06a] in Appendix A) summarized the problems encountered and presented good practices from finished COTS component-based projects. Examples of the problems encountered are:

- Could not negotiate requirements with the customer. - The learning effort was underestimated. - It was difficult to locate defects. - The COTS component could not be deployed as advocated by the vendor.

Risk management

35

Examples of good practices are: - Integrate unfamiliar COTS component first. - Do integration testing early and incrementally. - Investigate COTS component market and prepare to replace obsolescent

components. Since the OTS components involve both COTS and OSS components, risks and risk

management activities relevant to OTS components-based development are therefore the subset of risks and risk management in component-based development, COTS-based, and Open Source based development, as shown in Figure 4-2.

Figure 4-2 Risk management in OTS component-based development


This chapter has discussed risk management in software projects in general. Due to the specific attributes of OTS components, the OTS component-based project is more risky than a traditional software development project. The proposed risk identification and risk management activities are briefly introduced. The relevant research challenges in this thesis are:

− What are the results of the proposed risk management activities? Although many risk management activities have been proposed to avoid the typical risks in OTS component-based development, they are basically based either on common sense or few case studies. It is therefore necessary to empirically verify the effect of performing such risk management activities in projects with different profiles. This research challenge is formulated into research question RQ3 in this thesis

Risk management

36

− What are the differences of risks in COTS component-based development and OSS component-based development?

As we have shown earlier, there are both similarities and differences between the risks in COTS and OSS component-based development. It is therefore important to compare what had happened in finished COTS and OSS component-based projects. The intention is to help project managers to compare the possible benefits and risks of buying the COTS components, or downloading the OSS components. This research challenge is formulated into research question RQ4 in this thesis.

Research methods

37

5 Research Methods

This chapter provides a brief review of empirical research approaches and strategies in the empirical software engineering field. Since the survey method has been mainly used in several studies in this thesis, the advantages, disadvantages and guidelines for using this method are discussed. The research designs for each study in this thesis are described. Validity threats for all studies and in particular how to overcome these in our studies are further discussed.

5.1 Research Strategies in Empirical Software Engineering

Empirical research is research based on the scientific paradigm of observation, reflection and experimentation as a vehicle for the advancement of knowledge [Endres03]. Empirical studies may have different purposes, being exploratory (investigating parameters or doing a pre-study to decide whether all parameters of a study are foreseen), descriptive (finding distribution of a certain characteristics), or explanatory (investigating why certain phenomena happen).

There are three types of research paradigms that have different approaches to empirical studies [Wohlin00], [Creswell94], [Creswell03], [Seaman99]:

- Qualitative research is concerned with studying objects in their natural setting. A

qualitative researcher attempts to interpret a phenomenon based on explanations that people bring to them [Denzin94].

- Quantitative research is concerned with discovering causes noticed by the subject in the study, and understanding their view of the problem at hand. A quantitative study is mainly concerned with quantifying a relationship or to compare two or more groups [Creswell94]. The quantitative research is often conducted through setting up controlled experiments or collecting data through case studies or surveys.

- The mixed-method approach is evolved to compensate for limitations and biases of the above strategies, seeking convergence across other methods. The combination of quantitative and qualitative methods is usually more fruitful than either in isolation [Seaman99]. How to combine the qualitative and quantitative method in the design is described by [Basili86] and further discussed in [Seaman99].

Research methods

38

Depending on the purpose of the evaluation, whether it is techniques, methods, or tools, and depending on the conditions for the empirical investigation, the empirical research strategies can be classified into different categories. Zelkowitz and Wallace [Zelkowitz98] summarized 12 technology validation models and grouped these models into categories according to the data collection methods: observational, historical, and controlled. The validation models in [Zelkowitz98] include models to examine both the projects (e.g., case study, project monitoring, and field study) and products (e.g., static analysis, simulation, and dynamic analysis).

In the empirical software engineering field, three major different types of investigations are carried out frequently.

- Survey. A survey is often investigations performed in retrospect, when, for

example, a tool or technique, has been in use for a while [Pfleeger94]. The primary means of gathering qualitative or quantitative data are interviews or questionnaires. The results from the survey are then analysed to derive descriptive and explanatory conclusions. They are then generalized to the population from which the sample was taken. Surveys are discussed further in [Robson02] [Fowler01].

- Case study. Case studies are used for monitoring projects, activities or assignments. The case study is normally aimed at tracking a specific attribute or establishing relationships between different attributes. Case studies are discussed further in [Yin03].

- Experiment. Experiments are normally done in a laboratory environment, which provides a high level of control. When experimenting, subjects are assigned to different treatments at random. The object is to manipulate one or more variables and control all other variables at fixed levels. The effect of the manipulation is measured. Based on the measurement statistical analysis can be performed. A case study is an observation study while the experiment is a controlled study [Zelkowitz98]. Experiment design is discussed further in [Pfleeger94].

Some of the research strategies could be classified both as qualitative and

quantitative strategies depending on the design of the investigation [Wohlin00], as shown in Table 5-1.

Table 5-1 Qualitative vs. quantitative in empirical strategies

Strategy Qualitative/Quantitative Survey Both (mostly quantitative) Case study Both Experiment Quantitative

The important question in research design is when to use each strategy. If the

problem is identifying factors that influence an outcome or test the effect of some manipulation, quantitative approaches are chosen. If the problem is to understand why the results are as they are or to identify causes, a qualitative approach is better. The mixed method approach uses different methods in the various phases of a study. Yin [Yin03] answers that question of choosing an approach by listing three conditions:

Research methods

39

- The type of research question posed. How and why questions are explanatory, and usually should be studied over time in replicated experiments or case studies. What, who, where, how many, or how much questions asks about the frequency or describe the incidence of a phenomenon.

- The extent of control. Only in experiments, the researcher can control treatments or behavioural events. In a case study, the researcher cannot control treatment, but may control the measures to be collected.

- The focus. The experiment or case study focuses on the contemporary as opposed to the surveys, which focuses more on the historical events.

5.2 Survey Methods

The general objectives for conducting a survey are one of the following [Babbie90]:

- Explorative. Explorative surveys are used as a pre-study to a more thorough investigation to assure that important issues are not foreseen. Creating a loosely structured questionnaire and letting a sample from the population answer it could do it. The information is gathered and analysed, and the results are used to improve the full investigation.

- Descriptive. Descriptive surveys can be conducted to enable assertions about some population. This could be determining the distribution of certain characteristics or attributes. The concern is not about why the observed distribution exists, but instead what the distribution is.

- Explanatory. Explanatory surveys aim at making explanatory claims about the population. For example, when studying how developers use certain inspection techniques, we might want to know why some developers prefer one technique while the others prefer another.

The two most common data collection methods for survey are questionnaires and

interviews [Babbie90]. Questionnaires could both be provided in paper form or in some electronic form, for example, email or internet. Comparing with the questionnaire, the advantages of interviews are:

- Interview surveys typically achieve higher response rates than, for example, a

mail survey. - An interviewer generally decreases the number of “do not know” and “no

answer”. - It is possible for the interviewer to observe and ask questions. The disadvantage of an interview is that it usually costs more effort than a

questionnaire. However, the interview can include several open questions to help to get new insights on the research question.

Interviews come in three types. In [Lincoln85], a structured interview is described as one in which “the questions are in the hands of the interviewer and the response rests with the interviewee,” as opposed to an unstructured interview in which the interviewee is the source of both questions and answers. In an unstructured interview, the object is to elicit as much information as possible on a broadly defined topic. The interviewer

Research methods

40

does not know the form of this information ahead of time, so the questions asked must be as open-ended as possible. In a structured interview, the interviewer has very specific objectives for the type of information sought in the interview, so the questions can be fairly specific. Another form of the interview is the semi-structured interview. These interviews include a mixture of open-ended and specific questions, designed to elicit not only the information foreseen, but also unexpected types of information. One useful tool to gather the data during an interview is the interview guide [Taylor84]. It usually consists of a list of close questions and open-ended questions, possibly with some notes about the direction in which to steer the interview under different circumstances.

To collect the data of the survey, samples should be selected from the target population. The basic idea of sampling is that by selecting some of the elements in a population, we may draw conclusions about the entire population. The sample selection methods should consider [Cooper03]:

- What is the relevant population? - What are the parameters of interests? - What is the sampling frame? - What is the type of the sample? - What size sample is needed? - How much it will cost?

The sample can be selected on a probability basis, which is called probability

sampling. The probability sampling is a controlled procedure that assures that each population element is given a known non-zero chance of selection. Several strategies, such as simple random, systematic, stratified, cluster, and double, can be used in the probability sampling [Cooper03]. In contrast, non-probability sampling is arbitrary (non-random) and subjective. Each member does not have a known non-zero chance of being included. Typical non-probability sampling methods include convenient sampling, purposive sampling, and snowball sampling [Cooper03].

The result of the survey can be analysed either qualitatively or quantitatively. In the interview with open-ended questions, the analysis method can be constant comparison [Glaser67], as cross-case analysis [Eisenhardt89]. For the closed questions in the interview or questionnaire, the data analysis will usually use statistics methods. In order to present the result of the survey, the profile should be described and the details of the data analysis methods should be specified [Kitchenham02].

5.3 Research Design

5.3.1 Research Design of the Preliminary Study The purpose of the preliminary study (see paper P1 [Li04a] in Appendix A) is to investigate RQ1 (What is the status of component-based development based on in-house built components?).

To reuse in-house components successfully, developers must follow three basic steps [Mili95]:

- Step 1: Formulate the requirements in a way that supports retrieval of potentially useful reusable components.

Research methods

41

- Step 2: Understand the retrieved components. - Step 3: If the retrieved components are sufficiently close to the needs at hand and

are of sufficient quality, then adapt them.

From these steps, we have selected several key factors. For step 1, we focus on the efficiency of component-related requirements (re)negotiation and the value of component repository. For step 2, we study how knowledge about components can be transferred from a component provider to a component user. For step 3, we focus on definition and reasoning of quality attributes for components.

The sub-questions and hypotheses are as following. Detailed rationale for these sub-questions are in Section 3.1 of paper P1 [Li04a] in Appendix A.

- RQ1.1: Does requirements (re)negotiation for in-house components really work

as efficiently as people assume? - RQ1.2: Does the efficiency of component-related requirements (re)negotiation

increase with more in-house built components available?

Concerning the RQ1.2, the null hypothesis H01 and alternative hypothesis HA1 are: o H01. There is no relationship between the companies’ reuse level and the

efficiency of component-related requirements (re)negotiation. o HA1. There is a positive relationship between the companies’ reuse level and

the efficiency of component-related requirements (re)negotiation.

- RQ1.3: Does the value of component repository increase with more reusable components available?

Concerning the RQ1.3, the null hypothesis H02 and alternative hypothesis HA2 are: o H02. There is no relationship between the companies’ reuse level and the

value of component repository. o HA2. There is a positive relationship between the companies’ reuse level and

the value of component repository. - RQ1.4: How can a component user acquire sufficient information about relevant

components? - RQ1.5: Does the difficulty of component documentation and component

knowledge management increase with increased reuse level?

Concerning the RQ1.5, the null hypothesis H03 and alternative hypothesis HA3 are: o H03. There is no relationship between the companies’ reuse level and

developers’ satisfaction with component documentation. o HA3. There is a negative relationship between the companies’ reuse level

and developer’ satisfaction with component documentation.

Research methods

42

- RQ1.6: Do developers trust the quality specification of their in-house built

components? If the answer is no, how can they solve this problem?

In this study, we used a questionnaire to collect the data. The questionnaire included five parts. The questions in the first part were used to investigate the reuse level of the companies. The definition of reuse level in this study is the number of reused components vs. the number of total components in the organization. The other four parts were organized based on the four key factors. Each question in the questionnaire was used to study one or more research questions. The details of questions are given in the Table 5-2. The correspondences between the questions in the questionnaire and research questions are shown in Table 5-3. To increase the reliability of our survey, the questionnaire also included the definition of concepts used in the questionnaire, and the questions about the respondents’ personal information.

Table 5-2 Questionnaire used in the preliminary study

Reuse level Q1. What is the reuse level in your organization? Q2. To what extent do you feel affected by reuse in your work? Component-related requirements (re)negotiation Q3. Are requirements often changed/ (re)negotiated in typical development

projects? Q4. Are requirements usually flexible in typical projects? Q5. Do the component-related requirements (re)negotiation processes work

efficiently in typical projects? Value of component repository Q6. Would the construction of a reuse repository be worthwhile? Component understanding Q7. Do you know the architecture of the components well? Q8. Do you know the interface of the components well? Q9. Do you know the design rules of the components well? Q10a. Is the existing design/code of reusable components sufficiently

documented? Q10b. If the answer of Q10a is ‘sometimes’ or ‘no’, is this a problem? Q10c. If the answer of Q10a is ‘sometimes’ or ‘no’, what are the problems with

the documentation? Q10d. If the answer of Q10a is ‘sometimes’ or ‘no’, how would you prefer the

documentation? Q10e. What is your main source of information about reusable components

during implementation? Q10f. How do you decide whether to reuse a component ‘as-is’, ‘reuse with

modification’ or ‘make a new one from scratch’? Quality attributes specification of components Q11. Are specifications the quality attributes of components well defined? Q12. Do you test components after modification for their quality attributes

before integrating them with other components?

Research methods

43

Table 5-3 Correspondence between questions and RQs in the preliminary study RQ1.1 RQ1.2 RQ1.3 RQ1.4 RQ1.5 RQ1.6Q1-Q2 X X X Q3-Q5 X X Q6 X

Q7-Q10f X X Q11-Q12 X

For research questions RQ1.1, RQ1.4, and RQ1.6, the results were analysed by

examining their distribution. Since the research questions RQ1.2, RQ1.3, and RQ1.5 are related to the reuse level of the companies, the results were analysed with Spearman Rank Correlation Coefficient in SPSS version 11.0.

5.3.2 Research Design of the Pre-study The intention of the pre-study (see paper P2 [Li06a] in Appendix A) is to do an exploratory study on RQ2 (What were the development processes and OTS component-selection processes used in OTS component-based development projects?) and RQ3 (Which risk management activities have been performed and the result of performing them?)

Two sub-questions RQ2.1 and RQ2.2 examine RQ2 and the other, i.e., RQ3.1, examine the RQ3. The background and rationale for these sub-questions are in Section 3 of Paper P2 [Li06a] in Appendix A.

- RQ2.1: What was the actual process used in the projects using COTS components?

- RQ2.2: What are the commonalities and possible variations in COTS-based development processes?

- RQ3.1: What are process scenarios (variations) of the projects using COTS components successfully and not successfully? What are possible problems and good practices in different process scenarios?

Another intention of this study is to clarify research questions to test the newly

published theses in COTS-based development. The study performed by Torchiano and Morisio [Torchiano04] with seven interviews reveals six theses, which contradicted widely accepted (or simply undisputed) insights in COTS-based development. Since the sample size of that study was small, it is necessary to investigate the six theses in a larger and randomized sample. To test the theses, we decided to collect more data using a randomized sample. Our initial plan was to transfer the theses directly into hypotheses and make a quantitative survey with standardized questions in a formal questionnaire. We designed six fixed hypotheses according to the theses and a preliminary questionnaire. After two rounds of pre-testing, we discovered that it was difficult to reliably investigate the six theses without a pre-study to clarify our research questions. We therefore decided to resort to structured interviews, i.e., a qualitative pre-study, before using a larger and randomized sample.

We used an interview guide (see Appendix C) to collect data. The interview guide includes both closed and open-ended questions. The closed questions were used to

Research methods

44

solicit information on project context. For example, one question used the application domains of the final system (product), which was based on the North American Industrial Classification System (NAICS) [NAICS04]. The open-ended questions were used to gather information on actual COTS-based development processes. The questions covered the main life cycle process, the process changes due to using COTS components, and the experience and lessons learned in the whole COTS-based development process.

Interviews and data collection were conducted from Nov. 2003 to Feb. 2004. The contacted 34 IT companies were mostly involved in two Norwegian R&D Projects [INCO00], [SPIKE002] and one EU R&D project [FAMILIES03].

A pre-process was used to select relevant COTS-based projects. Contact people (IT managers in the most cases) in the above projects were first contacted by phone and email, using an introductory letter with our definition of COTS component and project (the same as the definitions in the interview guide). They were asked to check if they have a relevant COTS-based project. During this pre-process, 17 companies (among 34) did not match our definition of a COTS-based project, and therefore could not join the study. Of these 17 “rejected” companies, 13 used only in-house built components and 3 used only OSS components. There was one other company that could not participate because of confidentiality issues.

For the remaining 17 companies, 13 of these volunteered to join the study. The contact people in these 13 companies were asked to recommend people with most experience on the relevant projects. In all, 16 people were recommended from these 13 companies. For each recommended respondents, they were asked to select one relevant COTS-project. If a respondent claimed that he/she had experience from several COTS-based projects, the respondent was asked to select one that he/she was most familiar with. The whole process to find a suitable respondent and project in a prospective company, and later agree upon time and place for a physical interview took weeks.

The interview guide was sent to respondents a few days before the personal interview. In this way, the respondents were well-prepared and could have time to remember the details of the projects and find some relevant documents. Each interview was conducted by one of the two interviewers (none with two interviewers), took from 60 to 80 minutes, and was recorded.

The first step in the data analysis was to listen to the tape, and supplement and transcribe the interview “as-is” into a field note. After that, we used different analysis procedures based on the purpose of the different research questions.

The purpose of research question RQ2.1 is to investigate the actual development processes used in the project. The data was analysed by the constant comparison method [Strauss98].

The purpose of research question RQ2.2 is to see the similarities and differences between process changes in the studied projects. The main analysis method used is cross-case analysis [Strauss98].

The purpose of research question RQ3.1 is to summarize the process scenarios of the successful and unsuccessful projects. We first grouped projects into two groups based on the positive or negative effect of the COTS components regarding time-to-market and general quality of the system. In each group, we divided projects into sub-groups based on their main actual processes used. For each sub-group, we performed a similar cross-case analysis [Strauss98], as for research question RQ2.2.

Research methods

45

Detailed processes of data analysis of RQ2.1, RQ2.2 and RQ3.1 are in Section 3.6 of Paper P2 [Li06a] in Appendix A.

5.3.3 Research Design of the Main Study The research focus of the main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c], and P6 [Li06d] in Appendix A) includes four parts: 5.3.3.1. Part one: Verify our conclusions about RQ2 (What were the development processes and OTS component-selection processes used in OTS component-based development projects?).

Based on our findings in the pre-study, we designed sub-questions RQ2.3 to RQ2.8. The background and rationale for these sub-questions are in Sections 2.1 and 2.2 in Paper P4 [Li06b] in Appendix A.

- RQ2.3: What were the actual development processes in OTS-based projects? - RQ2.4: Was the actual development process decided before the make vs.

acquire decision or after the make vs. acquire decision. - RQ2.5: Who decided the actual development process? - RQ2.6: What was the actual selection process used to select one of the OTS

components (we asked for information about the component, named Comp. 1, that provides the most functionalities)?

- RQ2.7: When was the Comp. 1 selected? - RQ2.8 What was the relationship between the selection phase and the project

context, such as the familiarities with the Comp. 1 and the importance of the Comp. 1?

As research questions RQ2.3 to RQ2.7 focus on state-of-the-practice, we just use

statistical graphs to show the current trends. Since RQ2.8 is to examine the correlations between variables, statistical methods, such as Spearman correlation and Mann-Whitney, are used. Detailed information of the design of the data analysis is shown in Section 4 of Paper P4 [Li06b] in Appendix A.

5.3.3.2. Part two: Investigate our findings from RQ3 (Which risk management activities have been performed and what are the results of performing them?). In this study, we have used the conceptual measurement framework proposed by Bush and Fenton [Bush90]. The study includes one entity (i.e., the project) and two attributes (i.e., possible risks of the project and possible risk mitigation activities to avoid these risks). There are also two relationships, i.e., risks are related to a project and risk mitigation activities that IMPACTS risks. Before the start of a project, project managers must upfront estimate how often the risks may happen, how much effort they need to put in to certain risk mitigation activities, and how well the risks may be managed. The relationships are shown in Figure 5-1.

Research methods

46

Figure 5-1 Risk management before a project starts

To help project managers predict the possibility of risks and to plan the risk management efficiently before a project starts, it is necessary to summarize practices and results of risk management from finished projects. The research framework of risk management in the main study, as shown in Figure 5-2, is designed to do posteriori measurements (after the project is finished) on the occurrence of project risks and the impacts of performed risk management activities on these risks.

Figure 5-2 Research framework for the risk management in the main study The research questions RQ3.2, as shown in Figure 5-2, is to investigate the actual

risk incidence, i.e., how often had problems occurred in practice? The research question

Research methods

47

RQ3.3, is to investigate the execution of the risk management activities in projects, i.e., how often had certain risk management activities been performed in practice? The research question RQ3.4 is to examine the effect of performed risk management activities, i.e., how well the performed risk mitigation activities had reduced corresponding risks? For a specific risk, we are interested in the most effective mitigation activities.

In our study, we selected several typical risks and possible risk management activities proposed in [Boehm03], [FAA03], [Li06a], [Rose03], [CBSEnet04], [Vitharana03], [Lawton02], [Fitzgerald04], [Torchiano04]. The selection criteria are:

- First, the risks and risk management activities should address the common issues

in the CBD, COTS-based development, and OSS-based development, as shown in Figure 4-2.

- OTS component-based development encompasses three primary types of stakeholders, namely component developers, application assemblers/integrators, and customer. Different stakeholders have different risks and challenges [Vitharana03]. We have selected risks and challenges that are relevant only to the application assemblers/integrators.

- The selected risk management activities in our study focus mainly on process and organizational issues, instead of technical methods, such as wrapping, composition prediction, and testing.

The selected typical risks in OTS components-based cover different phases of a

project, as shown in Table 5-4. The typical risk management activities are shown in Table 5-5. In addition, the results from process improvement studies show that one added role in OTS component-based development, i.e., that of a knowledge keeper (dedicated project personnel to keep track of OTS component knowledge/information), may help to mitigate the risks in OTS component-based development [Li06a], [Morisio00], [Rose03]. We therefore propose the existence of knowledge keeper as one possible risk management activity.

Concerning research questions RQ3.2 and RQ3.3, we compared the occurrences of typical risks and performed risk management activities. The results are visualized with the boxplot graph.

To analyse the results of RQ3.4, we decided to use Nonlinear Canonical Correlation Analysis (Nonlinear CCA) [Gifi90], [Michailidis98], because we face a nonlinear CCA problem with a set of explanatory variable vector defined by a nominal (knowledge keeper) variable and several ordinal (risk management activities performed) variables and a dependent variable matrix defined by a series of ordinal (risk occurrences) scales. The linear combination on the explanatory variable side is undefined, because we have no metric to quantify the categories of each variable. The linear combination of the variables on the dependent side is also undefined, because the categories of each variable can be re-scaled by any nonlinear function the preserves monotonicity. Thus, we need to optimally scale or quantify the variables while simultaneously solving the traditional linear CCA problem of finding weights for each explanatory variable. Nonlinear CCA corresponds to categorical canonical correlation analysis with optimal scaling. The detailed introduction of Nonlinear Canonical Correlation Analysis and the rationales for using it are reported Section 3.2 in Paper P5 [Li06c] in Appendix A.

Research methods

48

Table 5-4 Typical risks in OTS component-based projects Category ID Name Explanation

R1 R_SelEff Effort to select OTS components was not satisfactorily estimated [Boehm03].

Estimation risks

R2 R_IntEff Effort to integrate OTS components was not satisfactorily estimated [Rose03].

R3 R_ComRel OTS components negatively affected system reliability [Vitharana03], [Ruffin04].

R4 R_ComSec OTS components negatively affected system security [Lawton02], [Vitharana03], [ Ruffin04]

Quality risks

R5 R_ComPerf OTS components negatively affected system performance [Lawton02], [Vitharana03], [ Ruffin04]

R6 R_ComAda OTS components could not be sufficiently adapted to changing requirements [Boehm03].

Requirements risks

R7 R_ReqRen It is not possible to (re) negotiate requirements with the customer, if OTS components could not satisfy all requirements [Li06a].

R8 R_DefLoc It was difficult to identify whether defects were inside or outside the OTS components [Boehm03].

Debug and deployment risks

R9 R_ComComp OTS components were not satisfactorily compatible with the production environment when the system was deployed [Vitharana03]

R10 R_MaintPlan It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles [Rose03].

Maintenance risks

R11 R_ComUpd It was difficult to update the system with the last OTS component version [Rose03].

R12 R_ProvInf Information on the reputation and technical support ability of provider were inadequate [Rose2003], [Fitzgerald04].

Vendor support risks

R13 R_MissSupp Provider did not provide enough technical support/ training [Rose03], [Fitzgerald04].

Research methods

49

Table 5-5 Typical risks management activities in OTS-based projects Category ID Name Explanation

Rm1 Rm_CstMkBuy Customer had been actively involved in the “acquire” vs. “make” decision of OTS components [FAA03], [Li06a].

Rm2 Rm_CstSel Customer had been actively involved in OTS component selection [FAA03].

Rm3 Rm_SelArch OTS components were selected mainly based on architecture and standards compliance, instead of expected functionality [Torchiano04].

Rm4 Rm_SelQual OTS component qualities (reliability, security etc.) were seriously considered in the selection process [Boehm03], [Li06a].

Rm5 Rm_LearnEff Effort in learning the OTS component was seriously considered in effort estimation [Boehm03].

Rm6 Rm_BBEff Effort in black-box testing of OTS components was seriously considered in effort estimation [Boehm03], [Li06a].

Rm7 Rm_UnfFirst Unfamiliar OTS components were integrated first [Rose03].

Development

Rm8 Rm_IncTest Did integration testing incrementally (after each OTS component was integrated [Li06a].

Rm9 Rm_UpdFoll Local OTS-experts actively followed updates of OTS components and possible consequences [Li06a].

Rm10 Rm_MktWtch Maintained a continuous watch on the market and looked for possible substitute components [Li06a].

Rm11 Rm_ProvWtch Maintained a continuous watch on provider support ability and reputation [Rose03].

Maintenance

Rm_KK Knowledge Keeper

Have a knowledge keeper to own the knowledge of the OTS components [Li06a].

5.3.3.3. Part three: Verify the six newly published theses Based on results of the pre-study (see paper P2 [Li06a] in Appendix A), we clarified the research questions that can be used to verify the six newly published theses

Research methods

50

[Torchiano04], as shown in Table 5-6. How we clarified the research questions is reported in Section 3.2 of Paper P3 [Li05a] in Appendix A.

We used different analysis methods to analyse the research questions. - Thesis T1 is relevant to OSS components. To study research questions T1.R1

and T1.R2, we compared the usage of OSS components with COTS components.

- Because T2, T3, T4, and T6 are not relevant to OSS components, only information from COTS components was analysed.

- Thesis T5 should be tested using data from the whole project rather than a specific COTS component. The unit to be analysed for research question T5.RQ1 is therefore the whole project instead of a specific OTS component.

Table 5-6 Revised research questions to investigate the newly published theses

Newly published theses Revised research questions T1: Open source software is often used as closed source.

T1.RQ1: Had the source code of OSS components been read by integrators during software development? T1.RQ2: Had the source code of OSS components been changed by integrators during software development?

T2: Integration problems result from lack of compliance with standards. Architecture mismatches constitute a secondary issue.

T2.RQ1: Were the standard mismatches more frequent than architecture mismatches?

T3: Custom code mainly provides additional functionalities.

T3.RQ1: Were the occurrences of building glueware more frequent than the occurrences of building addware?

T4: Developers seldom use formal selection procedures. Familiarity with either the product or the generic architecture is the leading factor in selection.

T4.RQ1: How often had a formal process been used in practice? T4.RQ2: How often had the familiarity-based selection process been used in practice? T4.RQ3: How often had the hands-on experimentation based selection process been used in practice?

T5: Architecture is more important than requirements for product selection.

T5.RQ1: Had architectural compliance been considered more important than functionality completeness in the whole COTS-based project?

T6: Integrators tend to influence the vendor on product evolution whenever possible.

T6.RQ1: How often the required changes (corrections or new functionalities) from integrators had been satisfied by the vendors.

Research methods

51

5.3.3.4. Part four: Compare COTS with OSS-based projects to investigate RQ4 (How did project managers make their make vs. acquire decision in using COTS instead of OSS, or vice versa?).

The high level question will be: Why do I use COTS instead of OSS components, or vice versa? To investigate this question, we asked three sub-questions RQ4.1 to RQ4.3 as follows:

- RQ4.1: What are the commonalities and differences in profiles of projects using

COTS components vs. those using OSS components? - RQ4.2: What are the commonalities and differences in the expectation of

projects using COTS components vs. those using OSS components? - RQ4.3: What are the commonalities and differences in possible risks (problems)

of projects using COTS components vs. those using OSS components?

The background and rationale for RQ4.1 to RQ4.3 are reported in Section 3 of Paper P6 [Li06d] in Appendix A.

We used a questionnaire (see Appendix D) to collect data in the main study. The

questionnaire was organized into three main parts. - Questions to collect background information on the company, project, and

respondents. - Questions to get answers of the research questions in process improvement, risk

management, and decision making. - Questions to collect information about OTS components actually used in the

project. In the questionnaire, we asked the respondents to give brief information (name, COTS/OSS, version) of all OTS used and detailed information of one of the most important COTS/OSS component (the one provides most functionality compared to other OTS components).

The final questionnaire was first designed and pre-tested in English (internal and

external reviews). It was then translated into the native languages (Norwegian, German, and Italian) and published on the SESE web survey tool [Sese04] at Simula Research Lab in OSLO.

We defined the unit of this study as a completed software development project, and its OTS-relevant properties. The projects were selected based on two criteria:

- The project should use one or more OTS components. - The project should be a finished project, possibly with maintenance, and

possibly with several releases.

Projects were collected randomly from IT companies in Norway, Italy, and Germany. Slightly different sampling procedures were used in each country due to limited resources. Detailed discussions on sample selection in this study are reported in Paper P8 [Conradi05a] in Appendix A.

Research methods

52

- In Norway, we gathered a company list from the Norwegian Census Bureau (SSB) [SSB02]. We included mainly companies which were registered as IT companies. Based on the number of employees, we selected the 115 largest IT companies (100 IT companies and 15 IT departments in the largest 3 companies in 5 other sectors), 200 medium-sized software companies (20-99 employees), and 100 small companies (5-19 employees) as the original contact list.

- In Italy, we first got 43580 software companies from the “yellow pages”. We then randomly selected companies from them. For these randomly selected companies, we read their web-site to ensure they really are software companies. 196 companies were finally clarified as software companies, and were included in the original contact list.

- In Germany, we composed a company list from a company list from an organization similar to the Norwegian Census Bureau and in a similar manner as for Norway. We then used the existing Fraunhofer IESE customer database to get contact information.

To avoid bias in the data collection procedure, we made a guideline about how to

contact respondents before the survey started. All countries – Norway, Italy, and Germany – followed the process in the guideline. Possible respondents were contacted first by telephone. If the candidate respondents had suitable OTS-based projects and would like to join our study, a username and password of the survey tool and an electric version of the questionnaire were sent to them. The respondents could use either the SESE web tool or the electronic version to fill in the questionnaire. The respondents who did not want to answer the questionnaire were also registered. We logged the main reasons of non-response, such as “no software development”, “no OTS-based projects”, and “busy”. Most respondents used the Web tool to fill in the questionnaire. However, phone interviews were used in Germany because of the confidentiality concern in some companies [Conradi05a].

5.3.4 Research Design of the Follow-up Study As we promised in the main study, we will hold a seminar to report our results. The follow-up study (see Paper P7 [Li05b] in Appendix A) was performed together with this industrial seminar. The intentions of the seminar included two parts:

- First, we would like to report our results by giving guidelines on how to improve processes and manage risks in OTS-based development.

- As most data of this study show only state-of-the-practice, we do not know possible cause-effect relationships of our conclusions. We therefore had one hour discussion as the second part of the seminar to get feedback from the participants.

We sent invitations to 100 industrial respondents and 100 academic researchers.

Twenty-three industrial respondents participated in the seminar. In this study, we first showed our survey results in one hour presentation. During our talk, participants were able to interrupt and ask questions. In the second part, we hypothesized some questions based on the survey result. We also listed our assumed explanations on these questions. Industrial participants gave their opinions to support or reject our explanations. The

Research methods

53

whole discussion section has been tape recorded. The data analysis was performed by listening to the tape afterwards.

5.4 Validity Issues

A fundamental discussion concerning results of a study is how valid they are. Empirical research usually uses definitions of validity threats that originate from statistics and not all the threats are relevant for all types of studies. Wohlin et al. [Wohlin00] define four categories of validity threats:

- Conclusion validity (for statistical analysis) - “right analysis”: this validity is concerned with the relationship between the treatment (the independent variable in a study) and outcome (the dependent variable). We want to make sure that there is a statistical relationship of significance. Threats are related to choice of statistical tests, sample sizes, reliability of measures etc.

- Internal validity (for explanatory and causal studies, not for exploratory or descriptive studies) - “right data”: we must make sure that there is a causal relationship between treatment and outcome and that is not a result of factors that are not measured. Threats are related to history, maturation, selection of subjects, unpredicted events and interactions, ambiguity about the direction of causal influence etc. Yin [Yin03] adds (experimental) reliability to this: demonstrating that the operations in a study can be repeated with the same results such as data collecting.

- Construct validity- “right metrics”: we must ensure that the treatment reflects the cause and the outcome reflects the effect. Threats are mono-operation bias (a single case may not reflect the constructs), mono-method bias (a single type of measure may be misleading), hypothesis guessing etc.

- External validity- “right context”: this validity is concerned with generalization of results outside the scope of a study. Three types of interactions with the treatment may happen: people (the subjects are not representative for the population), place (the setting is not representative), and time (the experiment is conducted in a special time for example right after a big software crash). Yin [Yin03] terms this establishing the domain to which the findings of a study can be generalized.

To avoid the possible threats to validity in the studies in this thesis, we performed

different methods. For example, in the pre-study (see Paper P2 [Li06a] in Appendix A) and the main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c], and P6 [Li06d] in Appendix A), we did several rounds of pre-tests of the interview guide and the questionnaire. These pre-tests helped to decrease the possible construct validity threats. In the main study, we used random sample selection strategy to decrease the threats to the external validity. Detailed information on the validity issue of the studies reported in this thesis can be found in the “possible threats to validity” sections in Papers from P1 to P8 in Appendix A.

Research methods

54


In this chapter, we first gave a brief review of empirical methods in software engineering fields. The rationale and steps of using different research methods in our studies are given. Finally, we discuss how to manage the four types of validity issues in empirical software engineering in our studies.

Results

55

6 Results

This chapter summarizes and interprets the collected data for research questions RQ1 to RQ4. Our lessons learned from performing the main study, i.e., the international survey, are described at the end.

6.1 Collected Samples in Each Study

6.1.1 Collected Samples in the Preliminary Study The data was collected from three IT companies, Ericsson in Grimstad, EDB Business Consulting in Trondheim, and Mogul Technology in Trondheim. There were 200 developers at Ericsson in Grimstad, where we sent out 10 questionnaires to developers in one development team and got 9 filled-in questionnaires back. There were 20 developers in EDB Business Consulting in Trondheim, and we gathered 10 filled-in questionnaires back out of 10. We distributed 10 questionnaires to 22 developers at Mogul Technology in Trondheim and got 7 back. Those developers were selected because their work was related to component reuse, and they could assign effort to participate in the survey. This is non-probability sampling, which is based on convenience. Most participants in this survey have a solid IT background. Six of 26 respondents have MSc degrees in computer science and all others have a bachelor’s degree in computer science or telecommunications. More that 80% of them have more than 5 years of programming experience. The detailed information of each respondent is in the Table 3 of Paper P1 [Li04a] in Appendix A.

6.1.2 Collected Samples in the Pre-study We interviewed 16 projects in 13 companies. All the companies are Norwegian IT companies. Nine of these 13 are stand-alone companies, with staff size between 5 and 500. The other companies are subsidiaries, with local staff size ranging from 50 to 320. Six companies are IT consultancies, 5 are software vendors, and 2 are telecom companies. Five of these companies are publicly traded companies, the others are privately owned. Detailed background information about the selected companies is given in Table 1 of Paper P2 [Li06a] in Appendix A.

The company projects were selected based on two criteria:

Results

56

- The project should use one or more COTS components - The project should be a finished project and possibly with maintenance, and

possibly with several releases. Generally, we selected only one project from a company. However, we selected two

projects in three companies because these projects differed largely in project members, development process, and COTS components used. Detailed background information about the 16 projects is given in Table 2 of Paper P2 [Li06a] in Appendix A.

We had one respondent from each project. Three respondents are IT managers, 4 are project managers, 5 are software architects, 3 are software developers, and 1 is a software development researcher. Nine of them have more than 10 years of software development experience, and 12 of them have more than 4 years of working experience with COTS-based development. Eight of them have a master’s degree, and the rest have a bachelor’s degree. Twelve of them have a principal degree in informatics or computer science.

Some projects used one or two COTS components, others used more. Because of time limitations, we asked the respondents to select no more than three typical COTS components which were used in their projects. Totally, we gathered information about 30 different COTS components in 16 projects. Some typical COTS components are listed in Table 3 of Paper P2 [Li06a] in Appendix A.

6.1.3 Collected Samples in the Main Study We have gathered results from 133 projects (47 from Norway, 48 from Germany, and 38 from Italy) from 127 companies. In general, we selected one project from each company. However, we selected more than one project in three Norwegian IT companies because those companies have many OTS-based projects and would like to share more experience with this study.

The responding companies involved small (5-19 employees), medium (20-99 employees), and large (100 or more employees) ones as shown in Figure 6-1. The main business areas of the sampled companies also cover different types as shown in Figure 6-2.

Figure 6-1 Distribution of company size

Results

57

Figure 6-2 Distribution of companies main business areas

The final systems produced by the 133 projects cover different application domains as shown in Figure 6-3.

Figure 6-3 Distribution of application domain of the systems

In the selected 133 projects, 83 used only COTS components, 44 used only OSS

components, and six used both COTS and OSS components. For the 44 projects using only OSS components, each of them gave detailed information of an OSS component. For the 83 projects using only COTS components, each of them gave detailed information of a COTS component. For the six projects using both COTS and OSS components, the respondents selected to give detailed information of five COTS components and one OSS component.

Most respondents of the 133 projects have a solid IT background. More than 90% of them are IT managers, project managers, or software architects. Most of them have more than 2 years of experience with OTS-based development. All of them have at least a bachelor’s degree in informatics, computer science, or telecommunications.

6.1.4 Collected Samples in the Follow-up Study Twenty-three Norwegian respondents attended the seminar. More than 40% of them came from large-sized IT companies (more than 100 employees). 40% came from a

Results

58

medium-sized (with 20-99 employees) and 20% came from small-sized companies (less than 20 employees).

6.2 Answers to Research Questions

6.2.1 Reusing In-House Built Components – RQ1 The companies’ reuse levels of the three investigated companies are classified as large, medium, and small. Most developers in Ericsson think that the reuse level in their company is very high or high. Most developers in EDB regard the reuse level in their company is high or medium. Most developers in Mogul think that the reuse level in their company is medium or little. Detailed data are shown in Figure 1 and 2 in Paper P1 [Li04a] in Appendix A. Therefore, we define the reuse level in Ericsson as high, the reuse level at EDB as medium, and the reuse level at Mogul as low.

Answers to sub-questions are summarized as following. Detailed information is in Section 4 of Paper P1 [Li04a] in Appendix A.

- RQ1.1: Does requirements (re)negotiation for in-house components really work

as efficiently as people assume? No respondents believe that the requirements were never changed/

(re)negotiated. Only 8% of respondents think the requirements of their typical project are not flexible. However, only 48% of respondents think component-related requirements (re)negotiation works well.

- RQ1.2: Does the efficiency of component related requirements (re)negotiation increase when more in-house built components are available?

The result of correlation between the reuse level and the efficiency of requirements (re)negotiation, using one-tailed Spearman Rank Correlation Coefficient analysis, is .112, and the significance is .306. This shows that there is no significant relationship between the reuse level and the efficiency of component-related requirements (re)negotiation.

- RQ1.3: Does the value of component repository increase with more reusable

components available? Seventy-one percent of the respondents in Mogul and EDB regard

constructing a component repository as worthwhile, against 57% in Ericsson. The result of correlation between the reuse level and the value of the repository, using one-tailed Spearman Rank Correlation Coefficient analysis, is -.124, and significance is .297. It shows that there is no obvious relationship between them.

- RQ1.4: How can a component user acquire sufficient information about relevant

components? Sixty-seven percent of the respondents think the component structure is well

understood, 61% say that the component interfaces are understood, and 63% regard the design rules of components are also well understood. However, no one considers that the design/code of components is well documented. Eighty-

Results

59

six percent believe that insufficient component documentation is a problem, e.g., documentation is not complete, not updated, and difficult to understand. The preferable way of documentation is web pages. Some of the developers’ knowledge about how to use components comes from informal communication sources, for example, previous experience and suggestions from local experts.

- RQ1.5: Does the difficulty of component documentation and component

knowledge management increase with increasing reuse level? The result of correlation between the reuse level and the satisfaction of

documentation, using one-tailed Spearman Rank Correlation Coefficient analysis, is -.469, and significance is .014, which shows a weak negative relationship between them. It means that the higher the companies’ reuse level, the less satisfied a developer is with the component documentation.

- RQ1.6: Do developers trust the quality specification of their in-house built

components? If the answer is no, how can they solve this problem? Seventy percent of the participants regard the design criteria for quality

requirements are not well defined, and 87% will test the quality attributes of components after component modification, before integrating them into the system.

6.2.2 Software Process Improvement – RQ2 The research question RQ2 is examined by sub-questions RQ2.1 to RQ2.8. RQ2.1 and RQ2.2 are investigated by the pre-study and detailed answers are shown in Section 5 of Paper P2 [Li06a] in Appendix A. Questions RQ2.3 to RQ2.8 are studied by the main study and detailed answers are shown in Section 4 of Paper P4 [Li06b] in Appendix A. Answers to these sub-questions are as follows:

- RQ2.1: What was the actual process used in the projects using COTS components?

Results show that the main development processes of the studied projects can be grouped into three categories: pure waterfall, waterfall mixed with prototyping, and incremental mixed with prototyping. Five projects used waterfall. One used waterfall mixed with a little prototyping (a small prototype was used for discussing requirements of the Graphical User Interface, i.e., GUI, part of the system). Ten projects used incremental mixed with some prototyping. None of the respondents claimed that they selected the main development process based on whether or not they intended to use COTS components. That is, they decided the main process before they started to think about using COTS components or not. They just added or changed some activities and roles in their traditional process to reflect the use of COTS components.

Therefore, the answer to research question RQ2.1 is: The so-called COTS-based development process is the customization of the traditional development process due to the use of COTS components.

- RQ2.2: What are the commonalities and possible variations in COTS-based

development processes?

Results

60

The commonalities are that new activities were added and possible one new role was added. The added new activities include make vs. acquire decision, COTS component selection, learn and understand the COTS components, and building glueware and addware. The possible added new role is: a COTS component knowledge keeper. One or more project members had some previous experience with selection and integration of the actual COTS components in ten of our studied projects. They provided suggestions on COTS component selection and integration. Although some of them were not dedicated to work as a COTS component knowledge keeper, their previous experience improved the speed and quality of COTS component selection and integration.

One variation in the customization of main development processes deals with the phase to make the make vs. acquire decision and COTS component selection. Another variation is the COTS component selection and evaluation process. The selection process in our studied projects can be summarized into two categories, i.e., familiarity-based selection process and Internet search, hands-on trial-based selection process.

Therefore, the answer to research question RQ2.2 is: There are some common new activities and one new role is added in COTS-based development processes. The possible variations are when and how to perform these new activities.

- RQ2.3: What were the actual development processes in OTS-based projects?

The results are summarized in Figure 6-4 and show that the waterfall, incremental, XP, German V-model, and prototyping model are those mainly used in practice. Between these, the incremental with prototyping model and XP was the most popular. The German V-model was used widely in Germany as the OTS-based development process.

Figure 6-4 The actual development process in the OTS-based project

Results

61

- RQ2.4: Was the actual development process decided before the make vs. acquire decision or after the make vs. acquire decision.

The results show that 75% projects decided their main development processes before they started to think about using OTS-components

- RQ2.5: Who decided the actual development process?

The answers reveal that in 29% of all projects the development process is predefined by the global company or the department rules. In addition, concerning 14% of the remaining projects the decision was at least affected by company/department rules. This trend is especially obvious in Germany as the company rules dominated the development processes in 65% of our studied projects.

- RQ2.6: What was the actual selection process used to select one of the OTS

components (we asked for information about the component, named Comp. 1, that provides the most functionalities)?

We listed six possible selection activities as named a) to f) in the questionnaire:

a) Searched Internet for possible OTS component candidates. b) Got recommendation of possible OTS component candidates from the customer. c) Got recommendation of possible candidates from a local colleague/OTS-expert. d) Used a formal decision-making method to compare possible OTS component candidates, e.g., with weighted evaluation criteria. e) Limited possible candidates into 1-3 components, by reading literature/documentation. f) Did “hands-on” try-out of 1-3 components, e.g., on a downloaded demo version. The results are summarized in Figure 6-5 and show that activities a, c, e, and

f were those mainly used. Thus, the familiarity-based selection process (activity c) and the Internet search, trial-based selection process (activity a, e and f) from the pre-study (see paper P2 [Li06a] in Appendix A) are proved to be more popular than a formal processes (activity d) in general. However, the further analysis with Chi-square shows that there are significant differences of the activities b and c between countries. In Norway and Italy, very few projects used the activity b. However, activity b was popular in Germany. In addition, almost all German projects performed activity c, while it was used only in half of the projects in Norway and Italy.

Results

62

Figure 6-5 What selection and evaluation actions were performed?

- RQ2.7: When was the Comp. 1 selected?

Figure 6-6 shows that OTS components were selected in the early stages, such as pre-study, requirement, and overall design, in most projects, especially in Germany.

Figure 6-6 When was the OTS component selected?

- RQ2.8: What was the relationship between the selection phase and the project

context, such as the familiarities with the Comp. 1 and the importance of the Comp. 1?

The results reveal that the selection phase was weakly (with Spearman correlation coefficient value -.356) correlated with the project members’ familiarity with it. It means that the project members preferred to decide the familiar OTS components in the early stage of the project. However, there is no

Results

63

significant connection between the importance of an OTS component and the phase it was selected.

In addition to answer the sub-questions RQ2.1 to RQ2.8, the pre-study (see paper P2

[Li06a] in Appendix A) and the main study (see paper P3 [Li05a] in Appendix A) verified six newly published theses [Torchiano04] in OTS component-based development. The results are summarized as following. Detailed information can be found in Section 5 of Paper P3 [Li05a] in Appendix A.

- Thesis T1: Open source software is often used as closed source.

We support T1. We have asked the respondents to answer whether they have read or changed the source code of the component. We used a five-point Likert scale to measure the answer. Respondents were asked to answer “very little”, “little”, “some”, “much”, “very much”, or “don’t know”. The results show that the median value of reading OSS code is 3 (meaning “some”). However, the source code has seldom been changed as the median value of changing the OSS code is 2 (meaning “little”). Further analysis discovered that 46% of the respondents claimed that they modified “very little” parts of the source code in OSS components. In addition, 29 (out of 88) selected COTS components provided source code to their users.

- Thesis T2: Integration problems result from lack of compliance with standards.

Architecture mismatches constitute a secondary issue.

We are against T2. For the selected 88 COTS components, only 4.5% of the COTS components had standard mismatches, compared with 28.4% COTS components having architecture mismatches.

- Thesis T3: Custom code mainly provides additional functionalities.

We support T3. Glueware was needed for 32.9% of the selected COTS

components and addware was needed for 65.9% of the selected COTS components.

- Thesis T4: Developers seldom use formal selection procedures. Familiarity with

either the product or the generic architecture is the leading factor in selection.

We support T4. Formal procedures were only used in evaluating 19% of the selected COTS components.

- Thesis T5: Architecture is more important than requirements for product

selection.

We are against T5. We asked the respondents whether COTS components were selected mainly based on architectural and standards compliance, or expected functionality. The answer is measured by “don’t agree at all”, “hardly agree”, “agree somewhat”, “agree mostly”, “strongly agree”, or “don’t know”.

Results

64

We assigned an ordinal number from 1 to 5 to the above alternatives (5 meaning strongly agree). Results are in Figure 6-7 and show that the median value of this question is 2 (meaning “hardly agree”). It indicates that most respondents hardly agree that COTS components were selected mainly based on architectural and standards compliance.

82N =

ARCH. VS. FUNC.

6

5

4

3

2

1

0

Figure 6-7 Result of thesis T5

- Thesis T6: Integrators tend to influence the vendor on product evolution

whenever possible.

We support T6. Sixty percent of the selected COTS components have been changed by vendors according to integrators’ requirements.

In the follow-up study (see paper P5 [Li05d] in Appendix A), we investigated the cause-effect reason of the phenomena discovered from the previous studies. The results are summarized in Table 6-1.

Table 6-1 Barriers to disseminating SPI theories to the IT industry

Observations from the main study

Industrial explanations

Our insights Barriers

Few projects used the proposed OTS-based development processes.

Few empirical studies have investigated the cost and benefit of the proposed COTS-based development processes.

The industrial practitioners will need evidence that the formal process is cost-effective. It is also necessary to know how to downsize a formal process according to different project contexts.

Without sufficient empirical evidence on OTS based development processes, industrial project managers would not like to change their de-facto development process.

Results

65

Formal OTS selection processes were seldom used in practice.

It is not necessary because OTS components are not the key parts of the system in some cases. Few studies showing that a formal process actually gave better results than informal processes.

Most formal selection processes were proposed without evaluations on their pre-conditions and their cost-benefit.

Little advice on how to customize a formal process according to project context and OTS itself prevents the use of a formal selection process.

6.2.3 Risk Management – RQ3 The research question RQ3 is investigated by sub-questions RQ3.1 to RQ3.4. RQ3.1 was examined by the pre-study and detailed answers to this question are in Section 5.3 of Paper P2 [Li06a] in Appendix A. Research questions RQ3.2 to RQ3.4 are examined in the main-study, and detailed answers to these questions are given in Section 5 of Paper P5 [Li06c] in Appendix A. The summaries of answers to these questions are as follows:

- RQ3.1: What are process scenarios (variations) of the projects using COTS components successfully and not successfully? What are possible problems and good practices in different process scenarios?

The results of the risk management in the pre-study are the 11 problems

encountered (PEs) and the 12 good practices (GPs). One example of the problems encountered is that the integration effort was under-estimated. An example of the good practice is that the integration testing was done incrementally. Details of these PEs and GPs are shown in Section 5.3 of Paper P2 [Li06a] in Appendix A.

- RQ3.2: Occurrences of Risks in OTS Component-Based Projects

We compared typical risks in Table 5-4 (see Section 5.3.3). Results are in Figure 6-8 and show that the most frequent risks are R2 (R_IntEff) and R8 (R_DefLoc). It means that the integration effort was usually wrongly estimated. In addition, it was difficult to figure out whether the defects are inside or outside the OTS components. The least frequent risks are R3 (R_ComRel), R4 (R_ComSec) and R5 (R_ComPerf), i.e., the problems relevant to the negative effect of the OTS components on the quality of the whole system.

Results

66

R_MissSupp

R_ProvInf

R_ComUpd

R_MaintPlan

R_ComComp

R_DefLoc

R_ReqRen

R_ComAda

R_ComPerf

R_ComSec

R_ComRel

R_IntEff

R_SelEff

4

2

Figure 6-8 Occurrences of typical risks

- RQ3.3: Which risk management actions have been performed in practice?

We compared typical management actions in Table 5-5 (see Section 5.3.3). The results for this question are shown in Figure 6-9. We can see that the activities Rm4 (Rm_SelQual), Rm5 (Rm_LearnEff), Rm7 (Rm_UnfFirst), Rm8 (Rm_IncTest) and Rm9 (Rm_UpdFoll) were used more frequently than others. It means that developers have seriously evaluated the quality of the OTS components in the selection phase. They have considered the possible learning effort in the effort estimation. They have done integration testing as early as possible and they usually have internal expert to follow the update of the OTS components. However, the activities relevant to the users, such as Rm1 (Rm_CstMkBuy) and Rm2 (Rm_CstSel), were least frequently used.

Results

67

Rm_ProvWtch

Rm_MktW

tch

Rm_UpdFoll

Rm_IncTest

Rm_UnfFirst

Rm_BBEff

Rm_LearnEff

Rm_SelQual

Rm_SelArch

Rm_CstSel

Rm_CstMkBuy

4

2

Figure 6-9 Risk management activities performed

- RQ3.4: Can performed risk mitigation actions help to mitigate the

corresponding risks? To investigate research question RQ3.4, we used the non-linear canonical

correlation analysis (the OVERALS in SPSS [SPSS01]) to find the relationships between the occurred problems (see Table 5-4 in Section 5.3.3) and the risk management activities performed (see Table 5-5 in Section 5.3.3). Since we put the data from three countries together, we also analysed whether the differences between the three nations will affect our conclusions. In addition, we analysed whether the differences between COTS and OSS components have effect on our results. The results are summarized in Table 6-2. Details are shown in Section 5.3 of Paper P5 [Li06c] in Appendix A.

Results

68

Table 6-2 Effective risk management activities

In the follow-up study (see paper P7 [Li05b] in Appendix A), we investigated two

risk management issues, i.e., the customer involvement and knowledge management. The details are shown in Table 6-3.

Table 6-3 Barriers to disseminating RM theories to the IT industry Observations from the main study

Industrial answers Our insight Barriers

Customers were seldom involved in OTS component selection and decision making.

Their customers either had no technical background to make a decision or cared only for the final product without considering the technical details.

As it is difficult to discover all limitations of OTS components at the very beginning, it is necessary for integrators to be able to (re)negotiate requirements with the customer.

Few case studies have observed successful requirements (re)negotiation in OTS-based development. There are therefore few guidelines about when and how to involve customers in requirements (re)negotiation because of OTS limitations.

Results

69

Proper knowledge management and sharing helped to avoid the risks.

They were interested in how much effort the knowledge keeper needs to spend on OTS component-based development and what kind of knowledge needs to be kept.

What knowledge should be documented and shared in OTS-based development has seldom been investigated. It is therefore difficult for industrial practitioners to collect and share relevant OTS-based development knowledge.

Few studies on knowledge accumulation and transfer delayed the dissemination of the OTS-based development experience.

6.2.4 Decision Making – RQ4 Research question RQ4 is investigated by sub-questions RQ4.1 to RQ4.3. These sub-questions were mainly examined in the main study, and detailed results are in Section 4 of the paper P6 [Li06d] in Appendix A and are summarized as follows:

- RQ4.1: What are the commonalities and differences in profiles of projects using COTS components vs. those using OSS components?

The results in Figure 6-10 and 6-11 illustrate that there are no significant

differences of the project profiles between COTS and OSS-based projects. The most emphasized requirements of the final systems were time-to-market, reliability, and performance. Security was surprisingly listed as the least emphasized requirement of the system.

Figure 6-10 Project profiles of the COTS component-based system

Results

70

Figure 6-11 Project profiles of the OSS component-based system

- RQ4.2: What are the commonalities and differences in the expectation of

projects using COTS components vs. those using OSS components? Results in Figure 6-12 and Figure 6-13 show that there are no significant

differences between the general expectation of using either COTS or OSS components. The main expectation of using OTS components is to shorten the time-to-market and to save development cost.

Figure 6-12 General motivations of using COTS components

Results

71

Figure 6-13 General motivations of using OSS components

For the specific expectations, results in Figure 6-14 show that COTS users believe that paid software will give good reliability and will follow the market trend. They also believe that the COTS vendor will provide good technical support. On the other hand, results in Figure 6-15 show that OSS components pay more attention to the free and available source code.

Figure 6-14 Specific motivations of using COTS components

Results

72

Figure 6-15 Specific motivations of using OSS components

- RQ4.3: What are the commonalities and differences in possible risks (problems)

of projects using COTS components vs. those using OSS components?

We compared the occurrences of typical risks in Table 5-4 (see Section 5.3.3) in COTS vs. OSS projects. Results are shown in Figures 6-16 and 6-17, and illustrate that COTS users had significantly more difficulties in R1 (R_SelEff), i.e., estimating the component selection effort and R6 (R_ComAda), i.e., following requirement changes.

Figure 6-16 Occurrences of risks in COTS component-based projects

Results

73

Figure 6-17 Occurrences of risks in OSS component-based projects

6.3 Our Lessons Learned on Performing the Studies

In addition to the results of the research question RQ1 to RQ4, we got several insights on methodology issues of implementing an international industrial survey. The details are reported in Paper P8 [Conradi05a] in Appendix A.

The insights on methodology issues focus on three categories. The first is how to sample the population. As our study object is to investigate the OTS component-based development projects in ICT (Information and Communication Technology) sector, there are two issues that need to be investigated. The first is how to define the ICT sector, and the second is how to define our study objects. To get the information about the ICT companies, we have used a complete set of registered ICT companies defined by a public source from three countries. However, our real problem is that the actual study object (experimental unit) – a special kind of project – is not identical to the primary object (accessible sampling unit), being a company or a respondent. Due to resource constraints and logistical problems, it seems totally unrealistic to perform an explorative pre-study to gather the information of all projects from each ICT company and randomly select samples from these projects. At best we can achieve a stratified-random sample of ICT companies, followed by a convenient sample of relevant projects. Other reflections on sampling and response rates are:

- For all countries, there were problems in classifying large and perhaps

international ICT corporations, with many subsidiary companies and divisions.

Results

74

- For a variety of reasons it seems impossible to perform the same random sampling and data collection in different countries.

- In Norway, 50 of 380 (13%) of the companies were no longer “contactable”, as the company lists were two years old in 2004. It represents a very dynamic marketplace.

- In Norway, we found that 70 of 285 (24%) software companies (NACE code 72.xx) claimed to have no software development at all.

- The response rate in Norway increases with decreasing company size. The response rate was especially low in large companies, cf. “gatekeeper [Corra02]” problem.

The second reflection from the main study is about the questionnaire design. A pre-

study often clarifies both research questions and concrete questions in a later questionnaire.

The third reflection is about contacting the companies. Through a list of potential companies, we had to find a contact person in each company. For convenience samples, there are few problems in getting hold of a potential respondent, as we often have personal and professional ties from earlier, joint activities. But for a large corporation, from a census or member list and with thousands of employees, it can be very frustrating to try to locate a quality manager, development/project manager, or just an “IT responsible”. The final struggle is to convince a possible respondent over the phone to use his/her time and effort to fill in a questionnaire, sent by email. The results show that random sampling and the ensuing contact process appears to be much more expensive (5 times?) than convenience sampling. In our case, this is because we got stuck with no previous contact person, and scant pre-knowledge and “goodwill” from such companies, especially the large ones.


In this chapter, we have illustrated the collected data of each study and the results to the research questions. Discussion of these results, the possible validity issues, and limitation of each study will be given in the next chapter.

Evaluation and discussion

75

7 Evaluation and Discussion

This chapter discusses our contributions to research questions RQ1 to RQ4. We evaluate our contributions against the state-of-the-art. The lessons learned from performing the main study are also discussed. In addition, we explain the possible threats to validity of each study to help the reader interpret our conclusions.

7.1 RQ1: How to Improve the Reuse of In-house Built Components?

For research question RQ1, we investigated the state-of-the-practice of reusing in-house built components and proposed how to prepare for more reuse through the preliminary study (see paper P1 [Li04a] in Appendix A). The study focused on four issues as following, and contributed to C1, i.e., increased understanding on the risks of reusing in-house built components.

7.1.1 Component-Related Requirements (Re) negotiation A lot of researchers focus on how to improve the efficiency of component-related requirements (re)negotiation in COTS-based development [Morisio00], [Sai03], [Tran99]. The main reason is that people think the challenges in requirements (re)negotiation are due to the lack of access to source code, to timely vendor supports, or to the lack of engineering expertise to modify the integrated components [Tran99]. The above constraints on COTS components are not considered as challenges with built in-house components. We found that although 92% of our investigated developers think that requirements of their typical projects are flexible, less than half think that the component-related requirements (re)negotiation in their typical projects works well. Since components are intended to be used ‘as-is’, it is possible that an in-house reusable component meeting all the requirements will not be found. So, even though the components are built in-house, requirements (re)negotiation is necessary. We do not claim that the requirements (re)negotiation based on in-house components is more difficult than COTS-based components. We just emphasize that requirements (re)negotiation based on in-house components is also important but not efficient. Moreover, we found that when IT companies move from a low reuse level to a higher reuse level, they probably cannot expect that component-related requirements (re)negotiation becomes easier and more efficient.


76

7.1.2 Component Repository Some researchers have claimed that a repository is important, but not sufficient for successful reuse [Rothenberger03], [Morisio02b]. Our data confirm that developers are positive, but not strongly positive to the value of a component repository. So, this result gives further support to their conclusion. In addition, we concluded that companies are not expected to invest in a repository to increase reuse.

7.1.3 Component Understanding Transferring component knowledge from the component developer to the component user is critical for successful component reuse. We found that most developers got the knowledge about components from informal channels, such as previous experience and local experts. The most important feature of a component is the separation of its interface from its implementation. Moreover, current component documentation technologies cannot describe all the information developers require, such as performance, reliability, and security. Therefore, informal knowledge transfer should be considered to supplement the insufficiency of formal component documentation and specification. One possible solution is to have special interest groups or mailing lists for a component (or a group of similar components) so that component users can share knowledge and experience of component usage. We also found that the higher a companies’ reuse level, the less satisfied a developer is with the component documentation. Rothenberger et al. [Rothenberger03] concluded that combining reuse education and training provided for staff with other reuse activity can lead to success in reuse. Our conclusion implies that when a company moves from a low reuse level to high level, more effort should be spent on the component documentation and component knowledge management.

7.1.4 Quality Attributes of Components Component-based development relies on the availability of high quality components to fill roles in a new intended system. When components are created or changed, we must ensure that they not only fulfil the functional requirements, but also quality requirements. We found that most developers are not satisfied with the specification of the quality attributes of components and therefore cannot use this information. Therefore, how can we model quality properties of both components and systems, and reason about them, particularly in the early stage of system development is still a key challenge in component-based development.

7.2 RQ2: How to Do SPI in OTS Component-Based Development?

For the research question RQ2, we compared theories from the academic field and industrial practice. We contributed to C2, i.e., summarizing what is the current state-of-the-practice of SPI and OTS selection in OTS component-based development (see papers P2 [Li06a], P4 [Li06b], and P7 [Li05b] in Appendix A). In addition, we contributed to C5, i.e., validated newly published theses T4 and T5 in [Torchiano04] that are relevant to the SPI in OTS component-based development (see paper P3 [Li05a]


77

in Appendix A). Our contributions can be used to help project managers to decide how to improve the whole life cycle of the OTS-based projects and how to select the OTS components based on the project contexts.

7.2.1 How to Improve the Whole Life Cycle Concerning the conclusions drawn by [Boehm99], we agree that using OTS components in system development involves potential danger, especially when using unfamiliar OTS components. Although the risk driven processes proposed in [Boehm99] may give benefits, results of RQ2.3 (see Section 6.2.2) show, however, that most projects integrate OTS components successfully with traditional processes, such as waterfall, incremental with prototyping, XP, or V-model. Results of RQ2.3 (see Section 6.2.2) show that most of the actual development processes were decided before the decision of using OTS components was made. In addition, results of RQ2.5 (see Section 6.2.2) show that company rules pre-defined the main development processes in many OTS-based projects. According to the results of RQ2.3 to RQ2.5 (see Section 6.2.2), we summarize the state-of-the-practice of OTS-based development processes into seven different scenarios. The classification of the scenarios is based on the phase of the make vs. acquire decision and the developers’ familiarity with the OTS candidates.

Scenarios 1 to 3: Scenarios of planned OTS-based software development In scenarios 1 to 3 (see Figure 7-1), project members prefer to evaluate the possibility of using OTS components right from the start. Due to time-to-market, cost, or capacity issues, project managers realize that it is difficult, impossible, or not cost-effective to make everything from scratch. In these scenarios, the first step is to do a make vs. acquire decision (see step A in Figure 7-1). In case project members decide not to use OTS components, the development process will be the same as the non-COTS based development. If project members decide to use OTS components, they then need to choose the main development process (see step B in Figure 7-1). To decide the main development process, main issues are the traditional risks in a project, such as the stability of the requirements. One of the other issues is their familiarity with possible OTS candidates. According to the project members’ familiarity with the OTS candidates, we define three scenarios as scenarios 1 to 3:

- Scenario 1 – planned OTS-based project with unfamiliar OTS candidate: In this scenario, the project members prefer to use OTS components to provide certain functionalities of the system. However, they are not familiar with the OTS candidate at all. To decide the main development process, the OTS relevant risks, such as the quality of OTS components and the vendor’s support, should be seriously considered. In addition, project members should pay attention to other risks, which are not relevant to an OTS component but the overall project (e.g., which process is more suitable for projects with unclear or unstable requirements - waterfall or an incremental with prototyping model?).

- Scenario 2 – planned OTS-based project with some familiar OTS candidates: In this scenario, the project members have certain experience with some of the possible OTS candidates. However, there are still certain candidates that they


78

have never used before. In order to decide the main development process, the issues are similar with those of scenario 1.

- Scenario 3 – planned OTS-based project with familiar OTS candidates: In this scenario, the project members are familiar with all possible candidate components. When deciding the main development process the OTS relevant risks are not as critical as in scenarios 1 or 2. Other non-OTS relevant factors, such as company rules or requirement flexibility may be used to decide the main development process. In this scenario, the main development process can be any, such as waterfall, incremental, or XP. It may not need to be changed because the project members are very familiar with the OTS component candidates.

Figure 7-1 Scenarios 1 to 3 The COTS-based development model in [Morisio00] illustrates the differences

between COTS-based development and traditional software development, such as new, reduced, or modified activities. The underlying assumption of this model is that the rate or degree of using OTS components is fixed in the very early phases of a project and that the main development process has to be modified accordingly. The model can therefore be used as a reference in scenarios 1 to 3. Scenarios 4 to 6: Scenarios of unplanned OTS-based software development In scenarios 4 to 6 (see Figure 7-2), there is no clear intention of using OTS components in the early phases of such a project. Project members decide on the main development process according to some non-OTS relevant factors, such as company rules and requirement stability. Due to the time-to-market pressure or internal capability, project members start to evaluate the possibility of using OTS components at certain stages of the project. At our follow-up study (see Paper P7 [Li05b] in Appendix A), several


79

industrial participants mentioned such conditions. According to the project members’ familiarity with the OTS candidates, we classify three scenarios from scenarios 4 to 6.

- Scenario 4 – Unplanned OTS-based project with unfamiliar OTS candidates: In this scenario, the project members decided to use OTS components in a late phase of the project, such as detailed design or coding. However, they do not have any previous experience with the possible OTS candidates. Therefore, the use of OTS components may bring several problems into the whole project. The project members need a second risk-evaluation to investigate whether the development process should be modified concerning the use of OTS components. For example, in projects with pure waterfall processes, the project members may need to add a short prototyping step to evaluate the OTS component and to negotiate the use of OTS components with their customer.

- Scenario 5 – Unplanned OTS-based project with some familiar OTS candidates: In this scenario, the project members decide to use OTS components in a late phase. However, they do not have previous experience with all OTS component candidates. In order to decide the main development process, the issues are similar to those of scenario 4.

- Scenario 6 – Unplanned OTS-based project with familiar OTS candidates: In this scenario, although the project members decide to use OTS components in a late development phase, they have enough experience with the OTS component candidates or are able to hire a person with such experience. In case the project members have used the OTS component many times before, the OTS component can be regarded as comparable to in-house built components. Although the use of OTS component is a little risky due to the same reasons as described in the scenarios 4 and 5, this scenario is less risky because of using familiar OTS components. It may not be necessary to do major changes to the current development process.

Figure 7-2 Scenarios 4 to 6


80

Scenario 7: The scenario of semi-planned OTS-based software development Scenario 7 is a mix of scenarios 1, 2, or 3 with scenarios 4, 5 or 6, whereby the use of OTS components is planned for some parts of the system. The main development process is decided according to the use of OTS components. However, in a late phase of the project, project members may need to add some unplanned OTS component due to the same reasons as for scenarios 4, 5, or 6.

The EPIC approach [Albert02], like RUP, consists of four phases and each phase consists of one or more EPIC iterations. The information of four spheres of influences, such as stakeholders’ needs, architecture design, risk, and marketplace, are gathered and evaluated in each iteration. The project management, customer requirements, and system architecture will be adjusted according to the trade-offs of the four spheres of influences. Since every iteration includes a complete sub-process to make the trade-off decision and select the OTS component, it is more flexible and suitable to be used as a reference in scenarios 4 to 7.

7.2.2 How to Select the OTS Components In these different scenarios above, various OTS component selection strategies and processes can be used. However, our results support the thesis T4 (see Section 6.2.2) proposed in [Torchiano04] and show that formal selection processes were rarely used. In addition, our results contradict T5 (see Section 6.2.2) in [Torchiano04] and conclude that functionality completeness was regarded more important than architectural compliance in COTS component selection.

In scenarios 1, 2, 4, 5, and 7, some unfamiliar OTS component candidates are going to be used. The selection process for unfamiliar OTS components can either be an Internet search with hands-on trial-based, or a formal selection process, such as processes shown in [Kontio96], [Tran97b], [Morisio97], [Ochs01]. In scenarios 3 and 6, all possible OTS component candidates have been used by the project members, the selection process could mainly be familiarity-based.

In scenarios 1 to 3, the use of OTS component is well-planned. In general, the OTS component will be selected in the early phase of the project in these scenarios. Selecting OTS components in the early phase can help to design the software architecture with consideration on constraints of the OTS components. Therefore, it is easier to avoid the mismatch between the OTS components and other components in the system.

In scenarios 4 to 7, the OTS components are usually evaluated at a phase in which most of the other components are already integrated in the system. Since the goal is to integrate the OTS component into the current system and to check its integration thoroughly, integrating the demo version of the OTS component and performing a hands-on trial is necessary. In case the OTS component is selected in the very late stage of the project, it is better to select a component that has a loose coupling with other components. OTS components tightly coupled with existing components may impose a large number of changes into the system.


81

7.3 RQ3: How to Mitigate Risks in OTS Component-Based Development?

For research question RQ3, we contributed to C3, i.e., empirical validation of risk management proposals in OTS component-based development (see Papers P2 [Li06a], P5 [Li06c], and P7 [Li05b] in Appendix A). In addition, we contributed to C5, i.e., empirically validated three newly published theses (see Paper P3 [Li05a]in Appendix A), i.e., T2, T3, and T6 in [Torchiano04], which are related to risk management in OTS component-based development.

Concerning the theses related to the risk management in OTS-based development, results of thesis T2 (see Section 6.2.2) show that architecture mismatches were more frequent than standard mismatches. With the maturity of COTS component standards, we believe that less standard mismatches will occur in the future. Results of thesis T3 (see Section 6.2.2) show that missing or incomplete features in COTS products raise significant problems. It is therefore important to develop effective techniques that allow seamless extension of existing products. Results of thesis T6 (see Section 6.2.2) show that COTS component users actually pushed vendors to change the COTS components either because of bug fixes or new features. However, this result does not imply that most COTS component users actually influenced COTS component evolution. In general, we believe that the COTS component evolution is decided by the vendors’ marketing strategy rather than the requirements of a specific user, unless this user is very powerful.

Concerning the validation on the risk management proposals, we first compared the frequency of some typical risks. The results can be used to help project managers to identify the most possible risks. We then investigated the correlations between the problems occurred and the risk management activities performed. The results are summarized in the following Section 7.3.1 to 7.3.6, and can be used to help project managers to do a proper risk management plan based on the risks identified.

7.3.1 How to Mitigate the Cost Estimation Risks Both the effort in selecting OTS components and the effort to integrate them sufficiently are difficult to estimate [Boehm03], [Mitre05]. Basili and Boehm [Basili01] hypothesized that COTS-based development is currently a high-risk activity, with effort and schedule overruns exceeding non-CBS software overruns. The results shown in Figure 6-8 illustrate that the wrong estimation of the integration effort happened more frequently than other problems.

To estimate the selection effort correctly, Boehm et al. [Boehm03] propose to identify the core “showstopper” requirements and filter all the OTS candidates that do not meet these during the initial assessment and then proceed for a more detailed assessment with the remaining OTS candidates. Although this method may be applicable to limit the number of OTS components that need to be evaluated, it is still difficult to estimate the possible effort to compare the limited OTS candidates, because both technical and non-technical issues of the OTS components need to be evaluated and compared [Rose03]. Our findings of RQ3.4 (see Section 6.2.3) show that in case the effort on learning in the OTS components has been seriously considered, it will be easier to estimate the selection effort correctly. Boehm et al. [Boehm03] found that one


82

common error in OTS-based development is the overly optimistic OTS learning curve and propose that a most likely OTS learning curve must be accounted for during planning and scheduling. Initially, the OTS component’s functional capability must be well understood, which can be a daunting task because there may be a significant amount of functionality to learn and that functionality might involve new technologies to the team. Also, there is more to learn than just the functionality. The methods of integration that the OTS component employs must be understood sufficiently to ensure that the integration can be succeeded into the current architecture. Since the intention of using OTS components is to shorten time-to-market and to save development cost, OTS component users are eager to acquire the components and to integrate them as soon as possible without understanding them thoroughly. Rose [Rose03] pointed out that developers are usually working with one or more OTS components for which they most likely have only a partial understanding. To improve the estimation on the OTS integration effort, the proposed activity is to use OTS-sensitive analytical and budget processes [Rose03], [FAA03], such as COCOTS [Abts00]. In our study, we did not investigate the effect of using COCOTS because our pre-study (see Paper P2 [Li06a] in Appendix A) did not find any companies using it. Therefore, we did not investigate this activity because we wondered whether this activity has been popularly used in industry. Although the results shown in Figure 6-9 reveal that the risk management activity Rm5 (i.e., effort in learning OTS component was seriously considered in effort estimation) has been widely used, we still wonder whether there are good tools to help the project managers to estimate the learning effort precisely.

7.3.2 How to Mitigate the Quality Risks To get a high-quality system, it is important to ensure that every component in the system does not bring negative effects on the whole system. The results shown in Figure 6-8 illustrate that OTS components rarely have a negative effect on system quality.

Our results of RQ3.3 and RQ3.4 (see Section 6.2.3) show that the black-box testing, and early integration testing, which have been very popular in practice, helped to avoid possible quality problems. Boehm et al. [Boehm03] discovered that the black-box testing is always imposed in OTS-based development and suggested that a risk based testing plan must be developed and accounted for in the project life cycle schedule. In the COTS component market, the salesman may make faulty claims about the functional and non-functional characters of their products [Boehm03], [FAA03], [Mitre05]. Thus, the users need to do some testing themselves to ensure that the quality of the component is the same as that claimed by the vendor. For the OSS components, there is never enough information to illustrate the full picture of the functions and the quality of the components. Thus, users need to test and evaluate the issues they are interested in by either reading the source code or doing black box testing.

However, the performance of the system highly depends a lot on the interaction between components and the environment, such as the hardware and the workload. It is therefore difficult to predict and evaluate by just incremental integration testing. Our investigated risk mitigation activities did not show effective reduction on the OTS components’ possible negative effect on system performance.

Although other solutions have been proposed to mitigate the possible quality issues of the overall system, such as selecting the OTS based on architecture instead of functionalities [Rose03] [Torchiano04], [Li06a], or leverage the commercial


83

infrastructure [FAA03]. Our results of RQ3.4 (see Section 6.2.3) did not show a strong effect from these activities. A possible reason is that most OTS components investigated in our studies followed the industrial standards or infrastructures - such as COM, CORBA, .Net, and EJB - by default. The possible quality problems caused by the architecture mismatch itself are therefore minimized.

7.3.3 How to Mitigate the Requirements Risks Another popular issue on OTS component integration is the requirements changes and mismatches with OTS component functionalities [Boehm03], [FAA03], [Rose03] [Mitre05].

Since it is difficult to change the OTS component to follow the customers’ changes in requirements, it is important for the OTS component integrator to be able to negotiate requirements with the customer [Boehm03], [Rose03], and involving customers into make vs. acquire decision and OTS component selection [FAA03]. However, our results in Figure 6-9 show that very few integrators have actually involved customers into the make vs. acquire decision and the OTS component selection. Results of the pre-study (see Paper P2 [Li06a] in Appendix A) illustrate that it is not easy to negotiate requirements with the customer in OTS component-based development. One reason is that customer only cares about the final product without willingness or capability of knowing the technical details.

To mitigate the possible requirements risks, our results of RQ3.4 (see Section 6.2.3) show that careful evaluation and testing of OTS components in the selection phase helped to increase the possibility to follow the customers’ requirements changes. The possible explanation is that people understand the OTS components better when they did testing and quality evaluation in the selection phase. They therefore know the capability and the possible extensions of the OTS components.

Another possible activity is to first integrate the unfamiliar, i.e., high risk or critical OTS components [Rose03], [Li06a]. Integrating the critical components first helped to provide the solid foundation for the architecture of the system [Rose03]. Integrating the high risk or most complex components first can help to find problems in the early phase of the project, which may give the integrator leeway to negotiate with their customers to redefine the requirements [Lauesen06]. To negotiate the requirements with the customer, educating the customer in requirements negotiation will be critical, but this must be accomplished in a manner that is politically viable for the particular customer [Rose03]. The possibility of negotiation depends on the willingness of the customer to negotiate issues and to accept changes. Our results of RQ3.4 (see Section 6.2.3) illustrate that customers in certain business culture, such as in Germany, are more difficult to accept requirements negotiation than others, as shown in Section 5.5 in Paper P5 [Li06c] in Appendix A.

7.3.4 How to Mitigate the Debugging and Deployment Risks Our results shown in Figure 6-8 reveal that the difficulty in locating the cause of defects happened more frequently than other problems. Kotonya and Rashid [Kotonya01] point out that OTS component offering similar functionalities may have very different system resource requirements (i.e., memory and processor requirements). When a problem is identified, it may not be obvious where the root cause of the problem is located. The


84

problem could be in one of the components, it could be in the integration code, or it could be a misinterpretation of the semantic operation of a component interface [Rose03].

The results of the research question RQ3.4 (see Section 6.2.3) show that this risk can be mitigated by giving developers enough time to learn and understand the OTS components. Since both COTS and OSS components are acquired from a third-party organization, it will require a lot of time for the users to understand the underlying design and code in these components. It is especially difficult for the COTS component users to use the components properly when source code is not available. In case there are defects in the OTS components or the interactions between the OTS components are not properly controlled, it may take longer time for the component users to figure out the problems than they had expected.

For the possible deployment problem, our findings of RQ3.4 (see Section 6.2.3) indicate that an effective way of controlling it is to start integration testing as early as possible. In the early days of integrating OTS into a system solution, there was a false assumption that the need for integration testing was either minimal or not needed at all. It seems that there was a belief (actually a false hope) that once the OTS components were installed, the system would be operational [Rose03]. The OTS providers are only knowledgeable in their own OTS components. If there is no strong evidence that problems in the integrated system are caused by a specific OTS component, the component vendor may not want to help the users to debug and solve the problems. The benefit of doing integration testing immediately after one OTS component is integrated is that it is easy to conclude that the problems lie in the newly added component. An additional possible mitigating activity is to do some analysis to determine the order in which the OTS components are to be integrated [Spc01].

7.3.5 How to Mitigate the Maintenance Risks Maintaining an OTS component-based system is difficult. The post-deployment costs may exceed the development costs [Basili01]. The different customer-vendor evolution cycles may result in an uncertainty about how often the OTS components in a system may have to be replaced and the extent of the impact of such a change on the rest of the system [Kotony01, [Rose03], [Mitre05], [FAA03].

One risk mitigation solution proposed is to build an explicit, formal strategic relationship that includes mutual non-disclosure agreements to ensure that the provider gives full disclosure of the products future upgrades [Rose03]. However, this strategy is only applicable in a few situations. In case the customer is not regarded as a major one by the COTS component provider, it is difficult to build such a relationship. In the case of using OSS components, it is also difficult to build such a relationship, because the OSS components are updated by volunteers and nobody knows what functions may be added in future releases.

Another risk mitigation activity is to have senior analysts (serving as knowledge keepers) directly involved in the analysis of the updates and deriving effective field upgrade procedures and schedules [Rose03], [Li06a]. This is a non-trivial system analyst task that requires appropriate level of expertise, as there is very little in the literature to help guide the analysts to a feasible solution [Rose03]. The existence of the knowledge keeper has been investigated by RQ3.4 (see Section 6.2.3) and show effective results on mitigating the possible risks due to the OTS component evolution.


85

Since the knowledge keepers know the current component version being used, it is possible for them to evaluate the new versions and to figure out the benefits and risks of updating the current version with the newer one. The personal capability and experience remain the dominant factors influencing OTS-based development [Basili01]. Therefore, it is important to involve OTS- knowledgeable individuals in all analytical processes [FAA03].

However, planning maintenance of an OTS component-based system is very difficult because the OTS component users have no control of the release cycle of the OTS components being used. Since the release cycle of the OTS component depends on the marketing strategy of the component vendors, the component users usually have no idea when the new release of the OTS component will be published. Our investigated risk management activities focus on the activities that can be controlled by the component users. Such risk management activities did not show effectiveness to mitigate the maintenance plan risk.

7.3.6 How to Mitigate the Provider Support Risks Another issue in OTS component-based development is to get enough technical support from the provider. Although the users can have access to the source code when using the OSS components, few of them have actually modified the code to meet their requirements [Li05a]. It is also advised to avoid modifications of OTS components, even when it is possible [FAA03]. COTS component users can get support from the provider by signing a support contract. However, COTS providers may still stop to support their products due to different reasons, or even go out of business [Rose03], or the actual version of a COTS component is too old to be supported [Basili01]. For the OSS components, some of them are currently well supported by the volunteers in the OSS projects. However, it does not ensure that the users can get the same level of support in the future. The users have no control of the technical support they want to get from the OSS community. As our study tried to find out the common risk mitigation activities for both COTS and OSS components, we did not investigate the contract and licensing techniques, which is not applicable for the OSS user. We therefore looked more on the activities that are under control among OTS component users.

For the provider support risks, our results of RQ3.4 (see Section 6.2.3) show that the most important remedy is still to select the right providers. It is necessary to perform continuous watch on the support reputation of the OTS component providers, as proposed in [FAA03], [Mitre05]. In addition, it is important to trace the history of the provider support by consulting the knowledge keeper.

7.4 RQ4: How to Make the Decision between COTS or OSS Components?

For research question RQ4, we compared the use of COTS and OSS components in practice. We contributed to the C4, i.e., increased understanding on the decision making practices in OTS-based development (see paper P6 [Li06d] in Appendix A).

We summarized who, why and how these components are used. We then proposed a brief guideline on selecting one from two possible options - COTS components and OSS components.


86

7.4.1 Why Did Developers Decide to Use OTS Components Using COTS and OSS components promises shorter time-to-market and increased productivity [Voas98c]. Our results of RQ4.2 (see Section 6.2.4) confirm that the main expectation of using OTS components is to shorten the time-to-market and to save development cost. Results also show that COTS users believe that paid software will give good reliability and will follow the market trend. They also believe that the COTS vendor will provide good technical support. On the other hand, the main motivation of using OSS components is that code could be acquired for free and that the source code is available. It means that the OSS component users care more about the initial investment. Another attraction of OSS components is that the marginal cost of scaling up is zero, i.e., using OSS does not require additional licences as the number of installations grows. This benefit is especially important when integrating OSS components into a software product with many installation bases.

7.4.2 How Were the OTS Components Used in Practice? If “white-box” vs. “black-box” is regarded as the main difference between COTS and OSS components, the results of T1 (see Section 6.2.2) show that COTS and OSS components are not as different as one might assume. COTS components were not always “black-box”, as 33% of the COTS components selected in this study opened their source code to users. If the source code is available, both COTS and OSS users tend to read the code. However, neither COTS nor OSS component users did change much of the available source code. From this result, we can generally conclude that the OSS component was used in the same way as COTS component, i.e., integrated without modification. Without source code, it is difficult for COTS component users to identify whether a defect is inside or outside the component. Although the source code of OSS components is available, OSS users might not have bothered to understand the source code of an OSS component before starting to use it. It is therefore difficult for them to figure out if a defect is inside an OSS component or not.

By analysing all activities normally performed when integrating OTS components, Giacomo, P. D. [Giacomo05] concluded that the OSS component is not completely different from the COTS component. Our results of RQ4.1 (see Section 6.2.4) show that both COTS and OSS components were used in projects and companies with similar profiles. The most emphasized requirements of the final systems were time-to-market, reliability, and performance. Security was surprisingly listed as the least emphasized requirement of the system. There are many discussions about whether OSS components are more secure than COTS components. Raymond [Raymond99] argued that “given enough eyeballs, all bugs are shallow.” Others argued that it is the quality, not the number of eyes looking at code that count [Hissam02]. Our results show that OTS component users are still very cautious of using either COTS or OSS components in the security-critical systems. So, software security aspects may not be distinguished between COTS and OSS components. However, trust in the security of third-party components may become an issue in the future.

Although COTS users believe that the paid components should have good quality and good technical support, our results of RQ4.3 (see Section 6.2.4) show that they were not more satisfied with reliability and performance of the components than OSS users. In addition, COTS users had significantly more difficulties in estimating the


87

component selection effort. The possible reason is that it is not easy or cost-effective for COTS users to switch to another COTS component once they have made the decision and paid for some selected components. Therefore, COTS users might be more cautious in selecting the right COTS component and spend more effort than estimated on selecting the best one. COTS users have also more difficulties on following the requirements changes. It might be because the source code is not available and it is therefore difficult to change the source code of the COTS component.

Although OSS users did not have a commercial support contract with OSS component providers, results of RQ4.3 (see Section 6.2.4) show that they were not less satisfied by the technical support than COTS users. The possible interpretation is that OSS users would like to read and change the code themselves, instead of asking for help, as we found in the key motivations of using OSS.

7.4.3 COTS or OSS? Our Suggestions In a system with undefined or fast changing requirements, OSS components might be better than COTS components. It might be easier for OSS users to change the source code of the OSS component if their customers’ requirements do change. Therefore, OSS components are ideal when the software solution requires high customization [Madanmohan04]. However, COTS components users can also ask for source code from the vendor in some situations. In this case, there will be fewer difficulties on following the requirements changes.

Concerning the use of domain-specific OTS components, it is important to ensure that the evolution of the selected OTS component will follow the market trend. Although a COTS user cannot directly influence the evolution of the selected COTS component, results of RQ4.2 (see Section 6.2.4) show that these users still trust that the COTS vendor will update the component according to market needs.

OSS components can be changed for similar market reasons, but can also be changed for political or social reasons – not necessarily in a direction suitable to the organization [Hissman01]. It is therefore important for OSS users to ensure that they are planning to use the software in the way it was intended to be used and the way most other users are using it. If OSS users stray too far away from the beaten path, the OSS developers might not want to help them when they run into trouble, or OSS developers will not accept the changes contributed by them [Madanmohan04]. More and more OSS components are supported by a commercial organization [Stewart05]. It is possible for OSS users to select components with commercial support in order to avoid the possible maintenance and evolution problems.

7.5 Evaluation of Validity Threats

Four groups of validity threats in empirical research are considered important and validity threats of the individual studies are discussed in this section.

7.5.1 Possible Validity Threats in the Preliminary Study The preliminary study (see paper P1 [Li04a] in Appendix A) is used to investigate RQ1 and contributed to C1. The study used the questionnaire to investigate the developers’


88

attitude on reuse in three IT companies. Some threats to validity of this study and how these are handled are as following:

• Internal validity. A major threat to internal validity is that we have not assessed the reliability of our measurement. Most variables are measured on a subjective ordinal scale. An important issue for future studies is to ensure the reliability and validity of all measurement. In this survey, we gave clearly specified concepts in the questionnaire and provided support to possible misunderstanding. These methods increased the reliability.

• Construct validity. In our case, the main construct issue applies to the variables chosen to characterize the data set. The independent variable, i.e., reuse level, is the most sensitive one. We asked several questions in the questionnaire to evaluate this variable. The results of these questions give a consistent value on this variable.

• External validity. The small sample size and lack of randomness in the choice of companies and respondents are threats to external validity.

• Conclusion validity. Although we used some statistical methods to analyse results of this study. This study is still an explorative study. Future studies will be implemented to give more statistically significant results.

7.5.2 Possible Validity Threats in the Pre-study The pre-study (see papers P2 [Li06a] in Appendix A) used semi-structured interviews to clarify research questions RQ2, RQ3, RQ4. Some threats to validity of this study and how these are handled are as following:

• Internal validity. The first possible threat to internal validity is possible misunderstanding of respondents’ answers. Although two interviewers carried out the interviews and there was only one in each interview, we taped all interviews. Listening to the tape helped ensure correct interpretation of answers and comments. However, having an independent (third) person to listen to the tape might increase data quality. The second possible threat to internal validity is that people tend to talk about examples of good practice instead of problems they encountered. As we had a confidentiality agreement with the respondents, the respondents were encouraged to tell the truth. However, the respondents might give only the most salient problems they encountered and forget some minor problems because we asked for data about past projects. The third possible threat to internal validity is that the life cycle phases, such as requirement, design, and implementation, may not be clearly separated in projects using prototyping and incremental processes. We called back some respondents to ask whether they had specific software architecture in mind before the make vs. acquire decision and COTS component selection. Their answers gave future clarification on whether COTS components were selected in the requirements or design phases. The fourth possible threat to internal validity is that some projects might not tailor the process because their processes already support COTS usage. So, we might miss some necessary process improvements


89

that have been performed before the project start because we asked only salient process changes within the studied projects.

• Construct validity. In this study, most variables are taken directly, or with little modification, from the existing literature. To ensure construct validity, we did two rounds of pre-tests on the interview guide in ten local IT companies. About 10% of the questions and alternatives in the interview guide were revised based on the feedback from the pre-test. In addition we used three meta-questions to evaluate whether the questions in the interview guide were understood consistently. Answers of these meta-questions illustrate that the questions are easy-to-understand and reliable. In the data analysis part, we divided the projects into successful and unsuccessful projects. The criterion to define a successful project is that the integrated COTS components should contribute positively both to time-to-market and to the quality of the final system. However, this criterion may not apply for all kinds of projects as some projects might emphasis only time-to-market, and others might emphasis only quality.

• External validity. The primary threat to external validity is that the study is based on few and possibly not typical companies in Norway. According to Norwegian “Census Bureau” (SSB) [SSB02], the mean value of employee number in Norwegian IT companies is 6. The first possible external validity threat to this study is that our selected companies are mostly medium and large sized. However, the selected projects covered several application domains. The second possible threat to external validity is that the size (based on person-hours) of the projects might bias the conclusions from this study. We have divided the projects into three groups: small (less than 10,000 person-hours), medium (between 10,000 and 100,000 person-hours), and large (more than 100,000 person-hours). We found out that the actual process used, the problems encountered, and the good practice crossed small, medium, and large projects. The third possible threat to external validity is that most COTS components are well-defined and provided only secondary (support) functionality of the delivered system. The process customizations may be different if the COTS components are intended as the core part of the system, or the COTS components are large packages, such as ERP and content management systems.

• Conclusion validity. This study is a qualitative study without statistical analysis. The conclusions of this study are descriptive and may need further refinement and verification.

7.5.3 Possible Validity Threats in the Main Study The main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c], and P6 [Li06d] in Appendix A) investigate the clarified research questions RQ2, RQ3, RQ4 from the pre-study. The data of the main study were collected by a questionnaire and analysed by statistical methods. The results of the main study show our contributions C2, C3, C4, and C5. Some threats to validity of this study and how these are handled are as following:

• Internal validity. We promised respondents in this study a final report and a seminar to share experience. The respondents were people who wanted to share


90

their experience and wanted to learn from others. In general, we think that the respondents answered the questionnaire truthfully. However, different people in the same project might have different opinions on the same project. Asking only one person in each project might not be able to get the whole picture of the project. Due to length limitation of a questionnaire, we asked the respondent to fill in information about only one component in the project. The possible threat is that other OTS components in the same project might give different answers to our questions.

• Construct validity. In this study, most variables and alternatives are taken directly, or with little modification, from existing literature. The questionnaire was pre-tested using a paper version by 10 internal experts and 8 industrial respondents before being published on the SESE tool [Sese04]. About 15% of the questions have been revised based on pre-test results. One possible threat to construct validity is that respondents from different companies may have slightly different understanding of the same concept, such as software architecture and architectural mismatches.

• External validity. We used different randomization methods to select samples in different countries. However, the sample selection processes were not exactly the same due to resource limitations. Another possible threat to external validity is that our study focused on OTS components followed the definition in Section 2.6. Conclusions may be different in projects using complex and large OTS packages, such as ERP, content management systems, and web services in general.

• Conclusion validity. This study is a state-of-the-practice one. We studied what had happened in industrial projects. The same size of this study is generally enough for statistical methods used to analyse the result. However, we generally treat the data from three countries the same. The cultural differences between three countries might affect the reliability of the conclusions.

7.5.4 Possible Validity Threats in the Follow-up Study The follow-up study (see paper P7 [Li05b] in Appendix A) was performed informally as an industrial seminar. This study is designed to briefly investigate the cause-effects of our results on research question R2, RQ3, and RQ4. It improves our understanding on contributes C2 to C5.

The possible threats to the construct validity are that there are no well-defined metrics to measure the answers of respondents. The possible threat to the external validity is that the conclusions are based on personal opinions without verifications on a large scale.

7.6 Summary

In this chapter, we have illustrated our contributions to each research questions. We compared our results with the state-of-the-art and gave some suggestions on process improvement, risk management, and decision-making in OTS component-based development. Due to possible threats to validity of each study, the conclusions of each study should be interpreted considering these validity issues

Conclusions and future work

91

8 Conclusions and Directions for Future Work

This thesis has presented the results of several empirical studies performed to investigate project management issues of OTS component-based development. The studies included several steps with mixed-method research design to take benefit of different empirical methods and answer the research questions reliably.

8.1 Conclusions

Empirical research can be performed to verify theories, develop new theories or extend existing ones, and improve practice. The thesis contributes to all these aspects by describing difference aspects of reusing components, i.e., the power of example [Parastoo04]; verifying existing theories and assessing existing method, i.e., the power of replication; generating new theories, hypotheses, or methods, i.e., the power of generalization.

8.1.1 Describing Different Aspects of Reusing Component We have investigated challenges related to four key factors for development based on in-house components, especially in development-with-reuse. These factors are component-related requirements (re)negotiation, component repository, component understanding, and components’ quality attribute specification. We also compared three IT companies with different reuse levels to study the possible trend and challenges in these factors when more and more code will be encapsulated as reusable components inside a company. We concluded that component-related requirements (re)negotiation for in-house built components is important but not efficient. The efficiency will not increase with higher reuse level. We confirmed that the component repository is not a key factor for successful reuse and the potential value of a component repository will not increase with higher reuse levels. For component understanding, we found that most developers are not satisfied with the component documentation, and developers’ satisfaction with component documentation decreased with a higher reuse level. We recommend that the informal communication channels (tacit knowledge), through which developers can get necessary information about the components, should be given more attention. For the quality attribute specification of components, we found that


92

developers still need to spend a lot of effort on testing, as they cannot get relevant information from the component specifications.

8.1.2 Verifying Existing Theories and Assessing Existing Methods We have assessed six newly published theses from the literature [Torchiano04] on OTS-based development. The results of our study gave support to four new theses and contradict the two others as follows:

- We support T1 (i.e., open source software is often used as closed source) and conclude that source code of OSS components was read frequently with very few modifications. OSS components were de facto used as COTS components, even if the source code was available.

- We support T3 (i.e., custom code mainly provides additional functionalities.). We conclude that more addware had been built than glueware in the COTS components integration.

- We support T4 (i.e., developers seldom use formal selection procedures. Familiarity with either the product or the generic architecture is the leading factor in selection.) We conclude that formal selection procedures were seldom used to select COTS components. Familiarity-based and hands-on experiment-based selection process were frequently used in practice.

- We support T6 (i.e., integrators tend to influence the vendor on product evolution whenever possible). We conclude that, in most cases, COTS users managed to get their required changes in the COTS components from COTS vendors.

- We are against T2 (i.e., integration problems result from lack of compliance with standards. Architecture mismatches constitute a secondary issue). We conclude that there were more architectural mismatches than standard mismatches in practice.

- We are against T5 (i.e., architecture is more important than requirements for product selection). We conclude that functionality completeness was regarded to be more important than architectural compliance in COTS component selection.

We have investigated state-of-the-practice of the main development processes and OTS component selection processes. The main findings are:

- The actual OTS-based development process was the traditional process with

OTS-specific activities. The development process was dominated by existing company/department rule instead of the decision of using OTS components.

- The actual OTS component selection can be done in different phases. The main evaluation processes are familiarity-based or hands-on-trial-based. The phase to select OTS component depends on the project members’ familiarity.


We have also investigated the proposed risks and risk management theories in OTS

component-based development. The main findings are:


93

- The most frequent problems in OTS-based projects are wrong estimation of the integration effort and the difficulty to locate defects. The least frequent problems are relevant to the negative effect of the OTS components on the quality of the whole system.

- To increase the accuracy of effort estimation, it is important to allocate enough time to learning and understanding the OTS components.

- To avoid the possible quality problems and to increase the efficiency of locating the defects, it is important to do black-box testing as part of OTS component selection and to do integration testing as early as possible.

- To mitigate the problems due to customers’ requirements changes, it is important to learn the OTS components thoroughly in the selection phases and to integrate the unfamiliar OTS components first. It is also important to be able to negotiate changed requirements with the customers.

- To increase the efficiency of debugging and system deployment, it is necessary to understand the components well and to do integration testing early.

- It is critical to keep and share the knowledge about OTS components and the component providers inside the organization. It shows to help mitigate the possible risks in system maintenance and evolution.

We have used stratified-random sample selection strategies to select representative

samples in three European countries. The lessons we learned show that all previous surveys have had problems in defining the study population and in selecting a valid sample out of this, especially since the study object is a special project, not a company. We also conclude that we can achieve a stratified-random (random based on size in three nations) sample of ICT companies, followed by a convenient sample of relevant projects at best.

8.1.3 Generating New Theories from New Perspectives By summarizing our findings on SPI in OTS component-based development, we have classified the state-of-the-practice OTS processes into seven scenarios. In addition, we have discussed how to decide/modify OTS-based development processes and how to select OTS components in each scenario.

By discussing practice with industrial representatives, we have briefly investigated the possible reasons for our discovered phenomena in SPI and risk management. These findings proposed some possible research issues in OTS component-based development in the future.

By summarizing our findings on risk management issues for OTS component-based development, we have recommended how to identify the possible risks and how to plan risk management in OTS-based projects.

By comparing the expectations and results of using COTS and OSS components, we have proposed how to select from two options, COTS and OSS components.


94

8.2 Future Work

The results of this thesis, its contributions and limitations, point to several possibilities for future research as following, which have relevance for theory, methodology, and practice.

8.2.1 Need to Study the Cause-Effects of the SPI Issues The main limitation of the study on SPI is that it is a state-of-the-practice survey. Although the discussion with industrial partners gave us some insights on the reasons of industrial practice, we need more detailed case studies or semi-structured interviews to propose and verify cause-effect issues. The intention is to deepen our understanding of process improvement in OTS component-based development and verify our proposals from this study.

Another limitation of the study on SPI is that we did not involve business critical projects, such as safety-critical projects and security-critical projects. Comparing the differences of SPI in these projects with projects in our study will provide more insights on SPI issues.

Due to the fast changes in the IT industry and OTS technologies, it is also necessary to replicate such a study to investigate the possible changes of the state-of-the-practice.

8.2.2 Risk Management Issues Need a Deeper Investigation We did an exploratory investigation of risk management issues in OTS component-based development. Due to the explorative nature of the study, further studies are needed to verify our conclusions. The assumption of our study is that the a priori risk probabilities of typical risks in the investigated projects are similar. However, it is possible that there are minor differences in the a priori risk probabilities between projects. It is therefore necessary to follow several OTS component-based projects from start by case studies. In the future, we plan to examine the several projects by estimating risk probabilities before the project start, following the project execution, and measuring the risk occurrences after the projects are finished. The intention is to investigate the cause-effect of the risk management activities on the corresponding risks.

Moreover, some of the conclusions of this study need further clarification. For example, which knowledge needs to be kept and shared in the organization and how to share it? How to estimate the possible learning effort needed when using the unfamiliar OTS components? These research questions need to be further studied by case studies or experiments.

8.2.3 Need to Better Understand OSS Projects and Products In order to provide guidelines for using OTS components in practice, it is important to understand how OSS projects are organized. Furthermore, it is important to know how to cooperate with OSS developers, for example, how to ensure that changes from users are accepted by the OSS community, and how to motivate developers to enhance and adapt OSS components into a desirable direction. Another important issue is the definition of suitable metrics to obtain reliable data about the concrete nature of OSS projects and products. Future studies have to examine OSS projects in order to identify, define, and validate more metrics.

References

95

References [Abts00] Abts, C., Boehm, B. W., and Clark, E. B.: COCOTS: A COTS Software

Integration Cost Model - Model Overview and Preliminary Data Findings. Proc. of the 11th European Software Control and Metrics Conference (ESCOM-SCOPE 2000), Apr. 2000, Munich, Germany, Shaker Publ., pp. 325-334.

[Albert02] Albert, C. and Brownsword L.: Evolutionary Process for Integrating COTS-Based System (EPIC): An Overview. SEI, Pittsburgh, 2002, available at: http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html.

[Alves03] Alves, C. and Fineklstein, A.: Investigating Conflicts in COTS Decision-Making. Journal of Software Engineering and Knowledge Engineering, 13 (5): 473-493, Oct. 2003.

[Atkinson02] Atkinson, C., Bayer, J., Bunse, C., Kamsties, E., Laitenberger, O., Laqua, R., Muthig, D., Paech, B., Wüst, J., and Zettel, J.: Component-based Product Line Engineering with UML. Addison-Wesley, 2002.

[Babbie90] Babbie, E.: Survey Research Methods. Wadsworth, 1990. [Bachmann00] Bachmann, F., Bass, L., Buhman, C., Comella-Dorda, S., Long, F.,

Robert, J., Seacord, R., and Wallnau, K.: Volume II: Technical Concepts of Component-Based Software Engineering. SEI Technical Report number CMU/SEI-2000-TR-008, 2000, available at: http://www.sei.cmu.edu/

[Barki93] Barki, H., Rivard, S., and Talbot, J.: Toward an Assessment of Software Development Risk. Journal of Management Information Technology, 22 (2): 359-371, Dec. 1993.

[Basili86] Basili, V. R., Selby, R. W., and Hutchens, D. H.: Experimentation in Software Engineering. IEEE Transaction on Software Engineering, 12 (7): 733-743, July 1986.

[Basili01] Basili, V. R. and Boehm, B. W.: COTS-Based Systems Top 10 List. IEEE Computer, 34 (5): 91-93, May 2001.

[Basili02] Basili V. R., Lindvall M., Rus I., Seaman C., and Boehm B.: Lessons-Learned Repository for COTS-Based SW Development. Software Technology Newsletter, 5 (3): 4-7, Sept. 2002, available at: http://fc-md.umd.edu/ll/index.asp

[Baskerville03] Baskerville, R., Ramesh, B., Levine, L., Pries-Heje, J., and Slaughter, S.: Is Internet-Speed Software Development Different? IEEE Software, 20 (6): 70-77, Nov./Dec. 2003.

[Bass00] Bass, L., Buhman, C., Comella-Dorda, S., Long, F., Robert, J., Seacord, R., and Wallnau, K: Volume I: Market Assessment of Component-based Software Engineering. SEI Technical Report number CMU/SEI-2001-TN-007, available at: http://www.sei.cmu.edu/

[Boehm88] Boehm, B. W.: A Spiral Model of Software Development and Enhancement. IEEE Computer, 21 (5): 61-72, May 1988.

[Boehm89a] Boehm, B. W.: Software Risk Management, Tutorial, IEEE CS Press, 1989.

References

96

[Boehm89b] Boehm, B. W. and Ross, R.: Theory-W Software Project Management: Principles and Examples. IEEE Transactions on Software Engineering, 15 (7): 902-916, July 1989.

[Boehm91] Boehm, B. W: Software Risk management: Principles and Practices. IEEE Software, 8 (1): 32-41, Jan.1991.

[Boehm98] Boehm, B. W., Egyed, A., Kwan, J., Port, D., Shah, A., and Madachy, R. J., Using the WinWin Spiral Model: A Case Study. IEEE Computer, 31 (7): 33-44, July 1998.

[Boehm99] Boehm B. W. and Abts, C.: COTS integration: Plug and Pray? IEEE Computer, 32 (1): 135-138, Jan. 1999.

[Boehm00] Boehm, B. W.: Requirements that Handle IKIWISI, COTS, and Rapid Change. IEEE Computer, 33 (7): 99-102, July 2000.

[Boehm03] Boehm, B. W., Port, D., Yang, Y. and Bhuta, J.: Not All CBS Are Created Equally COTS-intensive Project Types. Proc. of the 2nd International Conference on COTS-Based Software Systems (ICCBSS 2003), Feb. 2003, Ottawa, Canada, Springer LNCS Vol. 2580, pp. 36-50.

[Bosch00] Bosch, J.: Design and Use of Software Architectures: Adopting and Evolving a Product-line Approach. Addison-Wesley, 2000.

[Brooks87] Brooks, P. F. Jr.: No Silver Bullet: Essence and Accidents of Software Engineering. IEEE Computer, 20 (4): 10-19, Apr. 1987.

[Brownsword00] Brownsword, L., Oberndorf, T., and Sledge, C.: Developing New Processes for COTS-Based Systems. IEEE Software, 17 (4): 48-55, July/August 2000.

[Bush90] Bush, M. E. and Fenton, N. E.: Software Measurement: A Conceptual Framework. Journal of System and Software, 12 (3): 223-231, March 1990.

[Carr93] Carr, M., Kondra, S., Monarch, I., Ulrich, F., and Walker, C.: Taxonomy-Based Risk Identification. SEI report SEI-93-TR-006, Pittsburgh, USA, 1993.

[CBSEnet04] The CBSEnet project, July 5, 2004, available at: http://www.cbsenet.org [Cechich04] Cechich, A., and Piattini, M.: Quantifying COTS Component Functional

Adaptation. Proc. of the International Conference on Software Reuse, July, 2004, Madrid, Spain, Springer LNCS Vol. 3107, pp. 195-204.

[Charette89] Charette, R. N.: Software Engineering Risk Analysis and Management. Intertex Publications McGraw-Hill Book Col, 1989.

[Chittister96] Chittister, C. G., and Haimes, Y. Y.: System Integration via Software Risk Management. IEEE Transactions on Systems, Man, and Cybernetics, 26 (5): 521-532, Sep. 1996.

[Chung02] Chung, L. and Cooper, K.: A Knowledge-Based COTS-Aware Requirements Engineering Approach. Proc. of the 14th International Conference on Software Engineering and Knowledge Engineering, July 2002, Ischia, Italy, pp. 175-182.

[CMM06] http://www.sei.cmu.edu/cmm/

References

97

[Componentsource04] http://www.componentsource.com [Conradi05a] [P8]Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Torchiano, M.

and Morisio, M.: Reflections on conducting an international CBSE survey in ICT industry. Proc. of the 4th International Symposium on Empirical Software Engineering (ISESE 2005), Nov. 2005, Noosa Heads, Australia, IEEE Press, pp. 214-223.

[Conradi05b] [AP3] Reidar Conradi and Jingyue Li: Observations on Versioning of Off-the-Shelf Components in Industrial Projects (short paper). Proc. of the 12th International Workshop on Software Configuration Management (co-located with the ESEC/FSE 2005), Sept. 2005, Lisbon, Portugal, pp. 33-42, ACM Digital Library.

[Cooper03] Cooper, D. R. and Schindler, P. S.: Business Research Methods. 8th Edition, McGraw-Hill Press, 2003.

[Corra02] Corra, M. and Willer, D.: The Gatekeeper. Sociological Theory, 20 (2): 180-207, July 2002.

[Creswell94] Creswell, J.W.: Research Design, Qualitative and Quantitative Approaches. Sage Publications, 1994.

[Creswell03] Creswell, J.W.: Research Design, Qualitative, Quantitative, and Mixed Methods Approaches. 2nd edition, Sage Publications, 2003.

[Crnkovic01] Crnkovic, I. and Larsson, M.: A Case Study: Demands on Component-based Development. Proc. of the 22nd International Conference on Software Engineering, June 2000, Limerick, Ireland, pp. 21-31.

[Crnkovic02] Crnkovic, I., Larsson, M.: Building Reliable Component-Based System. Artech House, 2002.

[Denzin94] Denzin, N. K. and Lincoln, Y. S.: Handbook of Qualitative Research, Sage Publications, London, UK 1994.

[Eisenhardt89] Eisenhardt, K.M.: Building Theories from Case Study Research. Academy of Management Review, 14 (4): 532-550, 1989.

[Endres03] Endres, A. and Rombach, D.: A Handbook of Software and Systems Engineering, Empirical Observations, Laws, and Theories. Addison-Wesley Professional, 2003.

[FAA03] COTS risk factor, available at: http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm, 2003. [FAMILIES03] FAMILIES project, available at: http://www.esi.es/en/Projects/Families, 2003. [Feller02] Feller, J. and Fitzgerald, B.: Understanding Open Source Software

Development. Addison-Wesley, 2002. [Fenton97] Fenton, N. and Pfleeger, S.L.: Software Metrics: A Rigorous and Practical

Approach. International Thomson Computer Press, 2nd edition, 1997. [Finkelstein00] Finkelstein, A. and Kramer, J.: Software Engineering: a Road Map.

Proc. of the 22nd International Conference on Software Engineering, Future of Software Engineering Track, June 2000, Limerick Ireland, pp. 3-22.

References

98

[Fitzgerald04] Fitzgerald, B.: A Critical Look at Open Source. IEEE Computer, 37 (7): 92-94, July 2004.

[Frakes95] Frakes, W.B. and Fox, C.J.: Sixteen Questions about Software Reuse. Communications of the ACM, 38 (6): 75-78, June 1995.

[Freesoftware05] www.gnu.org, 2005. [Fox97] Fox, G., Lantner, K., and Marcom, S.: A Software Development Process for

COTS-based Information System Infrastructure. Proc. of the 5th International Symposium on Assessment of Software Tools (SAST '97), June, 1997, Pittsburgh, PA, USA, pp. 133-142.

[Fowler01] Fowler. F. J., Jr.: Survey Research Methods (Applied Social Research Methods). 3rd edition, Sage Publications, 2001.

[Fuggetta00] Fuggetta, A.: Software Process: A Roadmap. Proc. of the 22nd International Conference on Software Engineering, Future of Software Engineering Track, June 2000, Limerick Ireland, pp. 25-34.

[Gemmer97] Gemmer, A.: Risk Management: Moving Beyond Process. IEEE Computer, 30 (5): 33-41, May, 1997.

[Ghosh02] Ghosh, S.: Improving Current Component-Based Development Techniques for Successful Component-Based Software Development. Proc. of International Conference on Software Reuse (ICSR7), Workshop on Component-based Software Development Processes, 2002, available at: http://www.idt.mdh.se/CBprocesses/

[Giacomo05] Giacomo, P. D.: COTS and Open Source Software Components: Are They Really Different on the Battlefield? Proc. of the 4th International Conference on COTS-Based Software Systems, Feb. 2005, Bilbo, Spain, Springer, LNCS Vol. 3412, pp. 301-310.

[Gifi90] Gifi, A.: Nonlinear Multivariate Analysis, John Wiley & Sons, 1990. [Glaser67] Glaser, B.G. and Strauss, A. L.: The Discovery of Grounded Theory:

Strategies for Qualitative Research. Aldine Publishing, 1967. [Glass01] Glass, R. L.: Frequently Forgotten Fundamental Facts about Software

Engineering, IEEE Software, 18 (3): 110-112, May/June, 2001. [Griss93] Griss, M.L.: Software Reuse: From Library to Factory. IBM Systems Journal,

32 (4): 548-566, Nov./Dec. 1993. [Griss95] Griss, M.L. and Wosser, M.: Making Reuse Work in Hewlett-Packard. IEEE

Software, 12 (1): 105-107, Jan. 1995. [Hall98] Hall, E.: Managing Risk: Methods for Software Systems Development.

Readings, MA: Addision-Wesley, 1998. [Hecht04] Hecht, H.: Systems Reliability and Failure Prevention. Artech House

Publishers, 2003. [Heineman01] Heineman, G.T. and Councill, W.T.: Component-Based Software

Engineering, Putting Pieces Together. Addison-Wesley, 2001.

References

99

[Highsmith99]: Highsmith III, J. A.: Adaptive Software Development: A Collaborative Approach to Managing Complex Systems. Dorset House Publishing, New York, USA, 1999.

[Hissman01] Hissam, S. Weinstock, C. Plakosh, D., and Asundi, J: Perspectives on Open Source Software. Technical Report, SEI number, CMU/SEI-2001-TR-019, 2001.

[Hissam02] Hissam, S. A. Plakosh, D., and Weinstock, C.: Trust and Vulnerability in Open Source Software. IEE Proceedings - Software, 49 (1): 47-51, Feb. 2002.

[INCO00] INCO project, available at: http://www.ifi.uio.no/~isu/INCO/, 2000, [Infoweek04] InformationWeek Survey. Available at:

http://www.informationweek.com/story/showArticle.jhtml?articleID=51201599&tid=5979, 2004

[ISO06] http://www.iso.org/iso/en/ISOOnline.frontpage, 2006 [Jacobson97] Jacobson, I., Griss, M., and Jonsson, P.: Software Reuse: Architecture,

Process and Organization for Business Success. Addison-Wesley, 1997. [Jacobson99]Jacobson, I., Booch, G., Rumbaugh, J.: The Unified Software

Development Process, 1st edition, Addison-Wesley, 1999. [Jones84] Jones, T. C.: Reusability in Programming: A Survey of the State of the Art.

IEEE Transactions on Software Engineering, 10 (5): 488-494, Sep. 1984. [Kansala97] Kansala, K.: Integrating Risk Assessment with Cost Estimation. IEEE

Software, 14 (3): 69-74, May/June 1997. [Karlsson95] Karlsson, E.A. (Ed.): Software Reuse, a Holistic Approach. John Wiley &

Sons, 1995. [Keil98] Keil, M., Cule, P.E., Lyytinen, K., and Schmidt, R.C.: A Framework for

Identifying Software Project Risks. Communications of the ACM, 4 (11): 76-83, Nov. 1998.

[Kent99] Kent, B.: Extreme Programming Explained: Embrace Change. Addison-Wesley, 1999.

[Kitchenham02] Kitchenham, B.A., Pfleeger, S.L., Hoaglin, D.C., and Rosenberg, J.: Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering, 28 (8): 721-734, Aug. 2002.

[Kontio96] Kontio, J.: A Case Study in Applying a Systematic Method for COTS Selection. Proc. of the 18th International Conference on Software Engineering, Mar. 1996, Berlin, Germany, pp. 201 – 209.

[Kotonya01] Kotonya, G. and Rashid, A.: A Strategy for Managing Risk in Component-based Software Development. Proc. of the 27th EUROMICRO Conference, Sept. 2001, Warsaw, Poland, pp. 12-21.

[Kruchten00] Kruchten, P.: The Rational Unified Process. An Introduction. Addison-Wesley, 2000.

[Kruchten01] Kruchten, P.: The Nature of Software: What’s So Special about Software Engineering? The Rational Edge, Oct. 2001, http://www.therationaledge.com/.

References

100

[Lauesen06] Lauesen, S.: COTS Tenders and Integration Requirements. Journal of Requirements Engineering, 11 (2): 111-122, 2006.

[Lawlis01] Lawlis, P. K., Mark, K. E., Thomas, D.A., and Courtheyn, T.: A Formal Process for Evaluating COTS Software Products. IEEE Computer, 34 (5):58-63, May 2001.

[Lawton02] Lawton, G.: Open Source Security: Opportunity or Oxymoron? IEEE Computer, 35 (3): 18-21, March 2002.

[Leung02] Leung, K. R. P. H. and Leung, H. K. N.: On the Efficiency of Domain-Based COTS Product Selection Method. Journal of Information and Software Technology, 44 (12): 703-715, Sept. 2002.

[Li04a] [P1] Li, J., Conradi, R., Mohagheghi, P., Sæhle, O. A., Wang, Ø., Naalsund, E., and Walseth, O. A.: A Study of Developer Attitude to Component Reuse in Three IT Companies. Proc. of the 5th International Conference on Product Focused Software Process Improvement (PROFES'2004), Apr. 2004, Nara, Japan, Springer LNCS Vol. 3009, pp. 538-552.

[Li04b] [AP2] Li, J., Houmb, S. V., Kvale, A. A.: A Process to Combine AOM and AOP: A Proposal Based on a Case Study. Proc. of the 5th Aspect-Oriented Workshop (In conjunction with UML 2004), Oct. 11, 2004, Lisbon, Portugal. Available at: http://www.cs.iit.edu/~oaldawud/AOM/AcceptedPapers/

[Li04c] [AP6] Li, J., Bjørnson, F. O., Conradi, R. and Kampenes, V. B.: An Empirical Study of COTS Component Selection Processes in Norwegian IT companies. Proc. of the International Workshop on Models and Processes for the Evaluation of COTS Components (MPEC'04), May, 2004, Edinburgh, Scotland, IEE Press, pp. 27-30.

[Li04d] [AP7] Li, J., Bjørnson, F. O., and Conradi, R.: Empirical Study on COTS Component Classification, Proc. Intl. Workshop on COTS Terminology and Concepts, Co-located with International Conference on Component-Based Software Systems (ICCBSS'04), 1-4 Feb., 2004, Redondo Beach, USA, http://softeng.polito.it/iwctc/accepted.html - with list of accepted papers,

[Li05a] [P3] Li, J. Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and Morisio, M.: Validation of New Theses on Off-The-Shelf Component Based Development. Proc. of the 11th IEEE International Metrics Symposium (Metrics'05), Sep. 2005, Como, Italy, IEEE Press, pp.26, 10p.

[Li05b] [P7] Li, J. Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and Morisio, M.: Barriers to Disseminating Off-The-Shelf Based Development. ACM Software Engineering Notes (elec. version), Vol. 30, No. 4, July 2005.

[Li05c] [AP4] Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and Bunse, C.: Preliminary Results from a State-of-the-Practice Survey on Risk Management in Off-The-Shelf Component-Based Development. Proc. of the 4th International Conference on COTS-based Software System (ICCBSS 2005), Feb. 2005, Bilbao, Spain, Springer LNCS Vol. 3412, pp. 278-288.

[Li05d] [AP5] Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and Bunse, C.: An Empirical Study on Off-the-Shelf Component Usage in Industrial Projects. Proc. of the 6th International Conference on Product Focused Software

References

101

Process Improvement, Oulu, Finland (PROFES 2005), June 2005, Springer LNCS Vol. 3547, pp. 54-68.

[Li06a] [P2] Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B.: An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT Industry. To appear in the Journal of Empirical Software Engineering 2006 (Special issue from Metrics 2004).

[Li06b] [P4] Li, J., Torchiano, M., Conradi, R., Slyngstad, O. P. N., and Bunse, C.: A State-of-the-Practice Survey of the Off-the-Shelf Component-Based Development Processes, Proc. of the 9th International Conference on Software Reuse, Torino, Italy, June, 2006, Springer LNCS Vol. 4039, pp. 16-28.

[Li06c] [P5] Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and Morisio, M.: An Empirical Study on the Decision Making Process in Off-The-Shelf Component Based Development. Proc. of the International Conference on Software Engineering (ICSE 2006), Emerging Result Track, 20-28 May 2006, Shanghai, P.R. China, IEEE Press, pp. 897-900.

[Li06d] [P6] Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and Morisio, M.: An Exploratory Study on Risk Management in OTS Component-Based Development. Submitted to IEEE Transactions on Software Engineering.

[Li06e] [AP1] Li, J., Kvale, A. A., and Conradi, R.: A Case Study on Improving the Changeability of COTS-Based System Using Aspect-Oriented Programming. Journal of Information Science and Engineering, 22 (2): 370-390, March 2006.

[Lincoln85] Lincoln, Y.S. and Guba, E.G.: Naturalistic Inquiry. Sage, 1985. [Maccrimmon97] MacCrimmon, K. R.: An Overview of Multiple Objective Decision

Making. In Cochrane, J. L. and Zeleny, M. editors: Multiple Criteria Decision Making, University of South Carolina Press, 1973, pp.18-44.

[Madachy97] Madachy, R. J.: Heuristic Risk Assessment Using Cost Factors. IEEE Software, 14 (3): 69-74, May/June 1997.

[Madanmohan04] Madanmohan, T.R. and De’, R.: Open Source Reuse in Commercial Firms. IEEE Software, 21 (1): 62-69, January/February, 2004.

[Maiden98] Maiden, N. A. M, and Ncube, C.: Acquiring COTS Software Selection Requirement. IEEE Software, 15 (2): 46-56, Mar./Apr. 1998.

[McIlroy68] McIlroy, D.: Mass-produced Software Components. Proc. of the Software Engineering Concepts and Techniques Conference, 1968 Report on NATO Conference on Software Engineering, 1968, pp. 138-155.

[Michailidis98] Michailidis, G. and Leeuw, J. d.: The Gifi System of Descriptive Multivariate Analysis. Journal of Statistical Science, 13 (4): 307-336, 1998.

[Mili95] Mili, H., Mili, F., and Mili, A.: Reuse Software: Issues and Research Directions. IEEE Transactions on Software Engineering, 21 (6): 528-562, June 1995.

[Mitre05] Common Risks and Risk Mitigation Actions for a COTS-based System, www.mitre.org/work/sepo/toolkits/risk/taxonomies/files/CommonRisksCOTS.doc, 2005.

References

102

[Morisio97] Morisio, M. and Tsoukias, A.: IusWare: a Methodology for the Evaluation and Selection of Software Products. IEE Proceedings – Software Engineering, 144 (3): 162-174, Mar. 1997.

[Morisio00] Morisio, M., Seaman, S., Parra, A., Basili, V., Kraft, S., and Condon, S.: Investigating and Improving a COTS-Based Software Development Process, Proc. of the 22nd International Conference on Software Engineering, June 200, Limerick, Ireland, pp. 31-40.

[Morisio02a] Morisio, M. and Torchiano, M.: Definition and Classification of COTS: a Proposal. Proc. of the 1st International Conference on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, pp. 165-175.

[Morisio02b] Morisio, M., Ezran, M., and Tully, C.: Success and Failures in Software Reuse. IEEE Transactions on Software Engineering, 28 (4): 340-357, April 2002.

[Moynihan97] Moynihan, T.: How Experienced Project Managers Assess Risk. IEEE Software, 14 (3): 35-41, May/June 1997.

[NAICS04] NAICS Code, Available at: http://www.revenue.state.ne.us/tax/current/buscodes.pdf, 2004

[Ncube02] Ncube, C. and Dean, J. C.: The Limitation of Current Decision-Making Techniques in the Procurement of COTS Software Components. Proc. of the 1st International Conference on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, pp. 176-187.

[Netcraft05] Netcraft’s survey available at: http://news.netcraft.com/archives/2005/01/01/january_2005_web_server_survey.htm

[Oberndorf97] Oberndorf, T.: COTS and Open Systems - An Overview. 1997, available at: http://www.sei.cmu.edu/str/descriptions/cots.html#ndi

[Ochs01] Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N.: A Method for Efficient Measurement-based COTS Assessment and Selection - Method Description and Evaluation Results. Proc. of the 7th IEEE International Software Metrics Symposium, Apr. 2001, London, England, IEEE Press, pp. 285-297.

[OSS04] Open Source Initiative, 2004, available at: http://www.opensource.org/index.php

[Ould99] Ould, M. A.: Managing Software Quality and Business Risk. John Wiley & Sons, 1999.

[Paulson04] Paulson, J.W., Succi, G. and Eberlein, A.: An Empirical Study of Open-Source and Closed-Source Software Products. IEEE Transactions on Software Engineering, Vol. 30, No. 4, Apr. 2004, pp. 246-256.

[Parastoo04] Mohagheghi, P.: The Impact of Software Reuse and Incremental Development on the Quality of Large Systems. PhD thesis, ISBN: 82-471-6408-6.

[Pfleeger94] Pfleeger, S.: Experimental Design and Analysis in Software Engineering Part 1-5. ACM SIGSOFT Software Engineering Notes, 19 (4):16-20; 20 (1):22-26; 20 (2):14-16; 20 (3):13-15; and 20 (4):14-17, 1994-1995.

References

103

[Raymond99] Raymond, E. S. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O’Reilly, 1999.

[Robson02] Robson, C.: Real World Research: A Resource for Social Scientists and Practitioner-researchers (Regional Surveys of the World), 2nd edition, Blackwell Publishers, 2002.

[Ropponen00] Ropponen, J. and Lyytinen, K.: Components of Software Development Risk: How to Address Them? A Project Manager Survey. IEEE Transactions of Software Engineering, 26 (2): 98-112, Feb. 2000.

[Rose03] Rose, L. C.: Risk Management of COTS Based System Development. Component-based Software Quality, Springer LNCS 2693, 2003, pp. 352-373.

[Rothenberger03] Rothenberger, M. A., Dooley, K. J. and Kulkarni, U. R.: Strategies for Software Reuse: A Principal Component Analysis of Reuse Practices. IEEE Transactions on Software Engineering, 29 (9): 825-837, Sept. 2003.

[Royce87] Royce W. W.: Managing the Development of Large Software Systems: Concepts and Techniques. Proc. of the 9th International Conference on Software Engineering, Mar. 1987, Monterey, California, USA, pp. 328-338.

[Ruffin04] Ruffin, M. and Ebert, C.: Using Open Source Software in Product Development: A Primer. IEEE Software, 21 (1): 82-86, January/February 2004.

[Saaty90] Saaty, T. L.: How to make a decision: The Analytic Hierarchy Process (AHP). European Journal of Operational Research, 48 (1): 9-26, Jan. 1990.

[Sai03] Sai, V.: COTS Acquisition Evaluation Process: The Preacher’s Practice. Proc. of the 2nd International Conference on COTS-Based Software Systems (ICCBSS 2003), Feb. 2003, Ottawa, Canada, Springer LNCS Vol. 2580, pp. 196-206.

[Seaman99] Seaman, C.B.: Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering, 25 (4): 557-572, Apr. 1999.

[Sese04] SESE tool, available at: http://sese.simula.no, 2004. [Sherer95] Sherer, S. A.: The Three dimensions of Software Risk: Technical,

Organizational, and Environmental. Proc. of the 28th Hawaii International Conference on System Sciences, Maui, Hawaii, January, Jan. 1995, pp. 369-378.

[Sommerville04] Sommerville, I.: Software Engineering. 7th edition, Addison-Wesley, 2004.

[Spc01] Software Productivity Consortium: Phased-Integrated COTS Approach. Technical Report SPC-2001006-CMC, available at: http://www.software.org/pub/pic, 2001.

[SPIKE02] SPIKE project, available at: http://www.idi.ntnu.no/grupper/su/spike.html, 2002. [SPSS01] SPSS Categories 11.0, available at www.spss-sa.com, 2001. [SSB02] Norwegian Census Bureau (SSB), Oslo, ICT company data 2002,

http://www.ssb.no/emner/10 /03/ikt/. [Stanish95] The Stanish Group: http://www.stanishgroup.com/, 1995.

References

104

[Stewart05] Stewart, K. J., Ammeter, A. P., and Maruping, L. M.: Preliminary Analysis of the Influences of Licensing and Organizational Sponsorship on Success in Open Source Projects. Proc. of the 38th Hawaii International Conference on System Sciences, Jan. 2005, Hawaii, USA, pp. 197c.

[Strauss98] Strauss, A. and Corbin, J. M.: Basics of Qualitative Research: Grounded Theory Procedures and Techniques. 2nd edition, Sage Publications, 1998.

[Szyperski02] Szyperski, C., Gruntz, D., and Murer, S.: Component Software, Beyond Object-Oriented Programming. Addison Wesley, 2nd edition, 2002.

[Taylor84] Taylor, S. J. and Bogdan, R.: Introduction to Qualitative Research Methods. John Wiley&Sons, 1984.

[Torchiano04] Torchiano, M. and Morisio, M.: Overlooked Facts on COTS-Based Development. IEEE Software, 21 (2): 88-93, Mar./Apr. 2004.

[Tran97a] Tran, V., Liu, D. B., and Hummel, B.: Component-Based Systems Development: Challenges and Lessons Learned. Proc. of the 8th IEEE International Workshop on Software Technology and Engineering Practice, Jul. 1997, London, UK, pp. 452-462.

[Tran97b] Tran, V. Liu, D. B.: A Risk-Mitigating Model For the Development of Reliable and Maintainable Large-Scale Commercial-Off-the-Shelf Integrated Software Systems, Proc. of the Annual Reliability and Maintainability Symposium, Philadelphia, PA, USA, Jan. 1997, pp. 361-387.

[Tran99] Tran, V.N. and Liu, D. B.: Application of CBSE to Projects with Evolving Requirements- A Lesson-learned. Proc. of the 6th Asia-Pacific Software Engineering Conference (APSEC’ 99), Dec. 1999, Takamatsu, Japan, pp. 28-37.

[Verner05] Verner, J. M. and Evanco, W. M.: In-House Software Development: What Project Management Practices Lead to Success? IEEE Software, 22 (1): 86-93, Jan./Feb. 2005.

[Vigder96] Vigder, M. Gentleman, M. and Dean, J.: COTS Software Integration: State of the Art. Technical Report NRC No. 39190, 1996. [Vigder97] Vidger, M. and Dean, J.: An Architectural Approach to Building Systems

from COTS Software Components. Proc. of the 1997 Center for Advanced Studies Conference (CASCON 97), Toronto, Ontario, Nov. 1997, available at: http://seg.iit.nrc.ca/English/abstracts/NRC40221abs.html

[Vitharana03] Vitharana, P.: Risks and Challenges of Component-Based Software Development. Communications of the ACM, 46 (8): 67-72, Aug. 2003.

[V-Model05] German V-model: http://www.v-modell.iabg.de/#AU250, 2005. [Voas98a] Voas, J.M.: Certifying Off-the-Shelf Software Components. IEEE

Computer, 31 (6): 53-59, June 1998, pp. [Voas98b] Voas, J.M.: The Challenges of Using COTS Software in Component-Based

Development. IEEE Computer, 31 (6): 44-45, June 1998. [Voas98c] Voas, J.: COTS Software – the Economical Choice? IEEE Software, 15 (2):

16-19, March/April 1998.

References

105

[Wallace04] Wallace, L., Keil, M.: Software Project Risks and Their Effect on Outcomes, Communications of the ACM, 47 (4): 68-73, April 2004.

[Wohlin00] Wohlin, C., Runeseon, P., M. Höst, Ohlsson, M.C., Regnell, B., and Wesslén, A.: Experimentation in Software Engineering. Kluwer Academic Publications, 2000.

[Yin03] Yin, R.K.: Case Study Research, Design and Methods. Sage Publications, 2003.

[Zahran98] Zahran, S.: Software Process Improvement-Practical Guidelines for Business Success. Addison-Wesley, 1998.

[Zelkowitz98] Zelkowitz, M. V. and Wallace, D. R: Experimental Models for Validating Technology. IEEE Computer, 31 (5): 23-31, May 1998.

References

106

Appendix A

107

Appendix A: Selected papers

This chapter contains the papers in the order given in Section 1.5.

P1. Developer Attitude to Component Reuse

A Study of Developer Attitude to Component Reuse in Three IT Companies

Jingyue Li1, Reidar Conradi1,3, Parastoo Mohagheghi1,2,3, Odd Are Sæhle1, Øivind

Wang1, Erlend Naalsund1, and Ole Anders Walseth1 1 Department of Computer and Information Science,

Norwegian University of Science and Technology (NTNU), NO-7491 Trondheim, Norway

{jingyue, conradi}@idi.ntnu.no 2 Ericsson Norway-Grimstad, Postuttak, NO-4898 Grimstad, Norway

{parastoo.mohagheghi}@ericsson.com 3 Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysker, Norway

Abstract. The paper describes an empirical study to investigate the state of practice and challenges concerning some key factors in reusing of in-house built components. It also studies the relationship between the companies’ reuse level and these factors. We have collected research questions and hypotheses from a literature review and designed a questionnaire. 26 developers from three Norwegian companies filled in the questionnaire based on their experience and attitudes to component reuse and component-based development. Most component-based software engineering articles deal with COTS components, while components in our study are in-house built. The results show that challenges are the same in component related requirements (re)negotiation, component documentation and quality attributes specification. The results also show that informal communications between developers are very helpful to supplement the limitation of component documentation, and therefore should be given more attention. The results confirm that component repositories are not a key factor to successful component reuse.

Appendix A

108

1. Introduction Systematic reuse is generally recognized as a key technology for improving software productivity and quality [17]. With the maturity of component technologies, more and more companies have reused their software (program) in the form of components. Component reuse consists of two separate but related processes. The first deals with analysis of the application domain and development of domain-related components, i.e. development-for-reuse. The second process is concerned with assembling software system from prefabricated components, i.e. development-with-reuse. These two processes are tightly related, especially in reusing in-house built components. The number of components and the ratio of reused components to total components will determine the reuse benefits (e.g. improved productivity and quality) [11] [23].

To investigate the current state of practice and challenges for development-with- reuse in the IT industry, and to investigate the relationship between companies’ reuse level and some key factors in reusing of in-house components, an empirical study was performed as part of two Norwegian R&D projects. These projects were SPIKE (Software Process Improvement based on Knowledge and Experience) [29] and INCO (INcremental and COmponent-based development) [30]. From the literature review, we defined several research questions and hypotheses. A questionnaire was designed to investigate these questions. Developers from three Norwegian IT companies filled in the questionnaire based on their experience and attitudes to component reuse.

From the results of the survey, we found some new challenges in component reuse and component-based development based on in-house built components. The results support some commonly held beliefs and contradict others.

As the sample size of current research is still small, this study cannot provide statistically significant tests on hypotheses, and is therefore a pre-study. Later studies will be undertaken with refined hypotheses and on a larger sample.

The reminder of the paper is structured as follows: Section 2 presents some general concepts. Section 3 describes the research approach. Section 4 presents the survey results. Section 5 gives a detail discussion on the survey results. Conclusion and future research are presented in section 6.

2 Component reuse and component-based development Software reuse can take many different forms, from ad-hoc to systematic [16]. In the broad definition of reuse, it includes reusing everything associated with software projects, such as procedures, knowledge, documentation, architecture, design and code. In our research, we focus on systematic reuse of software code. The code reuse literature has identified reuse practice and success factors through several case studies and surveys. A major reuse effort is the REBOOT (Reuse Based on Object-Oriented Techniques) consortium [25]. This effort was one of the early reuse programs that recognized the importance of not only the technical, but also the organizational aspects of reuse [18]. As more experience become available from industrial studies, non-technical factors, such as organization, processes, business drivers and human involvement, appeared to be at least as important as technological issues [15][19].

Following the success of the structured design and OO paradigms, component -based software development has emerged as the next revolution in software development [27].

Appendix A

109

More and more IT companies have started to reuse code by encapsulating it into components. Whitehead defines a component as: A software component is a separable piece of executable software, which makes sense as a unit, and can interoperate with other components, within some supporting environment. The component is accessible only via its interface and is capable of use ‘as-is’, after any necessary installation and configuration procedures have been carried out [28].

Component-based development is assumed to have many advantages. These include more effective management of complexity, reduced time to market, increased productivity, improved quality, a greater degree of consistency and a wider range of usability [4][13]. It also brings many challenges, because it involves various stakeholders and roles, such as component developers, application developers, and customers. Different stakeholders and roles have different concerns [3], and face different issues and risks [2] [27].

Component-based development differs from traditional development, where the usual approach is for stakeholders to agree upon a set of requirements and then build a system that satisfies these requirements from scratch. Component-based development builds application by reusing existing components. Available components may not be able to satisfy all the requirements. Therefore, component-based projects must have flexibility in requirements, and must be ready to (re)negotiate the requirements with the customer. Moreover, components are intended to be used ‘as-is’. If some additional functionality is required, ‘glue-code’ is needed to be built to meet the differences between the requirement and component functionality. Another important feature of component-based development is the strong focus on the quality attributes (such as reliability, performance, and security etc.) and related testing. A major effort has to be put into checking how components perform, how well they interact, and to make sure that they are indeed compatible. Components may be developed in-house, acquired as COTS (commercial-off-the-shelf) [3], or even as OSS (Open Source Software) [5]. Most current research on component-based software engineering focuses on COTS-based development. Because COTS users cannot access the source code and must rely on vendors to give technical support, COTS-based development is assumed to be more challenging. Therefore, there is little research on the challenges based on in-house built components. 3 Research approach The difference between development based on in-house built components and development based on COTS is that the former is related very tightly with development-for-reuse. Component reuse is generally an incremental procedure. The company will build some reusable components in the beginning. In case of successful reuse, more and more code will be encapsulated into reusable components. The more reusable components are developed, the more complex will the development process be, and more support is required from the organization [8]. Our motivation is to investigate the relationship between companies’ reuse level and some key factors in component-based development so that company with low reuse level can make necessary software process improvements when moving to a higher reuse level.

Appendix A

110

3.1 Research questions To reuse in-house components successfully, developers must follow three basic steps [19]:

- Formulate the requirements in a way that supports retrieval of potentially useful reusable components.

- Understand the retrieved components. - If the retrieved components are sufficiently ‘close’ to the needs at hand and are of

sufficient quality, then adapt them. From these steps, we selected several key factors. For step 1, we focus on the

efficiency of component related requirements (re)negotiation and the value of component repository. For step 2, we study how knowledge about components can be transferred from a component provider to a component user. For step 3, our study focuses on definition and reasoning of quality attributes of components.

There is little research on the need for requirements (re)negotiation when components are built in-house. People assume that owning source code of in-house built components allows them to do any changes to meet the customers’ requirements. However, components are intended to be used ‘as-is’, even it is built in-house. So, our first research question is:

RQ1. Does requirements (re)negotiation for in-house components really work as efficiently as people assume?

Crnkovic et al. have proposed that to successfully perform the component-based

requirements (re)negotiation, a vast number of possible component candidates must be available, as well as tools for finding them [9]. Companies with a higher reuse level usually have more component candidates, more experience, and better experience than companies with a lower reuse level. So, our second research question is:

RQ2. Does the efficiency of component related requirements (re)negotiation increase with more in-house built components available?

To investigate this question, we formalized a null hypothesis H01 and an alternative hypothesis HA1 as follows:

H01. There is no relationship between the companies’ reuse level and the efficiency of component related requirements (re)negotiation.

HA1. There is a positive relationship between the companies’ reuse level and the efficiency of component related requirements (re)negotiation.

Concerning a component repository, Frakes claimed that it should not be given much

attention, at least initially [12]. So, our third research question is: RQ3. Does the value of component repository increase with more reusable

components available? To investigate this opinion more deeply, a null hypothesis H02 and an alternative

hypothesis HA2 was proposed: H02. There is no relationship between the companies’ reuse level and the value of

component repository. HA2. There is a positive relationship between the companies’ reuse level and the

value of component repository.

Appendix A

111

A complete specification of a component should include its functional interface, quality characteristics, use cases, tests, etc. While current component-based technologies successfully manage functional interfaces, there is no satisfactory support for expressing quality parts of a component [9]. So, our fourth research question is:

RQ4. How can a component user acquire sufficient information about relevant components?

Berglund claimed that growing reusable software components will create a new

problem, i.e. the information-overload problem. Therefore, learning which component to use and how to use them become the central part of software development [1]. Companies with a higher reuse level usually have more reusable components than companies with lower reuse level. So, our fifth research question is:

RQ5. Does the difficulty of component documentation and component knowledge management increase with increasing reuse level?

To study this question, we formalize null hypothesis H03 and alternative hypothesis HA3:

H03. There is no relationship between the companies’ reuse level and developers’ satisfaction with component documentation.

HA3. There is a negative relationship between the companies’ reuse level and developer’ satisfaction with component documentation.

One key issue in component-based development is trust, i.e. we want to build

trustworthy systems out of parts for which we have only partial knowledge [7]. Current component technologies allow systems builders to plug components together, but contribute little to ensure how well they will play together or to fulfill certain quality properties. So, the sixth research question is:

RQ6. Do developers trust the quality specification of their in-house built components? If the answer is no, how can they solve this problem? 3.2 The questionnaire The questionnaire included five parts. The questions in the first part were used to investigate the reuse level of the companies. The definition of reuse level in this study is the number of reused components vs. the number of total components in the organization. The other four parts were organized based on the four key factors. Each question in the questionnaire was used to study one or more research questions. The details of questions are shown in the following Table 1. The correspondences between the questions in the questionnaire and research questions are shown in Table 2. To increase the reliability of our survey, the questionnaire also included the definition of concepts used in the questionnaire, and the questions about the respondents’ personal information. 3.3 Data collection The study was performed in three Norwegian IT companies. Data collection was carried out by NTNU PhD and MSc students. Mohagheghi, Naalsund, and Walseth performed the first survey in Ericsson in 2002. In 2003, Li, Sæhle and Wang performed the survey

Appendix A

112

reusing the core parts of the questionnaire in two other companies (i.e. EDB Business Consulting and Mogul Technology). We selected those three companies because they have experience on component reuse and would like to cooperate with NTNU in this research. The respondents are developers in these three companies. They answered the questionnaires separately. The questionnaires were filled in either by hand or electronically (as a Word file). The MSc students provided support with possible problems in answering the questionnaire.

Table 1. Questions in the questionnaire Reuse level Q1. What is the reuse level in your organization? Q2. To what extend do you feel affected by reuse in your work? Component related requirements (re)negotiation Q3. Are requirements often changed/ (re)negotiated in typical develop projects? Q4. Are requirements usually flexible in typical projects? Q5. Do the component related requirements (re)negotiation processes work

efficiently in typical projects? Value of component repository Q6. Would the construction of a reuse repository be worthwhile? Component understanding Q7. Do you know the architecture of the components well? Q8. Do you know the interface of the components well? Q9. Do you know the design rules of the components well?

Q10a. Is the existing design/code of reusable components sufficiently documented?

Q10b. If the answer of Q10a is ‘sometimes’ or ‘no’, is this a problem? Q10c. If the answer of Q10a is ‘sometimes’ or ‘no’, what are the problems with

the documentation? Q10d. If the answer of Q10a is ‘sometimes’ or ‘no’, how would you prefer the

documentation? Q10e. What is your main source of information about reusable components

during implementation? Q10f. How do you decide whether to reuse a component ‘as-is’, ‘reuse with

modification’ or ‘make a new one from scratch’? Quality attributes specification of components Q11. Are specifications for components’ quality attributes well defined? Q12. Do you test components after modification for their quality attributes

before integrating them with other components?

Table 2. Correspondence between Questions in the questionnaire and Research Questions

RQ1 RQ2 RQ3 RQ4 RQ5 RQ6 Q1-Q2 X X X Q3-Q5 X X Q6 X

Q7-Q10f X X Q11-Q12 X

Appendix A

113

Below, we briefly characterize these three companies and respondents. 3.3.1 Companies Ericsson Norway-Grimstad started a development project five years ago and has successfully developed two large-scale telecommunication systems based on the same architecture and many reusable components in cooperation with other Ericsson organization. Their two main applications share more than 60% of ca. 1M lines of code [22].

EDB Business Consulting in Trondheim (now Fundator) is an IT-consultant firm which helps its customers to utilize new technology. It started to build reusable components from 2001. They have built some reusable components based on the Microsoft .Net in their eCportal framework (i.e. a web-application framework) 1.0 & 2.0. These components have been successfully reused in their new e-commence applications.

Mogul Technology (now Kantega) in Trondheim has large customers in the Norwegian finance- and bank sector. The main responsibilities are development and maintenance of the customers’ Internet bank application. The application was originally a monolithic system. After several years in production, the customer itself took initiative to reengineer the old system to a component-based solution based on EJB component model in 2002. At the time of the survey, some components have been created and reused in their new Internet bank system. 3.3.2 Respondents There were 200 developers at Ericsson in Grimstad, where we sent out 10 questionnaires to developers in one development team and got 9 filled-in questionnaires back. There were 20 developers in EDB Business Consulting in Trondheim, and we gathered 10 filled-in questionnaires back out of 10. We distributed 10 questionnaires to 22 developers at Mogul Technology in Trondheim and got 7 back. Those developers were selected because their work was related to component reuse, and they could assign effort to participate in the survey. This is non-probability sampling, which is based on convenience. Most participants in this survey have a solid IT background. 6 of 26 respondents have MSc degree in computer science and all others have a bachelor degree in computer science or telecommunication. More that 80% of them have more than 5 years of programming experience. The details of their position and their experience in the current organization are summarized in the following Table 3. 4 Survey Results In this section, we summarize the result of the survey. All the following statistical analyses are based on valid answers, i.e. Don’t Know answers are excluded. The statistical analysis tool we used is SPSS Version 11.0. 4.1 Different reuse level in these companies

Appendix A

114

First, we wanted to know the reuse level in those three companies. Q1 and Q2 were asked to get the answer based on developers’ subjective opinion on this issue. The result of Q1 is shown in Fig. 1, and the result of Q2 is shown in Fig. 2. From Fig. 1 and Fig. 2, we can see that most developers in Ericsson think that the reuse level in their company is very high or high. Most developers in EDB regard the reuse level in their company is high or medium. Most developers in Mogul think that the reuse level in their company is medium or little.

Table 3. Background of the respondents Company Position and working experience in the

organization Ericsson Norway-Grimstad 2 system architects, 7 designers.

1 person has 13 years of experience 7 persons have experience from 2-5 years, 1 person has 9 months of experience,

EDB Business Consulting in Trondheim

1 project manager, 5 developers and 4 IT consultants. 1 person has 17 years of experience 8 persons have experience from 3-8 years, 1 person has 2 years of experience.

Mogul Technology in Trondheim

6 developers and 1 maintainer (previous developer). 1 person has 10 years of experience, 6 persons have experience from 2-5 years.

4.2 Component related requirements (re)negotiation Questions Q3-Q5 were asked to investigate RQ1.We can see that no respondents to Q3 believe that the requirements were never changed/ (re)negotiated. Only 8% of respondents to Q4 think the requirements of their typical project are not flexible. However, only 48% of respondents to Q5 think component related requirements (re)negotiation works well. To study RQ2 and test the hypothesis H01, the correlation between the reuse level and response to Q5 is studied. We assign ordinal values to Ericsson, EDB and Mogul to represent their different reuse levels based on the responses to Q1 and Q2 (Ericsson = 3, EDB = 2, Mogul = 1). We also assign ordinal value to the answer of Q5 (Yes = 3, Sometimes = 2, No =1). The result of correlation between them using one-tailed Spearman Rank Correlation Coefficient analysis is .112, and the significance is .306. This shows that there is no significant statistical relationship between the reuse level and the efficiency of component related requirements (re)negotiation. 4.3 Value of component repository From the answer of Q6, we found that 71% of respondents in Mogul and EDB regard constructing a component repository as worthwhile, against 57% in Ericsson. To study RQ3 and test hypothesis H02, the relationship between the answer of Q6 and the reuse level is studied. We use the same ordinal number mapping as previously. The result of correlation between them using one-tailed Spearman Rank Correlation Coefficient

Appendix A

115

analysis is -.124, and significance is .297, which shows that there is no obvious relationship between them.

What i s t he r euse l evel i n your company?

0%

10%

20%

30%

40%

50%

60%

70%

80%

Ver y hi gh Hi gh Medi um Li t t l e Don' t knowReuse Level

Perc

enta

ge o

f an

swer

s

ERI CSSONEDB ASMOGUL

Fig. 1. Result of the question “What is the reuse level in your company?”

To what ext end do you f eel af f ect ed by r euse i n your wor k

0%

10%

20%

30%

40%

50%

60%

70%

80%

Ver y hi gh Hi gh Medi um Li t t l e Don' t knowEf f ect Level

perc

enta

ge o

f an

swer

s

ERI CSSONEDB ASMOGUL

Fig. 2. Result of the question “To what extend do you feel affected by reuse in your

work?” 4.4 Component understanding Questions Q7-Q10f were used to investigate RQ4. For Q7, Q8 and Q9, the results show that 67% of the respondents think the component structure is well understood, 61% say that the component interfaces are understood, and 63% regard the design rules of components are also well understood. But for the responses to question Q10a, no one thinks that the design/code of components is well documented, 73% think that they are sometimes well defined, and 27% believe that they are not well documented. Furthermore, the answers to questions Q10b and Q10c indicate that 86% believe that insufficient component documentation is a problem, e.g. documentation is not complete, not updated, and difficult to understand, etc. From responses to Q10d and Q10f, we can see that the preferable way of documentation is web pages. Some of the developers’ knowledge of how to use components comes from informal communication sources, for example, previous experience, suggestions from local experts, etc. To study RQ5 and test hypothesis H03, the association between reuse level and response to Q10a is

Appendix A

116

studied. We use the same ordinal number mapping as previously. The result of correlation between them using one-tailed Spearman Rank Correlation Coefficient analysis is -.469, and significance is .014, which shows that there is a weak negative relationship between them. It means that the higher the companies’ reuse level, the less satisfied a developer is with the component documentation. 4.5 Quality attributes of components Question Q11 and Q12 were used to investigate RQ6. From the responses to these questions, we see that 70% of the participants regard the design criteria for quality requirements are not well defined, and 87% will test the quality attributes of components after component modification, before integrating them into the system. 5 Discussions Based on the result of the survey, we discuss our research questions and hypotheses, and discuss the limitations and threats to validity. 5.1 Component related requirements (re)negotiation Much research focus on how to improve the efficiency of component related requirements (re)negotiation in COTS-based development [20] [24] [26]. The main reason is that people think the challenges in requirements (re)negotiation are due to the lack of access to source code, to timely vendor supports, or to the lack of engineering expertise to modify the integrated components [26]. In our case, the components are mostly built in-house. The above constrains on COTS components are not considered as challenges with built in-house components. From the responses to question Q3-Q5, we found that although 92% think that requirements of their typical projects are flexible, less than half think the component related requirements (re)negotiation in their typical projects works well.

Since components are intended to be used ‘as-is’, it is possible that an in-house reusable component meeting all the requirements will not be found. So, even though the components are built in-house, requirements (re)negotiation is necessary. For research question RQ1, we do not want to claim that the requirements (re)negotiation based on in-house components is more difficult than COTS-based components. We just want to emphasize that requirements (re)negotiation based on in-house components is also important but not efficient.

From the test result on H01, we cannot find a statistically significant relationship between the reuse level and the efficiency of component related requirements (re)negotiation. So, we cannot reject null hypothesis H01. Our conclusion to RQ2 is that when IT companies change from a low reuse level to a higher reuse level, they probably cannot expect that component-based requirements (re)negotiation becomes easier and more efficient.

5.2 Component repository

Appendix A

117

Some researchers have claimed that repository is important, but not sufficient for successful reuse [18] [21]. Our data confirms that developers are positive, but not strongly positive to the value of component repository. So, this result gives future support to the previous conclusion.

From the test result on H02, we can see that there is no statistically significant relationship between developers’ positive attitude to a component repository and reuse level. So, we cannot reject null hypothesis H02. Our conclusion to RQ3 is that companies are not expected to invest in a repository to increase reuse.

5.3 Component understanding Transferring component knowledge from the component developer to the component user is critical for successful component reuse. The answers of Q7-Q9 show that most developers understand the components in detail. However, the answers of Q10a-Q10c show that no one believes that the components are well documented because the documents are either incomplete or not updated. So, our question is “How can developers still understand the components without good documentation?” From the answers to question Q10e and Q10f, we found that most developers got the knowledge of components from informal channels, such as previous experience and local experts. The most important feature of a component is the separation of its interface from its implementation. The component implementation is only visible through its interface. Moreover, current component documentation technologies cannot describe all the information the developer required, such as performance, reliability, and security etc. Therefore, informal knowledge transfer should be considered to supplement the insufficiency of formal component documentation and specification. This point was shown in other empirical studies as well [6] [10]. For research question RQ4, we found that informal knowledge transfer is especially important in the component reuse. One possible solution is to have special interest groups or mailing lists for a components (or group of similar components) so that component users can share knowledge and experience of component usage.

From the test result on H03, we found a weak negative relationship between reuse level and developers’ satisfaction with the documentation. We reject the null hypothesis H03 and accept the alternative hypothesis HA3, It means the higher the companies’ reuse level, the less satisfied a developer is with components’ documentation. Marcus et al. concluded that combine reuse education and training provided for staff with other reuse activity can lead to all the success of reuse [18]. Our conclusion to RQ5 implies that when a company moves from a low reuse level to high level, more effort should be spent on the component documentation and component knowledge management.

5.4 Quality attributes of components Component-based development relies on the availability of high quality components to fill roles in a new intended system. When components are created or changed, we must ensure that they do not only fulfill the functional requirements, but also quality requirements. For research question RQ6, we found that most developers are not satisfied with the specification of components’ quality attributes and therefore cannot use this information. Therefore, how can we model quality properties of both

Appendix A

118

components and systems, and reason about them, particularly in the early stage of system development is still a key challenge in component-based development. 5.5 Threats to validity We now discuss the possible validity threats in this study. We use the definition given by Judd et al. [14].

Construct validity In our case, the main construct issue applies to the variables chosen to characterize the data set. The independent variable, i.e. reuse level, is the most sensible one. The results of questions Q1 and Q2 give a qualitative and consistent value on this variable.

Internal validity A major threat to this validity is that we have not assessed the reliability of our measurement. Most variables are measured on a subjective ordinal scale. An important issue for future studies is to ensure the reliability and validity of all measurement. In this survey, we gave clearly specified concepts in the questionnaire and provided support to possible misunderstanding. These methods partly increased the reliability.

External validity The small sample size and lack of randomness in the choice of companies and respondents are threats to external validity. In general, most empirical studies in industry suffer from non-representative participation, since companies that voluntarily engage in systematic improvement activities must be assumed to be better-than-average.

Conclusion validity This study is still a pre-study. Future studies will be implemented to give more statistically significant results.

6 Conclusion and future work This study has investigated challenges related to four key factors for development based on in-house components, especially in development-with-reuse. These factors are component related requirements (re)negotiation, component repository, component understanding and components’ quality attribute specification. Another contribution is that we compared three IT companies with different reuse levels to study the possible trend and challenges in these factors when more and more code will be encapsulated as reusable components inside a company.

- For component-based requirements (re)negotiation, the results of research questions RQ1 and RQ2 show that requirements (re)negotiation for in-house built components is important but not efficient. The efficiency will probably not increase with higher reuse level.

- For the component repository, the results of research question RQ3 confirm that a component repository is not a key factor for successful reuse. Furthermore, the potential value of a component repository will probably not increase with higher reuse levels.

- For component understanding, the results of research questions RQ4 and RQ5 show that most developers are not satisfied with the component documentation, and developers’ satisfaction with component documentation will probably decrease with higher reuse level. The results also show that informal

Appendix A

119

communication channels, which developers can get necessary information about the components through, should be given more attention

- For components’ quality attribute specification, the result of research question RQ6 shows that developers still need to spend much effort on testing, as they cannot get relevant information from component specifications.

The main limitation of our survey is that it depends on the subjective attitudes of

developers, and with few companies and participants involved. Later studies are planned to be undertaken with more precise quantitative methods and on more companies with more distinct reuse levels. Case studies will also be undertaken to follow the change of companies from lower reuse level to higher reuse level to future investigate our research questions.

7 Acknowledgements This study is supported by the SPIKE and INCO projects. We thank the colleagues in these projects, and all the participants in the survey. References 1. Erik Berglund: Writing for Adaptable Documentation. Proceedings of IEEE

Professional Communication Society International Professional Communication Conference and Proceedings of the 18th ACM International Conference on Computer Documentation: Technology & Teamwork, Cambridge, Massachusetts, September (2000) 497– 508

2. Pearl Brereton: Component-Based System: A Classification of Issues. IEEE Computer, November (2000), 33(11): 54– 62

3. Alan W. Brown: The Current State of CBSE. IEEE Software, September/October (1998) 37– 46

4. Alan W. Brown: Large-Scale Component-Based Development. Prentice Hall, (2000) 5. Alan. W. Brown and Grady. Booch: Reusing Open-source Software and Practices:

The Impact of Open-source on Commercial Vendors. Proceedings: Seventh International Conference on Software Reuse, Lecture Notes in Computer Science, Vol. 2319. Springer, (2002) 123– 136.

6. Reidar Conradi, Tore Dybå: An Empirical Study on the Utility of Formal Routines to Transfer Knowledge and Experience. Proceedings of European Software Engineering Conference, Vienna, September (2001) 268– 276

7. Bill Councill and George T. Heineman: Component-Base Software Engineering and the Issue of Trust. Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, June (2000) 661– 664

8. Ivica Crnkovic and Magnus Larsson: A Case Study: Demands on Component-based Development. Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, June (2000) 21– 31 .

9. Ivica Crnkovic: Component-based Software Engineering - New Challenges in Software Development. Proceedings of 25th International Conference on Information Technology Interfaces, Cavtat, Croatia, June (2003) 9– 18

Appendix A

120

10. Torgeir Dingsøyr, Emil Røyrvik: An Empirical Study of an Informal Knowledge Repository in a Medium-Sized Software Consulting Company. Proceedings of 25th International Conference on Software Engineering, Portland, Oregon, USA, May (2003) 84– 92

11. W. B. Frakes: An Empirical Framework for Software Reuse Research. Proceedings of the Third Annual Reuse Workshop, Syracuse University, Syracuse, N.Y. (1990)

12. W. B. Frakes, C.J. Fox: Sixteen Questions about Software Reuse. Communication of the ACM, June (1995), 38(6): 75– 87

13. Ivar, Jacobson, Martin Griss, Patrick Jonsson: Software Reuse-Architecture, Process and Organization for Business Success. Addison Wesley Professional, (1997)

14. C.M. Judd, E.R. Smith, L.H. Kidder: Research Methods in Social Relations. Sixth edition, Holt Rinehart and Winston, (1991)

15. Y. Kim and E.A. Stohr: Software Reuse: Survey and Research Directions. Journal of Management Information System, (1998), 14(4): 113– 147

16. C. Kruger: Software Reuse. ACM Computing Surveys, (1992), 24(2): 131– 183 17. N. Y. Lee, C. R. Litecky: An Empirical Study on Software Reuse with Special

Attention to Ada. IEEE Transactions on Software Engineering, September (1997), 23(9): 537– 549

18. Marcus A. Rothenberger, Kevin J. Dooley and Uday R. Kulkarni: Strategies for Software Reuse: A Principal Component Analysis of Reuse Practices. IEEE Transactions on Software Engineering, September (2003), 29(9): 825– 837

19. H. Mili, F. Mili, A. Mili: Reusing Software: Issues and Research Directions. IEEE Transactions on Software Engineering, June (1995), 21(6): 528– 561

20. M. Morisio, C.B. Seaman, A. T. Parra, V.R. Basili, S.E. Kraft, S.E. Condon: Investigating and Improving a COTS-Based Software Development Process. Proceeding of 22nd International Conference on Software Engineering, Limerick, Ireland, June (2000) 31– 40

21. Maurizio Morisio, Michel Ezran, Colin Tully: Success and Failure Factors in Software Reuse. IEEE Transactions on Software Engineering, April (2002), 28(4): 340– 357

22. Parastoo Mohagheghi and Reidar Conradi: Experiences with Certification of Reusable Components in the GSN Project in Ericsson, Norway. Proceedings of the 4th ICSE Workshop on Component-Based Software Engineering: Component Certification and System Prediction. Toronto, May (2001) 27– 31

23. Jeffrey S. Poulin: Measuring Software Reuse-Principles, Practices, and Economic Models. Addison-Wesley, (1997)

24. Vijay Sai: COTS Acquisition Evaluation Process: The Preacher’s Practice. Proceedings of 2nd International Conference on COTS-based software systems, Lecture Notes in Computer Science, Vol. 2580. Springer, 2003, Ottawa, Canada, February (2003) 196– 206

25. Guttorm Sindre, Reidar Conradi, and Even-Andre Karlsson: The REBOOT Approach to Software Reuse. Journal of System Software, (1995), 30(3): 201– 212

26. Vu N. Tran, Dar-Biau Liu: Application of CBSE to Projects with Evolving Requirements- A Lesson-learned. Proceeding of the 6th Asia-Pacific Software Engineering Conference (APSEC’ 99) Takamatsu, Japan, December (1999) 28– 37

Appendix A

121

27. Padmal Vitharana: Risks and Challenges of Component-based Software Development. Communications of the ACM, August (2003), 46(8): 67– 72

28. Katharine Whitehead: Component-Based Development: Principles and Planning for Business Systems. Addison-Wesley, (2002)

29. http://www.idi.ntnu.no/grupper/su/spike.html 30. http://www.ifi.uio.no/~isu/INCO/

Appendix A

122

Appendix A

123

P2. Variations in COTS-Based Development Process

An Empirical Study of Variations in COTS-based Software Development Processes in the Norwegian IT Industry

Jingyue Li 1, Finn Olav Bjørnson 1, Reidar Conradi 1,2, Vigdis By Kampenes 2

1 Dept. of Computer and Information Science Norwegian Univ. of Science and Technology

NO-7491 Trondheim, Norway {jingyue,bjornson,conradi}@idi.ntnu.no

2 Simula Research Laboratory P.O.BOX 134, NO-1325 Lysaker, Norway

[email protected]

Abstract. More and more software projects use Commercial-Off-The-Shelf (COTS) components. Although previous studies have proposed specific COTS-based development processes, there are few empirical studies that investigate how to use and customize COTS-based development processes for different project contexts. This paper describes an exploratory study of state-of-the-practice of COTS-based development processes. Sixteen software projects in the Norwegian IT companies have been studied by structured interviews. The results are that COTS-specific activities can be successfully incorporated in most traditional development processes (such as waterfall or prototyping), given proper guidelines to reduce risks and provide specific assistance. We have identified four COTS-specific activities – the make vs. buy decision, COTS component selection, learning and understanding COTS components, and COTS component integration – and one new role, that of a knowledge keeper. We have also found a special COTS component selection activity for unfamiliar components, combining Internet searches with hands-on trials. The process guidelines are expressed as scenarios, problems encountered, and examples of good practice. They can be used to customize the actual development processes, such as in which lifecycle phase to put the new activities into. Such customization crucially depends on the project context, such as previous familiarity with possible COTS components and flexibility of requirements. 1. Introduction COTS-based development has become more and more important in software and system development. Using COTS components promises faster time-to-market and increased productivity (Voas, 1998a). At the same time, COTS-based software introduces many risks. This means that unknown quality properties of the chosen COTS components can be harmful for the final product, and a COTS vendor may terminate the maintenance support of its COTS components (Voas, 1998b).

The use of COTS components introduces new system issues, which require revised software development processes. Although researchers and practitioners have been grappling with these new processes, most studies have been based on military or aerospace projects (Morisio et al., 2000; SEI, 2004), or similar large projects. In order to propose and design cost-effective COTS-based development processes, it is necessary

Appendix A

124

to empirically investigate how COTS-based projects were performed in different application domains, particularly in small or medium-sized companies.

This study has investigated the commonalities and differences between development processes in 16 finished COTS-based software projects in Norway. It also summarized process scenarios in successful vs. unsuccessful COTS-based projects. Eleven salient problems encountered and 12 examples of good practice are extracted. A process customization guideline is proposed to help software practitioners to integrate relevant COTS components activities successfully in different project contexts.

The remainder of the paper is structured as follows: Section 2 presents previous studies on COTS-based development processes. Section 3 describes the research approach. Contexts of selected projects are presented in Section 4. The results are presented in Section 5 and discussed in Section 6. Our conclusions and proposals for future research are presented in Section 7. 2 COTS-based software development processes 2.1 COTS component definition A pertinent question is “What do you mean by a COTS component?” There are many different definitions of COTS components (Basili and Boehm, 2001; Carney and Long, 2001). We have used the definition by Torchiano and Morisio (2004), where a COTS component should satisfy all the following requirements:

- It is either provided by some other organizations in the same company, or provided by external companies as a commercial product.

- It is integrated into the final delivered system. - It is not a commodity. It is not shipped with an operating system, not provided

with the development environment, and not generally included in any pre-existing platforms.

- It is not controllable by the user, in terms of provided features and their evolution.

- It is “black box” software. The source code is not available. The granularity of a COTS component can be different (Torchiano and Morisio,

2004). Some consider that COTS components could or should include very large software packages such as Microsoft Office. Others limit COTS components to GUI libraries. In this study, we focus on COTS components as software components. Such a component is a unit of composition, and must be specified so that it can be composed with other components and integrated into a system (product) in a predictable way (Crnkovic et al., 2002). That is, a component is an “Executable unit of independent production, acquisition, and deployment that can be composed into a functioning system.” This definition implies that we include not only components following COM, CORBA, and EJB standards, but also software libraries like those in C++ or Java. This definition is consistent with the scope used in the component marketplace (Componentsource, 2004).

Appendix A

125

2.2 COTS-based development processes Typically, a COTS-based process consists of four phases, comprising (Abts et al., 2000):

- COTS component assessment and selection - COTS component tailoring - COTS component integration - Maintenance of COTS and non-COTS parts of the system There is consensus that the use of a COTS component implies changes in the

software process (Brownsword et al., 2000). Most studies of a COTS-based development process focus on two dimensions: Process of the whole software development lifecycle and process of the specific phase, especially in COTS component selection and evaluation. 2.2.1 Process of the whole software development lifecycle. Boehm and Abts (1999) regard both the waterfall model and evolutionary development as unsuitable for COTS-based development because:

- In the sequential waterfall model, requirements are identified at an earlier stage and the COTS components are chosen at a later stage. This increases the likelihood of the COTS components not offering the required features.

- Evolutionary development assumes that additional features can be added if required. However, COTS components cannot be upgraded for one particular development team. The absence of source code bars the development team from adjusting the COTS components to their needs.

Based on the above arguments, Boehm and Abts (1999) proposed that development

models which explicitly take risk into account are more suitable for COTS-based development than the traditional waterfall or evolutionary approaches.

The Software Engineering Institute has a large ongoing effort to address the development of COTS-based systems, and they have developed the Evolutionary Process for Integrating COTS-based Systems (EPIC) (Albert and Brownsword, 2002). The proposed process integrates COTS-related roles and activities into a RUP process. The iterative and evolutionary nature inherent in this process allows developers to adjust the architecture and system design, as more knowledge is gained about the operations of the COTS components.

The National Aeronautic and Space Administration (NASA) has been developing IT systems with COTS components for many years. Their experience has been captured by Morisio et al. (2000). They have investigated the various processes that were used across 15 projects at NASA. Based on their insight, they have developed a COTS-based development process that was the most representative for the processes used in the projects. 2.2.2 COTS component selection and evaluation process. Based on case studies, researchers have proposed several COTS component selection processes and methods. One kind of process is based on direct assessment (Leung and Leung, 2002), such as

Appendix A

126

MBASE (Model Based Architecting and Software Engineering) (Boehm, 2000), OTSO (Off-the-Shelf-Option) (Kontio, 1996), CISD (COTS-based Integrated System Development) (Tran et al., 1997), PORE (Procurement-Oriented Requirement Engineering) (Maiden and Ncube, 1998), CAP (COTS Acquisition Process) (Ochs et al., 2001), IIDA (Infrastructure Incremental Development Approach)(Fox et al., 1997), CARE (COTS-Aware Requirements Engineering) (Chung and Cooper, 2002), RCPEP (Lawlis et al., 2001), and CRE (COTS-Based Requirements Engineering) (Alves and Finkelstein, 2003). In general, the direct assessment process includes three basic steps:

- Inspect all modules of each of the available COTS components to check whether they satisfy some or all of the functional requirements of the COTS-based system being developed.

- Check whether a COTS component also satisfies the non-functional requirements of the COTS-based system. Non-functional requirements may include properties such as the interoperability of the modules of the COTS product with other systems.

- Compare the COTS component candidates and select the most appropriate COTS component that satisfies both the functional and non-functional requirement of the COTS-based system.

Some of the direct assessment processes, such as OTSO (Kontio, 1996), CAP (Ochs

et al., 2001), and CISD (Tran et al., 1997), assume that the requirements are fixed and select the COTS components by comparing how well the COTS component candidates satisfy the requirements. A formal decision-making process is usually used to select the best COTS component (Ncube and Dean, 2002). The formal decision-making process usually includes three basic elements: selecting evaluation criteria (factors), collecting and assigning values to these criteria, and applying formal decision-making algorithms such as MAUT (MacCrimmon, 1973), MCDA (Morisio and Tsoukias, 1997), and AHP (Saaty, 1990). Other direct assessment processes, such as MBASE (Boehm, 2000), PORE (Maiden and Ncube, 1998), IIDA (Fox et al., 1997), CARE (Chung and Cooper, 2002), RCPEP (Lawlis et al., 2001), and CRE (Alves and Finkelstein, 2003) emphasize the trade-offs between the requirements and COTS component functionalities. These processes do not assume that the requirements are fixed and should not be changed. On the other hand, these processes propose that the requirements in COTS-based development should be flexible enough so that they can be negotiated or changed according to the available functionalities of the COTS components.

Another kind of process is based on a domain-model (Leung and Leung, 2002). It includes a set-up phase and a selection phase. In the set-up phase, the vendors need to map their COTS modules to those modules of the domain that they find are applicable. In the selection phase, the corresponding modules in a domain model are identified for each of the modules of the COTS-based system in question. Then, the COTS modules that claim to be applicable are identified by the mapping from the domain models to the COTS modules. After that, the non-functional properties of the identified COTS modules are assessed. In the end, the most appropriate COTS modules are selected with reference to all of the assessment results.

Appendix A

127

3 Research approach Most studies on COTS-based development processes are limited to proposals of new technology (e.g. revised processes) without empirical evidence, or to discovery of existing practice (e.g. component selection) from case studies. Empirical studies are few and mostly based on a small project sample or on large projects with rather similar contexts, such as from aerospace or defense. It is therefore difficult to identify the precise relationship between a project context and COTS-based development processes. It is consequently hard for project managers to customize their COTS-based processes according to their project context.

Our research was designed as an exploratory study, i.e. mostly a qualitative study. The focus is to investigate the variations in COTS-based development processes and summarize these variations by scenarios. The intent is to find out how to customize a COTS-based development process based on its project context. Our research focuses on the development phases, not on software maintenance and evolution.

3.1 Research questions 3.1.1 Research question RQ1. Boehm and Abts (1999) regard both the waterfall and evolutionary lifecycle process as unsuitable for COTS-based development. Most COTS-based projects must therefore adjust their development process to a risk-driven spiral process, instead of a waterfall and evolutionary one. However, there are still no conclusive empirical studies on this issue. So, our first research question is:

RQ1: What was the actual process used in the projects using COTS components?

3.1.2 Research question RQ2. Morisio et al. (2000) have proposed that COTS-based software processes differ considerably from “traditional” lifecycle processes, such as waterfall, or prototyping. Not modifying the traditional process can be a failure factor. They also pointed out that their proposed process has several variations for customization purposes. They also indicated that it is important to investigate under what conditions each variation is most suitable. However, there have been no further studies on this so far. Our second research question is then:

RQ2: What are the commonalities and possible variations in COTS-based development processes?

3.1.3 Research question RQ3. All previous studies on COTS-based development processes have mentioned the possible problems (or risks) in using COTS components (Boehm and Abts, 1999; Morisio et al., 2000; Albert and Brownsword, 2002). To successfully customize COTS-based processes, it is important to summarize and understand the experience from successful COTS-based projects vs. those that were not so successful. It is especially valuable to generalize the relationship between possible risks and the process adjustments (variations). It may help project managers to successfully adopt a COTS-based development process without taking unnecessary risks. So, our third research question is:

Appendix A

128

RQ3. What are process scenarios (variations) of the projects using COTS components successfully and not successfully? What are possible problems and good practices in different process scenarios? 3.2 Data collection method The initial plan for this study was to make a quantitative survey using a pre-stated hypothesis with standardized questions in a formal questionnaire. After two rounds of pre-tests on the questionnaire, we found that the questionnaire requires substantial work to give consistent and reliable results. We decided, as a first step, to go for an exploratory pre-study using semi-structured interview.

Face-to-face interviews offer the possibility of modifying one’s line of enquiry, following up interesting responses and investigating underlying motives in a way that questionnaire cannot (Robson, 2002). The choice of the semi-structured interview approach enabled more open research questions, not fixed hypotheses, but still relying on a rather detailed interview guide with both closed and open-ended questions. The advantages of open ended questions are that they allow us to go into depth and clear up misunderstandings, enable testing of limits of a respondent’s knowledge, allow us to make a truer assessment of what the respondents really believe, and opening for unexpected or unanticipated answers.

3.3 Interview guide We used a modified version of the original questionnaire to serve as an interview guide. The interview guide includes both closed and open-ended questions.

The closed questions were used to solicit information on project context. For example, one question used the application domains of the final system (product), which was based on the North American Industrial Classification System (NAICS) (NAICS, 2004).

The open-ended questions were used to gather information on actual COTS-based development processes. The questions covered the main lifecycle process, the process changes due to using COTS components, and the experience and lessons learned in the whole COTS-based development process. Some of the questions were arranged in an “if-then” structure that leads the interviewer along one of several paths depending on the answers to previous questions. For example, one question was to ask about whether they had changed or added some activities in their process because of using a COTS component. We then asked when and how did they make these changes if they added or changed some activities.

The interview guide was written in English. The reason was that we planned to extend this study to other countries later. On the first page, we gave the definitions of most concepts that were used in later questions, such as COTS component, project, and system (consisting of application, components, and addware/glueware). The rest of the interview guide had questions organized in three sections:

- Questions to characterize companies, projects, and COTS components. - Questions to characterize the individual respondents.

Appendix A

129

- Questions on COTS-based development processes, and covering mainly COTS component selection and integration.

To evaluate the construct validity of the interview guide, we added three meta-

questions following each question. These meta-questions asked the respondents, if the question was clear, if respondents needed clarification, and if respondents could give specific comments if the question was confusing. All this is a standard procedure from social science (Fowler, 2001).

Lastly, to ensure that a respondent had understood the COTS-relevant definitions correctly and could select a proper company project, we asked the respondent to give further examples of COTS components based on their understanding.

3.4 Sampling Interviews and data collection were conducted by Li and Bjørnson from Nov. 2003 to Jan. 2004. They contacted 34 IT companies, where most were involved in two Norwegian R&D Projects (INCO, 2000; SPIKE, 2002) and one EU R&D project (FAMILIES, 2003).

A pre-process was used to select relevant COTS-based projects. Contact persons (IT manager in most cases) in the above projects were first contacted by phone and email, using an introductory letter with our definition on COTS component and project (the same as definitions in the interview guide). They were asked to check if they have a relevant COTS-based project. During this pre-process, 17 companies (among 34) did not match our definition of a COTS-based project, and therefore could not join the study. Of these 17 “rejected” companies, 13 used only in-house built components and 3 used only open source components. There was one other company that could not participate because of confidentiality issues.

For the remaining 17 companies, 13 of these volunteered to join in the study. The contact persons in these 13 companies were asked to recommend persons with most experience on the relevant projects. In all, 16 persons were recommended from these 13 companies. For each recommended respondents, they were asked to select one relevant COTS-project. If a respondent claimed that he/she had experience from several COTS-based projects, the respondent was asked to select one that he/she was most familiar with. The whole process to find a suitable respondent and project in a prospective company, and later agree upon place and time for a physical interview took weeks.

Although the total sample was based on convenience, we ensured that we had included respondents from large, medium and small companies. 3.5 Data collection procedure The interview guide was sent to respondents a few days before the personal interview. In this way, the respondents were well-prepared and could have time to remember the details of the projects and find some relevant documents.

We offered the company respondent a compensation of 100 euro and/or a copy of a status report. Most companies did not want money, rather a status report. We emphasized that all answers and other information would be treated strictly as confidential.

Appendix A

130

Most of the answers and comments requested by the interview guide were filled-in on paper during the interview by the interviewer. Each interview was conducted by one of the two interviewers (none with two interviewers), took from 60 to 80 minutes, and was recorded.

3.6 Data analysis The first step in data analysis was to listen to the tape, and supplement and transcribe the interview “as-is” into a field note. After that, we used different analysis procedures according to the purpose of different research questions.

The purpose of the first research question RQ1 is to investigate the actual development processes used in the project. The data was analysed by the constant comparison method (Strauss and Corbin, 1998). As all respondents had a clear definition on the main process used in their project. These processes were early grouped into traditional process (waterfall, prototyping, or incremental) or a totally new process.

The purpose of the second research question RQ2 is to see the similarities and differences between process changes in the studied projects. The main analysis method used is cross-case analysis (Strauss and Corbin, 1998), i.e. treating each project as a separate “case”. The field notes for the first two projects were first reviewed. For each of these two projects, a list was compiled with short phases that describe the process changes in each project (i.e. new or changed activities, when and how these changes were performed). Then these two lists were compared to determine the similarities and differences. The next step was to list, in the form of propositions, conclusions one would draw if these two projects were the only two in the data set. Each proposition had associated process changes that supported it. After analysing the first two projects in this way, the third project was examined, and a list of its process changes was compiled. Then it was determined whether this third project supported or refuted any of the propositions formulated from the first two. If a proposition was supported, this third project was added to its list of supporting evidence. If it contradicted a proposition, then either the proposition was modified or the project was noted as refuting that proposition. Any additional propositions suggested by the third inspection were added to the list. This process was repeated for each project. The end result was a list of propositions, each with a set of supporting and refuting evidence (projects).

The purpose of the third research question RQ3 is to summarize the process scenarios of the successful and unsuccessful projects. We first grouped projects into two groups based on the positive or negative effect of the COTS components regarding time-to-market and general quality of the system. In each group, we divided projects into sub-groups based on their main actual processes used. For each sub-group, we performed a similar cross-case analysis as for research question RQ2. The difference is that we included more information for analysis. For each project, a list of short phases was compiled that describe not only the process changes in each project, i.e. new or changed activities, but also when and how these changes were performed. We also recorded project contexts, i.e. the developers’ experience with COTS components. To summarize the lesson learned in different process scenarios, we selected the main problems met and examples of good practice performed in each process scenario. The problems and examples of good practice that occurred in more than one project were grouped into one item. Some problems and examples of good practice, which occurred

Appendix A

131

in only one project, were also listed because they were regarded as the main factor that decided the success and failure of the project.

4 Collected samples 4.1 Companies As mentioned, we interviewed 16 projects in 13 companies. All the companies are Norwegian IT companies. Nine of these 13 are stand-alone companies, with staff size between 5 and 500. The other companies are subsidiaries, with local staff size ranging from 50 to 320. Six companies are IT consultancies, 5 are software vendors, and 2 are telecom companies. Five of these companies are publicly traded companies, the others are privately held. Some company background information is listed in Table 1.

Table 1. Background information on the 13 IT companies (called C1 to C13) Company ID

Staff Size Type Ownership Main business area

C1 42 Stand-alone Privately held IT Consulting C2 60 Subsidiary Privately traded IT Consulting C3 36 Stand-alone Privately held IT Consulting C4 17 Stand-alone Privately held IT Consulting C5 500 Stand-alone Publicly traded IT Consulting C6 320 Subsidiary Publicly traded Telecom Industry C7 120 Stand-alone Publicly traded Software Vendor C8 100 Stand-alone Privately held Software Vendor C9 130 Stand-alone Privately held IT Consulting C10 275 Subsidiary Publicly traded Telecom Industry C11 50 Subsidiary Publicly traded Software Vendor C12 5 Stand-alone Privately held Software Vendor C13 80 Stand-alone Privately traded Software Vendor

4.2 Respondents We had one respondent from each project. Three respondents are IT managers, 4 are project managers, 5 are software architects, 3 are software developers, and 1 is a software development researcher. Nine of them have more than 10 years of software development experience, and 12 of them have more than 4 years of working experience with COTS-based development. Eight of them have a master degree, and the rest have a bachelor’s degree. Twelve of them have a principal degree in informatics or computer science. 4.3 Projects The company projects were selected based on two criteria:

- The project should use one or more COTS components - The project should be a finished project and possibly with maintenance, and


Appendix A

132

Generally, we selected only one project from a company. However, we selected two projects in three companies because these projects differed largely in project members, development process, and COTS components used.

Background information includes effort spent, development language, and application domain. Table 2 lists the basic information on the 16 projects.

Table 2. Background information on the 16 projects (called P1 to P16)

Company ID

Project ID

Effort (person-hours)

Main development language

Application domain

C1 P1 7,000 Java Finance C1 P2 30,000 Power Builder Finance C2 P3 700 C, Java Scientific C2 P4 42,000 C Scientific C3 P5 16,000 C++ Internet Service

provider C3 P6 23,000 C++ Telecom. C4 P7 2,400 Java Agriculture C5 P8 3,000 Visual Basic El. Power C6 P9 58,000 C++, Java Telecom C7 P10 63,000 C++, VB Whole trade C8 P11 8,000 C++ El. Power C9 P12 10,000 C++ Construction C10 P13 128,000 C++ Telecom C11 P14 120,000 C++ Transport C12 P15 92,000 Ada, Java Finance C13 P16 15,000 C++ Scientific

4.4 COTS components Some projects used one or two COTS components, others used more. Because of time limitations, we asked the respondents to select no more than three typical COTS components which were used in their projects. Totally, we gathered information about 30 different COTS components in 16 projects. Some typical COTS components are listed in Table 3.

Table 3. Background information on some COTS components COTS component name Functionality CTI Computer Telephony Integration Sheridan GUI Intelligent Report Report generator Active Reports Report preview and printing Open MP Open parallel processing IMAPP Image processing SearchEngine Searching the context in the website ComWork Workflow management XML lite XML parser

Appendix A

133

The list is not exhaustive, and is only used to indicate the kinds of selected COTS components. Some of the COTS components are based on standards like COM, CORBA, and EJB, and some of them are C++ or Java libraries. All the COTS components follow our previous definition, i.e., they were integrated into the final product and were delivered together with the final system to the customer. 5 Results 5.1 Overall process used in COTS-based software development Research question RQ1 is to identify the actual processes used in projects with COTS components. To study this issue, we asked two open-ended questions: we first ask the respondents to describe the development processes they used. We also asked if they selected the actual development processes because of using COTS components.

Results of the first question show that the main development processes of the studied projects can be grouped into three categories: pure waterfall, waterfall mixed with prototyping, and incremental mixed with prototyping. Five projects used waterfall. One used waterfall mixed with little prototyping (a small prototype was used for discussing requirements of the GUI part of the system). Ten projects used incremental mixed with some prototyping.

Results of the second question show that none of the respondent claimed that they selected the main development process based on whether or not they intended to use COTS components. That is, they decided the main process before they started to think about using COTS components or not. They just added or changed some activities and roles in their traditional process to reflect the use of COTS component.

Therefore, the answer to research question RQ1: The so-called COTS-based development process is the customization of the traditional development process due to the use of COTS components. 5.2 Commonalities and variances in COTS-based software development process Research question RQ2 investigates the commonalities and possible variations in the development processes. To study the common changes in the develop processes, we asked if there are any new or changed activities, role and responsibilities in the project due to use of COTS components. The detailed answers are listed in Appendix A. The commonalities are that new activities were added and possible one new role was added. The added new activities include:

- Make vs. buy decision: In COTS-based development, the first extra activity in all the studied projects was to make the make vs. buy decision. Non-technical issues, such as costs, licensing, and market trends, as well as technical issues may dominate this stage.

- COTS component selection: After the respondents decided to use COTS component; the following extra activity in all the 16 projects was COTS component selection. However, different projects used different selection processes and criteria in their COTS component selection.

Appendix A

134

- Learn and understand the COTS components: Another new activity is to learn and understand the COTS components. Developers needed to spend time to study several possible candidates briefly before the final selection. After they have selected the COTS components, they needed to spend time to learn how to use the selected COTS components. Because the source code of a COTS component is often not available, it is difficult for developers to profoundly understand a COTS component.

- Build glueware and addware: A future step after COTS component selection is integration. All the projects built more or less glueware (the code to integrate the COTS components) and/or addware (the code to complete the required functionality of COTS components).

The possible added new role is: a COTS component knowledge keeper. As

mentioned above, it was difficult to really understand the COTS components. In 10 projects, one or more project members had some previous experience with selection and integration of the actual COTS components. They provided suggestions on COTS component selection and integration. Although some of them were not dedicated to work as a COTS component knowledge keeper, their previous experience improved the speed and quality of COTS component selection and integration.

To investigate the variations in the processes, we transcribed respondents’ description on the above new activities and compared when, how, and why these activities are performed. The detailed information on when and how the COTS components were selected is listed in Appendix B. The results show that the main variations in the customization of main development processes deal with the following activities:

- The phase to make the make vs. buy decision and the COTS component

selection: Morisio et al. (2000) proposed a two-phase make vs. buy decision. The first decision is in the requirement phase and the second in the design phase. Our results indicate that only 2 of the 5 waterfall projects made the make vs. buy decision and COTS component selection in the requirement phase. If the project members had no relevant experience with specific COTS component, it was very hard to make that choice in the requirement phase. Two waterfall process projects managed to make the make vs. buy decision and COTS component selection in the requirement phase, because they were already familiar with the possible COTS components.

- The COTS component selection and evaluation process: The selection process in our studied projects can be summarized in two categories: • Familiarity-based selection process (SP_fam): The previous familiarity

came either from the project members in the projects, i.e. the COTS component knowledge keeper, or from a colleague outside the project. If the project members had enough previous experience with the candidate COTS components, this experience will be the key factor in the COTS component selection. In all, 16 of the 30 given COTS components were selected based on the suggestion of the COTS component knowledge keeper inside the projects. Four COTS components were selected based on recommendations from colleagues outside the project or organization.

Appendix A

135

• Internet search, trial-based selection process (SP_unfam): This kind of selection was used when there was no previous experience available. This selection was more complex than SP_fam, and involved three steps: Step 1: First, developers used a search engine to browse the Internet, using keywords to express decided functionality. This usually gave them a handful of candidates. Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to 3 of them very quickly based on some key issues, such as licence issues, cost, and vendor reputation. Step 3: After a small number of candidate COTS components had been decided, they downloaded a demo version of these from the web and tested the COTS components.

This kind of selection is similar to some proposed processes, such as OTSO (Kontio,

1996), CAP (Ochs et al., 2001), and CISD (Tran et al., 1997). The main difference is that the actual process was much faster and no decision-making algorithms have been used. In addition, developers knew that it was impossible to test several COTS components completely due to limited time and cost. They generally tested only some of the key functionalities, and depended on comments from a newsgroup to evaluate the quality of the COTS components. Therefore, COTS components with more and better comments in the newsgroup or marketplace bulletin had a good chance to be selected, because they were assumed to be better tested and in general of higher quality.

The answer to research question RQ2: There are some common new activities and one new role is added in COTS-based development processes. The possible variations are when and how to perform these new activities. 5.3 Scenarios of projects that used COTS successfully and unsuccessfully Research question RQ3 is about summarizing the process scenarios of projects using COTS components successfully and unsuccessfully. The focus is to investigate the relationship between the possible risks and the development processes. We asked respondents to summarize their problems encountered (PEs) and their examples of good practices (GP) in the studied projects. First, we extracted process-relevant parts, i.e. what were the main development processes, and when and how were the new activities performed. We then summarized some other salient problems and examples of good practice, which may give guidance to COTS-based development.

In our definition, a project that used COTS components successfully means that the COTS components contributed positively to time-to-market and system quality. Otherwise, it is regarded as being used unsuccessfully.

5.3.1 Scenarios of projects that used COTS components unsuccessfully. Four of the 16 studied projects were regarded as using COTS components unsuccessfully. The scenarios are summarized in scenarios Sc1 to Sc3 in Table 4.

Appendix A

136

Table 4. The scenarios of the 4 projects that used COTS components unsuccessfully Scenario ID (Project ID)

Characterizations

Problems Encountered (PEs)

Good Practice (GP)

Sc1 (P1 and P14)

Two projects used waterfall processes. Made the make vs. buy decision and the COTS component selection in the implementation phase. Selected unfamiliar COTS components.

PE1: Selected the COTS component too late. PE2: Used only demonstration, without pilot. PE3: Could not (re) negotiate requirements with customer. PE4: It was difficult to formally verify the COTS component’s reliability.

GP1: Used external COTS expert.

Sc2 (P10)

One project used waterfall process. Selected COTS components in the design phase, but mainly on their functionality Selected unfamiliar COTS components.

PE5: Selection was based only on functionality. Ease of integration was not considered.

GP2: Used external COTS expert with internal developers.

Sc3 (P2)

One project used incremental & prototyping process. Selected the COTS components in the design phase, but mainly because of their functionality. Selected unfamiliar COTS components.

PE5: Selection was based only on functionality. Ease of integration was not considered. PE6: COTS component’s delay caused the delay of the project.

The detailed backgrounds of each problem encountered are following: PE1: The two waterfall projects (P1 and P14) started to select COTS components in the implementation phase. It was very difficult for them to select the suitable COTS components, when the customer requirements and design have been fixed. They built a lot of glueware to integrate the selected COTS components, and this brought many integration problems at the very end.

Appendix A

137

PE2: In project P1, the COTS component was selected only based on the vendor’s demonstration. They did not do any hands-on trial to investigate how much effort is needed to integrate the COTS component into the current system. As mentioned in PE1, they had to spend a lot of effort on integrating the COTS components into the system, which caused delay in the project. PE3: Although some previous studies proposed to (re)negotiate requirements with the customer when developers found later limitations of COTS components, developers in project P14 could not (re)negotiate the requirements with customers even if they wanted to. This is because the developer wanted to use COTS components, not the customer. The customer cared only about the final result. If developers were not allowed to use COTS components, (re)negotiation of requirements due to COTS limitations is difficult. PE4: The final product of project P14 is a highly safety-critical system. To ensure reliability of the product, each line of code was required not only to be tested strictly, but also to be formally proved. As the source code of the COTS component was not available, they could not formally prove the correctness of the COTS component even the COTS component passed all existing testing. They were therefore always worried about it until they bought the source code. PE5: In project P10, the project members did not know exactly how to integrate the COTS components into the system and how to integrate them with other components before COTS component selection. Their customer intervened in the COTS component selection and recommended a certain COTS component that could provide the needed functionality. This COTS component caused a lot of quality problems.

Although these projects were regarded as being unsuccessful, there were still some

examples of good practice that helped to improve the project results. The background of each example of good practices is the following:

- GP1: Project P1 hired one person from the vendor as a part-time consultant to configure and integrate the COTS. Although a lot of glueware was needed to integrate the COTS component because of the wrong selection, this external expert helped to shorten the integration time to some degree.

- GP2: In project P10, the whole component part was outsourced to a consulting company. The software company signed a fix-cost bid with the consulting company. The consulting company did all the work to make the integration work. As a result, the software company met the budget for the project. One of their architecture and project managers worked closely with consulting company. Therefore, these internal persons learned the COTS components in this project. They believed that they could save a lot of money in the following project, which will use the same COTS component.

5.3.2 Scenarios of projects that used COTS component successfully. Twelve projects successfully integrated the COTS components. The scenarios are summarized into scenarios Sc4 to Sc6 in Table 5.

Appendix A

138

Table 5.The scenarios of 12 projects that used COTS component successfully Scenairo ID (Proj.ID)

Characterization Problems Encountered(PEs)

Good Practice (GP)

Sc4 (P4 and P13) Two projects used a

waterfall process.

Made the make vs. buy decision and selected COTS components in the requirement phase.

Selected familiar COTS components.

PE7: Under-estimated learning effort.

PE8: Under-estimated the integration effort.

PE9: It was difficult to isolate and prove bugs.

GP3: COTS component selection was based on architecture.

GP4: Used COTS component newsgroup.

GP5: Prepared for replacement.

Sc5 (P12) One project used waterfall with some prototyping process.

Made the make vs. buy decision and selected COTS components in the design phase.

Selected familiar COTS components.


Sc6 (P3, P5, P6, P7, P8, P9, P11, P15, and P16)

Nine projects used incremental with prototyping processes.

Some projects selected COTS components in the requirements phase with familiar COTS components.

Some projects selected COTS components in the implementation phase with familiar COTS components

Some projects selected COTS component in the design phase with unfamiliar COTS components.

PE10: Met standard mismatch. PE11: Met deployment mismatch.

GP3: COTS component selection was based on architecture.


GP6: Using COTS components in the incremental & prototyping process helped the success of the project.

GP7: Integrated unfamiliar COTS component first.

GP8: Limited the amount of glueware.

GP9: Did not change COTS code.

Appendix A

139

GP10: Did integration testing incrementally.

GP11: Paid vendor incrementally.

GP12: Investigated COTS component market.

The detailed explanations of each problem encountered are following: - PE7: In project P4, they used a COTS component which includes more than 500

different classes. It took longer time to read the document and to understand how to use these classes than they originally estimated. As the COTS component provided most functionality, they still believed that it shortened the time-to-market of the system.

- PE8: In project P13, the project manager made a too optimistic estimation on the effort for the COTS component integration. Although they have used these COTS components before, it still took unexpectedly more time to integrate it into the new project. However, they still thought the COTS component provided positive effect on the time-to-market of the system.

- PE9: In project P13, they found some defects in the system during testing. However, it was difficult to convince the vendor that these defects were inside rather than outside the COTS components. The vendor took a long time to rectify these defects.

- PE10: In projects P3 and P9, developers found that the COTS components did not support the standard (such as EJB, DCOM) the vendor claimed. The COTS supported some old version standard instead of the new one as they claimed.

- PE11: In project P8, the COTS components could not work in the first round of integration testing. The COTS components failed because it could not work on the deployed platform. The reason is that developers used different platforms in the development and test.

To ensure the success of these projects, members of these projects performed some

other activities to reduce some possible negative effects of COTS components. Their examples of good practices are summarized in the following:

- GP3: As the developers had experience with the relevant COTS components, they knew how to integrate them and their limitations. In the COTS components selection, they focused mainly on the ease of integration. If the COTS components cannot provide all the required functionality, they built some addware. In project P4, selecting COTS components based on the ease of integration was highly recommended.

- GP4: In project P4, they not only consulted an internal expert, but also followed the newsgroups of a COTS component on the Internet. Comments in the newsgroups helped them to make the right selection.

- GP5: In project P13, they investigated the architecture of the system and the COTS component to ensure the substitutability of the COTS component. This

Appendix A

140

means that they can easily change to other COTS components, if they are not satisfied with the current COTS component. The same experience was also emphasized by respondents in projects P11 and P12.

- GP6: Several projects started to select COTS components in the requirements phase. This makes it simpler to match and possibly negotiate customers’ (volatile) requirements with the available functionality of the COTS components, because the customer will accept the functionality of the COTS components shown in the prototype. At the start of the prototyping process, the main benefit of using COTS components is to build a prototype rapidly. They could also learn and understand how to use the COTS components in building such a prototype.

- GP7: In project P3, integrating unfamiliar COTS components was ranked as a high risk task. These unfamiliar components had been integrated and tested before other components.

- GP8: In project P3, one experience was that they tried to limit the amount of glueware. They believed that the less glueware they need to produce, the easier it would be to update in case a vendor provides new versions.

- GP9: In project P13, they had the source code of one COTS component. However, they decided not to change the code because if the vendor released a new version, it will be hard for the customer to maintain possible changes himself.

- GP10: In some projects that used more than one COTS component, the experience is that the COTS components should be integrated and tested incrementally. It means that the integration test should be performed immediately after one COTS component was integrated, instead of doing the final integration testing. Without incremental integration testing, it may be difficult to locate defects after all the COTS components are integrated.

- GP11: In project P7, the project paid part of the agreed amount after the evaluation. As the project members discovered some limitations of the COTS component, they asked the vendor to perform changes. After they got the component as required, they paid the rest of the money to the vendor. They believe that this allowed them to persuade the vendor to change the component.

- GP12: In project P15, they investigated the COTS component market share before they selected it. They believed the vendor can survive for a long time if their COTS products were used sufficiently in the market.

The studied projects showed that the COTS-based development could succeed under

different process scenarios. The answer to research question RQ3: COTS-based development could be performed successfully using waterfall and evolutionary processes. However, different process scenarios may face various risks.

6 Discussion 6.1 Comparison with related work Results of this study confirmed some conclusions from previous studies and contradicted others.

Appendix A

141

Boehm and Abts (1999) regard both the waterfall model and evolutionary development as unsuitable for COTS-based development. The results from research question RQ3 showed however that some projects could integrate COTS components successfully, using waterfall or evolutionary processes. However, the results of RQ3 revealed that different process scenarios (Sc1 to Sc6) may have different risks. It is therefore necessary to perform risk management according to customized process scenarios, as proposed by Boehm and Abts (1999). In addition, results of RQ1 showed that the main development processes were decided before the decision was made about whether to use COTS components. That is, if the process is risk-driven as Boehm and Abts (1999) proposed, developers should use non-COTS related risks to decide the main development processes first. They should (only) start to consider COTS related risks, after they decide to use COTS components.

Both EPIC (Albert and Brownsword, 2002) and the process proposed by Morisio et al. (2000) indicated that some new activities and roles should be added to traditional development processes. Examples are the make vs. buy decision, COTS components selection, COTS components integration, and COTS component knowledge management. The results of RQ2 gave further support to their conclusions. This study shows that there are several possible variations in COTS-based development process, for example, the COTS components were selected not only in the requirement phase, but also in the design or coding phases.

Although the formal decision-making algorithms, such as MAUT (MacCrimmon, 1973), MCDA (Morisio and Tsoukias, 1997), or AHP (Saaty, 1990), was used in some previous studies, such as OTSO (Kontio, 1996), CAP (Ochs et al., 2001), and CISD (Tran et al., 1997). The results of RQ2 showed that none of the studied projects used these formal decision-making algorithms. The results support findings (named thesis T4) of Torchiano and Morisio (2004). The possible reason is that these techniques may not be applicable in selecting COTS (Ncube and Dean, 2002; Maiden et al., 2002). In practice, if developers had enough previous experience or tacit knowledge with specific components, COTS component selection was based on their experience (see SP_fam in Section 5.2). If they had no previous experience with the selected COTS components, the selection was based on hands-on trials (see SP_unfam in Section 5.2). A handful of candidate COTS components were compared very quickly with some key factors, for example price, platform, and vendor reputations. Two or three COTS components were selected for further hands-on trials.

Although methods to change or negotiate the requirements, such as MBASE (Boehm, 2000), PORE (Maiden and Ncube, 1998), IIDA (Fox et al., 1997), CARE (Chung and Cooper, 2002), RCPEP (Lawlis et al., 2001), and CRE (Alves and Finkelstein, 2003), have been proposed to select COTS products. The results of RQ2 showed that these methods were not used in our studied projects. The possible reason is that most COTS components being used in our studied projects are fine-grained COTS components with limited functionalities (see Table 3). In case of using these fine-grained COTS components, it is difficult for developers to negotiate the requirements with the customer. As shown in PE3 (see Section 5.3.1), the customers can easily ask the developers to build the required functionalities in-house instead of using a COTS component, if the candidate COTS component cannot satisfy the customers’ requirements.

Appendix A

142

COTS-based software introduces many risks (Voas, 1998b). Some previous studies summarized lessons learned and experiences by case studies (Boehm, et al., 2003; COTS lessons learned repository, 2004; Tran et al., 1997; Fox et al., 1997; Rose, 2003; Morris et al., 2003; FAA Guide, 2003; Kotonya and Rashid, 2001). The results of this study gave further support to some of their conclusions. For example, faulty vendor claims may result in feature loss and/or significant delays (Boehm, et al., 2003), and it is not recommend changing the source code of the COTS component even if such a code is available (COTS lessons learned repository, 2004; Morris et al., 2003). However, we discovered some new salient problems encountered and examples of good practice, such as incremental payment and preparation for replacement. In addition, previous studies did not investigate the relationship between the risks and corresponding process scenarios. Their lessons learned are proposed as general guidelines. It is therefore difficult for practitioners to assess and evaluate the possible risks according to their project contexts. The results of our study illustrated that some risks occur more frequently in some development processes, such as waterfall process, than in other development processes, such as incremental process.

6.2 Customizing the COTS-based development process Our results show that the COTS-based development process is not a totally new process. It needs a customization (change or add new activities) on the traditional development processes. Based on the results of this study, we propose how to design and customize a COTS-based development process. As our studied projects mainly used waterfall and evolutionary processes, our proposal focuses on these. The process design could include two elements:

- First, decide the main development processes based on the project profile. The main development process could be decided based on risk considerations such as proposed by Boehm and Abts (1999), and based on the typical problems discovered in this study.

- If the main development process is waterfall or evolutionary, the process could be customized based on the actual main development process and project members’ familiarity with relevant COTS components as following Figure 1. For each customization, some possible risks must be identified and managed as we discovered from this study.

6.2.1 Waterfall with unfamiliar COTS components. Scenarios Sc1 and Sc2 showed that it is difficult to use an unfamiliar COTS component successfully if the general process is waterfall. It will be difficult to find COTS components to fully satisfy the requirements after the requirements and design have been decided upon. Therefore, our recommendation is that the make vs. buy decision and the COTS components selection should be performed before the implementation phase. The COTS components selection process could be SP_unfam, and the ease of integration should be given much consideration in the make vs. buy decision. If the candidate COTS components do not appear to be easy to integrate, it is better to develop them in-house. Another recommendation is to inform the customer; if possible, about using COTS components after the make vs. buy decision. It may give the developers leeway to (re)negotiate the requirements if later limitations of COTS components are found.

Appendix A

143

6.2.2 Waterfall with familiar COTS components. If the project members are quite familiar with relevant COTS components, the make vs. buy decision and COTS component selection could be performed in the requirement phase, because the integrators know how to integrate them, and which functionality could be provided by the COTS components. Our results showed that COTS components selection drives the requirements to some extent. 6.2.3 Incremental & prototyping with unfamiliar COTS components. Here, the risk of using unfamiliar COTS components is relatively low, compared to waterfall projects. The integrator can select the COTS components based on the completeness of required functionalities and build a prototype in a short time. The customers’ requirements can be agreed upon by evaluating the prototype. For this customization, the recommendation is to integrate the unfamiliar COTS components first, if there are different phases to implement the system incrementally. 6.2.4 Incremental & prototyping with familiar COTS components. If the project members have enough experience with relevant COTS components; they can easily build a prototype of the system with the COTS component. In this case, our results showed that buying COTS components had more advantages than building them in-house. Here, the make vs. buy decision can be performed in the requirements and/or design phase. If there are different phases to implement the system incrementally, they do not need to integrate all COTS components upfront. In this case, COTS components could be considered as in-house built components. The process could be the same as a non-COTS one.

Design

Waterfall

Incremental& prototyping

SP_unfam

Waterfall with unfamiliar COTS component

Waterfall with familiar COTS component

Incremental & prototyping with familiar COTS component

Incremental & prototyping with unfamiliar COTS component

Requirement Implementation

RequirementSP_fam

DesignSP_fam

Implementation

RequirementSP_unfam

DesignSP_unfam

Implementation

RequirementSP_fam

DesignSP_fam

ImplementationSP_fam

Fig. 1. Possible COTS-based development process customizations

SP_fam: Familiarity-based selection process SP_unfam: Internet search, trial-based selection process

6.3 Threats to validity 6.3.1 Construct validity. In this study, most variables are taken directly, or with little modification, from the existing literatures. To ensure construct validity, we did two rounds of pre-tests on the interview guide in ten local IT companies. These companies

Appendix A

144

were selected based on convenience. For this, we wrote an introductory letter, sent by email. This was followed up by a phone call to see if the company had a willing contact person and a suitable COTS-based project. The interview guide (a MS-Word document) was then sent to the respondents by email. The respondents filled in the questions in the interview guide electronically, and gave free-text comments to each question and the following alternatives. 10% of the questions and alternatives in the interview guide were revised based on such comments.

The selected COTS component and further COTS component examples proposed from respondents show that the definition of a COTS component was understood correctly. Furthermore, the responses to the three meta-questions show that the questions in the interview guide were understood consistently.

However, one possible threat to construct validity is our chosen success criteria for a project successfully using COTS components, implying that the integrated COTS components should contribute positively both to time-to-market and to the quality of the final system. This may not apply for all kinds of projects as some projects might emphasis only time-to-market, and others might emphasis only quality.

6.3.2 Internal validity. The first possible threat to internal validity is our misunderstanding of respondents’ answers. Although two interviewers carried out the interviews and there was only one in each interview, we taped all interviews. Listening to the tape helped ensure correct interpretation of answers and comments. However, having an independent (third) person to listen to the tape might increase data quality.

The second possible threat to internal validity is that people tend to talk about examples of good practices instead of problems they encountered. As we had a confidentiality agreement with the respondents, the respondents were encouraged to tell the truth. The data in Tables 4 and 5 showed that 11 (out of 16) respondents stated at least one problem encountered in their projects. However, the respondents might give only the most salient problems they encountered and forget some minor problems because we asked for data about past projects.

The third possible threat to the internal validity is that the lifecycle phases, such as requirement, design, and implementation, may not be clearly separated in projects using prototyping and incremental processes. We called back some respondents to ask whether they had specific software architecture in mind before the make vs. buy decision and COTS component selection. Their answers gave future clarification on whether COTS components were selected in the requirements or design phases.

The fourth possible threat to the internal validity is that some projects might not tailor the process because their processes already support COTS usage. So, we might miss some necessary process improvements that have been performed before the project start because we asked only salient process changes within the studied projects.

We selected 10 projects in 10 companies. However, we selected 2 projects in each of the other 3 companies. The fifth possible threat is that they are the same instead of different. However, data in Appendix A showed that all projects in the same company are different:

- In the first company, one project used waterfall and another used incremental

process with prototyping. The project members were totally different, and the COTS selection and strategy to integrate COTS were also different.

Appendix A

145

- In the second company, one project used waterfall and another used incremental process with prototyping. The project members were totally different. The architectures of the final systems were also different.

- In the third company, the first project was finished two years before the second one started. The project members were totally different. The COTS selection method and strategy were also different.

6.3.3 External validity. The primary threat to external validity is that the study is based on few and possibly not typical companies in Norway. According to Norwegian “Census Bureau” (SSB) (SSB, 2004), the mean value of employee number in Norwegian IT companies is 6. The first possible external validity to this study is that our selected companies are mostly medium and large sized. However, the selected projects covered several application domains.

The second possible threat to external validity is that the size (based on person-hours) of the projects might bias the conclusion of this study. We have divided the projects into three groups: small (less than 10,000 person-hours), medium (between 10,000 and 100,000 person-hours), and large (more than 100,000 person-hours). We found out that the actual process used, the problems encountered, and the good practice crossed small, medium, and large projects.

The third possible threat to external validity is that most COTS components are well-defined and provided only secondary (support) functionality of the delivered system. The process customizations may be different if the COTS components are intended as the core part of the system, or the COTS components are large packages, such as ERP and content management systems.

7 Conclusions and future work This paper has presented an exploratory study of COTS-based development processes in 16 software projects in 13 Norwegian IT companies. We conclude that using COTS components can be done as part of traditional development processes (e.g. waterfall and evolutionary) – there is no special “COTS-based development process”. However, successful use of COTS components in such processes requires that some new activities and roles are introduced in order to reduce risks. Typical new activities are the make vs. buy decision, COTS component selection, and COTS component integration. A new role is that of a knowledge keeper. Two of the new activities, the make vs. buy decision and the COTS component selection, can be placed in different development phases (requirements, design, or implementation). This depends on project context, especially on the tacit familiarity with possible COTS components and the flexibility of requirements.

Although there were several different COTS component selection processes proposed from academia, we discovered that two COTS component selection processes were popular in practice. The first one is familiarity-based selection process. If the project members or external experts had enough previous experience with the candidate COTS component, their preference will decide which COTS component will be used in the new project. The second selection process is a process combining Internet searches with hands-on trials. Although several formal and complex COTS selection processes have been proposed, they were not used in this study. To reduce time-to-market,

Appendix A

146

industrial practitioners performed only the most necessary steps in COTS component selection and made the final decision based on the results of the hands-on trial.

A set of explanatory scenarios, encountered problems, and examples of good practices have been synthesized to assist in the customization of the traditional development processes with the new activities and roles. Although some of these problems and examples of good practice have been mentioned in other previous studies, we discovered several new ones, which decided the success or failure of the studied projects and deserve to be investigated further.

The main limitation of this research is that it is based only on software development projects in Norway, and that the sample size is small and perhaps not representative in profile for a typical COTS-based project. In the future, we will investigate further the actual COTS-based development processes. Although we proposed some possible customization of COTS-based development processes based on the results of this study, more samples are needed to validate them. The validation will focus on when, how, and why the new activities (make vs. buy decision, and COTS component selection) are performed.

We will also do a further study on systematic risk management in COTS-based development. The different process scenarios may meet different risks and need different risk mitigation methods. To manage risks in COTS-based development, it is necessary to identify risks, evaluate them and design risk mitigation strategies. Although several risk management methods have been proposed by case studies, there are few studies to verify these proposals and investigate how to use them in different process scenarios.

We are now formulating more explicit research hypotheses based on the findings of this study and the lessons learned. We plan to design a web-based questionnaire and use this to perform a quantitative survey in a large sample in Norwegian IT companies, in parallel with replicating the study in Germany and Italy. 8 Acknowledgements This study is supported by the INCO, SPIKE, and FAMILIES projects. Comments from our colleague Tore Dybå gave valuable input to this study. We thank the colleagues in these projects, and all the participants in this study. Appendix A: Detailed information on process customization

Comp. ID

Proj. ID

Succ. or failed

Main process

Other activities Added

New role added

C1 P1 Failed Waterfall Buy vs. build. Selection. Build glueware & addware.

External expert: Hired one person from the vendor as consultant.

C1 P2 Failed Incremental & prototyping

Customer decided COTS. Learning.

Appendix A

147

Build glueware. C2 P3 Succ. Incremental

& prototyping

Buy vs. build. Selection. Learning. Build glueware & addware.

Internal expert.

C2 P4 Succ. Waterfall Buy vs. build. Selection. Learning. Build glueware & addware.

Internal expert.

C3 P5 Succ. Incremental & prototyping

Buy vs. build. Selection. Learning. Build glueware.


Buy vs. build. No selection as it was the only alternative. Learning. Build glueware.


Buy vs. build. Selection. Learning. Build glueware & addware.


Buy vs. build. Selection. Build addware.

Internal expert.


Without make vs. buy decision because using COTS is a company strategy. Selection. Learning. Build glueware & addware.

Internal expert.

C7 P10 Failed Waterfall Buy vs. build. Selection. Build glueware Learning.

External expert: The integration was outsourced to a consulting company.

C8 P11 Succ. Incremental Buy vs. build. Internal

Appendix A

148

& prototyping

Selection. Learning. Build glueware.

expert to follow COTS evolution.

C9 P12 Succ. Waterfall & prototyping


C10 P13 Succ. Waterfall Buy vs. build Selection. Learning. Build glueware.

Internal expert.

C11 P14 Failed Waterfall Buy vs. build. Selection. Learning. Build glueware.



Internal expert.



Internal expert.

Appendix B: Detailed information on COTS component selection

Proj. ID

COTS ID

Selection process In which phase was COTS selected

P1 1 Demo (recommended from external expert).

Implementation

P1 2 Demo (recommended from external expert).

Implementation

P2 3 Decided by customer Design P3 4 Experience Design P3 5 Internet search with pilot Requirements P3 6 Experience Design P4 7 Internet search with pilot Requirements P4 8 Experience Requirements P4 9 Experience Requirements P5 10 Experience Implementation P5 11 Internet search with pilot Implementation P6 12 Experience Requirements P7 13 Internet search with pilot Design P8 14 Experience Implementation P8 15 Experience Implementation P9 16 Experience Design

Appendix A

149

P9 17 Experience Design P9 18 Experience Design P10 19 Demo. Design P10 20 Demo. Design P10 21 Demo. Design P11 22 Internet search with pilot Design P11 23 Internet search with pilot Design P12 24 Experience Design P13 25 Experience Requirements P14 26 Demo. Implementation P15 27 Experience Design P15 28 Recommended from external

expert Requirements

P16 29 Experience Design P16 30 Experience Design

References Abts, C., Boehm, B. W., and Clark, E. B. 2000. COCOTS: A COTS Software

Integration Cost Model - Model Overview and Preliminary Data Findings. Proc. of the 11th ESCOM Conf, Munich, Germany, 325 -- 333.

Albert, C. and Brownsword L. 2002. Evolutionary Process for Integrating COTS-Based System (EPIC): An Overview. SEI, Pittsburgh, Available at: http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html.

Alves, C. and Fineklstein, A. 2003. Investigating Conflicts in COTS Decision-Making. Journal of Software Engineering and Knowledge Engineering 13(5):473 – 493.

Basili, V.R. and Boehm, B. W. 2001. COTS-Based System Top 10 Lists. IEEE Computer, 34(5):91 -- 93.

Boehm, B.W. and Abts, C. 1999. COTS integration: Plug and Pray? IEEE Computer, 32(1):135 -- 138.

Boehm, B. W., Port, D., Yang, Y., and Bhuta, J. 2003. Not All CBS Are Created Equally COTS-intensive Project Types. Proc. of the 2nd Int. Conf. on COTS-Based Software Systems (ICCBSS’03), Ottawa, Canada, Springer, LNCS Vol. 2580, 36 -- 50.

Boehm, B. 2000. Requirements that Handle IKIWISI, COTS, and Rapid Change. IEEE Computer 33(7):99 --102.

Brownsword, L., Oberndorf, T. and Sledge, C. 2000. Developing New Processes for COTS-Based Systems. IEEE Software, 17(4):48 -- 55.

Carney, D. and Long, F. 2001. What do You Mean by COTS? Finally, a Useful Answer. IEEE Software, 7(2):83 -- 86.

Chung, L. and Cooper, K. 2002. A Knowledge-Based COTS-Aware Requirements Engineering Approach. Proc. of the 14th Int. Conf. on Software Engineering and Knowledge Engineering, Ischia, Italy, 175 – 182.

ComponentSource. 2004. Available at: http://www.componentsource.com/. COTS Lessons Learned Repository. 2004. Available at: http://fc-

md.umd.edu/ll/index.asp

Appendix A

150

Crnkovic, I., Hnich, B., Jonsson, T., and Kiziltan, Z. 2002. Specification, Implementation, and Deployment of Components. Communication of the ACM, 45(19):35 -- 40.

FAA Guide. 2003. Federal Aviation Administration, Software Engineering Resource Center. FAA COTS Risk Mitigation Guide: Practical Methods for Effective COTS Acquisition and Life Cycle Support. Available at: http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm

FAMILIES project. 2003. Available at: http://www.esi.es/en/Projects/Families. Fowler. F. J., Jr 2001. Survey Research Methods (Applied Social Research Methods).

3rd edition, Sage Publications. Fox, G., Lantner, K., and Marcom, S. 1997. A Software Development Process for

COTS-based Information System Infrastructure. Proc. of the 5th Int. Symposium on Assessment of Software Tools (SAST '97), Pittsburgh, PA, USA, 133 – 142.

INCO project. 2000. Available at: http://www.ifi.uio.no/~isu/INCO/. Kontio, J. 1996. A Case Study in Applying a Systematic Method for COTS Selection.

Proc. of the 18th Int. Conf. on Software Engineering, Berlin, Germany, 201 – 209. Kotonya, G. and Rashid, A. 2001. A Strategy for Managing Risk in Component-based

Software Development. Proc. of the 27th EUROMICRO Conference, Warsaw, Poland, 12 -- 21.

Lawlis, P. K., Mark, K. E., Thomas, D.A., and Courtheyn, T. 2001. A Formal Process for Evaluating COTS Software Products. IEEE Computer, 34(5):58 -- 63.

Leung, K. R. P. H. and Leung, H. K. N. 2002. On the Efficiency of Domain-Based COTS product Selection Method. Journal of Information and Software Technology 44(12):703 --715.

MacCrimmon, K. R. 1973. An Overview of Multiple Objective Decision Making. Proc. of the Multiple Criteria Decision Making, University of South Carolina Press, 18 -- 44.

Maiden, N. A. M., Kim, H., and Ncube, C. 2002. Rethinking Process Guidance for Selecting Software Components. Proc. of the 1st Int. Conf. on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, 151 -- 164.

Maiden, N. A. M, and Ncube, C. 1998. Acquiring COTS Software Selection Requirement. IEEE Software 15 (2):46-56.

Morisio, M. and Tsoukias, A. 1997. IusWare: a Methodology for the Evaluation and Selection of Software Products. IEE Proceedings – Software Engineering, 144(3):162 -- 174.

Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S. E. 2000. Investigating and Improving a COTS-Based Software Development Process. Proc. of the 22nd Int. Conf. on Software Engineering, Limerick, Ireland, 31 -- 40.

Morris, E., Albert, C., and Brownsword, L. 2003. COTS-Based Development: Taking the Pulse of a Project. Proc. of the 2nd Int. Conf. on COTS-Based Software Systems (ICCBSS 2003), Ottawa, Canada, Springer LNCS, VOL. 2580,168 – 177.

NAICS.2004. Available at: http://www.revenue.state.ne.us/tax/current/buscodes.pdf. Ncube, C. and Dean, J. C. 2002. The Limitation of Current Decision-Making

Techniques in the Procurement of COTS Software Components. Proc. of the 1st Int. Conf. on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, 176 -- 187.

Appendix A

151

Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N. 2001. A Method for Efficient Measurement-based COTS Assessment and Selection - Method Description and Evaluation Results. Proc. of the 7th Int. Software Metrics Symposium, London, England, 285 – 297.

Robson, C. 2002. Real World Research: A Resource for Social Scientists and Practitioner-researchers (Regional Surveys of the World), 2nd edition, Blackwell Publishers.

Rose, L. C. 2003. Risk Management of COTS Based Systems Development. In: Component-Based Software Quality, Springer LNCS, VOL. 2693, 352 -- 373.

Saaty, T. L. 1990. How to make a decision: The Analytic Hierarchy Process (AHP). European Journal of Operational Research, 48(1): 9 -- 26.

SPIKE project. 2002. Available at: http://www.idi.ntnu.no/grupper/su/spike.html. SEI (Software Engineering Institute). 2004. COTS-Based Initiative Description.

Available at http://www.sei.cmu.edu/cbs/cbs_description.html. SSB: Norwegian Census Bureau. 2004. Available at: http://www.ssb.no Strauss, A. and Corbin, J. M. 1998. Basics of Qualitative Research: Grounded Theory

Procedures and Techniques. 2nd edition, Sage Publications. Torchiano, M. and Morisio. M. 2004. Overlooked Aspects on COTS-based

Development. IEEE Software, 21(2):88 -- 93. Tran, V., Liu, D. B., and Hummel, B. 1997. Component-Based Systems Development:

Challenges and Lessons Learned. Proc. of the 8th IEEE Int. Workshop on Software Technology and Engineering Practice, London, UK, 452 -- 462.

Voas, J. 1998a. COTS Software – the Economical Choice? IEEE Software, 15(2):16 -- 19.

Voas, J. 1998b. The challenges of Using COTS Software in Component-Based Development. IEEE Computer, 31(6):44 -- 45.

Appendix A

152

Appendix A

153

P3. Validation of Theses on OTS-Based Development

Validation of New Theses on Off-The-Shelf Component Based Development

Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Umair

Khan3, Marco Torchiano4 and Maurizio Morisio4

1Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU),

NO-7491 Trondheim, Norway {jingyue, conradi, oslyngst}@idi.ntnu.no

2Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysaker, Norway 3 Fraunhofer IESE, Sauerwiesen 6, D- 67661 Kaiserslautern, Germany

{Christian.Bunse, khan}@iese.fraunhofer.de 4Dip. Automatica e Informatica, Politecnico di Torino Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy

{maurizio.morisio, marco.torchiano}@polito.it

Abstract: Using OTS (Off-The-Shelf) components in software development has become increasingly popular in the IT industry. OTS components can be either COTS (Commercial-Off-The-Shelf), or OSS (Open-Source-Software) components. A recent study with seven structured interviews concluded with six theses, which contradicted widely accepted (or simply undisputed) insight. Since the sample size of that study was very small, it is necessary to investigate these theses in a larger and randomized sample. A state-of-the-practice survey in three countries – Norway, Italy, and Germany – has been performed to validate these new theses. Data from 133 OTS component-based projects has been collected. Results of this survey support four and contradict two of the initial theses. The supported theses are: OSS components were mainly used without modification in practice; custom code mainly provided additional functionality; formal OTS selection processes were seldom used; OTS component users managed to get required changes from vendors. The unsupported theses are: standard mismatches were more frequent than architecture mismatches; OTS components were mainly selected based on architecture compliance instead of function completeness. 1. Introduction OTS components (Off-The-Shelf) includes COTS (Commercial-Off-The-Shelf) and OSS (Open Source Software) components. Developing with the OTS components is gaining more attention from both the research and industrial community. A newly performed study, in the form of seven structured interviews, proposed six new theses in OTS based development [12]. Since the sample size of that study was small, it is necessary to investigate the six theses in a larger and randomized sample. We therefore designed a state-of-the-practice study on OTS component-based development to validate these theses. The survey was performed in three European countries (Norway,

Appendix A

154

Italy, and Germany). We have gathered answer from 133 projects using either COTS or OSS components from a sample of 1500 companies.

Data of this survey support four of the six new theses and contradict the other two. How to interpret the survey result is also discussed in detail in order to give recommendations and suggestions on OTS based development.

The rest of this paper is organized as follows: Section 2 introduces the six theses we intend to investigate. Section 3 presents the research questions and survey design. Section 4 presents the actual sample. Section 5 lists results of each research question. Detailed discussion on the survey results is given in Section 6. Conclusions and future work are presented in Section 7. 2. Six new theses on OTS-based development After performing seven structured interviews of small and medium software companies in Norway and Italy, Torchiano and Morisio summarized six new theses which contradicted statements or implicit assumptions from previous literatures [12]. 2.1 New Thesis T1: Open source software is often used as closed source. In previous literatures, OSS components are usually considered to be completely different from closed-source components [6]. OSS products are provided by open source communities with source code available. COTS components are provided by a commercial vendor. The source code of COTS component is not available in most cases. Torchiano and Morisio discovered that developers didn’t look and change the source code of OSS components. The assumed explanation is that the users didn’t need to see and modify, or they lacked the knowledge, skill, or resources to do so [12]. 2.2 New thesis T2: Integration problems result from lack of compliance with standards. Architecture mismatches constitute a secondary issue. In previous literature, architectural mismatches were cited as one of the primary sources of integration problems [5, 13]. However, Torchiano and Morisio did not find any instances of this. The integration problem in their interviewed was that the selected components did not fully or correctly support the claimed standards [12]. 2.3 New thesis T3: Custom code mainly provides additional functionalities. The previous literature greatly emphasized the integration aspects of custom code (named glueware or gluecode) [3, 9]. However, Torchiano and Morisio concluded that one of the most common remedial solutions among interviewees was to add code (called addware) to provide missing functionalities [12]. 2.4 New Thesis T4: Developers seldom use formal selection procedures. Familiarity with either the product or the generic architecture is the leading factor in selection. Previous studies have proposed several structured, formal, or semi-formal selection procedures [1, 7]. However, no project interviewed by Torchiano and Morisio used any of them.

Appendix A

155

2.5 New thesis T5: Architecture is more important than requirements for product selection. For COTS component selection, previous literature proposes selection processes mainly based on requirements and their (re)negotiation [2, 7]. However, Torchiano and Morisio discovered that the architecture drove the selection process, and was the fulcrum of all trade-off activities [12]. 2.6 New thesis T6: Integrators tend to influence the vendor on product evolution whenever possible. Literature generally describes the lack of control over the features and evolution of COTS component as an immutable condition that integrators must tolerate [3]. However, Torchiano and Morisio found out that their interviewees were acting to modify this condition [12]. 3. Research design

To test the six new theses, we decided to collect more data using a randomized sample. Our initial plan was to transfer the theses directly into hypotheses and make a quantitative survey with standardized questions in a formal questionnaire. We designed six fixed hypotheses according to the new theses and a preliminary questionnaire. The process was as following:

- Jan.-June 2003: Read and discuss about the topic. - June-Aug. 2003: Make initial research questions and draft questionnaire (in

English). - Late Aug. 2003: first industrial pre-test in five local IT companies and with three

internal colleagues. - Sep.-Oct. 2003: Revision of the draft questionnaire. - Late Oct. 2003: Second pre-test in five companies.

After these two rounds of pre-testing, we discovered that it was difficult to reliably investigate the six theses without a pre-study to clarify our research questions. For example, we asked respondents whether they had influenced the vendors on product evolution. The question was intent to investigate T6. After the pre-tests, we concluded that it was not possible to get reliable results on this question. Some respondents had pushed the vendor to change a COTS component concerning bugs fixing or new functionalities. Although these respondents could tell whether they had got the required changes or not, they were not sure whether they had influenced the evolution of the COTS component. 3.1. A pre-study to clarify research questions To clarify the research questions, we decided to resort to structured interviews, i.e. a qualitative pre-study, before using a larger and randomized sample. The pre-study was designed to employee more open research questions, not fixed hypotheses, and with a

Appendix A

156

rather detailed interview guide. We used a convenience sample, i.e. a list of large, medium and small IT companies from existing project contacts. We ended up with 16 personal interviews in 13 Norwegian IT companies with an initial selection of 34 companies. The results of the pre-study were reported in [8]. 3.2. Clarified research questions After the pre-study, we were able to make testable research questions to investigate each new thesis. Although the research questions might not give complete answers to each thesis, they were designed to investigate each thesis in a reliable way. 3.2.1. Research questions for thesis T1. T1 claims that OSS components are being used as COTS components in practice. From the pre-study, we discovered that we had to interpret the word “used” correctly. The word “used” in [12] means that the source code of OSS component was seldom read and changed. OSS components were therefore “used” as “black-box”. As the meaning of “used” in T1 includes two activities, read and change, we designed two research questions T1.RQ1 and T1.RQ2 to study T1.

T1.RQ1: Had the source code of OSS components been read by integrators during software development? T1.RQ2: Had the source code of OSS components been changed by integrators during software development?

3.2.2. Research questions for thesis T2. T2 indicates that standard mismatches were the most frequent problem in COTS based development, and the architecture mismatches were a secondary issue. From the pre-study, we concluded that it was difficult to conclude that the architecture mismatches constituted a second issue. Although we can answer whether standard mismatches were more frequent than architecture mismatches, some other mismatches might be more frequent than these two. As a result, we limited our research question to compare the frequency of standard mismatches and architecture mismatches, as shown in T2.RQ1.

T2.RQ1: Were the standard mismatches more frequent than architecture mismatches?

3.2.3. Research questions for thesis T3. T3 argues that most custom code was used to provide additional functionalities rather than solving mismatches between components, or between components and application. In one word, T3 concludes that more addware (code to add missing functionalities) was built than glueware (code to solve various mismatches) in COTS-based development. In the pre-study, we asked the respondents to give the exact LOC (Line-Of-Code) of glueware and addware they developed in the project. However, most respondents could not give us the exact LOC of glueware/addware. Some respondents claimed that this information was confidential. Others complained that it was time-consuming to calculate this number. To avoid getting wrong or incomplete data, we limited the research question to investigate the

Appendix A

157

occurrences of building glueware and addware instead of comparing the exact LOC. So, the research question T3.RQ1 is:

T3.RQ1: Was the occurrences of building glueware more frequent than the occurrences of building addware?

3.2.4. Research question for thesis T4. T4 concludes that formal selection procedures, which generally come from the academia, were seldom used in practice. T4 also concludes that familiarity with either the product or the generic architecture was used as the leading factor in COTS component selection.

From the pre-study, we discovered that the definition of “formal selection procedure” in [12] was too general, because there are several different kinds of formal selection methods. After detailed analysis, we concluded the formal process defined in [12] is a kind of process includes three basic elements: selecting evaluation criteria (factors); collecting and assigning values to these criteria; making decisions using formal decision making methods. In the pre-study, we discovered two other selection processes were in frequent use [8]:

- Familiarity-based selection process: If the project members had enough previous

experience on the candidate COTS components, this experience will be the key factor in the COTS component selection.

- Hands-on experimentation based selection process: This kind of selection was used when there was no previous experience available. This selection process was more complex than familiarity-based selection process, and had three steps:

Step 1: First, developers used a search engine to browse the Internet, and used keywords to express decided functionality. This usually gave them a handful of candidates.

Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to 3 of them very quickly based on some key issues, such as licence issues, cost, or vendor reputation.

Step 3: After a small number of candidate COTS components had been decided, they downloaded a demo version of these from the web and tested the COTS components locally.

To test T4, we decided to compare how often these three kinds of selection processes

had been used in practice. So, the corresponding three research questions are:

T4.RQ1: How often had a formal process been used in practice? T4.RQ2: How often had the familiarity-based selection process been used in practice? T4.RQ3: How often had the hands-on experimentation based selection process been used in practice?

3.2.5. Research questions for thesis T5. T5 claims that architecture was more important than requirements for product selection. From the pre-study, we discovered that the architecture decision affected most components in the system and there were

Appendix A

158

trade-offs between architecture and COTS component selection. Some COTS components were selected to satisfy the required functionalities. Other COTS components in the same project were selected because they could easily be integrated, although they could not provide all required functionalities. Since different COTS components in the same project might be selected according to different criteria, our research question T5.RQ1 investigate which criteria were more important for the whole project rather than just for a specific COTS component.

T5.RQ1: Had architectural compliance been considered more important than functionality completeness in the whole COTS-based project?

3.2.6. Research questions for thesis T6. T6 argues that COTS users were not completely passive in the COTS market and they managed to influence the vendor on product evolution whenever possible. As mentioned in above, it was difficult to know whether the COTS users actually influenced COTS component evolutions or not, even if it was possible to know whether the required changes (corrections or new functionalities) from the COTS users had been satisfied. To get a reliable conclusion on T6, we limited our research question T6.RQ1 to evaluate how often the required changes had been implemented by the vendors.

T6.RQ1: How often the required changes (corrections or new functionalities) from integrators had been satisfied by the vendors.

3.3. Questionnaire design After we clarified the research questions based on results from the pre-study, we designed a new questionnaire. To validate thesis T1, several questions related to OSS components were also included.

The questions in the questionnaire were organized into three main parts: - Questions to collect background information of the company, project, and

respondents. - Questions to get answers of the research questions mentioned in Section 3.2

(Detailed information of each question is introduced in the Section 5). Some other questions to investigate process improvement and risk management were also included. However, answers to these questions are out of the scope of this paper.

- Questions to collect information about OTS components actually used in the project. In the questionnaire, we asked the respondents to give brief information (name, COTS/OSS, version) of all OTS used and detailed information of one of the most important COTS/OSS component (the one provides most functionality compared to other OTS components). For the selected most important COTS/OSS component, we gathered detailed background information, such as:

• Name, version, and functionality • How large a percentage of the application functionality in the whole system

is provided by the selected OTS component? • Whether the source code is available or not?

Appendix A

159

• What was the technical base, such as Library, COM; CORBA, EJB etc. Concepts used in this survey are listed in the first page of the questionnaire. In

general, the concepts are consistent with definitions proposed by Torchiano and Morisio [12]. The difference is that we extended the definition to involve OSS component. The key concepts are:

- Component: Software components are program units of independent

production, acquisition, and deployment and which can be composed into a functioning system. We limit ourselves to components that have been explicitly decided either to be built from scratch or to be acquired externally as an OTS-component. That is, to components that are not shipped with the operating system, not provided by the development environment, and not included in any pre-existing platform.

- An OTS component is a component provided (by a so-called provider) from a commercial vendor or the Open Source community. An OTS component may come with certain obligations, e.g. payment or licensing terms. An OTS component is not controllable, in terms of provided features and their evolution.

The final questionnaire was first designed and pre-tested in English (internal and

external reviews). It was then translated into the native languages and published on the SESE web survey tool [11] at Simula Research Lab in OSLO. 3.4. Sample definition and selection We define the unit of this study as a completed software development project, and its OTS-relevant properties. The projects were selected based on two criteria:


possibly with several releases. Projects were collected randomly from IT companies in Norway, Italy and Germany.

Slightly different sampling procedures were used in each country due to limited resources. Detailed discussions on sample selection in this study are reported in [4].

- In Norway, we gathered a company list from the Norwegian Census Bureau

(SSB) [10]. We included mainly companies which were registered as IT companies. Based on the number of employees, we selected the 115 largest IT companies (100 IT companies and 15 IT departments in the largest 3 companies in 5 other sectors), 200 medium-sized software companies (20-99 employees), and 100 small companies (5-19 employees) as the original contacting list.

- In Italy, we first got 43580 software companies from the “yellow pages”. We then randomly selected companies from them. For these randomly selected companies, we read their web-site to ensure they really are software companies. 196 companies were finally clarified as software companies, and were included in the original contact list.

Appendix A

160


3.5. Data collection procedure. To avoid bias in the data collection procedure, we made a guideline on how to contact respondents before the survey started. All countries – Norway, Italy, and Germany – followed the process in the guideline.

Possible respondents were contacted first by telephone. If the candidate respondents had suitable OTS-based projects and would like to join our study, a username and password of the survey tool and an electric version of the questionnaire were sent to them. The respondents could use either the SESE web tool or the electronic version to fill in the questionnaire. The respondents who didn’t want to answer the questionnaire were also registered. We logged the main reasons of non-response, such as “no software development”, “no OTS-based projects”, and “busy”. Most respondents used the Web tool to fill in the questionnaire. However, phone interview were used in Germany because of the confidential concerns in some companies [4]. 3.6. Data analysis method We used different analysis methods to analyse different research questions.

- Thesis T1 is relevant to OSS components. To study research questions T1.R1 and T1.R2, we compared the usage of OSS components with COTS components.

- Because T2, T3, T4, and T6 are not relevant to OSS components, only information from COTS components was analysed.

- As we mentioned in section 3.2.5, thesis T5 should be tested using data from the whole project rather than a specific COTS component. The unit to be analysed for research question T5.RQ1 is therefore the whole project instead of the selected OTS component.

4. Selected sample 4.1. Selected companies We have gathered results from 133 projects (47 from Norway, 48 from Germany, and 38 from Italy) from 127 companies. In general, we selected one project from each company. However, we selected more than one projects in three Norwegian IT companies because those companies have many OTS-based projects and would like to share more experience to this study.

The responding companies involved small (5-19 employees), medium (20-99 employees) and large (100 or more employees) ones as shown in Figure 1. The main business areas of the sampled companies also cover different types as shown in Figure 2.

Appendix A

161

Figure 1. Distribution of the company size

Figure 2. Distribution of the company’s main business area

4.2. Selected projects The final systems produced by the 133 projects cover different application domains as shown in the Figure 3.

Figure 3. Distribution of application domain of the systems produced by these

projects

Appendix A

162

In the selected 133 projects, 83 used only COTS components, 44 used only OSS components, and six used both COTS and OSS components. For the 44 projects using only OSS components, each of them gave the detailed information of an OSS component. For the 83 projects using only COTS components, 83 different COTS components were selected. For the six projects used both COTS and OSS components, the respondents selected to give detailed information of five COTS components and one OSS component. In total, we got detailed information of 88 COTS components and 45 OSS components. 4.3. Respondents Most respondents of the 133 projects have a solid IT background. More than 90% of them are IT managers, project managers, or software architects. Most of them have more than 2 year experiences with OTS-based development. All of them have at least a bachelor degree in informatics, computer science, or telecommunication. 5. Answers to research questions 5.1. Answers to T1.RQ1 and T1.RQ2 T1.RQ1 and T1.RQ2 is to test thesis T1, i.e. whether the OSS components had been used in the same way as COTS components. To investigate T1.RQ1, the corresponding question in the questionnaire is:

Have you read parts of the source code of OTS component? We used a five-point Likert scale to measure the answer. Respondents were asked to

answer “very little”, “little”, “some”, “much”, “very much”, or “don’t know”. For question T2.RQ2, we used the same measurement scale as for T1.RQ1, and the

corresponding question is: Have you modified parts of the source code of OTS component? We assigned an ordinal number from 1 to 5 to the above alternatives (5 meaning

“very much”). Results of these two research questions are summarized in Figure 4. The results show that the median value of reading OSS code is 3 (meaning “some”). However, the source code had seldom been changed as the median value of changing the OSS code is 2 (meaning “little”). Our future analysis discovered that 46% of the respondents claimed that they modified “very little” parts of the source code in OSS components.

Another question in the questionnaire asked if the source code of OTS components is available or not. Results show that not all COTS components in this study were “black-box”. 29 (out of 88) selected COTS components provided source code to their users. To compare with OSS components, we investigated whether the source code in “white-box” COTS components have been read or modified. Results are shown in Figure 5.

Appendix A

163

4444N =

CHANGE CODEREAD CODE

6

5

4

3

2

1

0

Figure 4. The source code usage of the OSS components

2726N =

CHANGE CODEREAD CODE

6

5

4

3

2

1

0

Figure 5. The source code usage of the “white-box” COTS components

From Figure 4 and Figure 5, we can see that the source code of COTS or OSS

components had been read to some degree, if it was available. However, the source code in both COTS and OSS components had seldom been changed. 5.2. Answers to T2.RQ1 T2.RQ1 asks whether the architecture mismatches were more frequent than standard mismatches. To investigate T2.RQ1, the corresponding question is:

Did you encounter some of the following aspects (risks) with the selected OTS

component? Standard mismatch and architecture mismatch were listed as option a) and b) of this

question: a) The OTS component did not follow industrial standards (COM, CORBA etc.) as

the provider claimed (i.e. glueware was needed) b) The OTS component had mismatching architectural assumptions towards other

parts of the system (i.e. glueware was needed) Respondents were asked to answer “yes”, “no”, or “don’t know”. For the selected 88

COTS components, only 4.5% of the COTS components had standard mismatches, compared with 28.4% COTS components having architecture mismatches.

Appendix A

164

5.3. Answers to T3.RQ1 T3.RQ1 is to test thesis T3, i.e. to compare glueware vs. addware. To investigate T3.RQ1, we asked respondents the same questions as in T2.RQ1. We listed another option c):

c) The OTS component could not provide all required functionality (i.e. addware needed)

Respondents were also expected to answer “yes”, “no”, or “don’t know”. In data analysis, we added all the “yes” answers for a) and b) options of this question

to count the occurrences of building glueware. The rationale is that glueware will be needed in case of either standard, or architecture mismatch. To count the occurrences of building addware, we added all the “yes” answers for option c).

Results show that glueware was needed for 32.9% of the selected COTS components and addware was needed for 65.9% of the selected COTS components. 5.4. Answers to T4.RQ1 to T4.RQ3 To answer T4.RQ1 to T4.RQ3, the corresponding question is:

Have you performed some of the following actions for the selected OTS component? The listed options are a) to f) as follows: - Concerning the identification of OTS components: a) Searched Internet for possible OTS component candidates. b) Got recommendation of possible OTS component candidates from the customer c) Got recommendation of possible OTS component candidates from a local

colleague/OTS-expert - Concerning the evaluation of components: d) Used a formal decision-making method to compare possible OTS component

candidates, e.g. with weighted evaluation criteria e) Limited possible candidates into 1-3 components, by reading literature or other

documentation f) Did “hands-on” try-out of 1-3 components, e.g. on a down-loaded demo version. Respondents were asked to answer “yes”, “no”, or “don’t know” for each option.

Results of the five options are listed in Table 1. Data in Table 1 shows that one common criterion for identifying OTS components is

familiarity-based. 58% COTS components were selected according to local expert (option c). Formal procedures (option d) were only used in evaluating 19% of the selected COTS components. The most common evaluation method was hands-on experimentation (option f). It was used to select 65% of the selected COTS components in this study.

Appendix A

165

Table 1. Results of T4.RQ1 – T4.RQ3 Option Yes No Don’t

know a) 59% 36% 5% b) 24% 73% 3% c) 58% 36% 6% d) 19% 74% 7% e) 48% 45% 7% f) 65% 27% 8%

5.5. Answers to T5.RQ1 To answer T5.RQ1, the corresponding question is:

What actions were performed during development or maintenance of the project? We listed several possible actions in the project. The action relevant to T5.RQ1 is:

OTS components were selected mainly based on architectural and standards compliance, instead of expected functionality.

Respondents were asked to answer “don’t agree at all”, “hardly agree”, “agree somewhat”, “agree mostly”, “strongly agree”, or “don’t know”. We assigned an ordinal number from 1 to 5 to the above alternatives (5 meaning strongly agree). Results of the 83 projects using only COTS components are shown in Figure 6.

82N =

ARCH. VS. FUNC.

6

5

4

3

2

1

0

Figure 6. Result of T5.RQ1

From Figure 6, we can see that the median value of this question is 2. It indicates that

most respondents hardly agree that COTS components were selected mainly based on architectural and standards compliance. So, we concluded that expected functionality still constituted the main concern in COTS components selection. 5.6. Answers to T6.RQ1

To investigate T6.RQ1, we asked the same question as T2.RQ1 with another option e):

Appendix A

166

e) The provider did not want to or could not change OTS component functionality as requested.

Results show that 60% of the selected COTS components have been changed by

vendors according to integrators’ requirements.

6. Discussion 6.1. Differences between COTS and OSS components

Our results support thesis T1. If “white-box” vs. “black-box” is regarded as the main difference between COTS and OSS components, results of T1.RQ1 and T1.RQ2 show that COTS and OSS component are not as different as people might assume. COTS components were not always “black-box”, as 33% of the COTS components selected in this study opened their source code to users.

If the source code is available, both COTS and OSS users tend to read the code. However, both COTS and OSS component users did not change much of the available source code. From this result, we can generally conclude that the OSS component was used the same as COTS component, i.e. integrated without modification.

6.2. COTS component selection and integration

Our results support thesis T4. For COTS component selection, results of T4.RQ1 to T4.RQ3 show that developers often choose a sub-optimal product only because they already know it. Formal selection procedures try to find out the most optimal product. However, these procedures might not be cost-effective in some industrial projects. As a result, selection techniques should also include familiarity as a main criterion [12].

Results of T5.RQ1 do not support thesis T5. COTS components users believe that functionality completeness is more important than architectural compliance in COTS component selection. However, we need to interpret this result carefully. The result does not claim that architectural and standard compliance are not important in COTS component selection. One lesson learned from our pre-study was that easiness of integration should be seriously considered before you decide to use a COTS component [8]. The possible explanation for result of T5 is that all candidate COTS components support required standards and architectures. So, the only comparable features are their functionalities. Then, COTS components were appeared to be selected based on their functionalities instead of architectural compliance.

For COTS component integration, results of T2.RQ1 contradict thesis T2, and show that architecture mismatches were more frequent than standard mismatches. With the maturity of COTS component standard, we believe that less standard mismatches will occur in the future.

Results of T3.RQ1 support thesis T3. It means that missing or incomplete features in COTS products raise significant problems. It is therefore important to develop effective techniques that allow seamless extension of existing products [12].

Appendix A

167

6.3. Vendor relationship in COTS based development

Results of T6.RQ1 support thesis T6. It shows that COTS component users actually pushed vendors to change the COTS components either because of bug fixes or new features. However, this result does not imply that most COTS component users actually influenced COTS component evolution. In general, we believe that the COTS component evolution is decided by the vendors’ marketing strategy rather than the requirements of a specific user, unless this user is very powerful. 6.4. Possible treats to validity 6.4.1. Construct validity. In this study, most variables and alternatives are taken directly, or with little modification, from existing literature. The questionnaire was pre-tested using a paper version by 10 internal experts and 8 industrial respondents before being published on the SESE tool. About 15% questions have been revised based on pre-test results. One possible threat to construct validity is that respondents from different companies may have slight different understanding on the same concept, such as software architecture and architectural mismatches. 6.4.2. Internal validity. We have promised respondents in this study a final report and a seminar to share experience. The respondents were persons who wanted to share their experience and wanted to learn from others. In general, we think that the respondents answered the questionnaire truthfully. However, different persons in the same project might have different opinions on the same project. Asking only one person in each project might not be able to get the whole picture of the project. Due to length limitation of a questionnaire, we asked the respondent to fill in information of only one component in the project. The possible threat is that other OTS components in the same project might give different answers to our questions. 6.4.3. Conclusion validity. This study is a state-of-the-practice study. We studied what had happened in industrial projects. However, we did not investigate the cause-effect relation of the phenomena discovered in this study. 6.4.4. External validity. We used different randomization to select samples in different countries. However, the sample selection processes were not exactly the same due to resource limitations [4]. Another possible threat to external validity is that our study focused on fine-grained OTS components. Conclusions may be different in projects using complex and large OTS packages, such as ERP, content management systems, and web services in general. 7. Conclusion and future work This paper has presented results of a state-of-the practice survey on OTS-based development in industrial projects. The study has investigated six new theses in OTS component based development. Results of this study gave support on four (T1, T3, T4, and T6) new theses and concluded that:

Appendix A

168

- T1: Source code of OSS components was read frequently with very few modifications. OSS components were de facto used as COTS components, even if the source code was available.

- T3: In the COTS components integration process, more addware had been built than glueware.

- T4: Formal selection procedures were seldom used to select COTS components. Familiarity-based and hands-on experiment based selection process were frequently used in practice.

- T6: In most cases, COTS users managed to get their required changes in the COTS components from COTS vendors.

Results of this study contradict the two (T2 and T5) other theses and concluded that: - T2: There were more architecture mismatches than standard mismatches in

practice. - T5: Functionality completeness was regarded more important than architectural

compliance in COTS component selection.

As this study is a state-of-the-practice study, we could not give cause-effect conclusions based on results of this survey. The next step is to do a further qualitative study with structured interviews to find out the possible explanations for the conclusions of this study. 8. Acknowledgements

This study was partially funded by the INCO (INcremental COmponent based development) project [14]. We thank the colleagues in these projects, and all the participants in the survey 9. References [1] Albert, C. and Brownsword, L. “Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview,” SEI, Pittsburgh, 2002. Available at: http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html.

[2] Boehm, B., “Requirements That Handle IKIWISI, COTS, and Rapid Change,” IEEE Computer, Vol. 33, No. 7, 2000, pp. 99–102.

[3] Boehm, B. and Abts, C., “COTS Integration: Plug and Pray?” IEEE Computer, Vol. 32, No. 1, 1999, pp. 135–138.

[4] Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Kampenes, V.B., Torchiano, M., and Morisio, M. “Reflections on conducting an international CBSE survey in ICT industry,” submitted to 4th International Symposium on Empirical Software Engineering, Nov. 2005, Noosa Heads, Australia, 11 pages.

[5] Egyed, A., Medvidovic, N., and Gacek, C., “Component-Based Perspective on Software Mismatch Detection,” IEE Proc.-Software, Vol. 147, No. 6, 2000, pp. 225–236.

[6] Lawton, G., “Open Source Security: Opportunity or Oxymoron?” IEEE Computer, Vol. 35, No. 3, 2002, pp. 18–21.

Appendix A

169

[7] Lawlis, P., Mark, K. E., Thomas, D.A., Courtheyn, T. “A Formal Process for Evaluating COTS Software Products,” IEEE Computer, Vol. 34, No. 5, 2001, pp. 58–63.

[8] Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B., “An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT Industry,” Proc. of the 10th IEEE International Metrics Symposium, Chicago, USA, September, 2004, IEEE CS Press (2004), pp. 72-83.

[9] Morisio, M. et al., “Investigating and Improving a COTS-Based Software Development Process,” Proc. of 22nd International Conference on Software Engineering, Limerick, Ireland, June, 2000, ACM Press, pp. 32–41.

[10] Norwegian Census Bureau (2004): Available at: http://www.ssb.no [11] SESE tool (2004), Available at: http://sese.simula.no [12] Torchiano, M. and Morisio, M., “Overlooked Aspects on COTS-based

Development,” IEEE Software, Vol. 21, No. 2, 2004, pp. 88-93. [13] Yakimovich, D., Bieman, J., and Basili, V., “Software Architecture Classification

for Estimating the Cost of COTS Integration,” Proc. 21st International Conference on Software Engineering (ICSE 99), ACM Press, 1999, pp. 296–302.

[14] INCO Project (2000): INCO project description, Available at: http://www.ifi.uio.no/~isu/INCO.

Appendix A

170

Appendix A

171

P4. SPI on OTS-Based Development Process

A State-of-the-Practice Survey of Off-the-Shelf Component-Based Development Processes

Jingyue Li1, Marco Torchiano2, Reidar Conradi1, Odd Petter N. Slyngstad1, and Christian Bunse3

1 Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU),


2 Dip. Automatica e Informatica, Politecnico di Torino Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy

[email protected] 3 Fraunhofer IESE, Fraunhoferplatz 1,

D-67663 Kaiserslautern, Germany [email protected]

Abstract. To gain competitive advantages software organizations are forced to develop systems quickly and cost-efficiently. Reusing components from third-party providers is one key technology to reach these goals. These components, also known as OTS (Off-the-Shelf) components, come in two different types: COTS (Commercial-Off-The-Shelf) and OSS (Open–Source-Software) components. However, the reuse of pre-fabricated components bears one major question: How to adapt development processes/methods with refer to system development using OTS components. To examine the state-of-the-practice in OTS component-based development a survey on 133 software projects in Norway, Italy and Germany was performed. The results show that OTS-based development processes are typically variations of well-known process models, such as the waterfall- or prototyping model, mixed with OTS-specific activities. One reason might be that often the process is selected before the use of OTS components is considered. Furthermore, the survey shows that the selection of OTS components is based on two processes: “Familiarity-based” and “Internet search-based”. Moreover, it appears that the lifecycle phase to select OTS components is significantly correlated with a project members’ previous familiarity with possible OTS candidates. Within this paper, we characterize the state-of-the-practice concerning OTS processes, using seven scenarios, and discuss how to decide or modify such processes and how to select OTS components. 1 Introduction Software development with OTS components is becoming increasingly popular in research and industrial communities. The use of OTS components introduces new requirements, which again require revised development processes. Although researchers and practitioners have been dealing with such processes quite a time, most studies are based on military or aerospace projects [10], [15], or other large projects. To propose

Appendix A

172

and design cost-effective OTS-based development processes, it is necessary to investigate how such projects are performed in different domains and project contexts.

Within a first, exploratory study, we investigated the commonalities and differences between development processes in 16 COTS-based software projects in Norway [6]. The study summarized several variations in COTS-based development processes and concluded that the customization of such development processes crucially depends on the project context, such as familiarity with possible COTS components and flexibility of requirements. Due to the small sample size it is necessary to verify these conclusions with a larger and more representative sample.

The study presented in this paper investigated several conclusions about variations in development processes, based on the exploratory study. The results show that the actual OTS component-based development processes are typically variations of well-known process models. In addition, two OTS component selection processes, such as familiarity-based and Internet search-based, are widely used in practice. By summarizing the state-of-the-practice of OTS component-based development processes using seven scenarios, we give systematic proposals on how to adopt the OTS-based development and OTS selection processes based on the project context.

The remainder of the paper is structured as follows: Section 2 presents previous studies and their research questions. Section 3 presents selected samples and Section 4 presents the empirical results. Discussions on customizing OTS-based development processes are given in Section 5 and Section 6 separately. Possible threats to validity are discussed in Section 7. Conclusion and future research are in Section 8. 2 Related work and research questions There is a consensus that the use of COTS components implies changes in the software process [2]. Some studies focused on the whole software development lifecycle [1], [3], and [10]. Others investigated the specific phase, especially in COTS component selection and evaluation [5], [8], [9], [11], [12], [13], [14]. 2.1 Process of the whole software development lifecycle Boehm et al. [3] regard both the waterfall model and evolutionary development as unsuitable for COTS-based development. In the waterfall model, requirements are identified at an earlier stage and COTS components chosen at a later stage. This increases the likelihood of COTS components not offering the required features. Evolutionary development assumes that additional features can be added if required. However, COTS components cannot be upgraded by one particular development team. The frequent lack of code availability hinders developers to adapt them to their needs. Therefore, Boehm et al. proposed that development models, which explicitly take risk into account, are more suitable for COTS-based development than the traditional waterfall or evolutionary approaches.

The National Aeronautic and Space Administration (NASA) has been developing systems using COTS components for many years (see [10] for the summary of experience made). Various processes, used across 15 projects, were examined and used as a basis for a common COTS-based development process.

Appendix A

173

The Software Engineering Institute developed the Evolutionary Process for Integrating COTS-based Systems (EPIC) [1]. EPIC integrates COTS component related roles and activities into a RUP process. The iterative and evolutionary nature inherent in EPIC allows developers to adjust the architecture and system design, as more knowledge is gained about the operations of the COTS components.

Our investigation on COTS-based development process in Norwegian IT companies, however, revealed that the main COTS-based development process is to customize the traditional development process to account for use of COTS components [6]. In addition, in all our investigated projects, the project members decided the main process before they started to think about using COTS components. To verify our findings, we needed more representative samples. Therefore, our first three research questions RQ1 to RQ3 are designed to examine the state-of-the-practice of the actual development process in OTS-based projects.

- RQ1: What were the actual development processes in OTS-based projects? - RQ2: Was the actual development process decided before the make vs. acquire

decision or after the make vs. acquire decision. - RQ3: Who decided the actual development process?

2.2 COTS component selection and evaluation process Based on case studies, researchers have proposed several COTS component selection processes and methods. Some of the direct assessment processes, such as OTSO [5], CAP [13], and CISD [8], assume that the requirements are fixed and select the COTS components by comparing how well the COTS component candidates satisfy the requirements. A formal decision-making process is usually used to select the “best” COTS component [12]. The formal decision-making process usually includes three basic elements: selecting evaluation criteria (factors), collecting and assigning values to these criteria, and applying formal decision-making algorithms such as MAUT [9], MCDA [11], and AHP [14]. However, both the study of Torchiano and Morisio [16] and our exploratory study [6] showed that these formal selection methods were seldom used. In fact, our exploratory study discovered two other popularly used selection processes. One is familiarity-based selection process and the other is Internet search with trial-based selection process. In addition, our exploratory study concluded that there are common new activities and that the possible variations are when and how to perform them, especially when and how to select COTS components. Therefore, research questions RQ4 to RQ6 are designed to investigate how OTS components were selected.

- RQ4: What was the actual selection process used? - RQ5: When was the OTS component selected? - RQ6: What was the relationship between the selection phase and the project

context, such as the familiarities with the OTS component candidate and the importance of the OTS component candidate?

Appendix A

174

3. Questionnaire design and sample selection The general questionnaire design, sample definition and selection, and data collection procedures are reported in [4], [7]. This study extended the exploratory study in two dimensions. First, it included OSS components because they represent an alternative to COTS components. Second, this study included much larger samples from three countries - Norway, Italy, and Germany. In addition, the sample was selected randomly instead of by convenience as in the exploratory study.

We have gathered results from 133 projects (47 from Norway, 48 from Germany, and 38 from Italy) from 127 companies. In general, we selected one project from each company. However, we selected more than one projects in three Norwegian IT companies because those companies have many OTS-based projects and would like to share more experience to this study. In the selected 133 projects, 83 used only COTS components, 44 used only OSS components, and six used both COTS and OSS components. Profiles of collected companies and projects are reported in [7]. 4. Empirical results 4.1. RQ1: what were the OTS component-based development processes? The first research question RQ1 is to investigate the actual development processes used in an OTS-based project. In the questionnaire, we asked the respondents to describe their development process in detail, whereby the answers were then summarized by different categories. The results are summarized in Figure 1 and show that the waterfall, incremental, XP [18], German V-model [19], and prototyping model are those mainly used in practice. Between these, the ‘incremental with prototyping’ model and XP was the most popular. The German V-model was also used widely in Germany as the OTS-based development process. 4.2. RQ2: when was the process decision made? The second research question RQ2 is to examine whether the actual development process was changed, considering the use of OTS-components. In the questionnaire, we asked the respondents to select whether the actual development process was decided before or after the make vs. acquire decision or after that. The results show that most (75%) projects decided their main development processes before they started to think about using OTS-components.

Appendix A

175

Fig. 1. The actual development process in the OTS-based project

4.3. RQ3: who was responsible for the process selection? The third research question RQ3 is to identify the decision maker concerning the actual development process. Within the questionnaire, we listed five options, such as company/department rules, project manager, software architect, software developer, and customer. The respondents were asked to select one or more from the options. The answers reveal that in 29% of all projects the development process is predefined by global company or department rules. In addition, concerning 14% of the remaining projects the decision was at least affected by company/department rules. This trend is especially obvious in Germany as the company rules dominated the development processes in 65% of our studied projects. 4.4. RQ4: what was the OTS component selection processes?

The research question RQ4 is aimed at summarizing the OTS selection and evaluation processes used in practice. Due to the length limitation of the questionnaire, we could not ask the respondents to fill in the details of every OTS component in their projects. Instead, they were asked to select one of the most important OTS components and fill in the details for this component, named Comp.1. Information provided for Comp. 1 was then used to investigate RQ5 and RQ6. To answer RQ4, we listed six possible activities as named a) to f) in the questionnaire:

a) Searched Internet for possible OTS component candidates. b) Got recommendation of possible OTS component candidates from the customer. c) Got recommendation of possible candidates from a local colleague/OTS-expert. d) Used a formal decision-making method to compare possible OTS component candidates, e.g., with weighted evaluation criteria. e) Limited possible candidates into 1-3 components, by reading literature/documentation.

Appendix A

176

f) Did “hands-on” try-out of 1-3 components, e.g., on a downloaded demo version.

The respondents were asked to fill in “yes”, “no”, or “do not know” for these

options. The results are summarized in Figure 2 and show that activities a, c, e, and f were those mainly used. Thus, the familiarity-based selection process (activity c) and the Internet search, trial-based selection process (activity a, e and f) from the exploratory study are proved to be more popular than a formal processes (activity d) in general. However, the further analysis with Chi-square shows that there are significant differences of the activities b and c between countries. In Norway and Italy, very few projects used the activity b. However, activity b was popularly used in Germany. In addition, almost all German projects performed activity c, while it was used only in half projects in Norway and Italy. 4.5. RQ5: when were OTS components selected?

From the exploratory study, we concluded that OTS components are selected in different development phases. Thus, research question RQ5 was postulated to investigate the selection phase of OTS component. The respondents were asked to select one of five phases, ranging from pre-study to coding, as selection time. Figure 3 shows that OTS components were selected in the early stages, such as pre-study, requirement, and overall design, in most projects, especially in Germany. 4.6. RQ6: what was the influence of the project context?

RQ6 aims at investigating the correlation between the project context and the OTS component selection process. We investigated two project context variables. The first is the project members’ familiarity with the Comp.1, measured by a five point Likert scale (i.e. very little, little, some, much, very much). The answers were coded into 1 to 5 (1 means very little and 5 means very much). The second variable is the importance of Comp.1 in the system, measured by the contribution (in %) of ‘Comp.1’ to the overall system functionality.

First, we investigated the correlation between the two context variables and the phase in which Comp.1 was selected. The selecting phases are treated as an ordinal variable and coded into the values 1 to 5 (1 means the pre-study phase and 5 means the coding phase). The importance of Comp.1 was also treated as an ordinal variable, although it is measured on a continuous scale. We then used the Spearman correlation test in SPSS 14.0 to investigate the correlation. The results are shown in Table 1 and reveal that the selection phase of Comp.1 was weakly (with correlation coefficient value -.356) correlated with the project members’ familiarity with it. It means that the project members preferred to decide the familiar OTS components in the early stage of the project. However, there is no significant connection between the importance of an OTS component and the phase it was selected.

Appendix A

177

Fig. 2. What selection and evaluation actions were performed?

Fig. 3. When was the OTS component selected?

Second, we investigated the relationship between the two context variables with the

selection and evaluation activities performed. Since possible answers on the selection and evaluation activity variable are “yes”, “no”, or “do not know”, we treated the projects with “yes” and “no” answers as two independent samples and then compared the mean differences of the two context variables. For the project members’ familiarity variable, we used the Mann-Whitney test in SPSS 14.0 to compare their rank differences. Results, presented in Table 2, show that project members will search the internet for a possible candidate, if there is no proper or familiar candidate at hand.

Appendix A

178

Table 1. The correlation between the selection phase and the context variables What phase was Comp.1 selected Context variables Spearman correlation Correlation Coefficient -.356 Project members’ experience

with the Comp.1 Sig (2-tailed) .000* The importance of the Comp.1 Correlation Coefficient -.132 Sig (2-tailed) .150 * P < .05 For the importance of Comp.1 variable, we used the independent T-test in SPSS 14.0

to compare mean differences, whereby the results are shown in Table 3. The overall result is that there is no significant relationship between the selection and evaluation activities performed with the importance of the OTS component.

Table 2. The relationship between selection activities and familiarity with the OTS

component Familiarity with Comp.1 Selection activities Mann-Whitney Z Asymp. Sig

(2-tailed) a) 1263.000 -2.407 .016* b) 1072.500 -1.622 .105 c) 1626.500 -1.043 .297 d) 964.000 -.533 .594 e) 1724.500 -.870 .385 f) 1481.500 -.913 .361 (Selection activities a) to f) are explained in Section 4.4) * P < .05

Table 3. The relationship between the selection activities with the importance the

OTS component Importance of the Comp.1 Selection activities t-test (equal variance not assumed) Sig (2-tailed)

a) .379 .706 b) -.971 .337 c) .804 .424 d) -.436 .666 e) 1.543 .126 f) 1.096 .278 (Selection activities a) to f) are explained in Section 4.4) * P < .05

5. How to customize the OTS-based development process Concerning the conclusions drawn in another study [3], we agree that using OTS components in system development is bearing potential danger, especially when using unfamiliar OTS components. Although the risk driven processes proposed in [3] may give benefits, results of RQ1 show, however, that most projects integrate OTS components successfully with traditional processes, such as waterfall, incremental with prototyping, XP, or V-model. Results of RQ2 show that most actual development

Appendix A

179

processes were decided before the decision of using OTS components was made. In addition, results of RQ3 show that company rules pre-defined the main development processes in many OTS-based projects. According to the results of RQ1 to RQ3, we summarize the state-of-the-practice of OTS-based development processes into seven different scenarios. The classification of the scenarios is based on the phase of the make vs. acquire decision and the developers’ familiarity with the OTS candidates. 5.1. Scenarios of planned OTS-based software development In these scenarios (see Figure 4), project members prefer to evaluate the possibility of using OTS components right from the start. Due to the time-to-market, cost, or capacity issues, project managers realize that it is difficult, impossible, or not cost-effective to make everything from scratch. In these scenarios, the first step is to do a make vs. acquire decision (see step A in Figure 4). In case project members decide not to use OTS components, the development process will be the same as the non-COTS based development. If project members decide to use OTS components, they then need to decide the main development process (see step B in Figure 4). To decide the main development process, main issues are the traditional risks in a project, such as the stability of the requirements. One of the other issues is their familiarity with possible OTS candidates. According to the project members’ familiarity with the OTS candidates, we define three scenarios from scenario 1 to 3.

- Scenario 1 – planned OTS-based project with unfamiliar OTS candidates: In this scenario, the project members prefer to use OTS component to provide certain functionalities of the system. However, they are not familiar with the OTS candidate at all. To decide the main development process, the OTS relevant risks, such as the quality of OTS components and the vendor’s support, should be seriously considered. In addition, project members should pay attention to other risks, which are not relevant to an OTS component but the overall project (e.g., which process is more suitable for projects with unclear or unstable requirements? Waterfall or an incremental with prototyping’ model?).

- Scenario 2 – planned OTS-based project with some familiar OTS candidates: In this scenario, the project members have a certain experience on some of the possible OTS candidates. However, there are still certain candidates that they have never used before. In order to decide the main development process, the issues are similar with those of the scenario 1.

- Scenario 3 – planned OTS-based project with familiar OTS candidates: In this scenario, the project members are familiar with all possible candidate components. When deciding the main development process the OTS relevant risks are not as critical as in scenario 1 or 2. Other non-OTS relevant factors, such as company rules or requirement flexibility may be used to decide the main development process. In this scenario, the main development process can be any, such as waterfall, incremental, or XP. It may not need to be changed because the project members are very familiar with the OTS component candidates.

In our investigated projects, only 25% can be classified into scenario 1 to 3, because

they decided their main development processes after they decided to use OTS

Appendix A

180

components. In these projects, 67% can be put into scenario 3, because they used familiar OTS components.

The COTS-based development model in [10] illustrates the differences between COTS-based development and traditional software development, such as new, reduced, or modified activities. The underlying assumption of this model is that the rate or degree of using OTS components is fixed in the very early phases of a project and that the main development process has to be modified accordingly. The model can therefore be used as a reference in scenarios 1 to 3.

Fig. 4. Scenario 1 to 3 5.2. Scenarios of unplanned OTS-based software development

In these scenarios (see Figure 5), the decision of using OTS component was not planned or made in advance, i.e. there is no clear intention of using OTS components in the early phases of such a project. Project members decide on the main development process according to some non-OTS relevant factors, such as company rules and requirement stability. Due to the time-to-market pressure or internal capability, project members start to evaluate the possibility of using OTS components at certain stages of the project. At an industrial seminar [17], several industrial participants mentioned such conditions. According to the project members’ familiarity with the OTS candidates, we classify three scenarios from scenario 4 to 6.

- Scenario 4 – Unplanned OTS-based project with unfamiliar OTS candidates: In this scenario, the project members decided to use OTS components in a late phase of the project, such as detailed design or coding. However, they do not have any previous experience with the possible OTS candidates. Therefore, the use of OTS components may bring several problems for the whole project. They need a second risk-evaluation to investigate whether the development process should be modified concerning the use of OTS components. For example, in

Appendix A

181

projects with pure waterfall processes, the project members may need to add a short prototyping step to evaluate the OTS component and to negotiate the use of OTS component with their customer.

- Scenario 5 – Unplanned OTS-based project with some familiar OTS candidates: In this scenario, the project members decided to use OTS components in a late phase. However, they do not have previous experience with all OTS component candidates. In order to decide the main development process, the issues are similar with those of scenario 4.

- Scenario 6 – Unplanned OTS-based project with familiar OTS candidates: In this scenario, although the project members decided to use OTS components in a late development phase, they have enough experience on the OTS component candidates or are able to hire a person with such experience. In case the project members have used the OTS component many times before, the OTS component can be regarded as comparable to in-house built components. Although the use of OTS component is a little risky due to the same reasons as described in scenarios 4 and 5, this scenario is less risky because of using familiar OTS components. It may not be necessary to do major changes to the current development process.

Fig. 5. Scenario 4 to 6 In our studied project, about 75% of them can be classified into scenario 4 to 6,

because they decided their main development processes before they started to think about using OTS components. In these projects, 65% used familiar OTS components and therefore can be classified into scenario 6.

5.3. The scenario of semi-planned OTS-based software development This scenario is a mix of scenarios 1, 2, or 3 with scenarios 4, 5 or 6, whereby the use of OTS components is planned for some parts of the system. The main development

Appendix A

182

process is decided according to the use of OTS components. However, in a late phase of the project, the project member may need to add some unplanned OTS component due to the same reasons as scenario 4, 5, or 6.

The EPIC approach [1], like the RUP, consists of four phases and each phase consists of one or more EPIC iterations. The information of four spheres of influences, such as stakeholders’ needs, architecture design, risk, and marketplace, was gathered and evaluated in each iteration. The project management, customer requirements, and system architecture will be adjusted according to the trade-offs of the four spheres of influences. Since every iteration includes a complete sub-process to make the trade-off decision and select the OTS component, it is more flexible and suitable to be used as a reference in scenarios 4 to 7.

6. How to select the OTS components The results of this study support our conclusion in the exploratory study [6], and shows that the familiarity-based and internet search-based processes are frequently used in practice. The results also confirm that OTS component selection can be performed in different development phases (requirements, design, or implementation), depending on the project context, especially on the familiarity with possible OTS components, and/or the flexibility of requirements [6]. In addition, results show project members aim at reusing familiar OTS components and selecting them in early phases. In our study, we measured the importance of an OTS component by their contribution on the functionality of the system. Although the results show no correlation between the OTS component’s importance with the phase it was selected, we still suggest project members to select OTS in the earlier phases, especially if the OTS component has a tight coupling with other components.

In different scenarios shown in Section 5, different OTS component selection strategies and processes can be used. In scenarios 1, 2, 4, 5, and 7, some unfamiliar OTS component candidates are going to be used. The selection process for unfamiliar OTS components can either be an internet search with hand-on trial-based, or a formal selection process, such as processes shown in [5], [8], [9], [11], [13], and [14]. In scenarios 3 and 6, all possible OTS component candidates have been used by the project members, the selection process could mainly be familiarity-based.

In scenarios 1 to 3, the use of OTS component is well-planned. In general, the OTS component will be selected in the early phase of the project in these scenarios. Selecting OTS component in the early phase can help to design the software architecture with consideration on constrains of the OTS components. Therefore, it is easier to avoid the mismatch between the OTS components and other components in the system.

In scenarios 4 to7, the OTS components are usually be evaluated at a phase which most of the other components are already integrated in the system. Since the goal is to integrate the OTS component into the current system and to check its integration thoroughly, integrate the demo version of the OTS component and do a hand-on trial is necessary. In case the OTS component was selected in the very late stage of the project, it is better to select a component that has a loose coupling with the other components. OTS components tightly coupled with existing components may impose a high number of changes onto the system.

Appendix A

183

7. Threats to validity To avoid possible threats to the construct validity of this study, we performed a pre-test to verify the questionnaire. In addition, we use a lot of effort to ensure the representative of the sample. Detailed discussion on other validities issue of this study are reported at [4], [7]. 8. Conclusion and future work This paper has presented a state-of-the-practice survey on the process issues in OTS-based development. We have studied 133 projects from Norway, Italy, and Germany. The main findings are:

- The actual OTS-based development process was the traditional process with OTS-specific activities. The development process was dominated by the company/department rule instead of the decision of using OTS components.

- The actual OTS component selection can be done in different phases. The main evaluation processes are familiarity-based or Internet search with hands-on-trial-based. The phase to select OTS component has relationship with the project members’ familiarity.


The main limitation of this study is that it is a state-of-the-practice survey. We are

going to perform more detailed case studies to get detailed information in projects with different contexts. The intention is to deepen our understanding the process improvement in OTS component-based development and to verify our proposals.

7. References 1. Albert, C. and Brownsword, L.: Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview. Software Engineering Institute, Pittsburgh, (2002), http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html.

2. Brownsword, L., Oberndorf, T., and Sledge, C.: Developing New Processes for COTS-Based Systems. IEEE Software, July/August (2000), 17(4):48-55.

3. Boehm, B. W. and Abts, C.: COTS integration: Plug and Pray? IEEE Computer, Jan. (1999), 32(1):135-138.

4. Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and Morisio, M.: Reflections on Conducting an International CBSE Survey in ICT Industry. Proc. of the 4th Int. Symposium on Empirical Software Engineering, Noosa Heads, Australia, Nov. (2005) 207-216.

5. Kontio, J.: A Case Study in Applying a Systematic Method for COTS Selection. Proc. of the 18th Int. Conf. on Software Engineering, Berlin, Germany, March (1996) 201-209.

6. Li, J., Bjørnson, F. O., Conradi, R. and Kampenes, V. B.: An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT

Appendix A

184

Industry. Proc. of the 10th IEEE Int. Metrics Symposium (Metrics'04), Chicago, USA, Sept. (2004) 72-83.

7. Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and Morisio, M.: Validation of New Theses on Off-The-Shelf Component Based Development. Proc. of the 11th IEEE Int. Metrics Symposium (Metrics’05), Como, Italy, Sept. (2005) 26.

8. Tran, V., Liu, D. B., and Hummel, B.: Component Based Systems Development: Challenges and Lessons Learned. Proc. of the 8th IEEE Int. Workshop on Software Technology and Engineering Practice, London, UK, (1997) 452-462.

9. MacCrimmon, K. R.: An Overview of Multiple Objective Decision Making. Proc. of the Multiple Criteria Decision Making, University of South Carolina Press, (1973) 18-44.

10. Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S. E.: Investigating and Improving a COTS-Based Software Development Process. Proc. of the 22nd Int. Conf. on Software Engineering, Limerick, Ireland, June (2000) 31-40.

11. Morisio, M. and Tsoukias, A.: IusWare: a Methodology for the Evaluation and Selection of Software Products. IEE Proceedings-Software Engineering, June (1997), 144(3):162-174.

12. Ncube, C. and Dean, J. C.: The Limitation of Current Decision-Making Techniques in the Procurement of COTS Software Components. Proc. of the 1st Int. Conf. on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Feb. (2002), LNCS Vol. 2255, Springer-Verlag Berlin Heidelberg New York 176-187.

13. Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N.: A Method for Efficient Measurement-based COTS Assessment and Selection - Method Description and Evaluation Results. Proc. of the 7th IEEE Int. Software Metrics Symposium, London, England, Apr. (2001) 285-297.

14. Saaty, T. L.: How to Make a Decision: The Analytic Hierarchy Process (AHP). European Journal of Operational Research, (1990), 48(1): 9-26.

15. SEI COTS-Based Initiative Description. Software Engineering Institute, Pittsburgh, (2004), http://www.sei.cmu.edu/cbs/cbs_description.html

16. Torchiano, M. and Morisio, M.: Overlooked Facts on COTS-based Development. IEEE Software, March/April (2004), 21(2):88-93.

17. Li, J. Conradi, R. Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and Morisio, M.: Barriers to Disseminating Off-The-Shelf Based Development Theories to IT Industry. Proc. of the ICSE2005/MPEC workshop, St. Louis, Missouri, USA, May (2005), 1-4.

18. Kent, B.: Extreme Programming Explained: Embrace Change. (1999) Addison-Wesley.

19. German V-model: http://www.v-modell.iabg.de/#AU250

Appendix A

185

P5. Risk Management in OTS-Based Development

A State-of-the-Practice Study on Risk Management in Off-the-Shelf Component-Based Development

Jingyue Li1, Marco Torchiano2, Reidar Conradi1, Odd Petter N. Slyngstad1, and Christian Bunse3

1 Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU),


2 Dip. Automatica e Informatica, Politecnico di Torino Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy

[email protected] 3 Fraunhofer IESE, Fraunhoferplatz 1,

D-67663 Kaiserslautern, Germany [email protected]

INDEX TERMS D.2.13. Software Engineering/Reusable Software, D.2.9. Software Engineering/Management, D.2.18. Software Engineering/Software Engineering Process ABSTRACT: This paper reports and discusses an international survey on risk management in OTS (off-the-shelf) component-based software development. Most studies on project risk management in this area are still limited to theoretical proposals or case studies in specific project contexts. Without experience derived from large scale verifications in different project environments, it is difficult for project managers to use available risk management guidelines properly and efficiently. This study investigates performed risk management activities and their results in OTS component-based development. Data from 133 projects in Norway, Italy, and Germany have been collected using the stratified-random sample selection strategy. Results reveal that putting more effort into learning OTS components, performing the integration testing early, evaluating the quality of OTS components thoroughly, and following the update of OTS components have helped to mitigate corresponding risks. However, some problems, such as wrong estimation of the integration efforts and inefficient debugging, still happen frequently in practice and need further investigations. 1 Introduction

Long term success in commercial software development is becoming increasingly challenging. As customers expect better, earlier, and cheaper software, software companies are finding it increasingly difficult to protect their technology investments and maintain productivity levels. The software industry is therefore increasingly turning to approaches which offer a larger than normal return on software development effort. Component-based software development (CBSD) [Szyperski02] is one of these approaches. In addition to reusing in-house built components, software companies are

Appendix A

186

using an increasing amount of COTS (Commercial-Off-The-Shelf) and OSS (Open Source Software) components. However, using such external components introduces a number of additional risks [Rose03], [Boehm03], [Voas98a], [Voas98b]. Thus, project managers have to identify and evaluate possible risks before deciding to acquire an external component, instead of developing an in-house component. Although several risks and risk management activities in OTS-based development have been identified from case studies [Rose03], [Boehm03], [Abts00], [Voas98a], [Voas98b], [Kotonya01], [Faa03], [Fitzgerald04], [Lawton02], [Vitharana03], [Ruffin04], few subsequent empirical studies have verified their conclusions. As a result, software project managers have few effective and well-proven guidelines to identify the relative effects of the various risks, and to select the effective activities to reduce such risks.

In this study, we did postmortem investigations on problems occurred and the risk management activities performed in 133 finished OTS component-based projects. The first research question was to find out which problems occurred more frequently than others. The second research question was to investigate which risk management activities have been performed frequently, and the third research question was to investigate the correlation between the (lower) occurrence of the problems and the application of risk mitigation activities. It was intended to find out which performed risk management activities were most effective on avoiding corresponding risks.

Results of this study show that the most frequent problems in OTS-based development are cost estimation and defect localization. It is notable that problems related to the quality of the OTS components do not appear as frequent as commonly assumed. To manage the risks in OTS-based development, it is critical to study and understand the OTS components thoroughly, to evaluate their quality by black-box testing, to keep track with the update of these OTS components, and to manage and share the knowledge on these OTS components.

The remainder of the paper is structured as follows: Section 2 presents work related to this study, while Section 3 discusses the research design and related research questions. Profiles of the collected samples are presented in Section 4 and the empirical results concerning the research questions are given by Section 5. Section 6 presents discussions on the study results, and Section 7 provides conclusions and open issues. 2 Related Work 2.1 Risk Management in Software Development

Risks are factors that may adversely affect a project, unless project managers take appropriate countermeasures. Risk management includes two primary steps each with three subsidiary steps as described by Boehm [Boehm91], and shown in Figure 1. Risk identification produces lists of project-specific risk items likely to compromise the success of a project. Risk analysis assesses the loss probability and loss magnitude for each identified risk item. Risk prioritization ranks the risk items identified and analysed. Risk management planning helps prepare to address each risk item. Risk resolution produces a situation in which the risk items are eliminated or otherwise resolved. Risk monitoring involves tracking the progress of a project towards resolving its risk items and taking corrective activities where appropriate.

Appendix A

187

Fig.1. Risk Management Framework

In order to identify risks, a list of possible project risks must be produced. Boehm [Boehm89] lists the top 10 risks by probing several large software projects and their common risks. Barki et al. [Barki93] identified 35 variables, which are subsequently used as a basis for creating scales in a project risk assessment instrument. The Software Engineering Institute (SEI) [Carr93] has contributed significantly to the literature with the SEI Taxonomy-Based risk identification instrument, which contains 194 questions. However, the notion of building single all-encompassing risk taxonomy to be globally used by all software developers is probably unrealistic. Thus, we may need different risk taxonomies in different project contexts [Moynihan97].

A few other studies have classified software risk items, i.e., consider software risks along several dimensions and have provided some empirically founded insights of typical software risks and their variation. The risks can be classified into technical, organizational, and environmental [Sherer95]. The possible risks can also be summarized into a hierarchical holographic modelling (HHM) framework [Chittister96]. Ropponen and Lyytinen [Ropponen00] empirically delineate six components of software development risks, such as scheduling and timing risks, system functionality risks, subcontracting risks, requirement management risks, resource usage and performance risks, and personnel management risks. Keil et al. [Keil98] extracted the experience of more than 40 software project managers and identified a universal set of risk factors, such as customer mandate, scope and requirements, environment, and execution. A following study investigated the relationship between these factors and their effect on the result of the project [Wallace04].

During the risk analysis process, project managers have to consider each identified risk and make a judgment about its probability to occur and seriousness. The project managers can tabulate the results of this analysis process using a table ordered according to the serious of the risk [Sommerville04]. The estimation of the seriousness and the rank of the risk should be combined with cost estimation [Madachy97],

Appendix A

188

[Kansala97]. Once the risks have been analysed and ranked, the project managers need to assess which are the most significant risks.

The purpose of identifying and analysing the risks is to manage these possible risks. The possible risk management activities have been proposed from processes [Boehm88], organization [Roppoen00], [Gemmer97], and technology [Hecht04] viewpoints. However, many researchers advocate rather than investigate. As a result, some advocated concepts are worth less than their advocates believe and there is a shortage of evaluative research to help determine the actual value of new tools and techniques [Glass01]. Although risk management proposals of the in-house software development have been verified by a survey-based research [Verner05], few studies have been performed to empirically investigate the risk management proposals in using the OTS components.

2.2 Specific Risk Management in OTS Component-Based Development

In addition to the classical risks associated with developing large systems, OTS components requires managers to modify their typical mitigation activities for some of the classic risks and to develop new mitigation activities for risks that are particular to the use of OTS components in a system.

Vigder et al. [Vigder96] investigated the specific risk issues in the COTS-based development and point out the fundamental differences between COTS-based development and building in-house are:

- COTS-based development may need infrastructure earlier to demonstrate and

validate the COTS component. - The COTS component may dictate standards, architecture, and design. - The COTS component may influence work flow. - Picking the wrong COTS component may be more expensive than fixing

problems in the in-house built software. - Issue resolution processes need to be in place earlier to resolve COTS

component issues. - Issue resolution processes may be more complicated because of the addition of

the vendor and possible incompatibilities between the vendor’s practices and yours.

Boehm et al. [Boehm03] summarized experience from their COTS-based university

projects. Examples of the summarized risks and corresponding risk management activities (In brackets) are:

- Requirement changes and mismatches (Prototyping, win-win negotiation among

all stakeholders). - Overly optimistic expectation on COTS quality attributes (Significant quality

features must be tested). - Overly optimistic COTS component learning curve (A most likely COTS

component learning curve must be accessed during planning and scheduling). - Version upgrade may result in retailoring of the COTS component (Ensure the

new version still can implement the capability of the old version).

Appendix A

189

The software engineering resource centre at the Federal Aviation Administration (FAA) [Faa03] summarized 10 risk factors in COTS-based development and proposed corresponding risk mitigation activities. The COTS lessons-learned in a repository made by Basili et al. [Basili02] summarized the possible risk management activities based on case studies. In these studies, COTS software can be either components or very large packages, such as ERP, CRM, Data Bases, or Operating Systems.

A few other studies contribute to investigate risk management in OSS-based development. For example, Ruffin and Ebert [Ruffin04] point out the possible licensing problem of using OSS as follows:

- If you reproduce or sell the product without the licensor’s permission, the

licensor might claim damages or force you to end the product’s production, delivery and sale.

- Another exposure is infringement of third party’s patents or other intellectual-property rights.

- A third risk area is the use of OSS in development activities when OSS is a tool. The developed product may need to follow the licences of the OSS tool, i.e., the code should also be open.

Fitzgerald [Fitzgerald04] mentioned that organizations should not expect

maintenance and support to be available at a lower cost that would apply for commercial software, even though OSS may be available at little or no cost. Other researchers mention that freely available source code addresses two typical concerns with using COTS components: unknown implementation quality and long-term vendor support [Madanmohan04].

Other researchers have focused only on the component-based development without telling the differences between in-house built COTS, or OSS components. Bass et al. [Bass00] mention inhibitors in component-based development as being: lack of available components, lack of standards for component technology, lack of certified components, and lack of engineering methods. Some challenges in each development phase and for a project as a whole, such as management, component-based software life cycle, composition predictability, component certification, component configuration, and tool support, are discussed in detail in [Crnkovic02], [Ghosh02], [Jacobson97]. Kotonya and Awais [Kotonya01] put risk management in component-based development into the risk-weighting scheme of [Moynihan97] and listed several questions in each risk item to help project managers to quantitatively assess the risks.

Some studies focus only on COTS components. Rose [Rose03] classifies possible risks in COTS component-based development based on the different phases. Examples of possible risks and corresponding risk management activities (In brackets) are:

- Lack of skilled personnel (Ensure that there is an appropriate number of senior

technical personnel assigned to the project). - Architecture mismatch (Design the architecture first and select COTS

component that best fits the architecture). - Product obsolescence (Maintain a continual watch on the marketplace looking

for substitute products as well as new technologies that might replace older ones).

Appendix A

190

- COTS components have asynchronous update cycles with the COTS-based system (Involve senior analysts directly involved in the analysis of the upgrades and derive effective field upgrade procedure and schedules)

The CBSEnet project [CBSEnet04] investigated the state-of-the practice in COTS

component-based development and summarized the challenges from business, information technology, and legal perspectives. The study performed by Li et al. [Li04] summarized the problems encountered and presented good practices from finished COTS component-based projects. Examples of the problems encountered are:

- Could not negotiate requirements with the customer. - The learning effort was underestimated. - It was difficult to locate defects. - The COTS component could not be deployed as advocated by the vendor. - Examples of good practices are: - Integrate unfamiliar COTS component first. - Do integration testing early and incrementally. - Investigate COTS component market and prepare to replace obsolescent

components.

Since the OTS components involve both COTS and OSS components, risks and risk management activities relevant to OTS components-based development are therefore the subset of risks and risk management in component-based development, COTS-based, and Open Source based development, as shown in Figure 2.

Fig. 2 OTS Component-based Development Risks

Although the mentioned risk management activities have been proposed to avoid the corresponding risks in OTS component-based development, they are generally based either on common sense or on small and non-representative case studies. Few empirical studies have subsequently validated their conclusions. In addition, it is important to compare the results of the performed risk management activities and to find out the most efficient ones that mitigated a specific risk. The results can help project managers put their (limited) resources on the most efficient risk mitigation activities to avoid identified risks.

Appendix A

191

3 Research Design In this study, we have used the conceptual measurement framework proposed by Bush and Fenton [Bush90]. The study includes one entity (i.e., the project) and two attributes (i.e., possible risks of the project and possible risk mitigation activities to avoid these risks). There are also two relationships, i.e., risks are related to a project and risk mitigation activities that IMPACTS risks. Before the start of a project, project managers must upfront estimate how often the risks may happen, how much effort they need to put in to certain risk mitigation activities, and how well the risks may be managed. The relationships are shown in Figure 3.

Fig. 3 Risk management before a project starts

To help project managers predict the possibility of risks and to plan the risk management efficiently before a project starts, it is necessary to summarize practices and results of risk management from finished projects. The research framework, as shown in Figure 4, of this study is designed to do posteriori measurements (after the project is finished) on the occurrence of project risks and the impacts of performed risk management activities on these risks.

The first research questions RQ1, as shown in Figure 4, is to investigate the actual risk incidence, i.e., how often had problems occurred in practice? The second research question RQ2, is to investigate the execution of the risk management activities in projects, i.e., how often had certain risk management activities been performed in practice? The third research question RQ3 is to examine the effect of performed risk management activities, i.e., how well the performed risk mitigation activities had reduced corresponding risks? For a specific risk, we are interested in the most effective mitigation activities.

Appendix A

192

Fig. 4 Research framework

In our study, we selected several typical risks and possible risk management activities proposed in [Boehm03], [Faa03], [Li04], [Rose03], [CBSEnet04], [Vitharana03], [Lawton02], [Fitzgerald04] [Torchiano04]. The selection criteria are:

- First, the risks and risk management activities should address the common issues

in the CBD, COTS-based development, and OSS-based development, as shown in Figure 2.

- OTS component-based development encompasses three primary types of stakeholders, namely component developers, application assemblers/integrators, and customer. Different stakeholders have different risks and challenges [Vitharana03]. We have selected risks and challenges that are relevant only to the application assemblers/integrators.

- The selected risk management activities in our study focus mainly on process and organizational issues, instead of technical methods, such as wrapping, composition prediction, and testing.

The selected typical risks in OTS components-based cover different phases of a

project, as shown in Table 1. The typical risk management activities are shown in Table 2. In addition, the results from process improvement studies show that one added role in OTS component-based development, i.e., that of a knowledge keeper (dedicated project personnel to keep track of OTS component knowledge/information), may help to mitigate the risks in OTS component-based development [Li04], [Morisio00], [Rose03]. We therefore propose the existence of knowledge keeper as one possible risk management activity.

Appendix A

193

Table 1 Typical Risks in OTS Component-Based Development

Table 2 Typical Risk Management Activities in OTS Component-Based Development

Appendix A

194

The data was collected with a web-based questionnaire using the SESE tool [Sese04] at Simula Research Laboratory. To answer research questions RQ1 to RQ3, we listed the risks and risk management activities shown in Table 1 and 2. Respondents are asked if these risks and risk management activities had occurred. The alternatives are “don’t agree at all”, “hardly agree”, “agree somewhat”, “agree mostly”, “strongly agree”, or “don’t know”. We assign an ordinal number from 1 to 5 to the above alternatives (5 meaning strongly agree). In addition, we asked if there are any dedicated personnel (or knowledge keeper) in the actual project to keep track of OTS component knowledge/information. The answer is “yes” or “no”.

3.1 Data Collection 3.1.1 Key definitions Questionnaire Design

The components investigate in our study have larger granularity than the component in “OO” theory [Crnkovic02]. The main definitions used in the questionnaire are:

- Component: Program units of independent production, acquisition, and deployment and which can be composed into a functioning system. We limit ourselves to components that have been explicitly decided either to be built from scratch or to be acquired externally as an OTS-component. That is, components that are not shipped with the operating system, not provided by the development environment, and not included in any pre-existing platform. That is, platform (“commodity”) software is not considered, e.g., an OS like Linux, DBMSes, various servers, or similar software. Furthermore, components usually follow some component model, as expressed by the COM, CORBA, EJB, .Net, or Web Service standards, or they can be a C++/Java library.

- OTS component: A component provided (by a so-called provider) from a commercial vendor or the Open Source community. An OTS component may come with certain obligations, e.g., payment or licensing terms. An OTS component is not controllable in terms of provided features and their evolution. An OTS component is mainly used as closed source, i.e., no source code is usually modified, even if it may be available.

3.1.2 Questionnaire structure The questionnaire includes six sections. The three sections relevant to this study are:

- Background questions to collect information on the company, project, and respondents.

- Main questions about risk and risk management. - Questions to collect information about concrete OTS components used in the

project. 3.1.3 Data collection procedure

We define the unit of this study as a completed software development project using OTS components. The projects were selected based on two criteria:

Appendix A

195



We used random sample selection strategy to collect representative companies and convenient strategy to select projects in a company in Norway, Italy and Germany:

- In Norway, we gathered a company list from the Norwegian Census Bureau

(SSB) [Ssb02] because they have a pool of large, medium, and small companies. Based on the number of employees, we selected the 115 largest IT companies (100 IT companies and 15 IT departments in the largest 3 companies in 5 other sectors), 200 medium-sized software companies (20-99 employees), and 100 small companies (5-19 employees) as the original contacting list.

- In Italy, we first got 43580 software companies from the “yellow pages”. We then randomly selected companies from this list. For these randomly selected companies, we read their web-site to ensure they really are software companies. 196 companies were finally clarified as software companies, and were included in the original contact list.


To avoid bias in the data collection procedure, we made a guideline on how to

contact respondents before the survey started. All countries – Norway, Italy, and Germany – followed the process from the guidelines explained below.

Possible respondents were contacted first by telephone. If the candidate respondents had suitable OTS-based projects and would like to join our study, a username and password of the survey tool and an electronic version of the questionnaire were sent to them. The respondents could use either the SESE web tool [Sese04] or the electronic version (.doc file) to fill in the questionnaire. The respondents who didn’t want to answer the questionnaire were also registered. We logged the main reasons of non-response, such as “no software development”, “no OTS-based projects”, and “busy”. Most respondents used the Web tool to fill in the questionnaire. However, phone interview were used in Germany because of the confidential concerns in some companies. In total, we contacted 1087 companies in three countries. In these companies, 425 of them have experience on OTS component-based development. From these relevant companies, we managed to collect data from 127 companies. This study is probably the first software engineering survey using census type data, and has revealed that the entire sampling and contact process can be unexpectedly expensive. The study reveals that, at best, we can achieve a stratified-random sample of ICT companies, followed by a convenience sample of relevant projects. Detailed discussions on sample selection, the contact process and the response rate are reported in [Conradi05].

Appendix A

196

3.2 Data Analysis

Concerning research questions RQ1 and RQ2, we compared the occurrences of typical risks and performed risk management activities. The results are visualized with the boxplot graph.

The intention of research question RQ3 is to find out the correlations between occurrences of risks and risk management activities performed. If there is a negative correlation between the risk management activity and the occurrence of possible problems, we assume that the performed risk management activity actually helped to mitigate the possible problems. Since several performed risk management activities may have helped to mitigate the same risk, we are only interested in finding out the most effective risk mitigation activities.

For research question RQ3, we decided to use Nonlinear Canonical Correlation Analysis (Nonlinear CCA) [Gifi90], [Michailidis98], because we face a nonlinear CCA problem with a set of explanatory variable vector defined by a nominal (knowledge keeper) variable and several ordinal (risk management activities performed) variables and a dependent variable matrix defined by a series of ordinal (risk occurrences) scales. The linear combination on the explanatory variable side is undefined, because we have no metric to quantify the categories of each variable. The linear combination of the variables on the dependent side is also undefined, because the categories of each variable can be re-scaled by any nonlinear function the preserves monotonicity. Thus, we need to optimally scale or quantify the variables while simultaneously solving the traditional linear CCA problem of finding weights for each explanatory variable. Nonlinear CCA corresponds to categorical canonical correlation analysis with optimal scaling. The detailed introductions and discussions on using the Nonlinear CCA instead of other analysis methods in this study are given in the Appendix A.

The nonlinear CCA used in this study is the OVERALS in SPSS version 14.0 [Spss06]. The outputs of the OVERALS in SPSS include several parts. The most important ones are:

- The fit and loss values tell how well the nonlinear CCA solution fits the

optimally quantified data with respect to the association between the sets. - The plot of component loading for the data. When there is no missing data, the

component loading is equivalent to the Pearson correlation between the quantified variables and the canonical variate. The distance from the origin to each variable point approximates the importance of that variable.

- The object score plots and centroids plots provide the remainder of the information we need to interpret a nonlinear CCA solution.

4 Collected Data

We have gathered results from 133 projects (47 from Norway, 48 from Germany, and 38 from Italy) in 127 companies. In general, we selected one project from each company. However, we selected more than one projects in three Norwegian IT companies because those companies have many OTS-based projects wanted to contribute more experience to this study.

Appendix A

197

The responding companies involved small (5-19 employees), medium (20-99 employees) and large (100 or more employees) ones. The main business areas of the sampled companies also cover different types and the final systems produced by the 133 projects cover various application domains. Detailed profiles of the selected companies and projects are shown in [Li05].

In the selected 133 projects, 83 used only COTS components, 44 used only OSS components, and six used both COTS and OSS components. Most respondents of the 133 projects have a solid IT background. More than 90% of them are IT managers, project managers, or software architects. Most of them have more than 2 year experience with OTS-based development. All of them have at least a bachelor degree in informatics, computer science, or telecom.

The OTS components used in the investigated projects include components that follow the COM, DCOM, and EJB models. There are also some components in the format of C++ or Java libraries. Examples of the selected components are listed in the Table 1 in Appendix B.

5 Results of Research Questions 5.1 RQ1: Occurrences of Risks in OTS Component-Based Projects

As the occurrences of possible risks are measured with an ordinal scale, we used boxplot to illustrate the distribution of the risk occurrences. The results are in Figure 5 and show that the most frequent risks are R2 (R_IntEff) and R8 (R_DefLoc). It means that the integration effort was usually wrongly estimated. In addition, it was difficult to figure out whether the defects are inside or outside the OTS components. The least frequent risks are R3 (R_ComRel), R4 (R_ComSec) and R5 (R_ComPerf), i.e., the problems relevant to the negative effect of the OTS components on the quality of the whole system.

R_MissSupp

R_ProvInf

R_ComUpd

R_MaintPlan

R_ComComp

R_DefLoc

R_ReqRen

R_ComAda

R_ComPerf

R_ComSec

R_ComRel

R_IntEff

R_SelEff

4

2

Fig. 5 Occurrences of Risks

Appendix A

198

5.2 RQ2: Performed risk management Activities

The occurrence of performed risk management activities are measured by an ordinal scale too. Figure 6 presents a boxplot illustrating the distribution of the risk management activities.

Rm_ProvWtch

Rm_MktWtch

Rm_UpdFoll

Rm_IncTest

Rm_UnfFirst

Rm_BBEff

Rm_LearnEff

Rm_SelQual

Rm_SelArch

Rm_CstSel

Rm_CstMkBuy

4

2

Fig. 6 Occurrences of Risk Management Activities From Figure 6, we can see that the activities Rm4 (Rm_SelQual), Rm5

(Rm_LearnEff), Rm7 (Rm_UnfFirst), Rm8 (Rm_IncTest) and Rm9 (Rm_UpdFoll) were used more frequently than others. It means that developers have seriously evaluated the quality of the OTS components in the selection phase. They have considered the possible learning effort in the effort estimation. They have done integration testing as early as possible and they usually have internal expert to follow the update of the OTS components. However, the activities relevant to the users, such as Rm1 (Rm_CstMkBuy) and Rm2 (Rm_CstSel), were least frequently used. 5.3 RQ3: Correlations between the Risks and Risk Management Activities

The analysis is organized in six parts (See section 5.3.1 to 5.3.6). In the first section, we analyze the correlation between the effort estimation risks, i.e., R1 and R2, with the risk management activities Rm1 to Rm8, i.e., for development. Although Rm1 to Rm8 have been proposed as effective activities to mitigate possible risks in OTS component-based development, there are few empirical studies to verify and compare which risks they can most effectively mitigate. The target of our analysis is to find out which of the activities Rm1 to Rm8 are most effective to avoid overly optimistic effort estimation. Due to the same rationale, we analyzed the correlation between the quality risks, i.e., R3 to R5, and the risk management activities Rm1 to Rm8 in the second section. In the third section, we analyzed the correlation between the requirements relevant risks, i.e., R6 and R7,

Appendix A

199

with the same Rm1 to Rm8. In the forth section, we analyzed the debugging and deployment risks, i.e., R8 and R9, also with Rm1 to Rm8.

In the fifth section, we analyze the maintenance risks, i.e., R10 and R11, with Rm9 to Rm13, i.e., for maintenance. In addition, we want to investigate whether the projects involving a knowledge keeper had less maintenance problems than projects without this role. Therefore, the existence of a knowledge keeper was also added as a risk management variable. In the sixth section, we analyzed the correlation between the vendor support risks, i.e., R12 and R13, with Rm9 to Rm13 and the existence of the knowledge keeper.

5.3.1 Effort estimation risks – R1 and R2

The result of the correlations between the effort estimation risks, i.e., R1 (R_SelEff) and R2 (R_IntEff) and risk management activities (Rm1 to Rm8) are analyzed with OVERALS (two dimensions, some variables not multiple nominal, multiple steps, random initial configuration) in SPSS. We selected to use two dimensions because the upper bound of the number of the dimensions, i.e., the number of the canonical variates, is the minimum of the number of the variables in each set [Gifi90]. In our study, the minimum number of the variable in our analysis is two. Another reason for selecting the two-dimension analysis is due to the interpretative convenience of two-dimension plots [Gifi90]. The risk management activities (i.e., Rm1 to Rm8) are given to the first set and each is defined as an ordinal variable with scale from 1 to 5. The effort estimation risks (i.e., R1 and R2) are given as the second set and each is also defined as ordinal variable with scale from 1 to 5.

The purpose of the analysis is to find out the most negative correlation between the risk management activities and the occurrence of possible problems. The analysis includes two steps:

- Step 1: For each possible problem, for example R1 (R_SelEff), we first check

the dimension (i.e., dimension 1 or 2) that R1 has been mostly loaded. If R1 is loaded more in one dimension, for example dimension 1, it means that the correlation between R1 and the dimension 1 is stronger than the correlation between the R1 and dimension 2.

- Step 2: In the most loaded dimension of a risk, for example R1 (R_SelEff), we then check the component loading to find out the risk management actions (i.e., Rms), which are in the peak positions of the opposite domain of R1.

The summary of OVERALS analysis show that 53% of the actual fit is accounted for

by the first dimension, and the 47% of the actual fit is accounted for by the second dimension. The component loadings plots are shown in Figure 7 and detailed component loading data are shown in Table 2 in Appendix B.

The results reveal that R1 (R_SelEff) is much more loaded (-.844) in the first dimension (i.e., the horizontal axis) than (-.496) in the second dimension (i.e., the vertical axis). R1 (R_SelEff) is in the negative domain of the first dimension and the Rm5 (Rm_LearnEff) is in the peak of its positive domain.

Appendix A

200

R2 (R_IntEff) is also much more loaded (-.859) in the first dimension than (-.424) in the second dimension. R2 (R_IntEff) is also in the negative domain of the first dimension and the Rm5 (Rm_LearnEff) is in the peak of its positive domain.

The results illustrate that seriously taking the learning effort in consideration and giving enough leeway on the learning curve helped to mitigate the effort estimation risks.

Fig. 7 Component loading plots of the cost estimation risks

5.3.2 Quality risks – R3 to R5

For the quality risks, the results of the OVERALS analysis show that 51% of the actual fit is accounted for by the first dimension and 49% by the second dimension. The component loading plots are shown in Figure 8 and detailed component loading is shown in Table 3 in Appendix B.

Results show that R3 (R_ComRel) is much more loaded (-.530) in the first dimension (i.e., the horizontal axis) than (.264) in the second dimension (i.e., the vertical axis). R4 (R_ComSec) is also much more loaded (-.643) in the first dimension than (-.258) in the second dimension. However, R5 (R_ComPerf) is loaded weakly both in the first dimension (.087) and the second dimension (.072).

Both R3 (R_ComRel) and R4 (R_ComSec) are in the negative domain of the first dimension and the Rm8 (Rm_IncTest) is in the peak (with loading .456) of its positive domain. In addition, Rm6 (Rm_BBEff) is very close (i.e., with loading .435, which is 4.6% less than Rm8) to Rm8 (Rm_IncTest) in the peak of the positive domain of the first dimension.

The results illustrate that early incremental testing and black-box testing of the OTS components in the selection phases helped to increase the reliability and security of the OTS component-based system. However, none of our investigated risk management activities helped to avoid the OTS components’ negative effect on the system performance.

Appendix A

201

Fig. 8 Component loading plots of the quality risk

5.3.3 Requirement risks – R6 and R7

For the requirements risks, the results of the OVERALS show that 52% of the actual fit is accounted for by the first dimension and the 48% by the second dimension. The component loading is shown in Figure 9 and detailed component loading data is shown in Table 4 in Appendix B.

Results show that R6 (R_ComAda) is loaded strongly both in the first dimension (.647) and in the second dimension (-.620). R7 (R_ReqRen) is also loaded strongly both in the first dimension (-.553) and the second dimension (-.761).

In the first dimension, R6 (R_ComAda) is in the positive domain and Rm6 (Rm_BBEff) is loaded (-.428) in the peak of the negative domain. In addition, Rm5 (Rm_LearnEff) is very close (i.e., with loading -.408, which is 4.6% less than Rm6) to Rm6 (Rm_BBEff) in the peak of the negative domain of the first dimension. In the second dimension, R6 (R_ComAda) is in the negative domain and Rm4 (Rm_SelQual) is in the peak of the positive domain.

R7 (R_ReqRen) is in the negative domain of the first dimension and Rm7 (Rm_UnfFirst) is in the peak of the positive domain. In the second dimension, R7 (R_ReqRen) is in the negative domain and the peak in the positive domain is Rm4 (Rm_SelQual). Moreover, the results show that R6 (R_ComAda) and R7 (R_ReqRen) are in different domains of the first dimension.

The results illustrate that evaluating the quality of OTS components carefully and doing a thorough black-box testing in the selection phase may help to reduce the effect of requirements changes. Integrating the unfamiliar OTS components first helped to facilitate requirements negotiation with the customer. If the developers have possibilities to (re) negotiate the requirements with their customer, it will also avoid the occurrences of OTS components being unable to satisfy requirements changes.

Appendix A

202

Fig. 9. Component loading plots of the requirements risks

5.3.4 Debugging and deployment risks – R8 and R9

For the debugging and deployment risks, the results of the OVERALS show that 52% of the actual fit is accounted for by the first dimension and the 48% by the second dimension. The component loading plots are shown in Figure 10 and detailed component loading data is shown in Table 5 in Appendix B.

Results show that R8 (R_DefLoc) is much more loaded in the second dimension (.961) than in the first dimension (.223). R9 (R_ComComp) is much more loaded in the first dimension (-.829) than in the second dimension (.303).

R8 (R_DefLoc) is in the positive domain of the second dimension and Rm5 (Rm_LearnEff) is in the peak of the negative domain. R9 (R_ComComp) is in the negative domain of the first dimension and Rm8 (Rm_IncTest) is in the peak of the positive domain.

The results illustrate that more effort on learning the OTS components helped to solve the most frequent risk, i.e., identifying whether the bugs were inside or outside the OTS components. Starting the integration testing early helped to mitigate the possible deployment problems.

Appendix A

203

Fig. 10 Component loading plots of the deployment and debugging risks 5.3.5 Maintenance risks – R10 and R11

For the maintenance risks, we used the same methods as others. However, the knowledge keeper variable is added as a single nominal with the value “Yes” (coded as 1) and “No” (coded as 2). The knowledge keeper variable is added in the same set as Rm9 to Rm11. The maintenance risks, i.e., R10 (R_MaintPlan) and R11 (R_CompUpd) are added in the second set. The results of the OVERALS show that 53% of the actual fit is accounted for by the first dimension and the 47% by the second dimension. The component loading plots are shown in Figure 11 and detailed component loading is shown in Table 6 in Appendix B.

Results show that R10 (R_MaintPlan) is loaded weakly both in the first dimension (.286) and in the second dimension (.072). R11 (R_CompUpd) is much more loaded in the first dimension (-.622) than the second dimension (.158).

The R11 (R_CompUpd) is loaded in the negative domain of the first dimension and Rm9 (Rm_UpdFoll) is in the peak of the corresponding positive domain. It illustrates that local experts following the update of the OTS component helped to make the system up-to-date with the newest OTS components.

Appendix A

204

Fig. 11. Component loading plots of the maintenance risk 5.3.6 Provider support risks – R12 and R13

For the provider support risks, we used the same methods as the maintenance risks. The knowledge keeper variable is also added in the same set as Rm9 to Rm11. The provider support risks, i.e., R12 (R_ProvInf) and R13 (R_MissSupp) are added in the second set. The results of the OVERALS show that 52% of the actual fit is accounted for by the first dimension and the 48% by the second dimension. The component loading plots are shown in Figure 12. The detailed component loading data is shown in Table 7 in Appendix B.

It shows that the risk R12 (R_ProvInf) is much more loaded in the first dimension (-.839) than in the second dimension (-.012). The R13 (R_MissSupp) is much more loaded in the second dimension (-.624) than the first dimension (-.117).

R12 (R_ProvInf) is in the negative domain of the first dimension. The knowledge keeper is also in the peak of the negative domain. The analysis on the centroids of the variable “knowledge keeper” shows that the projects without a knowledge keeper (code 2 in the Figure 1 in the Appendix B) are in the lower left domain as the project with high occurrences of risk R12 (coded 5 or 4 in the Figure 2 in Appendix B). It means that projects without the knowledge keeper have more problems on getting the information of providers than the project with this role.

R13 (R_MissSupp) is in the negative domain of the second dimension and Rm11 (Rm_ProvWtch) is in the peak of the corresponding positive domain.

The results reveal that the experience from the knowledge keeper and maintaining a continuous watch on the providers’ support ability and reputation is an effective way to ensure the providers’ support.

Appendix A

205

Fig. 12 Component loading plots of the provider support risks 5.4 Sensitivity analysis Nonlinear CCA is one kind of the Alternating Least Squares methods [Gifi90], [Burg94]. All least squares techniques are prone to be over sensitive to outliers, and the outlier problem is manifested in nonlinear CCA in a terms of variable categories with small number of observations (small usually being defined as less than about five observations). To avoid the outlier problem, samples with missing data are simply deleted. In addition, we checked the sensitivity of the results in two steps:

- The first step is to identify abnormal/unusual data points by using the plot of object scores (see Figures 3 to 8 in Appendix B). Results show that there are no outliers that might tend to dominate the solution.

- The second step to verify our conclusion on the random subsets of the data. We deleted one data point every 10th (i.e., 10th, 20th, 30th etc.) and run the same analysis on the remaining data, which accounts for about 90% of the original data. The conclusions based on the remaining data are the same as the conclusions shown in sections 5.3.1 to 5.3.6.

5.5 Analysis on possible moderator variables Since we put data from three countries together in our data analysis, we wonder whether the national differences will affect the direction and/or strength of the relationship between the risk management activities and risks occurrences we discovered. It is therefore necessary to verify the moderator effect of the country differences on our conclusions. As we also put the data from projects using COTS component and projects using OSS components into one analysis, the same validity analysis is done to figure out whether the differences between COTS and OSS will have moderator effect on our conclusions.

Appendix A

206

To analyze the effect of the national and OTS differences, we added the category variable country and OTS types as the third set in each correlation between the risks and risk management activities. The results show that our conclusions on the requirements risks indeed are affected by the country variable. The component loading plots are shown in Figure 13. It shows that the correlation between R7 (R_ReqRen) and Rm7 (Rm_UnfFirst) in Figure 9 is not valid anymore. Therefore, we need to interpret these conclusions cautiously in different countries. The further analysis on the centroids of the variable country and the centroids of R7 shows that Italy (coded 3 in Figure 9 in Appendix B) is in the positive domain of the first dimension. It means that it has fewer occurrences of R7 (R_ReqRen) (coded 1 in Figure 10 in Appendix B). On the other hand, Germany (coded 1 in the Figure 9 in Appendix B) is in the negative of the first dimension, which means that it has more occurrences of R7 (R_ReqRen) (coded 5 or 4 in Figure 10 in Appendix B). The occurrences of R7 (R_ReqRen) in Norway are between Germany and Italy. The possible explanation is from the business etiquette in different countries. The Geert Hofstede study [Geert06] indicates that “Germans do not like surprises. Sudden changes in business transactions, even if they may improve the outcome, are unwelcome.” On the other hand, a known Italian tacit is to dramatically change demands at the eleventh hour to unsettle or test the flexibility of their counterparts [kwintessential06].

Fig. 13 Component Loading Plots of Effect of Country on the Requirements Risks 5.6 Risks and effective risk management activities

The result of the correlations between the risks and risk management activities are summarized in the Table 3.

Appendix A

207

Table 3 Summary of the Risks and Most effective Risk Management Activities

Results in Table 3 show that some risk management actives are more effective on mitigating possible risks than the other activities. However, the effectiveness of the activities may not be different valid in different countries.

6 Discussion 6.1 Mitigation of the cost estimation risks

Both the effort in selecting OTS components and the effort to integrate them sufficiently are difficult to estimate [Boehm03], [Mitre05]. Basili and Boehm [Basili01] hypothesized that COTS-based development is currently a high-risk activity, with effort and schedule overruns exceeding non-CBS software overruns. The results of the first research question show that the wrong estimation of the integration effort happened more frequently than other problems.

To estimate the selection effort correctly, Boehm et al. [Boehm03] propose to identify the core “showstopper” requirements and filter all the OTS candidates that do not meet these during the initial assessment and then proceed for a more detailed assessment with the remaining OTS candidates. Although this method may be applicable to limit the number of OTS components that need to be evaluated, it is still difficult to estimate the possible effort to compare the limited OTS candidates, because both technical and non-technical issues of the OTS components need to be evaluated and compared [Rose03]. Our findings show that in case the effort on learning in the OTS components has been seriously considered, it will be easier to estimate the selection effort correctly. Boehm et al. [Boehm03] found that one common error in OTS-based development is the overly optimistic OTS learning curve and propose that a most likely OTS learning curve must be accounted for during planning and scheduling.

Appendix A

208

Initially, the OTS component’s functional capability must be well understood, which can be a daunting task because there may be a significant amount of functionality to learn and that functionality might involve new technologies to the team. Also, there is more to learn than just the functionality. The methods of integration that the OTS component employs must be understood sufficiently to ensure that the integration can be succeeded into the current architecture. Since the intention of using OTS components is to shorten time-to-market and to save development cost, OTS component users are eager to acquire the components and to integrate them as soon as possible without understanding them thoroughly. Rose [Rose03] pointed out that developers are usually working with one or more OTS components for which they most likely have only a partial understanding. To improve the estimation on the OTS integration effort, the proposed activity is to use OTS-sensitive analytical and budget processes [Rose03], [Faa03], such as COCOTS [Abts00]. In our study, we did not investigate the effect of using COCOTS because our pre-study [Li04] did not find any companies using it. Therefore, we did not investigate this activity because we wondered whether this activity has been popularly used in industry. Although the results shown in Figure 6 reveal that the risk management activity Rm5 (i.e., effort in learning OTS component was seriously considered in effort estimation) has been widely used, we still wonder whether there are good tools to help the project managers to estimate the learning effort precisely.

6.2 Mitigation of the quality risks

To get a high-quality system, it is important to ensure that every component in the system does not bring negative effects on the whole system. The results shown in Figure 4 illustrate that OTS components rarely have a negative effect on system quality.

Our results on the second research question show that the black-box testing, and early integration testing, which have been very popular in practice, helped to avoid possible quality problems. Boehm et al. [Boehm03] discovered that the black-box testing is always imposed in OTS-based development and suggested that a risk based testing plan must be developed and accounted for in the project life cycle schedule. In the COTS component market, the salesman may make faulty claims about the functional and non-functional characters of their products [Boehm03], [Faa03], [Mitre05]. Thus, the users need to do some testing themselves to ensure that the quality of the component is the same as that claimed by the vendor. For the OSS components, there is never enough information to illustrate the full picture of the functions and the quality of the components. Thus, users need to test and evaluate the issues they are interested in by either reading the source code or doing black box testing.

However, the performance of the system highly depends a lot on the interaction between components and the environment, such as the hardware and the workload. It is therefore difficult to predict and evaluate by just incremental integration testing. Our investigated risk mitigation activities did not show effective reduction on the OTS components’ possible negative effect on system performance.

Although other solutions have been proposed to mitigate the possible quality issues of the overall system, such as selecting the OTS based on architecture instead of functionalities [Rose03], [Torchiano04], [Li04], or leverage the commercial infrastructure [Faa03]. Our results did not show a strong effect from these activities. A

Appendix A

209

possible reason is that most OTS components investigated in our studies followed the industrial standards or infrastructures - such as COM, CORBA, .Net, and EJB - by default. The possible quality problems caused by the architecture mismatch itself are therefore minimized.

6.3 Mitigation of the requirements risks

Another popular issue on OTS component integration is the requirements changes and mismatches with OTS component functionalities [Boehm03], [Faa03], [Rose03] [Mitre05].

Since it is difficult to change the OTS component to follow the customers’ changes in requirements, it is important for the OTS component integrator to be able to negotiate requirements with the customer [Boehm03], [Rose03], and involving customers into make vs. acquire decision and OTS component selection [Faa03]. However, our results show that very few integrators have actually involved customers into the make vs. acquire decision and the OTS component selection. Results of the study [Li04] illustrate that it is not easy to negotiate requirements with the customer in OTS component-based development. One reason is that customer only cares about the final product without willingness or capability of knowing the technical details.

To mitigate the possible requirements risks, our results show that careful evaluation and testing of OTS components in the selection phase helped to increase the possibility to follow the customers’ requirements changes. The possible explanation is that people understand the OTS components better when they did testing and quality evaluation in the selection phase. They therefore know the capability and the possible extensions of the OTS components.

Another possible activity is to first integrate the unfamiliar, i.e., high risk or critical OTS components [Rose03], [Li04]. Integrating the critical components first helped to provide the solid foundation for the architecture of the system [Rose03]. Integrating the high risk or most complex components first can help to find problems in the early phase of the project, which may give the integrator leeway to negotiate with their customers to redefine the requirements [Lauesen06]. To negotiate the requirements with the customer, educating the customer in requirements negotiation will be critical, but this must be accomplished in a manner that is politically viable for the particular customer [Rose03]. The possibility of negotiation depends on the willingness of the customer to negotiate issues and to accept changes. Our results illustrate that customers in certain business culture, such as in Germany, are more difficult to accept requirements negotiation than others.

6.4 Mitigation of the debugging and deployment risks

Our results show that the difficulty in locating the cause of defects happened more frequently than other problems. Kotonya and Rashid [Kotonya01] point out that OTS component offering similar functionalities may have very different system resource requirements (i.e., memory and processor requirements). When a problem is identified, it may not be obvious where the root cause of the problem is located. The problem could be in one of the components, it could be in the integration code, or it could be a misinterpretation of the semantic operation of a component interface [Rose03].

Appendix A

210

The results of the research question RQ3 show that this risk can be mitigated by giving developers enough time to learn and understand the OTS components. Since both COTS and OSS components are acquired from a third-party organization, it will require a lot of time for the users to understand the underlying design and code in these components. It is especially difficult for the COTS component users to use the components properly when source code is not available. In case there are defects in the OTS components or the interactions between the OTS components are not properly controlled, it may take longer time for the component users to figure out the problems than they had expected.

For the possible deployment problem, our findings indicate that an effective way of controlling it is to start integration testing as early as possible. In the early days of integrating OTS into a system solution, there was a false assumption that the need for integration testing was either minimal or not needed at all. It seems that there was a belief (actually a false hope) that once the OTS components were installed, the system would be operational [Rose03]. The OTS providers are only knowledgeable in their own OTS components. If there is no strong evidence that problems in the integrated system are caused by a specific OTS component, the component vendor may not want to help the users to debug and solve the problems. The benefit of doing integration testing after one OTS component is integrated is that it is easy to conclude that the problems lie in a specific component. An additional possible mitigating activity is to do some analysis to determine the order in which the OTS components are to be integrated [Spc01].

6.5 Mitigation of the maintenance risks

Maintaining an OTS component-based system is difficult. The post-deployment costs may exceed the development costs [Basili01]. The different customer-vendor evolution cycles may result in an uncertainty about how often the OTS components in a system may have to be replaced and the extent of the impact of such a change on the rest of the system [Kotony01, [Rose03], [Mitre05], [Faa03].

One risk mitigation solution proposed is to build an explicit, formal strategic relationship that includes mutual non-disclosure agreements to ensure that the provider gives full disclosure of the products future upgrades [Rose03]. However, this strategy is only applicable in a few situations. In case the customer is not regarded as a major one by the COTS component provider, it is difficult to build such a relationship. In the case of using OSS components, it is also difficult to build such a relationship, because the OSS components are updated by volunteers and nobody knows what functions may be added in future releases.

Another risk mitigation activity is to have senior analysts (serving as knowledge keepers) directly involved in the analysis of the updates and deriving effective field upgrade procedures and schedules [Rose03], [Li04]. This is a non-trivial system analyst task that requires appropriate level of expertise, as there is very little in the literature to help guide the analysts to a feasible solution [Rose03]. The existence of the knowledge keeper has been investigated and show effective results on mitigating the possible risks due to the OTS component evolution. Since the knowledge keepers know the current component version being used, it is possible for them to evaluate the new versions and to figure out the benefits and risks of updating the current version with the newer one.

Appendix A

211

The personal capability and experience remain the dominant factors influencing OTS-based development [Basili01]. Therefore, it is important to involve OTS- knowledgeable individuals in all analytical processes [Faa03].

However, planning maintenance of an OTS component-based system is very difficult because the OTS component users have no control of the release cycle of the OTS components being used. Since the release cycle of the OTS component depends on the marketing strategy of the component vendors, the component users usually have no idea when the new release of the OTS component will be published. Our investigated risk management activities focus on the activities that can be controlled by the component users. Such risk management activities were not effective in mitigating the maintenance plan risk.

6.6 Mitigation of the provider support risks

Another issue in OTS component-based development is to get enough technical support from the provider. Although the users can have access to the source code when using the OSS components, few of them have actually modified the code to meet their requirements [Li05]. It is also advised to avoid modifications of OTS components, even when it is possible [Faa03]. COTS component users can get support from the provider by signing a support contract. However, COTS providers may still stop to support their products due to different reasons, or even go out of business [Rose03], or the actual version of a COTS component is too old to be supported [Basili01]. For the OSS components, some of them are currently well supported by the volunteers in the OSS projects. However, it does not ensure that the users can get the same level of support in the future. The users have no control of the technical support they want to get from the OSS community. As our study tried to find out the common risk mitigation activities for both COTS and OSS components, we did not investigate the contract and licensing techniques, which is not applicable for the OSS user. We therefore looked more on the activities that are under control among OTS component users.

For the provider support risks, our results show that the most important remedy is still to select the right providers. It is necessary to perform continuous watch on the support reputation of the OTS component providers, as proposed in [Faa03], [Mitre05]. In addition, it is important to trace the history of the provider support by consulting the knowledge keeper.

6.7 Possible Threats to Validity 6.7.1 International Validity Threats We have promised respondents in this study a final report and a seminar (in Norway) to share experience. The respondents were persons who wanted to share their experience and wanted to learn from others. In general, we think that the respondents answered the questionnaire truthfully. However, different persons in the same project might have different opinions on the same project. Asking only one person in each project might not be able to get the whole picture of the project. Although all questionnaires were filled in through the web tool in Norway, questionnaires were mostly filled in by

Appendix A

212

telephone interviews in Germany. The different fill-in methods may bring biases to the results of this study (a so-called “mode problem” in surveys). 6.7.2 External Validity Threats

Although we used different techniques to select samples in different countries, this study is the first survey on component-based software engineering and indeed software engineering, using stratified-random sampling of the IT companies in several countries. However, this study focused OTS components that satisfy our component definitions in Section 3.3.1. Conclusions may be different in projects using complex and large OTS packages, such as ERP, CRM, or content management systems solutions. 6.7.3 Construct Validity Threats

In this study, most variables and alternatives are taken directly, or with little modification, from existing literatures. However, we may omit some possible risks and risk management activities because they were not mentioned in previous literatures. The questionnaire was pre-tested using a paper version by 10 internal experts and 8 industrial respondents before being published on the SESE tool [Sese04]. About 15% questions have been revised based on pre-test results. 6.7.4 Conclusion Validity Threats

This study is a state-of-the-practice study. We studied what had happened in industrial projects. However, we did not investigate the cause-effect relation of the phenomena discovered in this study. In our study, we assume that the apriori probabilities of typical risks are similar in most OTS component-based projects and examined only the relations between the performed risk management activities and the actual risk incidence after projects are finished. Since different project might have slightly different a priori probability of typical risks, our assumption might bring biases to our conclusions. 7 Conclusion and future work OTS component-based development brings both benefits and risks. Although several studies have investigated and proposed risk management activities in OTS component-based development, few studies have verified and compared the effects of the risk management activities on the corresponding risks. This study is an exploratory study on examining the occurrences of typical risks in OTS-based projects and relations between the occurrence of these risks and the risk mitigation activities performed. The intention is to find out the most effective activities that helped to mitigate corresponding risks in practice. The data was collected from 133 OTS component-based projects in Norway, Italy and Germany. The analysis was performed with nonlinear canonical correlation analysis, i.e., OVERALS in SPSS.

The results of this study show that two of the most frequent problems in OTS-based projects are the incorrect estimation of the integration effort and inefficient debugging.

Appendix A

213

The least frequent problems are relevant to the negative effect of the OTS components on the quality of the whole system. We also conclude that:

- To increase the accuracy of effort estimation, it is important to allocate enough

effort to learn and understand the OTS components. - To avoid the possible quality problems and to increase the efficiency of locating

the defects, it is important to do black-box testing as part of OTS component selection and to do integration testing as early as possible.

- To mitigate the problems due to customers’ requirements changes, it is important to learn the OTS components thoroughly in the selection phases and to integrate the unfamiliar OTS components first. It is also important to be able to negotiate changed requirements with the customers.

- To increase the efficiency of debugging and system deployment, it is necessary to understand the components well and to do integration testing early.

- It is critical to keep and share the knowledge about OTS components and the component providers inside the organization. It shows to help mitigate the possible risks in system maintenance and evolution.

As this is an exploratory study, further studies are needed to confirm our conclusions.

The assumption of this study is that the a priori risk probabilities of typical risks in the investigated projects are similar. However, it is possible that there are minor differences in the a priori risk probabilities between projects. It is therefore necessary to follow several OTS component-based projects from start by case studies. In the future, we plan to examine the several projects by estimating risk probabilities before the project start, following the project execution, and measuring the risk occurrences after the projects are finished. The intention is to investigate the cause-effect of the risk management activities on the corresponding risks.

Moreover, some of the conclusions of this study need further clarification. For example, which knowledge needs to be kept and shared in the organization and how to share it? How to estimate the possible learning effort needed when using the unfamiliar OTS components? These research questions need to be further studied by case studies or experiments.

Acknowledgment

The authors wish to thank Barbara Kitchenham for commenting on the data analysis methods. References [Abts00] C. Abts, B. W. Boehm, and E. B. Clark, “COCOTS: A COTS Software

Integration Lifecycle Cost Model - Model Overview and Preliminary Data Collection Findings,” Technical report USC-CSE-2000-501, USC Center for Software Engineering, http://sunset.usc.edu/publications/TECHRPTS/2000/usccse2000-501/usccse2000-501.pdf, 2000.

Appendix A

214

[Barki93] H. Barki, S.Rivard, and J. Talbot, “Toward an Assessment of Software Development Risk,” J. Management Information Technology, vol. 22, no. 2, pp. 359-371, Dec. 1993.

[Basili01] V. R. Basili, and B. W. Boehm, “COTS-Based Systems Top 10 List,” IEEE Computer, vol. 34, no. 5, pp. 91-93, May 2001.

[Basili02] V. R. Basili, M. Lindvall, I. Rus, C. Seaman, and B. W. Boehm, “Lessons-Learned Repository for COTS-Based SW Development,” Software Technology Newsletter, vol. 5, no. 3, pp. 4-7, available at: http://fc-md.umd.edu/ll/index.asp, 2002.

[Bass00] L. Bass, C. Buhman, S. Comella-Dorda, F. Long, J. Robert, R. Seacord, and K. Wallnau, “Volume I: Market Assessment of Component-based Software Engineering,” SEI Technical Report number CMU/SEI-2001-TN-007, available at: http://www.sei.cmu.edu/, 2001.

[Boehm88] B. W. Boehm, “A Spiral Model of Software Development and Enhancement,” IEEE Computer, vol. 21, no. 5, pp. 61-72, May 1988.

[Boehm89] B. W. Boehm, Software Risk Management, Tutorial, IEEE CS Press, 1989. [Boehm91] B. W. Boehm, “Software Risk management: Principles and Practices,”

IEEE Software, vol. 8, no. 1, pp. 32-41, Jan.1991. [Boehm03] B. W. Boehm, D. Port, Y. Yang, and J. Bhuta, “Not All CBS Are Created

Equally COTS-intensive Project Types,” Proc. Second Intl. Conf. on COTS-Based Software Systems, pp. 36-50, 2003.

[Brakk90] C.J.F. Ter Brakk, “Interpreting canonical correlation analysis through biplots of structure correlations and weights,” Psychometrika, vol. 55, pp. 519-531, 1990.

[Burg94] E.V.D. Burg, J. D. Leeuw, and G. Dijksterhuis, “OVERALS – Nonlinear Canonical Correlation with k sets of Variables”, J. Computational Statistics & Data Analysis, vol. 18, pp. 141-163, 1994.

[Bush90] M. E. Bush, N. E. Fenton, “Software Measurement: A Conceptual Framework”, J. System and Software, vol. 12, no. 3, pp. 223-231, 1990.

[Carr93] M. Carr, S. Kondra, I. Monarch, F. Ulrich, and C. Walker, “Taxonomy-Based Risk Identification,” Technical Report SEI-93-TR-006, SEI, Pittsburgh, USA, 1993.

[CBSEnet04] Component-based Software Engineering Network. CBSEnet, available at http://www.cbsenet.org, 2004.

[Chittister96] C. G. Chittister and Y. Y. Haimes, “System Integration via Software Risk Management,” IEEE Trans Systems, Man, and Cybernetics, vol. 26, no. 5, pp. 521-532, Sep. 1996.

[Conradi05] R. Conradi, J. Li, O. P. N. Slyngstad, C. Bunse, M. Torchiano, and M. Morisio, “Reflections on Conducting an International CBSE Survey in ICT Industry,” Proc. of the forth Intl. Symp. on Empirical Software Engineering , pp. 214-223, 2005.

[Crnkovic02] I. Crnkovic and M. Larsson, “Building Reliable Component-Based System,” Artech House, 2002.

[Faa03] COTS risk factor, available at http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm, 2003.

[Fitzgerald04] B. Fitzgerald, “A Critical Look at Open Source,” IEEE Computer, vol. 37, no. 7. pp. 92-94, July 2004.

[Geert06] Geert Hofstede™ Cultural Dimensions, available at http://geert-hofstede.international-business-center.com/, 2006.

Appendix A

215

[Gemmer97] A. Gemmer, “Risk Management: Moving Beyond Process,” IEEE Computer, vol. 30, no. 5, pp. 33-41, May, 1997.

[Ghosh02] S. Ghosh, “Improving Current Component-Based Development Techniques for Successful Component-Based Software Development,” Proc. Intl. Conf. Software Reuse, Workshop on Component-based Software Development Processes, 2002, available at: http://www.idt.mdh.se/CBprocesses/

[Gifi90] A. Gifi, Nonlinear Multivariate Analysis, John Wiley & Sons, 1990. [Glass01] R. L. Glass, “Frequently Forgotten Fundamental Facts about Software

Engineering”, IEEE Software, vol. 18, no. 3, pp. 110-112, May/June, 2001. [Hecht04] H. Hecht, Systems Reliability and Failure Prevention. Artech House

Publishers, 2003. [INCO00] INCO project description, http://www.ifi.uio.no/~isu/INCO, 2000. [Jacobson97] I. Jacobson, M. Griss, and P. Jonsson, “Software Reuse: Architecture,

Process and Organization for Business Success,” Addison-Wesley, 1997. [Kansala97] K. Kansala, “Integrating Risk Assessment with Cost Estimation,” IEEE

Software, vol. 14, no. 3, pp. 69-74, May/June 1997. [Keil98] M. Keil, P.E. Cule, K. Lyytinen, and R.C. Schmidt, “A Framework for

Identifying Software Project Risks,” Communications of the ACM, vol. 4, no. 11, pp. 76-83, Nov. 1998.

[Kitchenham03] B. Kitchenham, S. L. Pfleeger, “Principles of Survey Research Part 6: Data Analysis,” Software Engineering Notes, vol. 28, no. 2, pp. 24-27, March 2003.

[Kotonya01] G. Kotonya and A. Rashid, “A Strategy for Managing Risk in Component-based Software Development,” Proc. 27th EUROMICRO Conf, pp. 12-21, 2001.

[Kwintessential06] http://www.kwintessential.co.uk/etiquette/doing-business-italy.html, 2006.

[Lawton02] G. Lawton, “Open Source Security: Opportunity or Oxymoron?” IEEE Computer, vol. 35, no. 3, pp. 18-21, March 2002.

[Li04] J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT Industry, “ Proc. 10th IEEE Intl. Metrics Symp. pp. 72-83, 2004.

[Li05] J. Li, R. Conradi, O. P. N. Slyngstad, C. Bunse, U. Khan, M. Torchiano, and M. Morisio, “Validation of New Theses on Off-The-Shelf Component Based Development,” Proc. 11th IEEE Intl. Metrics Symp., pp. 26, Sept. 2005.

[Lauesen06] S. Lauesen, “COTS Tenders and Integration Requirements,” J. Requirements Engineering, vol. 11, no. 2, pp. 111-122, 2006.

[Madachy97] R. J. Madachy, “Heuristic Risk Assessment Using Cost Factors,” IEEE Software, vol. 14, no. 3, pp. 69-74, May/June 1997.

[Madanmohan04] T.R. Madanmohan and R. De’, “Open Source Reuse in Commercial Firms,” IEEE Software, vol. 21, no. 1, pp. 62-69, January/February, 2004.

[Manly04] B. F. J. Manly, “Multivariate Statistical Methods: A Primer,” 3rd edition. Chapman & Hall/CRC, 2004.

[Michailidis98] G. Michailidis and J. d. Leeuw, “The Gifi System of Descriptive Multivariate Analysis,” J. Statistical Science, vol. 13, no. 4, pp. 307-336, 1998.

[Mitre05] Common Risks and Risk Mitigation Actions for a COTS-based System, www.mitre.org/work/sepo/toolkits/risk/taxonomies/files/CommonRisksCOTS.doc, 2005.

Appendix A

216

[Morisio00] M. Morisio, S. Seaman, A. Parra, V. Basili, S. Kraft, and S. Condon, “Investigating and Improving a COTS-Based Software Development Process,” Proc. 22nd Intl. Conf. on Software Engineering, pp. 31-40, 2000.

[Moynihan97] T. Moynihan, “How Experienced Project Managers Assess Risk,” IEEE Software, vol. 14, no. 3, May/June 1997.

[Ropponen00] J. Ropponen and K. Lyytinen, “Components of Software Development Risk: How to Address Them? A Project Manager Survey,” IEEE Trans. Software Engineering, vol. 26, no. 2, pp. 98-112, February 2000.

[Rose03] L. C. Rose, “Risk Management of COTS Based System Development,” Component-Based Software Quality - Methods and Techniques, Springer – Verlag LNCS vol. 2693, pp. 352-373, 2003.

[Ruffin04] M. Ruffin and C. Ebert, “Using Open Source Software in Product Development: A Primer,” IEEE Software, vol. 21, no. 1, pp. 82-86. January/February 2004.

[Sese04] Sese tool, available at http://www. Simula.no, 2004. [Sherer95] S. A. Sherer, “The Three dimensions of Software Risk: Technical,

Organizational, and Environmental,” Proc. 28th Hawaii International Conference on System Sciences, pp. 369-378, 1995.

[Ssb02] Norwegian Census Bureau (SSB), Oslo, ICT company data, available at http://www.ssb.no/emner/10 /03/ikt/, 2002.

[Sommerville04] I. Sommerville, Software Engineering, 7th edition, Addison-Wesley, 2004.

[Spc01] Software Productivity Consortium, “Phased-Integrated COTS Approach,” Technical Report SPC-2001006-CMC, http://www.software.org/pub/pic, 2001.

[Spearman1904] C. Spearman, “The Proof and Measurement of Association between Two Things”, American Journal of Psychology, vol. 15, pp. 72-101, 1904.

[Spss02] SPSS Categories 11.0, available at www.spss-sa.com, 2001. [Spss06] SPSS Categories 14.0, available at http://www.spss.com/categories/, 2006. [Szyperski02] C. Szyperski, D. Gruntz, and S. Murer, Component Software – Beyond

Object-Oriented Programming. Addison-Wesley, 2002. [Torchiano04] M. Torchiano and M. Morisio, “Overlooked Facts on COTS-based

Development,” IEEE Software, vol. 21, no. 2, pp. 88-93, March/April 2004. [Verner05] J. M. Verner and W. M. Evanco, “In-House Software Development: What

Project Management Practices Lead to Success?” IEEE Software, vol. 22, no. 1, pp. 86-93, Jan./Feb. 2005.

[Vigder96] M. Vigder, M. Gentleman, and J. Dean, “COTS Software Integration: State of the Art”, Technical Report NRC No. 39190, 1996.

[Vitharana03] P. Vitharana, “Risks and Challengs of Component-Based Software Development,” Communications of the ACM, vol. 46, no. 8, pp. 67-72, August 2003.

[Voas98a] J. Voas, “COTS Software – the Economical Choice?” IEEE Software, vol. 15, no. 2, pp. 16-19, March/April 1998.

[Voas98b] J. Voas, “The challenges of Using COTS Software in Component-Based Development,” IEEE Computer, vol. 31, no. 6, pp. 44-45, June 1998.

[Wallace04] L. Wallace and M. Keil, “Software Project Risks and Their Effect on Outcomes,” Communications of the ACM, vol. 47, no. 4, pp. 68-73, April 2004.

Appendix A

217

Appendix A: The rationale of using Nonlinear CCA Standard canonical correlation analysis (CCA) [Manly04] is an extension of multiple regressions, where the second set does not contain a single dependent variable but instead contain multiple dependent variables. The Nonlinear CCA approach expands the canonical correlation analysis in three crucial ways. First, Nonlinear CCA allows more than two sets of variables. Second, variables can be scaled as nominal, ordinal, or numerical. As a result, nonlinear relationships between variables can be analyzed. Finally, instead of maximizing correlations between the variable sets, the sets are compared to an unknown compromise set that is defined by the canonical variate [Spss02].

An efficient solution to the nonlinear CCA problem was first proposed by Gifi [Gifi90]. It was later extended to generalized canonical analysis with more than two sets of variables. The generalized nonlinear CCA program, called OVERALS is available in the SPSS CATEGORIES [Spss06] program suites. The OVERALS simultaneously determines both (1) optimal re-scalings of the nominal and ordinal variables and (2) explanatory variable weights, such that the linear combination of the weighted re-scaled variables in one set has the maximum possible correlation with the linear combination of weighted re-scaled variables in the second set. Both the variable weights and optimal category scores are determined by minimizing a loss function derived from the concept of “meet” in lattice theory. Graphical representations are very important in interpreting this plethora of results. In fact, several authors have argued that graphical representations are crucial in understanding the results of linear multivariate methods, particularly CCA, because patterns in the data can best be detected by the eyes [Brakk90].

In our study, a possible data analysis is to find out the correlations between the risk management activities performed and the occurrences of risks, using Spearman correlation analysis [Spearman1904]. To use this method, we must assume that the risks are independent and the effects of risk management activities are independent. However, we generally believe that each risk management activities performed may help to mitigate several risks. Therefore, we cannot ensure that the effects of risk management activities performed are independent.

Another possible data analysis method is to do a multiple regression analysis with the risk management activities as the independent variables and one specific risk as the dependent variables. The weights of each risk management activity in the regression equation can be used to compare their effectiveness. However, this method also assumes that the occurrences of different risks are independent. Since no previous studies have investigated the relations between these risks, we cannot ensure that they are independent of each other. Thus, we just divided them into different groups according to the phases in the project as in the study [Rose03].

The third possible data analysis method is to do a two- step analysis. The first step is to do a principal component analysis with the risks and risks management activities to figure out several risk factors and risk management factors. The second step is to find out the correlations between the risk factors and the risk activity factors. The same method has been used by [Ropponen00] to analyze risk management issues. However, Kitchenham and Pfleeger have commented this method on [Kitchenham03]: Principal component analysis can be used either to test a hypothesis or to generate hypotheses.

Appendix A

218

However, it cannot do both things at once. Thus, the risk factors identified using principal component analysis represent hypotheses/theories that should have been confirmed by an independent study before investigating risk strategies. The underlying problem in this study [Ropponen00] (and many others) was trying to do too much in one investigation.

Appendix B: Supplement Tables and Figures

Table 1 Examples of OTS components

LibraryCollection of embeddablemapping and GIS components

COTSMapObjects

J2EE, .NET, COMReport design and developmentCOTSCrystal Report

PHP classRSS ParserOSSSnoopy

LibraryLogging and tracingOSSLog4j

TechniqueFunctionOSS /COTS

Name of the OTScomponents

LibraryCollection of embeddablemapping and GIS components

COTSMapObjects

J2EE, .NET, COMReport design and developmentCOTSCrystal Report

PHP classRSS ParserOSSSnoopy

LibraryLogging and tracingOSSLog4j

TechniqueFunctionOSS /COTS

Name of the OTScomponents

Table 2 Component Loading of Effort Estimation Risks with RM Activities DimensionSet

1 2Rm1 ,035 -,242Rm2 ,349 ,095Rm3 ,214 ,330Rm4 ,400 ,024Rm5 ,562 -,021Rm6 ,164 ,402Rm7 ,444 -,060

1

Rm8 -,087 ,139R1 -,844 -,4962R2 -,859 ,424

Table 3 Component Loading of Quality Risks with RM Activities

DimensionSet1 2

Rm1 ,286 ,037Rm2 -,217 -,356Rm3 ,071 -,391Rm4 ,384 -,838Rm5 ,055 ,128Rm6 ,435 -,037Rm7 -,202 ,096

1

Rm8 ,456 ,059R3 -,530 ,264R4 -,643 -,258

2

R5 ,087 ,072

Table 4 Component Loading of Requirements Risks with RM Activities

Appendix A

219

DimensionSet1 2

Rm1 -,271 ,042Rm2 -,039 -,149Rm3 ,226 -,414Rm4 -,168 ,596Rm5 -,408 ,165Rm6 -,428 ,301Rm7 ,538 ,214

1

Rm8 ,309 ,281R6 ,647 -,6202R7 -,553 -,761

Table 5 Component Loading of Debugging and Deployment Risks with RM Activities

DimensionSet1 2

Rm1 -,029 -,170Rm2 -,576 ,073Rm3 -,111 -,194Rm4 -,033 -,083Rm5 ,003 -,675Rm6 ,140 -,230Rm7 -,382 -,159

1

Rm8 ,464 -,053R8 ,223 ,9612R9 -,829 ,303

Table 6 Component Loading of Maintenance Risks with RM Activities DimensionSet

1 2Knowledge keeper -,524 ,074Rm9 ,796 ,378Rm10 ,203 -,818

1

Rm11 ,240 ,144R10 ,286 ,0722R11 -,622 ,158

Table 7 Component Loading of Provider Support Risks with RM Activities DimensionSet

1 2Knowledge keeper -,734 -,092Rm9 ,199 -,139Rm10 -,362 ,509

1

Rm11 ,275 ,806R12 -,839 -,0122R13 -,117 -,624

Appendix A

220

Fig.1. Centroids of the Knowledge keeper in the analysis of provider support risks

Fig.2. Centroids of the R12 in the analysis of provider support risks

Fig. 3 Object Scores of the Cost Estimation Risks

Appendix A

221

Fig. 4 Object Scores of the Quality Risks

Fig. 5 Object Scores of the Requirements Risks

Fig. 6 Object Scores of the Debugging and Deployment Risks

Appendix A

222

Fig. 7 Object Scores of the Maintenance Risks

Fig. 8 Object Scores of the Provider support risks

1: Germany 2: Norway 3: Italy

Fig.9. Centroids of the countries in the analysis of requirements risks

Appendix A

223

Fig.10. Centroids of the R7 in the analysis of requirements risks

Appendix A

224

Appendix A

225

P6. Decision Making in OTS Component-Based Development

An Empirical Study on Decision Making in Off-The-Shelf Component-Based Development

Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Maurizio

Morisio4, and Marco Torchiano4 1Department of Computer and Information Science,

Norwegian University of Science and Technology (NTNU), NO-7491 Trondheim, Norway

{jingyue, conradi, oslyngst}@idi.ntnu.no 2Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysaker, Norway

3Fraunhofer IESE, Sauerwiesen 6, D- 67661 Kaiserslautern, Germany

[email protected] 4Dip.Automatica e Informatica, Politecnico di Torino Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy

{morisio, marco.torchiano}@polito.it

ABSTRACT: Component-based software development (CBSD) is becoming more and more important since it promotes reuse to higher levels of abstraction. As a consequence, many components are available being either open-source software (OSS) or commercial-off-the-shelf (COTS). However, it is still unclear how the decision for acquiring OSS or COTS components is made in practice. This paper describes an empirical study on why project decision-makers selected COTS instead of OSS components, or vice versa. The study was performed as an international survey in Norway, Italy and Germany. It focused on decision making on using off-the-shelf (OTS) components. We have gathered answers from 83 projects using only COTS components and 44 projects using only OSS components. Results of this study show significant differences and commonalities of integrating OSS or COTS components. Moreover, the study illustrates several research questions that warrant future research. Categories and Subject Descriptors D.2.13 [Software Engineering]: Reusable Software – Reuse model General Terms Management Keywords Commercial-Off-the-Shelf Software, Open Source Software, Empirical Study, Component-based Development

Appendix A

226

1. INTRODUCTION Long-term success in commercial software development is becoming increasingly challenging. The software industry is increasingly turning to approaches which offer a larger return on software development effort. Component-based software development (CBSD) [19] is one of these approaches. Besides reusing in-house built components, software companies are using an increasing amount of COTS and OSS components. Although both COTS and OSS component claim to save development effort, they are different in their nature and appearance. COTS components are owned by commercial vendors with agreed specific support. The users of COTS component normally have no access to the source code. On the other hand, OSS components are provided by open source communities with freely accessible source code but promise no specific support [14].

When planning a new software project, decision makers must decide whether they should buy a COTS component or acquire an OSS component, if it was decided to use OTS components. To make such a decision, it is important to investigate previous projects using such components. Unfortunately, the relevant decision-making processes are often intuitive, and if at all, only sparsely documented. Thus, further research is needed to examine the current state-of-the-practice and to define guidelines for OTS-based development.

This paper investigates the state-of-the-practice in OTS-based development on use of COTS and OSS components, by means of an international representative survey in Norway, Italy and Germany. We have gathered answers from 83 projects using only COTS components and 44 projects using only OSS components. Results of this study illustrated who are using OTS components, why was it decided to use such components, and what were the problems. Results also illustrated the commonalities and differences in integrating COTS or OSS components.

The remainder of this paper is structured as follows: Section two presents related research. Section three introduces research design. Section four presents obtained results and discussion. A short summary and some future work are given in Section five. 2. RELATED WORK COTS components promise faster time-to-market and increased productivity. At the same time, COTS software introduces many risks, such as the unknown quality of COTS components, or possible instability of the COTS vendor support [20]. Although some research stated that OSS makes only sense for operating systems [5], the use of OSS in industrial products is growing rapidly [11, 13, 16, 18]. OSS offer many advantages: it is usually freely available for public download. The collaborative, parallel efforts of globally distributed developers allow much OSS to be developed more quickly than conventional software. Many OSS products are recognized for high reliability, efficiency, and robustness [4]. Despite its wide appeal, OSS software faces several challenges and constraints: the documentation, testing, and field support may not be available [4], and long-term support is not guaranteed [11]. The OSS licence may restrict the user of changing and integrating the OSS components [18]. Previous studies have looked at using COTS and OSS components in software development [2, 13].

Appendix A

227

However, few studies have investigated a higher level question: Why do I use COTS instead of OSS components, or vice versa?

Some studies compared the differences between COTS and OSS products per se and concluded that there is still no empirical evidence that OSS fosters faster system growth and that OSS is more modular than closed source software [15]. However, there were few systematic empirical studies of the commonalities and differences in integrating COTS vs. OSS components in a software product. In order to guide the project decision maker in deciding between an OSS component and a COTS component, it is necessary to empirically investigate how the decision was made and what was the result of the decision in finished projects. 3. RESEARCH DESIGN The empirical study on decision making OTS based development followed a three step approach. We started from a pre-study of state-of-the-practice and the challenges in integrating COTS components. The pre-study was performed by structured interviews of 16 COTS projects in 13 Norwegian IT companies selected by convenience [8]. The study reported in this paper represents the second step, and investigated decision making in OTS component-based development by a questionnaire-based survey. This study extended the pre-study in two dimensions. First, it included OSS. Second, the study included randomly selected IT companies from Norway, Italy and Germany. The third future step will focus on investigating the underlining reasons of the conclusions in the second step and more studies on OSS projects. It will be performed with structured interviews and case studies. 3.1 Research Questions Based on the design of the study a couple of research questions were defined. First, we wanted to know the profiles of those using OTS components. Therefore, the first research question is:

RQ1: What are the commonalities and differences in profiles of projects using COTS components vs. those using OSS components?

We then wanted to know their expectation of using OTS components. The second

research question therefore is: RQ2: What are the commonalities and differences in the expectation of projects

using COTS components vs. those using OSS components? After we know who and why, we need to know the actual risks/problems of using

such components. We investigated whether the motivation and expectation of using OTS components are proper or not. So, the third research question is:

RQ3: What are the commonalities and differences in possible risks (problems) of projects using COTS components vs. those using OSS components?

Appendix A

228

3.2 Research Method In this study, we used a questionnaire to collect data. The detailed information of the questionnaire and sample selection are reported in [3, 9, 10] 4. RESULTS AND DISCUSSION In [10], we reported the preliminary results of this study with 71 projects using only COTS components and 39 projects using only OSS components. Now, the data collection is finished. In total, we gathered results from 83 projects using only COTS components, 44 projects using only OSS components, and six projects using both COTS and OSS components. 4.1 RQ1: What was the Profile of Projects Using OTS Components? To answer RQ1, we investigated the commonalities and differences of some requirements of the actual system, such as time-to-market, effort (cost), reliability, security, performance, maintainability, new functionality (first launch in the market), and improved functionality (over competitors).

We compared each requirement in systems using COTS vs. systems using OSS components. The results show that there are no such significant differences. The most emphasized requirements of the final systems were time-to-market, reliability, and performance. Security was surprising listed as the least emphasized requirement of the system. There are many discussions on whether OSS components are more secure than COTS components. Raymond [17] argued that “given enough eyeballs, all bugs are shallow.” Other argued that it is the quality, not the number, of the eyes looking at code that count [7]. Our results show that OTS component users are still very cautious of using either COTS or OSS components in the security-critical systems. So, software security aspect may not be distinguished between COTS and OSS components. However, trust in the security of third-party components may become an issue in the future.

4.2 RQ2: What were the Expectations of Using OTS Components? To answer RQ2, we investigated the respondent’s general expectations using OTS components, and specific expectations of using COTS and OSS components.

The general expectations of using OTS components are extracted from our pre-study [8] and literature reviews [4, 20]. Details of these expectations are shown in [10]. Results show again that there are no significant differences. The main expectation of using OTS components is to shorten the time-to-market and to save development cost. Pizka [16] reported experience on building the same system with three different strategies, such as wrapping an OSS component, adapting/changing the source code of the OSS component, or building the same component from scratch. Their results gave strong support on reusing OSS components and concluded that re-implementing existing OSS components should be avoided if there are no serious reasons for doing so.

The specific expectations of using COTS components listed in the questionnaire were shown in [10]. Results show that COTS users believe that paid software will give

Appendix A

229

good reliability and will follow the market trend. They also believe that the COTS vendor will provide good technical support. We also listed specific expectations of using OSS components listed in the questionnaire. Results show that the main motivation of using OSS components is that code could be acquired for free and that the source code is available.

In summary, results show that business considerations, such as reducing time and cost, are the main motivation for using OTS components. OSS component users care more about the initial investment. Another attraction of OSS components is that the marginal cost of scaling up is zero, i.e. using OSS does not require additional licences as the number of installations grows [18]. This benefit is especially important when integrating OSS components into a software product with many installation bases.

Concerning the use of domain-specific OTS components, it is important to ensure that the evolution of the selected OTS component will follow the market trend. Although a COTS user cannot directly influence the evolution of the selected COTS component, results of this study show that these users still trust that the COTS vendor will update the component according to market needs. OSS components can be changed out of similar market reasons, but can also be changed out of political or social reasons – not necessarily in a direction suitable to the organization [6]. It is therefore important for OSS users to ensure that they are planning to use the software in the way it was intended to be used and the way most other users are using it. If OSS users stray too far away from the beaten path, the OSS developers might not want to help them when they run into trouble, or OSS developers will not accept the changes contributed by them [11]. 4.3 RQ3: What were the Risks of Using OTS Components? To answer RQ3, we formulated 15 possible problems (see Table 3 in [10]) on OTS-based development, and asked the respondents if these problems had happened in their projects. Results showed that common problems both in projects using COTS and in those using OSS are: the effort to integrate OTS components was not satisfactorily estimated and it was difficult to identify whether the defects were inside or outside the OTS component.

The estimation of the component-integration effort in COTS-based development projects has been investigated by several studies [1], whereby the estimation of integration effort in OSS- based development has been neglected so far. It is therefore interesting to know the commonalities and differences in cost models for COTS-based vs. OSS-based projects.

Without source code, it is difficult for COTS component users to identify whether the defects is inside or outside the component. Although the source code of OSS components is available to their users, many OSS users did not change or even read its source code. As Hissam and Weinstock proposed in [6], OSS is the other commercial software, i.e. they were used as COTS in practice [9, 11]. OSS user might not have bothered to understand the source code of an OSS component before starting to use it. It is therefore difficult for them to figure out if the defects are inside an OSS component or not.

Results of RQ3 also show that COTS users had significantly more difficulties in estimating the component selection effort and following requirement changes. It is not

Appendix A

230

easy or cost-effective for COTS users to switch to another COTS component once they have made the decision and paid for some selected components. Therefore, COTS users might be more cautious in selecting the right COTS component and spend more effort than estimated on selecting the best one. Without paying for the OSS components, OSS users might be less cautious and might spend less effort on selecting the best OSS component from several candidates. In addition, it might be easier for OSS users to change the source code of the OSS component if their customers’ requirements do change. Therefore, OSS components are ideal when the software solution requires high customization [11].

OSS users had more problems on getting the information on a providers’ support reputation. Although the preliminary data [10] showed the differences between the OSS users and COTS users were significant, the current data (six months after the analysis of the preliminary data) does not show such significant differences any longer. This might be because more and more commercial organizations provide support to OSS components. In fact, OSS components with support by a commercial organization usually become more popular than components without such support [12]. In addition, more and more OSS portals, such as sourceforge.net and freshmeat.net, have started to measure the vitality and popularity of an OSS project. These metrics might have helped OSS users to estimate the quality and duration of support they can expect. 5. CONCLUSION AND FUTURE WORK This paper has presented the results of an international empirical study to investigate the decision making of integration COTS or OSS components in software products. In general, it appears that the motivation for using either COTS or OSS components and the risks involved are important factors to be concerned. Future research has to pay specific attention to these facts before defining systematic guidelines and methods. Results of this study have answered the following three questions:

RQ1: Who were using OTS components? Both COTS and OSS components are used in projects with different application

domains and different non-functional requirements. There was no significant difference between the profiles of COTS-based and OSS-based projects.

RQ2: Why was it decided to use OTS components? The main expectation of using either COTS or OSS components is to obtain shorter

time-to-market and less development effort. COTS users have higher expectation on COTS component quality and COTS vendor support. Possible no-cost source code is the key motivation of OSS users. OSS users prefer to have access the source code, so that they can revise it when necessary.

RQ3: What are the possible problems of using OTS components? It is more difficult for COTS component users to follow requirement changes than

OSS users. It is also more difficult for COTS users to estimate the selection effort. Based on these results we identified several emerging research questions that should

be investigated in the near future:

Appendix A

231

- Understand the reasons behind the current results

This study is a state-of-the-practice survey that provides evidence of some phenomena in OTS-based development. These phenomena have to be studied in more detail to better understand the nature of OTS projects and to provide systematic support. Examples of interesting issues are: How is the support capability of an OSS project estimated, how support for OSS components is realized, and how to avoid the licensing restriction.

- Understand the OSS projects per se

To provide guidelines on using OTS component in practice, it is important to

understand how OSS projects are organized. Furthermore, it is important to know how to cooperate with OSS developers, for example, how to ensure that changes of users are accepted by the OSS community, and how to motivate developers to enhance and adapt OSS component into a desirable direction. Another important issue is the definition of metrics to obtain reliable data about the concrete nature of OSS projects and products. For example, the organization support proved to give a positive effect on the popularity of an OSS component [12]. Future studies have to examine OSS projects in order to identify, define, and validate more such measures. A good candidate seems the question: whether the vitality and popularity of data provided by an OSS portal gives insight on the longevity of an OSS projects?

REFERENCES [1]Abts, C. et al. COCOTS: A COTS Software Integration Lifecycle CostModel -

Model Overview and Preliminary Data Collection Findings. TR: USC-CSE-2000-501, Mar. 2000.

[2]Brownsword, L. et al. Developing New Processes for COTS-Based Systems. IEEE Software. 17, 4 (Jul./Aug. 2000), 48-55.

[3]Conradi, R. et al. Reflections on conducting an international CBSE survey in ICT industry. Proc. of the 4th Intl. Symposium on Empirical Software Engineering (Noosa Heads, Australia, Nov. 2005), 214-223.

[4] Fitzgerald, B. A Critical Look at Open Source. IEEE Computer. 37, 7 (Jul. 2004), 92-94.

[5] Glass, R. L. A Look at the Economics of Open Source. Communication of the ACM, 47, 2 (Feb. 2004), 25-27.

[6] Hissam, S. A. and Weinstock, C. B. Open Source Software: The Other Commercial Software. Proc. of the 1st workshop on open source software engineering. (Toronto, Canada, May, 2001).

[7] Hissam, S. A. et al. Trust and Vulnerability in Open Source Software. IEE Proceedings - Software, 49, 1 (Feb. 2002) 47-51.

[8] Li, J. et al. An Empirical Study of Variations in COTS-based Software Development Processes in Norwegian IT Industry. Proc. of the 10th IEEE Intl. Metrics Symposium (Chicago, USA, Sep. 2004) 72-83.

Appendix A

232

[9] Li, J. et al. Validation of New Theses on Off-The-Shelf Component Based Development. Proc. of the 11th IEEE Intl. Metrics Symposium (Como, Italy, Sep. 2005), 26.

[10] Li, J. et al. An Empirical Study on Off-the-Shelf Component Usage in Industrial Projects. Proc. of the 6th Intl. Conf. on Product Focused Software Process Improvement (Oulu, Finland, Jun. 2005), Springer Verlag LNCS Vol. 3547, 54-68.

[11] Madanmohan, T. R. and De’, R. Open Source Reuse in Commercial Firms. IEEE Software, 21, 1 (Jan./Feb. 2004), 62-69.

[12] Stewart, K. J. et al. Preliminary Analysis of the Influences of Licensing and Organizational Sponsorship on Success in Open Source Projects. Proc. of the 38th Hawaii Intl. Conf. on System Sciences (Hawaii, USA; Jan. 2005), 197c.

[13] Norris, J. S. Mission-Critical Development with Open Source Software: Lessons Learned. IEEE Software. 21, 1 (Jan./Feb. 2004), 42-49.

[14] OSS Initiative: http://www.opensource.org/index.php [15] Paulson, J. W. et al. An Empirical Study of Open-Source and Closed-Source

Software Products. IEEE Transactions on Software Engineering. 30, 4 (Apr. 2004), 246-256.

[16] Pizka, M. Adaptation of Large-Scale Open Source Software – An Experience Report, Proc. of the 8th Euro. Conf. on Software Maintenance and Reengineering (Tampere, Finland, Mar. 2004) 147-153.

[17] Raymond, E. S. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O’Reilly, 1999.

[18] Ruffin, M. and Ebert, C. Using Open Source Software in Product Development: A Primer. IEEE Software, 21, 1 (Jan./Feb. 2004), 82-86.

[19] Szyperski, C. et al. Component Software – Beyond Object-Oriented Programming. Addison-Wesley, 2002.

[20] Voas, J. The challenges of Using COTS Software in Component-Based Development. IEEE Computer, 31, 6 (Jun. 1998), 44-45.

Appendix A

233

P7. Barriers to Disseminating Theory

Barriers to Disseminating Off-The-Shelf Based Development Theories to IT Industry

Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Umair

Khan3, Marco Torchiano4 and Maurizio Morisio4

1Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU),


2Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysaker, Norway 3 Fraunhofer IESE, Sauerwiesen 6, D- 67661 Kaiserslautern, Germany

{Christian.Bunse, khan}@iese.fraunhofer.de 4Dip. Automatica e Informatica, Politecnico di Torino Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy

{maurizio.morisio, marco.torchiano}@polito.it ABSTRACT: In this position paper, we have reported results of an industrial seminar. The seminar was intended to show our findings in an international survey, conducted in Norway, Italy and Germany, on off-the-shelf component-based development. Discussion in the second section of the seminar revealed several obstacles of popularizing the OTS based development theories into IT industry. Keywords: COTS component, OSS component, Empirical Study 1. INTRODUCTION Software developers are using an increasing amount of COTS (Commercial-Off-The-Shelf) and OSS (Open Source Software) components in their projects. However, using such external components introduces many risks [2, 5, 10]. Before project managers decide to acquire an external component, instead of building it in-house, they must identify and mitigate possible risks. Researchers and practitioners have proposed many new COTS selection and development processes [1, 14,19].

As most theories were based on local case studies, it is necessary to evaluate their conclusions on a larger sample. We have performed an international survey on process improvement and risk management in COTS and OSS component based development in Norway, Germany and Italy [12, 13]. After we analysed data, we hold an industrial seminar to report our preliminary results to the participants of this study and some interested project managers. An hour discussion revealed many new concerns from an industrial viewpoint.

Appendix A

234

2. BACKGROUND Our survey is the second phase of a systematic study on process improvement and risk management in OTS based development. We started from a pre-study, focusing on software process improvement in COTS based development, and being performed with structured interviews of 16 COTS projects in Norwegian IT companies [11]. The following main survey extended the pre-study in two dimensions. First, it included OSS components because they represent an alternative to COTS components. Second, this study included samples from Norway, Italy and Germany. In addition, the sample was selected randomly instead of on convenience as in the pre-study.

The survey was designed to investigate six main research questions: - RQ1: What are the main motivations of using OTS components instead of

building them in-house? - RQ2: What are the real common OTS-based process models, and what are the

possible variances? - RQ3: What is the OTS component selection process used in practice? - RQ4: What risks have been identified and what are their relationships with the

project contexts? - RQ5: Which risk management methods have been performed and with what the

results? - RQ6: What are the differences between COTS and OSS components?

The unit of study is a completed software development project. The projects were

selected based on two criteria: - The project should use one or more OTS components - The project should be a finished project, possibly with maintenance, and possibly

with several releases.

In the data collection process, we promised our participants that we would report the result of the study in the form of a seminar. After we have finished the data collection in Norway (47 projects) and gathered some data from Germany and Italy (23 from Germany and 20 from Italy), we held a seminar at Oslo in Feb. 2005 as we promised to the Norwegian participants.

The intentions of the seminar include two parts: - First, we would like to report our results by giving guidelines on how to improve

processes and manage risks in OTS based development. - As most data of this study show only the state-of-the-practice, we do not know

possible cause-effect relationships of our conclusions. We therefore had one hour discussion as the second part of the seminar to get feedback from the participants.

Twenty-three Norwegian industrial persons showed up in the seminar. More than

40% of them came from large-sized IT companies (more than 100 employees). 40% came from a medium-sized (with 20-99 employees) and 20% came from small-sized companies (less than 20 employees). We first showed our survey results in one hour presentation. On-going our talk, participants were able to interrupt and ask questions. In

Appendix A

235

the second part, we hypothesized some questions based on the survey result. We also listed our assumed explanations on these questions. Industrial participants gave their opinions to support or reject our explanations.

3. RESULTS Concerns and comments from industrial colleagues gave us valuable ideas on what we need to do in order to disseminate OTS based development theories into industry. We have summarized industrial concerns and comments into five barriers. 3.1 The barrier to disseminating the overall formal OTS-based development process The pre-study showed that most projects decided their main development process before they started to think about using OTS components [11]. The result from the survey follows the same trend. In the discussion section, we asked why they did not use the proposed OTS-based development processes as in [3,14]. Our assumed explanations were:

- Not necessary, as OTS is a minor part of the system - It is difficult to change (company level) Industrial answers: Industrial colleagues agreed with our second explanation. They

said people would not like to change the way they work. They think it is either too expensive or too difficult. They argued that there were few empirical studies to investigate the cost and benefit of the proposed COTS-based development processes. It is therefore difficult to convince industrial developers to use them. They do not want to change their development processes dramatically, being a painful procedure, without enough confidence.

Relevant studies: Some previous studies have analysed the barriers of using COTS

information technology based on lessons learned in small and medium sized companies (SMEs) [18]. The conclusions are:

- SMEs do not formally define their processes and tend to use tribal knowledge. - SMEs use flexible processes as a competitive advantage and take pride in that

flexibility. Changing this approach is difficult, because many SMEs equate process flexibility with having undocumented processes.

- The culture within SMEs is one of informal teamwork and therefore is resistant to formalism or change.

Our insight: In our study, we excluded companies with less than 5 employees

because most such companies do not have a formally defined process. Most companies in our study have one de-facto process, such as RUP or CMMI. The reason for not using a formal COTS process is not because they do not have any formal development process.

However, before adopting their process into a formal process, such as EPIC [1], the industrial practitioners will need evidences that the formal process is more cost-effective. It is also necessary for them to know how to downsize a formal process

Appendix A

236

according to different project contexts, such as time-to-market pressure, quality requirements and so on.

Barrier one: Without sufficient empirical evidence on OTS based development

processes, industrial project managers would not like to change their de-facto development process.

3.2 The barrier to disseminating the formal OTS selection process Although many formal OTS selection processes have been proposed [4, 9], results of this study and some other studies [20] showed that they were seldom used in practice. In the discussion section, we asked why they did not use these formal selection processes. Our assumed explanations were:

- Don’t know the formal selection process - Not necessary - It was not easy to use Industrial answers: Industrial participants agreed with our second explanation.

They argued that OTS component are not the key parts of the system in some cases. They just want to select one as fast as they can. These formal selection processes are not cost-effective for them. In addition, they argued that there were few empirical studies on when they should use the formal process instead of downloading a demonstration and trying it out, as the process discovered in study [11]. They were few studies showing that a formal process actually gave better results than informal processes.

Relevant studies: The balance between COTS uncertainty risks and risks resulting

from project delay has been investigated by an empirical study [16]. They concluded COTS assessment appears qualitatively intuitive for any given aspect. However there are often many factors that must be considered simultaneously. The impact of all considerations may be quite complex and counter intuitive.

Our insights: Most formal selection processes were proposed without evaluations on

their pre-conditions and their cost-benefit. For example, the selection process for fine-grained small OTS component (the object of this study) would be different with selection process for large OTS suites. The large OTS suites usually have more features than required from the customer. How to evaluate these extra features have not been investigated in current formal selection processes [15]. In addition, most selection processes assumed that the general software architecture was ready before OTS selection. However, our studies discovered that the OTS component can be selected in the pre-study phase, requirement phase, general design phase, detailed phase, and coding phase [11]. That is, how to customize the formal selection process according to little information needs future studies.

Barrier two: Little advice on how to customize a formal process according to

project context and OTS itself prevents the use of a formal selection process.

Appendix A

237

3.3 The barrier to sharing OTS based development knowledge inside company Our pre-study and some other studies [14] showed that a new role, an OTS knowledge keeper, is necessary in COTS-based development, Result of this survey showed that projects with a dedicate person keeping fresh of the knowledge of OTS component had fewer problems (see problems we listed in [12]) than projects without it. The results also show that the more project members had general experience with OTS-based development, the fewer problems they had in their projects. In the discussion section, we asked what other knowledge is necessary, in addition to the technical insights in concrete OTS components.

Industrial answers: Industrial participants argued that general risk management and cost estimation experience must be shared. However, they had no good idea on how to do it. Moreover, they were interested in how much effort the knowledge keeper needs to spend on OTS component based development and what kind of knowledge needs to be kept.

Relevant studies: In study [6], the authors concluded that a pre-fabricated, reusable

set of generic requirements, based upon the practical analysis in business environments of application and COTS middleware components characteristics helped to ensure the success of COTS middleware selection.

Our insight: Knowledge accumulation and transfer inside a software company and

software projects have been investigated in other software development fields. However, what knowledge should be documented and shared in OTS-based development has seldom been investigated. It is therefore difficult for industrial practitioners to collect and share relevant OTS based development knowledge.

Barrier three: Few studies on knowledge accumulation and transfer delayed the

dissemination of the OTS based development experience.

3.4 The barrier to using OSS components Our study compared the differences between projects using only COTS component with project using only OSS components. Results show that OSS users have serious problems in getting the support reputation information on OSS component providers [13]. We asked the question: How to evaluate the support reputation of OSS community?

Industrial answers: The conclusion from industrial participants is that it is not possible to get this information and ensure the technical support from OSS providers currently. They argued that most available information of OSS component support come from the previous experience, i.e. the previous response time, the bug fixing speed etc. Without a commercial contract, as with a COTS vendor, it is not possible to ensure the future response time and the quality of technical support because most technical support work is based on volunteers.

Appendix A

238

Relevant studies: OSS component have been used more popular in commercial projects. As discussed in [7], how to establish credible and mutually acceptable combinations of OSS delivery and procurement models therefore needs future research.

Our insight: Industrial practitioners inclined to use OSS components in their

projects. However, concerns on incredible technical support from OSS blocked the process.

Barrier four: Without credible and mutually acceptable combinations of OSS

components’ technical support, it is difficult for an industrial developer to use OSS in practice.

3.5 The barrier to doing requirements (re)negotiation in OTS-based development Our pre-study showed that customers’ involvement on OTS component selection and decision making could be a good risk mitigation strategy. It gives integrators the leeway to (re) negotiate requirements when the limitations of an OTS component are found at the very end [11]. However, data of this study shows that this strategy has been seldom used in practice. We asked the question: why customer had seldom been involved in COTS component decision making?

Industrial answers: Industrial colleagues explained that their customers either had no technical background to make such decisions or cared only the final product without considering the technical details. So, the integrators had to take all the responsibilities on OTS component decision making. As a result, the requirements (re)negotiation because of OTS component’s limitations is impossible in practice.

Relevant studies: Methods of requirements (re)negotiation have been proposed as

an important part of requirements engineering. However, both our pre-study [11] and another study [17] found out the same phenomena. If the COTS candidates could not feasibly satisfy pre-defined system project objectives, constraints, and priorities, these project objectives, constraints, and priorities could not be changed or negotiated to accommodate the constraints imposed by the available COTS. Therefore, building the required functionalities in-house became the only viable option.

Our insight: As it is difficult to discover all OTS components’ limitations at the

very beginning, it is necessary for integrators to be able to (re) negotiate the requirements with the customer. How to involve customers in OTS component decision making therefore needs future studies.

Barrier five: Few case studies have observed successful requirements

(re)negotiation in OTS based development. There are therefore few guidelines on when and how to involve customers in requirements (re)negotiation because of OTS limitations.

Appendix A

239

4. CONCLUSION AND FUTURE WORK This paper has reported some industrial obstacles to conduct OTS based theory into practice. We believe that answers to these barriers will help to improve OTS-based development practice in industry. In the future, we plan to study on these barriers by case studies and personal interviews. 5. ACKNOWLEDGMENTS This study was partially funded by the INCO (INcremental COmponent based development) project [8]. We thank the colleagues in these projects, and all the participants in the survey and seminar. REFERENCES Albert, C. and Brownsword L. Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview. SEI, Pittsburgh. http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html. 2002.

Boehm, B. W., Port, D., Yang, Y., and Bhuta, J. Not All CBS Are Created Equally COTS-intensive Project Types. In Proc. of the 2PndP International Conference on COTS-Based Software Systems (ICCBSS’03) (Ottawa, Canada, February 2003), LNCS Vol. 2580, Springer-Verlag, 2003, 36-50.

Brownsword, L., Oberndorf, T. and Sledge, C. Developing New Processes for COTS-Based Systems. IEEE Software, 71, 4, (July/August, 2000), 48-55.

Dorda, S. C., Dean, J. C., Morris, E., and Oberndorf, P. A Process for COTS Software Product Evaluation. In Proc. of the 1PstP Int. Conf. on COTS-Based Software Systems (ICCBSS’02), (Orlando, FL, USA, Feb. 2002), LNCS Vol. 2255, Springer-Verlag, 2002, 176-187.

Fitzgerald, B. A Critical Look at Open Source. IEEE Computer, 37, 7, (July 2004), 92-94.

Gorton, I. and Liu, A. Streamlining the Acquisition Process for Large-Scale COTS Middleware Components. In Proc. of the 1PstP Int. Conf. on COTS-Based Software Systems (ICCBSS’02), (Orlando, FL, USA, Feb. 2002), LNCS Vol. 2255, Springer-Verlag, 2002, 122-131.

Holck, J., Larsen, M. H., and Pedersen, M. K. Managerial and Technical Barriers to the Adoption of Open Source Software. In Proc. of the 4PthP Int. Conf. on COTS-Based Software Systems (ICCBSS’05), (Bibao, Spain Feb. 2005), LNCS Vol. 3412. Springer-Verlag, 2005, 289-300.

INCO project description, http://www.ifi.uio.no/~isu/INCO, 2000 Lawlis, P. K., Mark, K. E., Thomas, D. A., and Courtheyn, T. A Formal Processes for

Evaluating COTS Software Products. IEEE Computer, 34, 5, (May 2001), 58-63. Lawton, G. Open Source Security: Opportunity or Oxymoron? IEEE Computer, 35, 3,

(March 2002), 18-21. Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B. An Empirical Study of

Variations in COTS-based Software Development Processes in Norwegian IT Industry. In Proc. of the 10PthP IEEE Int. Metrics Symposium (Metrics'04), (Chicago, USA, September 2004), IEEE Press, 2004, 72-83.

Appendix A

240

Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and Bunse, C. A State-of-Practice Survey on Risk Management in Off-the-Shelf Component-Based Development. In Proc. of the 4PthP Int. Conf. on COTS-based Software System (ICCBSS’05), (Bibao, Spain Feb. 2005), LNCS Vol. 3412 Springer-Verlag, 2005, 278-288.

Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, M., Torchiano, M., and Morisio, M. An Empirical Study on Off-the-Shelf Component Usage in Industrial Projects. Forthcoming in the 6PthP Int. Conf. on Product Focused Software Process Improvement(PROFES 2005) (Oulu, Finland, June, 2005)

Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S. E. Investigating and Improving a COTS-Based Software Development Process. In Proc. of the 22PndP Int. Conf. on Software Engineering (ICSE 2000). (Limerick, Ireland, June, 2000). ACM Press, New York, NY, 2000, 31-40.

Perrone, V. A Wish List for Requirements Engineering for COTS-Based Information Systems. In Proc. of the 3PrdP Int. Conf. on COTS-Based Software Systems (ICCBSS’04) (Redondo Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-Verlage, 2004, 146-158.

Port, D. and Chen, S. Assessing COTS Assessment: How Much Is Enough? In Proc. of the 3PrdP Int. Conf. on COTS-Based Software Systems (ICCBSS’04) (Redondo Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-Verlage, Berlin Heidelberg New York (2004), 183-198.

Port, D. and Chen, S.,: Empirical Analysis of COTS Activity Effort Sequences. Proc. of the 3PrdP Int. Conf. on COTS-Based Software Systems (ICCBSS’04) (Redondo Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-Verlage, 2004, 169-182.

Robert, J., Buhman, C., Garcia, S., Allinder, D., Bringing COTS Information Technology into Small Manufacturing Enterprises. In Proc. of the 2PndP Int. Conf. on COTS-Based Software Systems (ICCBSS’03), (Ottawa Canada, Feb. 2003), LNCS Vol. 2580. Springer-Verlag, 2003, 187-195.

SEI (Software Engineering Institute). COTS-Based Initiative Description. http://www.sei.cmu.edu/cbs/cbs_description.html. 2004

Torchiano M. and Morisio M., “Overlooked Facts on COTS-based Development”, IEEE Software, 21, 2, (March/April 2004), 88-93.

Appendix A

241

P8. Reflections on Conducting a Survey

Reflections on Conducting an International Survey of Software Engineering

Reidar Conradi 1, 2, Jingyue Li 1, Odd Petter N. Slyngstad 1, Vigdis By Kampenes 2,

Christian Bunse 3, Maurizio Morisio 4, and Marco Torchiano 4 1 Department of Computer and Information Science Norwegian University of Science and Technology

NO-7491 Trondheim, Norway {conradi, jingyue, oslyngst}@idi.ntnu.no

2 Simula Research Lab P.O. Box 134, NO-1325, Lysaker, Norway

[email protected] 3 Fraunhofer IESE

Sauerwiesen 6, D-67661 Kaiserslautern, Germany [email protected]

4 Dip. Automatica e Informatica Politecnico di Torino Corso Duca degli Abruzzi 24, I-10129 Torino, Italy

{morisio, torchiano}@polito.it Abstract: Component-based software engineering (CBSE) with Commercial-Off-The-Shelf (COTS) or Open Source Software (OSS) components are more and more frequently being used in industrial software development. We therefore need to issue experience-based guidelines for the evaluation, selection and integration of such components. We have performed a survey on industrial COTS/OSS development in three countries – Norway, Italy and Germany. Concrete survey results, e.g. on risk management policies and process tailoring, are not being described here, but in other papers. This is a method paper, reporting on the challenges, approaches and experiences gained by conducting the main survey. The main contributions are as follows: At best we can achieve a stratified-random sample of ICT companies, followed by a convenience sample of relevant projects. This is probably the first software engineering survey using census type data, and has revealed that the entire sampling and contact process can be unexpectedly expensive. It is also hard to avoid national variations in the total process, possibly leading to uncontrollable method biases. Keywords: Surveys, component-based software engineering, COTS/OSS components, open source software, ICT industry. 1. Introduction Component-based software engineering (CBSE) is considered one of the most important technologies to reduce time-to-market and cost and to improve software quality. Although there are many proposed CBSE methods in research papers and some books, few representative industrial studies have been done to study such methods in practice.

Appendix A

242

Although the survey topic is CBSE, the focus of this paper is to convey some reflections and lessons learned in performing a large-scale, multinational survey of software engineering in the ICT industry. For a definition of “component”, see [30]. Let us summarize the overall survey process:

We scanned the literature to formulate research hypotheses and questions. Sampling was planned.

NTNU and the Simula Research Lab in Oslo performed a pre-study on COTS (Commercial-Off-The-Shelf) development through on-site, structured interviews of 16 projects in 13 companies in autumn 2003 [18].

Based on the pre-study, we started preparing the main study in late 2003. This included questions on OSS components, and extended the survey to Germany and Italy. We designed a more formal questionnaire for this study in spring 2004, translated it into Norwegian, German, and Italian and installed it on the SESE web server at the Simula Research Lab [28].

The main study was performed in May 2004 – March 2005. We got name lists of ca. 500 ICT companies in each country. For each company, we got hold of a possible respondent over the phone. This person was asked to identify a suitable OTS-based project (Off-The-Shelf, being either COTS or OSS), and encouraged to answer the questionnaire. Most respondents filled in the web-questionnaire him/herself, but some were filled in by phone interviews. Per 16 August 2005, 133 company responses have been recorded.

We are now analysing and publishing the results [19]. The survey has taken much more effort than anticipated, so we need to reflect on the choices, e.g. on sampling, approaching the companies, filling in the questionnaire etc.

The three research partners (NTNU/Simula, IESE, Torino group) have been working on software reuse, CBSE, process improvement, and industrial studies for over 15 years. We have planned and conducted the entire survey with no physical meetings, just email contact and two-weekly phone meetings – even the web server has been accessed remotely.

The main conclusions of this paper are that we at best can obtain a stratified-random sample of ICT companies, and thereafter a convenience sample of relevant projects. The sampling and contact process can be exceedingly expensive, and national process variations are difficult to avoid. This in turn can lead to uncontrollable method biases.

This paper is a method paper to reflect on the main study. The remainder of the paper is structured as follows. Section 2 presents some related work. Section 3 discusses definition of population, and Section 4 how to make a sample from this population. Section 5 discusses questionnaire design, and Section 6 discusses the company contact process. Section 7 has lessons learned, and Section 8 concludes the paper.

2. Related work on surveys of software technologies

2.1. Empirical study types for CBSE

There are many different types of empirical studies for software engineering [31]. Several guidelines have been published for how to conduct such studies [15] [4]. Aside from action research and case studies, the following two study types seem most relevant for CBSE, but are not discussed in [31]:

Appendix A

243

- Structured interviews (qualitative surveys) with an interview guide, to investigate rather open and qualitative research questions with some generalization potential. The Torino group has investigated the success of 32 reuse projects under the PIE program in EU [23]. In 2002, NTNU and the Torino group did structured interviews of seven small software companies in Norway and Italy. The results were synthesized as six theses [32] that contradicted many claims from the literature. This was followed up with a qualitative pre-study [18] in Norway by NTNU and the Simula Research Lab. The aim was to confirm these new findings and perhaps discover more revealing “truths”.

- Quantitative surveys with a questionnaire containing mostly closed questions. This is suited to investigate formalized hypotheses upon a sample from a larger population, where the results can be generalized (see 2.2). For instance, the above pre-study encouraged us to perform a main survey of industrial practice regarding OTS- (COTS- and OSS) based development [19].

Sometimes there is a sequence, even an iteration, in the studies. That is, we start with

local case studies and through interviews end up with a survey, reflecting increased theory building and generalization. As an example, the research partners have previously carried out case studies of CBSE at NASA [22], at Ericsson [20] and in German companies [2].

2.2. How to perform sound surveys in software engineering

There is a massive literature from the social sciences on how to perform valid and reliable surveys - e.g. regarding questionnaire design, sampling, and how to contact the respondents [10]. A challenge in international surveys is to stay methodologically similar across countries to avoid national bias, e.g. with sampling, contacting, and questionnaire fill-in.

Typical samples are based on convenience (reusing previous contacts), snowball (from a few seeds like a pyramid game), randomly selected subsets (from some population list), or stratified subsets (considering e.g. ICT sub-sector, company size, or country). Typical ways to fill in a questionnaire are by paper copy via post or possibly fax, by phone or site interviews, and recently by email or web. The questionnaire can be filled in either by the respondent her/himself or by an interviewer. Interviews, or phone contact in general, may result in a higher response rate, and may also contribute to quality-assure the returned data. However, they cost more to perform and some answers may be influenced by the interviewer, possibly resulting in dishonest answers (a “mode” problem, see later discussion in Section 5).

When asking about past events, in our case finished software projects, a respondent may have to think a bit, retrieve old documents, or speak to colleagues. This may take time and effort. We must therefore offer a print-out version, in addition to a web-questionnaire - read on the bus, or similar.

There have been published some more specific advice for surveys on software engineering [25], and for how to use web-based questionnaires in surveys [26]. However, even these more ICT-specific papers have not been of much help in our sampling and contact process.

Appendix A

244

2.3. A review of previous surveys in software engineering In light of the issues discussed in Sections 2.1 and 2.2, we will briefly present some published software engineering studies. The external validity or generalizability of all these surveys is speculatively or not at all discussed, because the population and associated sample are not clearly specified. First some non-CBSE surveys, summarized in Table 1, then an annotated list of CBSE surveys as shown in Table 2.

Table 1: Some non-CBSE surveys. Study topic Study object Sample selection Sample

size # responses

1.Software maintenance processes [13]

Portfolio of maintained software systems in ICT user-organizations

Random sample of 800 company members of Norwegian IFIP subsidiary

500 53 valid of 97 returned responses (Mean: 656 for all empl., 11 in IS-dept.)

2. SPI success criteria [6]

Software engineering processes of ICT companies

Complete member lists of two Norwegian ICT industrial organizations

154 120 (Mean: 196 empl., median: 55)

3. Estimation accuracy [21]

Finished software projects in ICT companies

From 100 largest Norwegian ICT companies, listed in a business magazine

37 18 companies with 52 projects (Mean: 141 empl., median: 75)

4. Effort estimation, COTS development [1]

Incremental COTS projects

Convenience sample of aerospace and defense contractors

100 20

5. Risk management policies [27]

Finished software projects in IT departments/ companies

Random sample of managers in IT companies, from a Finnish IT Assoc.

248 83

6. Efficiency of inspection proc’s [26]

State of the practice of inspections

Convenience sample of contact persons in German industry and academia

865 121

Appendix A

245

Table 2: Some CBSE studies, mostly surveys. Study topic Study object Sample selection Sample

size #

responses

1. Software reuse success factors [11]

Software organizations

Convenience sample of 113 persons in 29 organizations

NA

113 (Median: 25,000 empl., range 35 to 350,000)

2. Reuse proc’s with Ada [16]

Software professionals mostly in IT industry

Subscriber lists of three Ada magazines

NA 75

3. Software reuse success factors [23]

Finished EU projects on software reuse

From official list of 288 PIE projects in EU

32 projects

24 proj. interviews in 19 comp. (9 small, 9 medium, 6 large)

4. COTS dev. proc’s [32]

COTS-based projects

Convenience sample of 7 small sw companies

7 7 (Mean and median: 7 empl.)

5. Company attitudes towards OSS [3]

Companies using OSS

Snowball sample of ICT companies using OSS

275

146 (Mean: 17 empl., range 1-320)

6. Company experience and motivation on CBSE [7]

Companies practicing CBSE

Convenience sample 1300 109 (Median: ca. 100)

7. COTS/OSS dev. proc’s [19] (this study)

Finished sw projs with OTS dev.

From e.g. stratified lists of ICT companies from Census Bureau

1083

127 (Mean: 2240, median: 34)

As with these studies, most published surveys in our field fail to report on how the

population is defined, what ICT sub-sectors are covered, what sampling procedure has been applied, how representative the final sample is, what the mean and median company sizes are etc. Even most controlled experiments in our field, where the demand for rigor is high, have similar problems. Furthermore, generalizability raises more fundamental concerns than valid statistical sampling [17].

Appendix A

246

3. Defining the relevant population This has two parts: how to define the ICT sector (Section 3.1), and how to define our study objects inside this (3.2). We will use the terminology at (www.socialresearchmethods.net/kb/sampterm.htm):

- Theoretical population: all OTS-based software projects globally, - Study or target population: all OTS-based projects in the entire ICT industry in

the three countries. - Sampling frame: list of OTS-based projects in stratified subset of above (by

NACE codes and company size). - Sample: complete or random selection from the above sample frame. - Subsample: final or responding subset of sample.

3.1. Defining the population of ICT companies We generally have a problem in defining our study population of ICT companies, or the commercial ICT sector. In our study, this should be defined for three countries, and is expected to be representative for the ICT sector world-wide, the theoretical population. A quick glance at public statistics, however – using Standard Industry Codes from the US Census Bureau or their EU/OECD adaptation as four digit NACE codes [24] – gives little clue of what is ICT or not. For instance, the software business is bafflingly categorized under sector K for “Real estate, renting and business activities”. Thus, OECD, SSB (the Norwegian Census Bureau), and similar national agencies have jointly defined the following four sub-sectors to constitute the ICT sector in a country:

- 3x.xx ICT industry (Motorola, Nokia, etc.) - 5x.xx ICT trade (Computerland etc.) - 64.2x ICT telecom (AT&T, BT, Telenor etc.) - 72.xx ICT software, both vendors (Oracle etc.) and consultants (Accenture, Cap

Gemini etc.) We will use ICT sector as the general name. In Norway, this sector comprises 4-5%

of the economy, excluding public services, i.e. it is the biggest industrial sector after petroleum. Of course, the trade sub-section (5x.xx) is not very R&D-intensive, and hence outside the scope of our study.

In addition comes the ICT contents sector (database and web services, media industry etc.), expected to be finally classified by OECD et al. in 2007. There is also internal software development outside the ICT sector – as part of finance, engineering, public services etc. This is considered by the EU to be larger than the ICT sector itself. We will not deal with the contents sector or the non-ICT sector (including the public one) here, other than including some large non-ICT companies in our survey.

As a start, let us look at the structure of the ICT sector in Norway [29]. By employee is meant both owners and hired staff. By company (the Norwegian SSB uses the term enterprise) is meant a “legal” economic unit (typically a public shareholding company), not just an internal development department or a branch office. Some comments are:

Appendix A

247

- For software, ICT as well as the total, over 2/3 of the companies are small (1-4 employees) and with a median of 1 employee. The company average is 4.5 employees for software, 7 for all ICT, and 5 for the total. Only 0.5% (46) of the software companies, 1.5% of the ICT companies, and 0.5% of all companies have 100 or more employees.

- 8,228 (about 2/3) of the 12,861 ICT companies are software companies (NACE code 72.xx), having over 40% of the employees. Of the 7,171 software companies with 0-4 employees, 4,113 have one employee and 1610 have no employees!

- Ca. 30% of both software and ICT employees work in companies with less than 20 employees, 30% in companies with 20-99 employees, while 40% work in companies with 100 or more employees.

- There are regrettably some deviations in the totals, since they are drawn from different SSB sources.

- The ICT company distribution in Germany [8] and Italy [14] in 2002 shows that, as in Norway, most ICT companies are small. In Germany, the average software company has 7 employees and the ICT average is 13. 1% of German software companies have 100 employees or more. In Italy, the average software company has 4.5 employees, the median is 1 employee, and the average ICT company has 7 employees.

The software company distribution in the US in 1997 can be found in [9]. The US

pattern is similar to that described in Europe, with many small companies and a median of 1 (!) employee. However, the average company size is 150 employees in the US (4.5 in Norway, 7 in Germany, 7 in Italy), and 3% of the software companies had 100 or more employees.

The documented structure of the ICT sector in all these countries has clear consequences for defining our population and later samples of this (Section 4). We have four main sources for concrete names of ICT companies in our study population, called P1-P4:

- P1) A convenience subset of company contacts among the research partners, as in

the pre-study. Relevant contact persons are often available. - P2) The 100 largest ICT companies and similar, often summarized in annual lists

in business magazines. No contact person is usually given. - P3) Members of one or several industrial organizations. Norway has three such

organizations, Germany two, and Italy two. However, we do not know how comprehensive or accurate (“ICT-related”) such member bodies might be. In Norway, one of these organizations covers 5% of the total (600 of 13,000 companies). The listed contact is often an administrative person.

- P4) Complete set of registered ICT companies defined by public sources, e.g. by NACE-based lists from census bureaus or as sector lists in the yellow pages in phone books. Such sources should give a trustworthy coverage of the ICT sector.

In all such lists, there will be a substantial share of outdated information: ceased

companies, mergers or split-ups, or companies with e.g. new names, contact persons, locations, or phone/email addresses. All this reflects a very dynamic marketplace.

Appendix A

248

3.2. Defining the population of finished OTS-based projects For many ICT surveys, including this one, we do not only have a problem in defining the population of all ICT companies, and later samples with respondents of these. Our real problem is that the actual study object (experimental unit) – a special kind of project – is not identical to the primary object (accessible sampling unit), being a company or a respondent.

To be precise, our study object is “finished software project using OTS components”. The corresponding study topic is “OTS-based development processes”. In other studies, the study object could be “maintained software system” (to study software evolution processes), “finished software project” (to study estimation accuracy), or “applications of inspection methods” (to study related cost-efficiency).

Thus, we should first ask “every” ICT company (see 3.1) in the relevant countries to comprehensively document all recent instances of OTS-based projects, e.g. by an explorative pre-study. We might even conduct such a pre-study in two steps: first identify “typical” software projects, and then identify the “OTS” subset of this. But which project attributes should be collected to serve as later sampling criteria, and how to validate such criteria? We could consider company size, company profile, project size and duration, application domain, lifecycle process, developer skills, development technology, actual OTS-components being used etc. etc.

The above circumstances mean that many companies may have to give us project data, that later will be unused (“wasted”). Due to resource constraints and logistical problems, it seems totally unrealistic to perform such an explorative pre-study of a potential population of relevant study objects, both from the research and industry side.

Furthermore, only 1/3 of the sampled ICT companies in our survey had OTS-based development. This share would be close to 100%, if the study topic was effort estimation or software maintenance – i.e. activities that all companies routinely perform.

In surveys in medicine and social sciences, the population is often well-defined and coming from reliable public sources, although use of volunteers leads to an open-ended population. There, however, a responding person often matches the actual study object. 4. Sampling from the population A valid sample is a representative subset of the study population. Sections 3.1 and 3.2 have indicated some problems in defining the study population in software engineering surveys in the ICT sector. Then, how to pick a valid sample from a poorly or even un-documented population of study objects? Even if we had a complete list of all OTS-based projects, what would be the selection method – complete, stratified, random, or combinations?

At best, we can establish a hopefully representative population of primary objects (ICT companies) from which we can draw a valid sample. The study population of ICT companies may be defined as described in 3.1. We could then draw either a complete or a random-subset sample of primary objects as defined by P1-P3, or a stratified-random-subset sample from P4, as mostly done here.

The final sample will be a convenience sample of study objects (OTS-based projects), that the actual contact person (respondent) in the selected ICT company is

Appendix A

249

willing to provide. We can expect one or two such projects per company, but mostly none.

In our case, we needed about 50 study objects from each country to achieve statistical significance, given our research questions and related questionnaire questions. We therefore planned to ask about 250 companies in each country. Assuming an overall response rate of 1/3 or around 80 answers in each country, that would give us a satisfactory yield with some margin. (In reality, we needed 400-500 companies to get 50 responses per country). However, we should not draw a random sample from “all” ICT companies. Based on the documented structure of the ICT sector, our sample frame should ideally be stratified according to NACE code and company size.

For the Norwegian sample, our sample frame consists of three groups of ICT companies: large (all ICT sub-sectors including some non-ICT ones), medium (only software), and small (also software). Each group has roughly the same amount of software developers. The sample frame is then as follows:

- All large ICT companies (with 100 or more employees) with any ICT-relevant

NACE code; totally 114 companies and with average size of 330 employees. We sampled the topmost 100 of these 114 ICT companies, and 26 turned out to be software companies.

- Additionally, simply the 3 largest companies in 5 non-IT sectors (finance, engineering, energy, food processing, public/other). We sampled all these 15 companies.

- All medium-sized software companies, NACE codes 72.xx (20-99 employees), totally 228 companies with average size of 40 employees. From this we randomly sampled 100 companies (increased to 200, due to poor response rate).

- All small software companies, NACE codes 72.xx (5-19 employees), totally 783 companies and with average size of 9 employees. From these 783 companies we randomly sampled 100. We removed those with 0-4 employees, as they represent 87% (7171) of the companies, but only 19% (7514) of the employees. They are not expected to have very structured work processes, and are assumed irrelevant for our study.

We got company names for year 2002 for category 1, 3, and 4 of ICT-companies.

Randomization for category 3 and 4 was performed beforehand, and the data sent to us as spreadsheets via email. The 15 large non-ICT companies in category 2 are from a business magazine for year 2002.

Some observations: The Norwegian SSB struggles with incomplete or erroneous company lists. SSB has one part-time person on ICT and twelve on agricultural statistics! SSB has cooperated with one of the Norwegian industrial ICT organizations, IKT-Norge, since 1998 to help establish a reliable database of Norwegian ICT companies. IKT-Norge has recently set up the Infosector subsidiary [33] to sell services around this database. This subsidiary turned out to be the only practical way to get a stratified sample of the official database on electronic form, based on NACE category and company size. We hence got the requested company list very fast at a low price (1250 euro) – on the condition that we reported back anomalies in the data.

For the German sample, we got a list of company names for the years 2001 and 2002 from the German census bureau (Statistisches Bundesamt) for the four sub-sectors

Appendix A

250

of the ICT industry, enhanced by major non-ICT companies whose revenues are largely based on software (e.g., Siemens, DaimlerChrysler, Allianz). These lists were then validated using our customer database (e.g., removing falsely classified organizations or companies which clearly do not develop software, such as hardware stores). From the resulting sample frame of 430 companies, we selected a random sample for each category. Each sample was then cross-checked with the IESE customer database in order to obtain a contact point.

For the Italian sample, there is no national directory of ICT companies available to us. We therefore started with a study population of 43,580 assumed software companies from the yellow pages in 2001 (the same database as in the ELISS project on OSS [3]). From this population, we made a sample frame of 6,000 possible company contacts. From this sample frame, we randomly picked a sample of 747 companies, which we cross-checked (by web etc.) that their addresses were still valid and that they really did software development. We were left with 248 software companies, whereof 125 were using OTS components. 118 of these accepted to fill in the web-questionnaire. However, many potential respondents didn’t know what an OTS component was.

Tables 3 and 4 show the response rates in the three countries. In “Likely” we have pruned duplicates or irrelevant items such as holding companies. “Contactable” means the company coordinates were still valid.

Reflections on sampling and response rates: - For all countries, there were problems in classifying large and perhaps

international ICT corporations, with many daughter companies and divisions. That is, what is the relevant company unit? A company (legal economic unit) may have 10,000s of employees, but a local business unit (e.g. R&D department) less than 100. E.g., we twice “adjusted” a too high company size in Norway.

- The relatively high mean size for companies in Germany and especially in Italy is caused by a few large “outlier” companies.

- The local business unit size is considered the most representative measure. This does not vary much between the three countries, with the larger business units in Italy and Germany, as expected.

- In the pre-study in Norway, 17 of 34 (50%) of the ICT companies in our convenience sample had COTS-based development, and most of the remaining said they also used internal components. 13 of these 17 companies (almost 80%) agreed to participate and all responded.

- In the main study in Norway, 94 of 334 (28%) of the “contactable” companies had OTS-based development, and only 41 of the 94 (44%) eventually responded with a total of 47 projects. That gives an overall response rate of 12% (41 of 334).

- In Norway, 50 of 380 (13%) of the companies were no longer “contactable”, as the company lists were two years old in 2004. This is not surprising, and represents a very dynamic marketplace, as mentioned earlier.

- In Norway, we found that 70 of 285 (24%) software companies (NACE code 72.xx) claimed to have no software development at all. Many of these companies were sales subsidiaries of international ICT corporations, perhaps belonging to NACE code 51.84? However, no “penalty” for a company to be registered with a wrong NACE code exists!

Appendix A

251

- The response rate in Norway increases with decreasing company size. The response rate was especially low in large companies, cf. “gatekeeper” problem in Section 6.

Table 3: Response rate for Norwegian companies.

Init. sample

Likely (pruned s.)

Contactable

Do SW dev.

Use OTS comp.

Decl. willing

Final sample N

Large ICT comp. (where-of SW)

100 (26)

82 (23)

80 (23)

51 (21)

28 (13)

8 (6)

3 (2)

Large non-ICT comp.

15 15 15 9 1 0 0

Medium SW comp.

200 183 152 110 41 30 22

Small SW comp.

100 100 87 61 24 18 16

Sum main study

415 380 334 231 94 56 41

Sum pre-study

34 34 34 34 17 13 13

Table 4: Company responses in the three countries.

Initial sample

Likely (pruned s.)

Contact-able

Do SW dev.

Use OTS comp.

Declared willing

Final sample “N”

Mean/median in final sample :company

Mean/median in final sample: business unit

Norway

415 380 334 231 94 56 41 279 / 30 20 / 11

Germany

430 430 430 314 206 60 48 1933 / 200

74 / 40

Italy 747 747 323 248 125 118 38 3091/ 25

76 / 9

Total 1592 1557 1087 793 425 234 127 3371 / 44

56 / 20

Appendix A

252

5. Questionnaire design The (English) interview guide for the pre-study included standard meta-questions to test understandability, was held in two runs in fall 2003 on ten industrial colleagues in Norway, and lead to 10% of the questions being revised.

The main study questionnaire was designed in cooperation with the partners in Dec. 2003 – April 2004, relying on email correspondence and phone meetings. Most of the questions from the pre-study were reused, and we used much effort to reliably map the research questions and hypotheses into the questionnaire. The design went through 16 major revisions and this lead to changes in hypotheses, measure scale, and answer categories. Also, we created an analysis plan and conducted a questionnaire pre-test to increase survey quality.

The final questionnaire had an introductory letter, a set of concept definitions (component, OTS etc.) and some control questions to the reader. Thereafter 10 pages with 80 questions divided in 6 parts. Questions on personal or company matters were put at the end, emphasizing the confidentiality of the responses.

The pre-test had 18 respondents (industrial and academic) in the three countries. 15% of the questions were altered based on this. All work was done in English using an ms-word (.doc) file.

The questionnaire was finally translated into the native language of the three partners to ensure equal comprehension across cultures, and this revealed only few unclarities. Filling in took ca. 20-30 minutes.

Reflections on the questionnaire: - The questionnaire showed robust, with very few misunderstandings. Only four

incompletely filled-in forms (of 47) came in for Norway. - In Germany the fill-in procedure was improvised to be phone interviewing of

respondents. This introduced a mode problem that could affect “sensitive” answers, the response rate, or cause misunderstandings [12]. However, most questions are factual and technical, and the few national differences in German answer are not likely caused by mode problems, yet this mode difference represents an unplanned method bias.

- In Germany, we also had to offer extra anonymity for respondents and companies, in spite of an explicit confidentiality clause.

6. Contacting the companies Through a list of potential companies, we have to find a contact person in each company. The initial contact person (not the switchboard operator!) in a survey is called a gatekeeper [5]. Ideally, this person should connect us with a respondent for the actual study object. Note again, that the respondent is not our study object, neither is the company itself.

For convenience samples, there are few problems in getting hold of a potential respondent, as we often have personal and professional ties from earlier, joint activities. But for a large corporation, from a census or member list and with thousands of employees, it can be very frustrating to try to locate a quality manager, development/project manager, or just an “IT responsible”. The final struggle is to

Appendix A

253

convince a possible respondent over the phone to use his/her time and effort to fill in a questionnaire, sent by email.

Phone or site-interviews were discussed thoroughly in March 2004, but, despite recommendation [21] by researchers at Simula Research Lab, rejected, as we at that time estimated having 80-100 respondents per country and therefore did not have the capacity to do so (especially based on German arguments!).

Problems (low response rate etc.) were anticipated, so written procedural guidelines were made early:

1. Establish a representative sample of companies (Section 4). Generally, this meant

no pre-known gatekeeper, respondent, or phone number. 2. Rewards for survey participation were the final report, invitation to an industrial

sum-up seminar (www.idi.ntnu.no/grupper/su/cbse-survey/), and participation in a holiday lottery worth 1250 euro.

3. Call up the company, ask to be connected to a probable gatekeeper (first try the “IT manager”), and eventually to a prospective respondent. Explain the survey context including reward incentives, and ask if the company has software development and also OTS-based development. A positive return should yield an operative email address and a direct phone number.

4. The respondent is then emailed a “paper copy” of the questionnaire, telling her/him to please fill in the questionnaire online by the web (preferred option), on paper, or as an email attachment.

5. A few days later, re-phone and ask if the respondent is willing to participate in the survey. If yes, issue a userid/password to the web server.

6. Then wait for a final response, re-phone if necessary etc. This process could take months, even if the requested work took only half an hour.

7. In case of an incompletely or wrongly filled in questionnaire, we would try to call back.

8. Though this may sound smoothly, we have some reflections on the company contact process:

- All this turned out to be very costly and with a rather low response rate.

Given a person-year of total effort just in Norway, the cost of a filled-in questionnaire is ca. one person-week (2000 euro).

- As mentioned, the German fill-in procedure was performed through interviews, i.e. different from that used in Norway and Italy.

- Italy lost their research assistant (a summer student) in Oct. 2004, not replaced before March 2005. Likewise, IESE had 5 months delay (May-Sept. 2004) in PhD student upstart. Also, Norway hired an extra research assistant (upstarting PhD-student Slyngstad in March-June 2004). All this lead too much improvisation.

- There was marginal interest for any reward system - maybe we were not inventive enough?

Appendix A

254

7. Lessons learned 7.1. Method Aspects

- A pre-study often clarifies both research questions and concrete questions in a

later questionnaire. For instance, we “discovered” several variants of OTS-based development processes in our pre-study [18].

- A representative sample of national ICT companies assumes access to public data sources, like NACE-based company lists from census registers, but even such data may contain many errors. Close contact persons in such organs are needed to obtain stratified samples and expedite delivery of data in electronic form.

- Better criteria for selecting projects are needed. Since the study population (of finished OTS-based projects) was undefined, we had to do a two-phase approach: first pick a representative sample of companies, then a convenience sample of projects.

- For a variety of reasons it seems impossible to perform the same random sampling and data collection in different countries. The mentioned national differences now give an unknown mode effect in result interpretation between the countries. However, the observed national differences for Germany are unlikely to be linked to mode effects.

- Random sampling and the ensuing contact process appears to be much more expensive (5 times?) than convenience sampling. In our case, this is because we got stuck with no previous contact person, and scant pre-knowledge and “goodwill” from such companies, especially the large ones.

- If the final sample size is under 100 respondents, use phone interviews. This applies particularly when phone contact is anyhow needed, e.g. due to a non-convenience sample. A written procedure for the contact process is also needed.

- A baseline can be useful for comparison purposes. In our case, we did not ask the companies about non-OTS projects, so we missed this.

- Interviewing at this level requires solid background knowledge of the material; Company phone contact assumes at least PhD student qualifications.

7.2. Practical experiences

The industrial ICT environment seems extremely busy in recent years. It is very hard to ask for any extra commitment of effort. Also, of an audience of 25 at our final seminar, only 5 were respondents.

The SESE web server in Oslo has generally worked fine. One of the researchers had earlier experience as a webmaster, and spent two days in Oslo to learn about the SESE tool. The tool has already been used by over 20 studies, mostly controlled experiments, including a shared one by NTNU and the Simula Research Laboratory on software maintenance.

Appendix A

255

7.3. Research cooperation The researcher cooperation has gone very well, due to previous contacts, shared research interests, and good person chemistry. No physical meetings were held, and all data are sharable among the partners.

Extra resources through an external project would have helped to minimize national process differences, and having a social scientist on the team is strongly advised, as we found it highly useful.

Finally, the senior researchers must be thoroughly acquainted with details around sampling etc., i.e. getting their hands “dirty” by serving as research assistants. 8. Conclusion This study is the first survey on CBSE and indeed software engineering, which uses stratified-random sampling of ICT companies in several countries. All previous surveys have had problems in defining the study population and in selecting a valid sample of this, especially since the study object is a special project, not a company.

We had few real problems with questionnaire design and implementation, and some problems with randomized company sampling. On all these points we anticipated problems, and put in a lot of effort. However, unexpected much work went into contacting the companies and having the questionnaires filled in. National differences in sampling procedures and questionnaire fill-in may have introduced method biases, although not serious it seems.

We have learned a lot in running a representative, multinational survey, and also on OTS-based development. We plan to follow up the survey in other countries (China and US), and to supplement with more qualitative studies. 9. Acknowledgements This study was funded in part by INCO (Incremental COmponent based development, 2001-2004,http://www.ifi.uio.no/~isu/INCO) and SPIKE (SPI by better Knowledge and Experience), 2003-2005, http://www.idi.ntnu.no/grupper/su/spike.html). We thank our colleagues in these projects, social scientist Ola Listhaug at NTNU, and all participants in the survey. 10. References [1] C. Abts, B.W. Boehm, and E.B. Clark, “COCOTS: A COTS Software Integration

Lifecycle Cost Model – Model Overview and Preliminary Data Collection Findings”, Proc. of the ESCOM-SCOPE 2000, 18-20 April 2000, Munich, Germany, Shaker Publ., pp. 325-333.

[2] C. Atkinson, C. Bunse, and J. Wüst, “Driving Component-Based Software Development through Quality Modelling”, In A. Cechich, M. Piattini, and A. Vallecillo (Eds.), “Component-Based Software Quality: Methods and Techniques, State-of-the-Art Survey”, Springer Verlag LNCS 2693, 2003, pp. 207-224.

[3] Andrea Bonaccorsi and Cristina Rossi, "Why Open Source software may succeed”, Research Policy, 32(7):1243–1258, 2003.

Appendix A

256

[4] Reidar Conradi and Alf Inge Wang (Eds.), Empirical Methods and Studies in Software Engineering -- Experiences from ESERNET, Springer Verlag LNCS 2765, ISBN 3-540-40672-7, Aug. 2003, 278 pages.

[5] M. Corra and D. Willer, “The Gatekeeper”, Sociological Theory, 20(2):180-207, July 2002.

[6] Tore Dybå, "Factors of SPI Success in Small and Large Organizations: An Empirical Study in the Scandinavian Context", Proc. Joint ESEC'03/FSE-11 conf., Helsinki, Finland, 1-5 Sept., ACM Press, pp.148-157.

[7] Marisa Escalante and Pedro Gutierrez, “CBSE: State of the Practice”, Proc. 16th Int’l Conf. on Software & Systems Engineering and their Applications (ICSSEA'2003), 2-4 Dec. 2003, Paris, Vol. 2, Session 9: Reuse & Components, ISSN 1637-5033, 16 p.

[8] EuroStat office, EU, Luxembourg, http://europa.eu.int.comm/eurostat, 2002. [9] M. E. Fayad, M. Laitinen, R. P. Ward, “Software Engineering in the Small”,

CACM, 43(3):115-118, March 2000. [10] Floyd D. Fowler, Survey Research Methods, 3rd edition, Thousand Oaks, Sage,

1988. [11] W. B. Frakes and C. J. Fox, “Sixteen Questions About Software Reuse”, CACM,

38(6):75-87, June 1995. [12] Robert M. Groves et al., Survey Methodology, Hoboken, NJ, John Wiley, 2004. [13] Knut K. Holgeid, John Krogstie, Dag I. K. Sjøberg, “A study of development and

maintenance in Norway: assessing the efficiency of information systems support using functional maintenance”, Information & Software Technology, 42(10):687-700, 2000.

[14] Italian Census Bureau, www.istat.it, 2002. [15] Barbara A. Kitchenham et al., "Preliminary guidelines for empirical research in

software engineering", IEEE Trans. on Software Engr., 28(8):721-734, Aug. 2002. [16] N. Y. Lee and C. R. Litecky, “An Empirical Study of Software Reuse with Special

Attention to Ada”, IEEE Trans. on Sw. Engineering, 23(9):537-549, Sept. 1997. [17] A. S. Lee and R. L. Baskerville, “Generalizing Generalizability in Information

Systems Research”, Information Systems Research, 14(3):221-243, Sept. 2003. [18] J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An Empirical Study of

Variations in COTS-based Software Development Processes in Norwegian IT Industry", Proc. 10th IEEE Int’l Metrics Symposium (Metrics'04), Sept. 14-16 2004, Chicago,p.72-83 (pre-study).

[19] J. Li, R. Conradi, O. P. N. Slyngstad, C. Bunse, M. Torchiano, and M. Morisio, "Preliminary Results of a State-of-the-Practice Survey on Motivations of Using Off-The-Shelf Components", in Proc. 6th International Conference on Product Focused Software Process Improvement (PROFES'2005), 13-16 June, 2005, Oulu, Finland, 17 p. (on main study).

[20] P. Mohagheghi et al., "An Empirical Study of Software Reuse vs. Defect Density and Stability", Proc. 26th Int’l Conf. on Software Engineering (ICSE'2004), 23-28 May 2004, Edinburgh, Scotland, IEEE-CS Press, pp. 282-292.

[21] K. J. Moløkken-Østvold et al., “A Survey on Software Estimation in the Norwegian Industry”, Proc. 10th IEEE Int’l Metrics Symposium (Metrics'04), Chicago, USA, 14-16 Sept. 2004, p. 208-219.

Appendix A

257

[22] M. Morisio et al., “Investigating and Improving a COTS-Based Software Development Process”, Proc. 22nd Int’l Conference on Sw. Engineering (ICSE’00), Limerick, Ireland, IEEE-CS Press, 4-11 June 2000, pp.31-40.

[23] M. Morisio, M. Ezran, and C. Tully, “Success and Failure Factors in Software Reuse”, IEEE Trans. on Software Engineering, 28(4):340-357, April 2002.

[24] Nomenclature Generale des Activites Economiques dans I`Union Europeenne, European Union, Brussels, 2005, http://europa.eu.int/comm/competition/mer gers/cases/index/nace_all.html.

[25] S. L. Pfleeger and B. A. Kitchenham, “Principles of Survey Research”, 6 parts in ACM SIGSOFT Engineering Notes, Nov. 2001 – March 2003.

[26] T. Punter, M. Ciolkowski, B. Freimut, and I. John, ”Conducting On-line Surveys in Software Engineering”, Proc. 2nd Symp. of Emp. Sw. Eng. (ISESE’03), Rome, 30 Sept.–1 Oct. 2003, IEEE-CS Press, pp. 80-88.

[27] J. Ropponen and K. Lyytinen, “Components of Software Development Risk: How to Address”, IEEE Trans. on Software Engineering, 26(2):98-111, Feb. 2000.

[28] Dag I. K. Sjøberg et al., “Conducting Realistic Experiments in Software Engineering”, Proc. 1st Int’l Symp. on Empirical Sw. Eng. (ISESE’2002), 3-4 Oct. 2002, Nara, Japan, IEEE-CS Press, pp. 17-26

[29] Norwegian Census Bureau (SSB), Oslo, ICT company data 2002, http://www.ssb.no/emner/10 /03/ikt/.

[30] J. Li et al.: "An Empirical Study of COTS Component Selection Processes in Norwegian IT companies", MPEC'04 at ICSE ‘04, May 25, 2004.

[31] M. V. Zelkowitz and D. R. Wallace, "Experimental Models for Validating Technology", IEEE Computer, 31(5):23-31, May 1998.

[32] M. Torchiano, M. Morisio, “Overlooked facts on COTS-based Development”, IEEE Software, 21(2):88-93, March/April 2004.

[33] Infosector company, www.infosector.net

Appendix A

258

Appendix B

259

Appendix B: Other Papers [AP1] [Li06e] Jingyue Li, Axel Anders Kvale, and Reidar Conradi: A Case Study on Improving the Changeability of COTS-Based System Using Aspect-Oriented Programming. Journal of Information Science and Engineering, Vol. 22, No. 2, March 2006, pp. 370-390.

Abstract: More and more software projects are using COTS (commercial-off-the-shelf) components. One of the most challenging problems in COTS-based development is to evolve a system to follow changes in the customer requirements. It is therefore important to increase the changeability of the COTS-based system, so that new component can easily replace the problematic COTS components. Aspect-Oriented Programming (AOP) claims to make it easier to develop and maintain certain kinds of application. We performed a case study to investigate whether AOP can help to build an easy-to-change COTS-based system. This compared the changes when adding and replacing components between a COTS system implemented using Object-Oriented Programming (OOP) and the same system implemented using AOP. The results show that integrating COTS components using AOP may help to increase the changeability of the COTS-based system if the cross-cutting concerns in the code used to glue the COTS component are homogeneous (i.e., have a consistent application of the same or very similar policy in multiple places). Extracting heterogeneous or partially homogeneous cross-cutting concerns in glue-code as aspects does not give any benefits. This study also discovered some limitations in the AOP tool that makes it difficult or even impossible to integrate COTS components with AOP. Relevance to this thesis: This paper is relevant to the risk management section of this thesis. It reports a case study on mitigating the one typical risk, difficult to replace the problematic components, in COTS-based development.

My contribution: I contributed on 50% of the total work, including research design, data analysis and paper writing. Axel Anders Kvale worked on implementing the system, which includes 40% of the total work.

[AP2] [Li04b] Jingyue Li, Siv Hilde Houmb, Axel Anders Kvale: A Process to Combine AOM and AOP: A Proposal Based on a Case Study. Proceedings of the 5th Aspect-Oriented Workshop (In conjunction with UML 2004), Oct. 11, 2004, Lisbon, Portugal, http://www.cs.iit.edu/~oaldawud/AOM/AcceptedPapers/

Abstract: Traditional object-oriented programming (OOP) paradigm focused on structuring systems into distinguished objects that work together to realize a system. However, when dealing with non-functional or quality requirements, such as security and fault tolerance, these are not easily structured into separate objects, but do rather crosscut a set of objects. Aspect-oriented programming (AOP) separate crosscutting concerns into single units called aspects. Aspect-oriented modelling (AOM) techniques allow system developers to design and verify an aspect-oriented system on the modelling level. During a case study of re-engineering an object-oriented system using aspect-oriented programming,

Appendix B

260

we learned that well-designed aspect-oriented modelling (AOM) is essential to the success of aspect-oriented system. We also learned that current aspect-oriented programming tools (AOP) pose limitations on the design of an ideal aspect-oriented model. Based on lessons learned from this study we propose a process that handles aspects at two levels, both at the modelling level (AOM) and the programming language level (AOP). At the AOM level, aspects are identified and weaved together by AOM weaving to verify and do trade-off between different mechanisms. However, models are not woven together for the purpose of code generation based on a combined model. The actual weaving is done by the AOP compiler.

Relevance to this thesis: This paper is relevant to the risk management section of this thesis. It reports a proposal on avoiding the risks in COTS-based projects from the requirement and design phase.

My contribution: I contributed on 80% of the total work, including research design, theory building and paper writing.

[AP3][Conradi05b] Reidar Conradi and Jingyue Li: Observations on Versioning of Off-the-Shelf Components in Industrial Projects (short paper). Proceeding of the 12th International Workshop on Software Configuration Management (co-located with the ESEC/FSE 2005), Sept. 2005, Lisbon, Portugal, pp.33-42, ACM Digital Library.

Abstract: Using OTS (Off-The-Shelf) components in software projects has

become increasing popular in the IT industry. After project managers opt for OTS components, they can decide to use COTS (Commercial-Off-The-Shelf) components or OSS (Open Source Software) components. We have done a series of interviews and surveys to document and understand industrial practice with OTS-based development in Norwegian, German, and Italian IT industry. The perspective is that of a software architect or system integrator, not a developer or maintainer of such components. The study object is a completed development project using one or several OTS components. This paper reports on the versioning aspects of OTS components in such projects. We found that one third of the COTS components actually provided source code, in addition to all OSS components. However, OTS components were seldom modified (i.e. reused “as-is”), even if source code was available. Although backward compatibility of new releases did not cause noticeable problems for a single OTS component, using several different OTS components in a project caused difficulties in maintenance planning of asynchronous releases and system integration of new releases. Several new research questions have been formulated based on the results of this study.

Relevance to this thesis: This paper is relevant to the risk management section of this thesis. It reports on the versioning risks in OTS component-based development.

My contribution: I contributed 40% percent of the total work, including research design, data collection and data analysis.

[AP4] [Li05c] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Marco Torchiano, Maurizio Morisio, and Christian Bunse: Preliminary Results from a State-

Appendix B

261

of-the-Practice Survey on Risk Management in Off-The-Shelf Component-Based Development. Proceedings of the 4th International Conference on COTS-based Software System (ICCBSS 2005), Feb. 2005, Bilbao, Spain, Springer LNCS Vol. 3412, pp. 278-288.

Abstract: Software components, both Commercial-Off-The-Shelf and Open Source, are being increasingly used in software development. Previous studies have identified typical risks and related risk management strategies for what we will call OTS-based (Off-the-Shelf) development. However, there are few effective and well-proven guidelines to help project managers to identify and manage these risks. We are performing an international state-of-the-practice survey in three countries - Norway, Italy, and Germany - to investigate the relative frequency of typical risks, and the effect of the corresponding risk management methods. Preliminary results show that risks concerning changing requirements and effort estimation are the most frequent risks. Risks concerning traditional quality attributes such as reliability and security of OTS component seem less frequent. Incremental testing and strict quality evaluation have been used to manage the possible negative impact of poor component quality. Realistic effort estimation on OTS quality evaluation helped to mitigate the possible effort estimation biases in OTS component selection and integration.

Relevance to this thesis: This paper is relevant to the risk management section of this thesis. It reports risk management issues based on our preliminary data.

My contribution: I contributed on 70% of the total work, including research design, questionnaire design, data collection in Norway, data analysis and paper writing.

[AP5] [Li05d] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Marco Torchiano, Maurizio Morisio, and Christian Bunse: An Empirical Study on Off-the-Shelf Component Usage in Industrial Projects. Proceeding of the 6th International Conference on Product Focused Software Process Improvement, Oulu, Finland (PROFES 2005), June 2005, Springer LNCS Vol. 3547, pp. 54-68.

Abstract: Using OTS (Off-The-Shelf) components in software projects has become increasing popular in the IT industry. After project managers opt for OTS components, they can decide to use COTS (Commercial-Off-The-Shelf) components or OSS (Open Source Software) components instead of building these themselves. This paper describes an empirical study on why project decision-makers use COTS components instead of OSS components, or vice versa. The study was performed in form of an international survey on motivation and risks of using OTS components, conducted in Norway, Italy and Germany. We have currently gathered data on 71 projects using only COTS components and 39 projects using only OSS components, and 5 using both COTS and OSS components. Results show that both COTS and OSS components were used in small, medium and large software houses and IT consulting companies. The overall software system also covers several application domains. Both COTS and OSS were expected to contribute to shorter time-to-market, less

Appendix B

262

development effort and the application of newest technology. However, COTS users believe that COTS component should have good quality, technical support, and will follow the market trend. OSS users care more about the free ownership and openness of the source code. Projects using COTS components had more difficulties in estimating selection effort, following customer requirement changes, and controlling the component’s negative effect on system security.

Relevance to this thesis: This paper is relevant to the decision making section of the thesis. It reports our preliminary findings based on 115 projects.

My contribution: I contributed on 70% of the total work, including research design, questionnaire design, data collection in Norway, data analysis and paper writing.

[AP6] [Li04c] Jingyue Li, Finn Olav Bjørnson, Reidar Conradi, and Vigdis By Kampenes: An Empirical Study of COTS Component Selection Processes in Norwegian IT companies. Proceeding of the Int'l Workshop on Models and Processes for the Evaluation of COTS Components (MPEC'04 Arranged in co-location with ICSE'04), May, 2004, Edinburgh, Scotland, IEE Press, pp. 27-30.

Abstract: The use of commercial-off-the-shelf (COTS) software has become more and more important in software development. In COTS-based development, COTS component selection is the most crucial phase. Although some selection processes have been proposed, empirical studies are necessary to assess these processes. We describe an exploratory study by structured interviews of 16 COTS-based development projects in Norwegian IT companies. The results indicate that successful COTS component selection can be implemented without using formal processes, and projects with different contexts may use different selection processes. If members in new project has enough practical experience with actual COTS components, such experience can be the dominant factor in selection. In the case of using a new COTS component in the project, hands-on experimentation is needed as an effective way of evaluating the component.

Relevance to this thesis: This paper is relevant to the SPI section of the thesis. It reports the results related to COTS selection in the pre-study.

My contribution: I contributed to 80% of the total work, including research design, interview guide design, performing most interviews, data analysis and paper writing.

[AP7] [Li04d] Jingyue Li, Finn Olav Bjørnson, and Reidar Conradi: Empirical Study on COTS Component Classification, Proc. International Workshop on COTS Terminology and Concepts, Co-located with International Conference on Component-Based Software Systems (ICCBSS'04), 1-4 Feb., 2004, Redondo Beach, USA, http://softeng.polito.it/iwctc/accepted.html

Abstract: COTS-based development is gaining more and more attention. Effective COTS component classification will help integrators to successfully control the development process, such as selection and integration. In this paper, we present an empirical study to investigate the characterized classification

Appendix B

263

proposed by previous research. From the result of this study, we conclude that some attributes are not good because they are either not measurable or unnecessary. We also propose one other attribute that will affect the development process dramatically. Our future study will focus on investigating these attributes in a larger sample.

Relevance to this thesis: This paper is relevant to the SPI section of the thesis. It reports the results related to COTS classification.

My contribution: I contributed to 80% of the total work, including research design, interview guide design, performing most interviews, data analysis and paper writing.

Appendix B

264

Appendix C

265

Appendix C: Interview Guide for the Pre-study Our definition on concepts used in this questionnaire:

Project: Software development project, in which the respondent has been involved. We aim for finished projects and possibly with maintenance, and possibly with several releases. Component: Software components are executable units of independent production, acquisition, and deployment that can be composed into a functioning system. To enable composition, a software component adheres to a particular component model and targets a particular component platform. COTS (Commercial Of The Shelf) component: A component that:

• is either provided by some other organizations in the same company, or provided by external companies as a commercial product.

• is integrated into the final delivered system. • is not a commodity, i.e. not shipped with operating system, not provided with

the development environment, not generally included in any pre-existing platform

• is not controllable, in terms of provided features and their evolution System: Product delivered by the actual project, consisting of application plus components, but again excluding platform software (commodities like OS, DBMS etc.). Application: Code built in-house to provide certain functionalities of the system. Glueware: Code to solve possible mismatch between components, or between components and application/platform. Addware: Code to add the functionalities which are required but not provided by the components.

The relationship between the four last concepts is shown in the following figure.

ApplicationCOTS Components

Platform (OS, DBMS)

Glueware/ Addware

System

Questions for Comments: 1. Are the definitions easy to read as worded? □ Yes □ No 2. Do you need clarification for these definitions? □ Yes □ No 3. If you have comments on these definitions, please fill in here. 4.Can you give a example of the COTS component based on your understanding?

Appendix C

266

Section 1: Questions on the context of the project and system

1.1 Information about the project

The projects we want to investigate in this survey are finished software development project using one or more COTS components. If you have experience with several such projects, please select one that you are most familiar with and base your answers on this project. 1.1.1 What was the total staff size of the project? (full-time and part-time persons)

Full time: _________________ Part time: _________________ 1.1.2 What was the starting time of the project (mm/yy) / ? 1.1.3 What was the time of final delivery of the project (mm/yy) ___/____? 1.1.4 What was the development environment of the project?

• Programming Languages: ____________________ • CASE tools: ______________________________ • Overall develop process (e.g. waterfall, prototyping, incremental, etc.):

_________ 1.2 Information about the system

1.2.1 What were the main functionalities of the system? 1.2.2 What were the major types of industry / services (application domain) of the

customers of the system? [Mark one or more alternatives]

o Agriculture and forestry, fishing and hunting

o Mining and quarrying o Power, electricity, gas, water supply o Construction o Manufacturing o Whole trade o Retail trade o Transport, storage, and communication o Information(informatics/telecommunica

tion ) o Finance, Insurance o Real estate and rental and leasing

o Professional, scientific, and technical service

o Management of companies and enterprises

o Administration and support services

o Waste management services o Educational services o Health care and social assistance o Arts, entertainment, and

recreation o Accommodation and food

services o Public administration o Other sevices

Reference: North American Industrial Classification System (NAICS) Link: http://www.revenue.state.ne.us/tax/current/buscodes.pdf

1.2.3 What is the executing environment of the system? OS : ____________________________________________ DBMS: __________________________________________ Other Supporting Software: __________________________

Appendix C

267

1.2.4 What is the architecture of the system? If the architecture is essentially uniform, mark the one style which best describes

that architecture. If the architecture is a mix of multiple architectural styles, mark as many styles as needed. [Mark one or more alternatives] o Pipe & Filter o Real Time o Distributed (Client/Server) o Rule Based o Main/Subroutine o Transactional Database Centric

o Event Based o Layered o Multithreaded o Blackboard/ Repository o Closed Loop o Feedback Control o Other:____________________

1.2.5 What is the size of the delivered system?

Application : _____________ KB or __________________ (KLOC) Glueware : _____________ KB or __________________ (KLOC) Addware: _______________KB or __________________ (KLOC) COTS :__________________ KB or __________________ (KLOC)

1.2.6 What is the current version/release of the system (Version/Release) __ / __? 1.2.7 What is the total age of the current system after initial delivery (Years / Months)? 1.2.8 What was the emphasis on the Time-to-market of the system development?

• High: There was very restricted deadline. The time for system development was limited.

• Nominal: There was not very restricted deadline. The time for system development was fair.

• Low: There was no specific deadline. The time for system development was enough.

High Nominal Low Don’t know

1.2.9 What was the emphasis on the Reliability (failure rate) of the system?

• High: If a failure occurs, the losses are very high and are very hard to be recovered.

• Nominal: If a failure occurs, losses are fairly easily recoverable • Low: If a failure occurs, losses are easily recoverable (e.g., document

publishing) High Nominal Low Don’t know

1.2.10 What was the emphasis on the Security of the system?

• High: There were very specific requirements on preventing unauthorized access to, or modification of computer programs or data.

• Nominal: There were some specific requirements on preventing unauthorized access to, or modification of computer programs or data.

• Low: There were little specific requirements on preventing unauthorized access

Appendix C

268

to, or modification of computer programs or data. High Nominal Low Don’t know

1.2.11 What was the emphasis on the Performance of the system? • High: There were very strict requirements on the system response time. • Nominal: There were some requirements on the system response time. • Low: There were little requirements on system response time.

High Nominal Low Don’t know

Section 2. Questions on COTS components

2.1 What is the type of your COTS-based system?

[Mark one alternative] □ COTS-Solution system System relies on one (or more) core COTS

component(s). The system is built around these components. COTS components provide the main functionality of the system.

□ COTS-Integrated System System is built by integrating one or more COTS components around the central application. COTS components provide some functionality for the system.

2.2 Which COTS components were used in the project? And what kind of

functionality did they provide? (If there were more than three COTS components used in the project, please select the COTS components most relevant to your job in the project.) [Use text to answer the name and functionalities of COTS component]

COTS ID COTS Component Name

COTS Component Standard (COM, CORBA, EJB etc.)

Main Functionality

1 2 3

2.3 What was the highest level of stated management commitment to these COTS component in the project? (Use the COTS ID given to your components in question 2.2)

• Business top manager: The highest level of stated commitment to use this component is business top manager

• IT top-level manager: The highest level of stated commitment to use this component is IT-top manager

• IT project managers: The highest level of stated commitment to use this component is IT project manager

• No commitment: There is no management commitment to use this components, it is decided by software architects or

Appendix C

269

developers • Don’t know

[Mark one alternative per row below] COTS ID

Business Top manager

IT top-level manager

IT project manager

No commitment

Don’t know

1 2 3

2.4 What was the origin of these COTS components? (Use the COTS ID given to your components in question 2.2) • Component produced by contract: The COTS was acquired by asking a

component vendor to create a component for an agreed-upon fee. • Special version of commercial item: The COTS was produced by a vendor who

produces a customized version of a commercially available component. • Independent commercial item: The COTS was bought from market, the vendor

is selling a product to the world and has no knowledge of the individual purchaser’s requirements or constraints.

[Mark one alternative per row below] COTS ID Component

produced by contract

Special version of commercial item

Independent commercial item

1 2 3

2.5 What was your project members’ experience with these COTS components?

(Use the COTS ID given to your components in question 2.2) • Very much experience: All project members have the experience of using the

same COTS components. • Much experience: Roughly ¾ project members have the experience of using

the same COTS components. • Some experience: Roughly ½ project members have the experience of using the

same COTS components. • Little experience: Roughly ¼ project members have the experience of using the

same COTS components. • No experience: No project member has the experience of using the same COTS

components. [Mark one alternative per row below] COTS ID Very much

experience Much experience

Some experience

Little experience

No experience

1 2 3

Appendix C

270

2.6 What was the role of the COTS components at the architecture level of the system?

(Use the COTS ID given to your components in question 2.2) • Core: The COTS component provided the main functionality of the system.

The system was built around this COTS component. • Support: The COTS component provided some functionality to support the

central application of the system. [Mark one alternative per row below] COTS ID Core Support

1 2 3

2.7 What was the most influential information/method to support selection of COTS

components? (Use the COTS ID given to your components in question 2.2) • Experience: Based on own or other COTS users’ previous success or failure

experience. • Demonstration: Based on the result of the vendors’ demonstration. • Pilot: Based on the result of a short-term trial use or test of COTS. • Formal method: Based on the result of formal decision making process, such as

multi-attribute utility theory, Multi-criteria decision aid, weighted sum method, etc.

• Others: There are other information/methods except the alternatives listed above. Please specify

[Mark one alternative per row below] COTS ID Experience Demonstration Pilot Formal

method Others (Please Specify)

1 2 3

2.7.1 If you didn’t use Formal Selection Method (see question 2.7), why?

• I never heard about it • It isn’t cost-effective • It is not necessary • It is not easy/accurate to use • Others______________________

2.8 Please rank-order your top three non-technical factors based on their influence in COTS components selection. Mark three factors with rating 1, 2, and 3 (1 is the most influential factor, 3 is the least influential one)

Licensing arrangements Cost in general Market trend COTS reputation (maturity, stability)

Appendix C

271

Vendor reputation Vendor support capability

Others (please specify): _________________________

2.9 Please specify the importance of the following issues for the initial selection of COTS components.

• Integrations easiness: Whether the COTS components can be plugged into the system without extra integration code.

• Possible integrations effort: How much extra code may need to be produced to correct the possible mismatch between the COTS components and other parts of the system.

• Functionalities completeness: Whether the components have all the required functionalities.

• Possible functionalities supplement effort: How much effort may be used to add the required functionalities.

Scales: • Very Much: It is considered as the most important issue • Much: It is considered as a very important issue • Some: It is considered as an important issue • Little: It is not considered as an important issue • No: I didn’t consider this issue.

[Mark one alternative per row below] Very

much Much Some Little No

Integrations easiness Possible integrations effort Functionalities completeness Possible functionalities supplement effort

Other issues: 2.10 Characterize the causes of integration problems between COTS components, or integration problems between COTS components and other parts of the system.

• Do not support architecture style: The COTS components did not support the requirement of architecture style of the system.

• Support different architecture styles: The COTS component supported different architectural style with other components in the system.

• Do not support committed standard: The COTS components was claimed to support specific standard (COM, CORBA, EJB, etc.), but they didn’t

• Support different standards: The COTS components supported different standard (COM, CORBA, EJB, etc.)

• Support different standard versions: The COTS components supported different version of standard (COM, CORBA, EJB, etc.)

• Other problems: There are other integration problems, please specify; Scales:

Appendix C

272

• Very much: Almost all the integration effort was spent on solving this problem • Much: Roughly ¾ integration effort was spent on solving this problem • Nominal: Roughly ½ integration effort was spent on solving this problem • Some: Roughly ¼ integration effort was spent on solving this problem • Little: Very few effort was spent on solving this problem • No: No such problem, no integration effort was spent on solving this problem [Mark either No problem or one or several other alternatives per row below] Problem Very

much Much Nominal Som

e Little No

Do not support architecture style

Support different architecture style

Do not support committed standard

Support different standards Support different standard versions

Other problems (Specify): 2.11 How the COTS components were integrated into the system (possibly to solve the problems mentioned in 2.10)?

(Use the same COTS components and COTS ID as in question 2.4) • By Plug: Just plug in the COTS components to assemble an executable system

without any extra work. • By Configuration: Setting or defining shell parameters or configuration options

available for a COTS component, but which do not require modification of COTS source code.

• By Glueware: integrate COTS components mostly by using glueware (i.e. code to solve possible mismatch between components, or between components and application/platform.)

• By Addware : integrate COTS components mostly by using addware (i.e. code to add the functionalities which are required but not provided by the components.)

• By Source Modification: change the source code of COTS components so that it can be integrated

[Mark either By plug or one or several other four alternatives per row ] COTS ID

By Plug By Configuration

By Glueware

By Addware By Source Modification

1 2 3

2.12 What level of training and/or documentation for the COTS components was available AND obtained for the developers, either directly from the component vendors or through third parties?

Appendix C

273

• Very High: As much training and/or documentation obtained as needed • High: Roughly ¾ of the needed training and/or documentation obtained • Nominal: Roughly ½ of the needed training and/or documentation obtained • Low: Roughly ¼ of the needed training and/or documentation obtained • Very Low: No training and/or very little documentation obtained

[Mark one alternative per row below] COTS ID Very High High Nominal Low Very Low

1 2 3

2.13 How did you try to influence a COTS components vendor to change components to your own needs? [Mark one alternative per row below]

• No control: Cannot drive vendor to make the change and have to follow the market

• Pay vendor: Pay the vendors to change it to meet my requirements • Press vendor: Put pressure on the COTS vendor in order to make them change

the OTS wthout pay them • Merge with or buy vendor: Merge with or buy the COTS vendor firm and make

them change the component • Other methods:___________________

COTS ID No control Pay vendor Press

vendor Merge or buy vendor

Others (specify)

1 2 3

2.14 What was the effect on certain properties by using COTS components in your system?

• Strong positive effect: Using COTS contributed much in achieving the planned result

• Some positive effect: Using COTS contributed some in achieving the planned result.

• No effect: Using COTS had no contribution in achieving the planned result • Some Negative effect: Using COTS hindered some in achieving the planned

result • Strong Negative effect: Using COTS hindered much in achieving the planned

result [Mark one alternative per row below]

The effect on certain properties Perceived effect on certain properties

Strong positive effect

Some positive effect

No effect

Some Negative effect

Strong Negative effect

Don’t know

Appendix C

274

Time-to-market System Reliability

System security System performance

Development cost

Maintenance cost

2.15 Would you like to use these COTS components in the future projects? [Mark one alternative]

□ Yes, I would like to use these COTS components in the future projects. □ No, I don’t want to use these COTS components in the future projects. □ Don’t know.

Section 3. Questions on the context of respondent and business unit

3.1 Personal Information about the respondent 3.1.1 What is your position in the business unit? [Mark one or more alternatives]

□ IT manager □ IT project manager □ Software architect □ Software developer □ Other level (please specify): _______________

3.1.2 How long have you been working in the current business unit? Years: __________________________ 3.1.3 How long have you been working with software development?

Years: _________________________ 3.1.4 How long have you been working with COTS components?

Years: __________________________ 3.1.5 What is your principal education degree [Mark one alternative]?

□ Bachelor □ Master

□ Ph.D □ Other education: ___________________ 3.1.6 Is your principal degree in software (informatics/computer science/telematics)?

□ Yes □ No 3.2 Information about your Company 3.2.1 Type of your Company [Mark one alternative]

□ Stand-alone: The highest reporting entity with no parent organization above it. □ Subsidiary: An independent entity with majority interest held by a parent.

3.2.2 Ownership of your Company [Mark one alternative]

Appendix C

275

□ Publicly traded (A entity whose stock is publicly traded on stock exchange) □ Privately held company (A for-profit entity that is privately owned) □ Government, education, or nonprofit organization

3.2.3 Staff size of your Company (full-time and part-time persons)? Full time:____________________________________________ Part time:____________________________________________ 3.2.4 What is the main business areas of your Company [Mark one alternative]

□ IT/Telecommunication industry (e.g. Ericsson, Telenor, etc.) □ Software House / Software Vendor (e.g. Opera) □ Software / IT consulting company (e.g. Accenture) □ Others software intensive company (Please specify):

Section 4. Extra questions (will only be used in the personal interview, the alternatives of each question are different viewpoints that you can talk with.) 4.1 Motivation of using COTS components Why did you use COTS components instead of building components yourself?

□ Business issue (time-to-market, cost effective, partner adaptability… ) □ Technical issue (catch up with new technology, reliability, security, maintainability…) □ Organization (key technical person leave…) □ Others:_______________________________

4.2 Process issues 4.2.1 Please describe briefly the overall development process used in the project?

(Elaborate on question 1.1.4)

4.2.2 Did your business unit have any specific processes concerning the selection and use of COTS components before the project started?

If yes, please explain briefly what kind of processes: (e.g. activities, roles, responsibilities)

4.2.3 Did you have to introduce or make any process changes during the project related to the selection or use of COTS components? If yes, please explain briefly the situation and the changes introduced:

(e.g. new or changed activities, roles and responsibilities)

4.2.4 What impact on the development process did the use of COTS components have for future project? o ease of change of software o ease of integration of software o availability of certification o language/middleware lock-in o control on product customization o architectural constraints and

mismatches o marketplace competition

o redistribution issues (both legal and commercial)

o adaptability o learnability o setup effort o reusability across projects o availability of required functionality o different integration problems

Appendix C

276

o porting and adaptation effort o acquisition and maintenance cost o configuration management o platform constraints integration

effort

o the choice of the product can be dictated by external factors

o customization cost

4.3 Component selection 4.3.1 Did you (re) negotiate the requirements with the customer? What was the reason?

o The requirements had to be (re) negotiated, mainly because the software architecture style has been decided separately and could only partially satisfy the requirements.

o The requirements had to be (re) negotiated, mainly because the available COTS components could not fully satisfy them.

o Other reasons :___________________________________________ o No requirements (re) negotiation was performed.

4.4 System maintenance and evolution issues 4.4.1 What part of the system had the largest amount of defects (trouble reports) during the first 6 month of system maintenance? [Mark one alternative]

□ The application part of the system □ The COTS components of the system □ The glueware/addware of the system □ Don’t know

4.4.2 What part of the system had the largest amount of change requests (not covering trouble reports, e.g. upgrade) during the first 6 months of system maintenance? [Mark one alternative]

□ The application part of the system □ The COTS components of the system □ The glueware/addware of the system □ Don’t know

4.4.3 How could you ensure the quality (reliability, security, etc.) of your COTS-based system in the process of system evolution (system update, COTS components update, etc.)?

o Technical (testing, configuration management, version control, ...) o Vendor relationship (technical support, contract, …) o Others:______________________________

4.5 General lessons learned and experience of using COTS components in your system

o Business o Technical o Process o Organization o Others: ______________________________

Appendix C

277

In the questionnaire we used, there is a meta-question after some question (1.2.4, 1.2.5, 1.2.8 to 1.2.11, 2.1, 2.4 to 2.14, and 4.4) to evaluate the quality of the questions as following. Due to the space reason here, they have been removed. Questions for Comments: 1. Is this question easy to read as worded? □ Very easy to read □ Easy to read □ Hard to read

2. Do you need clarification for this question? □ I am totally confused with this question □ I need some clarifications for this question □ I don’t need clarifications for this question

3. If you have comments on this question, please fill in here.

Appendix C

278

Appendix D

279

Appendix D: Questionnaire in the Main Study Our definition of concepts used in this questionnaire: Project: A software development project, in which the respondent has been involved. We aim for completed projects, possibly with maintenance, and possibly with several releases. Customer: Representing either an end-user, or a contractor to which your organization is a subcontractor. Component: Software components are program units of independent production, acquisition, and deployment that can be composed into a functioning system. We limit ourselves to components that have been explicitly decided either to be built from scratch or to be acquired externally as an OTS-component. That is, to components that are not shipped with the operating system, not provided by the development environment, and not included in any pre-existing platform. That is, platform (“commodity”) software are not considered, e.g. an OS like Linux, DBMSes, various servers, or similar software. Furthermore, components usually follow some component model, as expressed by the COM, CORBA, EJB, .Net, or Web Service standards, or they can be a C++/Java library. OTS (Off-The-Shelf) component: A component that:

• is provided (by a so-called provider) from: - a commercial vendor (Commercial-Off-The-Shelf, COTS), or

- the Open Source community (Open Source Software, OSS). The components may come with certain obligations, e.g. payment or licensing terms. NB: we do not consider proprietary components developed by some other organization in the same company, i.e. internal reuse.

• is not controllable, in terms of provided features and their evolution. • is mainly used as closed source, i.e. no source code is usually modified, even it

is available. Typical COTS components are: Vector draw professional (3D modelling),

Shamman Reports for Excel (Database reporting), Dynamic Cube (OLAP) etc. Typical OSS components are: Xerces (XML parser), Kiwi Toolkit (Class library), OpenEJB, Python (Interpreter) etc.

System: Product delivered by the actual project, consisting of application plus components, but again excluding platform software. Application: Code built in-house to provide certain functionalities of the system. Glueware: Code to solve possible mismatch between components, or between components and application/platform. Addware: Code to add the functionalities which are expected but not provided by the components. The relationship between the four last concepts is shown in the following figure:

Appendix D

280

Question for comments. Can you mention some OTS components based on your understanding? Section 1: Questions on the context of the project and system

1.1 Information about the project The projects we want to investigate in this survey are completed software development projects using one or more OTS components. If you have experience with several such projects, please select the one that you are most familiar with, and base your answers on that project. 1.1.1 What was the total staff size of the project (full-time and part-time persons)?

Full time: _________________ Part time: _________________ 1.1.2 How large a percentage of the project members have previous experience with general OTS-based development __ (%)? 1.1.3 What was the starting time of the project (mm/yy) / ? 1.1.4 What was the time of first complete delivery from the project (mm/yy) ___/____? 1.1.5 What was the development environment of the project (describe textually)?

• Programming Languages: ____________________ • CASE (Computer Assistant Software Engineering) tools: _______________ • Main development method (RUP etc.): _______________

1.2 Information about the system 1.2.1 What were the names and the main functionalities of the system (describe textually)? 1.2.2 What was the emphasis on the following characteristics of the system?

[Mark one alternative per row] Very

low Low Medium High Very

high Don’t know

a) Time-to-market b) Effort (cost) c) Reliability d) Security e) Performance f) Maintainability

Appendix D

281

g) New functionality (first launch in the market)

h) Improved functionality (over competitors)

i) Other (please specify): 1.2.3 What were the major types of industry / services (application domain) of the customers of the system? [Mark one or more alternatives]

□ Bank/ Finance/ Insurance □ Other private services (consulting, wholesale, retail, entertainment, …) □ Public sector □ ICT sector □ Traditional industry/engineering/construction □ Other (please specify): _______________

1.2.4 What is the executing environment of the system (describe textually)? OS (Operating System):_____________________________________ DBMS (Database Management System): _______________________ Other Supporting Software: __________________________________

1.2.5 What is the current version/release of the system (Version/Release) ___/___? Section 2: Questions on the motivation of using OTS components

2.1 To what extent did you expect the chosen OTS component(s) to contribute to the

following project/system targets? [Mark one alternative per row]

Project/system targets Don’t agree at all

Hardly agree

Agree some what

Agree mostly

Strongly agree

Don’t know

a) Shorter time-to-market b) Less development effort/cost

c) Less maintenance effort/cost

d) Larger market share e) Compliance with industrial standards

f) Keeping up with the newest technology

g) Better system reliability h) Better system security i) Better system performance j) Other (please specify):

2.2 What were the specific motivations for using OSS Component(s) in your project? [NB: If you did not use an OSS Component in the project, please skip this question]

[Mark one alternative per row]

Appendix D

282

Motivations Don’t

agree at all

Hardly agree

Agree some what

Agree mostly

Strongly agree

Don’t know

a) Reduce risk of provider going out of business

b) Reduce risk of provider changing market strategies

c) Reduce risk of selected component evolving into an unwanted direction

d) Component can be acquired for free

e) Source code is available and can easily be changed

f) It was decided by the customer

g) Political reasons (company policy, licensing conditions)

h) Other (please specify):

2.3 What were the specific motivations of using COTS Component(s) in your project? [NB: If you did not use a COTS Component in the project, please skip this

question] [Mark one alternative per row]

Motivations Don’t agree at all

Hardly agree

Agree some what

Agree mostly

Strongly agree

Don’t know

a) Reduce risk of bad quality, because components were paid for

b) Reduce risk of poor security, because source code was closed/ unavailable

c) Provider could give adequate technical support

d) Provider may have investigated the market trends, and will give a component following them

e) It was decided by customer f) Political reasons (company policy, licensing conditions)

g) Other (please specify):

Appendix D

283

Section 3: Questions on the general development processes

3.1 Please describe the main development process (e.g. waterfall, prototyping, incremental, agile etc. – describe textually): 3.2 Who decided the main (i.e. lifecycle) process? [Mark one or more alternatives]

□ Company/department rules □ Project manager □ Software architect □ Software developer □ Customer □ Other (please specify): _______________

3.3 When was the main process decided? [Mark one alternative] □ Before the “acquire vs. build” decision, i.e. before the decision whether to

acquire OTS components instead of building them in-house □ After such an “acquire vs. build” decision

3.4 Were there any dedicated personnel in your project to keep track of OTS component knowledge/information? [Mark one alternative]

□ Yes □ No

If Yes, what is the role and OTS component-relevant experience of this person? Section 4: Questions on the project level risk management

4.1 What is your opinion on the following aspects (risks) of your OTS-based project?

[Mark one alternative per row] Don’t

agree at all

Hardly agree

Agree some what

Agree mostly

Strongly agree

Don’t know

a) The project was delivered long after schedule

b) Effort to select OTS components was not satisfactorily estimated

c) Effort to integrate OTS components was not satisfactorily estimated

d) OTS components negatively affected system reliability

e) OTS components negatively affected system security

f) OTS components negatively affected system performance

Appendix D

284

g) Requirements were changed a lot

h) OTS components could not be sufficiently adapted to changing requirements

i) Project could not (re)negotiate requirements with the customer, if OTS components could not satisfy all requirements

j) It was difficult to identify whether defects were inside or outside the OTS components

k) It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles

l) It was difficult to update the system with the last OTS component version

m) OTS components were not satisfactorily compatible with the production environment when the system was deployed

n) Information on the reputation and technical support ability of provider were inadequate

o) Provider did not provide enough technical support/ training

4.2 What actions were performed during development or maintenance of the project?

[Mark one alternative per row] Don’t

agree at all

Hardly agree

Agree somewhat

Agree mostly

Strongly agree

Don’t know

a) Customer had been actively involved in the “acquire” vs. “build” decision of OTS components

b) Customer had been actively involved in OTS component selection

c) OTS components were selected mainly based on architecture and standards compliance, instead of expected

Appendix D

285

functionality d) OTS components qualities (reliability, security etc.) were seriously considered in the selection process

e) Effort in learning OTS component was seriously considered in effort estimation

f) Effort in black-box testing of OTS components was seriously considered in effort estimation

g) Unfamiliar OTS components were integrated first

h) Did integration testing incrementally (after each OTS component was integrated)

i) Local OTS-experts actively followed updates of OTS components and possible consequences

j) Maintained a continual watch on the market and looked for possible substitute components

k) Maintained a continual watch on provider support ability and reputation.

Section 5: Questions on the OTS component level risk management 5.1 How many different OTS components were used in the project (Please fill in a

number)? 5.2 Which were the name, version and functionality of OTS components used in the

project? [NB: If there were more than five OTS components used in the project, please

choose five you are most familiar with. Also select what you think is the most important OTS component and fill it into component Comp.1 for later use.]

OTS component ID

Full name and version (descriptive free text)

Origin (write COTS or OSS)

Main functionality (descriptive free text)

Comp.1 Comp.2 Comp.3 Comp.4 Comp.5

Appendix D

286

5.3 How large a percentage of the application functionality in the whole system was provided by the selected OTS component Comp.1 ___________________(%)?

5.4 What was the source code status of the selected OTS component Comp.1? [Make one alternative per row] OSS, i.e. with source

code available COTS, but with source code available

COTS, without source code

Comp.1 5.5 Have you read parts of the source code of OTS component Comp.1?

[NB: If the source code of Comp.1 is not available, please skip this question] [Make one alternative per row] Very

little Little Some Much Very

much Don’t know

Comp.1 5.6 Have you modified parts of the source code of OTS component Comp. 1?

[NB: If the source code of Comp.1 is not available, please skip this question] [Make one alternative per row] Very

little Little Some Much Very

much Don’t know

Comp.1

5.7 What was the technical base of the selected OTS component Comp.1? [Mark one or more alternatives]

o .Net o ActiveX o Java Class/ Library o Java Applet/ Servlet o Java Bean/Enterprise Java Bean o C/C++ class/Library o MFC

o DLL o VBX/ VB Class Library o COM/DCOM o CORBA o Windows Static Link Library o Windows Foundation Class (WFC) o Others (specify):____________

5.8 What was your project members’ experience with the selected OTS component

Comp.1? [Make one alternative per row] Very little Little Some Much Very

much Don’t know

Comp.1

5.9 When were the selected OTS components Comp.1 selected? [Make one alternative per row]

Pre-study phase

Requirements/Analysis phase

Overall design phase

Detailed design phase

Coding phase

Don’t know

Comp.1

5.10 Did you encounter some of the following aspects (risks) with the selected OTS component Comp.1? [Make one alternative per row]

Appendix D

287

Yes No Don’t know

a) The OTS component did not follow the industry standard (COM, CORBA etc.) as the provider claimed (i.e. glueware needed)

b) The OTS component had mismatching architectural assumptions towards other parts of the system (i.e. glueware needed)

c) The OTS component could not provide all required functionality (i.e. addware needed)

d) The recent OTS component versions were not backwards-compatible with the first version

e) The provider did not want to or could not change OTS component functionality as requested

5.11 Have you performed some of the following actions for the selected OTS

component Comp.1?[Make one alternative per row] Yes No Don’

t know

a) Searched Internet for possible OTS component candidates.

b) Got recommendation of possible OTS component candidates from the customer

c) Got recommendation of possible OTS component candidates from a local colleague/OTS-expert

d) Used a formal decision-making method to compare possible OTS component candidates, e.g. with weighted evaluation criteria

e) Limited possible candidates into 1-3 components, by reading literature or other documentation

f) Did “hands-on” try-out of 1-3 components, e.g. on a down-loaded demo version.

5.12 What is your opinion on the following aspects of the selected OTS component

Comp.1? [Make one alternative per row] Don’t

agree at all

Hardly agree

Agree somewhat

Agree mostly

Strongly agree

Don’t know

a) Comp. 1 was integrated successfully

b) Comp. 1 was maintained successfully so far

Appendix D

288

Section 6: Questions regarding the context of respondent and business unit [NB: The supplied information in 6.1 and 6.2 will be treated confidentially.] 6.1 Personal Information about the respondent 6.1.1 What was your position in this project? [Mark one or more alternatives]

□ Project manager □ Software architect □ Software developer □ Other level (please specify): _______________

6.1.2 What is your current position in your business unit? [Mark one or more alternatives]

□ IT manager □ Project manager □ Software architect □ Software developer □ Other level (please specify): _______________

6.1.3 How long have you been working in the current business unit? Years: __________________________ 6.1.4 How long have you been working with software development? Years: _________________________ 6.1.5 How long have you been working with OTS components? Years: __________________________ 6.1.6 How many OTS component based projects have you been involved in? Projects: __________________________ 6.1.7 What is your principal education degree? [Mark one alternative]

□ Bachelor □ Master □ Ph.D. □ Other education: ___________________

6.1.8 Is your principal degree in software (informatics/computer science/telematics)? [Mark one alternative]

□ Yes □ No

6.2 Information about your company 6.2.1 What is the type of company? [Mark one alternative]

□ Stand-alone: The highest reporting entity with no parent organization above it. □ Subsidiary: An independent entity with majority interest held by a parent.

6.2.2 What is the ownership of your company? [Mark one alternative] □ Publicly traded on a stock exchange □ Privately held company □ Government, education, or nonprofit organization

6.2.3 What is the staff size of your embedding company (full- & part-time persons) ____?

What is the staff size of your local business unit (full- & part-time persons) ___? [This staff size may be the same for small and medium sized companies]

Appendix D

289

6.2.4 What is the main business areas of your company?[Mark one alternative]

□ IT/ Telecom industry (Ericsson, Nokia, NERA, Nordic VLSI etc.) □ Telecom service provider (Telenor, NetCom etc.) □ Software House / Software Vendor (Opera, MicroSoft etc.) □ Software / IT consulting company (Accenture, Cap Gemini E&Y, TietoEnator

etc.) □ Other software-intensive company (please specify): ___________________

Note: In the main study, the questionnaire was translated into local language (Norwegian, German, and Italian) and published in the web for filling in.

Process Improvement and Risk Management in Off-The · PDF fileProcess Improvement and Risk...

Documents

Transcript of Process Improvement and Risk Management in Off-The · PDF fileProcess Improvement and Risk...