Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering...

27
Maturing Requirements Engineering Process Maturity Models Pete Sawyer Computing Department, Lancaster University, Lancaster, UK, LA1 4YR tel: +44 1524 593780 fax: +44 1524 593608 [email protected] KEYWORDS Socio-Technical Systems, Requirements Analysis, Requirements Definition, Requirements Specification, Process Improvement

Transcript of Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering...

Page 1: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Maturing Requirements Engineering Process Maturity Models

Pete Sawyer

Computing Department, Lancaster University, Lancaster, UK, LA1 4YR

tel: +44 1524 593780

fax: +44 1524 593608

[email protected]

KEYWORDS

Socio-Technical Systems, Requirements Analysis, Requirements Definition, Requirements Specification, Process Improvement

Page 2: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Maturing Requirements Engineering Process Maturity Models

ABSTRACT

The interest in Software Process Improvement (SPI) in the early 1990s stimulated

tentative work on parallel models for Requirements Engineering (RE) process

improvement in the late 1990s. This chapter examines the role of SPI and the

implications of the exclusion of explicit support for RE in the most widely-used SPI

models. The chapter describes the principal characteristics of three RE-specific

improvement models that are in the public domain: the Requirements Engineering

Good Practice Guide (REGPG), The Requirements Engineering Process Maturity

Model (REPM) and the University of Hertfordshire model. The chapter examines the

utility of these models and concludes by considering the lessons learned from

industrial pilot studies.

INTRODUCTION

The risks posed to software development projects by weak requirements engineering

(RE) practice have become widely recognised during the last decade. This has

spawned a great deal of investment in RE methods, tools and training by practitioner

organisations, and in RE research by the wider software and systems engineering

communities.

The focusing of attention on RE during the early 1990s coincided with the

deployment of software process improvement (SPI) that was stimulated by

Humphrey's pioneering work in the 1980s (Humphrey, 1989). However, a European

survey of organisations engaged in SPI programmes during this period (Ibanez, 1996)

confirmed that the SPI models then available offered no panacea for RE problems.

Page 3: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Indeed, the organisations consulted identified requirements specification and the

management of customer requirements as the principal problem areas in software

development that they faced. Even enthusiastic adopters of SPI programmes found

that while SPI brought them significant benefits, their problems with handling

requirements remained stubbornly hard to solve.

The Software Engineering Institute's Capability Maturity Model for Software (SW-

CMM) (Paulk, 1993), which was becoming widely deployed at this time, touched on

RE practices but provided little specific guidance. To redress this, the Requirements

Engineering Adaptation and IMprovement for Safety and dependability (REAIMS)

project conducted the first systematic application of the principles of SPI specifically

for RE. This resulted in the publication of the Requirements Engineering Good

Practice Guide (REGPG) (Sommerville, 1997; Sawyer, 1999) in 1997.

This chapter reviews the state-of-the-art of process improvement for RE. It starts by

reviewing the background to process improvement in the software and systems

engineering industry. It then considers the nature of RE processes and the pressures

and trends that have merged in recent years. It argues that for socio-technical systems,

RE practice needs to be particularly strong. It then reviews three RE process

improvement methods and examines the extent to which they have been validated.

The chapter concludes by summarising the options open to organisations seeking to

systematically improve their RE processes.

SPI MODELS AND STANDARDS

Humphrey's pioneering work on Software Process Improvement (SPI) in the 1980s

(Humphrey, 1989) led to the development of the Capability Maturity Model for

Software (SW-CMM), developed at the Software Engineering Institute under the

Page 4: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

sponsorship of the US Department of Defence. Humphrey’s work reflected a

realization that the piecemeal adoption of better methods and tools would not deliver

the improvements in software quality increasingly demanded by customers. Rather,

the whole development life-cycle needed to be addressed in order to identify weak

areas and focus improvement efforts. From the customer’s perspective, SPI allows

software contractors to be assessed against a common model and provides a stimulus

for contractors to increase product quality and meet cost and delivery targets. From

the contractor’s perspective, SPI represents a strategic organizational tool for

containing costs and increasing market-share.

The SW-CMM does this by defining a generic model of the software development

process structured around 5 maturity levels. Process maturity represents the degree to

which a process is defined, managed, measured, controlled and effective (Paulk,

1993). The more mature a process, the more it is possible to accurately forecast and

meet targets for cost, time of delivery and product quality. As a process becomes

more mature, the range within which results fall is narrowed. The emphasis moves

from understanding the software process, through exerting control over the process, to

achieving on-going improvement.

The SW-CMM maturity levels range from level 1 (Initial) which is an ad hoc, risky,

process to level 5 (Optimizing) where a process is robust, adaptive and subject to

systematic tuning using experiential data (Fig. 1). Each maturity level has a focus that

is defined by a number of key process areas (KPAs). The KPAs effectively set

capability goals that should be met if the supporting key practices are adopted and

standardized. The set of key practices prescribed for each KPA define how the KPA

can be satisfied and, since SPI is concerned with organisational maturity,

institutionalised.

Page 5: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Leve l 1Initial

Leve l 2Repeatable

Level 4Managed

Level 5Optimizing

Leve l 3Defined

Figure 1. The 5-level SW-CMM process maturity model

SW-CMM level 2 (Repeatable) is interesting because its focus is on project

management and is concerned with gaining control of the process. It is also the one

level to explicitly address requirements engineering. It does this by defining

Requirements Management as a level 2 KPA (the others are Software project

planning, Software project tracking and oversight, Software subcontract management,

Software quality assurance, and Software configuration management).

Since achieving level 2 is the initial target of almost all SW-CMM-led improvement

programmes, the SW-CMM has had the effect of greatly raising awareness of

requirements management. It has stimulated the market for support tools and has

made it more widely (though still far from universally) practiced. In this respect,

therefore, it has been enormously beneficial. However, implementing requirements

management is hard (Fowler, 1998), in part because it exposes weaknesses elsewhere

in the requirements process. For example, one of the practices mandated for the

requirements management KPA is the allocation of requirements to software

subsystems. Arriving at the stage where a set of requirements are ready for allocation

(by successfully eliciting, validating, negotiating, and prioritizing them) is itself a

complex and error-prone process for which the SW-CMM provides no explicit help.

Page 6: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

The omission of detailed advice for the systematic improvement of requirements

processes is true of all SW-CMMs (including the SW-CMMI (SEI, 2002)), of the

draft ISO/IEC 15504 (Konrad, 1995; Garcia, 1997) international standard for software

process assessment and of the ISO9001-3 quality standard (Paulk, 1994). However,

help for requirements processes is not entirely lacking. There are software and system

engineering standards such as ESA PSS-05 (Mazza, 1994) which provide valuable

and explicit guidance on requirements practice. However, these do not address

systematic process improvement. They do not provide a method for assessing the

weak points in an RE process or map out a route for the incremental adoption of their

recommended practices. Few organizations can afford to make revolutionary changes

to their requirements processes so they need help in planning and controlling

evolutionary improvement so as to minimize the risk that inevitably accompanies

change. This is the great strength of SPI.

THE RE PROCESS

Perhaps the most orthodox model of the RE process is that represented by the

following three IEEE standards (it is interesting to note that RE process is defined

implicitly in terms of documents that are products of the process):

• IEEE std 1362-1998 Guide for Information Technology - System Definition -

Concept of Operations (ConOps) Document (IEEE, 1998a)

• IEEE std 1233-1998 Guide for Developing System Requirements Specifications

(IEEE, 1998b)

• IEEE std 830-1998 Recommended Practice for Software Requirements

Specifications (IEEE, 1998c)

This process begins with scoping a problem and, in very broad terms, the solution (the

concept of operations or ConOps for short). This is followed by a process in which

Page 7: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

requirements are elicited from customers, validated by developers and other technical

specialists, and constrained by factors associated with the system's environment. The

product of this is a System Requirements Specification document that defines the

requirements for the overall system. Following this, the system requirements are

allocated to components that will be configured in a system architecture that satisfies

the system requirements. As part of this, subsets of system requirements are allocated

to software components. For each software component, a further round of analysis is

performed in which a set of software requirements are derived from the allocated

system requirements. These are documented in a Software Requirements

Specification (SRS). This forms the definitive set of requirements that the component

must satisfy and is sufficiently detailed to allow development to commence.

This process has evolved to deal with large custom systems, comprising both

hardware and software, that are developed for single customers. These are typically

developed using a supply chain of sub-contractors responsible for delivering system

components and who are managed in a hierarchical relationship with, at its root, a

main contractor. Such projects still represent a substantial proportion of the economic

activity in the software industry, particularly since heterogeneous systems have

become increasingly software-intensive. Despite this, their relative significance has

declined during recent decades. This is not so much due to a decline in this sector of

the industry as increases elsewhere. These include the booming market for retail

software products and the development of the Internet as a medium for business and

entertainment.

As the industry has changed shape, the value of orthodox RE has been increasingly

called into question. Clearly, an RE process optimised for large heterogeneous

systems is unlikely to be easily applied to, for example, small short-duration projects.

Page 8: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Agile development methodologies, of which eXtreme Programming (XP) (Beck,

1999) is perhaps the best known, can be seen as a reaction against orthodox software

engineering and, by implication, orthodox RE.

Agile methodologies deviate from orthodox RE by eschewing detailed analysis,

modelling and documentation where a project might move straight from a concept of

operations-like activity to development. The promise of agile methodologies to be

lightweight and reactive is a seductive one. Although they are not entirely

incompatible with SPI (Paulk, 2001), they take what is essentially a programming

perspective on system development but for many domains, this is inappropriate. For

example, almost all large businesses are dependent on their computer systems, yet

programmers are seldom equipped to find business solutions. Many systems are

embedded within organisational contexts to support and manage business processes

that are enacted by people and software. These sociotechnical systems are typically

too critical to risk getting wrong, too sensitive to how the actors interact (which may

be dependent on, for example, tacit understanding between the actors) to be easily

understood, and too complex to make refactoring a viable option.

Sociotechnical systems need careful appraisal of change options. They need careful

analysis of the problem context and elicitation of user requirements. They need an

overall system specification that documents the validated customer and user

requirements and defines the new system in terms of its function, its quality attributes

and its context within its environment. They need responsibility for the satisfaction of

the system requirements to be carefully allocated to appropriate software and human

'components' (actors). And they need these components to be specified so that

development work, which increasingly takes the form of selecting and integrating off-

the-shelf components, can proceed. In these terms, most of the activities and products

Page 9: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

of orthodox RE processes still have a vital role to play in ensuring that systems are

developed that meet their users' real needs.

However, it would be naive to think that simply applying IEEE stds 1362, 1233 and

830 would guarantee success. There are many practices, techniques, methods and

tools that may be deployed to aid RE. Although they embody much accumulated

wisdom, RE standards can only set out the general principles or give detailed

guidance for particular activities or documents. They offer no aid for selecting

appropriate methods (for example) or for designing an RE process optimised for a

particular organisation.

Recognition of this problem was the motivation for the RE process improvement

work described in the next section.

PROCESS IMPROVEMENT FOR RE

Selection of the RE process improvement models for description below has been

based on the following criteria: they should include advice on RE practice, they

should present this advice within the framework of a maturity model, and they should

provide a method for assessing existing processes. This set of criteria omits other

valuable and pragmatic work (see below) that can be used for RE process

improvement. However, the criteria have the effect of isolating work designed

specifically to help organisations integrate RE process improvement within a general

SPI programme.

There are currently three RE process improvement models that meet the criteria; the

Requirements Engineering Good Practice Guide (REGPG), the University of

Hertfordshire model and the Requirements Engineering Process Maturity Model

(REPM). Others sources of improvement advice exist. In particular Young (2001) and

Page 10: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Wiegers (1999) are excellent textbooks based around sets of recommended RE

practices and include practical advice on how organisations can use them to improve

their RE processes. These are not reviewed further here simply because they do not

include a process maturity model or assessment method. However, they contain much

wisdom and practical advice for organisations who do not need or wish to undertake a

formal, calibrated improvement process.

The review of improvement models is mainly concerned with the improvement

framework and assessment rather than the validity of the practices that the models

recommend. All three models have derived the set of RE practices that they

recommend from a variety of sources including practices widely used in industry,

practices recommended or mandated in standards and practices learned from direct

experience. However, as Wiegers (1999) warns:

“… not all … have been endorsed as industry best practices …”

Nevertheless, the developers of each model have been careful not to recommend

practices that have not been validated in some form and shown to be practical and

practicable for practitioners.

The Requirements Engineering Good Practice Guide

The REGPG (Sommerville, 1997) was the first public-domain process improvement

and assessment model for RE. Like the SW-CMM, the REGPG uses an improvement

framework with several process maturity levels (Fig 2). The REGPG maturity levels

are design to mirror the SW-CMM levels. However, at the time of REGPG's design,

the adoption of RE practices across the industry was patchy (El Eman, 1995;

Forsgren, 1995) and there was insufficient empirical evidence for the existence of

requirements processes that could be characterised beyond Defined (level 3). The

Page 11: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

characteristics of highly mature RE processes were essentially unknown so the

REGPG mirrors only the SW-CMM's bottom three levels (initial, repeatable and

defined).

Level 1Initial

Level 2Repeatable

Level 4Managed

Le vel 5Optimizing

Level 3Defined

Figure 2. The 3-level REGPG process maturity model

In the REGPG model:

• Level 1 - Initial level organizations have an ad hoc requirements process. They

find it hard to estimate and control costs as, for example, many requirements have

to be reworked. The processes are not supported by planning and review

procedures or documentation standards. They are dependent on the skills and

experience of the individuals who enact the process.

• Level 2 - Repeatable level organizations have defined standards for requirements

documents which are more likely to be of a consistently high quality and to be

produced on schedule. They also have policies and procedures for requirements

management.

• Level 3 - Defined level organizations have a defined process model based on good

practices and defined methods. They have an active process improvement

Page 12: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

programme in place and can make objective assessments of the value of new

methods and techniques.

The key to improvement is in adopting appropriate good practices, in the right order,

at the right pace and with the required degree of strategic commitment. The REGPG

distils guidance on good practice adoption into 66 guidelines (key practices in SW-

CMM terms). The practices are classified according to whether they are Basic,

Intermediate or Advanced to reflect, for example, dependencies among the practices.

The practices are organised according to process activities. These are the

requirements document, requirements elicitation, analysis and negotiation, describing

requirements, system modelling, requirements validation, requirements management

and RE for critical systems. Although analogous to KPAs, REGPG process activities

serve only to classify good practices and form no part of the maturity framework.

Unlike the SW-CMM (but like the more recent SW-CMMI and ISO/IEC 15504), the

REGPG uses a continuous architecture. This means that the REGPG does not

prescribe the process activities to be targeted at each improvement stage. Instead

organisations can adopt practices across a range of process activities according to

their priorities.

Before improvement can be achieved, causality has to be established between the

evident problems and weaknesses in the organisation's RE process. Discovering

weaknesses and identifying priorities for targeted improvement is the role of process

assessment. Process assessment is based on a system of checklists designed to reveal

what practices are in use and the extent to which they are used. A checklist of the 66

REGPG guidelines is used as the starting point for the assessment. The activities used

in the assessment are:

Page 13: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

1 Prune checklist Assign a score of 0 (see below) to each practice that is

already known not to be used. This is simply a

pragmatic step for streamlining the assessment.

2 Select the right

people to interview

Knowledge of an organisation's RE practices may

reside with many people, particularly where there are

different units working on different products/projects.

This activity is tasked with selecting a small group of

people in order to gain a representative snapshot of RE

practice within the organisation.

3 Score process

against checklist

The purpose of this stage is to assign scores against

each practice for which a score can be confidently

assigned and to flag those practices where the extent of

adoption is uncertain.

4 Resolve uncertainty This stage focuses on seeking additional information

(perhaps by interviewing other people, looking for

documentary evidence, etc.) that allows a score to be

assigned to the uncertain practices identified at stage 3.

5 Compute maturity This stage derives an overall maturity score based on

the results of stages 1, 3 and 4.

During the assessment process, each good practice is assessed as being:

• Standardised (score 3). The practice has a documented standard in the

organisation and is integrated into a quality management process.

Page 14: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

• Normal use (score 2). The practice is widely followed in the organisation but is

not mandatory.

• Discretionary (score 1). Some project managers may have introduced the practice

but it is not universally used.

• Never (score 0). The practice is never or rarely applied.

The maturity level is calculated by summing the numerical scores for each practice

used:

• Level 1 (initial) processes score less than 55 in the basic guidelines.

• Level 2 (repeatable) processes score over 55 in the basic guidelines but less than

40 in the intermediate and advanced guidelines.

• Level 3 (defined) processes score over 85 in the basic guidelines and more than 40

in the intermediate and advanced guidelines.

The end result is an indicative maturity level for the organisation's RE process, and a

map of whether, and to what extent, each of the 66 RE practices described in the

REGPG is used in the RE process. Subsequent improvement is based on identifying

un- or under-used practices that:

• are likely to yield benefits that outweigh the cost of their introduction;

• can be feasibly introduced given dependencies on underpinning practices.

The practices' classifications (basic, intermediate, advanced) give some help in

selecting an appropriate subset for introduction. The classifications are supplemented

with additional qualitative information in the good practice guidelines about benefits,

how to implement them, and associated costs and problems. At the end of an REGPG

process assessment, therefore, an organisation should have the basis for an RE process

improvement agenda in terms of areas of weakness and options for improvement.

Page 15: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

The two more recent models described below both share broadly similar aims and

comprise a similar maturity framework.

The Requirements Engineering Process Improvement Model

The Requirements Engineering Process Maturity Model (REPM) (Gorschek, 2003) is

targeted at SMEs which, its authors observe, lack the resources to apply models like

the REGPG. Instead, REPM claims to take an approach to the RE process that is

sufficient to:

“… give an indication of whether a problem exists, and … where the problem areas

reside.” (Gorschek, 2003)

REPM uses a six-level maturity model. It actually defines five but since it is staged

model and level one has eleven actions (analogous to SW-CMM key practices or

REGPG practices) associated with it, there is an implicit level zero that broadly

corresponds to the SW-CMM's initial level. Each action is associated with one of the

levels one to five and, additionally, is classified according to three process activities

(called, in REPM, main process areas or MPAs): Elicitation, Analysis and

Negotiation, and Management. Each maturity level has a set of actions defined for it

under each of the three MPAs. Despite their superficial similarity, MPAs are therefore

unlike SW-CMM KPAs which are only associated with a single maturity level.

However, MPAs are also different to process activities in the REGPG where it is not

obligatory to implement practices for each process activity in order to achieve a given

maturity level.

MPAs may be further classified into sub-process areas (SPAs). For example, the

Elicitation MPA comprises Stakeholder identification, Stakeholder consulting,

Domain knowledge and Scenario elicitation. The authors argue that this allows the

Page 16: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

model to evolve easily since a composite action may be promoted to an SPA allowing

new actions to be derived from it.

REPM is designed for project rather than organisational assessment. Like the

REGPG, it uses checklist-driven interviews of key personnel in order to make an

assessment. Because an REPM assessment is scoped at the project level, selection of

personnel to interview defaults to the person responsible for RE in the project under

assessment. This obviates the need to select a range of people across an organisation

(c.f. step 1 in the REGPG assessment model) and minimises the interaction between

assessor and practitioners.

The project evaluation checklist is, like that of the REGPG, a list of all 60 actions.

REPM recognises that failing to implement an action does not necessarily fatally

weaken the RE process, provided there is a sound rationale for not doing so.

Producing a user manual draft (a level 2 action) may be meaningless for the

development of an embedded system, for example. This allows checklist actions to be

marked satisfied-explained. Actions in this category carry the same weight as

completed actions.

As with the REGPG, the authors of REPM have selected a set of RE practices (actions

in REPM terms) and made a judgement about how fundamental they are to an RE

process. Because REPM uses a staged model, each action is locked in to a particular

maturity level. To achieve a maturity level n, a process needs to have completed or

satisfied-explained all of the actions associated with levels one through n.

The University of Hertfordshire Model

The RE process improvement model (Beecham, 2003) developed at the University of

Hertfordshire is a direct adaptation of the SW-CMM framework for RE processes.

Page 17: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

This is intended to help dovetail RE process improvement with a wider SW-CMM-

based software process improvement programme.

The Hertfordshire model uses the 5 SW-CMM maturity levels (Fig 1) to classify RE

processes. Since it is based upon the SW-CMM it uses a staged architecture in which

a set of 68 practices (called processes) are mandated for each maturity level. Like the

REGPG and REPM, these are classified according to RE process activities, called

phases. These are management, elicitation, analysis and negotiation, documentation

and validation. Like REPM MPAs, each phase in the Hertfordshire model defines a

set of processes at each maturity level.

As with the other RE process improvement models reviewed here, the Hertfordshire

model draws its set of processes from analysis of RE practice in industry (including

(Hall, 2002)). However, to help integrate the model with the SW-CMM, some

processes are drawn directly from the SW-CMM. For example, the Hertfordshire level

2 process P1 below is essentially the same as the SW-CMM level 2 requirements

management KPA commitment to perform key practice.

P1. Follow a written organisational policy for managing the system requirements

allocated to the software project.

Process assessment is broadly similar to that of the REGPG even to the extent that, as

currently defined in the available documentation, it reflects a continuous rather than a

staged improvement model. In order to assess the extent to which a process is

satisfied by an organisation, the Hertfordshire model allocates a score to each process

against three assessment criteria. These are:

• Approach. A measure of the organisational commitment and capability.

• Deployment. A measure of the extent to which a process is implemented across

the organisation.

Page 18: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

• Results. A measure of the success of a process' implementation.

• Each process is assessed against each of these criteria and given a rating of:

− Outstanding (10)

− Qualified (8)

− Marginally qualified (6)

− Fair (4)

− Weak (2)

− Poor (score 0)

The average of the score for approach, deployment and results is recorded for each

process. Hence, if a process was marginally qualified in terms of approach, fair in

terms of deployment and weak in terms of results it would rate an average score of

fair (4). The scores are then summed for each phase (e.g. to assess the strength of

requirements management in an organisation) and the sum of all five phases yields an

overall score.

The Hertfordshire model is still under development. At the time of writing, the model

has not been calibrated to map overall scores onto maturity levels and the relationship

of this to the staged model implied by the association of processes with maturity

levels has not been worked through. It has undergone an initial validation phase that

has focused on the overall framework and the processes and process descriptions used

for maturity level 2. The set of processes and process descriptions for levels 3 to 5

currently exist in draft form.

Model Maturity levels CMM Key Process Area analogues?

CMM Key Practice analogues?

Staged or Continuous?

REGPG 3 Process activities but no direct

Good practice guidelines

Continuous

Page 19: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

analogy REPM 6 (1-5 with an

implicit level 0) Main process areas but no direct analogy

Actions Staged

Hertfordshire model

5 Phases but no direct analogy

Processes Staged but assessment is currently continuous

Table 1. Summary of RE Improvement Models

EXPERIENCE OF RE PROCESS IMPROVEMENT

Process improvement is a costly activity that requires substantial investment.

Organisations therefore require confidence in the improvement models that they use.

This is not simply confidence that the set of practices that a model embodies are

proven in industry. Confidence is required that the maturity framework and

assessment method will not caricature the organisation's processes. Organisations

need to know that the process weaknesses revealed by an assessment are real

weaknesses that inhibit process performance. They also need to know that the

remedial action proposed by process analysts will address real organisational

priorities in a cost-effective way. Validation is important for establishing confidence

in improvement models and each of the models reviewed in the previous section have

undergone some form of validation.

REGPG

Over 10000 copies of the REGPG have been sold and, as the longest-established and

most widely-disseminated model, it should be unsurprising that the REGPG has

undergone the most extensive validation. It has been the subject of two projects

Page 20: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

specifically intended to validate and improve the model: Qure (Kauppinen, 2002) and

Impression (Sommerville, 2003).

The Qure project conducted by the Helsinki University of Technology included an RE

process improvement theme. The REGPG was evaluated as part of this in which the

model was piloted in four companies working in a range of application domains. The

scope of the analysis was to discover whether the REGPG process analysis method

was accurate and usable. The question of whether a subsequent improvement cycle

based upon the results of the analysis resulted in real improvement was outside the

project's scope.

Qure found that, in the case studies used, the REGPG yielded useful results and was

appropriate for organisations beginning an RE process improvement programme. A

key weakness discovered was that the selection of good practices to address revealed

process weaknesses required more guidance and that, as a result, companies tended to

be over-ambitious in the improvement programmes that they undertook.

This general point was echoed in the results of the Impression project which

concluded that the assessment method was too passive. The scope of the REGPG

validation in Impression was wider than that of Qure. It involved nine companies

from a variety of domains and ran for a full improvement cycle from initial

assessment through good practice introduction to follow-up assessment. Like Qure,

assessment and improvement piloting was performed by a third party organisation.

However, REGPG authors acted as trainers and consultants for the staff performing

the process assessments.

The passivity of the assessment method emerged when, perhaps unsurprisingly, it

became apparent that the companies attached greater priority to actual improvement

than mere diagnostics and maturity classification. The assessment method was

Page 21: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

modified to include an explicit step for selecting good practices to address the

revealed weaknesses. This was backed up with the development of a decision support

tool that processed the analysis data and listed those best practices likely to provide

solutions most cost-effectively.

Other significant results included:

• Implicit dependencies between practices (e.g. that intermediate practices cannot be

introduced until appropriate basic practices have been deployed) were sometimes

wrong. This implies greater than anticipated freedom of choice in the selection of

practices for introduction and is something that would be hard to accommodate in

a staged architecture.

• The maturity model was not entirely generic and different norms of practice that

derive from different priorities across application domains were not accurately

reflected. The REGPG evolved from the experiences of the REAIMS project's

industrial partners who worked in dependable systems domains. The REGPG

inevitably retains some of this flavour which fits slightly awkwardly with, for

example, socio-technical systems.

At the end of the Impression assessment-improvement cycle a follow-up assessment

was performed. This established that, in terms of the REGPG maturity model, all but

one of the nine companies had improved their RE processes. Four had moved from

Initial to Repeatable. The remaining four had all improved their score but remained at

the Initial level. The strategies that companies used to improve ranged from

concentrating on further embedding existing practices within company standard

procedures to introducing large numbers of new practices (one company introduced

14).

Page 22: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Although Impression covered a whole improvement cycle and appears to demonstrate

the REGPG's utility for improving RE processes, there was insufficient time to follow

this through and establish whether increased process maturity in REGPG terms was

reflected in concrete business benefits. The signs are good but unproven.

REPM

(Gorschek, 2003) describes the evaluation of REPM using a pilot study involving four

companies. These were SMEs of the size for which REPM was designed. The scope

of this pilot was more restricted than that of Qure or Impression and had a dual aim of

using REPM to discover norms of RE practice (which we ignore here) and of

validating REPM.

Despite the limited scope of the REPM validation, the results indicate that its

lightweight assessment method yields useful results. REPM as a vehicle for

improvement has not been demonstrated but within the limited aims of REPM, the

model appears to show promise as an assessment tool for SMEs. REPM was explicitly

designed to be lightweight and the authors report no serious problems with applying

it. However, validation of the model's applicability does not appear to have been an

explicit goal of the evaluation.

University of Hertfordshire Model

The methodology for validation of the Hertfordshire model differs from that used for

the REGPG and REPM. The model's development was in part influenced by a study

of RE practice in twelve companies (Hall, 2002). Validation of the model itself,

however, has to date been performed by assessment by a panel of experts rather than

Page 23: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

deployment in practitioner companies. At the time of writing the results of this

exercise were being processed and will feed into maturation of the model.

CONCLUSIONS

Given the level of interest in SPI in recent years, RE process improvement has

received surprisingly little attention from the SPI and RE research communities or

from industry. This may be because RE process improvement takes place, but takes

place adequately without the formal framework of a CMM-like maturity model.

Nevertheless, RE process improvement is an important issue as surveys such as that

reported by Ibanez (1996), clearly demonstrate.

Organisations interested in RE process improvement and who, perhaps because of

investment in wider SPI programmes, wish to use a maturity level framework now

have a choice of at least the three improvement models reviewed here.

Of these, the REGPG has the benefit of being widely known. It has been validated

across a range of domains and it's strengths and weaknesses have been identified. The

REPM and the Hertfordshire models currently occupy more specialised niches.

REPM has been purposely designed as a lightweight method suitable for SMEs. It is

perhaps the model most in tune with the current mood for agile and lightweight

methodologies although it has not yet completed a full validation cycle.

The Hertfordshire model is a work in progress that is carefully aimed at the many

companies who have already invested in SW-CMM based improvement programmes.

If subsequent development can complete the model and integrate it within the SW-

CMM framework (and track CMM developments), then it has the potential to offer

substantial benefits to companies.

Page 24: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

REFERENCES

Beck, K. (1999). Extreme Programming Explained: Embrace Change. Boston:

Addison-Wesley Longman.

Beecham, S., Hall, T. & Rainer, A. (2003). Defining a Requirements Process

Improvement Model. Technical Report TR379. University of Hertfordshire.

El Eman, K. & Madhavji, N. (1995). A Field Study of Requirements Engineering

Practices in Information Systems Development. Proc. 2nd IEEE International

Symposium on Requirements Engineering (RE95), York, UK. 68-80.

Forsgren, P. & Rahkonen, T. (1995). Specification of Customer and User

Requirements in Industrial Control System Procurement Projects. Proc. 2nd IEEE

International Symposium on Requirements Engineering (RE95), York, UK. 81-88.

Fowler, P., Patrick, M., Carleton, A. & Merrin, B. (1988). Transition Packages: An

Experiment in the Introduction of Requirements Management. Proc. Third

International Conference on Requirements Engineering (ICRE’98), Colorado Springs,

Colorado. 138-147.

Garcia, S. (1997). Evolving Improvement Paradigms: Capability Maturity Models &

ISO/IEC 15504 (PDTR). Software Process: Improvement and Practice, 3(1), 47-58.

Gorscheck, T., Svahnberg, M. & Kaarina, T. (2003). Introduction and Application of

a Lightweight Requirements Engineering Process Evaluation Method. Proc.

Requirements Engineering Foundations for Software Quality '03 (REFSQ'03),

Klagenfurt/Velden, Austria. 83-92.

Hall, T., Beecham, S. & Rainer, A. (2002). Requirements problems in twelve software

companies: an empirical analysis, IEE Proceedings-Software, 149 (5), 153-160.

Humphrey, W. (1989). Managing the Software Process. Boston: Addison-Wesley

Longman.

Page 25: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Ibanez, M. & Rempp, H. (1996). European Survey Analysis. ESSI Project technical

Report. European Software Institute.

IEEE Std 1233-1998. (1998). IEEE Guide for Developing System Requirements

Specifications. New York: IEEE.

IEEE Std 1362-1998. (1998). IEEE Guide for Information Technology - System

Definition - Concept of Operations (ConOps) Document. New York: IEEE.

IEEE Std 830-1998. (1998). IEEE Recommended Practice for Software

Requirements. New York: IEEE.

Kauppinen, M., Aaltio, T., & Kujala, S. (2002). Lessons Learned from Applying the

Requirements Engineering Good Practice Guide for Process Improvement. Proc.

Seventh European Conference on Software Quality (QC2002), Helsinki. 73-81.

Konrad, M. & Paulk, M. (1995). An Overview of SPICE's Model for Process

Management. Proc. Fifth International Conference on Software Quality, Austin,

Texas. 291-301.

Mazza, C., Fairclough, J., Melton, B., De Pablo, D., Scheffer, A., Stevens, R., Jones,

M. & Alvisi, G. (1994). Software Engineering Guides. London: Prentice Hall.

Paulk, M. (1994). A Comparison of ISO 9001 and the Capability Maturity Model for

Software, CMU/SEI-94-TR-12, Software Engineering Institute.

Paulk, M. (2001) Extreme Programming from a CMM Perspective. IEEE Software,

18 (6), 19-26.

Paulk, M., Curtis, W., Chrissis, M. & Weber, C. (1993). Capability Maturity Model

for Software, Version 1.1. CMU/SEI-93-TR-24. Software Engineering Institute.

Sawyer, P., Sommerville, I. & Viller, S. (1999). Capturing the Benefits of

Requirements Engineering. IEEE Software, 16 (2), 78-85.

Page 26: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

SEI. (2002). CMMI for Software Engineering, Version 1.1, Continuous

Representation (CMMI-SW, V1.1, Continuous). CMU/SEI-2002-TR-028. Software

Engineering Institute.

Sommerville, I. & Ransom, J. (2003). An Industrial Experiment in Requirements

Engineering Process Assessment and Improvement. Technical report, Computing

Department, Lancaster University.

Sommerville, I. & Sawyer, P. (1997). Requirements Engineering - A Good Practice

Guide. Chichester: John Wiley.

Wiegers, K. (1999). Software Requirements, Redmond: Microsoft Press.

Young, R. (2001). Effective Requirements Practices, Boston: Addison-Wesley

Longman.

Page 27: Maturing Requirements Engineering Process Maturity Models · Maturing Requirements Engineering Process Maturity Models ABSTRACT The interest in Software Process Improvement (SPI)

Biography

Pete Sawyer holds a BSc and PhD from Lancaster University where he is a senior

lecturer in the Computing Department. His research interests are in the general area of

Software and Systems Engineering. In particular, he is interested in requirements

engineering, software process improvement and applications of natural language

processing to the systems engineering process. He is a member of the British

Computer Society (BCS), an affiliate member of the Institution of Electrical and

Electronics Engineers (IEEE) and a professional member of the Association of

Computing Machinery (ACM). He is also a member of the executive committee of the

BCS Requirements Engineering Specialist Group (RESG). He has over fifty peer-

reviewed publications and is the co-author (with Ian Sommerville) of Requirements

Engineering - A Good Practice Guide (John Wiley, 1997).