Project Portfolio Delivery Effectiveness Feedback Example Report

Project portfolio delivery effectiveness survey

ACME feedback report

Tim Pare, John Hall, John Kirkwood

January 17, 2009

© PA Knowledge Limited 2009. PPDE Survey - ACME Results

Foreword

In organizations that deliver a portfolio of many projects, there is perpetual pressure to do it better. This might be because their customers want to focus effort on more of the ‘right things’, do more with the same (or less) or improve speed and reliability of delivery.

It is clear that there are many drivers that can contribute to, or impede, overall delivery performance, spanning the fields of governance, portfolio, project, financial, workforce, resource and financial management.

Many emerging maturity models are predicated on the assumption that ‘best practice’ is an absolute, and if sufficient of the many prescribed practices are documented and complied with, an organization’s outcomes will be better. While this might be true, it is rather undiscriminating and might imply a huge and expensive challenge for an organization to contemplate.

We have observed that there usually isn't much guidance on the prioritization of prescribed practices and usually no direct matching on how those practices can help deliver an organization's specific improvement goals. Equally, the focus of the practices is generally on the ‘harder’ elements (organization, process and standards), with insufficient emphasis on the ‘softer’, capability aspects (the really difficult stuff – insightful intelligence, strong decision making, political influence and personal credibility).

We contend that a more succinct definition of best practice in this arena (as others) would be a set of practices that are ‘fit for purpose’. There are a lot of things an organisation could do; the real question is what they should do.

This report provides the results of ACME’s responses to our diagnostic and contrasts them with findings of other respondent organizations (anonymized for reasons of confidentiality) and wider research into those characteristics and key elements of practice that have been observed to contribute to tangible success in a variety of circumstances, in multiple industries and geographies.


Contents

1. Research demonstrates a clear correlation between maturity and overall effectiveness – with some caveats 4

2. Overall, ACME is towards the lower quartile of the maturity spectrum compared to other companies surveyed 5

3. ACME overall delivery effectiveness appears to offer scope for improvement around some key outcomes 6

4. The most successful organizations demonstrate maturity across all key areas of practice 7

5. ACME’s practice maturity assessment suggests a framework for prioritization 8

6. Detailed analysis indicates specific areas for improvement 9

7. These may suggest an adaptation to ACME’s own assessment of improvement priorities 10

8. Diagnostic methodology 11


Research demonstrates a clear correlation between maturity and overall effectiveness – with some caveatsAn earlier PA survey into portfolio management practices and their resulting effectiveness confirmed that firms which systematically plan for success are more likely to achieve it. In other words, the more an organization puts in to managing its portfolio of projects, and the means of delivery, the more its gets out. While this was to be expected, there were a number of surprising findings:

Outliers, very large organizations with apparently mature processes, reported unsatisfactory outcomes

Similarly, outliers (generally smaller organizations) with ad-hoc processes, reported more favourable outcomes

In addition, while one might expect that the nature and scale of an investment programme (ie what is at stake) should drive the need for more mature processes, the survey indicated that this was not always the case. Certainly the most developed practitioners had substantial project portfolios, however, the two organizations with the largest portfolios had the least effective approaches.

Clearly certain organizational characteristics, relating to size, complexity and culture, are also significant factors.

This suggested, to us, the following categorization:

Entrepreneurial – Success characterized by the extraordinary efforts of talented individuals applying ad-hoc processes and tools – generally in smaller, more ‘agile’ organizations

Planned – Success was planned for by developing and inserting the right processes, technologies, and people.

Informal – Ad-hoc approach to portfolio development and management has not resulted in effective delivery.

Bureaucratic – Significant investment in standards, tools and processes is not supported by the necessary behaviours to deliver success.

Note:

Maturity is a measure of the quality of practices

Effectiveness is a measure of the quality of outcomes

Additional information is provided in the section on diagnostic methodology

Source: PA Portfolio Management Survey

Bureaucratic

Entrepreneurial

Maturity vs Effectiveness

Ma

turi

tyEffectiveness

Planned

Informal

Most effective

Least effective


Overall, ACME is towards the lower quartile of the maturity spectrum compared to other companies surveyed

ACME has a medium sized portfolio of large in-house projects in support of the business and its customer base.

Approximate portfolio profile:

~30 candidate projects per year (75% initiated)

Average value $2 MM

A mix of product development, IT, business change and asset acquisition projects

Serving multiple business units

Planning is geographically dispersed / decentralized.

ACME’s responses indicate that they are in the lower half of the maturity spectrum.

The current survey in which ACME (and other organizations) have participated is more expansive than the original survey; the range of practices has been expanded beyond the confines of portfolio management into broader aspects of delivery.

The results from both studies are consistent in the correlation of maturity and effectiveness - those companies that demonstrate more mature practices observe better outcomes.

Source: PA Portfolio Delivery Effectiveness Survey, 2009

Maturity vs Effectiveness

0%

50%

100%

0% 50% 100%

Effectiveness

Matu

rity

ACME


ACME overall delivery effectiveness appears to offer scope for improvement around some key outcomes…


Dimension Implications

Supply capacity – understanding what resources you have

Capability and capacity to deliver projects and business-as-usual is not well understood. Operational issues frequently disrupt change activities.

Demand estimating – understanding what you need

Business cases are unreliable, with poor estimating of project and business-as-usual demand, and little understanding of expected benefits.

Allocation – matching demand and supply

It is difficult to balance resources between business-as-usual and project work; resources are frequently overcommitted and often unilaterally redeployed.

Utilization – understanding what you use

There is perpetually a shortage of skilled people; significant recourse to contract staff for operational and change activities.

Portfolio adaptability – tracking and adapting the portfolio to current needs

It is not clear how many projects are on the go, how they are tracking or what effort they absorb. Ability to adjust to changing priorities is limited.

Delivery reliability – what you achieve

There is a tendency to start everything, but not deliver it all.

Accountability – taking ownership of delivery

Accountability for delivery is unclear. It’s accepted that projects frequently run late or deliver short of original intentions.

Stakeholder satisfaction – how well your delivery is perceived

Stakeholders have low expectations of delivery. Little or no delivery metrics captured or available.

Process – achieving efficiency/effectiveness in delivery

Sequential development, siloed working, protract and sub-optimize delivery, and impede multi-disciplinary efforts.

The implications of shortcomings in other areas are potentially significant

Portfolio balance appears to be in good order and is the best rated outcome today. Opportunities for improvement exist in all other areas.

Outcomes

Portfolio outcomes

Project outcomes

Resourcing outcomes


The most successful organizations demonstrate maturity across all key areas of practice

In comparing the performance of the most and least developed (mature) practitioners in the original survey, the best performers rated themselves consistently across the range of practices; in other words, they did everything reasonably well.

In contrast, the least mature rated themselves poorly in a number of fundamental areas. This indicates a number of significant discontinuities in what is essentially an end-to-end process – or perhaps a highly integrated meta-process – but which is not treated as such. Hence, overall effectiveness is vulnerable to the weakest link in the chain.

This is illustrated in the diagram.

The ‘most developed’ firms exhibited capability in all dimensions and were most differentiated by the: Strength of linkages between projects and strategies

(portfolio management) Quality of their business cases (the front end of project

management) Proactive considerations of project constraints (resource and

workforce management).

Some features initially thought to be essential appear less so: Use of support systems (tools) to facilitate the process

was found to be necessary, but not sufficient to improve effectiveness at most levels of maturity

Similarly, the presence of project management standards does not directly correlate with delivery effectiveness

The formality of the process and the prioritization technique applied were not found to significantly impact the effectiveness of project delivery.

At face value, some of these seem to be contradictory findings. However, at early stages of the maturity curve, technology may be largely irrelevant – more important are defining the strategies to guide project identification and the quality of project definition. At the mid point of the maturity curve where standardization and consistency is required, often technology takes the place of solid governance and often distracts the focus of work towards checklist completion, rather than excellent delivery. At the mature end of the curve, technology is a critical enabler for larger scale, complex environments (ie with multiple projects, or distributed teams and geographic dispersion of activities).

The issue of project management standards is interesting in that, while it is not possible to have highly effective processes without such standards, having them is no guarantee of delivery effectiveness. More important are the values, attitudes and behaviours of those responsible for delivery.

Source: PA Portfolio Management Survey

Strongly agree

Ambivalent

Strongly disagree

Busines

s obje

ctiv

es c

lear

Strong li

nks b

etwee

n

object

ives

and s

elec

tion

Quality

busi

ness

case

s

Object

ive

and c

redib

le

sele

ctio

n pro

cess

Scenar

ios

rapid

ly e

valu

ated

Consider

all

signifi

cant

const

rain

ts

Flexi

ble a

pproac

h

resi

lient t

o chan

ge

Busines

s im

pact o

f

portfolio

under

stood

Effici

ent i

n use

of

man

agem

ent t

ime

Most effectiveLeast effective


0%

10%

20%

30%

40%

50%

60%

70%

80%

Governancepractices

PortfolioManagement

practices

Project Managementpractices

Adaptive Deliverypractices

WorkforceManagement

practices

ResourceManagement

practices

FinancialManagement

practices

SIMON OTTLEY - British American Tobacco 90th percentile 75th percentile 25th percentile 10th percentile

ACME’s practice maturity assessment suggests a framework for prioritization

As overall effectiveness is vulnerable to discontinuities in the delivery chain, the first priority must be to improve those areas that impede delivery.

ACME’s results suggest that, from a practice maturity perspective, priority should be given to aspects of workforce and resource management – these are the areas of greatest “internal” discontinuity, and also represent the most significant variances with higher performing organizations.

The results also suggest that enhancements in portfolio and project management, workforce management (the strategic aspects of resourcing) also merit attention.

Elements of adaptive delivery and financial management are relative strengths. Improvements in these areas are least important and quite likely of limited value at this time.

.


ACME


Practice Score

1 Governance practices 50%

2 Decision-making and oversight 75%

3 Clear delivery accountabilities 25%

4 Gated decision process 50%

5 Design and architecture blueprint enabled 75%

6 Design assurance 50%

7 Management information/Dashboards 25%

8 Portfolio Management practices 42%

9 Strategic Alignment 75%

10 Prioritization 25%

11 Portfolio monitoring & refinement 25%

12 Project Management practices 42%

13 Business cases & benefits models 50%

14 Program management 25%

15 Project management Approach 25%

16 Project management Controls 75%

17 Project/program assurance 75%

18 Continuous learning/ knowledge management 0%

19 Adaptive Delivery practices 56%

20 Leveraging standardized models 50%

21 Routed design paths 75%

22 Traceability management 50%

23 Collaborative toolset 50%

24 Workforce Management practices 15%

25 Capability development 0%

26 Staff sourcing strategy 0%

27 Career paths 0%

28 Performance management 25%

29 Learning & development (L&D) 50%

30 Resource Management practices 10%

31 Capacity planning 25%

32 Resource pools & centers of excellence 0%

33 Scheduling & allocating 0%

34 Time recording 0%

35 Resource forecasting 25%

36 Financial Management practices 58%

37 Budgeting 75%

38 Reallocation/ target revision 50%

39 Financial tracking and forecasting 50%

Practice Maturity

0% 20% 40% 60% 80% 100%

Detailed analysis indicates specific areas for potential improvementThe resourcing challenge is the most severe and covers the strategic, tactical and operational aspects of resourcing (from capacity planning and acquiring staff and external resources, their organization, scheduling and allocation).

The portfolio management challenge covers initial prioritization and ongoing adaptation of the portfolio, as well as enablers such as the management information/dashboards to guide decision-making and clarity around accountability of each aspect of delivery.

Those with a score less than this represent the greatest near-term improvement opportunities

The portfolio management

challenge

The project management

challenge

The broader resourcing challenge


These may suggest an adaptation to ACME’s own assessment of improvement priorities

Objective Description Priority

Delivery capacity

Enhancing the capacity to deliver more projects, without increasing cost

1=

Delivery quality

Ensuring fitness for purpose and attaining a better result every time

1=

Delivery speed

Delivering results faster, more efficiently and hence at lower cost

1=

Benefits realization

Achieving tangible business results regularly and reliably throughout the planning horizon

1=

ACME’s own assessment of improvement priorities Alignment of suggested improvements

Resourcing enhancements (from strategic workforce aspects to operational scheduling and allocation) would raise delivery capacity and enable projects to be adequately staffed for delivery.

Improving ACME’s portfolio management (initial prioritization/selection and ongoing adaptation of the portfolio during the year) would improve the benefits potential of the portfolio itself – i.e. choosing better candidates.

Real benefits realization requires credible business cases followed by active planning for, tracking and holding to account those responsible for their achievement.

Addressing the program and project management challenges, (including such as knowledge management), together with the application of sufficient, capable staff to projects, would contribute to achieving these objectives.

In addition, consideration of those aspects that are often covered by a “design authority” (aspects of architecture, the design/development process, design assurance, as well as enablers such as the collaborative toolset) may also be warranted to achieve project efficiency and quality objectives.


Diagnostic methodology

The diagnostic

The diagnostic measures respondents’ performance against key elements of practice that have been observed to contribute to tangible success in organizations in a variety of industries and geographies.

Questionnaire examines three areas:

– Objectives: What you want

– Practices: What you do

– Outcomes: What you get

Practices and Outcomes are self-assessed against a 5-point scale 1 (poor/immature) through 5 (excellent/world class)

Objectives are self-assessed against a similar scale 1 (needs minor improvement) through 5 (needs major improvement)

This report

The report is intended to highlight areas for a respondent company to focus on to improve overall portfolio delivery performance

Practice and outcomes responses are consolidated into single Maturity and Effectiveness scores to compare relative overall performance of all respondents

The respondent’s practice scores are contrasted with the 90th, 75th, 25th and 10th percentiles of all survey respondents to highlight key areas of discontinuity driving differences in performance

The chart used to identify specific areas for improvement are predicted on the hypothesis is that major discontinuities in the maturity of a given practice represent the most serious bottlenecks to performance improvements – without addressing these first, any efforts in other areas are unlikely to improve outcomes appreciably.

This survey is intended as an initial, rapid diagnostic, based on the perceptions of respondents, to identify areas for potential improvement. The findings need to be followed up by substantive analysis to validate the findings and prioritize specific improvement initiatives.


Contact details for further information concerning this report

For more information, contact:

John Kirkwood One Memorial DriveCambridge, MA 02142Tel: +1 571 227 9351Mobile: +1 202 320 0961E-mail: [email protected]

Tim Pare4601 N Fairfax DriveSuite 600 Arlington, VA 22203Tel: +1 571 227 9273Mobile: +1 571 215 1332E-mail: [email protected]

John Hall 4601 N Fairfax DriveSuite 600 Arlington, VA 22203Tel: +1 571 227 9351Mobile: +1 202 320 0961E-mail: [email protected]

Are you interested to know how your organization’s project delivery performance compares with your peers?

By participating in this research survey, you will have access to a complimentary benchmark report – like this one - that provides insight to how your organization compares to other respondents and areas to focus on.

Your organization will be one of more than 90 in the US and Europe that have begun to use the report to make improvements on their performance. Here are some respondents’ comments on their reports:

“The survey clearly showed areas where our approach works well, but also gave specific areas for focus in improvement. The report is extremely helpful in reporting the PMO's progress as

well as a tool for recommending improvement areas for the executive management.”

“This is unbelievably on the money for this group! As we’ve talked with internal and external stakeholders, these themes have consistently come up as major problems that they are facing.”

Not a bad return for 20 minutes of your time?

Please feel free to access the survey at: www.portfoliodiagnostic.com

Project Portfolio Delivery Effectiveness Feedback Example Report

Business

Transcript of Project Portfolio Delivery Effectiveness Feedback Example Report